Iou calculation device, region detection device, image processing device, and program

The method of dividing polygons into convex hulls and generating inner pixel maps for IoU calculations addresses the challenge of polygon IoU calculation, enabling precise and efficient neural network training for polygon detection.

JP2024157849APending Publication Date: 2024-11-08NIPPON HOSO KYOKAI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023072461
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing technologies lack a method for accurately calculating the Intersection over Union (IoU) for polygons, which is crucial for training neural networks to detect arbitrary polygon-shaped objects, as they are limited to rectangular areas and face challenges with rotated rectangles and complex shapes, leading to inaccuracies and high computational costs.

Method used

A method involving a convex hull division unit that divides polygons into convex hull polygons, generating inner pixel maps to calculate IoU values, allowing for continuous and differentiable error calculations between arbitrary polygons, eliminating restrictions on the number of vertices and shape complexity.

Benefits of technology

Enables accurate and efficient IoU calculations for arbitrary polygons, facilitating the training of neural networks to detect regions with high precision and flexibility, overcoming limitations of previous methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157849000001_ABST
    Figure 2024157849000001_ABST
Patent Text Reader

Abstract

To provide an IoU calculation device that can calculate an IoU (Intersection over Union) for two arbitrary polygons so as to enable machine learning of a model.SOLUTION: A convex hull division unit divides one polygon in a given image to obtain one or more convex hull polygons after the division. A convex hull inner pixel map generation unit generates inner pixel maps (IPMs) for the convex hull polygons after the division. A polygon inner pixel map generation unit generates an IPM for the polygon before the division by combining the IPMs for the convex hull polygons. An IoU approximation value calculation unit obtains an approximation value of the area of an intersection set region and an approximation value of the area of a union set region based on the IPM for each of two given polygons, and calculates an approximation value of an IoU value by dividing the former by the latter. This calculation process is performed by calculation within a range of a differentiable function, a maximum value function, and a minimum value function.SELECTED DRAWING: Figure 10
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an IoU calculation device, an area detection device, an image processing device, and a program. [Background technology]

[0002] There is an existing technique for detecting a specific region in an image using a machine learning technique. In the existing technique, a large amount of pairs of a learning image and a correct region contained in the image are first prepared as learning data. The correct region is a region having a specific characteristic. For example, the correct region is a region that includes a specific object or the like. Then, the learning data is used to train a neural network. In the training, the parameters inside the neural network are adjusted so that the error between the region estimated by the neural network at that time and the correct region is reduced.

[0003] When training a neural network, the above error calculation needs to be performed using a differentiable function. In other words, it is necessary to use a differentiable function that takes the estimated area and the correct answer area as inputs and outputs the error value. This is because the gradient of the error value in the multidimensional space of parameter values ​​is required when adjusting the parameters inside the neural network.

[0004] However, in practice, there are cases where it is no problem to include a non-differentiable function in error calculation. For example, an operation that selects a portion of multiple input values ​​(as an example, an operation that selects the maximum value of multiple input values) is locally non-differentiable, but it is widely known that neural networks can learn even if such an operation is included. A typical example of an operation that selects a portion of multiple input values ​​is Max Pooling.

[0005] In the conventional technology, because a differentiable error value function is used, a rectangular area is targeted when detecting a specific area in an image. In addition, an error function called IoU (Intersection over Union) is known to be suitable for calculating the error between the estimated area and the correct area (both of which are rectangular), and is used.

[0006] Furthermore, in the conventional technology, instead of constructing a neural network that directly obtains the coordinate values ​​of a specific region, a mask image consisting of pixel values ​​of 0 or more and 1 or less is output from the neural network, and a region whose pixel value exceeds a predetermined threshold value (e.g., 0.5) is output as a specific region of the estimated result. In this case, any shape, not limited to a rectangle, is obtained as the estimated result.

[0007] Non-Patent Document 1 describes a PIoU error function (PIoU loss) as an error function for calculating an error regarding a rotated rectangle. The PIoU error function is a loss function for detecting a rotated object, and is formulated to perform accurate rotated bounding box regression by using both the angle and IoU. The method described in Non-Patent Document 1 calculates an approximation of the number of pixels present inside a rotated rectangle using a differentiable function, and is a method that can calculate an error value similar to IoU.

[0008] Non-Patent Document 2 describes a technique for enabling area detection in more general polygon shapes. However, the technique in Non-Patent Document 2 uses a function (specifically, Smooth L1 Loss) that determines an error based on a simple distance measure between polygon vertices as an error function used for training a neural network. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Zhiming Chen,Kean Chen,Weiyao Lin,John See,Hui Yu,Yan Ke,Cong Yang,"PIoU Loss: Towards Accurate Oriented Object Detection in ComplexEnvironments", European Conference on Computer Vision(ECCV), 2020, https: / / arxiv.org / pdf / 2007.09584.pdf [Non-Patent Document 2] Xiang Zhang, Yongwen Su, Subarna Tripathi, Zhuowen Tu, "Text Spotting Transformers ", CVPR 2022, Computer Vision and Pattern Recognition (cs.CV), April 2022, arXiv:2204.01918 [cs.CV], https: / / arxiv.org / pdf / 2204.01918.pdf Summary of the Invention [Problem to be solved by the invention]

[0010] However, the conventional techniques have the following problems to be solved.

[0011] The IoU between polygons can be precisely defined using mathematical formulas and is often used as a measure of accuracy in technology for detecting polygon-shaped objects. However, there is a problem in that there is no method for calculating polygon IoU that can be used for training neural networks.

[0012] In the conventional technology, the error is calculated using an error function for a rectangular region. In other words, both the region estimated by the neural network and the correct region are limited to a rectangle. For this reason, the conventional technology has a problem in that it is not possible to configure a neural network that directly estimates the coordinate values ​​of regions with shapes other than a rectangle.

[0013] One of the methods described as the conventional technology is a method that allows estimation of any shape by outputting a mask image from a neural network. However, when determining a set of coordinate values ​​for identifying an estimated area based on a mask image generated by a neural network, errors may occur. For example, an area smaller than the size of one pixel of the generated mask image cannot be the estimated result. For example, a method of calculating by simple regression, such as treating the least square error between the correct coordinate value and the estimated coordinate value as an error, can be considered, but the accuracy of regression of coordinate values ​​is generally not high. In addition, there is a constraint that the number of vertices of the figure of the correct area must be the same as the number of vertices of the figure of the estimated area. In other words, for example, it is not possible to calculate an error based on the coordinate values ​​of the vertices between a rectangular area and a pentagonal area.

[0014] In addition, when using the PIoU error function described in Non-Patent Document 1, it is possible to obtain an error between regions for a rotated rectangle, but there is a problem that it is difficult to estimate a shape that is finer than the pixel size of the input image. In addition, the method described in Non-Patent Document 1 has a problem that it can be used only for rotated rectangles and cannot be applied to polygons with complex shapes.

[0015] In addition, in the technology described in Non-Patent Document 2, although the error between general polygons is calculated, the error is the error in the positions of the corresponding vertices of the two polygons. The technology using the vertex error described in Non-Patent Document 2 has a problem that the accuracy is inferior to the case where the polygon IoU is used as the error (does not well represent the degree of agreement (degree of overlap) between the two polygons). In addition, in the technology described in Non-Patent Document 2, the number of vertices of the polygon must be fixed, and therefore preprocessing of the data to match the number of vertices is required, resulting in a problem of high cost of creating a learning dataset.

[0016] The present invention has been made based on the recognition of the above problems, and aims to provide an IoU calculation device and program for calculating the error between any two polygons (polygons), not limited to convex hull polygons (including rectangles, etc.), for use in machine learning of a model, as well as an area detection device and image processing device realized using such an IoU calculation device. [Means for solving the problem]

[0017] [1] One aspect of the IoU calculation device of the present invention for solving the above problem includes a convex hull division unit that obtains one or more divided convex hull polygons by dividing one polygon in a given image, a convex hull inside pixel map generation unit that generates an inside pixel map (IPM) for each of the divided convex hull polygons, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image is inside the convex hull polygon, and a polygon inside pixel map generation unit that generates an inside pixel map for the polygon before it is divided by the convex hull division unit by combining the inside pixel maps of the divided convex hull polygons generated by the convex hull inside pixel map generation unit, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image is inside the polygon before it is divided. An inside pixel map is generated for each of the polygons based on the inside pixel map generated by the polygon inside pixel map generation unit, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map indicating, as a numerical value, whether or not each pixel in the image is inside the union area of ​​the two polygons, and an approximation of the area of ​​the intersection area is calculated based on the inside pixel map for the intersection area, an approximation of the area of ​​the union area is calculated based on the inside pixel map for the union area, and an IoU (Intersection over Unit) for the two given polygons is calculated by dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area.a computation process in which the convex hull inner pixel map generation unit generates an inner pixel map for the convex hull polygon, a computation process in which the polygon inner pixel map generation unit generates an inner pixel map for the polygon before division based on the inner pixel map for the convex hull polygon after division, and a computation process in which the IoU approximation calculation unit calculates an approximation of the IoU value based on the inner pixel map for each of the two polygons are calculated only within the range of a function that is continuous and differentiable within an image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments.

[0018] [2] Also, according to one aspect of the present invention, in the IoU calculation device of [1] above, the convex hull inside pixel map generation unit generates an inside pixel map for each edge of the convex hull polygon to be processed, the inside pixel map indicating, as a numerical value, whether each pixel in the image is on the same side or the opposite side of a specified pixel inside the convex hull polygon, based on the edge and an extension of the edge, and generates an inside pixel map indicating, as a numerical value, whether each pixel in the image is inside the convex hull polygon, by combining the inside pixel maps for all edges of the convex hull polygon.

[0019] [3] Also, according to one aspect of the present invention, in the IoU calculation device of [2] above, the convex hull inner pixel map generation unit generates an inner pixel map that numerically indicates whether or not the pixel is inside the convex hull polygon by selecting the minimum value of the values ​​for each pixel in the inner pixel map for each side when synthesizing the inner pixel maps for all sides of the convex hull polygon.

[0020] [4] Also, one aspect of the present invention is a region detection device including: a region estimation unit that includes an internal machine learning model and that estimates a region having a specific feature in an image by inputting an image passed from the outside to the machine learning model; a learning data supply unit that supplies pairs of learning images and correct regions corresponding to the learning images for learning the machine learning model included in the region estimation unit; and an error calculation unit that calculates an error between an estimated region that is a result of estimation by the region estimation unit based on the learning images supplied by the learning data supply unit and the correct region supplied by the learning data supply unit corresponding to the learning image, wherein both of a figure corresponding to the estimated region and a figure corresponding to the correct region are arbitrary polygons, and the error calculation unit includes an IoU calculation device according to any one of [1] to [3] above, and calculates the error based on an approximation of the IoU value calculated by the IoU calculation device for the polygon that is the figure corresponding to the estimated region and the polygon that is the figure corresponding to the correct region.

[0021] [5] Also, one aspect of the present invention is an image processing device comprising: an estimation unit that has an internal machine learning model and that outputs an estimated polygon as a result of estimating a polygon in an image by inputting an image passed from the outside into the machine learning model; a learning data supply unit that supplies pairs of learning images and ground truth polygons corresponding to the learning images for learning the machine learning model provided by the estimation unit; and an error calculation unit that calculates an error between the estimated polygon, which is a result of estimation by the estimation unit based on the learning images supplied by the learning data supply unit, and the ground truth polygon supplied by the learning data supply unit corresponding to the learning image, wherein the error calculation unit has an IoU calculation device according to any one of [1] to [3] above, and calculates the error based on an approximation of the IoU value calculated by the IoU calculation device for the estimated polygon and the ground truth polygon.

[0022] [6] Also, one aspect of the present invention includes a convex hull division unit that obtains one or more divided convex hull polygons by dividing one polygon in a given image; a convex hull inside pixel map generation unit that generates an inside pixel map (IPM) for each of the divided convex hull polygons, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image is inside the convex hull polygon; a polygon inside pixel map generation unit that generates an inside pixel map for the polygon before it is divided by the convex hull division unit by combining the inside pixel maps of the divided convex hull polygons generated by the convex hull inside pixel map generation unit; Based on the inside pixel map generated by the polygon inside pixel map generation unit, an inside pixel map is generated which numerically indicates whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map is generated which numerically indicates whether or not each pixel in the image is inside the union area of ​​the two polygons, and an approximation of the area of ​​the intersection area is calculated based on the inside pixel map for the intersection area, an approximation of the area of ​​the union area is calculated based on the inside pixel map for the union area, and an IoU (Intersection over Unit) for the two given polygons is calculated by dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area.and an IoU approximation calculation unit that calculates an approximation of an IoU (Interval over Unit) value (IoU over Unit Union) value, wherein the calculation process in which the convex hull inner pixel map generation unit generates an inner pixel map for the convex hull polygon, the calculation process in which the polygon inner pixel map generation unit generates an inner pixel map for the polygon before division, based on the inner pixel map for the convex hull polygon after division, and the calculation process in which the IoU approximation calculation unit calculates an approximation of the IoU value based on the inner pixel map for each of the two polygons are calculated only within the range of a function that is continuous and differentiable within an image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments.

[0023] In addition, several reference forms are as follows:

[0024] [Reference Aspect 1] A region detection device according to one reference aspect for solving the above problem includes a region estimation unit equipped with an internal machine learning model, and estimating a region having a specific feature in an image by inputting an image passed from outside to the machine learning model, a learning data supply unit supplied with a pair of a learning image and a correct region corresponding to the learning image for learning the machine learning model equipped in the region estimation unit, and an error calculation unit for calculating an error between an estimated region estimated by the region estimation unit based on the learning image supplied by the learning data supply unit and the correct region supplied by the learning data supply unit in correspondence with the learning image, wherein both a figure corresponding to the estimated region and a figure corresponding to the correct region are convex hull polygons, and the error calculation unit calculates an inclusion determination value for the figure corresponding to the estimated region and an inclusion determination value for the figure corresponding to the correct region for each of the virtual pixels, an error function value calculation unit which calculates a sum and a product of the virtual pixels and an inclusion judgment value, respectively, and determines a union value obtained by subtracting the product from the sum, the value of the product as an intersection value, and determines a value obtained by dividing the sum of the intersection values ​​for all of the virtual pixels by the sum of the union values ​​for all of the virtual pixels as an error, wherein the inclusion judgment value for a figure corresponding to the estimated area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more away from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more away from any side of the figure, and a function for calculating the inclusion judgment value for the figure corresponding to the estimated area is continuous and differentiable in the entire area within the image, and the inclusion judgment value for the figure corresponding to the correct answer area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more away from any side of the figure,When the virtual pixel exists outside the figure and is a predetermined distance or more away from any side of the figure, the value is 0 or nearly 0, the function for determining the inclusion judgment value for the figure corresponding to the correct answer region is continuous and differentiable in the entire region of the learning image, and the inclusion judgment value for the figure corresponding to the estimated region and the inclusion judgment value for the figure corresponding to the correct answer region are each 0 or more and 1 or less in the entire region of the image and the learning image.

[0025] [Reference Aspect 2] In one reference aspect, in the area detection device of the above [Reference Aspect 1], the error calculation unit further calculates, for each of the virtual pixels, an error vector obtained by calculating, for each side of the figure corresponding to the estimated area and each side of the figure corresponding to the correct area, the following equation (1) (where e is Napier's constant, k is a predetermined positive constant, d px is the distance from a reference line that passes through a specified point inside the figure and is parallel to the side to the virtual pixel, and d center an inclusion determination unit that determines an inclusion determination value for the virtual pixel for the relevant side by multiplying together the inclusion determination values ​​of the virtual pixel for all sides of the figure determined by the inclusion determination unit.

[0026] [Reference Aspect 3] Also, one reference aspect is a region detection device including: a region estimation unit that includes a machine learning model and that estimates a region having a specific feature in an image by inputting an image passed from the outside to the machine learning model; a learning data supply unit that supplies pairs of learning images and correct regions corresponding to the learning images for learning the machine learning model included in the region estimation unit; and an error calculation unit that calculates an error between an estimated region that is a result of estimation by the region estimation unit based on the learning images supplied by the learning data supply unit, and the correct region supplied by the learning data supply unit in correspondence with the learning image, wherein a figure corresponding to the estimated region and a figure corresponding to the correct region are both convex hull polygons, and the error calculation unit includes a virtual pixel generation unit that sets a large number of virtual pixels that are common in both the image and the learning image, and a sum and product of an inclusion determination value for a figure corresponding to the estimated region and an inclusion determination value for a figure corresponding to the correct region for each of the virtual pixels, and a union value (un an error function value calculation unit which calculates an error by dividing the sum of the intersection values ​​for all of the virtual pixels by the sum of the union values ​​for all of the virtual pixels, with the value of the product being an intersection value, and calculating an error by dividing the sum of the intersection values ​​for all of the virtual pixels by the sum of the union values ​​for all of the virtual pixels, wherein an inclusion determination value for a figure corresponding to the estimated area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more away from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more away from any side of the figure, and a function for calculating an inclusion determination value for a figure corresponding to the estimated area is continuous and differentiable in the entire area within the image, and an inclusion determination value for a figure corresponding to the correct answer area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more away from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more away from any side of the figure,and a function for determining an inclusion judgment value for the figure corresponding to the correct answer region is continuous and differentiable in the entire region of the learning image, and the inclusion judgment value for the figure corresponding to the estimated region and the inclusion judgment value for the figure corresponding to the correct answer region are each greater than or equal to 0 and less than or equal to 1 in the entire region of the image and the learning image. Effect of the Invention

[0027] According to the present invention, an approximation of IoU for two arbitrary polygons can be calculated in a form that can be used as an error for learning a machine learning model. The polygons to be targeted are not limited to convex hull polygons. In addition, there is no need to impose restrictions on the number of vertices of the polygons to be targeted. [Brief description of the drawings]

[0028] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an area detection device according to a first embodiment of the present invention. [Diagram 2] 4 is a block diagram showing a more detailed functional configuration inside the error calculation unit according to the first embodiment. FIG. [Diagram 3] 3 is a schematic diagram showing an example of an arrangement of real pixels and virtual pixels in an image in the first embodiment. FIG. [Figure 4] FIG. 2 is one schematic diagram for explaining the processing of the inclusion determination unit according to the first embodiment, showing a state in which one of the figures subject to error calculation exists on the xy plane. [Diagram 5] FIG. 13 is another schematic diagram for explaining the processing of the inclusion determination unit according to the first embodiment, showing a state in which the geometric plane is rotated so that one side is made horizontal. [Figure 6] 4 is a schematic diagram for explaining a process performed by an error function value calculation unit according to the first embodiment to calculate an error between two figures. FIG. [Figure 7] 1 is a block diagram showing an example of an internal configuration for realizing an area detection device according to a first embodiment. [Figure 8]FIG. 11 is a schematic diagram for explaining a method of determining an inclusion determination value according to a modified example of the first embodiment. [Figure 9] 2A to 2C are schematic diagrams for explaining an application example of the area detection device according to the first embodiment and examples of images used in an experiment for verifying the application example. [Figure 10] FIG. 11 is a block diagram showing a schematic functional configuration of an area detection device according to a second embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an example in which a convex hull division unit according to the second embodiment divides a given polygon (PA) into a plurality of convex hulls. [Figure 12] FIG. 11 is a schematic diagram showing an example in which a convex hull division unit according to the second embodiment divides a given polygon (PB) into a plurality of convex hulls. [Figure 13] 13 is a schematic diagram showing a polygon PA2 that is an example of a convex hull polygon that is a target of IPM (interior pixel map) generation processing by a convex hull IPM generation unit according to the second embodiment. FIG. [Figure 14] FIG. 11 is a schematic diagram for explaining a method in which a convex hull IPM generation unit according to the second embodiment determines an IPM for one side of a convex hull polygon PA2. [Figure 15] 13 is a schematic diagram showing an IPM calculated for an edge L1 of a convex hull polygon PA2 by a convex hull IPM generation unit according to the second embodiment. FIG. [Figure 16] 13 is a schematic diagram showing an IPM determined for side L2 of a convex hull polygon PA2 by a convex hull IPM generation unit according to the second embodiment. FIG. [Figure 17] 13 is a schematic diagram showing an IPM determined for side L3 of a convex hull polygon PA2 by a convex hull IPM generation unit according to the second embodiment. FIG. [Figure 18] 13 is a schematic diagram showing an IPM determined for side L4 of a convex hull polygon PA2 by a convex hull IPM generation unit according to the second embodiment. FIG. [Figure 19] 13 is a flowchart showing a processing procedure for determining an approximate value of IoU of two given polygons by the polygon IoU calculation device according to the second embodiment. [Figure 20]FIG. 13 is a block diagram showing a schematic functional configuration of an area detection device (using the polygon IoU calculation device of the second embodiment) according to a third embodiment. [Figure 21] FIG. 13 is a block diagram showing a schematic functional configuration of an image processing device (using the polygon IoU calculation device of the second embodiment) according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] [First embodiment] Next, a first embodiment will be described with reference to the drawings. In this embodiment, the area detection device 1 automatically detects an area having a specific characteristic in an input image by using a machine learning technique. In this embodiment, when learning the machine learning model, a method of calculating an error (also called loss or error) specific to this embodiment is used. The error calculation has the following two main features.

[0030] As a first feature, the error calculation unit 105, which will be described later, defines virtual pixels that are used only for convenience in calculation, regardless of the actual pixels that constitute the input image, and performs calculations based on the virtual pixels. Specifically, the error calculation unit 105 calculates the number of virtual pixels contained inside a figure that represents an area in the image. Note that the figure here is limited to a convex hull figure. In other words, the figure (polygon) here is limited to a figure in which the interior angle of any vertex that constitutes the figure is less than 180 degrees. Note that the error calculation unit 105 may calculate an approximation of the number of virtual pixels contained inside the figure. By performing calculations based on virtual pixels in this way, the error calculation unit 105 can calculate an error with high accuracy even when the resolution of the input image is low.

[0031] As a second feature, the error calculation unit 105 judges, for each side constituting the symmetrical figure, whether each virtual pixel is inside or outside the figure. Then, the error calculation unit 105 integrates the judgment results (inside or outside) for each side to determine whether each virtual pixel is inside or outside the symmetrical figure. Using this procedure, the error calculation unit 105 calculates the number of virtual pixels present inside a symmetrical figure (but a convex hull figure) having an arbitrary shape. A specific calculation method will be described later.

[0032] FIG. 1 is a block diagram showing a schematic functional configuration of the area detection device according to the present embodiment. As shown in the figure, the area detection device 1 includes an image input unit 101, an area estimation unit 102, a result output unit 103, a learning data supply unit 104, and an error calculation unit 105. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit has a storage unit as necessary. The storage unit is, for example, a variable in the program or a memory allocated by the execution of the program. Non-volatile storage units such as a magnetic hard disk device or a solid state drive (SSD) may be used as necessary. At least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0033] The area detection device 1 operates in either a learning mode or an estimation execution mode. In the learning mode, the area detection device 1 performs processing using learning data to optimize parameters of a machine learning model held by the area estimation unit 102. In the estimation execution mode, the area estimation unit 102 performs processing to estimate a specific area in an unknown input image using learned parameters. The functions of each unit are described below.

[0034] An image input unit 101 acquires an image input from outside and passes it to a region estimation unit 102 . The image that the image input unit 101 passes to the area estimation unit 102 is an image that is to be estimated in the operation in the estimation execution mode.

[0035] The region estimation unit 102 includes a machine learning model and estimates a region having a specific feature in an image by inputting an image passed from the outside into the machine learning model. Specifically, the region estimation unit 102 includes a neural network as the machine learning model. The region estimation unit 102 outputs a figure corresponding to the region as the estimation result. In this embodiment, the figure is a convex hull polygon. That is, the figure is a polygon in which all interior angles are less than 180 degrees. More specifically, the region estimation unit 102 may output the coordinate values ​​of each vertex of the convex hull polygon as the estimation result. Alternatively, the region estimation unit 102 may output equivalent information regarding the convex hull polygon.

[0036] When operating in the learning mode, the area estimation unit 102 passes information on the figure of the area of ​​the estimation result to the error calculation unit 105. As a result, the error calculation unit 105 calculates the error between the figure of the estimated area and the figure of the correct answer area. When operating in the estimation execution mode, the area estimation unit 102 passes information on the figure of the estimated area, which is the result of estimation based on an unknown image, to the result output unit 103.

[0037] The result output unit 103 outputs to the outside the information of the estimated area (graphic information) output by the area estimation unit 102 as an estimation result in the estimation execution mode. This information of the estimated area can be used for various purposes as information of an area having a specific characteristic.

[0038] When the area detection device 1 operates in a learning mode, the learning data supplying unit 104 supplies learning data for learning the neural network (machine learning model) held by the area estimation unit 102. The learning data is information on a set of pairs of a learning image and a figure (in this embodiment, a convex hull polygon) representing a correct area corresponding to the learning image. The learning data supplying unit 104 passes the learning image included in the pair to the area estimation unit 102. In addition, the learning data supplying unit 104 passes information on the figure of the correct area included in the pair to the error calculation unit 105. This allows the error calculation unit 105 to calculate the error between the figure of the area (estimated area) output by the area estimation unit 102 based on the learning image and the figure of the correct area supplied by the learning data supplying unit 104. When learning the neural network, the learning data supplying unit 104 sequentially supplies each pair for learning.

[0039] When the area detection device 1 operates in the learning mode, the error calculation unit 105 calculates the error between the figure of the area output by the area estimation unit 102 as an estimation result and the figure of the correct area supplied by the learning data supply unit 104. In this embodiment, the figure corresponding to the estimated area and the figure corresponding to the correct area are both convex hull polygons. The error calculated by the error calculation unit 105 is used to adjust (optimize) the internal parameters of the neural network of the area estimation unit 102 by the error backpropagation method. A more detailed configuration and processing contents of the error calculation unit 105 will be further explained later with reference to FIG. 2 etc.

[0040] The error calculated by the error calculation unit 105 is as follows. The internal configuration of the error calculation unit 105, which will be described later with reference to the block diagram in Fig. 2, realizes an example of a procedure for specifically realizing the following error.

[0041] The error calculation unit 105 uses an inclusion judgment value as basic data for calculating an error. The inclusion judgment value is a numerical value indicating whether or not a virtual pixel described later is included in a figure (a figure of an estimated region or a figure of a correct answer region). However, in this embodiment, the inclusion judgment value is not a binary value of, for example, 0 or 1, but can take a continuous value from 0 to 1.

[0042] The inclusion judgment value for the figure corresponding to the estimated region is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more away from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more away from any side of the figure, and the function for determining the inclusion judgment value for the figure corresponding to the estimated region is continuous and differentiable in the entire region of the image. That is, in the vicinity of the side of the figure (as already explained, the figure is a polygon) (the vicinity means the region within the above-mentioned predetermined distance from the side), the inclusion judgment value changes continuously (and suddenly) and smoothly from 0 (or approximately 0) to 1 (or approximately 1).

[0043] The same is true for the correct answer region. That is, the inclusion judgment value for the figure corresponding to the correct answer region is 1 or almost 1 when the virtual pixel exists inside the figure and is a predetermined distance or more away from any side of the figure, and is 0 or almost 0 when the virtual pixel exists outside the figure and is a predetermined distance or more away from any side of the figure, and the function for determining the inclusion judgment value for the figure corresponding to the correct answer region is continuous and differentiable in the entire region of the learning image. That is, in the vicinity of the side of the figure (which is a polygon) (the vicinity means the region within the above-mentioned predetermined distance from the side), the inclusion judgment value changes continuously (and suddenly) and smoothly from 0 (or almost 0) to 1 (or almost 1).

[0044] The inclusion determination value for a figure corresponding to the estimated region and the inclusion determination value for a figure corresponding to the correct region are each equal to or greater than 0 and equal to or less than 1 in the entire region of the image and the learning image.

[0045] In this embodiment, in order to calculate an inclusion determination value for a virtual pixel with respect to a figure, the error calculation unit 105 calculates an inclusion determination value for the virtual pixel with respect to each side of the figure, and multiplies these inclusion determination values ​​together to calculate an inclusion determination value for the virtual pixel with respect to the figure.

[0046] 2 is a block diagram showing a more detailed internal functional configuration of the error calculation unit 105 according to this embodiment. As shown in the figure, the error calculation unit 105 includes a virtual pixel generation unit 1051, an inclusion determination unit 1052, an inclusion determination integration unit 1053, and an error function value calculation unit 1054. The functions of each unit are as follows.

[0047] The virtual pixel generation unit 1051 sets (generates) a large number of virtual pixels that are common to both the image and the learning image. The virtual pixels will be described in more detail later.

[0048] The inclusion determination unit 1052 calculates, for each virtual pixel, the side of the figure corresponding to the estimated region and the side of the figure corresponding to the correct region, the following equation (1) (where e is Napier's constant, k is a predetermined positive constant, d px is the distance from a reference line that passes through a specified point inside the figure and is parallel to the side to the virtual pixel, and d center is the distance from the reference line to the side in question) to determine an inclusion determination value for the side in question of the virtual pixel.

[0049] The inclusion determination integrating unit 1053 integrates the inclusion determination values ​​for each side of one virtual pixel obtained by the inclusion determination unit 1052, and obtains an inclusion determination value for the graphic (polygon) of the virtual pixel. Specifically, the inclusion determination integrating unit 1053 multiplies all of the inclusion determination values ​​of the virtual pixel for all sides of the graphic obtained by the inclusion determination unit 1052, thereby obtaining an inclusion determination value for the graphic of the virtual pixel.

[0050] The error function value calculation unit 1054 calculates the error between the figure of the estimated region and the figure of the correct answer region based on the inclusion judgment value. Specifically, the error function value calculation unit 1054 calculates the error based on the inclusion judgment value, which is a value indicating whether or not the virtual pixel is included in each figure. More specifically, the error function value calculation unit 1054 calculates the sum and product of the inclusion judgment value for the figure corresponding to the estimated region and the inclusion judgment value for the figure corresponding to the correct answer region for each virtual pixel, respectively, and calculates the error by subtracting the product from the sum as a union value, the product as an intersection value, and dividing the sum of the intersection values ​​for all virtual pixels by the sum of the union values ​​for all virtual pixels.

[0051] Next, the flow of processing by the error calculation unit 105 will be described with an actual example.

[0052] The virtual pixel generation unit 1051 generates virtual pixels for the image for which the error is to be calculated. Virtual pixels are pixels virtually arranged at a predetermined interval on the surface of the image. When expressed in xy Cartesian coordinates, for example, when the y coordinate is from 0.0 to 1.5, the x coordinate is from 0.0 to 1.0, and the interval between each of the x and y coordinates of the virtual pixel is 0.5, the coordinate values ​​of the virtual pixel are the following 12: (x,y)= (0.0,0.0),(0.0,0.5),(0.0,1.0),(0.0,1.5),(0.5,0.0),(0.5,0.5),(0.5,1.0),(0.5,1.5),(1.0,0.0),(1.0,0.5),(1.0,1.0),(1.0,1.5),

[0053] The coordinate values ​​of a virtual pixel may differ from the coordinate values ​​of an actual pixel in the image. The spacing between virtual pixels (0.5 in the above example) may be set as appropriate, regardless of the spacing between actual pixels. The x and y coordinate values ​​of a virtual pixel are expressed as real numbers and do not need to be integers. As such, there is a high degree of freedom in the coordinate values ​​of a virtual pixel, making it suitable for handling small figures whose size in the x and y directions is one pixel or less. Note that the narrower the spacing between virtual pixels (i.e., the more virtual pixels per given length are used), the more calculations are required to calculate the error, but the calculation accuracy improves.

[0054] FIG. 3 is a schematic diagram showing an example arrangement of real and virtual pixels in an image. The figure shows only a part of the image. In the figure, real pixels 1901 are shown as rectangles. Also, virtual pixels 1902 are shown as black circles. In this example, both the real pixels 1901 and the virtual pixels 1902 are arranged in a square array. In this example, 16 virtual pixels 1902 are arranged corresponding to one real pixel 1901. That is, the density of the arrangement of the virtual pixels 1902 is four times the density of the arrangement of the real pixels 1901 in both the vertical and horizontal directions. The arrangement pattern shown in FIG. 3 is only an example. The virtual pixels may be arranged in a delta array, for example, instead of a square array. Also, the virtual pixels may be arranged in an irregular array. The relationship between the arrangement density of the real pixels and the arrangement density of the virtual pixels is also arbitrary. In either case, it is desirable that the arrangement density of the virtual pixels does not vary significantly depending on the position on the screen and is as uniform as possible.

[0055] The inclusion determination unit 1052 performs an inclusion determination for each side for each of the virtual pixels set above. That is, the inclusion determination unit 1052 determines on which side of each side each virtual pixel exists. As a processing procedure, the inclusion determination unit 1052 may rotate the image so that the side of interest is, for example, horizontal, and in that state, determine whether each virtual pixel exists above or below that side.

[0056] FIG. 4 is one of schematic diagrams for explaining the process by the inclusion determination unit 1052. The figure shows a state in which one of the figures for which an error is to be calculated is present on the xy plane. In the figure, 2001 is one of the figures for which an error is to be calculated. The figure 2001 is either a figure representing the result of the area estimated by the area estimation unit 102 or a figure representing the correct area supplied by the learning data supply unit 104. In this example, the figure 2001 is a quadrangle having sides 2011, 2012, 2013, and 2014. The figure for which an error is to be calculated is not limited to a quadrangle, and may be any polygon. However, the figure to be calculated is a convex hull polygon.

[0057] FIG. 5 is another schematic diagram for explaining the process by the inclusion determination unit 1052. The state shown in FIG. 5 is a result of rotating a plane (xy plane) including the figure 2001 shown in FIG. 4. The inclusion determination unit 1052 rotates the currently focused side (side 2011 in this example) of the sides of the figure 2001 so that it becomes horizontal. The dashed line 2021 shown in the figure is a horizontal line that serves as a reference. That is, the dashed line 2021 is a line parallel to the side 2011. The dashed line 2021 is a line that passes through the center point of the figure 2001. The center point may be determined by the average value of the x coordinate and the y coordinate of the vertices that constitute the figure 2001. That is, when the positional relationship in the vertical direction of this figure is viewed, the dashed line 2021 is located inside the figure 2001 more than the side 2011. Conversely, in terms of the vertical position, the side of side 2011 opposite to dashed line 2021 is outside figure 2001. Here, the distance from dashed line 2021 to side 2011 is 1.5. Also, the distance from dashed line 2021 to virtual pixel 2022 is 2.2, and the distance from dashed line 2021 to virtual pixel 2023 is 0.6. At this time, inclusion determination unit 1052 calculates an inclusion determination value by the following formula (1).

[0058]

number

[0059] In formula (1), d px is the distance from the reference line (dashed line 2021 in the example) to the virtual pixel to be determined. center is the distance from the reference line to the edge under consideration (edge ​​2011 in this example). k is a parameter that takes a predetermined positive value, for example, k=10. e is Napier's constant. d px d center If the value is smaller than , the inclusion judgment value calculated by formula (1) will be close to 1. px d center If the value of d is larger than 0, the inclusion judgment value calculated by formula (1) will be close to 0. px d center, the inclusion judgment value calculated by equation (1) will be 0.5. In other words, when viewed in the vertical position in the example of Figure 5, if the virtual pixel is located below side 2011, the inclusion judgment value approaches 0, and if the virtual pixel is located above side 2011, the inclusion judgment value approaches 1. Note that the inclusion judgment value changes smoothly and steeply in the vicinity of side 2011. The value of the parameter k above is a value for controlling the degree of abruptness of the change in the inclusion judgment value in the vicinity of that side 2011. However, the function for calculating the inclusion judgment value expressed by equation (1) is d px In the entire region of px It is differentiable with respect to

[0060] 5, the inclusion determination value of the virtual pixel 2022 with respect to the side 2011 is calculated using equation (1) and is approximately 0.0009 (close to 0). This indicates that the virtual pixel 2022 is located above the side 2011. Moreover, the inclusion determination value of the virtual pixel 2023 with respect to the side 2011 is calculated using equation (1) and is approximately 0.9999 (close to 1). This indicates that the virtual pixel 2023 is located below the side 2011.

[0061] The inclusion judgment integration unit 1053 integrates the inclusion judgments for each side by the inclusion judgment unit 1052 to obtain an inclusion judgment value for the entire graphic. Specifically, the inclusion judgment integration unit 1053 multiplies all the inclusion judgment values ​​calculated for each side for each virtual pixel to calculate an integrated inclusion judgment value for the virtual pixel. As already described, in a state where the graphic is rotated as shown in FIG. 5, that is, in a state where the side is horizontal and the center point of the graphic is located below the side, the inclusion judgment value of the virtual pixel below the side is close to 1, and the inclusion judgment value of the virtual pixel above the side is close to 0. In addition, the inclusion judgment value changes rapidly between 0 and 1 in the vicinity of the side. In other words, when all the inclusion judgment values ​​for each side are multiplied for a certain virtual pixel, the integrated inclusion judgment value of the virtual pixel present inside the graphic (for example, a rectangle) is close to 1, and the integrated inclusion judgment value of the virtual pixel present outside the graphic is close to 0.

[0062] The inclusion determination integration unit 1053 calculates an integrated inclusion determination value for all virtual pixels. The sum of the integrated inclusion determination values ​​for all virtual pixels in one image is an approximation of the number of virtual pixels contained in the figure, and this value is proportional to an approximation of the area of ​​the figure. In other words, the sum of the inclusion determination values ​​calculated by the inclusion determination integration unit 1053 is a value that represents the area of ​​the figure, and is a differentiable value that can be calculated simply by calculating formula (1) and adding up those values.

[0063] The error function value calculation unit 1054 uses the calculation result of the inclusion determination integration unit 1053 to calculate the error between the two figures input to the error calculation unit 105. Specifically, the error function value calculation unit 1054 performs the calculation described next with reference to FIG.

[0064] 6A and 6B are schematic diagrams for explaining the process performed by the error function value calculation unit 1054 to calculate the error between two figures. Fig. 6A shows the area of ​​a figure 2002, which is the first figure. FIG. 6B shows the area of ​​the second figure, figure 2003. The more these two figures match, the smaller the error between them; the more they differ, the larger the error between them. To find such an error, error function value calculation unit 1054 performs a calculation that can be used as an alternative to IoU (Intersection over Union), as described below. FIG. 6C shows a situation where there is only a partial overlap between the areas of figure 2002 and figure 2003. The calculation performed by the error function value calculation unit 1054 is based on the area of ​​the union of both sets and the area of ​​the intersection of both sets.

[0065] An intersection in the IoU is a common part (a part of a set intersection) of two figures. In this embodiment, as a calculation for obtaining a value corresponding to an intersection in the IoU, the error function value calculation unit 1054 obtains the product of the integrated inclusion judgment values ​​of the two figures for each virtual pixel. For one virtual pixel, when at least one of the inclusion judgment values ​​of the two figures is close to 0, the product of the two inclusion judgment values ​​is close to 0. When both of the inclusion judgment values ​​of the two figures are close to 1, the product of the two inclusion judgment values ​​is close to 1. Moreover, the product of the two inclusion judgment values ​​is differentiable. The sum of the above products for all virtual pixels is a value corresponding to the intersection in this embodiment. A union in the IoU is a part of the union of two figures. In this embodiment, as a calculation for obtaining a value corresponding to the intersection in IoU, the error function value calculation unit 1054 calculates the sum of the integrated inclusion judgment values ​​of the two figures for each virtual pixel, and calculates a value obtained by subtracting the above product value from the sum. For the common part of the two figures, the sum of the inclusion judgment values ​​is close to 2. Therefore, by subtracting the product (a value close to 1) of the two inclusion judgment values ​​from the sum, the value obtained is close to 1. For the exclusive sum part of the two figures, the sum of the inclusion judgment values ​​is close to 1. Since the product of the two inclusion judgment values ​​for the exclusive sum part is close to 0, the resulting value obtained is close to 1. For the other parts (parts that do not belong to either of the two figures), both the sum and the product of the inclusion judgment values ​​are close to 0. Therefore, the result of subtracting the product from the sum is close to 0. The sum of the values ​​obtained by subtracting the product from the above sum for all virtual pixels is the value corresponding to the union in this embodiment. This value is also differentiable.

[0066] That is, the error function value calculation unit 1054 calculates the error between the two figures based on the following formula.

[0067] Formula: sum(intersection) / sum(union)

[0068] In this formula, union is a value equivalent to the union of each virtual pixel calculated by the error function value calculation unit 1054 using the above method. Also, intersection in this formula is a value equivalent to the intersection of each virtual pixel calculated by the error function value calculation unit 1054 using the above method. Also, sum( ) represents an operation (calculation) for taking the sum of the values ​​of all virtual pixels in an image.

[0069] As described above, both the union and the intersection in the above formula are differentiable. The sum of all the virtual pixels of each of them is also differentiable. Furthermore, the value expressed by the above formula (sum(intersection) / sum(union)) (the error calculated by the error function value calculation unit 1054) is also differentiable.

[0070] As described above, the error calculation unit 105 can obtain an appropriate error by using the function of formula (1) for two figures (both of which are convex hull figures (convex hull polygons)). In addition, since the error is differentiable, it is possible to update parameters of the neural network contained within the area estimation unit 102 by error backpropagation. In other words, by using a sufficient amount of learning data and performing error backpropagation a sufficient number of times, the neural network (machine learning model) contained in the area estimation unit 102 is optimized.

[0071] As described above, in the area detection device 1 of this embodiment, the figure representing the area to be detected may be any convex hull polygon in any orientation (not limited to a rectangle, etc.). Furthermore, as is clear from the error calculation process described above, the number of vertices of the figure (convex hull polygon) of the area output by the area estimation unit 102 as the estimation result and the number of vertices of the figure (convex hull polygon) of the correct area supplied by the learning data supply unit 104 may be different. That is, the present embodiment solves the above-mentioned problem, and the error between any convex hull polygons can be calculated as a differentiable function value. This makes it possible to use a machine learning model to detect areas related to figures other than rectangles. This removes the constraints on the problem of detecting areas having specific characteristics in an image, and the area detection device 1 can be used to solve more general problems.

[0072] FIG. 7 is a block diagram showing an example of the internal configuration of the area detection device 1 described as an embodiment. The area detection device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." I / O port 903 is a port through which central processing unit 901 exchanges data with external I / O devices. I / O devices 904 and 905 are I / O devices. I / O devices 904 and 905 exchange data with central processing unit 901 via I / O port 903. Bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses an input / output port via the bus 906.

[0073] At least some of the functions of the area detection device 1 in the above-described embodiment can be realized by a computer and a program. In this case, the functions may be realized by recording a program for realizing the functions in a computer-readable recording medium, reading the program recorded in the recording medium into a computer system, and executing the program. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, as well as storage devices such as hard disks built into computer systems. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. The term "computer-readable recording medium" may also include a medium that temporarily and dynamically holds a program, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0074] Although the embodiment has been described above, the present invention can also be embodied in the following modified examples.

[0075] [Modification of the first embodiment] In the embodiment, the procedure has been described in which an inclusion determination value for one virtual pixel and one side is calculated by formula (1), and the inclusion determination values ​​for all sides are multiplied together to obtain an inclusion determination value for the graphic form (convex hull polygon) of the virtual pixel.As a modified example, a method for obtaining an inclusion determination value for the graphic form of one virtual pixel will be described here.

[0076] FIG. 8 is a schematic diagram for explaining how to obtain the inclusion judgment value according to this modified example. In the example shown in the figure, the figure to be judged is a convex pentagon. The convex pentagon has sides 2031, 2032, 2033, 2034, and 2035. In this figure, the region is divided into three regions using boundary lines (dashed lines). Region R1 is a region inside the figure (pentagon) and is a predetermined distance or more away from any of the sides. Region R3 is a region outside the figure (pentagon) and is a predetermined distance or more away from any of the sides. Region R2 is a region that is neither region R1 nor R3. That is, region R2 is a region in the vicinity of any of the sides 2031, 2032, 2033, 2034, and 2035. In other words, region R2 is a region within a predetermined distance from at least any of these sides. The width of region R2 in the direction perpendicular to each side is sufficiently small. In this modification, the inclusion determination value is also set to a value greater than or equal to 0 and less than or equal to 1. In this modification, the inclusion determination value in region R1 is set to 1 or approximately 1 (i.e., greater than or equal to 1-ε and less than or equal to 1). Here, ε is a positive constant that is sufficiently smaller than 1. The inclusion judgment value in region R3 is set to 0 or nearly 0 (i.e., 0 or more and ε or less). The function for calculating the inclusion judgment value for a virtual pixel in this image is set to be continuous and differentiable in the entire region of the image.

[0077] The error calculation unit 105 of the modified example calculates the error between the two figures based on such an inclusion determination value, using the calculation procedure already explained in the embodiment.

[0078] The inclusion determination value described with reference to Fig. 8 corresponds to a more general form. The method of obtaining the inclusion determination value described in the embodiment is a special example of this general form.

[0079] As described above, according to the embodiment (including the modified example), it is possible to detect areas of figures other than rectangles using a machine learning model. Also, according to the embodiment (including the modified example), it is possible to generate (provide) virtual pixels at any resolution and detect areas in detail.

[0080] The above describes in detail an embodiment of the present invention (including modified examples) with reference to the drawings. However, the specific configuration is not limited to this embodiment, and also includes designs that do not deviate from the gist of the present invention.

[0081] [Applications and Effects of the First Embodiment] A description will be given of an application example and its effects of the area detection device 1 of the above embodiment. As an application example, the area detection device 1 is used to detect an area of ​​text contained in an image.

[0082] FIG. 9 is a schematic diagram showing an example of an image to be processed in this application example. FIG. 9(A) shows an image to be input to the area detection device 1. The image contains the characters "abc" as an example. The area detection device 1 performs machine learning in advance using learning data so as to detect a character area. The area detection device 1 that has been trained is operated in an estimation execution mode, and the image of FIG. 9(A) is input. FIG. 9(B) shows an example of the detection result. The area detection device 1 detects areas that contain the characters "a", "b", and "c". Each of the detected areas is a square (not a rectangle). Here, based on the detection result by the area detection device 1, the detected areas are separated as independent images, and areas other than the detected areas are masked with black (pixel value 0). FIG. 9(C) shows three images showing the results. The hatched areas in FIG. 9(C) are areas masked with black (pixel value 0). This masking is to prevent unnecessary noise information from entering the character recognition process that is performed next. Then, character recognition was performed on these three images using a character recognition device (not covered by the present invention). As a result, the characters "a", "b", and "c" were correctly recognized. In other words, in this application example, detection of areas (character areas) having specific characteristics was performed as a preprocessing step for character recognition processing. When this demonstration experiment was performed, the character recognition accuracy (F value) improved from 70% when area detection was not performed to 73% when area detection preprocessing was performed. In other words, it was confirmed that the effects of the above embodiment were obtained.

[0083] [Second embodiment] Next, a second embodiment will be described. Note that the matters already described in the previous embodiment may not be described below. Here, the matters specific to this embodiment will be mainly described. Note that the polygons to be processed in this embodiment are polygons on a plane.

[0084] [Polygon IoU calculation device configuration and processing ocean] FIG. 10 is a block diagram showing a schematic functional configuration of a polygon IoU calculation device according to this embodiment. As shown in the figure, the polygon IoU calculation device 3051 includes a polygon information acquisition unit 3052, a convex hull division unit 3053, a convex hull IPM generation unit 3054, a polygon IPM generation unit 3055, and a polygon IoU approximation calculation unit 3056. As in the first embodiment, each functional unit of the polygon IoU calculation device 3051 of this embodiment can also be realized by, for example, a computer and a program. In addition, at least a part of the function of each functional unit may be realized as a dedicated electronic circuit rather than a program. An outline of each function will be described below.

[0085] The polygon information acquisition unit 3052 acquires information on two polygons for which IoU is to be calculated from the outside. The polygon information is information that indicates the shape, size, position, inclination (rotation angle) and the like of the polygon. The information on one polygon acquired by the polygon information acquisition unit 3052 may be expressed as information on the coordinates of all vertices of the polygon, for example. However, the representation form of the polygon information may be other than the form presented here (coordinates of all vertices). The polygon information acquisition unit 3052 passes the polygon information on the two polygons for which IoU is to be calculated to the convex hull division unit 3053.

[0086] The convex hull division unit 3053 divides each polygon into convex hull polygons based on the polygon information passed from the polygon information acquisition unit 3052. In other words, a convex hull polygon is a convex polygon. In other words, a convex hull polygon is a polygon that does not have any interior angle greater than 180 degrees. The convex hull division unit 3053 may divide the passed polygon into multiple convex hull polygons using various methods as appropriate. The method by which the convex hull division unit 3053 divides a polygon into convex hull polygons does not have to be differentiable, and any method may be used. As an example, the convex hull division unit 3053 may divide a given polygon into convex hull polygons using Delaunay triangulation (a triangle is always a convex hull). As another example, the convex hull division unit 3053 may divide a given polygon into convex hull polygons using a rule-based method. The division of an arbitrary polygon into a convex hull polygon can be realized by using existing technology.

[0087] In other words, the convex hull division unit 3053 obtains one or more divided convex hull polygons by dividing one polygon in a given image. If there is no need to divide a given polygon, the convex hull division unit 3053 may output the given polygon as it is as one divided convex hull polygon.

[0088] There is a problem in that the more the number of divisions into convex hulls as a result of the division by the convex hull division unit 3053, the more the amount of calculations in the subsequent processing increases. In other words, it is preferable for the convex hull division unit 3053 to perform processing to divide into as few convex hulls as possible. Also, when a polygon passed from the polygon information acquisition unit 3052 is a convex hull from the beginning, the convex hull division unit 3053 may omit (skip) the processing of dividing that polygon into convex hull polygons. An example of division by the convex hull division unit 3053 will be described later with reference to Figs. 11 and 12.

[0089] The convex hull IPM generation unit 3054, which will be described later, finds an IPM for each side of the convex hull polygon, and generates an IPM for the convex hull polygon by performing a set intersection operation (described in detail later) for the IPMs of all the sides. In order to make this possible, it is necessary that the polygon to be processed by the convex hull IPM generation unit 3054 is a convex hull polygon. If the polygon to be processed contains one or more interior angles of 180 degrees or more, the above-mentioned set intersection operation cannot be applied well. However, the convex hull division unit 3053 divides a polygon of an arbitrary shape, and passes a set consisting of only convex hull polygons to the convex hull IPM generation unit 3054 as the processing target. This allows the polygon IoU calculation device 3051 of this embodiment to correctly calculate an approximation value of IoU.

[0090] The convex hull IPM generating unit 3054 performs an inclusion determination for each of the convex hull polygons resulting from the division by the convex hull dividing unit 3053, and generates an IPM (Inner Pixel Map) for each of the convex hull polygons. The inclusion determination is a process for determining whether each pixel constituting an image is inside or outside the convex hull polygon when the convex hull polygon is depicted on an image made up of a large number of pixels. The IPM is data that expresses as numerical information whether each pixel in the image is inside a specific polygon. The convex hull IPM generated by the convex hull IPM generating unit 3054 in this embodiment is data that expresses as a map numerical values ​​indicating whether each pixel in the image is inside a given convex hull polygon. In other words, the IPM generated by the convex hull IPM generating unit 3054 is a map in which a value based on the result of the inclusion determination is assigned to each pixel. The pixel value in the IPM is 0.0 or more and 1.0 or less. Pixels that are determined to be inside the convex hull polygon are assigned a pixel value of the convex hull IPM close to 1.0. Pixels that are determined to be outside the convex hull polygon are assigned a pixel value of the convex hull IPM close to 0.0. To make this IPM usable in neural network training, the pixel values ​​of the IPM are not simply assigned discrete values ​​of 1 or 0, but are instead assigned values ​​of a differentiable function (continuous values).

[0091] That is, the convex hull IPM generator 3054 generates an inside pixel map (IPM) that numerically indicates whether or not each pixel in the image is inside the convex hull polygon for each of the convex hull polygons after division by the convex hull division unit 3053. The convex hull IPM generator 3054 is also called a "convex hull inside pixel map generator." Details of the process by which the convex hull IPM generator 3054 creates an IPM for one convex hull polygon will be described later with reference to Figures 13 to 18.

[0092] As a more specific process, the convex hull IPM generation unit 3054 may generate an inside pixel map (IPM) for each edge of the convex hull polygon being processed, which indicates as a numerical value whether each pixel in the image is on the same side or the opposite side of a specified pixel inside the convex hull polygon, based on the edge and its extension line, and generate an inside pixel map (IPM) by combining the inside pixel maps (IPM) for all edges of the convex hull polygon, which indicates as a numerical value whether each pixel in the image is inside the convex hull polygon.

[0093] Furthermore, as a more specific process, the convex hull IPM generation unit 3054 may generate an inside pixel map (IPM) that numerically indicates whether or not the pixel is inside the convex hull polygon by synthesizing the inside pixel maps for all sides of the convex hull polygon and selecting the minimum value of the values ​​for each pixel in the inside pixel maps for each side (a set intersection operation described below).

[0094] The polygon IPM generation unit 3055 generates an IPM of the polygon before division by the convex hull division unit 3053, by combining the IPMs of each convex hull polygon divided by the convex hull division unit 3053. In other words, the polygon IPM generation unit 3055 generates an IPM of the polygon before division (called a polygon IPM) by combining the IPMs of multiple convex hull polygons (called a convex hull IPM) generated by the convex hull IPM generation unit 3054.

[0095] That is, the polygon IPM generation unit 3055 generates an inside pixel map (IPM) that indicates, as a numerical value, whether or not each pixel in the image is inside the polygon before being divided by the convex hull division unit 3053, by combining the inside pixel maps (IPM) of the divided convex hull polygons generated by the convex hull IPM generation unit 3054 (convex hull inside pixel map generation unit). The polygon IPM generation unit 3055 is also called a "polygon inside pixel map generation unit." The method by which the polygon IPM generation unit 3055 generates the polygon IPM will be described in detail later.

[0096] The polygon IoU approximation calculation unit 3056 calculates the IoU of two polygons. Strictly speaking, the polygon IoU approximation calculation unit 3056 calculates an approximation of the IoU of two polygons. The polygon IPM generation unit 3055 generates an IPM (polygon IPM) for each of the two polygons. The polygon IoU approximation calculation unit 3056 obtains an IPM equivalent to a union polygon of the two polygons and an IPM equivalent to an intersection polygon of the two polygons based on the two polygon IPMs generated by the polygon IPM generation unit 3055. The polygon IoU approximation calculation unit 3056 calculates an approximation of the IoU of the two polygons using the IPM of the intersection polygon and the IPM of the union polygon.

[0097] In other words, based on the inside pixel map (IPM) generated by the polygon IPM generation unit 3055 (polygon inside pixel map generation unit) for each of two given polygons in the image, the polygon IoU approximation calculation unit 3056 generates an inside pixel map (IPM) that numerically indicates whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map (IPM) that numerically indicates whether or not each pixel in the image is inside the union area of ​​the two polygons, and calculates an approximation of the area of ​​the intersection area based on the inside pixel map (IPM) for the intersection area, calculates an approximation of the area of ​​the union area based on the inside pixel map (IPM) for the union area, and divides the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area, thereby calculating an approximation of the IoU (Intersection over Union) value for the two given polygons. The polygon IoU approximation calculation unit 3056 is also simply called an “IoU approximation calculation unit.” The method by which the polygon IoU approximation calculation unit 3056 calculates an approximation value of the polygon IoU will be described in detail later.

[0098] The polygon IoU calculation device 3051 having the above configuration has the following features. That is, the calculation process in which the convex hull IPM generation unit 3054 (convex hull inner pixel map generation unit) generates an inner pixel map for the convex hull polygon, the calculation process in which the polygon IPM generation unit 3055 (polygon inner pixel map generation unit) generates an inner pixel map (IPM) for the polygon before division based on the inner pixel map (IPM) for the convex hull polygon after division, and the calculation process in which the polygon IoU approximation calculation unit 3056 (IoU approximation calculation unit) calculates an approximation of the IoU value based on the inner pixel map for each of the two polygons are calculated only within the range of a function that is continuous and differentiable within the image area, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments. This makes it possible to learn a model by utilizing the error calculated based on the approximation of the IoU value.

[0099] [Details of processing in the polygon IoU calculation device 3051] Next, the processing in each part of the polygon IoU calculation device 3051 will be described in further detail.

[0100] [How to divide a given polygon into multiple convex polygons] Figures 11 and 12 show the relationship between two given polygons P A and P B 11A is a schematic diagram showing an example of how the convex hull dividing unit 3053 divides each of the polygons P A Polygon P A is an octagon, but is not a convex octagon. Also, (B) of FIG. 11 shows the above polygon P A This is an example of dividing a polygon P into multiple convex hull polygons. A is Polygon P A1 and Polygon P A2 and Polygon P A3 In FIG. 12, (A) is the polygon P before division.B Polygon P B is a pentagon, but not a convex pentagon. Also, (B) of FIG. 12 shows the polygon P B This is an example of dividing a polygon P into multiple convex hull polygons. B is Polygon P B1 and Polygon P B2 The polygon P can be divided into two convex hull polygons. B1 is the convex hull of the polygon P B2 is a convex hull triangle. The convex hull division unit 3053 passes information on the convex hull polygons obtained as a result of the division (for example, information on the coordinates of all vertices of each convex hull polygon) to the convex hull IPM generation unit 3054.

[0101] [How to generate the convex hull IPM] Next, the method in which the convex hull IPM generator 3054 generates the convex hull IPM will be described in detail with reference to FIGS.

[0102] FIG. 13 shows a polygon P, which is an example of a convex hull polygon to be processed by the convex hull IPM generation unit 3054. A2 13 is a schematic diagram showing a polygon P A2 In Fig. 11, polygon P A This is one of the convex hull polygons obtained by dividing the convex hull polygon P A2 The four edges (straight lines, line segments) that make up the line are L1, L2, L3, and L4.

[0103] FIG. 14 shows the convex hull polygon P A2 1 is a schematic diagram for explaining the process of processing by the convex hull IPM generation unit 3054 to generate an IPM of the convex hull polygon P A2 To generate the IPM of A2 Figure 14 shows the process of calculating the IPM for edge (straight line) L2 as an example.

[0104] In FIG. 14, an image 3501 is a convex hull polygon PA2 When you look at the edge (straight line) L2, the straight line L 2P is parallel to the edge (straight line) L2 and is the convex hull polygon P A2 The line passing through a certain point (pixel) that exists inside the convex hull polygon P A2 For example, the pixel of the center point of the convex hull polygon P A2 The average position of the coordinates of all vertices of the convex hull polygon P A2 The center point of the convex hull polygon P A2 A straight line L passing through the center point of 2P may be called the "center line". ij (i and j are horizontal and vertical coordinates in the image 3501, respectively) to find the IPM value at pixel (point) P ij From the above line L 2P Let the foot of the perpendicular line be pixel (point) P0. ijs is the pixel (point) P ij is the distance from to pixel (point) P0. Also, d s is the line L2 and the line L 2P At this time, the convex hull IPM generation unit 3054 calculates the distance between the pixel P ij IPM value inIPM ij is calculated by the following formula (2). Note that formula (2) has the same meaning as formula (1) used in the first embodiment. Note that k in formula (2) is a hyperparameter.

[0105]

number

[0106] The pixel P calculated by the convex hull IPM generation unit 3054 ij IPM value of IPM ij is the following value: IPM ij is greater than 0.0 and less than 1.0. Also, pixel P ijHowever, for the line of interest (line L2 in FIG. 14), the convex hull polygon (convex hull polygon P A2 ), if it is on the same side as a given point (e.g., the center point) in the IPM ij takes a value close to 1.0. Pixel P ij However, for the line of interest (line L2 in FIG. 14), the convex hull polygon (convex hull polygon P A2 ), if it is on the opposite side of a given point (e.g., the center point) in the IPM ij takes a value close to 0.0. Note that in the entire area of ​​image 3501, IPM ij The value of IPM does not change discretely but changes continuously. ij The value of is differentiable. Also, IPM ij The value of IPM changes rapidly between values ​​close to 0.0 and 1.0 in the vicinity of the line of interest (line L2 in FIG. 14). The degree of rapidity of this change depends on the value of k in equation (2). In other words, ij The value of is calculated based on the convex hull polygon (the convex hull polygon P A2 ) on the same side as the specified point, it is the side closer to 1.0, and on the opposite side to the specified point, it is the side closer to 0.0.

[0107] The higher the value of k, the greater the IPM near the edge under consideration. ij The value of k changes rapidly. In other words, the higher the value of k, the more accurate the IoU value that the polygon IoU calculation device 3051 can obtain. However, if the value of k is set too high, the IPM ijThe differential value of the value of k in the vicinity of the side of interest becomes too high, and in some cases, it may cause an overflow in the calculation during the machine learning process. Whether or not an overflow occurs in the calculation also depends on the calculation environment (bit length of the calculation element, etc.). According to the results of experiments by the inventors of the present application, when calculations are performed in a calculation environment with a 32-bit width that is widely used, the polygon IoU calculation device 3051 can obtain a more accurate IoU without causing an overflow in the calculation by using k in the range of 30≦k≦50.

[0108] The convex hull IPM generation unit 3054 generates a convex hull polygon (in FIG. 14, the convex hull polygon P A2 ) for each edge, we use the above formula (2) to calculate the IPM ij The IPM value calculated by the convex hull IPM generation unit 3054 is calculated. ij The value of is a base value for determining the degree to which each pixel is contained within the convex hull polygon.

[0109] Each of Fig. 15, Fig. 16, Fig. 17, and Fig. 18 shows the convex hull polygon P A2 1 is a schematic diagram showing the IPM calculated by the convex hull IPM generating unit 3054 for the sides L1, L2, L3, and L4 of the pixel P ij The IPM value of P ij,L1 ,P ij,L2 ,P ij,L3 ,P ij,L4 In FIG. 15, in the hatched area above side L1 and its extension, ij,L1 The value of is close to 1.0. In the unhatched area below side L1 and its extension, ij,L1 The value of is close to 0.0. In FIG. 16, in the hatched area on the upper right side of side L2 and its extension, ij,L2 The value of is close to 1.0. In the unhatched area on the lower left side of side L2 and its extension, ij,L2 The value of is close to 0.0. In FIG. 17, in the hatched area below side L3 and its extension,ij,L3 The value of is close to 1.0. In the unhatched area above side L3 and its extension, ij,L3 The value of is close to 0.0. In FIG. 18, in the hatched area on the upper left side of side L4 and its extension, ij,L4 The value of is close to 1.0. In the unhatched area on the lower right side of side L4 and its extension, ij,L4 The value of is close to 0.0. Note that in the entire area of ​​the image 3501, the IPM ij,L1 ,IPM ij,L2 ,IPM ij,L3 , and IPM ij,L4 Each value of is continuous and differentiable.

[0110] The convex hull IPM generation unit 3054 calculates the pixel P ij IPM is the IPM value for the convex hull polygon of ij The edges of the target convex hull polygon are generally expressed as L1, L2, . . . , L M (M is the number of sides of the convex hull polygon), the convex hull IPM generation unit 3054 generates the IPM ij,L1 ,IPM ij,L2 ,···,IPM ij,LM The IPM value of the convex hull polygon is calculated by performing an intersection operation of the ij Request.

[0111] Here, the intersection operation is an IPM that has values ​​close to 0.0 or close to 1.0 in most areas. ij,L1 ,IPM ij,L2 ,···,IPM ij,LM Regarding IPM ij,L1 ,IPM ij,L2 ,···,IPM ij,LM is close to 1.0, and otherwise returns a value close to 0.0 as the operation result.

[0112] The convex hull IPM generating unit 3054 realizes the intersection operation by, for example, calculating the following formula (3): In formula (3), minimum( ) is a function that returns the minimum value.

[0113]

number

[0114] Alternatively, the convex hull IPM generator 3054 may use multiplication as in the first embodiment. That is, the convex hull IPM generator 3054 may realize the intersection operation by performing the calculation of the following equation (4).

[0115]

number

[0116] The convex hull IPM generating unit 3054 may perform the intersection operation using a calculation process other than equation (3) or equation (4). When the convex hull IPM generating unit 3054 performs the intersection operation using equation (3), the amount of calculation required for the intersection operation can be reduced compared to when equation (4) is used. In addition, according to the experimental results of the present inventors, the intersection operation using equation (3) can calculate a more accurate IoU value compared to when equation (4) is used. This is considered to be because the error in the numerical values ​​used to calculate the IoU value (such as the degree to which the IPM value for pixels inside the convex hull polygon deviates below 1.0) is amplified by performing multiple multiplications.

[0117] The calculation shown in formula (3) is not differentiable in the entire region of the image 3501. However, even if a non-differentiable region exists locally in the image 3501, Experiments have confirmed that there are no problems with machine learning models such as neural networks.

[0118] That is, the convex hull IPM generation unit 3054 creates an IPM for the convex hull polygon by performing an intersection operation on the IPMs generated for each edge of the convex hull polygon. The convex hull IPM generation unit 3054 passes the generated IPM for the convex hull polygon to the polygon IPM generation unit 3055.

[0119] [How to generate a polygon IPM based on a convex hull polygon IPM] The polygon IPM generation unit 3055 receives the IPMs of the convex hull polygons from the convex hull IPM generation unit 3054, and by combining them, generates the IPM of the original polygon before it was divided by the convex hull division unit 3053. Specifically, the polygon IPM generation unit 3055 generates the IPM of the original polygon by performing a union operation on the IPMs of each convex hull polygon resulting from the division. The K convex hull polygons obtained by dividing the original polygon P are denoted as P1, P2,..., P K Let pixel P in the image ij Convex hull polygons P1, P2, . . . , P K The IPM values ​​of ij,P1 ,IPM ij,P2 ,···,IPM ij,PK The polygon IPM generation unit 3055 generates the IPM ij,P1 ,IPM ij,P2 ,···,IPM ij,PK The IPM of the original polygon P is obtained by performing a union operation on ij Request.

[0120] Here, the union operation is an operation that combines IPMs that have values ​​close to 0.0 or close to 1.0 in most areas. ij,P1 ,IPM ij,P2 ,···,IPM ij,PK Regarding IPM ij,P1 ,IPM ij,P2 ,···,IPM ij,PK is close to 0.0, and otherwise returns a value close to 1.0 as the operation result.

[0121] The polygon IPM generating unit 3055 realizes the union operation by, for example, calculating the following formula (5): In formula (5), maximum( ) is a function that returns the maximum value.

[0122]

number

[0123] Alternatively, the polygon IPM generator 3055 may implement the union operation using arithmetic operations (addition and multiplication). The union operation for two convex hull polygons (e.g., P1 and P2) can be implemented by the following formula (6).

[0124]

number

[0125] Using the above formula (6), we can perform a union operation on K convex hull polygons for any K (K≧2), as shown in the following formula (7).

[0126]

number

[0127] That is, the polygon IPM generation unit 3055 performs a union operation on the convex hull polygons P1 and P2 based on equation (6), then performs a similar union operation on the union polygon (P1∪P2) and the convex hull polygon P3, and so on, ultimately generating the union polygon (P1∪P2∪∪P K ) we can obtain the IPM for this union polygon (P1 ∪ P2 ∪ ∪ P K ) is the polygon P before being divided by the convex hull division unit 3053.

[0128] The polygon IPM generating unit 3055 can generate an IPM for the polygon P by using either a method using equation (5), or a method using equations (6) and (7), or by other methods. Note that the calculation shown in equation (5) is not differentiable in the entire region of the image 3501. However, it has been confirmed through experiments that even if a non-differentiable region exists locally in the image 3501, this does not pose a problem in machine learning of a model such as a neural network.

[0129] It has been experimentally confirmed that the method using equation (5) requires less computational effort than the methods using equations (6) and (7), and that the accuracy of the final IoU approximation is higher.

[0130] That is, the polygon IPM generation unit 3055 can obtain the IPM of the polygon before division by performing a union operation based on the IPM of each convex hull polygon resulting from division by the convex hull division unit 3053. A and Polygon P B In order to obtain the error between the polygon P A and Polygon P B The IPM of both polygons is passed to the polygon IoU approximation calculation unit 3056.

[0131] [How to calculate an approximation of polygon IoU as the error between two polygons] The polygon IoU approximation calculation unit 3056 receives the IPM of each of the two polygons for which an error is to be calculated from the polygon IPM generation unit 3055. To calculate the IoU of these two polygons, the polygon IoU approximation calculation unit 3056 first calculates the IPM of the union polygon and the IPM of the intersection polygon of these two polygons. The method of calculating the IPM of the union polygon of two polygons has already been described as part of the processing of the polygon IPM generation unit 3055. The method of calculating the IPM of the intersection polygon of two polygons has already been described as part of the processing of the convex hull IPM generation unit 3054. Specifically, the IPM of the union polygon of two polygons can be calculated using equation (5) or equation (6), etc. The IPM of the intersection polygon of two polygons can be calculated using equation (3) or equation (4), etc.

[0132] The IPM has information on the IPM value for each pixel. Therefore, the polygon IoU approximation calculation unit 3056 calculates the area (approximation) of the union polygon of two polygons as the sum of the values ​​of each pixel in the IPM of the union polygon of the two polygons. The polygon IoU approximation calculation unit 3056 also calculates the area (approximation) of the intersection polygon of two polygons as the sum of the values ​​of each pixel in the IPM of the intersection polygon of the two polygons. The unit of the area calculated here is the number of pixels. Then, the polygon IoU approximation calculation unit 3056 calculates the IoU approximation for two pixels given by the following formula (8).

[0133]

number

[0134] [Processing procedure for calculating the IoU approximation] 19 is a flowchart showing the procedure of a process in which polygon IoU calculation device 3051 obtains an approximation value of IoU of two given polygons. As a premise of this process, polygon IoU calculation device 3051 is given information on the shapes and positions of a first polygon and a second polygon (for example, coordinate values ​​on an image of all vertices). Polygon information acquisition unit 3052 acquires information on these polygons. The process procedure will be described below with reference to this flowchart.

[0135] First, in step S1, the convex hull division unit 3053 divides the first polygon into convex hull polygons. The division process has been described above. If it has already been determined that the first polygon is a convex hull polygon and division is unnecessary, the given first convex hull polygon itself may be used as the convex hull polygon resulting from the division in this step.

[0136] Next, in step S2, the convex hull IPM generation unit 3054 calculates the IPM of each convex hull polygon obtained by the division in step S1. The method by which the convex hull IPM generation unit 3054 generates the IPM of the convex hull polygon is as already described.

[0137] Next, in step S3, the polygon IPM generation unit 3055 obtains the IPM of the original polygon before division (the first polygon) by combining the IPMs of the convex hull polygons obtained in step S2. As already explained, the polygon IPM generation unit 3055 can combine the IPMs of multiple convex hull polygons by using a union operation.

[0138] Next, in step S4, the convex hull division unit 3053 divides the second polygon into convex hull polygons. If it can be determined that the second polygon is already a convex hull polygon and division is unnecessary, the given second convex hull polygon itself may be used as the convex hull polygon resulting from the division in this step.

[0139] Next, in step S5, the convex hull IPM generation unit 3054 calculates the IPM of each of the convex hull polygons obtained by the division in step S4.

[0140] Next, in step S6, the polygon IPM generation unit 3055 obtains the IPM of the original polygon before division (the second polygon) by combining the IPMs of the convex hull polygons obtained in step S5.

[0141] Next, in step S7, the polygon IoU approximation calculation unit 3056 uses the IPM of the first polygon obtained in step S3 and the IPM of the second polygon obtained in step S6 to obtain the IPM of the union area of ​​the first polygon and the second polygon. To generate the IPM of the union area, the polygon IoU approximation calculation unit 3056 uses the union operation described above.

[0142] Next, in step S8, the polygon IoU approximation calculation unit 3056 uses the IPM of the first polygon obtained in step S3 and the IPM of the second polygon obtained in step S6 to obtain the IPM of the intersection area of ​​the first polygon and the second polygon. To generate the IPM of the intersection area, the polygon IoU approximation calculation unit 3056 uses the above-mentioned intersection operation.

[0143] Next, in step S9, polygon IoU approximation calculation unit 3056 divides the sum of pixel values ​​of the IPM of the intersection area calculated in step S8 (the sum of all pixels) by the sum of pixel values ​​of the IPM of the union area calculated in step S7 (the sum of all pixels). The sum of pixel values ​​of the IPM of the intersection area is an approximation of the area of ​​the intersection area. In addition, the sum of pixel values ​​of the IPM of the union area is an approximation of the area of ​​the union area. The result of the above division in this step is an approximation of the IoU of the two given polygons.

[0144] [Implementation using a computer program] At least a part of the functions of the polygon IoU calculation device 3051 of this embodiment can be realized by using a computer and a program. Note that the realization by using a computer and a program is as described in the first embodiment with reference to FIG.

[0145] [Using IoU values ​​in neural network training] The IoU value (IoU approximation) obtained by the above method can be used in training the neural network. Specifically, when training the neural network, the error function (1-IoU approximation) can be used to train the neural network. The neural network training itself can be performed using a method of backpropagating the above error to the neural network. As already explained, the error function (1-IoU approximation) is a differentiable function, or a function that is not differentiable (has locally non-differentiable parts) but can be used in training the neural network. When training, the error between a hypothetical polygon estimated based on input data included in the training data and a correct answer polygon as correct answer data associated with the input data in the training data is calculated.

[0146] By calculating the error using the method of this embodiment, the constraints that the conventional technology had regarding polygons to be processed can be eliminated or alleviated. For example, in the method of this embodiment, there is no constraint on the number of vertices of the polygon to be processed (for example, not limited to a rectangle). In addition, in the method of this embodiment, the polygon to be processed is not limited to a convex polygon (convex hull polygon). In addition, in the method of this embodiment, the number of vertices of two polygons to be the subject of error calculation may be different from each other. Therefore, for example, both a data set in which the correct answer is given as a 4-vertex polygon and a data set in which the correct answer is given as a 16-vertex polygon can be used as correct answer data, and these two can be mixed as correct answers. In other words, when preparing correct answer data for learning, additional work such as correction to match the number of vertices is not required.

[0147] In addition, when the error is calculated by the method of this embodiment, it becomes possible to use a random crop technique to expand the training data. Random cropping is a technique for artificially increasing the amount of training data, and randomly cuts out a part of the image included in the training data. In other words, by performing random cropping, one or more of the top, bottom, right, and left of the image are cut off. In a polygonal object detection task, when random cropping is performed on a correct answer image included in the training data, a part of the correct answer polygon may be cut off (the cropping line may divide the correct answer polygon). This may change the number of vertices of the correct answer polygon, so there are restrictions on the use of random cropping when using conventional techniques. However, when the error between polygons is calculated by this embodiment, there is no restriction on the number of vertices of the polygon, so the random crop technique can be used without restrictions. In other words, this embodiment can automatically increase the variation in the shape of the polygons included in the training data.

[0148] [Experimental Verification] For the task of detecting character string regions (polygons) contained in a planar image, an experiment was conducted using the error calculation method (IoU approximation calculation method) of this embodiment during neural network training. As a result, it was confirmed that the accuracy of detecting character string regions was improved by about 2% (about 2 points) compared to the conventional technology.

[0149] [Third embodiment] Next, a third embodiment of the present invention will be described. Note that the matters already described in the previous embodiments may not be described below. Here, the matters specific to this embodiment will be mainly described. This embodiment is an invention of an area detection device that uses the polygon IoU calculation device 3051 of the second embodiment.

[0150] 20 is a block diagram showing a schematic functional configuration of the area detection device according to this embodiment. As shown in the figure, the area detection device 2 includes an image input unit 201, an area estimation unit 202, a result output unit 203, a learning data supply unit 204, and an error calculation unit 205. Each functional unit of the area detection device 2 of this embodiment can also be realized by, for example, a computer and a program. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0151] The image input unit 201, the area estimation unit 202, the result output unit 203, and the learning data supply unit 204 in this embodiment have the same functions as the image input unit 101, the area estimation unit 102, the result output unit 103, and the learning data supply unit 104 in the first embodiment (see FIG. 1), respectively. However, in this embodiment, the area to be detected is not limited to a convex hull polygon, and may be any polygon. Accordingly, the correct area included in the learning data is also not limited to a convex hull polygon, and may be any polygon.

[0152] When the area detection device 2 operates in the learning mode, the error calculation unit 205 calculates the error between the figure of the area output by the area estimation unit 202 as the estimation result and the figure of the correct area supplied by the learning data supply unit 204. As mentioned above, these areas are not limited to convex hull polygons. The error calculated by the error calculation unit 205 is used to adjust (optimize) the internal parameters of the neural network of the area estimation unit 202 by the error backpropagation method.

[0153] As shown in Fig. 20, the error calculation unit 205 includes a polygon IoU calculation device 3051 therein. This polygon IoU calculation device 3051 has been described in detail in the second embodiment. In this embodiment, the polygon IoU calculation device 3051 calculates (an approximation of) the IoU between the figure (polygon) of the area output by the area estimation unit 202 as an estimation result and the figure (polygon) of the correct area supplied by the learning data supply unit 204. The error calculation unit 205 uses (the approximation of) the IoU calculated by the polygon IoU calculation device 3051 to calculate and output (1-IoU approximation) as an error value.

[0154] That is, the region estimation unit 202 includes a machine learning model and estimates a region having a specific feature in an image by inputting an image passed from the outside to the machine learning model. The learning data supply unit 204 supplies a pair of a learning image and a correct region corresponding to the learning image for learning the machine learning model included in the region estimation unit 202. The error calculation unit 205 calculates an error between an estimated region, which is a result of estimation by the region estimation unit 202 based on the learning image supplied by the learning data supply unit 204, and the correct region supplied by the learning data supply unit 204 in correspondence with the learning image. In addition, both the figure corresponding to the estimated region and the figure corresponding to the correct region are arbitrary polygons. In addition, the error calculation unit 205 includes an IoU calculation device 3051, and calculates the error (e.g., 1-IoU) based on an approximation of the IoU value calculated by the IoU calculation device 3051 for a polygon that is a figure corresponding to the estimated area and a polygon that is a figure corresponding to the correct answer area.

[0155] [Implementation using a computer program] At least a part of the functions of the area detection device 2 of this embodiment can be realized by using a computer and a program. Note that the realization by using a computer and a program is as described in the first embodiment with reference to FIG.

[0156] With the configuration of this embodiment described above, the area detection device 2 can learn a machine learning-enabled model (e.g., a neural network) that the area estimation unit 202 has internally, based on the error value calculated by the error calculation unit 205. That is, the area detection device 2 according to this embodiment can accurately detect an area formed as a polygon of any shape, not limited to a convex hull polygon.

[0157] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. Note that the matters already described in the previous embodiments may not be described below. Here, the matters specific to this embodiment will be mainly described. This embodiment is an invention of an image processing device that uses the polygon IoU calculation device 3051 of the second embodiment.

[0158] 21 is a block diagram showing a schematic functional configuration of an image processing device according to this embodiment. As shown in the figure, the image processing device 3 includes an image input unit 301, an estimation unit 302, a result output unit 303, a learning data supply unit 304, and an error calculation unit 305. Each functional unit of the image processing device 3 of this embodiment can also be realized by, for example, a computer and a program. Furthermore, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0159] The functions of the image input unit 301, the estimation unit 302, the result output unit 303, the learning data supply unit 304, and the error calculation unit 305 are similar to the functions of the image input unit 201, the area estimation unit 202, the result output unit 203, the learning data supply unit 204, and the error calculation unit 205 in the third embodiment, respectively. The following mainly describes the differences between the third and fourth embodiments.

[0160] While the area detection device 2 of the third embodiment is a device for detecting an area having a specific characteristic (for example, an area of ​​a character string), the image processing device 3 of this embodiment is not limited to the task of area detection, but performs any estimation on an input figure. In other words, the estimation unit 302 of this embodiment performs any estimation process, not limited to the task of detecting an area having a specific characteristic. However, the result estimated by the estimation unit 302 is a polygon in an image. Aside from the task of detecting an area having a specific characteristic, the estimation unit 302 can execute a task of generating a figure (polygon) that satisfies a condition represented by input information (for example, text information, numerical information, image information, composite information of two or more of text, numerical information, and image, or other general information). One example of this is a task of inputting a map (topographical map) and the number of vertices of a polygon as constraint condition data, and outputting the shape of a lake or pond included in the map (shape when viewed in a plan view) by approximating the shape with a polygon having a specified number of vertices.

[0161] The estimation unit 302 is realized by having a model capable of machine learning, such as a neural network, built in. The parameters held in the estimation unit 302 can be adjusted (optimized) based on the error calculated by the error calculation unit 305.

[0162] As shown in Fig. 21, the error calculation unit 305 includes a polygon IoU calculation device 3051 therein. This polygon IoU calculation device 3051 has been described in detail in the second embodiment. In this embodiment, the polygon IoU calculation device 3051 calculates (an approximation of) the IoU between a figure (polygon) of an area output by the estimation unit 302 as an estimation result and a figure (polygon) of a correct answer area supplied by the learning data supply unit 304. The error calculation unit 305 uses (the approximation of) the IoU calculated by the polygon IoU calculation device 3051 to calculate and output (1-IoU approximation) as an error value.

[0163] That is, the estimation unit 302 includes a machine learning model and outputs an estimated polygon as a result of estimating polygons in an image by inputting an image passed from the outside into the machine learning model. The learning data supply unit 304 supplies a pair of a learning image and a ground truth polygon corresponding to the learning image for learning the machine learning model included in the estimation unit 302. The error calculation unit 305 calculates an error between the estimated polygon, which is a result of estimation by the estimation unit based on the learning image supplied by the learning data supply unit 304, and the ground truth polygon supplied by the learning data supply unit 304 corresponding to the learning image. The error calculation unit 305 includes an IoU calculation device 3051 and calculates the error (for example, 1-IoU) based on an approximation of the IoU value calculated by the IoU calculation device 3051 for the estimated polygon and the ground truth polygon.

[0164] [Implementation using a computer program] At least a part of the functions of the image processing device 3 of this embodiment can be realized by using a computer and a program. Note that the realization by using a computer and a program is as described in the first embodiment with reference to FIG.

[0165] With the configuration of this embodiment described above, the image processing device 3 can learn a machine-learnable model (e.g., a neural network) that the estimation unit 302 has internally, based on the error value calculated by the error calculation unit 305. That is, the image processing device 3 according to this embodiment can accurately perform a task in which the estimation result is not limited to a convex hull polygon, but a polygon of any shape.

[0166] According to at least one of the above-described embodiments, it is possible to calculate the IoU or its approximation for two given polygons. Moreover, the IoU or its approximation is calculated as a value of a function that is differentiable for most of the area of ​​the image. Even if there is a locally non-differentiable part in the image for the function for calculating the IoU or its approximation, due to the characteristics of the maximum function or minimum function, learning of a neural network or the like can be performed without problems using such an error. In other words, it is possible to learn a machine learning model that treats a predetermined polygon (for example, any polygon with no limitations on calculation, which may be a polygon with a complex shape) as the correct answer. This makes it possible to detect areas in the image and perform other image-related processing with high accuracy. [Industrial Applicability]

[0167] The present invention can be used, for example, to perform image processing. Specifically, the present invention can be used for machine learning of models for image processing. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0168] 1,2 Area detection device 3. Image Processing Device 101 Image input unit 102 Area estimation part 103 Result output section 104 Learning Data Supply Unit 105 Error calculation section 205 Error calculation section 301 Image Input Unit 302 Estimation section 303 Result output section 304 Learning Data Supply Unit 305 Error calculation section 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus 1051 Virtual Pixel Generation Unit 1052 Inclusion Judgment Section 1053 Inclusion Judgment Integration Department 1054 Error function value calculation unit 3051 Polygon IoU Calculation Device (IoU Calculation Device) 3052 Polygon Information Acquisition Unit 3053 Convex hull division 3054 Convex Hull IPM Generation Unit (Convex Hull Inner Pixel Map Generation Unit) 3055 Polygon IPM Generation Unit (Polygon Inner Pixel Map Generation Unit) 3056 Polygon IoU Approximation Calculation Unit (IoU Approximation Calculation Unit)

Claims

1. a convex hull division unit for dividing a polygon in a given image to obtain one or more divided convex hull polygons; a convex hull inside pixel map generator that generates an inside pixel map (IPM) for each of the divided convex hull polygons, the IPM indicating, as a numerical value, whether or not each pixel in the image is inside the convex hull polygon; a polygon inside pixel map generator that generates an inside pixel map by combining the inside pixel maps of the convex hull polygons after division generated by the convex hull inside pixel map generator, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image exists inside the polygon before being divided by the convex hull division unit; an IoU approximation calculation unit that generates an inside pixel map for an intersection area of ​​two given polygons in the image, the inside pixel map indicating as a numerical value whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map indicating as a numerical value whether or not each pixel in the image is inside the union area of ​​the two polygons, based on the inside pixel map generated by the polygon inside pixel map generation unit for each of the two polygons given in the image, and calculates an approximation of an IoU (Intersection over Union) value for the two given polygons by calculating an approximation of an area of ​​the intersection area based on the inside pixel map for the intersection area, calculating an approximation of the area of ​​the union area based on the inside pixel map for the union area, and dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area; Equipped with a calculation process in which the convex hull inside pixel map generation unit generates an inside pixel map for the convex hull polygon, a calculation process in which the polygon inside pixel map generation unit generates an inside pixel map for the polygon before division based on the inside pixel map for the convex hull polygon after division, and a calculation process in which the IoU approximation calculation unit calculates an approximation of the IoU value based on the inside pixel maps for each of the two polygons are calculated only within the range of a function that is continuous and differentiable within an image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments. IoU calculation device.

2. the convex hull inside pixel map generation unit generates an inside pixel map for each side of the convex hull polygon to be processed, the inside pixel map numerically indicating whether each pixel in the image is on the same side or the opposite side of a specified pixel inside the convex hull polygon with respect to the side and an extension line of the side as a reference, and generates an inside pixel map numerically indicating whether each pixel in the image is inside the convex hull polygon or not by combining the inside pixel maps for all sides of the convex hull polygon. The IoU calculation device according to claim 1.

3. the convex hull inner pixel map generation unit generates an inner pixel map that numerically indicates whether or not the pixel is inside the convex hull polygon by selecting a minimum value for each pixel in the inner pixel map for each side when synthesizing the inner pixel maps for all sides of the convex hull polygon; The IoU calculation device according to claim 2 .

4. A region estimation unit that includes a machine learning model and inputs an image passed from the outside into the machine learning model to estimate a region having a specific feature in the image; A learning data supply unit that supplies pairs of learning images and correct answer regions corresponding to the learning images for learning the machine learning model provided in the region estimation unit; an error calculation unit that calculates an error between an estimated area that is a result of estimation by the area estimation unit based on the learning image supplied by the learning data supply unit, and the correct area that is supplied by the learning data supply unit in correspondence with the learning image; An area detection device comprising: a figure corresponding to the estimated area and a figure corresponding to the correct answer area are both arbitrary polygons, The error calculation unit includes the IoU calculation device according to any one of claims 1 to 3, and calculates the error based on an approximation of the IoU value calculated by the IoU calculation device for a polygon that is a figure corresponding to the estimated area and a polygon that is a figure corresponding to the correct answer area. Area detection device.

5. an estimation unit that includes a machine learning model and that inputs an image provided from outside into the machine learning model to estimate polygons in the image and output an estimated polygon; a learning data supply unit that supplies pairs of learning images and ground truth polygons corresponding to the learning images for learning the machine learning model included in the estimation unit; an error calculation unit that calculates an error between the estimated polygon that is a result of estimation by the estimation unit based on the learning image supplied by the learning data supply unit, and the ground truth polygon that is supplied by the learning data supply unit in correspondence with the learning image; An image processing device comprising: The error calculation unit includes an IoU calculation device according to any one of claims 1 to 3, and calculates the error based on an approximation of the IoU value calculated by the IoU calculation device for the estimated polygon and the ground truth polygon. Image processing device.

6. a convex hull division unit for dividing a polygon in a given image to obtain one or more divided convex hull polygons; a convex hull inside pixel map generator that generates an inside pixel map (IPM) for each of the divided convex hull polygons, the IPM indicating, as a numerical value, whether or not each pixel in the image is inside the convex hull polygon; a polygon inside pixel map generator that generates an inside pixel map by combining the inside pixel maps of the convex hull polygons after division generated by the convex hull inside pixel map generator, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image exists inside the polygon before being divided by the convex hull division unit; an IoU approximation calculation unit that generates an inside pixel map for an intersection area of ​​two given polygons in the image, the inside pixel map indicating as a numerical value whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map indicating as a numerical value whether or not each pixel in the image is inside the union area of ​​the two polygons, based on the inside pixel map generated by the polygon inside pixel map generation unit for each of the two polygons given in the image, and calculates an approximation of an IoU (Intersection over Union) value for the two given polygons by calculating an approximation of an area of ​​the intersection area based on the inside pixel map for the intersection area, calculating an approximation of the area of ​​the union area based on the inside pixel map for the union area, and dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area; Equipped with a calculation process in which the convex hull inside pixel map generation unit generates an inside pixel map for the convex hull polygon, a calculation process in which the polygon inside pixel map generation unit generates an inside pixel map for the polygon before division based on the inside pixel map for the convex hull polygon after division, and a calculation process in which the IoU approximation calculation unit calculates an approximation of the IoU value based on the inside pixel maps for each of the two polygons are calculated only within the range of a function that is continuous and differentiable within an image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments. A program for causing a computer to function as an IoU calculation device.