Region detection device and program

Abstract

To provide an area detection device and a program capable of detecting an area (polygon) of an appropriate shape.SOLUTION: The estimation unit inputs the image to the model and outputs a plurality of numerical values output from the model as an estimation result. The conversion unit determines, based on the plurality of numerical values, positions on the image of the polygon center line in a state in which (N / 2-1) (N is an even number of 4 or more) line segments are sequentially connected so as not to branch, and determines, at each of both end points (two points) of the polygon center line and a connection point (N / 2-2) which is a point at which one line segment and another line segment are connected in the polygon center line, the positions of two points at positions extended in the width direction with the polygon center line as the center are determined as vertex positions, and an estimated polygon formed by connecting the determined vertices by sides is output. When performing machine learning of the model, backpropagation is performed based on the IoU error calculated by the error calculation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an area detection device and a program. [Background technology]

[0002] Research is being conducted into technologies for detecting areas with specific characteristics in an image. A specific example is technology for detecting the area of ​​an object in an image. Another specific example is the automatic detection of signboards with well-designed characters in a street photo.

[0003] It is not obvious what shape of region should be detected when detecting a region in an image. Traditionally, much research has been done on detecting an object region as an unrotated rectangle in conventional technology. An "unrotated rectangle" refers to a state in which the top and bottom sides of the rectangular region corresponding to the object region are parallel to the top and bottom sides of the entire image (the entire image is rectangular), and the left and right sides of the rectangular region corresponding to the object region are parallel to the right and left sides of the entire image.

[0004] Recent research has proposed methods for detecting polygon-shaped object regions within an image. For example, when detecting regions such as "text written on stylishly designed signs" found around town, if the region is represented using a rectangle, it may contain many non-text areas, making it difficult to accurately represent the text region. However, even in such cases, it is possible to successfully detect text regions by using a method for detecting polygon-shaped regions. A polygon is an N-sided polygon consisting of three or more sides. In other words, region detection using polygon shapes has become increasingly popular in recent years.

[0005] There are two methods for representing polygon shapes. The first method represents a polygon as a list of the coordinates of the vertices that make up the polygon. If a polygon on a plane has N vertices, the polygon is represented as a list of N two-dimensional coordinates. The second method represents a polygon as a list of Bezier curve parameters that represent the polygon area. Both the first and second methods represent a polygon as a list of coordinates on the image plane. When detecting areas using a neural network, the structure and optimization methods of the neural network are equivalent between the first and second methods. From this common perspective, there is no particular need to distinguish between the first and second representation methods. In other words, a list of coordinates on the image plane can generally be treated as polygon area parameters.

[0006] In conventional technology, a typical training method for a neural network to detect polygon regions is as follows: In neural network training, a large number of pairs of training images (input images) and data (ground truth data) representing the object region (ground truth region) contained in the images are prepared. Next, based on the input training images, the internal parameters are updated so as to reduce the error between the object region (estimated region) estimated by the neural network to be trained using the internal parameters at that time and the given ground truth region. By repeating this process, the values ​​of the internal parameters are gradually optimized.

[0007] Non-Patent Document 3 describes an example of neural network training for detecting polygonal object regions. Specifically, in the case of a neural network that detects polygonal object regions, the error calculation uses the distance between the polygon region parameters output by the neural network and the correct polygon region parameters, as described in Non-Patent Document 3. A specific example of the calculated error is smooth L1 loss. When using this method, the number of polygon region parameters output by the neural network must be the same as the number of polygon region parameters of the correct polygon. In other words, when training a neural network that detects polygons with 16 vertices, the training data (correct data) must also be data that represents a polygon with 16 vertices.

[0008] On the other hand, there is also a neural network training method for detecting polygon-shaped object regions that uses the polygon IoU approximation value as an error function. IoU stands for Intersection over Union and is an index that represents the degree to which two regions overlap. IoU is a value obtained by dividing the area (area, etc.) of the common part (intersection region) of two regions by the union area of ​​the two regions. If two regions completely overlap with no overlap, the IoU value of the two regions is 1.0. If two regions do not overlap at all (the area of ​​the common part is zero), the IoU value of the two regions is 0.0. When IoU is used as a value representing the error between regions in neural network training, the IoU value is given a negative sign, for example.

[0009] It is possible to calculate the polygon IoU approximation even between two polygons with different numbers of vertices. Therefore, when used for neural network training, there is an advantage that the number of parameters of the estimated polygon area output by the neural network does not need to be the same as the number of parameters of the polygon area given as the correct answer.

[0010] In machine learning methods, the greater the amount (number) of training data, the greater the learning effect and the higher the accuracy of region detection. In methods that use polygon IoU approximations to train neural networks, there are no restrictions on the number of polygon region parameters or expression formats, so it is possible to use training data for polygon region detection in a variety of formats that exist in the world. In other words, the advantage is that it is easy to use a larger amount of training data, making it easier to create a highly accurate region detector.

[0011] Non-Patent Document 1 describes a PIoU error function (PIoU loss) as an error function for calculating an error regarding a rotated rectangle. The PIoU error function is a loss function for detecting rotated objects, and is formulated to perform accurate rotated bounding box regression by utilizing both the angle and IoU. The method described in Non-Patent Document 1 uses a differentiable function to calculate an approximation of the number of pixels present inside a rotated rectangle, and is a method that can calculate an error value similar to IoU.

[0012] Furthermore, Non-Patent Document 2 describes a technique for enabling area detection in more general polygonal shapes. However, the technique in Non-Patent Document 2 uses a function (specifically, Smooth L1 Loss) that calculates an error based on a simple distance measure between polygon vertices as the error function used for training the neural network. [Prior art documents] [Non-patent literature]

[0013] [Non-Patent Document 1] Zhiming Chen,Kean Chen,Weiyao Lin,John See,Hui Yu,Yan Ke,Cong Yang,"PIoU Loss: Towards Accurate Oriented Object Detection in ComplexEnvironments", European Conference on Computer Vision(ECCV), 2020, https: / / arxiv.org / pdf / 2007.09584.pdf [Non-patent document 2] Xiang Zhang, Yongwen Su, Subarna Tripathi, Zhuowen Tu, "Text Spotting Transformers ", CVPR 2022, Computer Vision and Pattern Recognition (cs.CV), April 2022, arXiv:2204.01918 [cs.CV], https: / / arxiv.org / pdf / 2204.01918.pdf [Non-patent document 3] Xiang Zhang, Yongwen Su, Subarna Tripathi, Zhuowen Tu, Text Spotting Transformers, CVPR 2022, arXiv:2204.01918 [cs.CV], 2022. Summary of the Invention [Problem to be solved by the invention]

[0014] However, in the method of training a neural network using polygon IoU approximation, the training is unstable compared to conventional distance-based optimization, and there are cases where the training does not proceed as intended.

[0015] In other words, because the polygon IoU approximation is a similarity index for the entire polygon region, when using the polygon IoU approximation to optimize a neural network, there is a problem in that there is no control at all over what specific values ​​the individual values ​​of the polygon region parameters (for example, the coordinate value of one of the vertices that make up the polygon) will take. In other words, learning based on the polygon IoU approximation is a learning method that only requires that an appropriate shape be achieved across all the parameters that represent the polygon.

[0016] For this reason, neural networks trained using polygon IoU approximations may proceed in a way that leads to irrational inference results (polygon regions). Experience has shown that this tendency is particularly pronounced when the number of polygons generated by the neural network is large or when the training images contain a large number of object regions.

[0017] The present invention has been made based on the above-mentioned problem recognition, and aims to provide an area detection device and a program capable of detecting an area (polygon) of an appropriate shape. [Means for solving the problem]

[0018] [1] In order to solve the above problem, an area detection device according to one aspect of the present invention includes an image input unit that acquires an image to be processed, and an estimation unit that has a machine learning model, and inputs the image acquired by the image input unit into the model, and outputs a plurality of numerical values ​​output from the model as estimation results corresponding to the image, and determines the position on the image of a polygon center line, which is a state in which (N / 2-1) line segments (N is an even number equal to or greater than 4) are sequentially connected so that they do not branch, based on the plurality of numerical values ​​output by the estimation unit, and determines both ends of the polygon center line The area detection device includes a conversion unit that determines, as vertex positions, the positions of two points extending in the width direction from the polygon center line as the center, for each of the points (two points) and the connecting points ((N / 2-2) points) where a line segment connects to another line segment on the polygon center line, and outputs, as the detected area, an estimated polygon formed by connecting a total of N of the determined vertex positions with edges; and an error calculation unit that calculates the IoU (Intersection over Union) error between the estimated polygon determined based on the image acquired by the image input unit and a correct polygon that is given corresponding to the image, and is configured so that, when machine learning of the model possessed by the estimation unit is performed, the internal parameters of the model can be adjusted by error backpropagation based on the IoU error calculated by the error calculation unit.

[0019] [2] Furthermore, one aspect of the present invention is that in the area detection device of [1] above, the (N / 2-1) line segments determined by the conversion unit are all of equal length, and the multiple numerical values ​​output by the estimation unit are numerical values ​​representing the coordinates of the start and end points of an initial line segment, which is a line segment located at either end of the determined polygon center line, the magnitude of a displacement angle representing the direction of (N / 2-2) line segments sequentially connected from the initial line segment, and the (N / 2) width magnitudes when determining the positions of two points located widthwise from each of the total (N / 2) points, namely the end points (two points) of the polygon center line and the connecting points ((N / 2-2) points), as vertex positions.

[0020] [3] In addition, one aspect of the present invention is the area detection device according to the above [2], wherein a displacement angle Δ representing the difference in direction of the two line segments connected at the connection point is used. i (where i is an index for identifying the connection point) |Δ i |≦2π / ((N / 2)-1) The following constraint is imposed.

[0021] [4] Also, one aspect of the present invention includes an image input unit that acquires an image to be processed; an estimation unit that has a model capable of machine learning and inputs the image acquired by the image input unit into the model, and outputs a plurality of numerical values ​​output from the model as an estimation result corresponding to the image; a conversion unit that determines, based on the plurality of numerical values ​​output by the estimation unit, a position on the image of a polygon center line in which (N / 2-1) line segments (N is an even number equal to or greater than 4) are sequentially connected so as not to branch; determines, as vertex positions, positions of two points located in a width direction extending from the polygon center line as a center, at both end points (two points) of the polygon center line and at connection points ((N / 2-2) points) where a line segment is connected to another line segment on the polygon center line; and outputs, as a detected region, an estimated polygon formed by connecting a total of N vertex positions with sides; and a conversion unit that calculates an IoU (Intersection over Unit) between the estimated polygon determined based on the image acquired by the image input unit and a correct polygon that is a correct answer given corresponding to the image. and an error calculation unit that calculates an IoU (Integration over Union) error, wherein the internal parameters of the model can be adjusted by error backpropagation based on the IoU error calculated by the error calculation unit when performing machine learning of the model possessed by the estimation unit.

[0022] In addition, several reference embodiments (2022-210) are as follows.

[0023] [A1] One aspect of the IoU calculation device includes a convex hull division unit that divides one polygon in a given image to obtain one or more divided convex hull polygons; a convex hull inside pixel map generation unit that generates an inside pixel map (IPM) for each of the divided convex hull polygons, the inside pixel map indicating, as a numerical value, whether each pixel in the image is inside the convex hull polygon; a polygon inside pixel map generation unit that generates an inside pixel map for the polygon before it is divided by the convex hull division unit, the inside pixel map indicating, as a numerical value, whether each pixel in the image is inside the polygon before it is divided, by combining the inside pixel maps of the divided convex hull polygons generated by the convex hull inside pixel map generation unit; and a polygon inside pixel map generation unit that generates an inside pixel map for each of two given polygons in the image, the inside pixel map indicating, as a numerical value, whether each pixel in the image is inside the polygon before it is divided. Based on the inside pixel maps generated by the polygon inside pixel map generation unit for each of the polygons, an inside pixel map is generated that numerically indicates whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map is generated that numerically indicates whether or not each pixel in the image is inside the union area of ​​the two polygons, and an IoU (Intersection over Unit) is calculated for the two given polygons by calculating an approximation of the area of ​​the intersection area based on the inside pixel map for the intersection area, and dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area.and an IoU approximation calculation unit that calculates an approximation of an IoU (Interpolated over Unit) value, wherein the calculation process in which the convex hull inner pixel map generation unit generates an inner pixel map for the convex hull polygon, the calculation process in which the polygon inner pixel map generation unit generates an inner pixel map for the polygon before division based on the inner pixel map for the convex hull polygon after division, and the calculation process in which the IoU approximation calculation unit calculates an approximation of the IoU value based on the inner pixel maps for each of the two polygons are calculated only within the ranges of a function that is continuous and differentiable within an image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments.

[0024] [A2] Also, in one reference aspect, in the IoU calculation device of [A1] above, the convex hull inside pixel map generation unit generates an inside pixel map for each edge of the convex hull polygon to be processed, which generates an inside pixel map for that edge, which numerically indicates whether each pixel in the image is on the same side or the opposite side as a specified pixel inside the convex hull polygon, based on the edge and an extension of that edge, and generates an inside pixel map that numerically indicates whether each pixel in the image is inside the convex hull polygon, by combining the inside pixel maps for all edges of the convex hull polygon.

[0025] [A3] Also, in one reference aspect, in the IoU calculation device of [A2] above, the convex hull inside pixel map generation unit generates an inside pixel map that numerically indicates whether or not the pixel is inside the convex hull polygon by selecting the minimum value of the value for each pixel in the inside pixel map for each side when synthesizing the inside pixel maps for all sides of the convex hull polygon.

[0026] [A4] Also, one reference aspect of the region detection device includes: a region estimation unit that includes an internal machine learning model and that estimates a region having a specific feature within an image by inputting an externally passed image into the machine learning model; a training data supply unit that supplies pairs of training images and ground truth regions corresponding to the training images, for training the machine learning model included in the region estimation unit; and an error calculation unit that calculates the error between an estimated region that is a result of estimation by the region estimation unit based on the training images supplied by the training data supply unit, and the ground truth region that is supplied by the training data supply unit corresponding to the training images, wherein both the figure corresponding to the estimated region and the figure corresponding to the ground truth region are arbitrary polygons, and the error calculation unit includes the IoU calculation device described in any one of [A1] to [A3] above, and calculates the error based on approximations of the IoU values ​​calculated by the IoU calculation device for the polygon that is the figure corresponding to the estimated region and the polygon that is the figure corresponding to the ground truth region.

[0027] [A5] Also, one reference embodiment of an image processing device is an image processing device comprising: an estimation unit that includes an internal machine learning model and that outputs estimated polygons as a result of estimating polygons in an image received from outside by inputting the image into the machine learning model; a training data supply unit that supplies pairs of training images and ground truth polygons corresponding to the training images for training the machine learning model provided by the estimation unit; and an error calculation unit that calculates the error between the estimated polygons that are the result of estimation by the estimation unit based on the training images supplied by the training data supply unit, and the ground truth polygons that are supplied by the training data supply unit corresponding to the training images, wherein the error calculation unit includes the IoU calculation device described in any one of [A1] to [A3] above, and calculates the error based on an approximation of the IoU values ​​calculated by the IoU calculation device for the estimated polygons and the ground truth polygons.

[0028] [A6] Also, one reference aspect is a program comprising: a convex hull division unit that divides one polygon in a given image to obtain one or more divided convex hull polygons; a convex hull inside pixel map generation unit that generates an inside pixel map (IPM) for each of the divided convex hull polygons, the inside pixel map indicating as a numerical value whether each pixel in the image is inside the convex hull polygon; a polygon inside pixel map generation unit that generates an inside pixel map for the polygon before it is divided by the convex hull division unit, by combining the inside pixel maps of the divided convex hull polygons generated by the convex hull inside pixel map generation unit; an inside pixel map for each of the two polygons based on the inside pixel map generated by the polygon inside pixel map generation unit, the inside pixel map indicating, as a numerical value, whether or not each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map for a union area of ​​the two polygons indicating, as a numerical value, whether or not each pixel in the image is inside the union area of ​​the two polygons; an approximation of the area of ​​the intersection area is calculated based on the inside pixel map for the intersection area; an approximation of the area of ​​the union area is calculated based on the inside pixel map for the union area; and an IoU (Intersection over Unit) for the two given polygons is calculated by dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area.and an IoU approximation calculation unit that calculates an approximation of an IoU (Interpolated over Unit) value, wherein the calculation process in which the convex hull inner pixel map generation unit generates an inner pixel map for the convex hull polygon, the calculation process in which the polygon inner pixel map generation unit generates an inner pixel map for the polygon before division based on the inner pixel map for the convex hull polygon after division, and the calculation process in which the IoU approximation calculation unit calculates an approximation of the IoU value based on the inner pixel maps for each of the two polygons are performed only within the ranges of a function that is continuous and differentiable within an image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments.

[0029] Further references are given below.

[0030] [B1] A region detection device according to one reference aspect includes a region estimation unit that includes an internal machine learning model and that estimates a region within the image that has a specific feature by inputting an image passed from the outside into the machine learning model; a training data supply unit that supplies pairs of training images and gold standard regions corresponding to the training images for training the machine learning model provided in the region estimation unit; and an error calculation unit that calculates an error between an estimated region that is a result of estimation by the region estimation unit based on the training images supplied by the training data supply unit and the gold standard region that is supplied by the training data supply unit in response to the training image, wherein both a figure corresponding to the estimated region and a figure corresponding to the gold standard region are convex hull polygons, and the error calculation unit includes a virtual pixel generation unit that sets a large number of virtual pixels that are common within the image and the training image, and a union value ( an error function value calculation unit that calculates an error by dividing the sum of the intersection values ​​for all of the virtual pixels by the sum of the union values ​​for all of the virtual pixels, using the value of the product as a union, and the value of the intersection as an intersection value, wherein the inclusion determination value for the figure corresponding to the estimated area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more from any side of the figure, and the function for calculating the inclusion determination value for the figure corresponding to the estimated area is continuous and differentiable over the entire area within the image, and the inclusion determination value for the figure corresponding to the correct answer area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more from any side of the figure,Furthermore, the function for determining the inclusion judgment value for the figure corresponding to the correct answer region is continuous and differentiable in the entire region of the learning image, and the inclusion judgment value for the figure corresponding to the estimated region and the inclusion judgment value for the figure corresponding to the correct answer region are each greater than or equal to 0 and less than or equal to 1 in the entire region of the image and the learning image.

[0031] [B2] In addition, in one reference aspect, in the area detection device of the above [B1], the error calculation unit further calculates, for each of the virtual pixels, the following equation (1) (where e is Napier's constant, k is a predetermined positive constant, d px is the distance from a reference line that passes through a specified point inside the figure and is parallel to the side to the virtual pixel, and d center is the distance from the reference line for the side to the side), and an inclusion determination integration unit that multiplies the inclusion determination values ​​of the virtual pixel for all sides of the figure determined by the inclusion determination unit to determine the inclusion determination value for the virtual pixel for the figure.

[0032] [B3] Also, one reference aspect is a program for detecting regions including a region estimation unit that includes a machine learning model and that estimates a region having a specific feature in an image by inputting an image passed from the outside into the machine learning model; a training data supply unit that supplies pairs of training images and correct regions corresponding to the training images for training the machine learning model provided in the region estimation unit; and an error calculation unit that calculates an error between an estimated region that is a result of estimation by the region estimation unit based on the training images supplied by the training data supply unit and the correct region that is supplied by the training data supply unit in correspondence with the training image, wherein both a figure corresponding to the estimated region and a figure corresponding to the correct region are convex hull polygons, and the error calculation unit includes a virtual pixel generation unit that sets a large number of virtual pixels that are common to both the image and the training image, and a union calculation unit that calculates, for each of the virtual pixels, the sum and product of an inclusion determination value for the figure corresponding to the estimated region and an inclusion determination value for the figure corresponding to the correct region, and calculates a value obtained by subtracting the product from the sum. an error function value calculation unit that calculates an error by dividing the sum of the intersection values ​​for all of the virtual pixels by the sum of the union values ​​for all of the virtual pixels, with the value of the product as a union value and the value of the product as an intersection value, wherein the inclusion determination value for a figure corresponding to the estimated area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more from any side of the figure, and the function for calculating the inclusion determination value for the figure corresponding to the estimated area is continuous and differentiable over the entire area within the image, and the inclusion determination value for the figure corresponding to the correct answer area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more from any side of the figure,and a function for determining an inclusion determination value for the figure corresponding to the correct answer region is continuous and differentiable in the entire region of the learning image, and the inclusion determination value for the figure corresponding to the estimated region and the inclusion determination value for the figure corresponding to the correct answer region are each greater than or equal to 0 and less than or equal to 1 in the entire region of the image and the learning image. [Effects of the Invention]

[0033] According to the present invention, it is possible to impose appropriate constraints on the area (polygon area) detected by the area detection device. [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a functional block diagram showing a schematic functional configuration of an area detection device according to an embodiment of the present invention. [Figure 2] FIG. 10 is one diagram for explaining a procedure in which the area detection device according to the embodiment generates a polygon based on predetermined numerical values, and is a schematic diagram showing an initial vector used in the procedure. [Figure 3] FIG. 10 is one of diagrams for explaining the procedure by which the area detection device according to the embodiment generates a polygon based on predetermined numerical values, and is a schematic diagram illustrating the first expansion of the initial vector. [Figure 4] FIG. 10 is one of diagrams for explaining the procedure by which the area detection device according to the embodiment generates a polygon based on predetermined numerical values, and is a schematic diagram showing the second extension of the initial vector. [Figure 5] FIG. 10 is one of the diagrams for explaining the procedure by which an area detection device according to an embodiment generates polygons based on predetermined numerical values, and is a schematic diagram showing the arrangement of the center lines of the polygons at the time when the fifth expansion is completed. [Figure 6] This is one of the figures for explaining the procedure by which an area detection device according to an embodiment generates a polygon based on specified numerical values, and is a schematic diagram showing a polygon with 12 vertices on the image plane generated by expanding in the width direction based on the above-mentioned center line. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of a polygon (after conversion by a conversion unit 13) output as an estimation result by an estimation unit 12 (learned) according to an embodiment. [Figure 8] FIG. 1 is a schematic diagram showing an example of an inappropriate polygon that may be output by a region detection technique using a neural network according to the prior art. [Figure 9] FIG. 2 is a block diagram showing a more detailed functional configuration inside an error calculation unit according to an embodiment (first embodiment related to error calculation). [Figure 10] 1 is a schematic diagram showing an example of the arrangement of real pixels and virtual pixels in an image in an embodiment (first embodiment relating to error calculation). FIG. [Figure 11] FIG. 1 is one of schematic diagrams for explaining the processing of the inclusion determination unit according to the embodiment (first embodiment relating to error calculation), showing a state in which one of the figures to be subjected to error calculation exists on the xy plane. [Figure 12] FIG. 10 is another schematic diagram for explaining the processing of the inclusion determination unit according to the embodiment (first embodiment relating to error calculation), showing a state in which the graphic plane is rotated so that one side is horizontal. [Figure 13] 10 is a schematic diagram for explaining a process performed by an error function value calculation unit according to an embodiment (first embodiment relating to error calculation) to calculate an error between two figures. FIG. [Figure 14] FIG. 10 is a schematic diagram for explaining a method for determining an inclusion determination value according to a modified example of the embodiment (first embodiment relating to error calculation). [Figure 15] 1 is a schematic diagram for explaining an application example of the area detection device according to the embodiment (first embodiment relating to error calculation) and an example of an image used in an experiment for verifying the application example. FIG. [Figure 16] FIG. 1 is a block diagram showing a schematic functional configuration of a polygon IoU calculation device according to an embodiment (second embodiment relating to error calculation). [Figure 17] FIG. 10 is a schematic diagram showing an example in which a convex hull dividing unit according to an embodiment (second embodiment relating to error calculation) divides a given polygon (PA) into a plurality of convex hulls. [Figure 18]FIG. 10 is a schematic diagram showing an example in which a convex hull dividing unit according to an embodiment (second embodiment relating to error calculation) divides a given polygon (PB) into a plurality of convex hulls. [Figure 19] FIG. 10 is a schematic diagram showing a polygon PA2, which is an example of a convex hull polygon that is a target of processing for generating an IPM (interior pixel map) by a convex hull IPM generating unit according to an embodiment (second embodiment related to error calculation). [Figure 20] FIG. 10 is a schematic diagram for explaining a method in which a convex hull IPM generation unit according to an embodiment (second embodiment relating to error calculation) calculates an IPM for one side of a convex hull polygon PA2. [Figure 21] FIG. 10 is a schematic diagram showing an IPM calculated for an edge L1 of a convex hull polygon PA2 by a convex hull IPM generation unit according to an embodiment (second embodiment relating to error calculation). [Figure 22] 10 is a schematic diagram showing an IPM calculated for a side L2 of a convex hull polygon PA2 by a convex hull IPM generation unit according to an embodiment (second embodiment relating to error calculation). FIG. [Figure 23] FIG. 10 is a schematic diagram showing an IPM calculated for side L3 of convex hull polygon PA2 by a convex hull IPM generation unit according to an embodiment (second embodiment relating to error calculation). [Figure 24] FIG. 10 is a schematic diagram showing an IPM calculated for a side L4 of a convex hull polygon PA2 by a convex hull IPM generation unit according to an embodiment (second embodiment relating to error calculation). [Figure 25] 10 is a flowchart showing the procedure of a process performed by a polygon IoU calculation device according to an embodiment (second embodiment relating to error calculation) to find an approximate value of IoU between two given polygons. [Figure 26] FIG. 10 is a block diagram showing a schematic functional configuration of an area detection device (using the polygon IoU calculation device of the second embodiment) according to an embodiment (third embodiment related to error calculation). [Figure 27] FIG. 10 is a block diagram showing a schematic functional configuration of an image processing device (using the polygon IoU calculation device of the second embodiment) according to an embodiment (fourth embodiment related to error calculation). [Figure 28]FIG. 2 is a block diagram showing an example of the internal configuration of each device such as the area detection device 1, the area detection device 2, the image processing device 3, and the polygon IoU calculation device 3051 according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0035] Next, an embodiment of the present invention will be described with reference to the drawings.

[0036] The embodiment described below relates to a technology for detecting an area having specific characteristics from an image by performing machine learning using a neural network or the like. In this embodiment, a polygon IoU approximation calculation method is used. The polygon IoU approximation calculation method will be described in detail later. This embodiment enables stable learning of a model (neural network) for detecting a specific area. A feature of this embodiment is the method by which the model (neural network) generates a polygon shape. This embodiment stabilizes learning of an area detection model (neural network) using a polygon IoU approximation value.

[0037] Experiments using the method of this embodiment have confirmed that data sets that could not be used for learning with conventional methods can now be used. In other words, by using this embodiment, more training data can be used than before, and improvement in detection accuracy can be expected.

[0038] The area detection device 1 according to this embodiment detects an area having specific characteristics within an input image. In other words, the area detection device 1 estimates an area having a polygonal shape based on the input image. In doing so, constraints are imposed on the shape of the polygon estimated by the area detection device 1. In other words, the area detection device 1 estimates and outputs a polygon within the constraints. Specifically, the area detection device 1 generates a reference vector, and then generates a polygon by extending the vector under the above constraints. Because of these constraints, when the area detection device 1 performs machine learning, it avoids learning that could generate extremely irrational polygon shapes.

[0039] Among the training data used to train the area detection device 1, there is one that can automatically find a reasonable and desirable shape. In this case, not only optimization based on the IoU approximation value of the polygon but also distance-based optimization using a reasonable and desirable shape is used in combination. This allows the area detection device 1 of this embodiment to achieve more stable training.

[0040] FIG. 1 is a block diagram showing a schematic functional configuration of an area detection device according to this embodiment. As shown in the figure, the area detection device 1 includes an image input unit 11, an estimation unit 12, a conversion unit 13, a learning data supply unit 21, and an error calculation unit 22. At least a portion of each functional unit constituting the area detection device 1 can be realized, for example, by a computer and a program. Each functional unit also has a storage means as necessary. The storage means is, for example, a program variable or memory allocated by program execution. Non-volatile storage means such as a magnetic hard disk drive or solid-state drive (SSD) may also be used as necessary. At least a portion of the functions of each functional unit may also be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows:

[0041] The image input unit 11 acquires an image to be processed. The image input unit 11 passes the acquired input image to the estimation unit 12.

[0042] The estimation unit 12 has a model capable of machine learning, and by inputting the image acquired by the image input unit 11 into this model, outputs a plurality of numerical values ​​output from this model as estimation results corresponding to the image. The model is realized using, for example, a neural network.

[0043] The conversion unit 13 determines an area based on the multiple numerical values ​​output by the estimation unit 12. The area determined by the conversion unit 13 is a polygon. That is, the conversion unit 13 converts the multiple numerical values ​​output by the estimation unit 12 into a polygon area. Specifically, based on the multiple numerical values ​​output by the estimation unit 12, the conversion unit 13 determines the position on the image of a polygon center line, which is a state in which (N / 2-1) line segments (N is an even number greater than or equal to 4) are sequentially connected so as not to branch, and determines, as vertex positions, the positions of two points located in the width direction extending from the polygon center line as the center, at each of the two end points (two points) of the polygon center line and the connection points ((N / 2-2) points) where a line segment is connected to another line segment on the polygon center line. The conversion unit 13 outputs, as the detected area, an estimated polygon formed by connecting a total of N of the determined vertex positions with sides.

[0044] Specifically, the (N / 2-1) line segments determined by the conversion unit 13 may all be of equal length. The multiple numerical values ​​output by the estimation unit 12 may represent (1) the coordinates of the start and end points of the initial line segment, which is a line segment located at either end of the determined polygon center line, (2) the magnitude of the displacement angle representing the direction of the (N / 2-2) line segments sequentially connected to the initial line segment, and (3) the (N / 2) width magnitudes when determining the positions of two points extending in the width direction from each of the total (N / 2) points of the polygon center line, namely, the two end points (two points) and the connecting points ((N / 2-2) points). The above-mentioned "line segments" (line segments constituting the polygon center line) may be given a direction so that they can be processed as vectors.

[0045] Also, the difference in direction between the two line segments at the connection point is expressed as a displacement angle Δ i (where i is an index for identifying the connection point) |Δ i |≦2π / ((N / 2)-1) Here, || represents the absolute value of the numerical value.

[0046] The details of the procedure by which the conversion unit 13 determines polygons based on the numerical values ​​passed from the estimation unit 12 will be explained later with reference to FIGS.

[0047] Furthermore, when adjusting (optimizing) the internal parameters of the model held by the estimation unit 12 by backpropagating the error calculated by the error calculation unit 22, the conversion unit 13 passes the value obtained by the differential calculation of the above conversion process to the estimation unit 12. In other words, all calculations performed by the conversion unit 13 must be composed of differentiable functions. The implementation of the present invention can be composed only of differentiable functions such as the four arithmetic operations and trigonometric functions.

[0048] When the area detection device 1 operates in the learning mode, the learning data supply unit 21 supplies learning data for machine learning of the model (neural network) held by the estimation unit 12. The learning data is a set of pairs of an input image and a ground truth area corresponding to the input image. In other words, the learning data is a set of pairs of an input image and the coordinates (x and y coordinates on the image) of all vertices of a ground truth polygon corresponding to the input image.

[0049] Specifically, the learning data supply unit 21 passes the input image included in the pair to the image input unit 11. The learning data supply unit 21 also passes information on the coordinates of the vertices of the polygon in the correct region included in the pair to the error calculation unit 22. This enables the error calculation unit 22 to calculate the error between the polygon output by the estimation unit 12 based on the input image for learning (the polygon converted by the conversion unit 13) and the correct polygon supplied by the learning data supply unit 21. When learning a model, the learning data supply unit 21 sequentially supplies each pair for learning.

[0050] The error calculation unit 22 calculates an error for machine learning of the model held by the estimation unit 12. Specifically, the error calculation unit 22 calculates the error between a polygon (estimated polygon) obtained by estimation by the estimation unit 12 and conversion by the conversion unit 13, and a correct polygon that corresponds to the estimated polygon. In this embodiment, the error calculated by the error calculation unit 22 is an IoU (Intersection over Union) error. The IoU error calculated by the error calculation unit 22 may be an error based on a strict IoU that is based on the degree of overlap between two polygons, or may be an error based on an approximation of the IoU (IoU approximation error). A method by which the error calculation unit 22 calculates the IoU approximation error using a realistic procedure will be described in detail later. The error calculated by the error calculation unit 22 is used for error backpropagation for machine learning of the model held by the estimation unit 12.

[0051] The area detection device 1 operates in either an estimation mode or a machine learning mode. When operating in the estimation mode, the area detection device 1 estimates an area (polygon) corresponding to an input image using a machine-learned model. When operating in the machine learning mode, the area detection device 1 adjusts the internal parameters of the model (neural network) held by the estimation unit 12 using training data supplied by the training data supply unit 21. When performing machine learning of the model (neural network) held by the estimation unit 12, the area detection device 1 is configured to be able to adjust the internal parameters of the model by performing error backpropagation based on the IoU error calculated by the error calculation unit 22.

[0052] Polygon generation using a typical neural network uses a projection neural network, such as a linear combination layer or a multilayer perceptron layer. This neural network converts feature values ​​obtained from an image or other source into a list of polygon coordinates. For example, in the case of a model that generates a polygon with 12 vertices, the projection neural network generates 24 numerical values ​​(i.e., the x and y coordinate values ​​of each of the 12 vertices). Using this method, the neural network can generate any polygon shape. However, because such neural networks have too many degrees of freedom, they are not suitable for optimization using polygon IoU approximation. This is because there is a possibility that the network will be trained to generate unnatural polygon shapes due to the high degrees of freedom.

[0053] Therefore, as described above, the area detection device 1 of this embodiment imposes constraints on the shape of the polygon to be generated (estimated). Specifically, the area detection device 1 first generates a reference vector. Then, the area detection device 1 generates a polygon by following a procedure to expand the vector under certain constraints.

[0054] 2 to 6 are schematic diagrams showing the procedure by which the area detection device 1 generates a polygon. The projection neural network of the model included in the area detection device 1 outputs (the number of vertices of the polygon to be generated + 3) numerical values. For example, a model for generating a 12-vertex polygon generates 15 numerical values. These 15 numerical values ​​consist of four numerical values ​​(x and y coordinate values ​​of the starting point and the ending point of the reference vector) representing the reference vector (also called the initial vector), five numerical values ​​(= (number of vertices / 2 - 1)) representing the angle of change of the polygon center line, and six numerical values ​​(= (number of vertices / 2)) representing the thickness (width) of the polygon. These 15 numerical values ​​for generating a 12-vertex polygon can be converted into a 12-vertex polygon using the procedure described below.

[0055] In general, to generate a polygon with N vertices, four numerical values ​​representing the reference vector, (N / 2-1) numerical values ​​representing the angle of change of the polygon center line, and (N / 2) numerical values ​​representing the thickness (width) of the polygon are used. In other words, to generate a polygon with N vertices, (N+3) numerical values ​​are used.

[0056] [Steps to generate a 12-vertex polygon based on 15 numerical values] In steps S1 to S3 below, the conversion unit 13 of the area detection device 1 extends (expands) the center line of the polygon. Then, in step S4, the coordinate values ​​of each vertex of the polygon are determined by expanding the polygon in the width direction around the center line (broken line). This determines the shape and position of the polygon.

[0057] Step S1: Figure 2 shows the initial vector v init Schematic diagram of the initial vector v init The starting point of the vector v is the pixel position p0 on the image. The pixel position is expressed as a discrete x and y coordinate value. init When the starting point of is placed so as to coincide with the pixel position p0, the initial vector v init The pixel position of the end point of the initial vector v init The x and y coordinates of the starting point of the vector v init A total of four scalar values ​​are determined: the x and y coordinate values ​​of the end points.

[0058] Step S2: initial vector v init Length (absolute value) d base and the phase angle θ0. The phase angle is an angle that increases counterclockwise, with the horizontal direction of the image plane at the right side being 0. These values ​​are calculated based on the initial vector v init It is possible to calculate the starting point from the x and y coordinate values ​​of the end point. s, y s ), and the end point is (x e, y e ), then dbase is calculated by taking the norm using the formula below.

[0059] d base =((y e -y s )^2+(x e -x s )^2)^0.5

[0060] In addition, θ0 is calculated using the following formula.

[0061] θ0=atan2(y e -y s ,x e -x s )

[0062] Here, atan2() is the inverse function of the trigonometric function tan (tangent), and returns a value in the interval [-π,π]. base The two scalar values ​​of the phase angle θ0 are determined. base The equations for calculating θ and θ are differentiable.

[0063] Step S3: The initial vector v init Extend the initial vector v several times. Figures 3 to 5 show an example of extending the initial vector v five times. init This shows an extension of

[0064] Figure 3 shows the initial vector v init As shown in the figure, in the first expansion, the conversion unit 13 converts the initial vector v init The starting point of the vector v is fixed at the pixel position p0. init The length of d base While keeping it fixed, the initial vector v init The phase angle is changed to θ1. However;

[0065] θ1=θ0+Δ1

[0066] Here, Δ1 is the displacement angle, which may be positive, negative, or 0. The initial vector v when the phase angle is changed to θ1 is init The pixel position of the end point is set to p1.

[0067] Figure 4 shows the initial vector v init As shown in the figure, in the second expansion, a new vector is placed on the image plane with the pixel position p1 as the starting point. However, the length (absolute value) of the newly placed vector is also d base The phase angle of the newly placed vector is θ2.

[0068] θ2=θ1+Δ2

[0069] Here again, the displacement angle Δ2 may be positive, negative, or 0. Here, the pixel position of the end point of the newly placed vector is set to p2.

[0070] Similar to this second extension, the center line of the polygon can be extended (extended) a third time, a fourth time, etc. However, the number of extensions is limited.

[0071] FIG. 5 is a schematic diagram showing the arrangement of centerlines at the end of the fifth expansion in step S3. As shown in the figure, the start point of the vector newly placed in the third expansion is p2 and the end point is p3. The start point of the vector newly placed in the fourth expansion is p3 and the end point is p4. The start point of the vector newly placed in the fifth expansion is p4 and the end point is p5. The lengths (absolute values) of the vectors newly placed in the third, fourth, and fifth expansions are all d, as in the first and second expansions. baseThe phase angle of the newly placed vector in the third expansion is θ3. The phase angle of the newly placed vector in the fourth expansion is θ4. The phase angle of the newly placed vector in the fifth expansion is θ5. However;

[0072] θ3=θ2+Δ3 θ4=θ3+Δ4 θ5=θ4+Δ5

[0073] Each of the displacement angles Δ3, Δ4, and Δ5 can be positive, negative, or 0. As mentioned above, the center line of the polygon can be generated by successively extending the initial vector. In other words, the five line segments p0, p1, p2, p3, p4, and p5 (each of which has a length d base ) is generated as the center line of the polygon. The coordinates of p1 to p5 are calculated using the formula below. i The coordinates of (x i ,y i ), then we get:

[0074] x i =x i-1 +d base ×cos(θ i ) y i =y i-1 +d base ×sin(θ i )

[0075] Step S4: Polygons are generated by expanding the width direction based on the center line generated by the procedure up to step S3.

[0076] FIG. 6 is a schematic diagram showing a polygon with 12 vertices on the image plane generated as a result of the operation of step S4. As shown in the figure, the positions of the 12 vertices are determined by expanding in a direction (width direction) approximately perpendicular to the center line, centered on the six positions p0, p1, p2, p3, p4, and p5 shown in FIG. 5. Note that the width of the expansion in the width direction passing through pixel position p0 is d0. Similarly, the width of the expansion in the width direction passing through pixel position p1 is d1. The width of the expansion in the width direction passing through pixel position p2 is d2. The width of the expansion in the width direction passing through pixel position p3 is d3. The width of the expansion in the width direction passing through pixel position p4 is d4. And the width of the expansion in the width direction passing through pixel position p5 is d5.

[0077] Note that the expansion is performed equally on both sides (left and right) when viewed from the center line. That is, the distance between two vertices determined based on pixel position p0 is d0, and the midpoint between those two vertices is pixel position p0. Similarly, the distance between two vertices determined based on pixel position p1 is d1, and the midpoint between those two vertices is pixel position p1. The distance between two vertices determined based on pixel position p2 is d2, and the midpoint between those two vertices is pixel position p2. The distance between two vertices determined based on pixel position p3 is d3, and the midpoint between those two vertices is pixel position p3. The distance between two vertices determined based on pixel position p4 is d4, and the midpoint between those two vertices is pixel position p4. The distance between two vertices determined based on pixel position p5 is d5, and the midpoint between those two vertices is pixel position p5.

[0078] As shown in the figure, the expansion based on pixel position p0 is performed perpendicular to the vector with p0 as the starting point and p1 as the end point. Also, the expansion based on pixel position p5 is performed perpendicular to the vector with p4 as the starting point and p5 as the end point. i The widthwise expansion based on (i=0,1,2,...) is performed at pixel position p i is the start point or end point, and the adjacent pixel position p i-1 or pi+2 A vector extending in the direction of the arrow (if such a vector exists) is made perpendicular to the average vector.

[0079] The expansion in the width direction from the center line can be expressed mathematically as follows. The two vertices obtained by expanding in both directions with P0=(x0, y0) as the center are called P 0a =(x 0a ,y 0a ) and P 0b =(x 0b ,y 0b ) then;

[0080] x 0a =x0+(d0 / 2)×cos(θ i +π / 2) y 0a =y0+(d0 / 2)×sin(θ i +π / 2) x 0b =x0+(d0 / 2)×cos(θ i -π / 2) y 0b =y0+(d0 / 2)×sin(θ i -π / 2)

[0081] In the above equation, ±π / 2 means a rotation of 90 degrees in both the positive and negative directions.

[0082] In general, the expansion of P0, P1, P2,... is expressed as follows: i =(x i ,y i ) (where i=0,1,2...) is expanded to P ia =(x ia ,y ia ) and P ib =(x ib ,y ib ) can be calculated.

[0083] x ia =x i +(d i / 2)×cos(θ´ i +π / 2) yia =y i +(d i / 2)×sin(θ´ i +π / 2) x ib =x i +(d i / 2)×cos(θ´ i -π / 2) y ib =y i +(d i / 2)×sin(θ´ i -π / 2)

[0084] However, when i=0 (the first point), θ´ i =θ i+1 Also, when i=5 (the last point), θ' i =θ i In other cases, i.e., when i=1,2,3,4, which correspond to the intermediate points, for example, by taking the average of the orientations of the center lines on both sides, θ' i =(θ i +θ i+1 ) / 2, etc.

[0085] In other words, the procedure for generating a polygon with 12 vertices is as follows: In the first step, an initial vector is determined based on four numerical values. These four numerical values ​​are the X and Y coordinate values ​​of the starting point of the initial vector and the X and Y coordinate values ​​of the end point. In the second step, the length d of the initial vector determined above is used. base and angle θ0. baseThe angle θ0 and the initial vector can be easily calculated from the coordinates of the start and end points of the initial vector. In the third step, the initial vector is sequentially extended by adding five values ​​(Δ1 to Δ5; if the polygon has 12 vertices) representing the change angle of the polygon center line to the angle θ0. The broken line formed by these extended initial vectors is the center line of the polygon. In the fourth step, the center line is extended horizontally (widthwise) using six values ​​(d0 to d5; if the polygon has 12 vertices) representing the thickness (width) of the polygon. Note that this horizontal extension is perpendicular to the line segment forming the center line. A polygon is generated through this procedure. In other words, the conversion unit 13 converts the set of values ​​passed from the estimation unit 12 into a polygon.

[0086] In the above polygon generation procedure, Δ i A constraint may be placed on the value of (1≦i≦5). An example of a specific constraint is expressed by the inequality below.

[0087] Constraint: |Δ i |≦2π / ((number of polygon vertices / 2)-1)

[0088] As mentioned above, Δ i By limiting the value of , it is possible to prevent the generation of polygons with extremely unnatural shapes. It has been confirmed that the above constraints work well.

[0089] The above explanation is the procedure for expanding the initial vector five times, expanding the width direction around six positions p0, p1, p2, p3, p4, and p5, and finding 12 vertices of the polygon. The number of times the initial vector is expanded is not limited to five. In general, expanding the initial vector to p0, p1, p2, ..., p (N / 2)-1 The (N / 2) points (points that make up the polygon center line) can be found, and from there the N vertices of the polygon can be found.

[0090] When generating a polygon with an odd number of vertices (N-1; N is an even number), the polygons are assigned p0, p1, p2, . . . , p (N / 2)-1 (N / 2) points (points that make up the polygon center line) are found, and their end points (p0 or p (N / 2)-1 The width may be set to 0 when expanding in the width direction from either

[0091] As described above, the calculations performed by the conversion unit 13 are differentiable calculations.

[0092] As explained above, this embodiment does not generate completely free polygons, but rather restricts the polygons that can be generated by the above procedure. This method of restriction is effective, and it has been confirmed that good results can be obtained in area detection for various purposes. In other words, this embodiment does not generate extremely unnatural polygons in area detection for various purposes.

[0093] 7 is a schematic diagram showing an example of a polygon (after conversion by the conversion unit 13) output as an estimation result by the estimation unit 12 (which has been trained) of this embodiment. The polygon shown in FIG. 7 is a polygon resulting from conversion by the conversion unit 13 based on the numerical values ​​estimated by the estimation unit 12 as an area having text in an image (for example, a landscape photograph, etc., which is omitted here).

[0094] Fig. 8 (reference diagram) shows an example of a polygon that may be output by a region detection technique using a neural network according to a conventional technique, not according to this embodiment. Depending on how the neural network is trained, a polygon that is inappropriate as a region, as shown in Fig. 8, may be output.

[0095] In other words, this embodiment can prevent the generation of inappropriate polygons.

[0096] [IoU approximation calculation method] The IoU approximation calculation method will be explained below.

[0097] When training the neural network (estimation unit 12), the error must be calculated using a differentiable function. That is, it is necessary to use a differentiable function that takes the estimated region and the correct region as input and outputs an error value (calculation in error calculation unit 22). This is because the gradient of the error value in the multidimensional space of parameter values ​​is required when adjusting the parameters inside the neural network.

[0098] However, in practice, there are cases where it is not a problem to include some non-differentiable functions when calculating errors. For example, an operation that selects a portion of multiple input values ​​(for example, an operation that selects the maximum value of multiple input values) is locally non-differentiable, but it is widely known that neural networks can be trained even if such an operation is included. A typical example of an operation that selects a portion of multiple input values ​​is max pooling.

[0099] In conventional techniques, rectangular regions were targeted for detection when detecting specific regions within an image, due to the use of a differentiable error function. Furthermore, an error function called IoU (Intersection over Union) is known to be suitable for calculating the error between the estimated region and the ground truth region (both of which are rectangular), and is therefore used.

[0100] Furthermore, in the prior art, instead of configuring a neural network that directly obtains the coordinate values ​​of a specific region, a mask image consisting of pixel values ​​of 0 or more and 1 or less is output from the neural network, and a region whose pixel value exceeds a predetermined threshold (e.g., 0.5) is output as the specific region of the estimation result. In this case, any shape, not limited to a rectangle, is obtained as the estimation result.

[0101] However, the conventional techniques have the following problems to be solved.

[0102] The IoU between polygons can be precisely defined using mathematical formulas and is often used as an evaluation measure for accuracy in polygon-shaped object detection technology. However, there is a problem in that there is no method for calculating polygon IoU that can be used for neural network training.

[0103] In the conventional technology, the error is calculated using an error function for rectangular regions. In other words, both the region estimated by the neural network and the correct region are limited to rectangular shapes. Therefore, the conventional technology has the problem that it is not possible to configure a neural network that can directly estimate the coordinate values ​​of regions with shapes other than rectangular.

[0104] One of the techniques described as prior art is a method that outputs a mask image from a neural network, enabling estimation of any shape. However, errors can occur when calculating a set of coordinate values ​​for identifying an estimated area based on a mask image generated by a neural network. For example, an area smaller than the size of one pixel of the generated mask image cannot be estimated. While simple regression methods, such as treating the least squares error between the correct coordinate values ​​and the estimated coordinate values ​​as the error, are generally not very accurate, coordinate regression is also constrained by the requirement that the number of vertices in the shape of the correct area must be the same as the number of vertices in the shape of the estimated area. In other words, it is not possible to calculate the error based on the vertex coordinate values ​​between a rectangular area and a pentagonal area, for example.

[0105] Furthermore, when the PIoU error function described in the aforementioned Non-Patent Document 1 is used, it is possible to calculate the error between regions for a rotated rectangle, but there is a problem that it is difficult to estimate a shape that is finer than the pixel size of the input image. Also, the method described in Non-Patent Document 1 has a problem that it can only be used for rotated rectangles and cannot be applied to polygons with complex shapes.

[0106] Furthermore, while the technology described in the aforementioned Non-Patent Document 2 calculates the error between general polygons, the error is the error in the positions of corresponding vertices of two polygons. The technology using the vertex error described in Non-Patent Document 2 has the problem that it is less accurate than a technology using polygon IoU as the error (does not properly represent the degree of match (degree of overlap) between two polygons). Furthermore, the technology described in Non-Patent Document 2 requires that the number of vertices of a polygon be fixed, which necessitates preprocessing of the data to match the number of vertices, resulting in the problem of high costs for creating a training dataset.

[0107] In this embodiment, based on the above-mentioned problem recognition, an approximation value of IoU is calculated in a better way.

[0108] [First form of error calculation] The area detection device 1 uses a machine learning technique to automatically detect areas having specific characteristics within an input image. In this embodiment, when training the machine learning model, a method for calculating an error (also called loss or error) specific to this embodiment is used. The error calculation has the following two main features.

[0109] As a first feature, the error calculation unit 22 defines virtual pixels used solely for calculation convenience, regardless of the actual pixels constituting the input image, and performs calculations based on these virtual pixels. Specifically, the error calculation unit 22 calculates the number of virtual pixels contained within a figure representing an area within the image. Note that the figure here is limited to a convex hull figure. In other words, the figure (polygon) here is limited to a figure in which the interior angle of any vertex constituting the figure (polygon) is less than 180 degrees. Note that the error calculation unit 22 may calculate an approximation of the number of virtual pixels contained within the figure. By performing calculations based on virtual pixels in this way, the error calculation unit 22 can calculate the error with high accuracy even when the resolution of the input image is low.

[0110] As a second feature, the error calculation unit 22 determines, for each side that constitutes the symmetrical figure, whether each virtual pixel is inside or outside the figure. Then, the error calculation unit 22 integrates the determination results (inside or outside) for each side to determine whether each virtual pixel is inside or outside the symmetrical figure. Using this procedure, the error calculation unit 22 calculates the number of virtual pixels that are inside a symmetrical figure (however, a convex hull figure) having an arbitrary shape. The specific calculation method will be described later.

[0111] The area detection device 1 operates in either a learning mode or an estimation execution mode. In the learning mode, the area detection device 1 performs processing using learning data to optimize the parameters of the machine learning model held by the estimation unit 12. In the estimation execution mode, the estimation unit 12 performs processing to estimate a specific area in an unknown input image using learned parameters. The functions of each unit are described below.

[0112] The image input unit 11 acquires an image input from the outside and passes it to the estimation unit 12. The image that the image input unit 11 passes to the estimation unit 12 is an image that is to be estimated in operation in the estimation execution mode.

[0113] The estimation unit 12 includes a machine learning model and estimates an area within the image having a specific feature by inputting an externally provided image into the machine learning model. Specifically, the estimation unit 12 includes a neural network as the machine learning model. The estimation unit 12 outputs a numerical value that can be converted into a figure corresponding to the area, which is the estimation result. In this embodiment, the figure is a convex hull polygon. That is, the figure is a polygon in which all interior angles are less than 180 degrees. The relationship between the numerical value output by the estimation unit 12 and the convex hull polygon (polygon) has already been described with reference to FIGS. 2 to 6.

[0114] When the estimation unit 12 operates in the learning mode, it passes the numerical values ​​of the estimation result to the conversion unit 13. The conversion unit 13 calculates the coordinate values ​​of the vertices of a convex hull polygon (polygon) based on the passed numerical values ​​and passes them to the error calculation unit 22. As a result, the error calculation unit 22 calculates the error between the graphic of the estimated area and the graphic of the correct area. Furthermore, when the estimation unit 12 operates in the estimation execution mode, it passes the numerical values ​​that are the result of estimation based on an unknown image to the conversion unit 13. Similarly, the conversion unit 13 calculates the coordinate values ​​of the vertices of the convex hull polygon (polygon) based on the passed numerical values ​​and outputs this convex hull polygon (polygon) as the area of ​​the estimation result (detection result).

[0115] That is, in the estimation execution mode, the conversion unit 13 can output to the outside information (graphic information) of the estimated area based on the numerical values ​​output by the estimation unit 12. This information of the estimated area can be used for various purposes as information of an area having a specific characteristic.

[0116] When the area detection device 1 operates in the learning mode, the error calculation unit 22 The error between the figure of the estimated area (vertex coordinate values ​​of the polygon) passed from the conversion unit 13 and the figure of the correct area (vertex coordinate values ​​of the correct polygon) supplied by the learning data supply unit 21 is calculated. Note that the figure corresponding to the estimated area and the figure corresponding to the correct area are both convex hull polygons (polygons). The error calculated by the error calculation unit 22 is used to adjust (optimize) the internal parameters of the neural network held by the estimation unit 12 by the error backpropagation method. The configuration and processing contents of the error calculation unit 22 will be described in more detail later with reference to FIG. 9 etc.

[0117] The error calculated by the error calculation unit 22 is as follows: The internal configuration of the error calculation unit 22, which will be explained later with reference to the block diagram in Fig. 9, realizes an example of a procedure for specifically realizing the following error.

[0118] The error calculation unit 22 uses an inclusion determination value as basic data for calculating the error. The inclusion determination value is a numerical value that indicates whether a virtual pixel, which will be described later, is contained in a figure (the figure of the estimated region or the figure of the correct region). However, in this embodiment, the inclusion determination value is not a binary value, for example, 0 or 1, but can take a continuous value greater than or equal to 0 and less than or equal to 1.

[0119] The inclusion judgment value for a figure corresponding to the estimated area is 1 or approximately 1 when the virtual pixel is inside the figure and is a predetermined distance or more from any side of the figure, and is 0 or approximately 0 when the virtual pixel is outside the figure and is a predetermined distance or more from any side of the figure, and the function for determining the inclusion judgment value for the figure corresponding to the estimated area is continuous and differentiable in the entire area of ​​the image. That is, in the vicinity of the side of the figure (as already explained, the figure is a polygon) (the vicinity means the area within the predetermined distance from the side), the inclusion judgment value changes continuously (and abruptly) and smoothly from 0 (or approximately 0) to 1 (or approximately 1).

[0120] The same applies to the correct answer region. That is, the inclusion judgment value for the figure corresponding to the correct answer region is 1 or almost 1 when the virtual pixel is inside the figure and is a predetermined distance or more from any side of the figure, and is 0 or almost 0 when the virtual pixel is outside the figure and is a predetermined distance or more from any side of the figure, and the function for calculating the inclusion judgment value for the figure corresponding to the correct answer region is continuous and differentiable in the entire region of the training image. That is, in the vicinity of the sides of the figure (which is a polygon) (the vicinity being the region within the predetermined distance from the side), the inclusion judgment value changes continuously (and abruptly) and smoothly from 0 (or almost 0) to 1 (or almost 1).

[0121] The inclusion determination value for the figure corresponding to the estimated region and the inclusion determination value for the figure corresponding to the correct region are both 0 or more and 1 or less in the entire region of the image and the learning image.

[0122] In this embodiment, in order to calculate the inclusion determination value for a virtual pixel with respect to a figure, the error calculation unit 22 calculates the inclusion determination value for the virtual pixel with respect to each side of the figure, and multiplies these inclusion determination values ​​together to calculate the inclusion determination value for the virtual pixel with respect to the figure.

[0123] 9 is a block diagram showing a more detailed internal functional configuration of the error calculation unit 22 according to this embodiment. As shown in the figure, the error calculation unit 22 includes a virtual pixel generation unit 1051, an inclusion determination unit 1052, an inclusion determination integration unit 1053, and an error function value calculation unit 1054. The functions of each unit are as follows:

[0124] The virtual pixel generation unit 1051 sets (generates) a large number of virtual pixels that are common to both the image and the learning image. Virtual pixels will be described in more detail later.

[0125] The inclusion determination unit 1052 calculates, for each virtual pixel, the side of the figure corresponding to the estimated area and the side of the figure corresponding to the correct area, the following equation (1) (where e is Napier's constant, k is a predetermined positive constant, d px is the distance from a reference line that passes through a specified point inside the figure and is parallel to the side to the virtual pixel, and d center is the distance from the reference line to the side in question) to determine the inclusion determination value for the side in question of the virtual pixel.

[0126] The inclusion determination integrating unit 1053 integrates the inclusion determination values ​​for each side of one virtual pixel obtained by the inclusion determination unit 1052, and obtains an inclusion determination value for the graphic (polygon) of that virtual pixel. Specifically, the inclusion determination integrating unit 1053 multiplies all of the inclusion determination values ​​of the virtual pixel for all sides of the graphic obtained by the inclusion determination unit 1052, thereby obtaining an inclusion determination value for the graphic of that virtual pixel.

[0127] The error function value calculation unit 1054 calculates the error between the graphic of the estimated region and the graphic of the correct region based on the inclusion determination value. Specifically, the error function value calculation unit 1054 calculates the error based on the inclusion determination value, which is a value indicating whether a virtual pixel is contained in each graphic. More specifically, the error function value calculation unit 1054 calculates the sum and product of the inclusion determination value for the graphic corresponding to the estimated region and the inclusion determination value for the graphic corresponding to the correct region for each virtual pixel, determines the value obtained by subtracting the product from the sum as a union value, determines the value obtained by the product as an intersection value, and calculates the error by dividing the sum of the intersection values ​​for all virtual pixels by the sum of the union values ​​for all virtual pixels.

[0128] Next, the flow of processing by the error calculation unit 22 will be described with an example.

[0129] The virtual pixel generation unit 1051 generates virtual pixels for the image for which error calculation is to be performed. Virtual pixels are pixels virtually arranged at predetermined intervals on the surface of the image. When expressed in xy Cartesian coordinates, for example, if the y coordinate ranges from 0.0 to 1.5, the x coordinate ranges from 0.0 to 1.0, and the interval between the x and y coordinates of the virtual pixels is 0.5, the virtual pixels have the following 12 coordinate values: (x,y)= (0.0,0.0),(0.0,0.5),(0.0,1.0),(0.0,1.5),(0.5,0.0),(0.5,0.5),(0.5,1.0),(0.5,1.5),(1.0,0.0),(1.0,0.5),(1.0,1.0),(1.0,1.5),

[0130] The coordinate values ​​of a virtual pixel may differ from the coordinate values ​​of an actual pixel in the image. Furthermore, the spacing between virtual pixels (0.5 in the above example) can be set as appropriate, regardless of the spacing between actual pixels. The x and y coordinate values ​​of a virtual pixel are expressed as real numbers and do not need to be integers. Because there is such a high degree of freedom in the coordinate values ​​of virtual pixels, this method is also suitable for handling small shapes, for example, those whose size in the x and y directions is one pixel or less. Note that the narrower the spacing between virtual pixels (i.e., the greater the number of virtual pixels per given length), the more calculations are required to calculate the error, but the accuracy of the calculation improves.

[0131] FIG. 10 is a schematic diagram showing an example of the arrangement of real pixels and virtual pixels in an image. This diagram shows only a portion of the image. In this diagram, real pixels 1901 are represented by squares, and virtual pixels 1902 are represented by black circles. In this example, both the real pixels 1901 and the virtual pixels 1902 are arranged in a square array. In this example, 16 virtual pixels 1902 are arranged corresponding to one real pixel 1901. In other words, the density of the virtual pixels 1902 is four times that of the real pixels 1901 in both the vertical and horizontal directions. The arrangement pattern shown in FIG. 3 is merely an example. The virtual pixels may be arranged in a delta array, for example, instead of a square array. The virtual pixels may also be arranged in a non-regular array. The relationship between the density of the real pixels and the density of the virtual pixels is also arbitrary. In either case, it is desirable that the density of the virtual pixels be as uniform as possible and not vary significantly depending on the position on the screen.

[0132] The inclusion determination unit 1052 performs an inclusion determination for each side for each of the virtual pixels set above. That is, the inclusion determination unit 1052 determines on which side of each side each virtual pixel is located. As a processing procedure, the inclusion determination unit 1052 may rotate the image so that the side of interest is, for example, horizontal, and in that state determine whether each virtual pixel is located above or below that side.

[0133] FIG. 11 is a schematic diagram illustrating the processing performed by the inclusion determination unit 1052. This diagram shows a state in which one of the figures for which an error is to be calculated exists on the xy plane. In this diagram, 2001 is one of the figures for which an error is to be calculated. Figure 2001 is either a figure representing the resulting area estimated by the estimation unit 12 (after conversion by the conversion unit 13) or a figure representing the correct area supplied by the learning data supply unit 104. In this example, figure 2001 is a quadrangle having sides 2011, 2012, 2013, and 2014. The figure for which an error is to be calculated is not limited to a quadrangle, and may be any polygon. However, the figure to be calculated is a convex hull polygon (polygon).

[0134] FIG. 12 is another schematic diagram illustrating the processing performed by the inclusion determination unit 1052. The state shown in FIG. 5 is the result of rotating a plane (xy plane) including the graphic 2001 shown in FIG. 11. The inclusion determination unit 1052 rotates the graphic 2001 so that the currently focused edge (edge ​​2011 in this example) of the graphic 2001 becomes horizontal. The dashed line 2021 shown in the figure is a horizontal reference line. That is, the dashed line 2021 is a line parallel to the edge 2011. The dashed line 2021 is a line passing through the center point of the graphic 2001. The center point may be determined by the average values ​​of the x and y coordinates of the vertices constituting the graphic 2001. In other words, when viewed in the vertical direction in this figure, the dashed line 2021 is located more inward of the graphic 2001 than the edge 2011. Conversely, in the vertical direction, the side of side 2011 opposite dashed line 2021 is outside figure 2001. Here, the distance from dashed line 2021 to side 2011 is 1.5. Also, the distance from dashed line 2021 to virtual pixel 2022 is 2.2, and the distance from dashed line 2021 to virtual pixel 2023 is 0.6. In this case, the inclusion determination unit 1052 calculates the inclusion determination value using the following formula (1).

[0135]

number

[0136] In equation (1), d px is the distance from the reference line (the dashed line 2021 in the example) to the virtual pixel to be determined. center is the distance from the reference line to the edge in question (edge ​​2011 in this example). k is a parameter that takes a predetermined positive value, for example, k=10. e is Napier's constant. d px d center If d is smaller than , the inclusion judgment value calculated by formula (1) will be close to 1. px d center If d is larger than , the inclusion judgment value calculated by formula (1) will be close to 0. px d center, the inclusion judgment value calculated by formula (1) is 0.5. In other words, when viewed from the vertical position in the example of FIG. 12, if the virtual pixel is located below the side 2011, the inclusion judgment value approaches 0, and if the virtual pixel is located above the side 2011, the inclusion judgment value approaches 1. Note that the inclusion judgment value changes smoothly and steeply in the vicinity of the side 2011. The value of the parameter k above is a value for controlling the degree of abruptness of the change in the inclusion judgment value in the vicinity of the side 2011. However, the function for calculating the inclusion judgment value expressed by formula (1) is d px In the whole area of ​​d px is differentiable with respect to

[0137] In the example shown in FIG. 12, the inclusion determination value of virtual pixel 2022 with respect to side 2011 is calculated using equation (1) to be approximately 0.0009 (close to 0). This indicates that virtual pixel 2022 is located above side 2011. Furthermore, the inclusion determination value of virtual pixel 2023 with respect to side 2011 is calculated using equation (1) to be approximately 0.9999 (close to 1). This indicates that virtual pixel 2023 is located below side 2011.

[0138] The inclusion determination integration unit 1053 integrates the inclusion determinations for each side by the inclusion determination unit 1052 to determine an inclusion determination value for the entire graphic. Specifically, the inclusion determination integration unit 1053 calculates an integrated inclusion determination value for each virtual pixel by multiplying all the inclusion determination values ​​calculated for each side. As already explained, when the graphic is rotated as shown in FIG. 12 , that is, when the sides are horizontal and the center point of the graphic is located below the sides, the inclusion determination value of the virtual pixels below the sides is close to 1, and the inclusion determination value of the virtual pixels above the sides is close to 0. Furthermore, the inclusion determination value changes rapidly between 0 and 1 near the sides. In other words, when all the inclusion determination values ​​for each side of a certain virtual pixel are multiplied, the integrated inclusion determination value of the virtual pixels inside the graphic (e.g., a rectangle) is close to 1, and the integrated inclusion determination value of the virtual pixels outside the graphic is close to 0.

[0139] The inclusion determination integration unit 1053 calculates an integrated inclusion determination value for all virtual pixels. The sum of the integrated inclusion determination values ​​for all virtual pixels in an image is an approximation of the number of virtual pixels contained in the figure, and this value is proportional to an approximation of the area of ​​the figure. In other words, the sum of the inclusion determination values ​​calculated by the inclusion determination integration unit 1053 is a value that represents the area of ​​the figure, and is a differentiable value that can be calculated simply by calculating equation (1) and adding up those values.

[0140] The error function value calculation unit 1054 uses the calculation result of the inclusion determination integrating unit 1053 to calculate the error between the two figures input to the error calculation unit 105. Specifically, the error function value calculation unit 1054 performs the calculation described below with reference to FIG.

[0141] FIG. 13 is a schematic diagram illustrating the process performed by the error function value calculation unit 1054 when calculating the error between two figures. FIG. 13(A) shows the area of ​​the first figure, figure 2002. FIG. 13(B) shows the area of ​​the second figure, figure 2003. The more these two figures match, the smaller the error between them; the more they differ, the larger the error between them. To calculate such an error, the error function value calculation unit 1054 performs a calculation that can serve as an alternative to IoU (Intersection over Union), as described below. FIG. 13(C) shows a situation in which only a portion of the area of ​​figure 2002 and the area of ​​figure 2003 overlap. The calculation performed by the error function value calculation unit 1054 is based on the area of ​​the union of the two figures and the area of ​​the intersection of the two figures.

[0142] An intersection in IoU is the common part (intersection part) of two figures. In this embodiment, as a calculation to obtain a value corresponding to an intersection in IoU, the error function value calculation unit 1054 calculates the product of the integrated inclusion judgment values ​​of two figures for each virtual pixel. For one virtual pixel, if the inclusion judgment value of at least one of the two figures is close to 0, the product of the two inclusion judgment values ​​will be close to 0. If the inclusion judgment values ​​of both of the two figures are close to 1, the product of the two inclusion judgment values ​​will be close to 1. Moreover, the product of the two inclusion judgment values ​​is differentiable. The sum of the above products for all virtual pixels is the value corresponding to the intersection in this embodiment. A union in IoU is the union part of two figures. In this embodiment, to obtain a value corresponding to the intersection in IoU, the error function value calculation unit 1054 calculates the sum of the integrated inclusion judgment values ​​of the two shapes for each virtual pixel and then subtracts the above product from that sum. For the common portion of the two shapes, the sum of those inclusion judgment values ​​is close to 2. Therefore, by subtracting the product (close to 1) of the two inclusion judgment values ​​from that sum, the calculated value is close to 1. For the exclusive sum portion of the two shapes, the sum of those inclusion judgment values ​​is close to 1. Since the product of the two inclusion judgment values ​​for the exclusive sum portion is close to 0, the calculated value is close to 1. For other portions (portions that do not belong to either of the two shapes), both the sum and the product of those inclusion judgment values ​​are close to 0. Therefore, the result of subtracting that product from that sum is close to 0. The sum of the values ​​obtained by subtracting the product from the above sum for all virtual pixels is the value corresponding to the union in this embodiment. This value is also differentiable.

[0143] That is, the error function value calculation unit 1054 calculates the error between the two figures based on the following formula.

[0144] Formula: sum(intersection) / sum(union)

[0145] In this equation, union is a value equivalent to the union of each virtual pixel calculated by the error function value calculation unit 1054 using the above method. Also, intersection in this equation is a value equivalent to the intersection of each virtual pixel calculated by the error function value calculation unit 1054 using the above method. Also, sum(·) represents the operation (calculation) of taking the total sum of the values ​​of all virtual pixels in the image.

[0146] As explained above, both the union and the intersection in the above equation are differentiable. Also, the sum of each of them over all virtual pixels is also differentiable. Furthermore, the value expressed by the above formula (sum(intersection) / sum(union)) (the error calculated by the error function value calculation unit 1054) is also differentiable.

[0147] As described above, the error calculation unit 22 can calculate an appropriate error by using the function of formula (1) for two figures (both of which are convex hull figures (convex hull polygons)). Furthermore, because the error is differentiable, it is possible to update the parameters of the neural network that the estimation unit 12 has internally by error backpropagation. In other words, by using a sufficient amount of learning data and performing error backpropagation a sufficient number of times, the neural network (machine learning model) that the estimation unit 12 has is optimized.

[0148] As described above, in this embodiment, the figure representing the area to be detected may be any convex hull polygon of any orientation (not limited to a rectangle, etc.). Furthermore, as is clear from the error calculation process described above, the number of vertices of the figure (convex hull polygon) of the area (the area after conversion by the conversion unit 13) output as the estimation result by the estimation unit 12 may differ from the number of vertices of the figure (convex hull polygon) of the correct area provided by the training data supply unit 21. In other words, this embodiment solves the problems of the prior art and makes it possible to calculate the error between any convex hull polygons as the value of a differentiable function. This makes it possible to detect areas related to shapes other than rectangles using a machine learning model. This removes constraints on the problem of detecting areas with specific characteristics in an image, allowing the area detection device 1 to be used to solve more general problems.

[0149] Although the embodiment has been described above, the following modifications can also be implemented.

[0150] [Modification of the first embodiment regarding error calculation] In the embodiment already described, the procedure was explained in which an inclusion determination value for one virtual pixel and one side is calculated using formula (1), and the inclusion determination values ​​for all sides are multiplied to determine an inclusion determination value for the graphic (convex hull polygon) of that virtual pixel.As a modified example, here, a method for determining an inclusion determination value for the graphic of one virtual pixel will be described.

[0151] FIG. 14 is a schematic diagram illustrating how to determine the inclusion determination value according to this modified example. In the example shown in FIG. 14, the figure to be determined is a convex pentagon. The convex pentagon has sides 2031, 2032, 2033, 2034, and 2035. In this diagram, the region is divided into three regions using boundary lines (dashed lines). Region R1 is a region inside the figure (pentagon) and is a region that is a predetermined distance or more from all sides. Region R3 is a region outside the figure (pentagon) and is a region that is a predetermined distance or more from all sides. Region R2 is a region that is neither region R1 nor R3. In other words, region R2 is a region near one of sides 2031, 2032, 2033, 2034, and 2035. In other words, region R2 is a region that is within a predetermined distance from at least one of these sides. The width of region R2 in the direction perpendicular to each side is set to be sufficiently small. In this modified example, the inclusion judgment value is set to a value greater than or equal to 0 and less than or equal to 1. In this modified example, the inclusion judgment value in region R1 is set to 1 or approximately 1 (i.e., greater than or equal to 1-ε and less than or equal to 1), where ε is a positive constant sufficiently smaller than 1. In addition, the inclusion judgment value in region R3 is set to 0 or approximately 0 (i.e., greater than or equal to 0 and less than or equal to ε). In addition, the function for calculating the inclusion judgment value for a virtual pixel in this image is set to be continuous and differentiable in all regions within the image.

[0152] The error calculation unit 22 of the modified example calculates the error between the two figures based on such an inclusion determination value, using the calculation procedure already explained in the embodiment.

[0153] 14 corresponds to a more general form. The method of obtaining the inclusion determination value described in the embodiment is a special example of this general form.

[0154] As described above, according to this embodiment (including the modified examples), it is possible to use a machine learning model to detect areas of shapes other than rectangles. Furthermore, according to the embodiment (including the modified examples), it is possible to generate (provide) virtual pixels at any resolution and detect areas with precision.

[0155] The above has described in detail an embodiment of the present invention (including modified examples) with reference to the drawings, but the specific configuration is not limited to this embodiment, and also includes designs within the scope that do not deviate from the gist of the present invention.

[0156] [Application examples and effects of the first form of error calculation] A description will be given of an application example and the effects of the above-described area detection device 1. As an application example, the area detection device 1 is used to detect an area of ​​text contained in an image.

[0157] FIG. 15 is a schematic diagram showing an example of an image to be processed in this application example. FIG. 15(A) shows an image to be input to the area detection device 1. This image contains, as an example, the characters "abc." The area detection device 1 performs machine learning in advance using learning data so as to detect character areas. The trained area detection device 1 is operated in estimation execution mode, and the image of FIG. 15(A) is input. FIG. 15(B) shows an example of the detection results. The area detection device 1 detects areas containing the characters "a," "b," and "c." Each of the detected areas is quadrangular (not rectangular). Here, based on the detection results by the area detection device 1, the detected areas are separated into independent images, and areas other than the detected areas are masked with black (pixel value 0). FIG. 15(C) shows three images showing the results. The hatched areas in FIG. 15(C) are areas masked with black (pixel value 0). This masking is done to prevent unnecessary noise information from being included in the subsequent character recognition process. Then, character recognition was performed on these three images using a character recognition device (not covered by the present invention). As a result, the characters "a," "b," and "c" were correctly recognized. In other words, in this application example, detection of areas (character areas) having specific characteristics was performed as preprocessing for character recognition. In this demonstration experiment, the character recognition accuracy (F-value) improved from 70% when area detection was not performed to 73% when area detection preprocessing was performed. In other words, it was confirmed that the effects of the above embodiment were obtained.

[0158] [Second form of error calculation] Next, a second embodiment relating to error calculation will be described. Note that the matters already explained in the previous embodiment may not be explained below. Here, the explanation will focus on matters specific to this embodiment. Note that the polygons that are the subject of processing in this embodiment are polygons on a plane.

[0159] [Configuration and processing overview of polygon IoU calculation device] FIG. 16 is a block diagram showing a schematic functional configuration of a polygon IoU calculation device according to this embodiment. As shown in the figure, a polygon IoU calculation device 3051 includes a polygon information acquisition unit 3052, a convex hull division unit 3053, a convex hull IPM generation unit 3054, a polygon IPM generation unit 3055, and a polygon IoU approximate value calculation unit 3056. As with the previous embodiment, each functional unit of the polygon IoU calculation device 3051 according to this embodiment can also be realized by, for example, a computer and a program. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program. An overview of each function will now be described.

[0160] The polygon information acquisition unit 3052 externally acquires information about the two polygons for which IoU is to be calculated. The polygon information is information that indicates the shape, size, position, tilt (rotation angle), etc. of the polygon. The information about one polygon acquired by the polygon information acquisition unit 3052 can be expressed, for example, as information about the coordinates of all vertices of that polygon. However, the representation form of the polygon information may be other than the form presented here (coordinates of all vertices). The polygon information acquisition unit 3052 passes the polygon information about the two polygons for which IoU is to be calculated to the convex hull division unit 3053.

[0161] The convex hull division unit 3053 divides each polygon into convex hull polygons based on the polygon information received from the polygon information acquisition unit 3052. A convex hull polygon is, in other words, a convex polygon. A convex hull polygon is, in other words, a polygon with no interior angles greater than 180 degrees. The convex hull division unit 3053 may divide the received polygon into multiple convex hull polygons using various methods as appropriate. The method used by the convex hull division unit 3053 to divide a polygon into convex hull polygons does not need to be differentiable, and any method may be used. For example, the convex hull division unit 3053 may divide a given polygon into convex hull polygons using Delaunay triangulation (triangles are always convex hulls). As another example, the convex hull division unit 3053 may divide a given polygon into convex hull polygons using a rule-based method. The division of an arbitrary polygon into convex hull polygons can be realized using existing technology.

[0162] That is, the convex hull division unit 3053 obtains one or more divided convex hull polygons by dividing one polygon in the given image. If there is no need to divide one given polygon, the convex hull division unit 3053 may output the given polygon as is as one divided convex hull polygon.

[0163] Note that the greater the number of divisions into convex hulls as a result of division by the convex hull division unit 3053, the greater the calculation load in subsequent processing. In other words, it is preferable for the convex hull division unit 3053 to perform processing that divides into as few convex hulls as possible. Furthermore, if the polygon passed from the polygon information acquisition unit 3052 is a convex hull from the beginning, the convex hull division unit 3053 may omit (skip) the process of dividing that polygon into convex hull polygons. Examples of division by the convex hull division unit 3053 will be described later with reference to FIGS. 11 and 12.

[0164] The convex hull IPM generation unit 3054, which will be described later, calculates an IPM for each edge of the convex hull polygon and generates an IPM for the convex hull polygon by performing a set intersection operation (described in detail below) on the IPMs of all those edges. To achieve this, the polygon to be processed by the convex hull IPM generation unit 3054 must be a convex hull polygon. If the polygon to be processed contains one or more interior angles of 180 degrees or greater, the above-described set intersection operation cannot be applied effectively. However, the convex hull division unit 3053 divides a polygon of an arbitrary shape, and passes a set consisting only of convex hull polygons to the convex hull IPM generation unit 3054 as the processing target. This allows the polygon IoU calculation device 3051 of this embodiment to accurately calculate an approximate IoU value.

[0165] The convex hull IPM generator 3054 performs an inclusion determination for each convex hull polygon resulting from division by the convex hull division unit 3053, and generates an IPM (Inner Pixel Map) for each convex hull polygon. Inclusion determination is a process for determining whether each pixel constituting an image, when a convex hull polygon is depicted on an image consisting of a large number of pixels, is inside or outside the convex hull polygon. The IPM is data that numerically indicates whether each pixel in the image is inside a specific polygon. The convex hull IPM generated by the convex hull IPM generator 3054 in this embodiment is data that represents, as a map, numerical values ​​indicating whether each pixel in the image is inside a given convex hull polygon. In other words, the IPM generated by the convex hull IPM generator 3054 is a map in which a value based on the result of the inclusion determination is assigned to each pixel. The pixel values ​​in the IPM are greater than or equal to 0.0 and less than or equal to 1.0. Pixels determined to exist inside the convex hull polygon are assigned a pixel value of the convex hull IPM close to 1.0. Pixels determined to exist outside the convex hull polygon are assigned a pixel value of the convex hull IPM close to 0.0. To make this IPM usable in neural network training, the pixel values ​​of the IPM are not simply assigned discrete values ​​of 1 or 0, but are instead assigned values ​​of a differentiable function (continuous values).

[0166] That is, the convex hull IPM generator 3054 generates an inside pixel map (IPM) that numerically indicates whether each pixel in the image is inside the convex hull polygon for each of the convex hull polygons obtained by the convex hull division unit 3053. The convex hull IPM generator 3054 is also called a "convex hull inside pixel map generator." Details of the process by which the convex hull IPM generator 3054 creates an IPM for one convex hull polygon will be described later with reference to Figures 19 to 24.

[0167] As a more specific process, the convex hull IPM generation unit 3054 may generate an inside pixel map (IPM) for each edge of the convex hull polygon being processed, which numerically indicates whether each pixel in the image is on the same side or the opposite side of a specified pixel inside the convex hull polygon, based on the edge and its extension line, and by combining the inside pixel maps (IPM) for all edges of the convex hull polygon, generate an inside pixel map (IPM) which numerically indicates whether each pixel in the image is inside the convex hull polygon.

[0168] Furthermore, as a more specific process, the convex hull IPM generation unit 3054 may generate an inside pixel map (IPM) that numerically indicates whether or not the pixel is inside the convex hull polygon by selecting the minimum value (a cross-set operation described later) for each pixel in the inside pixel map for each side when synthesizing the inside pixel maps for all sides of the convex hull polygon.

[0169] The polygon IPM generation unit 3055 generates an IPM of the polygon before division by the convex hull division unit 3053 by combining the IPMs of each convex hull polygon divided by the convex hull division unit 3053. In other words, the polygon IPM generation unit 3055 generates an IPM of the polygon before division (called a polygon IPM) by combining the IPMs of multiple convex hull polygons (called convex hull IPMs) generated by the convex hull IPM generation unit 3054.

[0170] That is, the polygon IPM generation unit 3055 generates an inside pixel map (IPM) that numerically indicates whether each pixel in the image is inside the polygon before being divided by the convex hull division unit 3053, by combining the inside pixel maps (IPM) of the divided convex hull polygons generated by the convex hull IPM generation unit 3054 (convex hull inside pixel map generation unit). The polygon IPM generation unit 3055 is also called a "polygon inside pixel map generation unit." The method by which the polygon IPM generation unit 3055 generates the polygon IPM will be described in detail later.

[0171] The polygon IoU approximation calculation unit 3056 calculates the IoU of two polygons. Strictly speaking, the polygon IoU approximation calculation unit 3056 calculates an approximation of the IoU of the two polygons. The polygon IPM generation unit 3055 generates an IPM (polygon IPM) for each of the two polygons. Based on the two polygon IPMs generated by the polygon IPM generation unit 3055, the polygon IoU approximation calculation unit 3056 calculates an IoU approximation for the two polygons, an IPM corresponding to the union polygon of the two polygons, and an IPM corresponding to the intersection polygon of the two polygons. The polygon IoU approximation calculation unit 3056 uses the IPM of the intersection polygon and the IPM of the union polygon to calculate an approximation of the IoU of the two polygons.

[0172] In other words, based on the inside pixel maps (IPM) generated by the polygon IPM generation unit 3055 (polygon inside pixel map generation unit) for each of two given polygons in the image, the polygon IoU approximation calculation unit 3056 generates an inside pixel map (IPM) for the intersection area of ​​the two polygons, which numerically indicates whether each pixel in the image is inside the intersection area of ​​the two polygons, and an inside pixel map (IPM) for the union area of ​​the two polygons, which numerically indicates whether each pixel in the image is inside the union area of ​​the two polygons; calculates an approximation of the area of ​​the intersection area based on the inside pixel map (IPM) for the intersection area; calculates an approximation of the area of ​​the union area based on the inside pixel map (IPM) for the union area; and calculates an approximation of the IoU (Intersection over Union) value for the two given polygons by dividing the approximation of the area of ​​the intersection area by the approximation of the area of ​​the union area. The polygon IoU approximate value calculation unit 3056 is also simply referred to as an “IoU approximate value calculation unit.” The method by which the polygon IoU approximate value calculation unit 3056 calculates the approximate value of the polygon IoU will be described in detail later.

[0173] The polygon IoU calculation device 3051 having the above configuration has the following features. That is, the calculation process in which the convex hull IPM generation unit 3054 (convex hull inner pixel map generation unit) generates an inner pixel map for the convex hull polygon, the calculation process in which the polygon IPM generation unit 3055 (polygon inner pixel map generation unit) generates an inner pixel map (IPM) for the polygon before division based on the inner pixel map (IPM) for the convex hull polygon after division, and the calculation process in which the polygon IoU approximation calculation unit 3056 (IoU approximation calculation unit) calculates an approximation of the IoU value based on the inner pixel map for each of the two polygons are calculated only within the ranges of a function that is continuous and differentiable within the image region, a maximum function that returns the maximum value of arguments, and a minimum function that returns the minimum value of arguments. This makes it possible to train a model using errors calculated based on the approximation of the IoU value.

[0174] [Details of processing within the polygon IoU calculation device 3051] Next, the processing in each part of the polygon IoU calculation device 3051 will be described in further detail.

[0175] [How to divide a given polygon into multiple convex hull polygons] 17 and 18 show the polygons P A and P B 17A is a schematic diagram showing an example in which the convex hull dividing unit 3053 divides each of the polygons P into a plurality of convex hull polygons. A Polygon P A is an octagon, but it is not a convex octagon. Also, (B) of FIG. 17 shows the polygon P A This is an example of dividing a polygon into multiple convex hull polygons. A is Polygon P A1 and Polygon P A2 and Polygon P A3 In FIG. 18, (A) shows the polygon P before division.B Polygon P B is a pentagon, but it is not a convex pentagon. Also, (B) of FIG. 12 shows the polygon P B This is an example of dividing a polygon into multiple convex hull polygons. B is Polygon P B1 and Polygon P B2 The polygon P can be divided into two convex hull polygons. B1 is the convex hull quadrilateral, and polygon P B2 is a convex hull triangle. The convex hull division unit 3053 passes information about the convex hull polygons obtained as a result of the division (for example, information about the coordinates of all vertices of each convex hull polygon) to the convex hull IPM generation unit 3054.

[0176] [How to generate the convex hull IPM] Next, the method by which the convex hull IPM generation unit 3054 generates the convex hull IPM will be described in detail with reference to FIGS.

[0177] FIG. 19 shows a polygon P, which is an example of a convex hull polygon to be processed by the convex hull IPM generation unit 3054. A2 19 is a schematic diagram showing the polygon P A2 is the polygon P in FIG. A This convex hull polygon P A2 Let the four edges (straight lines, line segments) that make up the rectangle be L1, L2, L3, and L4.

[0178] FIG. 20 shows the convex hull polygon P A2 1 is a schematic diagram illustrating the process of the convex hull IPM generation unit 3054 for generating the IPM of the convex hull polygon P A2 To generate the IPM, first we define the convex hull polygon P A2 The IPM for each edge of is calculated. Figure 20 shows the process of calculating the IPM for edge (straight line) L2 as an example.

[0179] In FIG. 20, an image 3501 is a convex hull polygon PA2 When we look at the edge (straight line) L2, the line L 2P is parallel to the edge (straight line) L2 and is part of the convex hull polygon P A2 The "predetermined point" here is, for example, a line passing through a predetermined point (pixel) that exists inside the convex hull polygon P A2 For example, the pixel at the center of the convex hull polygon P A2 The average position of the coordinates of all vertices of the convex hull polygon P A2 The center point of the convex hull polygon P A2 A line L passing through the center point of 2P can be called the "center line". ij (i and j are horizontal and vertical coordinate values ​​in the image 3501, respectively) to calculate the IPM value at pixel (point) P ij From the above line L 2P Let the foot of the perpendicular line be pixel (point) P0. ijs is the pixel (point) P ij is the distance from the pixel (point) P0. s is the line L2 and the line L 2P At this time, the convex hull IPM generation unit 3054 calculates the distance between the pixel P ij IPM value in IPM ij is calculated by the following formula (2). Note that formula (2) has the same meaning as formula (1) used in the first embodiment. Note that k in formula (2) is a hyperparameter.

[0180]

number

[0181] The pixel P calculated by the convex hull IPM generation unit 3054 ij IPM value of IPM ij is the following value: IPM ij is a value greater than 0.0 and less than 1.0. Also, pixel P ijHowever, for the line in question (line L2 in FIG. 20), the convex hull polygon (convex hull polygon P A2 ) is on the same side as a given point (e.g., the center point), ij takes a value close to 1.0. Pixel P ij However, for the line in question (line L2 in FIG. 20), the convex hull polygon (convex hull polygon P A2 ) on the opposite side of a given point (e.g., the center point), ij takes a value close to 0.0. Note that in the entire area of ​​image 3501, IPM ij The value of IPM does not change discretely but changes continuously. ij The value of is differentiable. ij The value of changes rapidly between values ​​close to 0.0 and values ​​close to 1.0 in the vicinity of the line of interest (line L2 in FIG. 20). The degree of rapidity of this change depends on the value of k in equation (2). In other words, IPM ij The value of is the sum of the convex hull polygon (in FIG. 20, the convex hull polygon P A2 ) on the same side as the predetermined point is the side closer to 1.0, and on the opposite side to the predetermined point is the side closer to 0.0.

[0182] The higher the value of k, the greater the IPM near the edge under consideration. ij The value of k changes rapidly. In other words, the higher the value of k, the more accurate the IoU value the polygon IoU calculation device 3051 can calculate. However, if the value of k is set too high, the IPM ijThe differential value of the value of k in the vicinity of the edge of interest becomes too high, and in some cases, this may cause an overflow in the calculation process during machine learning. Whether or not an overflow occurs in the calculation also depends on the calculation environment (such as the bit length of the calculation element). According to the results of experiments conducted by the inventors of the present application, when calculations are performed in a commonly used 32-bit width calculation environment, the polygon IoU calculation device 3051 can calculate a more accurate IoU without causing an overflow in the calculation by using k in the range of 30≦k≦50.

[0183] The convex hull IPM generation unit 3054 generates a convex hull polygon (in FIG. 20, the convex hull polygon P A2 ) for each edge of the IPM using the above formula (2). ij The convex hull IPM generation unit 3054 calculates the value of ij The value of is a base value for determining the degree of inclusion of each pixel in the convex hull polygon.

[0184] 21, 22, 23, and 24 are the convex hull polygon P shown in FIG. A2 1 is a schematic diagram showing the IPM calculated by the convex hull IPM generator 3054 for the edges L1, L2, L3, and L4 of the pixel P ij The IPM value of P ij,L1 ,P ij,L2 ,P ij,L3 ,P ij,L4 In FIG. 21, in the hatched area above the side L1 and its extension, ij,L1 The value of is close to 1.0. In the unhatched area below side L1 and its extension, ij,L1 The value of is close to 0.0. In Fig. 22, in the hatched area on the upper right side of the side L2 and its extension, ij,L2 The value of is close to 1.0. In the unhatched area on the lower left side of side L2 and its extension, ij,L2 The value of is close to 0.0. In Fig. 23, in the hatched area below the side L3 and its extension,ij,L3 The value of is close to 1.0. In the unhatched area above side L3 and its extension, ij,L3 The value of is close to 0.0. In Fig. 24, in the hatched area on the upper left side of the side L4 and its extension, ij,L4 The value of is close to 1.0. In the unhatched area on the lower right side of side L4 and its extension, ij,L4 The value of is close to 0.0. Note that in the entire area of ​​the image 3501, the IPM ij,L1 ,IPM ij,L2 ,IPM ij,L3 , and IPM ij,L4 Each value of is continuous and differentiable.

[0185] The convex hull IPM generation unit 3054 generates a pixel P ij IPM is the IPM value for the convex hull polygon of ij The edges of the target convex hull polygon are generally expressed as L1, L2, . . ., L M (M is the number of sides of the convex hull polygon), the convex hull IPM generation unit 3054 generates the IPM ij,L1 ,IPM ij,L2 ,···,IPM ij,LM By performing the intersection operation of the two, the IPM value of the convex hull polygon is obtained. ij Ask for.

[0186] Here, the intersection operation is an IPM that has values ​​close to 0.0 or close to 1.0 in most areas. ij,L1 ,IPM ij,L2 ,···,IPM ij,LM Regarding IPM ij,L1 ,IPM ij,L2 ,···,IPM ij,LM is an operation that returns a value close to 1.0 as the operation result if all of are close to 1.0, and returns a value close to 0.0 as the operation result in all other cases.

[0187] The convex hull IPM generation unit 3054 realizes the intersection operation by, for example, calculating the following equation (3): In equation (3), minimum( ) is a function that returns the minimum value.

[0188]

number

[0189] Alternatively, the convex hull IPM generator 3054 may use multiplication, as in the first embodiment. That is, the convex hull IPM generator 3054 may realize the intersection operation by performing the calculation of the following equation (4):

[0190]

number

[0191] The convex hull IPM generator 3054 may perform the intersection operation using a calculation process other than equation (3) or equation (4). When the convex hull IPM generator 3054 performs the intersection operation using equation (3), the amount of calculation required for the intersection operation can be reduced compared to when equation (4) is used. Furthermore, according to experimental results conducted by the inventors of the present application, performing the intersection operation using equation (3) can calculate a more accurate IoU value than when equation (4) is used. This is thought to be because errors in the numerical values ​​used to calculate the IoU value (such as the degree to which the IPM value for pixels inside the convex hull polygon deviates below 1.0) are amplified by performing multiple multiplications.

[0192] Note that the calculation shown in equation (3) is not differentiable in the entire region of image 3501. However, it has been confirmed through experiments that even if a non-differentiable region exists locally in image 3501, this does not pose a problem in machine learning of a model such as a neural network.

[0193] That is, the convex hull IPM generator 3054 generates an IPM for the convex hull polygon by performing an intersection operation on the IPMs generated for each edge of the convex hull polygon. The convex hull IPM generator 3054 passes the generated IPM for the convex hull polygon to the polygon IPM generator 3055.

[0194] [How to generate a polygon IPM based on a convex hull polygon IPM] The polygon IPM generation unit 3055 receives the IPMs of the convex hull polygons from the convex hull IPM generation unit 3054 and synthesizes them to generate the IPM of the original polygon before it was divided by the convex hull division unit 3053. Specifically, the polygon IPM generation unit 3055 generates the IPM of the original polygon by performing a union operation on the IPMs of each convex hull polygon resulting from the division. The K convex hull polygons obtained by dividing the original polygon P are denoted as P1, P2, . . . , P K Let pixel P in the image ij Convex hull polygons P1, P2, , P K The IPM values ​​of ij,P1 ,IPM ij,P2 ,···,IPM ij,PK The polygon IPM generation unit 3055 generates the IPM ij,P1 ,IPM ij,P2 ,···,IPM ij,PK The IPM of the original polygon P is obtained by performing a union operation on ij Ask for.

[0195] Here, the union operation is the operation to find IPMs that have values ​​close to 0.0 or close to 1.0 in most areas. ij,P1 ,IPM ij,P2 ,···,IPM ij,PK Regarding IPM ij,P1 ,IPM ij,P2 ,···,IPM ij,PK is an operation that returns a value close to 0.0 as the operation result if all of are close to 0.0, and returns a value close to 1.0 as the operation result in all other cases.

[0196] The polygon IPM generating unit 3055 realizes the union operation by, for example, calculating the following equation (5): In equation (5), maximum( ) is a function that returns the maximum value.

[0197]

number

[0198] Alternatively, the polygon IPM generator 3055 may implement the union operation using arithmetic operations (addition and multiplication). The union operation for two convex hull polygons (e.g., P1 and P2) can be implemented by the following formula (6):

[0199]

number

[0200] Using the above formula (6), we can perform a union operation on K convex hull polygons for any K (K≧2), as shown in the following formula (7).

[0201]

number

[0202] That is, the polygon IPM generation unit 3055 performs a union operation on the convex hull polygons P1 and P2 based on equation (6), then performs a union operation on the union polygon (P1∪P2) and the convex hull polygon P3 in the same way, and so on, to finally obtain the union polygon (P1∪P2∪∪P K ) we can obtain the IPM for this union polygon (P1 ∪ P2 ∪ ∪ P K ) is the polygon P before being divided by the convex hull dividing unit 3053.

[0203] The polygon IPM generation unit 3055 can generate an IPM for the polygon P by using either a method using equation (5), a method using equations (6) and (7), or another method. Note that the calculation shown in equation (5) is not differentiable over the entire region of the image 3501. However, experiments have confirmed that even if non-differentiable regions exist locally within the image 3501, this does not pose a problem for machine learning of models such as neural networks.

[0204] It has been confirmed through experiments that the method using equation (5) requires less computational effort than the methods using equations (6) and (7), and that the accuracy of the final IoU approximation is higher.

[0205] That is, the polygon IPM generation unit 3055 can obtain the IPM of the polygon before division by performing a union operation based on the IPM of each convex hull polygon resulting from division by the convex hull division unit 3053. A and Polygon P B When calculating the error between the polygon P A and Polygon P B The IPMs of both polygons are passed to the polygon IoU approximation calculation unit 3056.

[0206] [Method for calculating an approximate value of polygon IoU as the error between two polygons] The polygon IoU approximation calculation unit 3056 receives the IPMs of the two polygons for which the error is to be calculated from the polygon IPM generation unit 3055. To calculate the IoU of these two polygons, the polygon IoU approximation calculation unit 3056 first calculates the IPM of the union polygon and the IPM of the intersection polygon of these two polygons. The method for calculating the IPM of the union polygon of two polygons has already been described as part of the processing by the polygon IPM generation unit 3055. The method for calculating the IPM of the intersection polygon of two polygons has already been described as part of the processing by the convex hull IPM generation unit 3054. Specifically, the IPM of the union polygon of two polygons can be calculated using equation (5) or equation (6), etc. The IPM of the intersection polygon of two polygons can be calculated using equation (3) or equation (4), etc.

[0207] The IPM has information on the IPM value for each pixel. Therefore, the polygon IoU approximation calculation unit 3056 calculates (an approximate value of) the area of ​​the union polygon of two polygons as the sum of the values ​​of each pixel in the IPM of the union polygon of the two polygons. The polygon IoU approximation calculation unit 3056 also calculates (an approximate value of) the area of ​​the intersection polygon of two polygons as the sum of the values ​​of each pixel in the IPM of the intersection polygon of the two polygons. The unit of the calculated area is the number of pixels. Then, the polygon IoU approximation calculation unit 3056 calculates the IoU approximation for the two pixels given by the following equation (8):

[0208]

number

[0209] [Processing procedure for calculating the IoU approximation] 25 is a flowchart showing the procedure of processing by which polygon IoU calculation device 3051 calculates an approximate value of IoU for two given polygons. As a prerequisite for this processing, polygon IoU calculation device 3051 is given information on the shapes and positions of a first polygon and a second polygon (for example, coordinate values ​​on the image of all vertices). Polygon information acquisition unit 3052 acquires information on these polygons. The processing procedure will be explained below with reference to this flowchart.

[0210] First, in step S1, the convex hull division unit 3053 divides the first polygon into convex hull polygons. The division processing method has already been described. If it has already been determined that the first polygon is a convex hull polygon and division is unnecessary, the given first convex hull polygon itself may be used as the convex hull polygon resulting from the division in this step.

[0211] Next, in step S2, the convex hull IPM generation unit 3054 calculates the IPM for each convex hull polygon obtained by the division in step S1. The method by which the convex hull IPM generation unit 3054 generates the IPM for the convex hull polygon has already been described.

[0212] Next, in step S3, the polygon IPM generation unit 3055 obtains the IPM of the original polygon (first polygon) before division by combining the IPMs of the convex hull polygons obtained in step S2. As already explained, the polygon IPM generation unit 3055 can combine the IPMs of multiple convex hull polygons by using a union operation.

[0213] Next, in step S4, the convex hull division unit 3053 divides the second polygon into convex hull polygons. If it has already been determined that the second polygon is a convex hull polygon and division is unnecessary, the given second convex hull polygon itself may be used as the convex hull polygon resulting from the division in this step.

[0214] Next, in step S5, the convex hull IPM generation unit 3054 calculates the IPM of each convex hull polygon obtained by the division in step S4.

[0215] Next, in step S6, the polygon IPM generation unit 3055 obtains the IPM of the original polygon (second polygon) before division by combining the IPMs of the convex hull polygons obtained in step S5.

[0216] Next, in step S7, the polygon IoU approximation calculation unit 3056 calculates the IPM of the union area of ​​the first polygon and the second polygon using the IPM of the first polygon obtained in step S3 and the IPM of the second polygon obtained in step S6. To generate the IPM of the union area, the polygon IoU approximation calculation unit 3056 uses the union operation described above.

[0217] Next, in step S8, the polygon IoU approximation calculation unit 3056 calculates the IPM of the intersection area of ​​the first polygon and the second polygon using the IPM of the first polygon obtained in step S3 and the IPM of the second polygon obtained in step S6. To generate the IPM of the intersection area, the polygon IoU approximation calculation unit 3056 uses the above-mentioned intersection operation.

[0218] Next, in step S9, the polygon IoU approximation calculation unit 3056 divides the sum of the pixel values ​​of the IPM of the intersection area calculated in step S8 (the sum of all pixels) by the sum of the pixel values ​​of the IPM of the union area calculated in step S7 (the sum of all pixels). The sum of the pixel values ​​of the IPM of the intersection area is an approximation of the area of ​​the intersection area. The result of the above division in this step is an approximation of the IoU of the two given polygons.

[0219] [Implementation using a computer program] At least a part of the functions of the polygon IoU calculation device 3051 of this embodiment can be realized using a computer and a program.

[0220] [Using IoU values ​​in neural network training] The IoU value (IoU approximation) calculated by the above method can be used in neural network training. Specifically, when training a neural network, an error function called (1-IoU approximation) can be used to train the neural network. The neural network training itself can be performed using a method of backpropagating the above error to the neural network. As already explained, the error function called (1-IoU approximation) is either a differentiable function or a function that is not differentiable (has locally non-differentiable parts) but can be used for neural network training. Note that when training is performed, the error between a hypothetical polygon estimated based on input data included in the training data and a ground truth polygon that is the ground truth data associated with the input data in the training data is calculated.

[0221] By calculating errors using the method of this embodiment, it is possible to eliminate or alleviate the constraints imposed by conventional techniques regarding polygons to be processed. For example, the method of this embodiment eliminates the constraint on the number of vertices of polygons to be processed (e.g., not limited to rectangles). Furthermore, the method of this embodiment does not limit the polygons to be processed to convex polygons (convex hull polygons). Furthermore, the method of this embodiment allows the two polygons to be used for error calculation to have different numbers of vertices. Therefore, for example, both a data set in which a correct answer is given as a 4-vertex polygon and a data set in which a correct answer is given as a 16-vertex polygon can be used as correct answer data, and these two data sets can also be mixed as correct answers. In other words, when preparing correct answer data for learning, additional work such as correction to match the number of vertices is not required.

[0222] Furthermore, when calculating errors using the method of this embodiment, it becomes possible to use a random crop technique to expand the training data. Random cropping is a technique for artificially increasing the amount of training data, by randomly cropping a portion of an image included in the training data. In other words, random cropping results in cropping off one or more of the top, bottom, right, or left sides of the image. In a polygonal object detection task, random cropping of a ground truth image included in the training data may result in part of the ground truth polygon being cropped (the cropping line may divide the ground truth polygon). This may change the number of vertices of the ground truth polygon, which has limited the use of random cropping in conventional techniques. However, when calculating the error between polygons using this embodiment, there is no limit to the number of vertices in the polygon, so the random cropping method can be used without any restrictions. In other words, this embodiment can automatically increase the variation in polygon shapes included in the training data.

[0223] [Experimental verification] For the task of detecting character string regions (polygons) contained in a planar image, an experiment was conducted using the error calculation method (IoU approximation calculation method) of this embodiment during neural network training. As a result, it was confirmed that the accuracy of character string region detection improved by approximately 2% (approximately 2 points) compared to the conventional technology.

[0224] [Third form of error calculation] Next, a third embodiment relating to error calculation will be described. Note that the following description may omit matters that have already been described in the embodiments. Here, the description will focus on matters unique to this embodiment. This embodiment is an invention of an area detection device that uses the polygon IoU calculation device 3051 of the "second embodiment relating to error calculation."

[0225] 26 is a block diagram showing a schematic functional configuration of the area detection device according to this embodiment. As shown in the figure, the area detection device 2 includes an image input unit 211, an estimation unit 212, a conversion unit 213, a learning data supply unit 221, and an error calculation unit 222. Each functional unit of the area detection device 2 of this embodiment can also be realized by, for example, a computer and a program. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0226] The image input unit 211, the estimation unit 212, the conversion unit 213, and the training data supply unit 221 in this embodiment have the same functions as the image input unit 11, the estimation unit 12, the conversion unit 13, and the training data supply unit 21 in Fig. 1, respectively. However, in this embodiment, the detection target region is not limited to a convex hull polygon and may be any polygon. Therefore, the correct region included in the training data is also not limited to a convex hull polygon and may be any polygon.

[0227] When the area detection device 2 operates in the learning mode, the error calculation unit 222 calculates the error between the figure of the area (area after conversion by the conversion unit 213) output as the estimation result by the estimation unit 212 and the figure of the correct area supplied by the learning data supply unit 21. As mentioned above, these areas are not limited to convex hull polygons. The error calculated by the error calculation unit 222 is used to adjust (optimize) the internal parameters of the neural network of the estimation unit 212 by the error backpropagation method.

[0228] As shown in FIG. 26, the error calculation unit 222 includes an internal polygon IoU calculation device 3051. This polygon IoU calculation device 3051 has been described in detail in "Third Mode Related to Error Calculation." The polygon IoU calculation device 3051 calculates (an approximate value of) the IoU between the graphic (polygon) of the area (the area after conversion by the conversion unit 213) output as an estimation result by the estimation unit 212 and the graphic (polygon) of the correct area supplied by the training data supply unit 221. The error calculation unit 222 uses (the approximate value of) the IoU calculated by the polygon IoU calculation device 3051 to calculate and output (1-IoU approximate value) as an error value.

[0229] That is, the estimation unit 212 includes an internal machine learning model, and estimates a region having a specific feature in an image by inputting an externally provided image into the machine learning model. The training data supply unit 221 supplies pairs of training images and correct regions corresponding to the training images for training the machine learning model included in the estimation unit 212. The error calculation unit 222 calculates the error between an estimated region (an estimated region after conversion by the conversion unit 213) estimated by the estimation unit 212 based on the training image supplied by the training data supply unit 221, and the correct region supplied by the training data supply unit 221 corresponding to the training image. Furthermore, both the figure corresponding to the estimated region and the figure corresponding to the correct region are arbitrary polygons. The error calculation unit 222 also includes an IoU calculation device 3051, which calculates the error (e.g., 1-IoU) based on an approximation of the IoU value calculated by the IoU calculation device 3051 for a polygon that is a figure corresponding to the estimated area and a polygon that is a figure corresponding to the ground truth area.

[0230] [Implementation using a computer program] At least a part of the functions of the area detection device 2 of this embodiment can be realized using a computer and a program.

[0231] The configuration of the "third embodiment related to error calculation" described above allows the area detection device 2 to learn a machine-learnable model (for example, a neural network) that the estimation unit 212 has internally, based on the error value calculated by the error calculation unit 222. In other words, the area detection device 2 according to this embodiment is not limited to detecting convex hull polygons, and can accurately detect areas formed as polygons of any shape.

[0232] [Fourth form of error calculation] Next, a fourth embodiment relating to error calculation will be described. Note that the matters already explained in the previous embodiments may not be explained below. Here, the explanation will focus on matters specific to the fourth embodiment relating to error calculation. The fourth embodiment relating to error calculation is an invention of an image processing device that uses the polygon IoU calculation device 3051 of the "second embodiment relating to error calculation."

[0233] 27 is a block diagram showing a schematic functional configuration of an image processing device according to the "fourth embodiment of error calculation." As shown in the figure, the image processing device 3 includes an image input unit 311, an estimation unit 312, a conversion unit 313, a learning data supply unit 321, and an error calculation unit 322. Each functional unit of the image processing device 3 according to the "fourth embodiment of error calculation" can also be realized by, for example, a computer and a program. Furthermore, at least some of the functions of each functional unit may be realized by a dedicated electronic circuit rather than a program.

[0234] The functions of the image input unit 311, the estimation unit 312, the conversion unit 313, the learning data supply unit 321, and the error calculation unit 322 are similar to the image input unit 211, the estimation unit 212, the conversion unit 213, the learning data supply unit 221, and the error calculation unit 222 in the "third embodiment of error calculation." The following description will focus on the differences between the "third embodiment of error calculation" and the "fourth embodiment of error calculation."

[0235] While the area detection device 2 of the "third embodiment of error calculation" is a device for detecting an area having a specific characteristic (for example, an area of ​​a character string), the image processing device 3 of this embodiment is not limited to the task of area detection but also performs any estimation of an input figure. That is, the estimation unit 312 of this embodiment is not limited to the task of detecting an area having a specific characteristic but performs any estimation process. However, the result of estimation by the estimation unit 312 is a polygon within an image. Tasks that the estimation unit 312 can execute other than the task of detecting an area having a specific characteristic include, for example, a task of generating a figure (polygon) that satisfies conditions expressed by input information (e.g., text information, numerical information, image information, composite information of two or more of text, numerical information, and image, or other general information). One example is a task that takes a map (topographical map) and the number of polygon vertices as constraint data as input, and approximates the shape (shape in plan view) of a lake or pond included in the map with a polygon having a specified number of vertices, and outputs the result.

[0236] The estimation unit 312 is realized by having a machine learning model such as a neural network inside. The parameters held inside the estimation unit 312 can be adjusted (optimized) based on the error calculated by the error calculation unit 322.

[0237] As shown in FIG. 27, the error calculation unit 322 includes an internal polygon IoU calculation device 3051. This polygon IoU calculation device 3051 has been described in detail in "Second Embodiment Related to Error Calculation." In this embodiment, the polygon IoU calculation device 3051 calculates (an approximate value of) the IoU between the graphic (polygon) of the area (however, the area after conversion by the conversion unit 313) output as the estimation result by the estimation unit 312 and the graphic (polygon) of the correct area supplied by the training data supply unit 321. The error calculation unit 322 uses (the approximate value of) the IoU calculated by the polygon IoU calculation device 3051 to calculate and output (1-IoU approximate value) as an error value.

[0238] That is, the estimation unit 312 includes an internal machine learning model, and outputs an estimated polygon as a result of estimating polygons in an image received from an external device by inputting the image into the machine learning model. The training data supply unit 321 supplies pairs of training images and ground truth polygons corresponding to the training images for training the machine learning model included in the estimation unit 312. The error calculation unit 322 calculates the error between the estimated polygon, which is the result of estimation by the estimation unit based on the training images supplied by the training data supply unit 321, and the ground truth polygon supplied by the training data supply unit 321 corresponding to the training image. The error calculation unit 322 includes an IoU calculation device 3051, and calculates the error (e.g., 1-IoU) based on an approximation of the IoU value calculated by the IoU calculation device 3051 between the estimated polygon and the ground truth polygon.

[0239] [Implementation using a computer program] At least a part of the functions of the image processing device 3 of the "fourth embodiment relating to error calculation" can be realized using a computer and a program.

[0240] With the configuration of this embodiment described above, the image processing device 3 can perform learning of a machine-learnable model (for example, a neural network) that the estimation unit 312 has internally, based on the error value calculated by the error calculation unit 322. That is, the image processing device 3 according to this embodiment can accurately perform a task in which the estimation result is not limited to a convex hull polygon, but a polygon of any shape.

[0241] According to at least one of the above-described embodiments, it is possible to calculate the IoU or its approximation for two given polygons. Furthermore, the IoU or its approximation is calculated as a value of a function that is differentiable with respect to most of the image area. Even if the function used to calculate the IoU or its approximation has locally non-differentiable locations within the image, the characteristics of the maximum function or minimum function allow for training of a neural network or the like without any problems. In other words, it is possible to train a machine learning model that accepts a specific polygon (e.g., any polygon with no limitations on size, even a polygon with a complex shape) as the correct answer. This enables highly accurate detection of areas within an image and other image-related processing.

[0242] FIG. 28 is a block diagram showing an example of the internal configuration of each device, such as the area detection device 1, area detection device 2, image processing device 3, and polygon IoU calculation device 3051. Each device can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0243] When the area detection device 1, area detection device 2, image processing device 3, polygon IoU calculation device 3051, etc. are implemented by a computer and a program, the program for realizing the functions of each device is recorded on a computer-readable recording medium. The program recorded on this recording medium may then be loaded into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include devices that temporarily and dynamically store programs, such as communication lines when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a certain period of time, such as volatile memory within the computer systems that serve as the server or client in such cases. Furthermore, the above program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

[0244] The method for realizing a predetermined function using machine learning techniques is as follows. Here, for convenience, a unit of function that can be machine-learned is referred to as a "function unit." The function unit calculates data to be output based on input data. The function unit uses internal parameter values ​​when calculating the data to be output. The internal parameters can be updated and optimized through a learning process. When training the function unit, training data is used. The training data includes input data to the function unit and a correct answer for output data calculated based on the input data. During the learning process, the function unit reads the input data included in the training data and references output data based on the input data and using the internal parameter values ​​at that time. This output data is an estimated value calculated based on the internal parameters at that time and does not necessarily coincide with the correct answer. To update the internal parameters, the difference between the estimated value calculated and output by the function unit based on the input data at that time and the correct answer corresponding to the input data is calculated. This difference is called an error, loss, etc. The difference calculated here may be, for example, the absolute value of the difference between scalars, a squared error, a cross-entropy error, or a difference calculated by other methods. Based on the calculated difference, the internal parameter values ​​can be updated using backpropagation. This operation adjusts the internal parameter values ​​so as to reduce the error. By performing the above operation multiple times (multiple times) using a predetermined amount (large amount) of training data, the internal parameter values ​​are optimized. In other words, the functional unit is adjusted to perform the processing exemplified by the training data provided. After a sufficient amount of training has been performed, the training process may be terminated. The set of trained internal parameter values ​​at this time may be stored in a storage device or the like, enabling estimation based on the training results. A function that enables data input and output, including the internal parameters, may be called a "model." In other words, a model includes internal parameters. Training the functional unit may be called "model training."One way to implement the model is to use a neural network, and depending on the type of processing to be performed, an appropriate form of neural network can be used.

[0245] According to the embodiment described above, completely free polygons are not generated, but rather are constrained to polygons that can be generated by a predetermined procedure. This prevents extremely unnatural polygons from being generated in area detection for various purposes. In other words, this embodiment stabilizes polygon generation. This makes it possible to use data sets for training that could not be used for training with conventional methods. Therefore, since more training data can be used, improvement in area detection accuracy can be expected. In other words, according to this embodiment, appropriate and favorable constraints can be applied to the area (polygon area) detected by the area detection device.

[0246] Experiments using the method of this embodiment have confirmed that data sets that could not be used for learning with conventional methods can now be used. In other words, by using this embodiment, more training data can be used than before, and improvement in detection accuracy can be expected.

[0247] Furthermore, according to the embodiment described above, an approximate IoU value for two arbitrary polygons can be calculated in a form that can be used as an error for training a machine learning model. The target polygons are not limited to convex hull polygons. Furthermore, there is no need to impose restrictions on the number of vertices of the target polygons.

[0248] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0249] The present invention can be used, for example, to perform image processing. Specifically, the present invention can be used for machine learning of models for image processing. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0250] 1,2 Area detection device 3. Image processing device 11 Image input unit 12 Estimation part 13 Conversion unit 21 Learning Data Supply Unit 22 Error calculation section 211 Image input unit 212 Estimation Department 213 Conversion Unit 221 Learning Data Supply Department 222 Error calculation section 311 Image input unit 312 Estimation Department 313 Conversion Unit 321 Learning Data Supply Department 322 Error calculation section 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus 1051 Virtual Pixel Generation Unit 1052 Inclusion determination section 1053 Inclusion Judgment Integration Unit 1054 Error function value calculation unit 3051 Polygon IoU Calculation Device (IoU Calculation Device) 3052 Polygon Information Acquisition Unit 3053 Convex hull division 3054 Convex Hull IPM Generation Unit (Convex Hull Inner Pixel Map Generation Unit) 3055 Polygon IPM Generation Unit (Polygon Inner Pixel Map Generation Unit) 3056 Polygon IoU Approximation Calculation Unit (IoU Approximation Calculation Unit)

Claims

1. an image input unit for acquiring an image to be processed; an estimation unit having a machine learning model, which inputs the image acquired by the image input unit into the model, and outputs a plurality of numerical values ​​output from the model as estimation results corresponding to the image; a conversion unit that determines, based on the plurality of numerical values ​​output by the estimation unit, the position on the image of a polygon center line formed by sequentially connecting (N / 2-1) line segments (N is an even number equal to or greater than 3) so that they do not branch, determines, as vertex positions, two points at both end points (two points) of the polygon center line and at connection points ((N / 2-2) points) where a line segment on the polygon center line connects to another line segment, the positions of two points extending in the width direction from the polygon center line as the center, and outputs, as the detected area, an estimated polygon formed by connecting a total of N of the determined vertex positions with sides; an error calculation unit that calculates an Intersection over Union (IoU) error between the estimated polygon determined based on the image acquired by the image input unit and a ground truth polygon that is a ground truth given corresponding to the image; Equipped with When performing machine learning of the model possessed by the estimation unit, an internal parameter of the model can be adjusted by error backpropagation based on the IoU error calculated by the error calculation unit. Area detection device.

2. The (N / 2-1) line segments determined by the conversion unit are all equal in length, The plurality of numerical values ​​output by the estimation unit are coordinates of the positions of the start point and end point of an initial line segment, which is a line segment located at either end of the determined polygon center line; The magnitude of the displacement angle representing the direction of (N / 2-2) line segments sequentially connected from the initial line segment; (N / 2) widths when determining the positions of two points extending in the width direction from each of the (N / 2) total points, namely, the both end points (two points) of the polygon center line and the connecting points ((N / 2-2) points), as vertex positions; is a number representing The area detection device according to claim 1 .

3. The displacement angle Δ represents the difference in direction between the two line segments connected at the connection point. ｉ (where i is an index for identifying a connection point), |D ｉ |≦2π / ((N / 2)-1) The constraint that The area detection device according to claim 2 .

4. an image input unit for acquiring an image to be processed; an estimation unit having a machine learning model, which inputs the image acquired by the image input unit into the model, and outputs a plurality of numerical values ​​output from the model as estimation results corresponding to the image; a conversion unit that determines, based on the plurality of numerical values ​​output by the estimation unit, the position on the image of a polygon center line formed by sequentially connecting (N / 2-1) line segments (N is an even number equal to or greater than 3) so that they do not branch, determines, as vertex positions, two points at both end points (two points) of the polygon center line and at connection points ((N / 2-2) points) where a line segment on the polygon center line connects to another line segment, the positions of two points extending in the width direction from the polygon center line as the center, and outputs, as the detected area, an estimated polygon formed by connecting a total of N of the determined vertex positions with sides; an error calculation unit that calculates an Intersection over Union (IoU) error between the estimated polygon determined based on the image acquired by the image input unit and a ground truth polygon that is a ground truth given corresponding to the image; Equipped with When performing machine learning of the model possessed by the estimation unit, an internal parameter of the model can be adjusted by error backpropagation based on the IoU error calculated by the error calculation unit. A program that enables a computer to function as an area detection device.

Images