Area detection device and program

Abstract

To provide an area detection device that uses a machine learning model to detect an area with a specific feature from an image, while preventing a shape of an area to be detected from being limited to a rectangle.SOLUTION: An area estimation part comprises a machine learning model in its interior to estimate an area with a specific feature in the image by inputting an externally provided image to the machine learning model. A learning data supply part supplies a pair of a learning image and a correct area corresponding to the learning image to perform learning of the machine learning model that the area estimation part comprises. An error calculation part calculates an error between the estimated region that is the result estimated by the area estimation part based on the learning image supplied by the learning data supply part and the correct area that corresponds to the learning image and is supplied by the learning data supply part. The error calculated by the error calculation part is to be differentiable over the entire image area when the area to be detected is a convex closure polygon.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a region detection device and a program. [Background technology]

[0002] Existing techniques exist for detecting specific regions within images using machine learning methods. In these existing techniques, a large amount of training data is prepared, consisting of pairs of training images and the ground truth regions contained within those images. Ground truth regions are regions that possess specific features. For example, ground truth regions are regions that contain specific objects, etc. Then, a neural network is trained using this training data. During training, the internal parameters of the neural network are adjusted so that the error between the region estimated by the neural network at that point and the ground truth region is minimized.

[0003] When training a neural network, the above error calculation must be performed using a differentiable function. That is, a differentiable function must be used that takes the estimated region and the correct region as input and outputs the error value. This is because the gradient of the error value in the multidimensional space of parameter values ​​is required when adjusting the internal parameters of the neural network.

[0004] Conventional techniques, partly due to the use of differentiable error function, have targeted rectangular regions when detecting specific areas within an image. Furthermore, the Intersection over Union (IoU) error function has been found to be suitable for calculating the error between the estimated region and the ground truth region (both of which are rectangular), and has been used for this purpose.

[0005] Furthermore, in conventional techniques, instead of constructing a neural network that directly determines the coordinate values ​​of a specific region, a mask image consisting of pixel values ​​between 0 and 1 is sometimes output from the neural network, and the region where the pixel values ​​exceed a predetermined threshold (e.g., 0.5) is output as the specific region of the estimation result. In this case, any shape, not limited to rectangles, can be obtained as the estimation result.

[0006] Non-Patent Document 1 describes the PIoU error function (PIoU loss) as an error function for determining the error for a rotated rectangle. The PIoU error function is a loss function for detecting rotated objects and is formulated to perform accurate rotation bounding box regression by utilizing both angle and IoU. The method described in Non-Patent Document 1 approximates the number of pixels inside the rotated rectangle using a differentiable function, and is a method that can obtain an error value similar to IoU. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Zhiming Chen,Kean Chen,Weiyao Lin,John See,Hui Yu,Yan Ke,Cong Yang,"PIoU Loss: Towards Accurate Oriented Object Detection in Complex Environments", European Conference on Computer Vision(ECCV), 2020, https: / / arxiv.org / pdf / 2007.09584.pdf [Overview of the project] [Problems that the invention aims to solve]

[0008] However, conventional technologies have the following problems that need to be solved.

[0009] Conventional techniques calculate errors using an error function for rectangular regions. In other words, both the region estimated by the neural network and the correct region are limited to rectangles. Therefore, conventional techniques have the problem that they cannot construct a neural network that can directly estimate the coordinate values ​​of regions with shapes other than rectangles.

[0010] One of the methods described as conventional techniques involves outputting a mask image from a neural network to enable estimation of arbitrary shapes. However, errors can occur when determining the set of coordinate values ​​to identify the estimation region based on the mask image generated by the neural network. For example, it is not possible to estimate a region smaller than the size of one pixel of the generated mask image. One possible method is to calculate the error using simple regression, such as treating the least-squares error between the ground truth coordinates and the estimated coordinates as the error, but generally, regression of coordinate values ​​is not highly accurate. Furthermore, there is a constraint that the number of vertices in the shape of the ground truth region and the number of vertices in the shape of the estimated region must be the same. In other words, it is not possible to calculate the error based on the coordinate values ​​of vertices between, for example, a quadrilateral region and a pentagonal region.

[0011] Furthermore, while it is possible to calculate the error between regions using the PIoU error function described in Non-Patent Document 1 for a rotated rectangle, there is a problem in that it is difficult to estimate the shape in finer detail than the pixel size of the input image.

[0012] This invention was made based on the above-mentioned problem recognition, and aims to provide a region detection device and program that can detect regions with specific features within an image using machine learning models such as neural networks, and in which the detected regions are not limited to rectangles. [Means for solving the problem]

[0013] [1] To solve the above problems, a region detection device according to one aspect of the present invention comprises: a region estimation unit that internally includes a machine learning model and estimates a region having specific features in an image by inputting an image passed from the outside into the machine learning model; a learning data supply unit that supplies a pair of a learning image and a correct region corresponding to the learning image for training the machine learning model included in the region estimation unit; an error calculation unit that calculates the error between an estimated region which is the result of the region estimation unit's estimation based on the learning image supplied by the learning data supply unit and the correct region supplied by the learning data supply unit corresponding to the learning image, wherein the figure corresponding to the estimated region and the figure corresponding to the correct region are both convex hull polygons, and the error calculation unit includes: a virtual pixel generation unit that sets a number of virtual pixels common to the image and the learning image, respectively; and for each of the virtual pixels, the sum and product of the inclusion determination value for the figure corresponding to the estimated region and the inclusion determination value for the figure corresponding to the correct region, respectively, and the value obtained by subtracting the product from the sum The system comprises an error function value calculation unit that calculates an error by dividing the sum of the intersection values ​​for all virtual pixels by the sum of the union values ​​for all virtual pixels, with the value of the product being used as the union value, and the inclusion determination value for the figure corresponding to the estimated region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and is 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure, and the function for calculating the inclusion determination value for the figure corresponding to the estimated region is continuous and differentiable over the entire region of the image, and the inclusion determination value for the figure corresponding to the correct region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and is 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure,Furthermore, the function for determining the intensity judgment value for the figure corresponding to the correct answer region is continuous and differentiable throughout the entire region of the training image, and the intensity judgment value for the figure corresponding to the estimated region and the intensity judgment value for the figure corresponding to the correct answer region are, respectively, greater than or equal to 0 and less than or equal to 1 throughout the entire region of the image and the training image.

[0014] [2] In another aspect of the present invention, in the region detection device described in [1] above, the error calculation unit further calculates, for each of the virtual pixels, the following formula (1) (where e is Napier's number, k is a predetermined positive constant, d px d is the distance from a reference line parallel to the side and passing through a predetermined point inside the figure to the virtual pixel, center The system comprises: an inclusion determination unit that determines an inclusion determination value for the edge of the virtual pixel by (where is the distance from the reference line for that edge to that edge); and an inclusion determination integration unit that determines the inclusion determination value for the virtual pixel for the figure by multiplying the inclusion determination values ​​for all edges of the figure determined by the inclusion determination unit.

[0015] [3] Another aspect of the present invention is a region detection device comprising: a region estimation unit that internally includes a machine learning model and estimates regions having specific features in an image by inputting an image passed from the outside into the machine learning model; a learning data supply unit that supplies pairs of learning images and correct regions corresponding to the learning images for training the machine learning model provided by the region estimation unit; and an error calculation unit that calculates the error between an estimated region which is the result of the region estimation unit's estimation based on the learning images supplied by the learning data supply unit and the correct region supplied by the learning data supply unit corresponding to the learning image, wherein the figure corresponding to the estimated region and the figure corresponding to the correct region are both convex hull polygons, and the error calculation unit includes a virtual pixel generation unit that sets a number of virtual pixels common to the image and the learning image, respectively, and for each of the virtual pixels, calculates the sum and product of the inclusion determination value for the figure corresponding to the estimated region and the inclusion determination value for the figure corresponding to the correct region, respectively, and calculates the value obtained by subtracting the product from the sum as a union value (unio The system comprises an error function value calculation unit that calculates an error by dividing the sum of the intersection values ​​for all virtual pixels by the sum of the union values ​​for all virtual pixels, where n) is the value of the product, and the value obtained by dividing the sum of the intersection values ​​for all virtual pixels by the sum of the union values ​​for all virtual pixels, wherein the inclusion determination value for the figure corresponding to the estimation region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure, and the function for calculating the inclusion determination value for the figure corresponding to the estimation region is continuous and differentiable over the entire region of the image, and the inclusion determination value for the figure corresponding to the correct region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure,A function for obtaining an inclusion determination value for a figure corresponding to the correct region is continuous and differentiable in all regions within the learning image, and the inclusion determination value for the figure corresponding to the estimated region and the inclusion determination value for the figure corresponding to the correct region are each 0 or more and 1 or less in all regions of the image and the learning image. A program for causing a computer to function as a region detection device.

Effect of the Invention

[0016] According to the present invention, a region detection device for detecting a region having a specific feature using a machine learning model such as a neural network can detect a region other than a rectangle (a region corresponding to a figure of a convex hull polygon). In addition, since the resolution of virtual pixels can be arbitrarily set, it is possible to process a detailed image without depending on the resolution of real pixels.

Brief Description of the Drawings

[0017] [Figure 1] It is a block diagram showing a schematic functional configuration of a region detection device according to an embodiment of the present invention. [Figure 2] It is a block diagram showing a more detailed functional configuration inside an error calculation unit according to the same embodiment. [Figure 3] It is a schematic diagram showing an arrangement example of real pixels and virtual pixels in an image in the same embodiment. [Figure 4] It is one of the schematic diagrams for explaining the processing of the inclusion determination unit according to the same embodiment, and shows a state in which one of the figures to be the target of error calculation exists on the xy plane. [Figure 5] It is another schematic diagram for explaining the processing of the inclusion determination unit according to the same embodiment, and shows a state in which the figure plane is rotated so that one side is horizontal. [Figure 6] It is a schematic diagram for explaining the processing when the error function value calculation unit according to the same embodiment obtains the error between two figures. [Figure 7] It is a block diagram showing an example of an internal configuration for realizing the region detection device according to the same embodiment. [Figure 8] This is a schematic diagram illustrating how to determine the inclusion determination value using a modified example of the same embodiment. [Figure 9] This is a schematic diagram illustrating an application example of the region detection device of the same embodiment and an example of an image used in an experiment to verify it. [Modes for carrying out the invention]

[0018] Next, one embodiment of the present invention will be described with reference to the drawings. In this embodiment, the region detection device 1 automatically detects regions having specific features within an input image by using machine learning techniques. In this embodiment, when training the machine learning model, a method for calculating the error (also called loss or error, etc.) specific to this embodiment is used. The main features of this error calculation are as follows:

[0019] The first characteristic is that the error calculation unit 105, which will be explained later, defines virtual pixels for convenience in calculations only, regardless of the actual pixels that make up the input image, and performs calculations based on these virtual pixels. Specifically, the error calculation unit 105 calculates the number of virtual pixels contained within a figure representing a region in the image. Note that the figure here is limited to a convex hull figure. That is, the interior angle of any vertex constituting the figure (polygon) is limited to less than 180 degrees. Note that the error calculation unit 105 may find an approximate value of the number of virtual pixels contained within the figure. By performing calculations based on virtual pixels in this way, the error calculation unit 105 can calculate the error with high accuracy even when the resolution of the input image is low.

[0020] The second feature is that the error calculation unit 105 determines whether each virtual pixel exists inside or outside the symmetrical figure for each edge that makes up the figure. Then, the error calculation unit 105 integrates the determination results (inside or outside) for each edge to determine whether each virtual pixel exists inside or outside the symmetrical figure. Using this procedure, the error calculation unit 105 finds the number of virtual pixels that exist inside a symmetrical figure of any shape (however, a convex hull figure). The specific calculation method will be explained later.

[0021] Figure 1 is a block diagram illustrating the schematic functional configuration of the region detection device according to this embodiment. As shown in the figure, the region detection device 1 includes an image input unit 101, a region estimation unit 102, a result output unit 103, a training data supply unit 104, and an error calculation unit 105. Each of these functional units can be implemented, for example, by a computer and a program. Each functional unit also has storage means as needed. The storage means is, for example, a variable in the program or memory allocated by the execution of the program. Alternatively, non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may be used as needed. Furthermore, at least some of the functions of each functional unit may be implemented as a dedicated electronic circuit instead of a program.

[0022] The region detection device 1 operates in either a learning mode or an estimation execution mode. In learning mode, the region detection device 1 optimizes the parameters of the machine learning model of the region estimation unit 102 by processing training data. In estimation execution mode, the region estimation unit 102 uses the trained parameters to estimate a specific region in an unknown input image. The functions of each part are described below.

[0023] The image input unit 101 acquires an image input from an external source and passes it to the region estimation unit 102. The image that the image input unit 101 passes to the region estimation unit 102 is the image that will be used for estimation in the estimation execution mode.

[0024] The region estimation unit 102 internally incorporates a machine learning model and estimates regions within an image that have specific features by inputting an image received from an external source into the machine learning model. Specifically, the region estimation unit 102 internally incorporates a neural network as the machine learning model. The region estimation unit 102 outputs a figure corresponding to the estimated region. In this embodiment, the figure is a convex hull polygon. That is, the figure is a polygon in which all interior angles are less than 180 degrees. More specifically, the region estimation unit 102 may output the coordinate values ​​of each vertex of the estimated convex hull polygon. Alternatively, the region estimation unit 102 may output equivalent information regarding the convex hull polygon.

[0025] When the region estimation unit 102 is operating in learning mode, it passes information about the shape of the estimated region to the error calculation unit 105. The error calculation unit 105 then calculates the error between the shape of the estimated region and the shape of the correct region, etc. When the region estimation unit 102 is operating in estimation execution mode, it passes information about the shape of the estimated region, which is the result of estimation based on an unknown image, to the result output unit 103.

[0026] The result output unit 103 outputs to the outside the estimated region information (graphic information) output by the region estimation unit 102 as the estimation result in the estimation execution mode. This estimated region information can be used for various purposes as information about a region with specific characteristics.

[0027] The training data supply unit 104 supplies training data for the neural network (machine learning model) of the region estimation unit 102 when the region detection device 1 is operating in training mode. The training data is information of a set of pairs of training images and shapes (convex hull polygons in this embodiment) that represent the correct region corresponding to the training image. The training data supply unit 104 passes the training images included in the pairs to the region estimation unit 102. The training data supply unit 104 also passes the information of the shapes of the correct region included in the pairs to the error calculation unit 105. This enables the error calculation unit 105 to calculate the error between the shape of the region (estimated region) output by the region estimation unit 102 based on the training images and the shape of the correct region supplied by the training data supply unit 104. When training the neural network, the training data supply unit 104 sequentially supplies each pair for training.

[0028] The error calculation unit 105 calculates the error between the region shape output by the region estimation unit 102 as an estimation result and the correct region shape supplied by the learning data supply unit 104 when the region detection device 1 is operating in learning mode. In this embodiment, both the shape corresponding to the estimated region and the shape corresponding to the correct region are convex hull polygons. The error calculated by the error calculation unit 105 is used to adjust (optimize) the internal parameters of the neural network of the region estimation unit 102 using backpropagation. The configuration and processing details of the error calculation unit 105 will be explained further later with reference to Figure 2, etc.

[0029] The error calculated by the error calculation unit 105 is as follows. The internal configuration of the error calculation unit 105, which will be explained later with reference to the block diagram in Figure 2, is an example of a procedure for specifically realizing the following error.

[0030] The error calculation unit 105 uses an inclusion determination value as basic data for calculating the error. The inclusion determination value is a numerical value that indicates whether or not a virtual pixel, described later, is contained within a shape (the shape of the estimation region or the shape of the correct answer region). However, in this embodiment, the inclusion determination value is not a binary value such as 0 or 1, but can take on a continuous value between 0 and 1.

[0031] The inclusion determination value for a figure corresponding to the estimated region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure. Furthermore, the function for determining the inclusion determination value for the figure corresponding to the estimated region is continuous and differentiable over the entire region of the image. In other words, in the neighborhood of an edge of a figure (as already explained, the figure is a polygon) (the neighborhood is the region within the predetermined distance from the edge), the inclusion determination value changes continuously (and abruptly) and smoothly from 0 (or approximately 0) to 1 (or approximately 1).

[0032] The same applies to the correct answer region. That is, the inclusion determination value for a figure corresponding to the correct answer region is 1 or approximately 1 if the virtual pixel is inside the figure and is more than a predetermined distance away from any side of the figure, and 0 or approximately 0 if the virtual pixel is outside the figure and is more than a predetermined distance away from any side of the figure, and the function for determining the inclusion determination value for the figure corresponding to the correct answer region is continuous and differentiable over the entire area of ​​the training image. In other words, in the neighborhood of the sides of the figure (which is a polygon) (the neighborhood is the area within the predetermined distance from the side), the inclusion determination value changes continuously (and abruptly) and smoothly from 0 (or approximately 0) to 1 (or approximately 1).

[0033] Furthermore, the intensification judgment value for the figure corresponding to the estimated region and the intensification judgment value for the figure corresponding to the correct answer region are both 0 or greater and 1 or less across the entire region of the image and the training image, respectively.

[0034] In this embodiment, in order to determine the inclusion determination value for a shape of a virtual pixel, the error calculation unit 105 determines the inclusion determination value for each side of the shape of the virtual pixel, and multiplies these inclusion determination values ​​to determine the inclusion determination value for that shape of the virtual pixel.

[0035] Figure 2 is a block diagram showing a more detailed functional configuration of the internal components of the error calculation unit 105 according to this embodiment. As shown in the figure, the error calculation unit 105 includes a virtual pixel generation unit 1051, an inclusion determination unit 1052, an inclusion determination integration unit 1053, and an error function value calculation unit 1054. The functions of each unit are as follows.

[0036] The virtual pixel generation unit 1051 sets (generates) a large number of virtual pixels that are common to both the image and the training image. Virtual pixels will be explained in more detail later.

[0037] The internal determination unit 1052 calculates, for each virtual pixel, the respective edges of the figure corresponding to the estimated region and the respective edges of the figure corresponding to the correct region using the formula (1) described later (where e is Napier's number, k is a predetermined positive constant, d px d is the distance from a reference line parallel to the side and passing through a predetermined point inside the figure to the virtual pixel, center The inclusion determination value for the edge of the virtual pixel is determined by the distance from the aforementioned reference line to the edge (where is the distance from the edge to the edge).

[0038] The Encompassing Determination Integration Unit 1053 integrates the encompassing determination values ​​for each side obtained by the Encompassing Determination Unit 1052 for a single virtual pixel to determine the encompassing determination value for the shape (polygon) of that virtual pixel. Specifically, the Encompassing Determination Integration Unit 1053 determines the encompassing determination value for the shape of that virtual pixel by multiplying all the encompassing determination values ​​for all sides of the shape obtained by the Encompassing Determination Unit 1052.

[0039] The error function value calculation unit 1054 calculates the error between the figure in the estimated region and the figure in the correct region based on the inclusion determination value. Specifically, the error function value calculation unit 1054 calculates the error based on the inclusion determination value, which is a value that indicates whether or not a virtual pixel is contained within each figure. More specifically, for each virtual pixel, the error function value calculation unit 1054 calculates the sum and product of the inclusion determination value for the figure corresponding to the estimated region and the inclusion determination value for the figure corresponding to the correct region, respectively. The value obtained by subtracting the product from the sum is taken as the union value, and the value of the product is taken as the intersection value. The error is obtained by dividing the sum of the intersection values ​​for all virtual pixels by the sum of the union values ​​for all virtual pixels.

[0040] Next, we will explain the processing flow by the error calculation unit 105 with an example.

[0041] The virtual pixel generation unit 1051 generates virtual pixels for the image subject to error calculation. Virtual pixels are pixels virtually placed on the surface of the image at predetermined intervals. When expressed in xy Cartesian coordinates, for example, if the y coordinate ranges from 0.0 to 1.5, the x coordinate ranges from 0.0 to 1.0, and the interval between the x and y coordinates of the virtual pixels is 0.5, then the coordinate values ​​of the virtual pixels are the following 12. (x,y= (0.0,0.0),(0.0,0.5),(0.0,1.0),(0.0,1.5), (0.5,0.0),(0.5,0.5),(0.5,1.0),(0.5,1.5), (1.0,0.0),(1.0,0.5),(1.0,1.0),(1.0,1.5),

[0042] The coordinate values ​​of virtual pixels may differ from the coordinate values ​​of actual pixels in the image. Furthermore, the spacing between virtual pixels (0.5 in the example above) can be determined appropriately, independently of the actual spacing between pixels. The x and y coordinate values ​​of virtual pixels are represented as real numbers and do not need to be integers. Because of this high degree of freedom regarding the coordinate values ​​of virtual pixels, it is suitable for handling small shapes, such as those where the size in the x and y directions is 1 pixel or less. Note that narrowing the spacing between virtual pixels (i.e., increasing the number of virtual pixels per given length) increases the computational load required for error calculation, but improves calculation accuracy.

[0043] Figure 3 is a schematic diagram showing an example of the arrangement of real and virtual pixels in an image. The figure shows only a portion of the image. In the figure, real pixels 1901 are represented by squares, and virtual pixels 1902 are represented by black circles. In this example, both real pixels 1901 and virtual pixels 1902 are arranged in a square pattern. In this example, 16 virtual pixels 1902 are arranged to correspond to 1 real pixel 1901. In other words, the density of virtual pixels 1902 is four times the density of real pixels 1901 in both the vertical and horizontal directions. The arrangement pattern shown in Figure 3 is just one example. Virtual pixels may not be arranged in a square pattern, but for example, a delta pattern. Also, virtual pixels may be arranged in an irregular pattern. The relationship between the density of real pixels and the density of virtual pixels is also arbitrary. In any case, it is desirable that the density of virtual pixels does not vary greatly depending on their position on the screen and is as uniform as possible.

[0044] The inclusion determination unit 1052 performs an inclusion determination for each of the virtual pixels set above, edge by edge. In other words, the inclusion determination unit 1052 determines which side each virtual pixel is on of each edge. As a processing procedure, the inclusion determination unit 1052 may rotate the image so that the edge of interest is, for example, horizontal, and then determine whether each virtual pixel is above or below that edge in that state.

[0045] Figure 4 is one of the schematic diagrams illustrating the processing performed by the inclusion determination unit 1052. This figure shows a state in which one of the figures to be used for error calculation lies on the xy-plane. In this figure, 2001 is one of the figures to be used for error calculation. Figure 2001 is either a figure representing the region estimated by the region estimation unit 102, or a figure representing the correct region supplied by the learning data supply unit 104. In this example, figure 2001 is a quadrilateral with sides 2011, 2012, 2013, and 2014. The figure to be used for error calculation is not limited to a quadrilateral, but can be any polygon. However, the figure to be used is a convex hull polygon.

[0046] FIG. 5 is another schematic diagram for explaining the processing by the inclusion determination unit 1052. The state shown in FIG. 5 is the state resulting from rotating the plane (xy plane) including the figure 2001 shown in FIG. 4. The inclusion determination unit 1052 rotates the figure so that the currently focused side (in this example, side 2011) among the sides of the figure 2001 becomes horizontal. The dashed line 2021 shown in the figure is a reference horizontal line. That is, the dashed line 2021 is a line parallel to the side 2011. The dashed line 2021 is set as a line passing through the center point of the figure 2001. The center point may be determined by the average value of the x - coordinate and y - coordinate of the vertices constituting the figure 2001. That is, when looking at the positional relationship in the vertical direction of this figure, the dashed line 2021 is located inside the figure 2001 compared to the side 2011. Conversely, in the vertical position, the side of the side 2011 on the opposite side of the dashed line 2021 is outside the figure 2001. Here, the distance from the dashed line 2021 to the side 2011 is set to 1.5. Also, the distance from the dashed line 2021 to the virtual pixel 2022 is set to 2.2, and the distance from the dashed line 2021 to the virtual pixel 2023 is set to 0.6. At this time, the inclusion determination unit 1052 calculates the inclusion determination value according to the following formula (1).

[0047]

Equation

[0048] In Equation (1), d px is the distance from the reference line (the dashed line 2021 in the example) to the virtual pixel to be determined. Also, d center is the distance from the reference line to the focused side (in this example, side 2011). Also, k is a parameter taking a predetermined positive value, for example, k = 10. Here, e is the Napier number. When d px is smaller than d center , the inclusion determination value obtained by Equation (1) approaches a value of 1. When d px is larger than d center , the inclusion determination value obtained by Equation (1) approaches a value of 0. When d px is equal to d centerIf it is exactly equal to , the inclusion determination value obtained by equation (1) will be 0.5. In other words, when viewed from the vertical position in the example in Figure 5, if the virtual pixel is located below edge 2011, the inclusion determination value approaches 0, and if the virtual pixel is located above edge 2011, the inclusion determination value approaches 1. Note that in the vicinity of edge 2011, the inclusion determination value changes smoothly and steeply. The value of the parameter k above is a value that controls the degree of abruptness of the change in the inclusion determination value in the vicinity of edge 2011. However, the function for determining the inclusion determination value expressed by equation (1) is d px In the entire domain d px It is differentiable with respect to .

[0049] In the example shown in Figure 5, the inclusion determination value for virtual pixel 2022 with respect to edge 2011 is calculated by equation (1) and is approximately 0.0009 (close to 0). This indicates that virtual pixel 2022 is located above edge 2011. Similarly, the inclusion determination value for virtual pixel 2023 with respect to edge 2011 is calculated by equation (1) and is approximately 0.9999 (close to 1). This indicates that virtual pixel 2023 is located below edge 2011.

[0050] The inclusion determination integration unit 1053 integrates the inclusion determinations for each edge made by the inclusion determination unit 1052 to obtain an inclusion determination value for the entire figure. Specifically, the inclusion determination integration unit 1053 calculates the integrated inclusion determination value for each virtual pixel by multiplying all the inclusion determination values ​​calculated for each edge. As already explained, when the figure is rotated as shown in Figure 5, that is, when the edges are horizontal and the center point of the figure is located below the edges, the inclusion determination value of the virtual pixels below the edges is close to 1, and the inclusion determination value of the virtual pixels above the edges is close to 0. Also, the inclusion determination value changes rapidly between 0 and 1 in the vicinity of the edges. In other words, when all the inclusion determination values ​​for each edge are multiplied together for a given virtual pixel, the integrated inclusion determination value of the virtual pixels inside the figure (for example, a rectangle) is close to 1, and the integrated inclusion determination value of the virtual pixels outside the figure is close to 0.

[0051] The enumeration determination integration unit 1053 calculates an integrated enumeration determination value for all virtual pixels. The sum of the integrated enumeration determination values ​​for all virtual pixels in a single image is an approximation of the number of virtual pixels contained within the shape, and this value is proportional to an approximation of the area of ​​the shape. In other words, the sum of the enumeration determination values ​​obtained by the enumeration determination integration unit 1053 is a value that represents the area of ​​the shape, and moreover, it is a differentiable value that can be obtained only by the calculation of equation (1) and the addition of those values.

[0052] The error function value calculation unit 1054 uses the calculation result of the internalization determination integration unit 1053 to calculate the error between the two figures input to the error calculation unit 105. Specifically, the error function value calculation unit 1054 performs the calculation described below with reference to Figure 6.

[0053] Figure 6 is a schematic diagram illustrating the process by which the error function value calculation unit 1054 calculates the error between two figures. Figure 6(A) shows the region of the first figure, figure 2002. Figure 6(B) shows the region of the second figure, figure 2003. The more these two figures coincide, the smaller the error between them; the more they differ, the larger the error between them. To find such an error, the error function value calculation unit 1054 performs a calculation that can serve as an alternative to IoU (Intersection over Union), as explained below. Figure 6(C) shows a situation where there is only a partial overlap between the regions of figure 2002 and figure 2003. The calculation performed by the error function value calculation unit 1054 is based on the size of the region of the union of the two figures and the size of the region of the intersection of the two figures.

[0054] In IoU, the intersection is the common part (the intersection portion) of two shapes. In this embodiment, to obtain a value corresponding to the intersection in IoU, the error function value calculation unit 1054 calculates the product of the integrated intensity determination values ​​of the two shapes for each virtual pixel. For a single virtual pixel, if the intensity determination value of at least one of the two shapes is close to 0, the product of those two intensity determination values ​​will be close to 0. If the intensity determination values ​​of both shapes are close to 1, the product of those two intensity determination values ​​will be close to 1. Moreover, the product of the two intensity determination values ​​is differentiable. The sum of the above products for all virtual pixels is the value corresponding to the intersection in this embodiment. In IoU, the union is the part of the union of two shapes. In this embodiment, to obtain a value corresponding to the intersection in IoU, the error function value calculation unit 1054 calculates the sum of the integrated intensity determination values ​​of the two shapes for each virtual pixel, and then subtracts the value of the product from that sum. For the common part of the two shapes, the sum of their intensity determination values ​​is close to 2. Therefore, by subtracting the product of the two intensity determination values ​​(close to 1) from that sum, the resulting value is close to 1. For the exclusive sum portion of the two shapes, the sum of their intensity determination values ​​is close to 1. Since the product of the two intensity determination values ​​for the exclusive sum portion is close to 0, the resulting value is close to 1. For the other parts (parts that do not belong to either of the two shapes), both the sum and the product of their intensity determination values ​​are close to 0. Therefore, the result of subtracting the product from the sum is close to 0. The sum of the values ​​obtained by subtracting the product from the above sum for all virtual pixels is the value corresponding to the union in this embodiment. This value is also differentiable.

[0055] In other words, the error function value calculation unit 1054 calculates the error between the two figures using the following equation (2).

[0056] sum(intersection) / sum(union) ···(2)

[0057] In equation (2), union is the value corresponding to the union of each virtual pixel obtained by the error function value calculation unit 1054 using the method described above. Also, intersection in equation (2) is the value corresponding to the intersection of each virtual pixel obtained by the error function value calculation unit 1054 using the method described above. Furthermore, sum(·) represents the operation (calculation) of taking the sum of the values ​​for all virtual pixels in the image.

[0058] As explained above, both the union and intersection in equation (2) are differentiable. Furthermore, the sum of their values ​​over all virtual pixels is also differentiable. Moreover, the value expressed by equation (2) (the error calculated by the error function value calculation unit 1054) is also differentiable.

[0059] As explained above, the error calculation unit 105 can find an appropriate error by using the function of equation (1) with respect to the two figures (both convex hull figures (convex hull polygons)). Furthermore, since this error is differentiable, the neural network internally contained in the region estimation unit 102 can be updated by backpropagation. In other words, by using a sufficient amount of training data and performing backpropagation a sufficient number of times, the neural network (machine learning model) contained in the region estimation unit 102 is optimized.

[0060] As explained above, in the region detection device 1 of this embodiment, the figure representing the region to be detected may be any convex hull polygon with any orientation (not limited to rectangles, etc.). Furthermore, as is clear from the error calculation process described above, the number of vertices of the region figure (convex hull polygon) output by the region estimation unit 102 as an estimation result may be different from the number of vertices of the correct region figure (convex hull polygon) supplied by the learning data supply unit 104. In other words, this embodiment solves the aforementioned problems, and the error between any two convex hull polygons can be obtained as the value of a differentiable function. This makes it possible to detect regions of shapes other than rectangles using a machine learning model. This removes constraints in the problem of detecting regions with specific features in an image, and the region detection device 1 can be used to solve more general problems.

[0061] Figure 7 is a block diagram showing an example of the internal configuration of the region detection device 1 described as an embodiment. The region detection device 1 can be implemented using a computer. As shown in the figure, the computer is composed of a central processing unit 901, RAM 902, input / output ports 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in programs read from RAM 902, etc. The central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations according to each instruction. RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using that address. RAM stands for "Random Access Memory". Input / output ports 903 are ports for the central processing unit 901 to exchange data with external input / output devices, etc. Input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via input / output port 903. Bus 906 is a common communication channel used within the computer. For example, the central processing unit 901 reads and writes data to RAM 902 via bus 906. Also, for example, the central processing unit 901 accesses input / output ports via bus 906.

[0062] At least some of the functions of the area detection device 1 in the above-described embodiment can be realized by a computer and a program. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, DVD-ROMs, USB memory, and storage devices such as hard disks built into a computer system. In other words, "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Moreover, "computer-readable recording medium" may also include those that temporarily and dynamically hold programs, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in that case. Furthermore, the above-mentioned program may be for realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0063] Although embodiments have been described above, the present invention can also be implemented in the following modified forms.

[0064] [Differentiation] In the embodiment, a procedure was described in which an inclusion determination value for one virtual pixel and one edge is calculated using equation (1), and the inclusion determination value for the shape (convex hull polygon) of that virtual pixel is obtained by multiplying the inclusion determination values ​​for all edges. As a modified example, a method for obtaining an inclusion determination value for the shape of one virtual pixel is described here.

[0065] Figure 8 is a schematic diagram illustrating how to determine the inclusion determination value using this modified example. In the example shown in the figure, the figure to be determined is a convex pentagon. This convex pentagon has sides 2031, 2032, 2033, 2034, and 2035. In this figure, the region is divided into three areas using boundary lines (dashed lines). Region R1 is the area inside the figure (pentagon) and is at least a predetermined distance from any of its sides. Region R3 is the area outside the figure (pentagon) and is at least a predetermined distance from any of its sides. Region R2 is the area that is neither Region R1 nor R3. That is, Region R2 is the area near any of the sides 2031, 2032, 2033, 2034, and 2035. In other words, Region R2 is the area that is at least a predetermined distance from any of these sides. The width of region R2 in the direction perpendicular to each edge is assumed to be sufficiently small. In this modification as well, the intensity determination value is a value between 0 and 1. In this modification, the intensity determination value in region R1 is 1 or approximately 1 (i.e., between 1-ε and 1), where ε is a positive constant sufficiently small compared to 1. Also, the intensity determination value in region R3 is 0 or approximately 0 (i.e., between 0 and ε). Furthermore, the function for calculating the intensity determination value for a virtual pixel in this image is assumed to be continuous and differentiable over the entire region of the image.

[0066] The modified error calculation unit 105 calculates the error between the two figures based on such an inclusion determination value, using the calculation procedure already described in the embodiment.

[0067] The inclusion determination value explained with reference to Figure 8 corresponds to a more general form. The method for determining the inclusion determination value explained in the embodiment is one special case within this general form.

[0068] As described above, according to the embodiments (including modified versions), it is possible to use a machine learning model to detect areas other than rectangles. Furthermore, according to the embodiments (including modified versions), it is possible to generate (create) virtual pixels at any resolution to detect areas in detail.

[0069] Although embodiments of this invention (including modified examples) have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0070] [Applications and Effects] The application examples and effects of the region detection device 1 of the above embodiment will be described. As an application example, the region detection device 1 is used to detect the region of characters contained in an image.

[0071] Figure 9 is a schematic diagram showing an example of an image processed in this application. Figure 9(A) shows the image that will be input to the region detection device 1. This image contains the letters "abc" as an example. The region detection device 1 is pre-trained using training data to detect character regions. The trained region detection device 1 is operated in estimation execution mode and the image in Figure 9(A) is input. Figure 9(B) is an example of the detection result. The region detection device 1 detects regions containing the letters "a", "b", and "c", respectively. Each of the detected regions is a rectangle (not a square). Based on the detection result by the region detection device 1, the detected regions are separated into independent images, and the areas other than the detected regions are masked with black (pixel value 0). Figure 9(C) shows three images showing the result. In Figure 9(C), the hatched regions are the regions masked with black (pixel value 0). This mask is to prevent unnecessary noise information from being included in the subsequent character recognition processing. Then, character recognition was performed on these three images using a character recognition device (not covered by the present invention). As a result, the characters "a," "b," and "c" were correctly recognized. In other words, in this application example, as a preprocessing step for character recognition, a region with specific features (a region containing characters) was detected. In this demonstration experiment, the character recognition accuracy (F-value) improved from 70% when no region detection was performed to 73% when region detection preprocessing was performed. In other words, it was confirmed that the effects of the above embodiment were obtained. [Industrial applicability]

[0072] The present invention can be used, for example, to detect specific features present in an image. More specifically, the present invention can be used to detect objects in an image, to detect areas containing text in an image, or to detect other specific areas. However, the scope of use of the present invention is not limited to those exemplified herein. [Explanation of symbols]

[0073] 1. Area detection device 101 Image Input Section 102 Area estimation part 103 Result Output Section 104 Training Data Supply Unit 105 Error calculation section 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus 1051 Virtual Pixel Generation Unit 1052 Inclusion determination unit 1053 Internalization Determination Integration Unit 1054 Error Function Value Calculation Unit

Claims

1. A region estimation unit that internally incorporates a machine learning model and estimates regions within an image that have specific features by inputting an image provided from an external source into the machine learning model, A training data supply unit that supplies pairs of training images and corresponding ground truth regions for training the machine learning model provided by the region estimation unit, An error calculation unit that calculates the error between the estimated region, which is the result of the region estimation unit's estimation based on the training image supplied by the training data supply unit, and the correct region supplied by the training data supply unit in correspondence with the training image. A region detection device comprising, The figure corresponding to the estimation region and the figure corresponding to the correct answer region are both convex hull polygons, The error calculation unit, A virtual pixel generation unit sets a number of virtual pixels common to both the aforementioned image and the aforementioned training image, An error function value calculation unit calculates, for each of the virtual pixels, the sum and product of the inclusion determination value for the figure corresponding to the estimation region and the inclusion determination value for the figure corresponding to the correct answer region, respectively, the value obtained by subtracting the product from the sum as the union value, the value of the product as the intersection value, and the value obtained by dividing the sum of the intersection values ​​for all the virtual pixels by the sum of the union values ​​for all the virtual pixels as the error. Equipped with, The inclusion determination value for the figure corresponding to the estimated region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and is 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure, and the function for determining the inclusion determination value for the figure corresponding to the estimated region is continuous and differentiable over the entire region of the image. The inclusion determination value for a figure corresponding to the correct answer region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and is 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure, and the function for determining the inclusion determination value for a figure corresponding to the correct answer region is continuous and differentiable over the entire region of the training image. The inclusion determination value for the figure corresponding to the estimation region and the inclusion determination value for the figure corresponding to the correct answer region are, respectively, 0 or greater and 1 or less in all regions of the image and the training image. Area detection device.

2. The error calculation unit, moreover, For each of the virtual pixels, for each edge of the figure corresponding to the estimated region and each edge of the figure corresponding to the correct region, the following equation (1) [Math 1] (where e is Napier's number, k is a predetermined positive constant, d ｐｘ d is the distance from a reference line parallel to the side and passing through a predetermined point inside the figure to the virtual pixel, ｃｅｎｔｅｒ (This is the distance from the aforementioned reference line to the side in question.) An inclusion determination unit that determines an inclusion determination value for the edge of the virtual pixel, An integration unit for determining the inclusion determination value of a virtual pixel for that figure is obtained by multiplying the inclusion determination values ​​of the virtual pixels for all sides of the figure obtained by the inclusion determination unit, The region detection device according to claim 1, comprising:

3. A region estimation unit that internally incorporates a machine learning model and estimates regions within an image that have specific features by inputting an image provided from an external source into the machine learning model, A training data supply unit that supplies pairs of training images and corresponding ground truth regions for training the machine learning model provided by the region estimation unit, An error calculation unit that calculates the error between the estimated region, which is the result of the region estimation unit's estimation based on the training image supplied by the training data supply unit, and the correct region supplied by the training data supply unit in correspondence with the training image. A region detection device comprising, The figure corresponding to the estimation region and the figure corresponding to the correct answer region are both convex hull polygons, The error calculation unit, A virtual pixel generation unit sets a number of virtual pixels common to both the aforementioned image and the aforementioned training image, An error function value calculation unit calculates, for each of the virtual pixels, the sum and product of the inclusion determination value for the figure corresponding to the estimation region and the inclusion determination value for the figure corresponding to the correct answer region, respectively, the value obtained by subtracting the product from the sum as the union value, the value of the product as the intersection value, and the value obtained by dividing the sum of the intersection values ​​for all the virtual pixels by the sum of the union values ​​for all the virtual pixels as the error. Equipped with, The inclusion determination value for the figure corresponding to the estimated region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and is 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure, and the function for determining the inclusion determination value for the figure corresponding to the estimated region is continuous and differentiable over the entire region of the image. The inclusion determination value for a figure corresponding to the correct answer region is 1 or approximately 1 if the virtual pixel is inside the figure and is at least a predetermined distance from any side of the figure, and is 0 or approximately 0 if the virtual pixel is outside the figure and is at least a predetermined distance from any side of the figure, and the function for determining the inclusion determination value for a figure corresponding to the correct answer region is continuous and differentiable over the entire region of the training image. The inclusion determination value for the figure corresponding to the estimation region and the inclusion determination value for the figure corresponding to the correct answer region are, respectively, 0 or greater and 1 or less in all regions of the image and the training image. A program to make a computer function as a region detection device.

Images