Learning device, object detection device, learning method, object detection method

Abstract

To provide a technology that improves learning accuracy of a neural network model that detects an object from an image.SOLUTION: A learning device acquires a degree of similarity between a correct answer region indicating a region of an object in an image and each of a plurality of anchor boxes preset in the image, and selects an anchor box with a degree of similarity equal to or greater than a predetermined threshold among the plurality of anchor boxes for the correct answer region. The learning device learns, on the basis of the correct answer region and the selected anchor box, a neural network model for object detection. The learning device changes, when anchor boxes of the upper limit number are selected for the correct answer region, on the basis of the degree of similarity obtained for the anchor boxes selected for the correct answer region, the upper limit number of the anchor boxes for the correct answer region.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to learning techniques.

Background Art

[0002] As a method for detecting an object from an image using a neural network model, a method using an anchor box, such as the object detection method described in Non-Patent Document 1, is often used. In the learning of the object detection method using an anchor box, an anchor box having pre-defined coordinates, width, and height is assigned to a ground truth label having the coordinates, width, and height of the region of the object to be detected in the image, and the coordinates of the anchor box assigned to the ground truth label are moved, or the width and height are scaled, etc., and the parameters of the neural network model are learned so that the same value as the ground truth label is output. The assignment is performed by calculating the IoU (Intersection over Union) between the ground truth label and the anchor box, and using that value as the similarity, and is done from the pair with the highest value.

[0003] The ground truth label can take arbitrary coordinates in the image. On the other hand, since the coordinates of the anchor box are defined discretely, depending on the positional relationship between the ground truth label and the anchor box, there is a problem of position dependence where learning is not stable for pairs with a large distance.

[0004] For such a problem, for example, in Non-Patent Document 2, a technique for solving this problem by assigning a plurality of anchor boxes to one ground truth label is disclosed.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] In the method disclosed in Non-Patent Document 2, multiple anchor boxes are assigned, but the maximum number of assignments is constant. Therefore, a problem may arise where anchor boxes that should be assigned are not assigned because the number of assigned anchor boxes exceeds the maximum number. As a result, the neural network training fails, and the detection confidence decreases. The present invention provides a technique for improving the training accuracy of a neural network model that detects objects from images. [Means for solving the problem]

[0007] One aspect of the present invention is a ground truth region that indicates the area of ​​an object in an image, and each of a plurality of anchor boxes that are pre-set in the image, Based on IoU (Intersection over Union) A means of obtaining similarity, A selection means for selecting, from among the plurality of anchor boxes, an anchor box whose similarity is equal to or greater than a predetermined threshold for the correct answer region, A learning means that trains a neural network model for detecting the aforementioned object based on the ground truth region and the anchor boxes selected by the selection means. Equipped with, If the selection means selects an upper limit number of anchor boxes for the correct answer region, it changes the upper limit number of anchor boxes for the correct answer region based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region. It is characterized by the following: [Effects of the Invention]

[0008] According to the configuration of the present invention, the learning accuracy of a neural network model that detects objects from images can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing an example configuration of the learning device 100. [Figure 2] A flowchart of the process by which the assignment processing unit 121 assigns anchor boxes to the correct labels. [Figure 3] A flowchart of the processing performed by the arithmetic processing unit 120. [Figure 4] (a) is a diagram showing an example of a correct label, (b) is a diagram showing an example of an anchor box, and (c) is a diagram showing an example of the similarity between the correct label and each anchor box. [Figure 5] A block diagram showing an example configuration of the object detection device 500. [Figure 6] (a) is a figure showing an example of a detection frame, (b) is a figure showing an example of the confidence level of the detection frame, and (c) is a figure showing an example of the similarity with detection frame 612-1. [Figure 7] A block diagram showing an example of a computer device hardware configuration. [Figure 8](a) is a diagram showing an example of a human face in an image of a usage scenario, (b) is a diagram showing an example of the center position and size values ​​of a human face, (c) is a diagram showing an example of the size of a face in each image region, and (d) is a diagram showing an example of the reference values ​​for the size of a face and the upper limit of anchor boxes. [Figure 9] A flowchart of the process by which the assignment processing unit 121 assigns anchor boxes to the correct labels using the second similarity metric. [Figure 10] (a) is a diagram showing an example of an anchor box, and (b) is a diagram showing an example of the first and second similarity between the correct label and each anchor box. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] [First Embodiment] This embodiment describes a learning device that trains a neural network model for detecting a human face from an image containing the face. An example of the configuration of the learning device 100 according to this embodiment will be explained using the block diagram in Figure 1.

[0012] The information storage unit 110 is a memory device such as a HDD (Hard Disk Drive), SSD (Solid State Drive), optical disk, RAM (Random Access Memory), flash memory, etc. The information storage unit 110 stores an OS (Operating System), computer programs and data for causing the arithmetic processing unit 120 to execute or control various processes described as being performed by the learning device 100. The data stored in the information storage unit 110 includes first information defining a correct label indicating the region of the face in an image (learning image) including a human face, and second information defining a plurality of anchor boxes preset for a learning image including a human face. Further, the data stored in the information storage unit 110 includes a similarity threshold value used in the processes described later, and an upper limit number of anchor boxes that can be assigned to one correct label.

[0013] The first information includes, for example, the coordinates, width, and height of the correct label (region of a human face) in the learning image. For example, as illustrated in FIG. 4(a), the correct label indicates the face region 411 of the person 412 in the learning image 410 used for learning the neural network model. In this case, the first information includes the coordinates of the region 411 in the learning image 410, the width and height of the region 411. Note that the correct label is not limited to a human face in the image, and may be any object that the neural network model detects. In that case, the first information is information defining the target object in the image.

[0014] The second information includes the coordinates, width, and height of each of the plurality of anchor boxes. For example, as illustrated in FIG. 4(b), a plurality of anchor boxes are arranged within a certain range 420 (for example, a range having the image size of the learning image 410). In FIG. 4(b), anchor boxes 421-1, 421-2, 421-3, and 421-4 are arranged. In this case, the second information includes the coordinates, width, and height of each of the anchor boxes 421-1, 421-2, 421-3, and 421-4. Note that the anchor box is not limited to being rectangular, and the arrangement pattern is not limited to the arrangement pattern illustrated in FIG. 4(b).

[0015] Further, the information storage unit 110 also stores parameters (such as weight values) of the neural network model. Further, information to be treated as known information in the following description is also stored in the information storage unit 110.

[0016] The neural network model used in this embodiment is configured to perform a conversion such that when an image including a human face is input, the coordinates, width, and height of the anchor box become the coordinates, width, and height of the region of the human face. Note that the neural network model may be any model as long as it uses an anchor box to detect an object from an image.

[0017] The arithmetic processing unit 120 is an electronic circuit such as a CPU (Central Processing Unit). The arithmetic processing unit 120 executes various processes using computer programs and data stored in the information storage unit 110. Thereby, the arithmetic processing unit 120 controls the operation of the entire learning device 100 and executes or controls various processes described as being performed by the learning device 100. Note that the arithmetic processing unit 120 may be an integrated circuit such as an FPGA (Field Programmable Gate Array). The arithmetic processing unit 120 includes an allocation processing unit 121 and a learning processing unit 122.

[0018] The allocation processing unit 121 allocates (selects) one or more anchor boxes to each correct label in the learning image. The allocation process of the anchor box to the correct label by the allocation processing unit 121 will be described according to the flowchart of FIG. 2.

[0019] In step S200, the allocation processing unit 121 reads out the first information of each correct label in the learning image, the second information defining each of the plurality of anchor boxes, the similarity threshold value, and the upper limit number from the information storage unit 110 into its internal memory.

[0020] In step S201, the assignment processing unit 121 calculates the similarity between each of the multiple anchor boxes (identified by the second information) for each of the ground truth labels (identified by the first information) in the training image. The assignment processing unit 121 calculates the IoU (Intersection over Union) between the ground truth label and the anchor box as the similarity between the ground truth label and the anchor box. However, the similarity between the ground truth label and the anchor box is not limited to the IoU. For example, another similarity metric such as GIoU (Generalized Intersection over Union) between the ground truth label and the anchor box may be used as the "similarity between the ground truth label and the anchor box".

[0021] Figure 4(c) shows an example of the similarity between the correct label 411 in Figure 4(a) and each of the anchor boxes 421-1 to 421-4 in Figure 4(b). As shown in Figure 4(c), the similarity for anchor box 421-1 is 0.8, the similarity for anchor boxes 421-2 and 421-3 is 0.5, and the similarity for anchor box 421-4 is 0.3.

[0022] Next, the assignment processing unit 121 performs the process in step S202 for each correct label in the training image. First, before starting the process in step S202, the assignment processing unit 121 creates an unassigned list in which all anchor boxes are registered for each correct label in the training image. Then, the assignment processing unit 121 selects one of the unselected correct labels in the training image as the selected correct label. Then, the assignment processing unit 121 identifies the anchor box with the highest similarity to the selected correct label from among the anchor boxes registered in the unassigned list for the selected correct label. Then, the assignment processing unit 121 registers the identified anchor box in the assigned list for the selected correct label and removes the identified anchor box from the unassigned list for the selected correct label. In this way, the assignment processing unit 121 assigns the anchor box with the highest similarity to the selected correct label from among the anchor boxes registered in the unassigned list for the selected correct label to the selected correct label.

[0023] For example, if the correct label 411 in Figure 4(a) is selected as the correct answer label, then, as shown in Figure 4(c), the anchor box with the highest similarity to the correct answer label 411 among anchor boxes 421-1 to 421-4 is anchor box 421-1. In this case, the assignment processing unit 121 assigns anchor box 421-1 to the correct answer label 411. In other words, the assignment processing unit 121 registers anchor box 421-1, which is registered in the unassigned list for the correct answer label 411, to the assigned list for the correct answer label 411, and removes anchor box 421-1 from the unassigned list.

[0024] The unassigned and assigned lists for each correct label may be stored in the internal memory of the arithmetic processing unit 120, or in the information storage unit 110, and the storage location is not limited to a specific location.

[0025] Next, the assignment processing unit 121 performs the processing in steps S203 to S206 for each correct label and each anchor box registered in the unassigned list for that correct label. In step S203, the assignment processing unit 121 selects one of the unselected correct labels from all the correct labels as the selected correct label, and selects one of the unselected anchor boxes registered in the unassigned list for the selected correct label as the selected anchor box. The order in which the anchor boxes are selected is not limited to a specific order, but for example, they may be selected in order from the anchor boxes with the highest similarity to the selected correct label. If there are two or more anchor boxes with the same similarity to the selected correct label, for example, the anchor box with the smaller index may be selected first.

[0026] For example, if the correct label 411 in Figure 4(a) is selected as the correct answer label, then anchor boxes 421-2 to 421-4 are registered in the unassigned list for the correct label 411. The similarity scores corresponding to anchor boxes 421-2 to 421-4 are "0.5", "0.5", and "0.3", respectively, as shown in Figure 4(c). Therefore, anchor boxes 421-2, 421-3, and 421-4 will be selected in that order.

[0027] The assignment processing unit 121 then determines whether the similarity S between the selected correct label and the selected anchor box is equal to or greater than the similarity threshold. If the similarity S is equal to or greater than the similarity threshold, the process proceeds to step S204; otherwise, the process proceeds to step S203.

[0028] For example, suppose the similarity threshold is "0.4". If the correct label 411 in Figure 4(a) is selected as the correct label and anchor box 421-2 is selected as the selected anchor box, the similarity of 0.5 corresponding to anchor box 421-2 is greater than or equal to the similarity threshold, so the process proceeds to step S204. Subsequently, if anchor box 421-3 is selected as the selected anchor box, the similarity of anchor box 421-3 is 0.5, which is greater than or equal to the similarity threshold, so the process proceeds to step S204. Subsequently, if anchor box 421-4 is selected as the selected anchor box, the similarity of anchor box 421-4 is 0.3, which is less than the similarity threshold, so the process proceeds to step S203.

[0029] In step S204, the assignment processing unit 121 determines whether the number of anchor boxes registered in the assigned list of correct selected labels is less than the upper limit. If the result of this determination is that the number of anchor boxes registered in the assigned list of correct selected labels is less than the upper limit, the process proceeds to step S206. On the other hand, if the number of anchor boxes registered in the assigned list of correct selected labels is not less than the upper limit, the process proceeds to step S205.

[0030] For example, let's assume the upper limit is "2". When the correct label 411 in Figure 4(a) is selected as the correct answer label and anchor box 421-2 is selected as the selected anchor box, the number of anchor boxes registered in the assigned list for the correct label 411 is "1" (anchor box 421-1), which is less than the upper limit of "2", so the process proceeds to step S206. Subsequently, when anchor box 421-3 is selected as the selected anchor box, the number of anchor boxes registered in the assigned list for the correct label 411 is "2" (anchor boxes 421-1 and 421-2), which is not less than the upper limit of "2", so the process proceeds to step S205. Subsequently, when anchor box 421-4 is selected as the selected anchor box, the number of anchor boxes registered in the assigned list for the correct label 411 is "2" (anchor boxes 421-1 and 421-2), which is not less than the upper limit of "2", so the process proceeds to step S205.

[0031] In step S205, the assignment processing unit 121 determines whether the difference Δ between the similarity S and the minimum similarity M is less than or equal to a specified value, using the minimum similarity M as the smallest similarity among the similarities registered in the assigned list of selected correct labels. The difference Δ is, for example, Δ = |SM|. If the difference Δ is less than or equal to the specified value, the process proceeds to step S206; if the difference Δ is greater than the specified value, the process proceeds to step S203.

[0032] In step S206, the assignment processing unit 121 registers the selection anchor box registered in the unassigned list for the correct selection label to the assigned list for the correct selection label, and removes the selection anchor box from the unassigned list. As a result, the assignment processing unit 121 assigns the selection anchor box to the correct selection label.

[0033] For example, suppose the correct label 411 in Figure 4(a) is selected as the correct answer label. At this time, the assigned list for the correct answer label 411 contains anchor boxes 421-1 and 421-2, and the minimum similarity M is 0.5. If anchor box 421-3 is selected as the selected anchor box, the similarity corresponding to anchor box 421-3 is 0.5. If the default value is 0, the difference Δ from the minimum similarity M = 0.5 is less than or equal to the default value, so the process proceeds to step S206. Then, anchor box 421-3, which is registered in the unassigned list for the correct answer label 411, is registered in the assigned list for the correct answer label 411, and anchor box 421-3 is removed from the unassigned list. This assigns anchor box 421-3 to the correct answer label 411.

[0034] After performing the processes in steps S203 to S206 for each correct label and each anchor box registered in the unassigned list for that correct label, the process proceeds to step S207.

[0035] In step S207, the assignment processing unit 121 assigns an anchor box registered in the unassigned list of selected correct labels to the background. For example, if the correct label 411 in Figure 4(a) is selected as the selected correct label, then as described above, anchor box 421-4 will remain in the unassigned list of correct labels, and so anchor box 421-4 is assigned to the background.

[0036] In step S208, the assignment processing unit 121 outputs to the learning processing unit 122 the correct labels for each training image, the anchor boxes registered in the assigned list for those correct labels, and the anchor boxes assigned to the background.

[0037] The learning processing unit 122 performs the learning process of the neural network model based on the correct labels and anchor boxes received from the assignment processing unit 121. For example, the learning processing unit 122 performs the learning process in accordance with the learning method described in Non-Patent Literature 1, such that anchor boxes assigned to the correct labels are converted to the correct labels, and anchor boxes not assigned to the correct labels become the background. In this learning process, the learning image is input to the neural network model to find the error between the converted anchor boxes and the correct labels, and the parameters of the neural network model are updated based on the found error.

[0038] Thus, in this embodiment, when the maximum number of anchor boxes is assigned to the correct label, one or more anchor boxes are assigned to the correct label from the unassigned anchor boxes based on the similarity obtained for the anchor boxes already assigned to the correct label and the similarity obtained for the anchor boxes not assigned to the correct label. This resolves the problem that "anchor boxes that should have been assigned are not assigned because the number of anchor boxes assigned to the correct label exceeds the maximum number." Consequently, the problem of the neural network model failing to learn and the detection confidence decreasing can be resolved.

[0039] <Example 1> The following process may be used as the step (S202) for determining the anchor boxes to be assigned to the correct labels in the training images. That is, the sum of similarity is calculated for each combination of anchor boxes to be assigned to the set of correct labels in the training images, and the combination of anchor boxes that maximizes the sum of similarity is determined using an algorithm such as the Hungarian algorithm or the primal dual method.

[0040] <Modification 2> The maximum number of anchor boxes that can be assigned to a single correct label stored in the information storage unit 110 may be changed for each region of the training image. For example, when adjusting the neural network model trained by the training device 100 according to the scene in which it is used, training is performed again using the training device 100 with images taken in the scene in which it is used. In this case, the smaller the size of the human face in the image of the scene in which it is used, the higher the maximum number of assignable anchor boxes may be. An example of changing the maximum number of assignable anchor boxes is illustrated using Figure 8. When the scene in which it is used is fixed, the central position (x,y) and size S of the human face in the image can be expressed as S = ax + by + c using variables a, b, and c, and the variables a, b, and c can be obtained by the least squares method if there are at least three combinations of the central position (x,y) and size S of the human face. The variables a, b, and c may also be obtained from at least two combinations of the central position and size of the human face by setting a=0 or b=0. The combination of the central position (x,y) and size S of a person's face can be specified from the image of the scene the user is using, or it can be extracted from a set of face detection results obtained from a set of images of the scene being used. As shown in Figure 8(a), if the sizes of people that may be included in the image are person 800, person 801, and person 802, the central position and size of each person's face can be represented as shown in Figure 8(b). Calculating the variables a, b, and c from this, we get a=0.1, b=0.2, and c=15. The size of a person's face can be represented in pixels, in real-world size, or as a standardized value based on the image size, etc. Using a=0.1, b=0.2, and c=15 to determine the size of a person's face in each image region, we can divide the image region into three areas as shown in Figure 8(c): Image region 803 where the size of a person's face S is 15 or greater but less than 25; Image region 804 where the size of a person's face S is 25 or greater but less than 35; and Image region 805 where the size of a person's face S is 35 or greater but less than 45. As shown in Figure 8(d), based on the size of the object and the reference value of the upper limit of the number of anchor boxes, we assign an upper limit of "4" to image region 803, an upper limit of "3" to image region 804, and an upper limit of "2" to image region 805.The user determines the reference values ​​for object size and the maximum number of anchor boxes based on the detection accuracy of the trained neural network model and the width and height of the anchor boxes being used. According to this, assigning more anchor boxes to smaller objects, which are more difficult to detect, improves the detection accuracy of the neural network model.

[0041] <Variation 3> When the maximum number of anchor boxes to be assigned is exceeded, the following process may be adopted for assigning the correct label (steps S205 and S206). The process of assigning anchor boxes to the correct label using this process will be explained according to the flowchart in Figure 9. The same points as the above process of assigning anchor boxes to the correct label will be omitted from the explanation.

[0042] If the minimum similarity and the similarity within the assigned anchor box match in step S900, the process proceeds to step S901. If they do not match, the process proceeds to step S203.

[0043] In step S901, a second similarity score is calculated for two anchor boxes with matching similarity scores, and the anchor box with the higher second similarity score is used as the correct label, updating the information of the anchor boxes registered in the assigned list.

[0044] The second similarity metric may be the center distance between the anchor box and the ground truth label, the size difference between the anchor box and the ground truth label, or the sum of the center distance and size difference between the anchor box and the ground truth label. The size difference is, for example, the squared error of the width and height. Furthermore, a smaller value for these three types indicates a higher degree of similarity. An example of the second similarity metric is illustrated using Figure 10. As shown in Figure 10(a), anchor box 1000 and anchor box 1001 are anchor boxes with equal width and height but different center coordinates. Also, anchor box 1002 and anchor box 1003 are anchor boxes with equal area and center coordinates but different width and height. As shown in Figure 10(b), when the similarity between the ground truth label 411 and each anchor box is calculated, the first similarity between anchor box 1000 and anchor box 1001 is equal. Therefore, if the second similarity metric is the central distance, or the sum of the differences between the central distance and the size, the second similarity of anchor box 1000 is "0" or "1", and the second similarity of anchor box 1001 is "0.5" or "1.5", so anchor box 1000, which has a smaller value and higher similarity, is selected. Furthermore, the first similarity of anchor box 1002 and anchor box 1003 are equal. Therefore, if the second similarity metric is the difference in size, or the sum of the differences between the central distance and the size, the second similarity of anchor box 1002 is "1.9", and the second similarity of anchor box 1003 is "1.3", so anchor box 1003 is selected. By using the second similarity metric for each anchor box to determine which anchor box to assign, it is possible to select an anchor box that is closer to the correct label in the anchor box situation shown in Figure 10(a). According to this, the detection accuracy of the neural network model can be improved by assigning only anchor boxes that have a higher similarity to the correct label.

[0045] [Second Embodiment] In this embodiment, the differences from the first embodiment will be explained, and unless otherwise specified below, it will be assumed to be the same as the first embodiment. In this embodiment, an object detection device will be described that obtains a final face detection result by integrating the face detection results output from a neural network model when an image containing a human face is input to a neural network model trained by the learning device 100 according to the first embodiment. If the neural network model is trained to detect objects other than human faces, the object detection device will obtain a final object detection result by integrating the object detection results output from the neural network model when an image containing the object is input to the neural network model.

[0046] First, an example of the configuration of the object detection device 500 according to this embodiment will be explained using the block diagram in Figure 5. The object detection device 500 may be the same device as the learning device 100 according to the first embodiment (one device being incorporated as a functional part of the other device), or it may be a separate device. In other words, as long as the object detection device 500 can utilize the neural network model trained by the learning device 100, the configuration of the system including the learning device 100 and the object detection device 500 may be any configuration.

[0047] The information storage unit 510 is a memory device such as an HDD, SSD, optical disc, RAM, or flash memory. The information storage unit 510 stores the OS and computer programs and data for causing the arithmetic processing unit 520 to execute or control various processes that will be described as being performed by the object detection device 500. The data stored in the information storage unit 510 includes images including human faces, a first similarity threshold and a second similarity threshold (<first similarity threshold) used in the processing described later. The data stored in the information storage unit 510 also includes the "upper limit of anchor boxes that can be assigned to a single correct label" as described in the first embodiment, and a neural network model trained by the learning device 100.

[0048] The arithmetic processing unit 520 is an electronic circuit such as a CPU. The arithmetic processing unit 520 performs various processes using computer programs and data stored in the information storage unit 510. In this way, the arithmetic processing unit 520 controls the operation of the entire object detection device 500 and performs or controls the various processes described as being performed by the object detection device 500. The arithmetic processing unit 520 may also be an integrated circuit such as an FPGA. The arithmetic processing unit 520 includes an object detection unit 521 and an integrated processing unit 522.

[0049] The processing performed by the arithmetic processing unit 120 (integration processing that combines human face detection frames obtained from the image using a neural network model) will be explained according to the flowchart in Figure 3.

[0050] In step S301, the object detection unit 521 reads an image containing a human face, first and second similarity thresholds, the "upper limit of anchor boxes that can be assigned to a single correct label," and a trained neural network model from the information storage unit 110 into its internal memory. The object detection unit 521 then inputs the image into the neural network model and operates the neural network model to obtain the detection result output from the neural network model (as described in Non-Patent Document 1, etc.). The detection result includes multiple detection region frames (detection frames) that are detected as areas of a human face in the image, and a confidence level (confidence level of the detection frame) that indicates the likelihood that the detection region is an area of ​​a face.

[0051] The integrated processing unit 522 acquires the detection results from the object detection unit 521. The integrated processing unit 522 then registers all detection frames included in the detection results into an unprocessed list. To provide a specific explanation, we will describe a case in which detection frames 612-1, 612-2, and 612-3 were obtained as detection frames for a person's face 611 in image 610 by the neural network model, as shown in Figure 6(a). Furthermore, as shown in Figure 6(b), we assume that the confidence scores for detection frames 612-1, 612-2, and 612-3 are 0.9, 0.7, and 0.7, respectively.

[0052] In step S302, the integrated processing unit 522 sorts the detection frames 612-1, 612-2, and 612-3 registered in the unprocessed list in descending order of confidence. Note that the sorting order is not limited to descending, and sorting may be omitted.

[0053] In step S304, the integration processing unit 522 reads detection frame 612-1, which has the highest confidence level, from the unprocessed list as the first detection frame, and deletes detection frame 612-1 from the unprocessed list. The integration processing unit 522 then initializes the integration count counter, which is used to count the number of detection frames to be integrated into the first detection frame, to 0.

[0054] The integrated processing unit 522 then performs the series of processes in steps S305 to S307 for each of the detection frames 612-2 and 612-3 registered in the unprocessed list.

[0055] In step S305, the integrated processing unit 522 obtains the detection frame with the highest confidence level from the unprocessed list as the second detection frame and deletes the second detection frame from the unprocessed list. If there are two or more detection frames with the highest confidence level in the unprocessed list, the integrated processing unit 522 may, for example, select the detection frame with the smaller index first. The integrated processing unit 522 then calculates the similarity between the second detection frame and the first detection frame. The integrated processing unit 522 calculates the IoU (Intersection over Union) between the second detection frame and the first detection frame as the similarity between the second detection frame and the first detection frame. However, as in the first embodiment, the similarity is not limited to IoU, and other similarity indicators such as GIoU (Generalized Intersection over Union) may be used. The integrated processing unit 522 then determines whether the similarity between the second detection frame and the first detection frame is equal to or greater than the first similarity threshold.

[0056] As a result of this determination, if the similarity between the second detection frame and the first detection frame is equal to or greater than the first similarity threshold, the process proceeds to step S307. If the similarity between the second detection frame and the first detection frame is less than the first similarity threshold, the process proceeds to step S306.

[0057] When detection frame 612-2 is obtained as the second detection frame, the similarity between the first detection frame and the second detection frame is 0.9, as shown in Figure 6(c). Here, assuming the first similarity threshold is 0.9, the similarity between the first detection frame and the second detection frame is greater than or equal to the first similarity threshold, so the process proceeds to step S307.

[0058] When detection frame 612-3 is obtained as the second detection frame, the similarity between the first detection frame and the second detection frame is 0.8, as shown in Figure 6(c). If the first similarity threshold is 0.9, then the similarity between the first detection frame and the second detection frame is less than the first similarity threshold, so the process proceeds to step S306.

[0059] In step S306, the integration processing unit 522 determines whether the following conditions are met: "the number of integrations is less than the maximum number of anchor boxes that can be assigned to a single correct label, and the similarity between the second detection frame and the first detection frame is equal to or greater than the second similarity threshold."

[0060] As a result of this determination, if the conditions are met, the process proceeds to step S307; otherwise, the process proceeds to step S305. In this embodiment, detection frame 612-3 is the target of step S306, and if the second similarity threshold is 0.5, the conditions are met, so the process proceeds to step S307.

[0061] In step S307, the integration processing unit 522 performs integration processing on the first detection frame and the second detection frame, making the integrated detection frame a new first detection frame. Various integration processes can be applied to the integration processing of the first detection frame and the second detection frame, and are not limited to a specific integration process. For example, the two detection frames may be integrated by calculating a weighted average of the coordinates, width, and height, weighted by the confidence level of the detection frame, as in Soft-NMS (Non-Maximum Suppression). In other words, the coordinates, width, and height of the new first detection frame may be the sum of the coordinates, width, and height of the first detection frame, each weighted by the confidence level of the first detection frame, and the coordinates, width, and height of the second detection frame, each weighted by the confidence level of the second detection frame. The integration processing unit 522 then registers the second detection frame in the processed list and increments the number of integrated frames by 1.

[0062] Through this process, the first detection frame can integrate not only detection frames whose similarity to the first detection frame is equal to or greater than the first similarity threshold, but also detection frames whose similarity to the first detection frame is equal to or greater than the second similarity threshold, provided that the number of integrated frames is less than a certain number.

[0063] The integration processing unit 522 then outputs the integrated detection frame as the final "detection frame for a human face in the image." The output destination and output format of the integrated detection frame are not limited to a specific destination or output format. For example, the object detection device 500 may display an image containing a human face on a display device that it has or that can communicate with the object detection device 500, and the integrated detection frame may be displayed superimposed on that image.

[0064] Thus, in this embodiment, by suppressing the increase in the detection frame caused by assigning multiple anchor boxes to a single correct label by the learning device 100, it is possible to prevent an increase in false detections and reduce the decrease in detection accuracy while eliminating position dependence.

[0065] [Third Embodiment] In each of the embodiments described above, the functional units of the learning device 100 shown in Figure 1 and the functional units of the object detection device 500 shown in Figure 5 were described as being implemented in hardware. However, the assignment processing unit 121, the learning processing unit 122, the object detection unit 521, and the integration processing unit 522 may be implemented in software (computer program). In this case, a computer device having a memory that functions as an information storage unit 110, or a computer device that can access said memory, and that can execute computer programs to realize the respective functions of the assignment processing unit 121 and the learning processing unit 122, is applicable to the learning device 100. Furthermore, a computer device having a memory that functions as an information storage unit 510, or a computer device that can access said memory, and that can execute computer programs to realize the respective functions of the object detection unit 521 and the integration processing unit 522, is applicable to the object detection device 500.

[0066] An example of a computer device hardware configuration applicable to such a learning device 100 and object detection device 500 will be explained using the block diagram in Figure 7. Note that the hardware configuration example shown in Figure 7 is just one example.

[0067] The CPU 701 executes various processes using computer programs and data stored in the RAM 702 and ROM 703. In doing so, the CPU 701 controls the operation of the entire computer system and executes or controls the various processes described above as being performed by the learning device 100 and the object detection device 500.

[0068] RAM 702 has areas for storing computer programs and data loaded from ROM 703 and external storage device 706, and areas for storing data received from the outside via I / F (interface) 707. Furthermore, RAM 702 has a work area used by CPU 701 when executing various processes. In this way, RAM 702 can provide various areas as appropriate.

[0069] ROM703 stores configuration data for the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and so on.

[0070] The operation unit 704 is a user interface such as a keyboard, mouse, or touch panel, and allows the user to input various instructions to the CPU 701 by operating it. For example, the user can operate the operation unit 704 to input instructions to the CPU 701 such as specifying a training image, starting training, specifying an image for object detection, and starting object detection.

[0071] The display unit 705 has a display screen such as an LCD screen or a touch panel screen, and displays the processing results of the CPU 701 in the form of images, text, etc. For example, the display unit 705 displays a training image, an image for detecting an object, a detection frame, information related to the detected object, etc. The display unit 705 may also be a projection device such as a projector that projects images and text.

[0072] The external storage device 706 is a large-capacity information storage device such as a hard disk drive. The external storage device 706 stores computer programs and data that cause the CPU 701 to execute or control various processes described as being performed by the OS, the learning device 100, and the object detection device 500. The data stored in the external storage device 706 includes the first information, second information, similarity threshold, upper limit of the number of anchor boxes that can be assigned to a single correct label, trained neural network models (parameters), training images including human faces, images to be used for human face detection, first similarity threshold, second similarity threshold, and so on.

[0073] Computer programs and data stored in the external storage device 706 are loaded into the RAM 702 as appropriate according to the control of the CPU 701 and become subject to processing by the CPU 701. The above-mentioned information storage units 110 and 510 can be implemented using RAM 702, the external storage device 706, or a combination thereof.

[0074] I / F707 is a communication interface for data communication with external devices via a network such as a LAN or the Internet. For example, a computer device may acquire images captured by an imaging device via I / F707 and store the acquired images in RAM702 or external storage device 706 as "images to be detected by the object detection device 500". Alternatively, if, for example, first information, second information, similarity threshold, upper limit of the number of anchor boxes that can be assigned to a single correct label, trained neural network model (parameters), training images including human faces, images to be detected for human faces, first similarity threshold, second similarity threshold, etc., are stored in an external device, the computer device may acquire this information via I / F707 and store the acquired information in RAM702 or external storage device 706.

[0075] The CPU 701, RAM 702, ROM 703, control unit 704, display unit 705, external storage device 706, and I / F 707 are all connected to the system bus 708.

[0076] Furthermore, although this embodiment describes a case in which the learning device 100 and the object detection device 500 are implemented using a single computer device, the learning device 100 and the object detection device 500 may also be implemented using multiple computer devices.

[0077] Furthermore, the numerical values, processing timings, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments and modifications are given as examples for the purpose of providing specific explanations, and are not intended to limit the scope to such examples.

[0078] Furthermore, some or all of the embodiments and modifications described above may be used in appropriate combinations. Alternatively, some or all of the embodiments and modifications described above may be used selectively.

[0079] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0080] The disclosures herein include the following learning devices, object detection devices, learning methods, object detection methods, and computer programs. (Item 1) A means for obtaining the similarity between the ground truth region, which represents the area of ​​an object in an image, and each of the multiple anchor boxes pre-set within the image, A selection means for selecting, from among the plurality of anchor boxes, an anchor box whose similarity is equal to or greater than a predetermined threshold for the correct answer region, A learning means that trains a neural network model for detecting the aforementioned object based on the ground truth region and the anchor boxes selected by the selection means. Equipped with, If the selection means selects an upper limit number of anchor boxes for the correct answer region, it changes the upper limit number of anchor boxes for the correct answer region based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region. A learning device characterized by the following features. (Item 2) The learning device according to item 1, characterized in that when the selection means selects an upper limit number of anchor boxes for the correct answer region, the acquisition means changes the upper limit number of anchor boxes for the correct answer region based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region and the similarity obtained by the acquisition means for anchor boxes that have not yet been selected for the correct answer region. (Item 3) The learning device according to item 2, characterized in that, when the selection means selects an upper limit number of anchor boxes for the correct answer region, it selects for the correct answer region an anchor box from among the anchor boxes not yet selected for the correct answer region whose similarity difference with the smallest similarity among the anchor boxes selected for the correct answer region is less than or equal to a predetermined value. (Item 4) The learning device according to item 2, characterized in that the selection means calculates the sum of similarities for each combination of anchor boxes selected for the group of correct regions in the image, and determines the combination of anchor boxes that maximizes the sum. (Item 5) The learning device according to any one of items 1 to 4, characterized in that the learning means performs the learning such that, in the image, anchor boxes selected for the correct region by the selection means are converted into the correct region, and anchor boxes not selected for the correct region by the selection means become the background region. (Item 6) The learning device according to any one of items 1 to 5, characterized in that the acquisition means calculates the IoU (Intersection over Union) between the correct answer region and each of the plurality of anchor boxes as the similarity. (Item 7) The learning device according to any one of items 1 to 5, characterized in that the acquisition means calculates the GIoU (Generalized Intersection over Union) between the correct answer region and each of the plurality of anchor boxes as the similarity. (Item 8) The learning device according to any one of items 1 to 7, wherein the selection means further changes the upper limit of the number of anchor boxes based on the size of the object in the image. (Item 9) The learning device according to item 8, characterized in that the selection means changes the upper limit of the anchor boxes based on the size of the object in the image and a predetermined reference value which is set such that the smaller the size of the object, the greater the upper limit of the anchor boxes. (Item 10) The acquisition means further acquires a second similarity between the correct answer region and each of the plurality of anchor boxes, The learning device according to item 2, characterized in that the selection means selects the anchor box based on the second similarity if the difference between the similarity obtained by the acquisition means for the anchor box selected for the correct answer region and the similarity obtained by the acquisition means for the anchor box not yet selected for the correct answer region is less than or equal to a predetermined value. (Item 11) The learning device according to item 10, characterized in that the second similarity is a similarity based on at least one of the distance between the center of the ground truth region and the anchor box, and the difference in size between the ground truth region and the anchor box. (Item 12) The learning device according to item 10, characterized in that the second similarity is the sum of the difference in center distance and size between the correct answer region and the anchor box. (Item 13) An acquisition means for acquiring multiple detection frames for an object, and the confidence level of the detection frames, which are output from a neural network model that has been trained by any one of the learning devices described in item 1 to 12, by inputting an image containing an object into the neural network model. An integrating means for integrating the first detection frame with the highest confidence level among the plurality of detection frames, and the second detection frames, excluding the first detection frame, that are less than the upper limit number among the plurality of detection frames. An object detection device characterized by comprising the following features. (Item 14) The object detection device according to item 13, characterized in that the integrating means integrates a second detection frame whose similarity to the first detection frame is equal to or greater than a first similarity threshold into the first detection frame. (Item 15) The object detection device according to item 14, characterized in that, if the number of detection frames integrated into the first detection frame is less than the upper limit, the integrating means integrates into the first detection frame second detection frames, which have a similarity to the first detection frame less than the first similarity threshold and a similarity to the second similarity threshold that is smaller than the first similarity threshold. (Item 16) A learning method performed by a learning device, The acquisition means of the learning device includes an acquisition step of acquiring the similarity between the ground truth region, which indicates the region of an object in the image, and each of the multiple anchor boxes that are pre-set in the image, The selection means of the learning device includes a selection step of selecting an anchor box from among the plurality of anchor boxes whose similarity is equal to or greater than a predetermined threshold for the correct answer region, The learning means of the learning device performs a learning step in which it learns a neural network model for detecting the object based on the correct answer region and the anchor boxes selected in the selection step. Equipped with, In the selection step, if the maximum number of anchor boxes is selected for the correct answer region, the maximum number of anchor boxes for the correct answer region is changed based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region. A learning method characterized by the following: (Item 17) An object detection method performed by an object detection device, The acquisition means of the object detection device includes an acquisition step of acquiring a plurality of detection frames for the object, and the confidence level of the detection frames, which are output from the neural network model when an image containing the object is input to a neural network model trained by any one of the learning devices described in item 1 to 12. The integration means of the object detection device includes an integration step of integrating a first detection frame with the highest reliability among the plurality of detection frames and a second detection frame among the plurality of detection frames that is less than the upper limit number excluding the first detection frame. An object detection method characterized by comprising the following features. (Item 18) A computer program that causes a computer to function as one of the means of a learning device described in any one of items 1 through 12. (Item 19) A computer program for causing a computer to function as one of the means of an object detection device described in any one of items 13 through 15.

[0081] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0082] 100: Learning device 110: Information storage unit 120: Arithmetic processing unit 121: Assignment processing unit 122: Learning processing unit

Claims

1. An acquisition means for obtaining the similarity between the ground truth region, which indicates the area of ​​an object in the image, and each of the multiple anchor boxes pre-set in the image, based on IoU (Intersection over Union), A selection means for selecting, from among the plurality of anchor boxes, an anchor box whose similarity is equal to or greater than a predetermined threshold for the correct answer region, A learning means that trains a neural network model for detecting the aforementioned object based on the ground truth region and the anchor boxes selected by the selection means. Equipped with, If the selection means selects an upper limit number of anchor boxes for the correct answer region, it changes the upper limit number of anchor boxes for the correct answer region based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region. A learning device characterized by the following features.

2. The learning device according to claim 1, characterized in that when the selection means selects an upper limit number of anchor boxes for the correct answer region, it changes the upper limit number of anchor boxes for the correct answer region based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region and the similarity obtained by the acquisition means for anchor boxes not yet selected for the correct answer region.

3. The learning device according to claim 2, characterized in that, when the selection means selects an upper limit number of anchor boxes for the correct answer region, it selects for the correct answer region an anchor box from among the anchor boxes not yet selected for the correct answer region whose similarity difference with the smallest similarity among the anchor boxes selected for the correct answer region is less than or equal to a predetermined value.

4. The learning device according to claim 2, characterized in that the selection means calculates the sum of similarities for each combination of anchor boxes selected for the group of correct regions in the image, and determines the combination of anchor boxes that maximizes the sum.

5. The learning device according to claim 1, characterized in that the learning means performs the learning such that, in the image, anchor boxes selected for the correct region by the selection means are converted into the correct region, and anchor boxes not selected for the correct region by the selection means become the background region.

6. The learning device according to claim 1, characterized in that the acquisition means determines the IoU (Intersection over Union) between the correct answer region and each of the plurality of anchor boxes as the similarity.

7. The learning device according to claim 1, characterized in that the acquisition means calculates the GIoU (Generalized Intersection over Union) between the ground truth region and each of the plurality of anchor boxes as the similarity.

8. The learning device according to claim 1, wherein the selection means further changes the upper limit of the number of anchor boxes based on the size of the object in the image.

9. The learning device according to claim 8, characterized in that the selection means changes the upper limit of the anchor boxes based on the size of the object in the image and a predetermined reference value which is set such that the upper limit of the anchor boxes increases as the size of the object decreases.

10. The acquisition means further acquires a second similarity between the correct answer region and each of the plurality of anchor boxes, The learning device according to claim 2, characterized in that the selection means selects the anchor box based on the second similarity if the difference between the similarity obtained by the acquisition means for the anchor box selected for the correct answer region and the similarity obtained by the acquisition means for the anchor box not yet selected for the correct answer region is less than or equal to a predetermined value.

11. The learning device according to claim 10, characterized in that the second similarity is a similarity based on at least one of the center distance between the correct answer region and the anchor box, and the size difference between the correct answer region and the anchor box.

12. The learning device according to claim 10, characterized in that the second similarity is the sum of the difference in center distance and size between the correct answer region and the anchor box.

13. An acquisition means for acquiring a plurality of detection frames for an object, and the confidence level of the detection frames, which are output from a neural network model by inputting an image containing an object into a neural network model trained by the learning device described in claim 1. An integrating means for integrating the first detection frame with the highest confidence level among the plurality of detection frames, and the second detection frames, excluding the first detection frame, that are less than the upper limit number among the plurality of detection frames. An object detection device characterized by comprising the following features.

14. The object detection device according to claim 13, characterized in that the integrating means integrates a second detection frame whose similarity to the first detection frame is equal to or greater than a first similarity threshold into the first detection frame.

15. The object detection device according to claim 14, characterized in that, if the number of detection frames integrated into the first detection frame is less than the upper limit, the integrating means integrates into the first detection frame second detection frames, which have a similarity to the first detection frame less than the first similarity threshold, and which have a similarity to the first detection frame greater than or equal to a second similarity threshold that is smaller than the first similarity threshold.

16. A learning method performed by a learning device, The acquisition means of the learning device includes an acquisition step of acquiring the similarity based on IoU (Intersection over Union) between the ground truth region indicating the region of an object in the image and each of the multiple anchor boxes pre-set in the image, The selection means of the learning device includes a selection step of selecting an anchor box from among the plurality of anchor boxes whose similarity is equal to or greater than a predetermined threshold for the correct answer region, The learning means of the learning device performs a learning step in which it learns a neural network model for detecting the object based on the correct answer region and the anchor boxes selected in the selection step. Equipped with, In the selection step, if the maximum number of anchor boxes is selected for the correct answer region, the maximum number of anchor boxes for the correct answer region is changed based on the similarity obtained by the acquisition means for the anchor boxes selected for the correct answer region. A learning method characterized by the following:

17. An object detection method performed by an object detection device, The acquisition means of the object detection device includes an acquisition step of acquiring a plurality of detection frames for the object and the confidence level of the detection frames, which are output from the neural network model when an image containing the object is input to the neural network model trained by the learning device described in claim 1. The integration means of the object detection device includes an integration step of integrating a first detection frame with the highest reliability among the plurality of detection frames and a second detection frame among the plurality of detection frames that is less than the upper limit number excluding the first detection frame. An object detection method characterized by comprising the following features.

18. A computer program for causing a computer to function as one of the means of a learning device according to any one of claims 1 to 12.

19. A computer program for causing a computer to function as one of the means of an object detection device according to any one of claims 13 to 15.

Images