Information processing device, control method and program for information processing device

Abstract

To integrate detection frames to select an appropriate attribute for a result of the integration even when multiple different attributes are detected for one object.SOLUTION: An information processing apparatus is configured to: acquire, from an input image, detection frames where an object exists and acquire reliability indicating the possibility of presence of the object, and a class probability; calculate overlap rates of the detection frames based on a set of arbitrary two detection results; output an overlap detection group of the detection frames with overlap rates equal to or higher than a threshold; determine, as a representative frame, a detection frame having the highest reliability from among the detection results included in the overlap detection group; and calculates class indices on the basis of the representative frame and the overlap rates, to determine a class with the highest class index as a class of the representative frame.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] In particular, the present invention relates to a technique for detecting an object from an image. [Background technology]

[0002] In recent years, image analysis has been used in a variety of situations to detect and track objects using images captured by imaging devices such as surveillance cameras, estimate the attributes of those objects, and estimate the number of objects using the results of such image analysis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-180945 [Non-patent literature]

[0004] [Non-Patent Document 1] J.Redmon, A.Farhadi,"YOLO9000:Better Faster Stronger", Computer Vision and Pattern Recognition (CVPR) 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 avoids merging object detection frames of different types (attributes) by using a threshold adjusted according to the type of detection target object included in the detection processing frame. Therefore, if multiple different types of detection results are output for a single object, the object detection frames cannot be merged into one.

[0006] In view of the above-mentioned problems, the present invention aims to integrate detection frames into one, even when detection results for multiple different attributes are obtained for a single object, and to enable the selection of an appropriate attribute for the integrated result. [Means for solving the problem]

[0007] The information processing device according to the present invention comprises an input means for acquiring an input image, a detection means for detecting a candidate area where an object exists from the acquired input image and acquiring candidate attributes of the object, an overlap rate acquisition means for acquiring an overlap rate between the plurality of candidate areas when there are a plurality of candidate areas, an integration means for setting the candidate area with the highest reliability indicating the possibility that the object may be included in the candidate area as a representative area for each combination of the candidate areas and deleting candidate areas whose overlap rate with the representative area is equal to or greater than a threshold, and an object detection means for detecting an object included in the candidate areas. Attributes and the object and determining means for determining the attribute of the object in the representative region based on the probability of the attribute and the overlap rate with the representative region. [Effects of the Invention]

[0008] According to the present invention, even when detection results with multiple different attributes are obtained for one object, the detection frames can be integrated into one and an appropriate attribute can be selected for the integrated result. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 3] 5 is a flowchart illustrating an example of a procedure for object detection processing according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining an object detection process according to the first embodiment. [Figure 5] 10 is a flowchart illustrating an example of a procedure for object detection processing according to the second embodiment. [Figure 6] FIG. 10 is a diagram for explaining an object detection process according to the second embodiment. [Figure 7] 10A and 10B are diagrams for explaining a pattern in which a detection region and a detection frame are in contact with each other; [Figure 8] 11 is a flowchart illustrating an example of a procedure for object detection processing according to the third embodiment. [Figure 9] 13 is a flowchart showing an example of a detailed procedure of a frame integration process according to the third embodiment. [Figure 10] FIG. 10 is a diagram for explaining an object detection process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (First embodiment) In object detection, for example, the position and size of the object to be detected, the attributes of the object, the reliability of the object's existence, etc. are output. In object detection, multiple detection results may be generated for one object. This can lead to problems such as a decrease in the reliability of the detection results or the reliability of statistical data. This embodiment describes a method for determining the optimal detection result when multiple detection results are generated for one object. Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. 1 is a block diagram showing an example of the hardware configuration of an information processing device 100 according to this embodiment. The information processing device 100 according to this embodiment has an object detection function that detects a target object from an image captured by an imaging device such as a surveillance camera. The following describes the case of detecting a person's face as an example, but the present invention is not limited to this and can be applied to any system that analyzes an image to detect a predetermined object.

[0011] The information processing device 100 according to this embodiment includes a CPU (Central Processing Unit) 101, a memory 102, a communication interface (I / F) unit 103, a display unit 104, an input unit 105, and a storage unit 106. These components are communicably connected via a system bus 107. The information processing device 100 according to this embodiment may further include other components.

[0012] The CPU 101 is responsible for overall control of the information processing device 100. The CPU 101 controls the operation of each functional unit connected via, for example, a system bus 107. The memory 102 stores data, programs, etc. used by the CPU 101 for processing. The memory 102 also functions as the main memory, work area, etc. of the CPU 101. The CPU 101 executes processing based on the programs stored in the memory 102, thereby realizing the functional configuration of the information processing device 100 shown in FIG. 2 (described later) and the processing of the flowchart shown in FIG. 3 (described later).

[0013] The communication I / F unit 103 is an interface that connects the information processing device 100 to a network. The display unit 104 has a display member such as a liquid crystal display, and displays the results of processing by the CPU 101, etc. The input unit 105 has operation members such as a mouse or buttons, and inputs user operations to the information processing device 100. The storage unit 106 stores, for example, various data and the like required when the CPU 101 performs processing related to a program. The storage unit 106 also stores, for example, various data and the like obtained when the CPU 101 performs processing related to a program. Note that data, programs, etc. used by the CPU 101 for processing may be stored in the storage unit 106.

[0014] 2 is a block diagram showing an example of the functional configuration of the information processing device 100 according to this embodiment. The information processing device 100 includes an image acquisition unit 201, an object detection unit 202, an overlap determination unit 203, a representative frame determination unit 204, a class determination unit 205, a result correction unit 206, a result output unit 207, and a storage unit 208.

[0015] The image acquisition unit 201 acquires an image that is to be subjected to object detection. In this embodiment, the image that is to be subjected to object detection is acquired from the outside via the communication I / F unit 103. Hereinafter, the data of the image that is to be subjected to object detection and acquired by the image acquisition unit 201 will also be simply referred to as an "input image." In addition, in the following description, the input image is, as an example, an RGB image of 1080 x 720 pixels, with a horizontal (width) width of 1080 pixels and a vertical (length) height of 720 pixels. Note that the input image is not limited to an RGB image of 1080 x 720 pixels, and any image can be used as the input image, and for example, the horizontal width and vertical height may be different.

[0016] The object detection unit 202 detects objects related to multiple attributes (classes) from an image. In this embodiment, the object detection unit 202 detects human faces from the input image acquired by the image acquisition unit 201. The object detection unit 202 also outputs the detection result using a machine learning model (trained model) that has been trained to be able to detect "faces with glasses" and "faces without glasses" included in the image. The detection of "faces with glasses" and "faces without glasses" can be achieved by applying the technology described in Non-Patent Document 1, for example.

[0017] Here, the detection results output by the object detection unit 202 include the position and size of the detected face (candidate area), the detection confidence score, and class probabilities, which indicate the probability to which attribute (class) the face belongs, as well as the detection confidence score. The position and size of the face are output, for example, using coordinates that define a rectangular frame (candidate area) surrounding the face (for example, the upper left coordinates (x1, y1) and the lower right coordinates (x2, y2) of the rectangle). The detection confidence represents, for example, the confidence that a face is included in the rectangular frame (candidate area), and is output as a real number between 0 and 1, with 0 representing the lowest confidence level and 1 representing the highest confidence level when obtaining the confidence score. The face class probability indicates the probability that the face is a face wearing glasses and the probability that the face is a face not wearing glasses, and the sum of these probabilities is 1 (100%). Hereinafter, the rectangular frame surrounding the face, the detection reliability, and the face class probability will be simply referred to as the "detection frame," "reliability," and "class probability," respectively. Note that the method of outputting the detection results is not limited to the example described above, and it is sufficient if the position and range of the detected face, the detection reliability, and the face class probability can be recognized.

[0018] The overlap determination unit 203 determines whether or not the detection results overlap based on the detection results (particularly the positions and sizes of the candidate areas) obtained by the object detection unit 202. The overlap determination unit 203 groups any two detection frames from all detection results obtained by the object detection unit 202 into a pair and calculates the overlap rate of the detection frames for each pair. The overlap determination unit 203 determines that there is overlap if the calculated overlap rate is equal to or greater than a threshold, that is, if there is a pair of detection frames whose detection frame areas overlap by a predetermined percentage or more, and outputs the pair of detection results as an "overlapping detection group." In this embodiment, when acquiring the overlap rate, the overlap rate is calculated using IoU (Intersection over Union), and the threshold is set to 0.5, for example. In other words, if the value obtained by dividing the area of ​​the common part of the areas of the two detection frames by the union of the areas of the two areas is 0.5 or greater, the overlap determination unit 203 determines that there is overlap. If there is no pair of detection frames that overlap by more than the threshold, the overlap determination unit 203 determines that there is no overlap.

[0019] The representative frame determination unit 204 determines one detection frame to serve as a representative region for each overlapping detection group output by the overlap determination unit 203, based on the detection results (particularly the reliability of the candidate regions) obtained by the object detection unit 202. For each overlapping detection group, the representative frame determination unit 204 determines the detection frame corresponding to the detection result with the highest reliability among the detection results contained therein as the representative frame (representative region) for that overlapping detection group. Note that if there are multiple detection results with the highest reliability, the detection frame with the largest area within the corresponding detection frame is determined as the representative frame. Note that, when there are multiple detection results with the highest reliability in one overlapping detection group, an index other than the area within the detection frame may be used to determine the representative frame. Note that the reliability of all object detection results (candidate regions) may be sorted in descending order, and the top N candidate regions or candidate regions with a reliability equal to or greater than a threshold may be determined as the representative region. A specific example of this process will be described in embodiment 3.

[0020] The class determination unit 205 determines the class of the representative frame determined by the representative frame determination unit 204, using the class probability of each detection result included in the overlapping detection group. Details of the class determination process by the class determination unit 205 will be described later. This embodiment can improve the object detection accuracy by using not only the class probability of the object in the representative region, but also the class probability of the object in the candidate region that overlaps with the representative region.

[0021] The result correction unit 206 corrects the detection results obtained by the object detection unit 202 using the outputs of the overlap determination unit 203, the representative frame determination unit 204, and the class determination unit 205. For each overlapping detection group output by the overlap determination unit 203, the result correction unit 206 deletes all detection results other than those corresponding to the representative frame determined by the representative frame determination unit 204. Furthermore, for a detection result whose overlap rate with any other frame is less than a threshold, the result correction unit 206 determines the class with the highest class probability as the class of that detection result. Through the above result correction process, only one detection result corresponding to the representative frame remains for each overlapping detection group, and the class of that result is set to the class determined by the class determination unit 205. The class of each of the other detection results that did not overlap is also determined.

[0022] The result output unit 207 outputs the results of the processing by the result correction unit 206. The format may be data of the coordinates and class of the detection frame, or an image in which the detection result is superimposed on the input image may be output. The storage unit 208 stores data used in the processes performed by the image acquisition unit 201 to the result output unit 207 of the information processing device 100, data obtained as the process results, and the like.

[0023] Next, processing performed by the information processing device 100 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a flowchart showing an example of the procedure of object detection processing according to this embodiment. Fig. 4 is a diagram for explaining the object detection processing according to this embodiment. In step S301, the image acquisition unit 201 acquires an input image (an image to be subjected to object detection). Fig. 4(a) shows an example of an input image 410 in this embodiment. In this embodiment, the input image 410 is assumed to be an image of 1080 x 720 pixels, as described above.

[0024] In step S302, the object detection unit 202 performs face detection processing to detect the face of a person to be detected in the input image. Then, it outputs a confidence level and a class probability (a probability that the face is in the "glasses worn" class and a probability that the face is in the "glasses not worn" class) for each detected face. FIG. 4(b) shows an example of the detection results of the face detection processing for the input image, and FIG. 4(c) shows an example of an image in which the detection results are superimposed on the input image. In the example shown in FIG. 4(b), three detection results A to C are obtained, and the upper left coordinates (x1, y1) and lower right coordinates (x2, y2) of each rectangular detection frame, the confidence level, and the class probability (candidates are "glasses worn" and "glasses not worn"). In the example shown in FIG. 4(c), rectangular detection frames 411 to 413 corresponding to the detection results A to C are superimposed on the input image 410 and displayed on the display unit 104.

[0025] In step S303, overlap determination unit 203 calculates the overlap rate of the detection frames by pairing any two of the detection results for the input image. In the example of FIG. 4(b), the upper left coordinates of detection frame 411 of detection result A are (143, 165) and the lower right coordinates are (417, 418). Furthermore, the upper left coordinates of detection frame 412 of detection result B are (166, 190) and the lower right coordinates are (450, 446). Therefore, the overlap rate of the detection frames of detection result A and detection result B is IoU(A,B)=((417-166)×(418-190))÷((417-143)×(418-165)+(450-166)×(446-190)-(417-166)×(418-190))≒0.67 For other combinations, the overlap rate of the detection window is 0.

[0026] In step S304, overlap determination unit 203 determines whether there is a combination of detection results whose overlap rate calculated in step S303 is equal to or greater than a threshold. If overlap determination unit 203 determines that there is a combination of detection results whose overlap rate of detection frames is equal to or greater than the threshold (YES in step S304), it outputs the combination of detection results whose overlap rate is equal to or greater than the threshold (overlapping detection group) and proceeds to step S305. On the other hand, if overlap determination unit 203 determines that there is no combination of detection results whose overlap rate of detection frames is equal to or greater than the threshold (NO in step S304), it proceeds to step S309. In this embodiment, the overlap rate threshold is 0.5 as described above. Here, the overlap rate of the detection frames of detection result A and detection result B is calculated to be 0.67, which is equal to or greater than the threshold of 0.5. Therefore, overlap determination unit 203 outputs the combination whose overlap rate is equal to or greater than 0.5 as an overlapping detection group (A, B) and proceeds to step S305.

[0027] In step S305, the representative frame determination unit 204 compares the reliability of each detection result included in the overlapping detection group output in step S304, and determines the detection frame corresponding to the detection result with the highest reliability as the representative frame of that overlapping detection group. For the overlapping detection group (A, B) in this example, according to Figure 4(b), the reliability of detection result A is 0.80 and the reliability of detection result B is 0.75, so the representative frame is determined to be the detection frame corresponding to detection result A, which has the highest reliability.

[0028] In step S306, the class determination unit 205 determines the class of the representative frame determined in step S305 using the class probability and overlap rate of each detection result included in the overlap detection group output in step S303. In the case of the overlap detection group (A, B) in this example, the class index of this overlap detection group is calculated as follows, as a weighted sum in which the class probability of each detection frame shown in Figure 4(b) is weighted by the above-mentioned overlap rate (limited to the overlap rate with the representative frame). Glasses wearer class index = 1 x 0.55 + 0.67 x 0.15 ≒ 0.65 Non-glasses wearer class index = 1 x 0.45 + 0.67 x 0.85 ≒ 1.02 Note that the overlap rate with itself is 1, so the first term on the right side of the above equation is multiplied by 1.

[0029] The class with the largest calculated class index is determined as the representative frame class of the target. In this example, since the non-glasses wearer class index is the largest, the class of this representative frame is the non-glasses wearer class. Note that if there are multiple classes with the same largest calculated class index, the class with the highest original class probability of the representative frame is adopted. For example, since the original information of the detection frame that became the representative frame in this example is detection result A, if both of the above class indices are the same value, the class with the higher class probability of detection result A, i.e., the glasses wearer class, is determined as the representative frame class. The class determination unit 205 sets the class probability of the class determined as above to 1 and the other classes to 0, and overwrites and updates the class probabilities of the detection results corresponding to the representative frame determined in step S305.

[0030] In step S307, the result correction unit 206 deletes the detection results other than the detection result corresponding to the representative frame from the overlapping detection group.

[0031] In step S308, result modification unit 206 determines whether or not processing has been completed for all combinations of detection results for which the overlap rate of detection frames is equal to or greater than the threshold. If result modification unit 206 determines that processing has been completed for all combinations of detection results for which the overlap rate is equal to or greater than the threshold (YES in step S308), it proceeds to step S309. On the other hand, if result modification unit 206 determines that there are unprocessed combinations of detection results for which the overlap rate is equal to or greater than the threshold (NO in step S308), it proceeds to step S305, and executes the processing from step S305 onwards for the unprocessed combinations.

[0032] In step S309, the result modification unit 206 determines a class for each detection result. For a detection result that has become representative of a group of overlapping detections after going through the processes of steps S305 to S308, the class is determined to be the class determined in step S306. For a detection result that has not gone through the processes of steps S305 to S308 and for which the output of step S302 remains unchanged, the class with the maximum class probability is determined to be the class of that detection result. By this process, one class is determined for each detection result (detection frame), as shown in FIG. 4(d).

[0033] In step S310, the result output unit 207 outputs the corrected detection result data as shown in Fig. 4(d), terminates the process, and moves on to processing the next input image. This output data can be used in a format in which a rectangular detection frame is superimposed on the input image 410, as shown in Fig. 4(e). In Fig. 4(e), a dashed rectangular frame 414 representing the non-glasses class is superimposed on the face of the person on the left as the detection frame for detection result A, and a dotted rectangular frame 415 representing the glasses class is superimposed on the face of the person on the right as the detection frame for detection result C.

[0034] As described above, according to this embodiment, when multiple overlapping detection results occur in object detection for an input image, they can be merged into a single, most appropriate detection frame. Furthermore, the attribute (class) of the merged detection frame is calculated based on the class probability of the multiple detection results before merging and the overlap rate of the detection frames, making it possible to select the most appropriate attribute (class). This makes it possible to ultimately output a more appropriate detection result as the detection result for object detection for the input image.

[0035] Note that the object detection process in object detection unit 202 is not limited to the technique disclosed in Non-Patent Document 1, and various other techniques can be applied as long as they are capable of detecting the desired object. The representative frame determined by representative frame determination unit 204 may be any region that includes the detected object. For example, a circumscribing rectangle for a union of detection frames included in an overlapping detection group may be defined as the representative frame. Alternatively, a circumscribing rectangle for a union of detection frames with the highest reliability or overlap rate among those included in an overlapping detection group may be defined as the representative frame.

[0036] Furthermore, in this embodiment, an example in which two detection frames overlap has been described, but three or more detection frames may also be detected. For example, if three detection results M, N, and O overlap, and the overlap rates of detection results M, N, N, and M, O are all 0.5 or greater, the overlap determination unit 203 outputs them as an overlap detection group (M, N, O). Then, for example, if the reliability of detection result M is the highest, the class determination unit 205 calculates each class index using the overlap rates of the overlap detection groups (M, N) and (M, O), and does not use the overlap rate of the overlap detection group (N, O).

[0037] (Second embodiment) In the first embodiment, a process for appropriately integrating multiple overlapping detection results into one is described. In the second embodiment, a process for integrating multiple detection results when multiple detection areas are set on an image to be detected is described. In the following description, the same reference numerals are used for components common to the first embodiment, and descriptions thereof will be omitted.

[0038] Fig. 5 is a flowchart showing an example of the procedure of the object detection process performed by the information processing device 100 in this embodiment, and parts common to the flowchart shown in Fig. 3 are assigned the same reference numerals as in Fig. 3. Fig. 6 is a diagram for explaining the object detection process according to this embodiment. In step S301, the image acquisition unit 201 acquires an input image (an image to be subjected to object detection). Fig. 6(a) shows an example of an input image 610 in this embodiment. In this embodiment, as in the first embodiment, the input image 610 is assumed to be an image of 1080 x 720 pixels.

[0039] In step S501, the object detection unit 202 sets an area (detection area) to be subjected to detection processing in the input image. FIG. 6(b) shows how detection areas a (611) and b (612) have been set. The upper left coordinates of detection area a are (99,127) and the lower right coordinates are (719,747), while the upper left coordinates of detection area b are (546,10) and the lower right coordinates are (1076,540). Note that there is no limit to the number of detection areas that can be set, but for the sake of explanation, it is assumed here that the aforementioned two areas are set. Furthermore, in this embodiment, detection areas are set so that multiple detection areas overlap, as shown in FIG. 6(b), depending on the characteristics of the scene, etc., shown in the input image.

[0040] In step S502, the object detection unit 202 performs face detection processing for each detection area set in step S501. The face detection processing performed for each detection area is the same as the processing performed in step S302 in the first embodiment. FIG. 6(c) shows an example of the detection results obtained by performing face detection processing for each detection area set in the input image, and FIG. 6(d) shows an example of an image in which the detection results are superimposed on the input image. Because detection area b (612) contains a part of a person at the left edge, part of the face is detected incompletely, as in detection results B and C. On the other hand, because the same person's face is detected completely in detection area a (611), processing is performed to correctly integrate these detection results. Subsequent processing handles the detection results for all detection areas simultaneously. In this example, three detection results A to C are obtained, and in the example shown in FIG. 6(d), rectangular detection frames 613 to 615 corresponding to detection results A to C are superimposed on the input image 610 and displayed on the display unit 104.

[0041] In step S503, the overlap determination unit 203 calculates the overlap rate of the detection frames by pairing any two of the multiple detection results. In the first embodiment, the overlap rate here is defined as IoU, and the IoU threshold in the next step S304 is set to 0.5. However, as mentioned above, if an incomplete detection result is output due to the presence of part of a person's face at the edge of the detection area, the overlap rate will be calculated as low even if the detection results are the same face if the overlap rate is defined as IoU. For example, the IoU of the detection frames of detection result A and detection result B in Figure 6(c) is IoU(A,B)=((685-546)×(414-145))÷((685-410)×(414-145)+(705-546)×(440-113)-(685-546)×(414-145))≒0.42 The IoU of the detection frames of detection results A and C is IoU(A,C)=((660-567)×(384-186))÷((685-410)×(414-145))≒0.25 The IoU of the detection frames of detection results B and C is IoU(B,C)=((660-567)×(384-186))÷((705-546)×(440-113))≒0.20 Therefore, if the threshold value is set to 0.5 as in the first embodiment, none of the combinations of detection results A to C will be integrated because they are below the threshold value.

[0042] Therefore, in this embodiment, when calculating the overlap rate, the Simpson coefficient is introduced, which expresses a sufficiently high overlap rate even when one region is included in a larger portion of the other. The overlap rate using the Simpson coefficient is defined as the area of ​​the common portion of the regions of the two detection frames divided by the area of ​​the smaller detection frame region of the two detection frames. The Simpson coefficient for the detection frames of detection results A and B is: Simpson(A,B)=((685-546)×(414-145))÷((705-546)×(440-113))≒0.72 The Simpson coefficient for the detection frames of detection results A and C is: Simpson(A,C)=1 The Simpson coefficient for the detection frames of detection results B and C is: Simpson(B,C)=1 Since both are equal to or greater than the threshold value of 0.5, the process can proceed to the subsequent integration process.

[0043] From the above, in step S503, overlap determination unit 203 calculates both the IoU and the Simpson coefficient as the overlap rate of the detection frames. The Simpson coefficient calculated here is used in step S304 as the overlap rate for determining whether or not to subject the detection frames to the integration process executed in steps S304 to S308. On the other hand, the IoU calculated here is used in step S306 as the overlap rate for determining the class of the representative frame into which multiple frames are integrated.

[0044] In step S304, the overlap determination unit 203 determines whether there is a combination of detection results whose overlap rate according to the Simpson coefficient calculated in step S503 is equal to or greater than a threshold. If the overlap determination unit 203 determines that there is a combination of detection results whose overlap rate of the detection frames is equal to or greater than the threshold (YES in step S304), it outputs the combination of detection results whose overlap rate is equal to or greater than the threshold (overlapping detection group) and proceeds to step S504. On the other hand, if the overlap determination unit 203 determines that there is no combination of detection results whose overlap rate of the detection frames is equal to or greater than the threshold (NO in step S304), it proceeds to step S309. In this embodiment, the overlap rate threshold is 0.5 as described above. In this example, the overlap rate (Simpson coefficient) of the detection frames of detection result A and detection result B is 0.72, and the overlap rate (Simpson coefficient) of the detection frames of detection result A and detection result C and detection result B and detection result C is 1, all of which are equal to or greater than the threshold of 0.5. In this case, the combinations with an overlap rate of 0.5 or more become overlap detection groups (A, B), (A, C), and (B, C), which are overlapping combinations. Therefore, the overlap determination unit 203 outputs the combinations with an overlap rate of 0.5 or more as overlap detection group (A, B, C), and proceeds to step S504.

[0045] In step S504, the object detection unit 202 determines whether any of the detection frames of each detection result included in the overlapping detection group output in step S304 touches the boundary of the detection area. Whether a detection frame touches the boundary of a detection area is determined by checking whether any of the four sides of the detection frame of each detection result touches any of the four sides of the detection area from which that result was obtained. In the examples of FIGS. 6(c) and 6(d), the left-edge x-coordinate of the detection frame 614 of detection result B matches 546 with the left-edge x-coordinate of detection area b, the detection area from which that result was obtained. Therefore, it is determined that the detection frame 614 of detection result B touches the boundary of detection area b. Note that the detection frame 613 of detection result A does not touch the boundary of detection area a, and similarly, the detection frame 615 of detection result C does not touch the boundary of detection area b. If it is determined that any of the detection frames touches the boundary of the detection area (YES in step S504), information about the detection frame that touches the boundary of the detection area is output, and the process proceeds to step S505. On the other hand, if it is determined that there is no detection frame that touches the boundary of the detection area (NO in step S504), the process proceeds to step S305.

[0046] In step S505, the object detection unit 202 performs a process of adjusting the reliability of the detection result corresponding to the detection frame that abuts the boundary of the detection area output in step S504. A detection frame that abuts the boundary of the detection area can be interpreted as possibly representing a detection result for part of a face, and therefore may be incomplete as face detection information. Therefore, the reliability is adjusted to reduce the contribution rate to the representative frame and representative class probability when integrating multiple detection results. The reliability adjustment here is performed, for example, by multiplying the reliability by a predetermined coefficient. In this embodiment, the predetermined coefficient is 0.8. As described above, since the detection frame 614 of detection result B abuts the boundary of detection area b, the reliability of detection result B (0.85) shown in FIG. 6(c) is multiplied by the predetermined coefficient 0.8 to obtain an adjusted reliability of 0.68. FIG. 6(e) shows the detection result after this adjustment, in which the reliability of detection result B has been reduced to 0.68.

[0047] Following the processing up to this point, the information processing device 100 executes the same processing from step S305 onwards as in the first embodiment. In the example shown in Fig. 6(e), in step S305, the representative frame determination unit 204 determines the detection frame 613 of the detection result A, whose reliability is 0.80, as the representative frame.

[0048] In the next step S306, the class determination unit 205 calculates each class index using the overlap rate of the two overlap detection groups (A, B) and (A, C) associated with the representative frame, and determines the representative frame class. Note that, as described above, the overlap rate used to calculate the class index in step S306 applies IoU as in the first embodiment. This is because, as in this embodiment, the contribution rate of detection result C of detection frame 615, which is completely contained in detection frame 613 of detection result A, is a more appropriate value than the Simpson coefficient. In the example shown in FIG. 6(e), each class index is calculated as a sum weighted by the overlap rate as follows: Glasses wearer class index = 1 x 0.15 + 0.42 x 0.30 + 0.25 x 0.60 ≒ 0.426 Non-glasses wearer class index = 1 x 0.85 + 0.42 x 0.70 + 0.25 x 0.40 ≒ 1.244 As a result, the class determination unit 205 determines the non-glasses wearing class as the representative frame class. Note that the overlap rate of the overlap detection group (B, C) not related to the representative frame is not the overlap rate with the representative frame, and therefore is not used in calculating the class index.

[0049] The detection result data output by the result output unit 207 in step S310 is, for example, the result shown in Fig. 6(f). This detection result can be used in a format in which a rectangular detection frame is superimposed on the input image 610. In Fig. 6(g), a dashed rectangle 616 representing the non-glasses wearing class is superimposed on the person's face.

[0050] As described above, according to this embodiment, when a plurality of detection areas are set for an input image, it is possible to appropriately integrate a plurality of detection results for detection targets near the boundaries of the detection areas.

[0051] In step S505, the predetermined coefficient by which the reliability of the detection result by the object detection unit 202 is multiplied is not limited to a fixed value as described above, and may be determined, for example, according to the positional relationship between the detection area and the detection frame. For example, as shown in the conceptual diagrams of FIGS. 7(a) to 7(c), the predetermined coefficient may be changed according to the number of sides where the detection area shown by the dotted line and the detection frame shown by the solid line meet. For example, in the case of FIG. 7(a), the number of sides where the detection area and the detection frame meet is 0, so the predetermined coefficient is 1; in the case of FIG. 7(b), the number of sides where the detection area and the detection frame meet is 1, so the predetermined coefficient is 0.8; and in the case of FIG. 7(c), the number of sides where the detection area and the detection frame meet is 2, so the predetermined coefficient is 0.6.

[0052] Alternatively, the detection frame may be divided as shown in the examples of FIGS. 7(d) to 7(g), and the predetermined coefficient may be calculated as follows, depending on the length of the side of the detection frame that contacts the detection area boundary relative to the perimeter length within the detection frame. For example, the calculation may be performed as follows: predetermined coefficient = 1 - (length of the contacting side ÷ perimeter length). In this case, the predetermined coefficient is calculated as 1 for FIG. 7(d), 0.88 for FIG. 7(e), 0.63 for FIG. 7(f), and 0.5 for FIG. 7(g). The predetermined coefficient may also be determined depending on other geometric characteristics.

[0053] (Third embodiment) In this embodiment, a method is described in which the order in which multiple detection results are integrated is changed and integration processing is performed based on the reliability of the detection results. In the following description, the same reference numerals are used for components common to the first and second embodiments, and their description will be omitted.

[0054] Fig. 8(a) is a flowchart showing an example of object detection processing performed by the information processing device 100 in this embodiment, and parts common to the flowcharts shown in Fig. 3 and Fig. 5 are assigned the same reference numerals as in Fig. 3 and Fig. 5. Fig. 10 is a diagram for explaining the object detection processing according to this embodiment. In step S301, the image acquisition unit 201 acquires an input image. Fig. 10(a) shows an example of an input image 1010 in this embodiment. In this embodiment, as in the first embodiment, the input image 1010 is assumed to be an image of 1080 x 720 pixels.

[0055] Then, in step S302, the object detection unit 202 performs face detection processing to detect the face of a person to be detected in the input image, and outputs a confidence level and a class probability for each detected face. Note that, if multiple detection areas are set as in the second embodiment, steps S501 and S502 in FIG. 5 are performed instead of step S302. Note that FIG. 10 will be described assuming that steps S501 and S502 in FIG. 5 have been performed. FIG. 10(b) shows an example of the detection results obtained by performing face detection processing for each detection area set in the input image, and FIG. 10(c) shows an example of an image in which the detection results are superimposed on the input image. As shown in FIG. 10(c), two detection areas a (1011) and b (1012) are set, and four detection results A to D are obtained. In the example shown in FIG. 10(c), rectangular detection frames 1013, 1014, 1016, and 1017 corresponding to detection results A to D are superimposed on the input image 1010 and displayed on the display unit 104.

[0056] In step S810, the object detection unit 202 performs a reliability adjustment process. Details will be described later with reference to Fig. 8(b). Note that if multiple detection areas are not set as in the first embodiment, this process may be omitted. In step S820, the representative frame determination unit 204 performs processing order list creation processing, the details of which will be described later with reference to FIG. In step S900, the overlap determination unit 203, the representative frame determination unit 204, and the class determination unit 205 perform frame integration processing, the details of which will be described later with reference to FIG. In step S310, the result output unit 207 outputs the detection result data.

[0057] FIG. 8B is a flowchart showing an example of the detailed procedure of the reliability adjustment process in step S810. In step S811, the object detection unit 202 determines whether the reliability adjustment process has been performed on all detection results. If the object detection unit 202 determines that the reliability adjustment process has been performed on all detection results (YES in step S811), it ends the reliability adjustment process of Fig. 8(b). On the other hand, if the object detection unit 202 determines that there are still detection results for which the reliability adjustment process has not been performed (NO in step S811), it moves on to the next detection result and proceeds to step S812.

[0058] In step S812, the object detection unit 202 defines the positional relationship between the detection frame included in the detection result to be processed and the detection area where the detection was performed. As explained in the second embodiment using FIG. 7, this positional relationship is defined by the number of sides where the perimeter of the detection frame and the perimeter of the detection area meet, the ratio of their lengths, etc.

[0059] In step S813, the object detection unit 202 adjusts the reliability of the detection result of the processing target in accordance with the positional relationship defined in step S812. This adjustment is also as described in the second embodiment using Fig. 7. Thereafter, the process returns to step S811, and if there are any detection results remaining for the next processing target, steps S812 and S813 are repeated to perform the reliability adjustment process on all detection results.

[0060] Figure 10(d) is an example of a detection result after performing a reliability adjustment process on the example detection result in Figure 10(b), in which the reliability of detection result B, in which one side of the detection frame overlaps detection area b, has been reduced from 0.85 to 0.68.

[0061] FIG. 8C is a flowchart showing an example of the detailed procedure of the processing order list creation process in step S820. In step S821, the representative frame determination unit 204 sorts all detection results in descending order of reliability. In the case of detection results A to D shown in Fig. 10(d), the respective reliability levels are 0.80, 0.68, 0.85, and 0.75, so the sorted results are C, A, D, and B in descending order of reliability.

[0062] In step S822, the representative frame determination unit 204 lists the results sorted in step S821 and stores them as a processing order list in the storage unit 208. Fig. 10(e) shows an example of the stored processing order list. Note that, although only the correspondence between the rank and the detection result is shown as list information here, the coordinate information, reliability, and class probability of the detection frame included in the detection result can also be included in the list information.

[0063] FIG. 9 is a flowchart showing an example of the detailed procedure of the frame integration process in step S900. In step S901, the representative frame determination unit 204 determines whether or not the processing order list created in step S822 contains any detection results to be processed. If the representative frame determination unit 204 determines that the processing order list does not contain any detection results to be processed and is empty (YES in step S901), the frame integration process ends. On the other hand, if the representative frame determination unit 204 determines that the processing order list contains detection results to be processed (NO in step S901), the process proceeds to step S902.

[0064] In step S902, the representative frame determination unit 204 sets the detection frame corresponding to the processing result at the top of the processing order list as the representative frame. For example, if the processing order list at this point is the information shown in Figure 10(e), detection result C is first in the processing order, so detection frame 1016 of detection result C is set as the representative frame. Subsequent steps S903 to S909 are processes for determining and integrating detection frames to be integrated into the representative frame set here.

[0065] In step S903, the representative frame determination unit 204 sets the class probability of the representative frame to the initial value of each class index for the representative frame set in step S902. For example, with reference to Figure 10(d), the class probabilities corresponding to the detection frame 1016 of detection result C that became the representative frame are 0.55 for the glasses-wearing class and 0.45 for the non-glasses-wearing class. Therefore, the initial values ​​of the class index of the representative frame are 0.55 for the glasses-wearing class and 0.45 for the non-glasses-wearing class.

[0066] In step S904, the overlap determination unit 203 determines whether or not there are any detection results in the processing order list for which the overlap rate with the representative frame has not been calculated. If the overlap determination unit 203 determines that the overlap rate with the representative frame has been calculated for all of the detection results in the processing order list (YES in step S904), the process proceeds to step S908. On the other hand, if the overlap determination unit 203 determines that there are any detection results in the processing order list for which the overlap rate with the representative frame has not been calculated (NO in step S904), the process proceeds to step S905.

[0067] In step S905, the overlap determination unit 203 calculates the overlap rate between the representative frame and a detection frame corresponding to one of the detection results lower than the representative frame in the processing order list. One of the detection results lower than the representative frame in the processing order list may be selected from those whose overlap rate has not been calculated, starting from the highest. According to the processing order list shown in FIG. 10(e), the overlap rate between the representative frame (detection frame 1016 of detection result C) and the detection frame 1013 of detection result A is first calculated. As can be seen from FIG. 10(c), the overlap rate between these two frames is 0. Note that when multiple detection regions are set as in the example of FIG. 10, the overlap rate is calculated using both IoU and Simpson coefficient, as in the second embodiment, and when multiple detection regions are not set as in the first embodiment, the overlap rate is calculated using IoU.

[0068] In step S906, the overlap determination unit 203 determines whether the overlap rate calculated in step S905 is equal to or greater than a predetermined threshold. When multiple detection regions are set as in the example of FIG. 10, the overlap rate based on the Simpson coefficient is compared with the threshold, as in the second embodiment. When multiple detection regions are not set as in the first embodiment, the overlap rate based on the IoU is compared with the threshold. When the overlap determination unit 203 determines that the overlap rate is less than the threshold (NO in step S906), this combination is not a target for frame integration, and the process returns to step S904 to move the processing target to the next detection result in the processing order list. On the other hand, when the overlap determination unit 203 determines that the overlap rate is equal to or greater than the threshold (YES in step S906), this combination is a target for frame integration, and the process proceeds to step S907. The overlap rate between the representative frame (detection frame 1016) in FIG. 10 and the detection frame 1013 of detection result A is 0, and the determination result in step S906 is NO. Note that the overlap rate between the detection frame 1017 of the detection result D, which is next in the processing order, and the representative frame is equal to or greater than the threshold, so the determination result in step S906 is YES.

[0069] In step S907, the representative frame determination unit 204 and the class determination unit 205 perform a process of integrating the detection frames to be integrated into a representative frame. In the integration process into a representative frame, the class determination unit 205 adds a numerical value obtained by multiplying the probability of each class of the detection frames to be integrated by the overlap rate (IoU) to each class index of the representative frame. Furthermore, the representative frame determination unit 204 deletes the detection result corresponding to the processing frame to be integrated from the processing order list, and deletes the detection result itself. In the example of FIG. 10, the detection frame 1017 of detection result D is integrated into the representative frame. Therefore, a numerical value obtained by multiplying the probability of each class of detection result D by the overlap rate with the representative frame is added to each class index of the representative frame, and detection result D is deleted from the processing order list. The processing order list at this time becomes that shown in FIG. 10(f). Furthermore, information about detection result D is deleted from FIG. 10(d). When the integration process into a representative frame is completed, the process returns to step S904 to move the processing target to the detection result in the next processing order list.

[0070] Then, in the example of Figure 10, the overlap rate is calculated for the detection frame 1014 corresponding to detection result B, which is lower in the processing order list, with respect to the representative frame of detection result C, but since the overlap rate is 0, the detection frame 1014 of detection result B is not integrated into the representative frame.

[0071] As described above, the overlap rate of one representative frame with other detection frames is calculated, and when frame integration processing is completed as necessary, the process proceeds from step S904 to step S908. In step S908, class determination unit 205 determines the class with the maximum value of the class indices calculated in step S903 or S907 as the class of that representative frame. In the example of Fig. 10, the class of the representative frame of detection result C is determined to be "glasses worn."

[0072] Next, in step S909, the representative frame determination unit 204 deletes from the processing order list the detection result corresponding to the representative frame for which processing has been completed up to this point. In the example of FIG. 10, the representative frame was the detection frame 1016 of detection result C in the processing up to this point, so detection result C is deleted from the processing order list. As a result, the processing order list becomes the list shown in FIG. 10(g). Then, to proceed to processing for the next representative frame, the process returns to step S901. In the subsequent processing, the detection frame 1013 of detection result A, which is at the top of the processing order list, is set as the representative frame, frame integration processing is performed on detection result B, and the processing order list becomes the list shown in FIG. 10(h). Then, in step S908, the class of the representative frame of detection result A is determined to be "glasses not worn," and detection result A is deleted from the processing order list in step S908. As a result, in step S901, it is determined that the processing order list is empty, and the processing shown in FIG. 9 ends.

[0073] Then, in step S310 of Fig. 8, the result output unit 207 outputs the detection result data. Fig. 10(i) is an example of the detection result data. Fig. 10(a) is the input image, and when the detection results of Fig. 10(i) are output, Fig. 10(j) is an example of an image in which the detection results are superimposed on the input image. Note that in Fig. 10(j), the non-glasses-wearing class is represented by a dashed rectangle 1018, and the glasses-wearing class is represented by a long-dashed rectangle 1019.

[0074] As described above, according to this embodiment, the order in which multiple detection results are integrated is determined based on reliability, the overlap rate is always calculated on a one-to-one basis, and the frame integration process is executed each time. This simplifies the process and improves computational efficiency even when there are many frames to be integrated.

[0075] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0076] 201 Image acquisition unit, 202 Object detection unit, 203 Overlap determination unit, 204 Representative frame determination unit, 205 Class determination unit, 206 Result correction unit

Claims

1. an input means for acquiring an input image; a detection means for detecting a candidate region in which an object exists from the acquired input image and acquiring candidate attributes of the object; an overlap rate acquisition means for acquiring an overlap rate between the plurality of candidate areas when the plurality of candidate areas are present; an integration means for setting, for each combination of the candidate regions, the candidate region with the highest reliability indicating that the candidate region may contain an object as a representative region, and for deleting candidate regions whose overlap rate with the representative region is equal to or greater than a threshold; a determination means for determining an attribute of an object in the representative region based on an attribute of the object included in the candidate region, a probability of the attribute of the object, and an overlap rate with the representative region; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the determination means determines, for each combination of the candidate areas, the attribute of the object in the representative area to be the attribute of the object in the representative area that maximizes the sum of the probabilities of the attribute of the object weighted by the overlap rate with the representative area.

3. The method further includes a combination acquisition unit that acquires a combination of candidate areas whose overlapping rates acquired by the overlapping rate acquisition unit are equal to or greater than a threshold value, 3. The information processing apparatus according to claim 1, wherein the integration unit sets the representative area for each combination of candidate areas, acquired by the combination acquisition unit, whose overlap rate is equal to or greater than a threshold.

4. 3. The information processing apparatus according to claim 1, wherein the overlap rate acquisition means acquires an overlap rate between the representative area set by the integration means and one of the plurality of candidate areas.

5. The information processing device according to any one of claims 1 to 4, characterized in that the overlap rate acquisition means acquires an overlap rate which is a value obtained by dividing the area of ​​the common part of two candidate areas by the union of the areas of the two candidate areas.

6. a detection means for detecting a candidate region where an object exists from an input image and acquiring candidate attributes of the object; an overlap rate acquisition means for acquiring an overlap rate between the plurality of candidate areas when the plurality of candidate areas are present; an integration means for setting a representative region for each combination of the candidate regions based on a reliability indicating a possibility that the candidate region includes an object, and for eliminating candidate regions whose overlap rate with the representative region is equal to or greater than a threshold; a determination means for determining an attribute of an object in the representative region based on an attribute of the object included in the candidate region, a probability of the attribute of the object, and an overlap rate with the representative region; An information processing device comprising:

7. The information processing device described in Claim 6, characterized in that the determination means determines the attribute of the object in the representative area as the attribute of the object in the representative area, with the sum of the probability of the attribute of the object weighted by the overlap rate with the representative area being the largest for each combination of the candidate areas.

8. an input step of acquiring an input image; a detection step of detecting a candidate region in which an object exists from the acquired input image and acquiring candidate attributes of the object; an overlap rate acquisition step, when there are a plurality of candidate regions, of acquiring an overlap rate between the plurality of candidate regions; an integration step of setting, for each combination of the candidate regions, the candidate region with the highest reliability indicating that the candidate region may contain an object as a representative region, and deleting candidate regions whose overlap rate with the representative region is equal to or greater than a threshold; a determination step of determining attributes of an object in the representative region based on attributes of the object included in the candidate region, the probability of the attributes of the object, and an overlap rate with the representative region; 1. A method for controlling an information processing device, comprising:

9. The control method for an information processing device described in Claim 8, characterized in that the determination process determines the attribute of the object in the representative area as the attribute of the object in the representative area, with the probability of the attribute of the object being the largest when the sum is weighted by the overlap rate with the representative area for each combination of the candidate areas.

10. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 7.

Images