Information processing device, information processing method, and program

Abstract

In object detection, where a target subject is detected from an image, the detection accuracy is improved. [Solution] The information processing device includes: a first detection means for estimating a detection region and a detection score related to a subject to be detected from an input image; a first candidate generation means for generating a first detection candidate related to the detection of a subject to be detected from the first image using the first detection means; a region calculation means for calculating a cropping region based on the first detection candidate if there is no detection candidate among the first detection candidates whose detection score is equal to or greater than a first threshold; an image generation means for generating a second image by cropping from the first image based on the cropping region; and a second candidate generation means for generating a second detection candidate related to the detection of a subject to be detected from the second image using the first detection means.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Object detection for detecting a region of a specific object from an image is performed. For example, face detection for detecting a region of a person's face from an image having a person as a subject is performed. As object detection technology, in recent years, technology for learning using a neural network has been developed. Non-Patent Document 1 discloses a method for detecting an object by training a neural network to output a keypoint indicating the position of an object to be detected as a heatmap.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When performing object detection, it is common to output the location and size of the region containing the object to be detected on the input image, as well as a detection score. The detection score is a numerical value that represents the likelihood of detection. A neural network trained to detect a specific object from an image will produce a high output for image features that are likely to be detected and a low output for image features that are not likely to be detected. The detection score is calculated based on, for example, the values ​​of a heatmap output from the neural network. A low detection score indicates a low likelihood of detection, i.e., a high probability of false positive. Therefore, if the detection score is below a predetermined threshold, the object is treated as not being detected (not detected).

[0005] In images where the size of the area to be detected, i.e., the image size of the subject, is small, the detection score tends to be low because the image features that are likely to be detected are unclear. Therefore, when the image size of the subject to be detected is small, the detection score often falls below a predetermined threshold, resulting in failure to detect the object. In addition, the larger the input image size, the more computational load is required for object detection processing. To reduce the computational load, the input image size is reduced before performing object detection processing. In this case, the image size of the subject to be detected becomes even smaller, making it even more likely to fail to detect the object.

[0006] This invention has been made in view of these circumstances, and aims to improve the detection accuracy in object detection, which detects a subject to be detected from an image. [Means for solving the problem]

[0007] The information processing device according to the present invention is characterized by comprising: a first detection means for estimating a detection region and a detection score relating to a subject to be detected for an input image; a first candidate generation means for generating a first detection candidate relating to the detection of the subject to be detected for the first image using the first detection means; a region calculation means for calculating a cropped region based on the first detection candidate if there is no detection candidate among the first detection candidates whose detection score is equal to or greater than a first threshold; an image generation means for generating a second image by cropping from the first image based on the cropped region; and a second candidate generation means for generating a second detection candidate relating to the detection of the subject to be detected for the second image using the first detection means. [Effects of the Invention]

[0008] According to the present invention, in object detection where a subject to be detected is detected from an image, the detection accuracy can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of the information processing device in Embodiment 1. [Figure 2] This is a diagram illustrating the neural network in Embodiment 1. [Figure 3] This flowchart shows an example of processing performed by the information processing device in Embodiment 1. [Figure 4] This flowchart shows an example of the process for calculating the cut-out area in Embodiment 1. [Figure 5] This flowchart shows an example of the process for calculating the cut-out area in Embodiment 1. [Figure 6] This flowchart shows an example of the process for calculating the cut-out area in Embodiment 1. [Figure 7] This figure shows an example of the configuration of the information processing device in Embodiment 2. [Figure 8] This is a diagram illustrating the neural network in Embodiment 2. [Figure 9]This flowchart shows an example of the process for calculating the cut-out area in Embodiment 2. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. Furthermore, identical or similar components in the drawings are denoted by the same reference numerals, and redundant explanations are omitted. The information processing device in each embodiment described below performs object detection to detect a target subject from an input image. While an example where the target subject is a person's face is described below, the method is not limited to this, and the target subject can be any subject.

[0011] <Embodiment 1> Figure 1 shows an example configuration of an information processing device in Embodiment 1. The information processing device in this embodiment includes a CPU 101, a first memory 103, a second memory 104, an input unit 105, a display unit 106, and a communication unit 107. The CPU 101, the first memory 103, the second memory 104, the input unit 105, the display unit 106, and the communication unit 107 are connected to each other via a bus 102.

[0012] The CPU (Central Processing Unit) 101 controls the entire information processing device. The first memory 103 and the second memory 104 store control programs and various data for the information processing device in this embodiment to execute various processes. The first memory 103 and the second memory 104 are implemented by, for example, memory or auxiliary storage devices. The input unit 105 is implemented by an input device such as a keyboard or touch panel and accepts input from the user. The display unit 106 is implemented by a display device such as a liquid crystal display and displays various information such as processing results to the user. The communication unit 107 transmits and receives data through communication with other devices.

[0013] In the example shown in Figure 1, the first memory 103 mainly stores the control program, and the second memory 104 mainly stores various data. Note that the control program and various data stored in the first memory 103 and the second memory 104 are not limited to the example shown in Figure 1.

[0014] The second memory 104 stores a neural network 120, which is a model trained to detect the target subject. In this embodiment, as shown in Figure 2, the neural network 120 is trained to take an input image 201 as input, and when the input image 201 is input to the neural network 202, an inference map 203 is obtained. The neural network 120 is trained to output a map in which, for example, the values ​​are high in areas of the image where the target subject exists, and low in other areas. For the sake of simplicity in illustration, the inference map 203 is shown as a binary image with high values ​​in black and low values ​​in white, but it is inferred that the map value is higher towards the center of the target and lower as you move away from the center of the target. Generally, the neural network 120 is configured so that the inference map 203 is output at a size smaller than the input image 201, but for the sake of simplicity in explanation, it will be described as a configuration in which the size of the inference map 203 is the same as the input image 201.

[0015] The control program stored in the first memory 103 includes at least a program for executing the processing described later in this embodiment. When the CPU 101 executes the control program, the image acquisition unit 110, the first detection unit 111, the first candidate generation unit 112, the determination unit 113, the cropping region calculation unit 114, the cropped image generation unit 115, the second candidate generation unit 116, and the result determination unit 117 function. Note that each of these units may be implemented in software using the CPU 101, or some may be implemented in hardware such as electronic circuits.

[0016] In this embodiment, the first detection unit 111 performs an object detection process of detecting a predetermined object (a subject to be detected) from an input image using a pre-trained neural network 120. For example, the first detection unit 111 uses the neural network 120 to detect the region of a person's face from an image. As shown in FIG. 2, the neural network 120 outputs an inference map for the input image. The detection result by the first detection unit 111 is output as information indicating the detection region (the region where the subject to be detected exists) and the detection score in the input image. For example, the first detection unit 111 derives a bounding box that encloses a region where the map value in the inference map output by the neural network 120 is higher than a predetermined threshold, and uses it as the detection region. The bounding box can be represented by the detection position and the detection size, for example, the center coordinates and the width and height information of a rectangular region. Note that, not limited to the center coordinates of the rectangular region, the information regarding the bounding box may be represented using the vertex coordinates of the rectangular region. Regarding the detection score, for example, the highest map value within the bounding box region may be used as the detection score. In this way, the first detection unit 111 estimates and outputs the detection region and the detection score regarding the subject to be detected for the input image. The first detection unit 111 outputs a list of detection results with a set of the detection region and the detection score as one detection result. There may be a case where there is no detection result for one input image, or there may be one or more. Therefore, the first detection unit 111 outputs a list of zero or more detection results.

[0017] FIG. 3 is a flowchart showing an example of the processing of the information processing apparatus in Embodiment 1. Referring to FIG. 3, the object detection process of detecting a subject to be detected from the input image will be described.

[0018] In step S301, the image acquisition unit 110 acquires a detection target image for performing object detection processing. Based on the acquired detection target image, the image acquisition unit 110 generates an input image of a predetermined size for input to the neural network 120 and stores it in the memory. The detection target image may be acquired by reading an image specified by the user via the input unit 105, or may be acquired by receiving an image from an external imaging device via the communication unit 107. In this embodiment, it is assumed that the detection target image acquired by the image acquisition unit 110 is an image with a higher resolution than the input image 122. The image acquisition unit 110 stores the acquired high-resolution detection target image in the second memory 104 as the high-resolution image 121. Thereafter, the image acquisition unit 110 resizes the high-resolution image 121 to an image of a predetermined input image size (reducing the resolution) and stores it in the second memory 104 as the input image 122. The input image 122 is an example of the first image. The image size of the input image 122 is an image size that can be input to the neural network 120, and is assumed to be pre-determined during the learning of the neural network 120 and pre-stored in the second memory 104 as the input image size 123. Although it is assumed that the detection target image acquired by the image acquisition unit 110 is an image with a higher resolution than the input image 122, it is not limited thereto. For example, the image acquisition unit 110 may acquire an image having a size equal to the input image size 123 as the detection target image. In that case, the above-described image resizing process is unnecessary. Also, for example, the image acquisition unit 110 may acquire an image having a size smaller than the input image size 123 as the detection target image, and in this case, a resizing process for enlarging the image size may be performed and stored as the input image 122.

[0019] In step S302, the first candidate generation unit 112 performs object detection on the input image 122 using the neural network 120 to generate a first detection candidate, which is then stored in the second memory 104 as the first detection candidate 124. The first candidate generation unit 112 inputs the input image 122 to the first detection unit 111 and stores the obtained detection result (a pair of detection region and detection score) in the second memory 104 as the first detection candidate 124. As described above, the second memory 104 stores a list of zero or more detection results as the first detection candidate 124.

[0020] In step S303, the determination unit 113 determines whether there is a detection result (detection candidate) among the first detection candidates 124 stored in step S302 whose detection score is equal to or greater than a predetermined detection score threshold. The detection score threshold is pre-stored as the detection score threshold 125 in the second memory 104. The detection score threshold 125 is a threshold that determines whether or not the detection score included in the detection result should be considered as detected, and can be adjusted according to the degree to which the user is willing to tolerate false detections or undetected objects. The detection score threshold 125 is an example of the first threshold. The lower the value of the detection score threshold 125, the more likely false detections are but the less likely undetected objects are to occur, and the higher the value, the less likely false detections are but the more likely undetected objects are to occur. If the determination unit 113 determines that there is no detection result (detection candidate) among the first detection candidates 124 whose detection score is equal to or greater than the detection score threshold 125 (NO), the process proceeds to step S304. On the other hand, if the determination unit 113 determines that there is a detection result (detection candidate) in the first detection candidate 124 whose detection score is equal to or greater than the detection score threshold 125 (YES), the process proceeds to step S309. In step S309, the result determination unit 117 stores the detection result (detection candidate) in the first detection candidate 124 whose detection score is equal to or greater than the detection score threshold 125 as the detection result 129 in the second memory 104, and then the process proceeds to step S308.

[0021] In step S304, the cutout area calculation unit 114 calculates the cutout area based on the first detected candidate 124 and stores it in the second memory 104 as the cutout area 126. Details of the cutout area calculation process in step S304 will be described later.

[0022] In step S305, the cropped image generation unit 115 generates a cropped image based on the cropping region 126 and stores it in the second memory 104 as the cropped image 127. The cropped image 127 is an example of the second image. In this embodiment, the cropped image generation unit 115 generates the cropped image 127 from the high-resolution image 121 based on the cropping region 126, but is not limited to this. For example, if a high-resolution image has not been acquired as the image to be detected, as described in the above explanation regarding the image acquisition unit 110, the cropped image generation unit 115 may generate the cropped image 127 from the input image 122. Although not shown in Figure 3, if the cropping region 126 is empty when the processing of step S305 begins, the process may proceed directly to step S308 and terminate the processing of this flowchart as there is no detection result.

[0023] The generation of the cropped image 127 by the cropped image generation unit 115 will be described below. The cropped image generation unit 115 first calculates the position corresponding to the center of the cropped area 126 in the high-resolution image 121. This can be calculated, for example, by recording the conversion magnification (reduction magnification) used when the image size conversion was performed in the image acquisition unit 110 in the second memory 104, and then converting the center position of the cropped area 126 using that conversion magnification. Next, the cropped image generation unit 115 calculates a rectangular area centered on the center of the cropped area 126 in the high-resolution image 121, with a size equal to the input image size 123, and crops a partial image from the high-resolution image 121 according to the calculated rectangular area. Then, the cropped image generation unit 115 stores the partial image cropped from the high-resolution image 121 as the cropped image 127 in the second memory 104. Note that when cropping a partial image from the high-resolution image 121, if the rectangular area extends beyond the high-resolution image 121, the pixel values ​​of the extended area may be filled with 0. Alternatively, the rectangular area may be shifted so that the cropping area 126 is within the rectangle, but the rectangular area does not extend beyond the high-resolution image 121. The cropped image 127 generated by the cropped image generation unit 115 in this way will be an image that is like an enlarged version of the portion of the input image 122 corresponding to the cropping area 126. Since the image is cropped from the high-resolution image 121 and not enlarged by pixel value interpolation, an image of the cropping area 126 can be obtained without image degradation. However, as described in the explanation of the image acquisition unit 110, there are cases where an image of the same size as the input image size 123 is acquired as the image to be detected, in which case an image of the cropping area 126 may be obtained by image enlargement using pixel value interpolation.

[0024] In step S306, the second candidate generation unit 116 performs object detection on the cropped image 127 using the neural network 120 to generate a second detection candidate, which is then stored in the second memory 104 as the second detection candidate 128. The second candidate generation unit 116 inputs the cropped image 127 to the first detection unit 111 and stores the obtained detection result (a pair of detection region and detection score) in the second memory 104 as the second detection candidate 128. Similar to the processing in step S302, the second memory 104 stores a list of zero or more detection results as the second detection candidate 128. However, in step S306, the detection region information included in the detection result is converted from the image coordinates of the cropped image 127 to the image coordinate system of the input image 122 based on the information of the cropped region 126 and then stored.

[0025] In step S307, the result determination unit 117 determines the final detection result based on the detection results for the subject to be detected obtained in the processing up to this point, and stores it in the second memory 104 as the detection result 129. Details of the detection result determination process in step S307 will be described later.

[0026] In step S308, the result determination unit 117 outputs the detection result based on the detection result 129. For example, the result determination unit 117 may superimpose a rectangular frame or the like indicating the detection area, which is the detection result, onto the input image 122 and display it on the display unit 106. If no detection results are stored in the detection result 129, the result determination unit 117 may display on the display unit 106 or the like that no objects to be detected were detected. Note that the use of the detection result is not limited to displaying the detection result. It is also conceivable to perform other processing using the detection result. For example, the following use is conceivable: The information processing device receives an image acquired by an image sensor in an external imaging device as the input image 122 via the communication unit 107. The information processing device then performs object detection processing on the input image 122 and transmits the object detection result 129 to the external imaging device via the communication unit 107. The external imaging device performs autofocus control to focus the imaging device based on the face detection area indicated by the received detection result 129. After performing the process in step S308, the process in this flowchart is terminated.

[0027] In the above explanation, in step S303, the process proceeds to step S304 only if the determination unit 113 determines that there are no detection results (detection candidates) among the first detection candidates 124 whose detection score is equal to or greater than the detection score threshold 125 (NO). However, the process is not limited to this, and the process may always proceed to step S304 regardless of whether or not there are detection results (detection candidates) among the first detection candidates 124 whose detection score is equal to or greater than the detection score threshold 125. If the process always proceeds to step S304, in step S307, the result determination unit 117 may determine the final detection result from both the first detection candidate 124 and the second detection candidate 128. By doing so, it becomes possible to generate the second detection candidate 128 regardless of the result of the first detection candidate 124, and to determine the candidate with the higher detection score and the higher probability of correct detection from both detection candidates as the final detection result.

[0028] Next, the process for calculating the cutout region in step S304 will be explained. Several examples of the process for calculating the cutout region are possible, as described below. In any of the examples described below, if there are no detection results among the first detection candidates 124, the cutout region calculation unit 114 may clear the cutout region 126 and terminate the process.

[0029] For example, the extraction region calculation unit 114 calculates an extraction region 126 based on the detection result (detection candidate) among the first detection candidates 124 whose detection score is equal to or greater than a predetermined detection candidate score threshold, and which has the highest detection score. An example of this extraction region calculation process will be explained with reference to Figure 4. The detection candidate score threshold is set in advance as the detection candidate score threshold 130 in the second memory 104. Since the extraction region calculation process is executed when it is determined that there are no detection results (detection candidates) among the first detection candidates 124 whose detection score is equal to or greater than the detection score threshold 125, the detection candidate score threshold 130 is set to a value lower than the detection score threshold 125. The detection candidate score threshold 130 is an example of a second threshold.

[0030] Figure 4 is a flowchart showing an example of the process for calculating the extraction area. In step S401, the extraction region calculation unit 114 selects a detection result (detection candidate) from the first detection candidates 124 whose detection score is equal to or greater than the detection candidate score threshold 130. If there are no detection results (detection candidates) among the first detection candidates 124 whose detection score is equal to or greater than the detection candidate score threshold 130, the process of this flowchart may be terminated. In step S402, the extraction region calculation unit 114 selects one detection result (detection candidate) with the highest detection score from among the detection results (detection candidates) selected in step S401, and stores it in the second memory 104 as the extraction criterion candidate 132. In step S403, the cropping area calculation unit 114 calculates the cropping area based on the information regarding the detected area of ​​the cropping criterion candidate 132 and stores it in the second memory 104 as the cropping area 126. The cropping area calculation unit 114 may store the detected area of ​​the cropping criterion candidate 132 as is as the cropping area 126, or it may store an area that has been enlarged, reduced, etc., according to separately defined rules as the cropping area 126.

[0031] In this way, if there are no detection results (detection candidates) among the first detection candidates 124 with a detection score equal to or greater than the detection candidate score threshold 130, it is determined that there is no target to be detected in the input image 122, and no further processing is performed. This avoids the execution of excessive processing and reduces the possibility of false detections by cropping the image and performing re-detection. Alternatively, by setting the detection candidate score threshold 130 to 0, one detection result (detection candidate) with the highest detection score among the first detection candidates 124 will always be selected and stored as the cropping criterion candidate 132.

[0032] Another example is when the cropping area calculation unit 114 selects a cropping criterion candidate from among the first detection candidates 124, on the condition that the detection size of the detection area is within a predetermined detection candidate size range, and calculates the cropping area 126. An example of this cropping area calculation process will be explained with reference to Figure 5. The detection candidate size range is set in advance as the detection candidate size range 131 in the second memory 104. In object detection processing, the detection score may be low and the object may not be detected if the image size of the subject is small, but if the detection area size is large in the first detection candidate 124, it is unlikely that the low detection score is due to the image size of the subject. If the detection area size is large but the detection score is low, it is highly likely that the image does not look like a target for detection.

[0033] Figure 5 is a flowchart showing an example of the process for calculating the extraction area. In step S501, the extraction region calculation unit 114 selects a detection result (detection candidate) from the first detection candidates 124 in which the detection size of the detection region is within the detection candidate size range 131. If there is no detection result (detection candidate) among the first detection candidates 124 in which the detection size of the detection region is within the detection candidate size range 131, the process of this flowchart may be terminated. In step S502, the extraction region calculation unit 114 selects a detection result (detection candidate) from the detection results (detection candidates) selected in step S501 whose detection score is 130 or higher than the detection candidate score threshold. If there are no detection results (detection candidates) among the detection results (detection candidates) selected in step S501 whose detection score is 130 or higher than the detection candidate score threshold, the processing of this flowchart may be terminated. In step S503, the extraction region calculation unit 114 selects one detection result (detection candidate) with the highest detection score from among the detection results (detection candidates) selected in step S502, and stores it in the second memory 104 as the extraction criterion candidate 132. In step S504, the cropping area calculation unit 114 calculates the cropping area based on the information regarding the detection area of ​​the cropping criterion candidate 132 and stores it in the second memory 104 as the cropping area 126. The cropping area calculation unit 114 may store the detection area of ​​the cropping criterion candidate 132 as is as the cropping area 126, or it may store an area that has been enlarged, reduced, or moved according to a separately defined rule based on the information regarding the detection area as the cropping area 126. For example, if the detection target is a person's eye, a rule may be defined based on the detection position and size of the eye so that the entire face of the person is included in the area. Alternatively, the cropping area may be calculated using a rule such that the ratio of the size of the detection area to the size after cropping is a predetermined value. If the accuracy of the detection process is affected by the size ratio between the input image and the detection target, the detection accuracy can be expected to improve by calculating the cropping area to have an appropriate size ratio.

[0034] This approach prioritizes the detection of candidates whose detection scores were not sufficiently high due to the small image size of the subject, and allows for the calculation of the cropping area based on these candidates. It also avoids unnecessary processing of candidates that received low detection scores simply because the image content did not resemble the target of detection. Alternatively, by setting the detection candidate score threshold 130 to 0, one detection result (detection candidate) with the highest detection score among the detection results (detection candidates) selected in step S501 will always be selected and stored as the cropping criterion candidate 132.

[0035] Another example is that the cropping region calculation unit 114 performs region division based on the detected region of the cropping criterion candidate 132 in the example shown in Figures 4 and 5, and calculates the cropping region 126 based on the result of the region division. An example of this cropping region calculation process will be explained with reference to Figure 6. Region division here refers to a method of dividing an image into foreground and background, and is sometimes called blob detection. For example, methods using graph cuts are known. A graph cut is one method of dividing an image into a foreground containing a region given as a seed region and the rest of the background.

[0036] Figure 6 is a flowchart showing an example of the process for calculating the extraction area. In step S601, the cutting area calculation unit 114 calculates the detection area of ​​the cutting reference candidate 132 using the method shown in the example in Figure 4 or Figure 5 described above. In step S602, the cropping area calculation unit 114 performs region division based on the detected region of the cropping criterion candidate 132 calculated in step S601. For example, the cropping area calculation unit 114 sets the detected region of the cropping criterion candidate 132 as the seed region for graph cutting and performs graph cutting to divide the image into a foreground and the rest of the background. In step S603, the cropping area calculation unit 114 calculates a rectangular area that includes the area designated as the foreground in the area division performed in step S602, calculates a cropping area based on that, and stores it in the second memory 104 as a cropping area 126.

[0037] This approach makes it possible to crop a larger area that includes the region defined as the detection target. For example, if the neural network 120 is trained to recognize the central part of a person's face as the target subject, the detection region of the detection result will only be the central part of the face. However, by segmenting the region based on the detection region, it becomes possible to crop an image that includes the entire head or the entire body. This improves the accuracy of the detection process when generating a second detection candidate in a later process. This is because the neural network 120, trained to detect the target subject, learns not only the sub-region containing the target but also the surrounding image, and therefore may be able to detect with greater accuracy on an image that includes the surrounding area than on an image containing only the target. For example, object detection may be more accurate when performed on an image showing the entire head, its surroundings, or the entire body, than when performing object detection on an image cropped only from the central part of the face. Furthermore, compared to cropping around the detection region of the cropping criterion candidate 132, it becomes possible to crop by setting the region of the object containing that region in a more balanced manner. For example, instead of cropping around the face, it becomes possible to crop an image that crops the entire body around the center of the body, resulting in more accurate detection. This effect is more pronounced when the model is trained to detect only a small portion of an object. Note that while graph cuts were used as an example of region segmentation in this example, the method of region segmentation is not limited to this.

[0038] Next, we will explain the process for determining the detection result in step S307. Several examples of the detection result determination process are possible, as described below. Note that the detection result determination process described below is just one example, and the method for determining the final detection result is not limited to these. For example, the result determination unit 117 selects the detection result (detection candidate) with the highest detection score among the second detection candidates 128 and determines it as the final detection result, storing it as the detection result 129. In this example, the result determination unit 117 determines the detection result 129 based solely on the detection score included in the detection result.

[0039] Another example is when the result determination unit 117 selects a detection result (detection candidate) from the second detection candidate 128 in which the detection size of the detection region included in the detection result is within a predetermined range. Subsequently, the result determination unit 117 determines the detection result (detection candidate) with the highest detection score among the selected detection results (detection candidates) as the final detection result and stores it as the detection result 129. In object detection processing, the detection score may be low and the object may not be detected if the image size of the subject is small. However, if the detection size of the detection region is large in the second detection candidate 128, it is unlikely that the detection score in the first detection candidate 124 was low due to the image size of the subject. By selecting a detection result (detection candidate) from the second detection candidate 128 in which the detection size of the detection region is within a predetermined range, it is possible to prevent a candidate that is not likely to have had a low detection score due to the image size of the subject from becoming the final detection result. This predetermined range for the detection size may be the same range as the detection candidate size range 131 in the second memory 104, or a different range value may be set for this process.

[0040] Another example is that the result determination unit 117 compares the detection region of the cropping criterion candidate 132 selected by the cropping region calculation unit 114 from the first detection candidate 124 with the detection region of the second detection candidate 128 to determine the final detection result, which is then stored as the detection result 129. In this embodiment, the cropping criterion candidate 132 is determined based on the detection candidate whose detection score was less than the detection score threshold 125 in the first detection candidate 124, and the second detection candidate 128 is generated from the cropped image 127 based on the cropping criterion candidate 132. Therefore, if the detection is performed correctly, the detection result corresponding to the detection region of the cropping criterion candidate 132 will be in the second detection candidate 128. Any other detection results (detection candidates) in the second detection candidate 128 are newly detected results obtained by performing object detection processing on the cropped image 127 again, and there is a possibility that new false detections have occurred. In this example, the intention is to avoid such false detections. The result determination unit 117, in the second detection candidate 128, determines the detection result (detection candidate) with the highest detection score among the detection results (detection candidates) whose position and size of the detection area match that of the cropping criterion candidate 132 as the final detection result, and stores it as the detection result 129. The matching of the position and size of the detection area can be determined based on the degree of overlap between the two detection areas. If there is no overlap, a detection result with similar values ​​for the position and size of the detection area may be selected.

[0041] In the explanation above, an example was described where there is only one cropping criterion candidate 132, but it is also possible to select multiple cropping criterion candidates. For example, from the first detection candidates 124, multiple detection results (detection candidates) whose detection score is 130 or higher and whose detection score is high may be prioritized as cropping criterion candidates. Alternatively, from the first detection candidates 124, multiple detection results (detection candidates) whose detection region size is within the detection candidate size range 131, whose detection score is 130 or higher and whose detection score is high may be prioritized as cropping criterion candidates. Furthermore, the process of generating cropped images and the process of generating second detection candidates, which are executed later, may be performed appropriately for each of the multiple cropping criterion candidates.

[0042] In this embodiment, if an object is not detected in the input image 122, a cropped image 127 is generated based on the detection result of object detection in the input image 122, and object detection is performed on the cropped image 127. In this way, objects that would have been undetected conventionally due to a low detection score caused by the small size of the subject image can now be detected. Furthermore, by cropping the image based on the detection result of undetected objects, the position on the image where the target subject is likely to exist can be efficiently identified, and object detection can be performed, thereby improving detection performance. Thus, according to this embodiment, the detection accuracy in object detection from images can be improved. Furthermore, if the final detection result is determined based on both the first and second detection candidates, which were not detected due to low detection scores, it becomes less likely to cause false positives resulting from image cropping and subsequent detection processing.

[0043] <Embodiment 2> Figure 7 shows a diagram of the information processing device in Embodiment 2. The configuration of the information processing device in this embodiment is generally the same as that of Embodiment 1 described above, but the first memory 103 further includes a second detection unit 118 and a third detection unit 119. In addition, the configuration of the neural network 120 and the processing for calculating the extracted region in Embodiment 2 differ from those of Embodiment 1 described above. These points will be explained below.

[0044] The configuration of the neural network 120 in Embodiment 2, and the second detection unit 118 and third detection unit 119 shown in Figure 7 will be described below. Figure 8 illustrates the configuration of the neural network 120 in Embodiment 2. The neural network 120 in Embodiment 2 is configured to output multiple inference maps for a single input image and is trained to output separate inference maps for different detection targets. In the neural network 120, a portion of the network is shared, and the network is configured to branch off from a certain point. Such a neural network is called a multitasking neural network.

[0045] In Figure 8, 801 shows an example of an input image. In this example, input image 801 is an image of a person and a tree. 802 is a multitasking neural network, and 803-805 show examples of the output inference maps.

[0046] The multitask neural network 802 is trained to output a first inference map 803 in which the map value is high in the region of a person's face. The first inference map 803 illustrates an example in which it responds to the region of a person's face 806 and the region of tree bark 807, which is not a person's face. The hatching of regions 806 and 807 indicates that the map values ​​of these regions are lower than the map value of region 808 on the second inference map 804, which will be described later. In Embodiment 2, the first detection unit 111 performs face detection based on this first inference map 803. The processing content of the first detection unit 111 in Embodiment 2 is the same as that of the first detection unit 111 in Embodiment 1.

[0047] Furthermore, the multitasking neural network 802 is trained to output a second inference map 804 that has a high map value in the tree region. In this example of the second inference map 804, a high map value is output in the tree region 808. The second detection unit 118 performs the same processing as the first detection unit 111 does to detect a person's face, and detects trees based on this second inference map 804. In this embodiment, the second detection unit 118 is described as detecting trees, but it is not limited to this and may detect other subjects to be detected. For example, it may detect animals such as dogs and cats, or automobiles.

[0048] Furthermore, the multitasking neural network 802 is trained to output a third inference map 805 that has high map values ​​in areas that appear to be objects, without specifying the category of the subject to be detected. In this example of the third inference map 805, high map values ​​are output for the tree region 808 and the entire person region 809. The third detection unit 119 performs the same processing as the first detection unit 111 when detecting a person's face, and detects an arbitrary object based on this third inference map 805.

[0049] Thus, the second detection unit 118 and the third detection unit 119 perform object detection based on the inference maps output by the neural network 120. Note that the processing in the second detection unit 118 and the third detection unit 119 is the same as that in the first detection unit 111, so a detailed explanation is omitted. Each inference map output by the multitasking neural network 802 is learned for a different purpose, and each purpose is called a task. In this example, these are the human face detection task, the tree detection task, and the arbitrary object detection task.

[0050] Next, the process for calculating the cropped area in Embodiment 2 will be described. Here, we will show an example where we attempt to detect a human face as the detection target, similar to Embodiment 1, and explain the case where the detection score is not high due to the small size of the subject image of the human face, and a cropped image is generated and the object detection process is performed again.

[0051] Figure 9 is a flowchart showing an example of the process for calculating the cut-out area in Embodiment 2. In step S901, the cropping region calculation unit 114 compares the first inference map 803 of the human face detection task with the second inference map 804 of another task that has been trained to detect specific objects other than the arbitrary object detection task. By comparing the first inference map 803 of the human face detection task with the second inference map 804 of the other task, the cropping region calculation unit 114 calculates regions in the input image 122 that have a high probability of containing the subject that the other task is trying to detect. This can be done by calculating regions where the value of the second inference map 804 is higher than the value of the first inference map 803. In the example shown in Figure 8, regions where the value of the second inference map 804 of the tree detection task is higher than the value of the first inference map 803 of the human face detection task are calculated. Therefore, in this example, the tree region 808 is calculated.

[0052] In step S902, the cropping region calculation unit 114 removes the detection result (detection candidate) from the first detection candidate 124 for the human face detection task that corresponds to the region calculated in step S901. In the example shown in Figure 8, the detection result (detection candidate) that corresponds to region 807 in the first inference map 803 of the human face detection task is excluded.

[0053] In step S903, the extraction region calculation unit 114 selects the detection result (detection candidate) with the highest detection score from among the detection results (detection candidates) that were not excluded in step S902, and stores it in the second memory 104 as the extraction criterion candidate 132. In the example shown in Figure 8, for example, the detection result (detection candidate) corresponding to region 806 is selected and stored as the extraction criterion candidate 132.

[0054] In step S904, the cropping region calculation unit 114 obtains an object region from the third inference map 805 of the arbitrary object detection task based on the detection position of the cropping criterion candidate 132. This can be done by calculating a bounding box that includes the detection position of the cropping criterion candidate 132 and encompasses an area where the map value of the third inference map 805 of the arbitrary object detection task is greater than or equal to a predetermined value. In this case, the bounding box may be calculated considering the detection size of the cropping criterion candidate 132. In the example shown in Figure 8, a bounding box that encompasses region 809 is obtained as the object region. Then, the cropping region calculation unit 114 calculates a cropping region based on the obtained object region and stores it in the second memory 104 as the cropping region 126.

[0055] In this embodiment, we have explained an example where the detection score for human face detection is low and human faces are not detected due to the small size of the subject image. However, if we focus on tree detection, there may also be cases where the detection score for tree detection is low and trees are not detected. In such cases, the same processing can be performed by swapping the face detection task, which was given as an example of the task to focus on in the explanation above, with the tree detection task, which was given as an example of other tasks.

[0056] In this way, the extracted region can be calculated based on the results of multiple detection tasks, preventing the calculation of an extracted region that is inappropriate for the task of interest. Furthermore, compared to the extraction region calculation process using region segmentation described in Embodiment 1, Embodiment 2 allows a region that appears to be any object to be obtained from a single neural network, thus eliminating the need for region segmentation and reducing processing load.

[0057] (Other embodiments of the present invention) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0058] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0059] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) A first detection means that estimates the detection region and detection score related to the subject to be detected from the input image, A first candidate generation means generates a first detection candidate for the detection of the subject to be detected using the first detection means for the first image, If there are no detection candidates among the first detection candidates whose detection score is equal to or greater than the first threshold, the region calculation means calculates a region to be cut out based on the first detection candidates, Image generation means for generating a second image by cutting out the first image based on the aforementioned cutout region, An information processing device characterized by having a second candidate generation means for generating a second detection candidate for the detection of the subject to be detected using the first detection means with respect to the second image. (Configuration 2) The information processing apparatus according to configuration 1, further comprising determination means for determining whether or not there is a detection candidate among the first detection candidates whose detection score is equal to or greater than the first threshold. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the image generation means generates a second image of the same size as the first image based on an image extracted from the first image. (Composition 4) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the region calculation means prioritizes selecting from the first detection candidates the detection candidate whose detection score is equal to or greater than the second threshold and whose detection score is high as the extraction criterion candidate, and calculates the extraction region based on the detection region of the extraction criterion candidate. (Composition 5) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the region calculation means prioritizes selecting as extraction criterion candidates detection candidates from among the first detection candidates, the size of the detection region being within a predetermined range, the detection score being equal to or greater than a second threshold, and the extraction region being calculated based on the detection region of the extraction criterion candidates. (Composition 6) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the region calculation means calculates the cut-out region based on the detection region of the detection candidate with the highest detection score among the first detection candidates. (Composition 7) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the region calculation means calculates the cut-out region based on the detection region of the detection candidate among the first detection candidates, wherein the size of the detection region is within a predetermined range and the detection score is the largest. (Composition 8) An information processing device according to any one of configurations 1 to 7, characterized by having a result determination means for determining a detection result for detecting the subject to be detected based on the first detection candidate and the second detection candidate. (Composition 9) The information processing apparatus according to configuration 8, characterized in that the result determination means determines the detection candidate corresponding to the first detection candidate from among the second detection candidates as the detection result. (Composition 10) The information processing apparatus according to configuration 8, characterized in that the result determination means determines the second detection candidate as the detection result. (Composition 11) The information processing apparatus according to any one of configurations 8 to 10, characterized in that the result determination means determines the detection candidate among the second detection candidates, in which the size of the detection area is within a predetermined range, as the detection result. (Composition 12) An information processing apparatus according to any one of configurations 1 to 11, characterized by having an image acquisition means that acquires an image with a higher resolution than the first image, and reduces the resolution of the acquired high-resolution image to obtain the first image. (Composition 13) The information processing apparatus according to configuration 12, characterized in that the image generation means generates the second image by cutting out the image from the high-resolution image acquired by the image acquisition means based on the cut-out region. (Composition 14) A second detection means that has been trained to detect a subject different from the subject to be detected from the input image, It has a third detection means that is trained to detect any subject from an input image, The information processing apparatus according to any one of configurations 1 to 13, characterized in that the region calculation means calculates the cropped region based on the detection results of the first detection means, the second detection means, and the third detection means on the first image. (Composition 15) The information processing apparatus according to configuration 14, characterized in that the region calculation means does not use the detection region detected by the second detection means as the cut-out region. (Method 1) An information processing method performed by an information processing device, A first detection step estimates the detection area and detection score for the subject to be detected based on the input image. A first candidate generation step is performed on the first image to generate a first detection candidate for the detection of the subject to be detected, If there are no detection candidates among the first detection candidates whose detection score is equal to or greater than the first threshold, the region calculation step calculates a region to be cut out based on the first detection candidates. An image generation step of generating a second image by cutting out the first image based on the aforementioned cutout region, An information processing method characterized by comprising: a second candidate generation step of performing the first detection step on the second image to generate a second detection candidate for the detection of the subject to be detected. (Program 1) A first detection step involves estimating the detection region and detection score for the subject to be detected based on the input image. A first candidate generation step involves performing the processing of the first detection step on the first image to generate a first detection candidate for the detection of the subject to be detected, If there are no detection candidates among the first detection candidates whose detection score is equal to or greater than the first threshold, the region calculation step involves calculating an excision region based on the first detection candidates. Image generation step of generating a second image by cutting out the first image based on the aforementioned cutout region, A program for causing a computer to perform the following steps: a first detection step and a second candidate generation step, which involves performing the processing of the second image to generate a second detection candidate for the detection of the subject to be detected. [Explanation of Symbols]

[0060] 110: Image acquisition unit 111: First detection unit 112: First candidate generation unit 113: Determination unit 114: Cropping region calculation unit 115: Cropped image generation unit 116: Second candidate generation unit 117: Result determination unit 118: Second detection unit 119: Third detection unit

Claims

1. A first detection means that estimates the detection region and detection score related to the subject to be detected from the input image, A first candidate generation means generates a first detection candidate for the detection of the subject to be detected using the first detection means for the first image, If there are no detection candidates among the first detection candidates whose detection score is equal to or greater than the first threshold, the region calculation means calculates a region to be cut out based on the first detection candidates, Image generation means for generating a second image by cutting out the first image based on the aforementioned cutout region, An information processing device comprising: a second candidate generation means for generating a second detection candidate for the detection of the subject to be detected using the first detection means with respect to the second image.

2. The information processing apparatus according to claim 1, further comprising determination means for determining whether or not there is a detection candidate among the first detection candidates whose detection score is equal to or greater than the first threshold.

3. The information processing apparatus according to claim 1, characterized in that the image generation means generates a second image of the same size as the first image based on an image cut out from the first image.

4. The information processing apparatus according to claim 1, characterized in that the region calculation means prioritizes selecting from the first detection candidates the detection candidate whose detection score is equal to or greater than the second threshold and whose detection score is high as the extraction criterion candidate, and calculates the extraction region based on the detection region of the extraction criterion candidate.

5. The information processing apparatus according to claim 1, characterized in that the region calculation means prioritizes selecting as extraction criterion candidates detection candidates from among the first detection candidates, where the size of the detection region is within a predetermined range, the detection score is equal to or greater than a second threshold, and the detection score is high, and calculates the extraction region based on the detection region of the extraction criterion candidates.

6. The information processing apparatus according to claim 1, characterized in that the region calculation means calculates the cut-out region based on the detection region of the detection candidate with the highest detection score among the first detection candidates.

7. The information processing apparatus according to claim 1, characterized in that the region calculation means calculates the cut-out region based on the detection region of the first detection candidate whose size is within a predetermined range and whose detection score is the highest among the first detection candidates.

8. The information processing apparatus according to claim 1, further comprising a result determination means for determining a detection result for detecting the subject to be detected based on the first detection candidate and the second detection candidate.

9. The information processing apparatus according to claim 8, characterized in that the result determination means determines the detection candidate corresponding to the first detection candidate from among the second detection candidates as the detection result.

10. The information processing apparatus according to claim 8, characterized in that the result determination means determines the second detection candidate as the detection result.

11. The information processing apparatus according to claim 8, characterized in that the result determination means determines the detection candidate among the second detection candidates, in which the size of the detection area is within a predetermined range, as the detection result.

12. The information processing apparatus according to any one of claims 1, 8, 9, 10, or 11, characterized by having an image acquisition means that acquires an image with a higher resolution than the first image, and reduces the resolution of the acquired high-resolution image to obtain the first image.

13. The information processing apparatus according to claim 12, characterized in that the image generation means generates the second image by cutting out the image from the high-resolution image acquired by the image acquisition means based on the cut-out region.

14. A second detection means that has been trained to detect a subject different from the subject to be detected from the input image, It has a third detection means that is trained to detect any subject from an input image, The information processing apparatus according to any one of claims 1, 8, 9, 10, or 11, characterized in that the region calculation means calculates the cropped region based on the detection results of the first detection means, the second detection means, and the third detection means, respectively, on the first image.

15. The information processing apparatus according to claim 14, characterized in that the region calculation means does not use the detection region detected by the second detection means as the cut-out region.

16. An information processing method performed by an information processing device, A first detection step estimates the detection region and detection score related to the subject to be detected for the input image, A first candidate generation step is performed on the first image to generate a first detection candidate for the detection of the subject to be detected, If there are no detection candidates among the first detection candidates whose detection score is equal to or greater than the first threshold, the region calculation step calculates a region to be cut out based on the first detection candidates. An image generation step of generating a second image by cutting out the first image based on the aforementioned cutout region, An information processing method characterized by comprising: a second candidate generation step of performing the first detection step on the second image to generate a second detection candidate for the detection of the subject to be detected.

17. A first detection step involves estimating the detection region and detection score related to the subject to be detected from the input image. A first candidate generation step involves performing the processing of the first detection step on the first image to generate a first detection candidate for the detection of the subject to be detected, If there are no detection candidates among the first detection candidates whose detection score is equal to or greater than the first threshold, the region calculation step involves calculating a region to be cut out based on the first detection candidates. Image generation step of generating a second image by cutting out the first image based on the aforementioned cutout region, A program for causing a computer to perform the following steps: a first detection step, which involves processing the second image to generate a second candidate for detection of the subject to be detected; and a second candidate generation step, which involves generating a second candidate for detection of the subject to be detected.

Images