Information processing apparatus, information processing method, and program

Abstract

To stably detect objects close to each other and overlapping each other without learning a task with high level of difficulty.SOLUTION: An information processing apparatus includes: detection means which detects a region of an object to be detected, from an input image; region processing means which integrates a plurality of ground-truth regions close to each other, out of ground-truth regions to be detected in the input image, into one ground-truth region; loss calculation means which calculates a loss on the basis of the region detected by the detection means and the ground-truth region processed by the region processing means, the ground-truth region being an output target of the detection means; and update means which updates a parameter to be used by the detection means for detecting the object to be detected.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There has been proposed a device that detects an object region, such as the position and size of a detection target object in a captured region, based on an image captured by a camera or the like. In recent years, many deep learning methods using a convolutional neural network (CNN) have been developed as a technology for detecting an object region that includes an object. Non-Patent Document 1 discloses a method for learning the center position of a detection target object and the size of a rectangle that includes the object, as an object detection technology using a CNN.

[0003] When detecting an object using a detection model based on deep learning using CNN, if multiple objects overlap in an image, it is difficult to accurately detect each of these objects individually. Patent Document 1 proposes a technology that makes it possible to detect an object that is partially occluded in an image. In Patent Document 1, when multiple detection candidate frames are output and overlap each other, the existence probability of an object in the rear detection candidate frame is calculated, and if the existence probability is equal to or greater than a threshold, each detection candidate frame is output, and if it is less than the threshold, the front and rear detection candidate frames are integrated and output. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6891984 [Non-patent literature]

[0005] [Non-Patent Document 1] Objects as Points, Xingyi Zhou, Dequan Wang, Philipp Krahenbuhl; arXiv1904.07850, Summary of the Invention [Problem to be solved by the invention]

[0006] In learning an object detection model, it is a difficult task to learn to separate closely overlapping objects in an image and infer their individual regions. Forcing a model to learn such a difficult task can have significant drawbacks. If you try to force the model to learn by separating them, the detection rate itself will decrease, and processing costs will increase.

[0007] Therefore, an object of the present invention is to enable stable detection of even closely spaced and overlapping objects without the need to learn highly difficult tasks. [Means for solving the problem]

[0008] The information processing device of the present invention is characterized in having a detection means for detecting a region of an object to be detected from an input image, a region processing means for merging a plurality of adjacent correct answer regions within the correct answer region of the object to be detected in the input image into a single correct answer region, a loss calculation means for calculating a loss based on the region detected by the detection means and the correct answer region that is the output target of the detection means processed by the region processing means, and an update means for updating parameters used by the detection means to detect the object to be detected based on the calculated loss. Effect of the Invention

[0009] According to the present invention, it is possible to stably detect objects that are close to each other and overlap each other without learning highly difficult tasks. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a data flow in an information processing device. [Diagram 3] 5 is a flowchart showing an example of processing by the information processing device in the first embodiment. [Figure 4] 5 is a flowchart showing an example of processing by a region processing unit in the first embodiment. [Diagram 5] FIG. 4 is a diagram illustrating a process related to a correct solution area in the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating a process according to the second embodiment. [Figure 7] 13 is a flowchart showing an example of processing by a region processing unit in the second embodiment. [Figure 8] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a third embodiment. [Figure 9] 13 is a flowchart showing an example of processing by an information processing device according to the third embodiment. [Figure 10] 13 is a flowchart illustrating an example of a unification determination and selection process in the third embodiment. [Figure 11] FIG. 1 shows an example of an image showing closely overlapping people. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] <First embodiment> First, an example will be described in which multiple objects overlap each other, making it difficult to detect objects using a detection model based on deep learning using a CNN (Convolutional Neural Network). For example, as in the case of human body 1101 and human body 1102 shown in FIG. 11(A), when people overlap each other in a crowd, it becomes difficult to accurately recognize the regions of these people individually. Human bodies can take various postures, for example, as in the case of human body 1103, where the head and feet of the same person are not necessarily aligned vertically. Therefore, it is a highly difficult task for the detection model to accurately determine which of the heads and feet of human body 1101 and human body 1102 correspond to the same person.

[0013] The correct answer region (correct answer rectangle) when the detection model is trained is usually divided for each object as shown by the dotted line in Fig. 11(A), and these regions are trained so that they can be inferred separately. However, as an inference result, it is possible that the respective regions are inferred to be unified as shown by the dotted lines surrounding the human body 1101 and the human body 1102 as shown in Fig. 11(B). It is difficult to accurately associate detailed parts of the human body 1101 and the human body 1102, such as the head, torso, and legs, with the same person. In addition, the human body 1101 and the human body 1102 overlap and become the foreground and are separated from the background, so they are likely to be inferred to be unified.

[0014] Therefore, in this embodiment, in the learning of a detection model for detecting a detection target object, closely overlapping objects are not learned to be forcibly separated, but are learned as a unified object, thereby enabling stable detection. By learning as a unified object, the detection model can focus only on separating foreground and background, and does not need to learn to separate closely overlapping objects, which is highly difficult.

[0015] The information processing device in this embodiment uses CNN to construct a detection model for detecting a detection target object from an image. In this embodiment, an example will be described in which the detection target is a region of a human body visible in an image.

[0016] 1 is a block diagram showing an example of the configuration of an information processing device in this embodiment. The information processing device in this embodiment has a CPU 100, a ROM 110, an input unit 120, a RAM 130, a storage unit 140, a display unit 150, and a communication unit 160. The CPU 100, the ROM 110, the input unit 120, the RAM 130, the storage unit 140, the display unit 150, and the communication unit 160 are communicatively connected via a bus.

[0017] A CPU (Central Processing Unit) 100 controls the entire device by executing a control program stored in a ROM (Read Only Memory) 110. For example, the CPU 100 can realize each of the functional units of a detection unit 111, an area processing unit 112, a map generation unit 113, a loss calculation unit 114, and an update unit 115 by executing the control program stored in the ROM 110. Each of these functional units will be described later. A RAM (Random Access Memory) 130 temporarily stores various data, and also develops a program read from the ROM 110 to make it executable by the CPU 100.

[0018] The storage unit 140 stores various data used by the information processing device in this embodiment. The storage unit 140 stores, for example, data used for learning and a learned model. As a storage medium of the storage unit 140, a HDD (hard disk drive), an SSD (solid state drive), a flash memory, various optical media, etc. can be used. The input unit 120 is composed of a keyboard, a touch panel, etc., and accepts input from a user, and can receive an instruction to start learning, etc. The display unit 150 is composed of a liquid crystal display, etc., and displays various data and processing results to the user. The communication unit 160 connects the information processing device in this embodiment to another device so that they can communicate with each other. The information processing device in this embodiment may receive, for example, an instruction from a user from another device via the communication unit 160, or may output a processing result to the other device.

[0019] FIG. 2 is a diagram showing an example of a data flow in the information processing device of this embodiment. First, the CPU 100 acquires information on the input image 201 and the correct answer region 211 corresponding to the input image 201 from the storage unit 140. The correct answer region 211 is a group of predetermined shapes surrounding the range of each detection target in the input image 201, and is an output target of the detection unit 111. In the case of this embodiment, the detection target is a human body, and the predetermined shape is a rectangle. Here, when starting new learning, the CPU 100 initializes the parameters of the learning model 202 with random values ​​and expands it on the RAM 130. Also, when performing additional learning for an existing model, the CPU 100 reads the learning model 202 from the storage unit 140 and expands it on the RAM 130.

[0020] The detection unit 111 is an image detector that detects a detection target object (a human body in this embodiment) from an image. The detection unit 111 receives an input image 201 and outputs a detection map 203 via a learning model 202. The learning model 202 is composed of a CNN consisting of multiple layers. The CNN has an input layer that receives the input image 201, and outputs the detection map 203 from the output layer through multiple intermediate layers. Each layer of the CNN holds a weighting factor as a learning parameter, and each layer generates the detection map 203 that is finally output by sequentially executing a process of passing the result of weighting the input to the next layer. The detection map 203 is a two-dimensional map, and each point on the map has three-dimensional information, namely, an existence probability indicating whether the detection target exists at the location of the corresponding pixel, and the width and height of a rectangle that contains the detection target at the corresponding pixel. The existence probability takes a value between 0 and 1, and the closer the value is to 1, the higher the probability that the pixel in the two-dimensional map is an area where the detection target exists is inferred.

[0021] The region processing unit 112 receives the correct answer regions 211 as input, and determines whether or not to merge the respective correct answer regions 211. If the region processing unit 112 determines to merge the regions, it corrects the correct answer regions 211. Details of the operation of the region processing unit 112 will be described later with reference to FIG.

[0022] The map generation unit 113 receives the correct region 211 corrected by the region processing unit 112 as an input, and generates a correct map 212. The correct map 212 is a two-dimensional map similar to the detection map 203, and each point on the two-dimensional map has three-dimensional information of the existence probability of the detection target and the width and height of a rectangle containing the detection target, similar to the detection map 203. The existence probability takes a value between 0 and 1 for each pixel in the two-dimensional map, with the center of the detection target being 1 and approaching 0 the further away from the center of the detection target. The existence probability is generally made to have a spread following a Gaussian distribution.

[0023] The loss calculation unit 114 calculates the loss by comparing the detection map 203 with the correct map 212. In comparing the existence probability in the detection map 203 with the correct map 212, the loss calculation unit 114 calculates the cross entropy for each pixel according to (Equation 1). In (Equation 1), for each pixel i, the value p i and the value y of the correct map 212 i Cross entropy E i is calculated.

[0024]

number

[0025] Furthermore, the loss calculation unit 114 calculates the absolute error for each pixel when comparing the width and height of the rectangle in the detection map 203 and the correct map 212. As described above, the loss calculation unit 114 calculates the cross entropy and absolute error for each pixel for each dimension of the detection map 203 and the correct map 212, and calculates the loss by taking the sum of all pixels as shown in (Equation 2). In (Equation 2), the cross entropy E for each pixel is i , absolute error of width W i , and the absolute height error H i The sum of all pixels is calculated and is considered a loss.

[0026]

number

[0027] The update unit 115 uses backpropagation or the like based on the loss calculated by the loss calculation unit 114 to update the weight coefficients of each layer, which are parameters of the learning model 202, thereby constructing the learning model 202. That is, the update unit 115 updates the parameters used by the detection unit 111 to detect the detection target object based on the loss calculated by the loss calculation unit 114.

[0028] Fig. 3 is a flowchart showing an example of processing performed by the CPU 100 of the information processing device in the first embodiment. The processing shown in Fig. 3 is an example of a learning processing for learning a learning model (detection model) 202 used for detecting a detection target object.

[0029] In step S301, the detection unit 111 acquires from the storage unit 140 the input image 201 to be processed. In step S302, the detection unit 111 estimates and outputs a detection map 203 for the input image 201 acquired in step S301, using the learning model 202. In the process of step S302, the detection unit 111 obtains the detection map 203 by inputting the input image 201 to the CNN and transmitting signals to each layer in sequence.

[0030] In step S303, the region processing unit 112 acquires the correct region 211, which is to be an output target of the detection unit 111, from the storage unit 140. This correct region 211 corresponds to the input image 201 acquired in step S301. In step S304, the region processing unit 112 performs a correction process on the correct region 211, and merges adjacent correct regions 211 among the correct regions 211 acquired in step S303. Details of the process in step S304 will be described later with reference to FIG. In step S305, the map generating unit 113 generates and outputs the correct map 212 based on the correct area 211 corrected by the area processing unit 112 in step S304.

[0031] In step S306, the loss calculation unit 114 calculates a loss from the detection map 203 obtained by the detection unit 111 in step S302 and the correct answer map 212 generated by the map generation unit 113 in step S305. In step S307, the update unit 115 updates the learning parameters of the learning model 202 based on the loss calculated by the loss calculation unit 114 in step S306, using the backpropagation algorithm or the like.

[0032] In step S308, CPU 100 determines whether or not a predetermined end condition is satisfied. If CPU 100 determines that the end condition is satisfied (YES in step S308), the learning process shown in Fig. 3 is terminated. On the other hand, if CPU 100 determines that the end condition is not satisfied (NO in step S308), the process returns to step S301, and the above-mentioned processes of steps S301 to S307 are performed on the next input image 201. The end condition in step S308 can be any condition previously set by the user, and can be, for example, that the accuracy for the evaluation image is equal to or higher than a predetermined value, or that the learning process has been repeated a predetermined number of times or for a predetermined period of time.

[0033] The operation of region processing unit 112 in this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of processing by region processing unit 112 in step S304 in Fig. 3. When a group of rectangles indicating the range of a detection target object is input to region processing unit 112 as correct solution region 211, in step S401, a loop process is started for all rectangles in the group of rectangles input to region processing unit 112. In this loop process of steps S401 to S408, the processes of steps S402 to S407, in which one rectangle selected from the group of input rectangles is treated as the processing target, are executed for each of the input rectangles. In the following description, the one selected rectangle to be processed is treated as the target rectangle, and rectangles other than the target rectangle are treated as the group of candidate rectangles.

[0034] In step S402, the region processing unit 112 determines whether the candidate rectangle group is empty, i.e., whether or not at least one rectangle exists in the candidate rectangle group. If the region processing unit 112 determines that the candidate rectangle group is not empty, i.e., that one or more rectangles exist in the candidate rectangle group (YES in step S403), the process proceeds to step S403. On the other hand, if the region processing unit 112 determines that the candidate rectangle group is empty, i.e., that no rectangle exists in the candidate rectangle group (NO in step S403), the process proceeds to step S408.

[0035] In step S403, the region processing unit 112 calculates the area overlap rate with the target rectangle for each of the rectangles in the candidate rectangle group. Then, the region processing unit 112 selects the rectangle with the largest area overlap rate with the target rectangle from the candidate rectangle group as the candidate rectangle. The area overlap rate of two rectangles is the common area of ​​the two rectangles divided by the area of ​​the union of the two rectangles (so-called IoU: Intersection over Union).

[0036] In step S404, the region processing unit 112 calculates the vertical overlap rate between the target rectangle and the candidate rectangle, and determines whether the vertical overlap rate is 80% or more. The vertical overlap rate of two rectangles is the vertical length of the common part of the two rectangles divided by the vertical length of the union of the two rectangles. That is, in step S404, the region processing unit 112 determines whether the human bodies are at approximately the same height on the input image 201 and whether they are approximately the same size. If the region processing unit 112 determines that the vertical overlap rate is 80% or more (YES in step S404), the process proceeds to step S405. If the region processing unit 112 determines that the vertical overlap rate is less than 80% (NO in step S404), the process proceeds to step S407.

[0037] In step S405, region processing unit 112 calculates the horizontal overlap rate between the target rectangle and the candidate rectangle, and determines whether the horizontal overlap rate is 10% or more. The horizontal overlap rate of two rectangles is the horizontal length of the common part of the two rectangles divided by the horizontal length of the union of the two rectangles. If region processing unit 112 determines that the horizontal overlap rate is 10% or more (YES in step S405), the process proceeds to step S406. If region processing unit 112 determines that the horizontal overlap rate is less than 10% (NO in step S405), the process proceeds to step S407. Through the processing of steps S404 and S405, the region processing unit 112 can perform control so as not to merge the target rectangle with the candidate rectangle if the difference in size between the target rectangle and the candidate rectangle is equal to or greater than a predetermined threshold value.

[0038] In step S406, the region processing unit 112 generates a single rectangle by merging the target rectangle and the selected rectangle, and adds the rectangle to the merged rectangle group. The region processing unit 112 also adds the target rectangle and the selected rectangle to the deletion rectangle group. The merged rectangle group and the deletion rectangle group are stored separately from the rectangle group of the correct solution region 211, and are not included in the loop target in step S401 or the candidate rectangle group in step S402.

[0039] In step S407, region processing unit 112 determines that the candidate rectangle is not a target for merging with the target rectangle, so it deletes the candidate rectangle from the group of candidate rectangles and repeats the process from step S402.

[0040] In step S408, the area processing unit 112 executes the process of step S401 if the above-mentioned process has not been completed for all rectangles input as the correct answer area 211. After the above-mentioned process has been completed for all rectangles input as the correct answer area 211, the area processing unit 112 executes the process of step S409.

[0041] In step S409, the region processing unit 112 deletes the rectangles of the deleted rectangle group from the correct solution region 211. In addition, the region processing unit 112 adds the rectangles of the merged rectangle group to the correct solution region 211. Then, the processing shown in FIG. 4 ends.

[0042] FIG. 5 is a diagram illustrating the super-correct region 211 before and after processing by the region processing unit 112 described with reference to FIG. 4. FIG. 5(A) shows an example of the super-correct region 211 before processing, and FIG. 5(B) shows an example of the super-correct region 211 after processing. In both FIG. 5(A) and FIG. 5(B), the super-correct region 211 is drawn by a dotted line for the same input image 201. When the region processing unit 112 determines that two super-correct regions 211 are close to each other, it merges the two super-correct regions 211 into one super-correct region 211. The merged super-correct region 211 becomes one large super-correct region 211 that includes the set of super-correct regions 211 before the merger.

[0043] In FIG. 5(A), the correct answer regions 211 of the human body 500 and the human body 501 are distant from each other and are not close to each other, so the region processing unit 112 does not merge the correct answer regions 211, and outputs the original correct answer region 211 as shown in FIG. 5(B). Also, in FIG. 5(A), the correct answer regions 211 of the human body 502 and the human body 503 are close to each other, so the region processing unit 112 merges the correct answer regions 211, and outputs the large correct answer region 211 as shown in FIG. 5(B). In FIG. 5(A), the correct answer regions 211 of the human body 504 and the human body 505 are close to each other in the horizontal direction, but are distant from each other in the vertical direction. Therefore, the region processing unit 112 determines that they are not targets for merging, and outputs the original correct answer region 211 as shown in FIG. 5(B). In FIG. 5(A), the correct answer region 211 of the human body 507 is close to both the correct answer regions 211 of the human body 506 and the human body 508. The region processing unit 112 merges the correct answer region 211 of the human body 507 with the correct answer regions 211 of both the human body 506 and the human body 508 individually. Therefore, the three correct answer regions 211 before the merger become two correct answer regions 211 after the merger as shown in FIG. 5(B). In FIG. 5(A), the correct answer regions 211 of the human body 509 and the human body 510 are close to each other, but the difference in size between the human bodies is larger than the allowable range. Therefore, the region processing unit 112 determines that they are not targets for merging, and outputs the original correct answer region 211 as shown in FIG. 5(B).

[0044] According to this embodiment, in the learning of a detection model for detecting a detection target object, by learning closely overlapping objects as a single object without trying to learn to separate them, it is possible to prevent learning that is unduly difficult. In this way, it is possible to learn to stably detect the detection target object, and it is possible to improve the performance of object detection.

[0045] <Second embodiment> In the first embodiment, the case where there is one image detector is described. In this embodiment, an example is described in which the detection unit 111 has a first detector and a second detector, and after the adjacent regions are merged and detected by the first detector, the adjacent regions are detected by another second detector, so that the adjacent regions can be detected individually with high accuracy. In the first embodiment, the case where there are two correct answer regions to be merged is described. However, depending on the characteristics of the detector and the detection target, there may be cases where it is better to merge the correct answer regions in a combination of three or more regions, not limited to two. In this embodiment, a case is described in which the combination of the correct answer regions to be merged is set to an arbitrary number. Note that in the second embodiment described below, the description of the same configuration and operation (processing) as in the first embodiment is omitted.

[0046] FIG. 6 is a diagram for explaining the processing of the information processing device in the second embodiment. With reference to FIG. 6, an overview of the processing of the first detector and the second detector in this embodiment will be explained. The detection target of the first detector is a human body as in the first embodiment, and the detection target of the second detector is a human head. FIG. 6(A) shows an example of the correct answer region 211 before processing by the region processing unit 112 in the input image 201, and FIG. 6(B) shows an example of the correct answer region 211 after processing. In the first embodiment, the region processing unit 112 merges the correct answer regions 211 for each pair, but in the second embodiment, it is assumed that it can merge up to three correct answer regions 211. A method for arbitrarily setting the number of correct answer regions 211 to be merged will be described later with reference to FIG. 7. In the second embodiment, for example, the region processing unit 112 merges three adjacent correct answer regions 211 of the human body 601, the human body 602, and the human body 603 shown in FIG. 6(A) into one correct answer region 211 as shown in FIG. 6(B). Using the corrected correct region 211 as shown in FIG. 6(B), the first detector is trained in the same manner as in the first embodiment.

[0047] When the input image 201 is inferred using the first detector trained in this way, the rectangle indicating the area of ​​the detection target object inferred by the first detector is as shown in FIG. 6(B). Then, the information processing device cuts out each rectangular area of ​​the input image 170 using the group of rectangles inferred by the first detector and enlarges it by two times. For example, an input image as shown in FIG. 6(C) is obtained by cutting out the rectangular area indicated by the dotted line in FIG. 6(B) from the input image 201 and enlarging it by two times. FIG. 6(C) is a diagram showing the result of inference by the second detector after inputting the enlarged image of the rectangular part inferred by the first detector to the second detector. Here, the second detector has not been trained to merge adjacent areas. Therefore, by performing detection using a larger image, the second detector can easily distinguish the contours even when the detection targets overlap, and can easily detect the detection target object without merging. In addition, the detection target of the second detector is the human head, which is unlikely to overlap with the human body and its shape does not change as much as the human body, so even if it does overlap, it is easy to separate, and therefore easy to detect while remaining separated.

[0048] Fig. 7 is a flowchart showing a processing example of the area processing unit 112 in the second embodiment. Fig. 7 shows a processing example of the area processing unit 112 when the number of combinations of correct answer areas (the number of correct answer areas that can be merged) is set arbitrarily. In the second embodiment, the processing of step S304 in Fig. 3 in the first embodiment is replaced by the processing shown in Fig. 7, and other processing is similar to that in the first embodiment.

[0049] In step S701, area processing unit 112 determines whether the process has been repeated (maximum number of combinations-1) times. If area processing unit 112 determines that the process has been repeated (maximum number of combinations-1) times (YES in step S701), the process shown in FIG. 7 ends; otherwise (NO in step S701), the process proceeds to step S702. The maximum number of combinations is set before the start of the process, and is 3 in this embodiment. That is, if the maximum number of combinations is 3, the processes of steps S702 and S703 will be executed a maximum of two times.

[0050] In step S702, the region processing unit 112 performs adjacent pair rectangle merging processing. This adjacent pair rectangle merging processing in step S702 is the same as the processing described in the first embodiment with reference to Fig. 4. After this adjacent pair rectangle merging processing, the rectangles are treated as the same rectangle group regardless of whether they are merged or not.

[0051] In step S703, the region processing unit 112 determines whether or not there is a rectangle that has been changed by the adjacent pair rectangle merging process in step S702. If the region processing unit 112 determines that there is a changed rectangle (YES in step S703), the process returns to step S701. On the other hand, if the region processing unit 112 determines that there is no changed rectangle (NO in step S703), the process shown in FIG. 7 ends.

[0052] According to the second embodiment, the first detector detects the detection target area without missing anything, thereby limiting the detection target area of ​​the second detector. Then, the second detector can easily separate the detection targets individually by performing inference at high resolution for a limited range. In this way, even if the detection targets overlap, it is possible to detect the detection targets individually by separating them by using two detectors. It is also possible to deal with cases where it is better to detect three or more detection targets in a unified manner.

[0053] Although an example in which the first detector and the second detector have different detection targets has been described, the first detector and the second detector may have the same detection target. For example, based on the detection result of the detection target by the first detector, an image near the detection target is enlarged and input to the second detector, so that the resolution of the input image is higher and it is easier to separate the detection target than with the first detector. In the above example, the enlarged image obtained by enlarging the rectangular portion inferred by the first detector by a factor of two is input to the second detector, but the magnification is not limited to two and any magnification greater than one can be applied.

[0054] <Third embodiment> Although it is possible to improve the detection rate by detecting nearby objects in a unified manner, there are cases where it is desired to detect the objects to be detected while keeping them separate depending on the application. In the third embodiment, an example of detecting the objects in such a situation by separating them without reducing the detection rate will be described. Note that in the third embodiment described below, the description of the same configuration and operation (processing) as those in the first and second embodiments will be omitted.

[0055] A description will now be given of a learning method for the detection unit 111 in this embodiment. The learning method for the detection unit 111 described below can be realized by, for example, the information processing device shown in FIG.

[0056] In the first embodiment, the detection map 203 and the correct answer map 212 had three-dimensional information on the existence probability of the detection target and the width and height of the rectangle containing the detection target. In addition, the rectangles were unified by the region processing unit 112 for adjacent rectangles. In the first embodiment, the detection map 203 and the correct answer map 212 had three-dimensional information on the existence probability of the unified rectangle and the width and height of the rectangle. In the third embodiment, in addition to the above, the detection map 203 and the correct answer map 212 have seven-dimensional information that includes four-dimensional information on the existence probability of the rectangle that remains separated, the width and height of the rectangle, and the probability of whether or not the rectangles have been unified by the region processing unit 112 (hereinafter, the unification probability). That is, when the input image 201 is input, the detection unit 111 outputs the detection map 203 having seven-dimensional information on the three-dimensional existence probability, width, and height of the unified detection target, the three-dimensional existence probability, width, and height of the separated detection target, and the one-dimensional unification probability.

[0057] In the correct answer map 212, the probability of unification takes a value between 0 and 1 for each pixel in the map, similar to the probability of existence, with the center of the unified object being 1 and the value approaching 0 the further away from the detected object. The probability of unification is generally made to have a spread according to a Gaussian distribution. Also, if there is no unification, the value is 0 regardless of the distance from the detected object.

[0058] The loss calculation unit 114 calculates the loss based on (Equation 1) for the probability in the corresponding dimensions in the detection map 203 and the correct answer map 212, as in the first embodiment, calculates the absolute error for the width and height, and calculates the loss by taking the sum of all pixels, as in (Equation 2).

[0059] By learning in this way, the detection unit 111 can output not only the unified rectangles, but also the rectangles that remain separated and the probability of whether or not they have been unified.

[0060] A case where inference is performed using the detection unit 111 learned as described above will be described. Inference using the detection unit 111 learned as described above can be realized, for example, by an information processing device as shown in FIG. 8. FIG. 8 is a block diagram showing an example of the configuration of an information processing device in the third embodiment, and performs inference processing using the detection unit 111 learned as described above. The CPU 100 of the information processing device shown in FIG. 8 can realize each of the functional units of the detection unit 111, the post-processing unit 811, and the determination unit 812 by executing, for example, a control program stored in the ROM 110. Note that the detection unit 111 has a first detector and a second detector for detecting the same detection target, and the first detector detects adjacent areas related to the detection target in a unified manner, and the second detector detects areas related to the detection target without unifying them.

[0061] Fig. 9 is a flowchart showing an example of processing performed by the CPU 100 of the information processing device in the third embodiment shown in Fig. 8. Fig. 9 shows processing when the information processing device performs inference using the detection unit 111.

[0062] In step S901, the detection unit 111 acquires an image to be inferred as an input image 201, and estimates and outputs a detection map 203 for the input image 201 using the detection model learned as described above.

[0063] In step S902, the post-processing unit 811 outputs the adjacent-portion-united rectangle group, the probability of unity (uniting probability) for each rectangle, and the rectangle group that remains separated from the detection map 203 obtained in step S901. The processing of the post-processing unit 811 for the adjacent-portion-united rectangle group will be described. First, the post-processing unit 811 acquires the coordinates of the maximum value point and the value of the probability at that time in the dimension of the existence probability of the detection map 203. Then, the post-processing unit 811 deletes points whose probability is less than a threshold value (30% in the example of this embodiment). Next, the post-processing unit 811 acquires the width, height, and uniting probability value at the coordinates of the maximum point that has not been deleted from the dimensions of the width, height, and uniting probability of the detection map 203, and outputs information on the position, size, and uniting probability of the rectangle. The same applies to the processing of the post-processing unit 811 for the rectangle group that remains separated.

[0064] In step S903, the determination unit 812 executes a merge determination and selection process based on the adjacent-portion merged rectangle group obtained in step S902, the merge probability, and the rectangle group that remains separated. In this merge determination and selection process, it is determined whether each merged rectangle output in step S902 is a merged rectangle, and a selection is made as to whether to output a merged rectangle or a separated rectangle. The details of the merge determination and selection process in step S903 will be described later with reference to FIG. 9. In step S904, the detection unit 111 groups the rectangles output by the unification determination and selection process in step S903 into a rectangle group, and finally outputs the group as an inferred rectangle.

[0065] FIG. 10 is a flowchart showing an example of the unification determination and selection process in step S903 of FIG. The loop process of steps S1001 to S1009 is executed by selecting the input united rectangles one by one.

[0066] In step S1002, the determination unit 812 determines whether the probability of unification corresponding to the selected united rectangle is 30% or less. If the determination unit 812 determines that the probability of unification is 30% or more (YES in step S1002), the process proceeds to step S1003. On the other hand, if the determination unit 812 determines that the probability of unification is less than 30% (NO in step S1002), the process proceeds to step S1007. Here, the determination unit 812 does not immediately determine that there is a unification even if the probability of unification is 30% or more. In the process after step S1003, the determination unit 812 checks whether there is a separated rectangle corresponding to the united rectangle, and determines that there is a unification when there is a separated rectangle corresponding to the united rectangle.

[0067] In step S1003, the determination unit 812 determines whether or not there are two or more separated rectangles in the united rectangle. If the determination unit 812 determines that there are two or more separated rectangles in the united rectangle (YES in step S1003), the process proceeds to step S1004, and if not (NO in step S1003), the process proceeds to step S1007. In this embodiment, the number of combinations of adjacent rectangles that are united is set to a maximum of two, as in the first embodiment. If there are two or more separated rectangles in the united rectangle, it is determined that there is a possibility that a separated rectangle corresponding to the united rectangle exists, and the process of step S1004 is executed.

[0068] In step S1004, the determination unit 812 performs the unification process shown in FIG. 4 on the separated rectangles in the selected united rectangle in the same manner as the region processing unit 112. This unification process creates a unified separated rectangle group by unifying the output separated rectangles. The determination unit 812 determines whether the maximum area overlap rate between the unified separated rectangle group and the selected united rectangle is 50% or more. If the determination unit 812 determines that the maximum area overlap rate is 50% or more (YES in step S1004), the process proceeds to step S1005. On the other hand, if the determination unit 812 determines that the maximum area overlap rate is less than 50% (NO in step S1004), the process proceeds to step S1007. This process in step S1004 can prevent, for example, separated rectangles that are too small in size compared to the united rectangle from being output.

[0069] In step S1005, the determination unit 812 determines that the selected united rectangle is united, and in step S1006, the determination unit 812 selects two separated rectangles corresponding to the selected united rectangle as output rectangles.

[0070] In step S1007, the determination unit 812 determines that the selected united rectangle is a non-united rectangle. Here, a non-united rectangle is a rectangle that is not a united rectangle, or a united rectangle that has been unified but does not have a suitable corresponding separated rectangle. Then, in step S1008, the determination unit 812 selects the selected united rectangle as an output rectangle.

[0071] In step S1009, if the determination unit 812 has not completed the above-mentioned processing for all the united rectangles, the process returns to step S1001. After completing the above-mentioned processing for all the united rectangles, the determination unit 812 outputs the rectangle group selected as the output rectangles and ends the unity determination and selection process shown in FIG.

[0072] According to the third embodiment, it is possible to accurately separate and detect the detection target from the merged target. When two-stage detection is performed as in the second embodiment, multiple detections are required, which increases the processing time. However, according to the third embodiment, the operation of the detection unit 111 can perform detection in one operation, which makes it possible to minimize the increase in processing time.

[0073] In the third embodiment, the probability of unification output by the detection unit 111 is used as one of the means for determining unification by the determination unit 812. The means for determining unification is not limited to this, and other methods can be used. For example, when detecting detailed parts instead of outputting separated rectangles, the result can be used. For example, when the detection unit 111 detects a human body as a global part and a head as a detailed part, the determination unit 812 checks how many detailed parts are detected in each human body rectangle. Then, the determination unit 812 determines that there is unification when there are two or more heads in one human body rectangle, and determines that there is no unification when there is only one head.

[0074] In each of the above-described embodiments, an example has been described in which a human body is the detection target. However, the detection target is not limited to a human body, and for example, the detection target may be an animal, a vehicle, or the like.

[0075] (Another embodiment of the present invention) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0076] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0077] The disclosure of the present embodiment includes the following configurations, methods, etc. (Configuration 1) A detection means for detecting a region of a detection target object from an input image; a region processing means for integrating a plurality of adjacent correct answer regions of the detection target in the input image into one correct answer region; a loss calculation means for calculating a loss based on the area detected by the detection means and a correct area which is an output target of the detection means and which has been processed by the area processing means; and updating means for updating a parameter used by said detecting means for detecting an object based on the calculated loss. (Configuration 2) 2. The information processing device according to configuration 1, wherein the region processing means does not merge the two adjacent correct regions when the difference in size between the two correct regions is equal to or greater than a predetermined threshold value. (Configuration 3) The information processing device according to configuration 2, wherein the area processing means merges two adjacent correct answer areas into one correct answer area when the vertical overlap rate of the two adjacent correct answer areas is equal to or greater than a first threshold value and the horizontal overlap rate of the two adjacent correct answer areas is equal to or greater than a second threshold value. (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the loss calculation means calculates the loss by comparing a detection map generated based on the area detected by the detection means with a correct map generated based on the correct area processed by the area processing means. (Configuration 5) 5. The information processing device according to configuration 4, wherein the detection map and the correct answer map have at least information regarding the probability of the presence of a detection target at a position of a corresponding pixel and a width and height of an area. (Configuration 6) A first detection means for detecting a plurality of adjacent detection target objects from an input image as a single region; and second detection means for detecting a detection target object from an enlarged image relating to the area detected by the first detection means. (Configuration 7) A first detection means for detecting a plurality of adjacent detection target objects from an input image as a single region; a second detection means for detecting the detection target object from the input image without merging the object; a determination means for determining whether or not the area detected by the first detection means is a unified area; an output means for outputting the detection result by the second detection means when the determination means determines that the detected objects are united, and for outputting the detection result by the first detection means when the determination means determines that the detected objects are not united. (Configuration 8) 8. The information processing device according to configuration 7, wherein the determination means makes a determination using information on the probability of whether or not the signals are merged, which is output from the detection means. (Configuration 9) 9. The information processing device according to any one of configurations 6 to 8, comprising the first detection means obtained by learning using the information processing device according to any one of claims 1 to 5. (Method 1) An information processing method executed by an information processing device, a detection step of detecting an area of ​​a detection target object from an input image by a detection means; a region processing step of merging a plurality of adjacent correct answer regions of the detection target in the input image into one correct answer region; a loss calculation step of calculating a loss based on the area detected in the detection step and a correct area that is an output target of the detection means and that is processed in the area processing step; and updating a parameter used by the detection means for detecting the detection target based on the calculated loss. (Method 2) An information processing method executed by an information processing device, A first detection step of integrating a plurality of adjacent detection target objects from an input image and detecting them as one region; and a second detection step of detecting a detection target object from an enlarged image relating to the area detected in the first detection step. (Method 3) An information processing method executed by an information processing device, A first detection step of integrating a plurality of adjacent detection target objects from an input image and detecting them as one region; a second detection step of detecting the detection target object from the input image without merging the object; a determination step of determining whether or not the regions detected in the first detection step are detected as a unified region; an output process for outputting the detection result of the second detection process when it is determined in the determination process that the detected objects are united, and outputting the detection result of the first detection process when it is determined in the determination process that the detected objects are not united. (Program 1) The computer of the information processing device a detection step of detecting an area of ​​a detection target object from an input image by a detection means; a region processing step of merging a plurality of adjacent correct answer regions of the detection target in the input image into one correct answer region; a loss calculation step of calculating a loss based on the area detected in the detection step and a correct area that is an output target of the detection means and that is processed in the area processing step; and an updating step of updating parameters used by the detection means for detecting the detection target based on the calculated loss. (Program 2) The computer of the information processing device A first detection step of integrating a plurality of adjacent detection target objects from an input image and detecting them as one region; and a second detection step of detecting a detection target object from an enlarged image relating to the area detected in the first detection step. (Program 3) The computer of the information processing device A first detection step of integrating a plurality of adjacent detection target objects from an input image and detecting them as one region; a second detection step of detecting the detection target object from the input image without merging the object; a determination step of determining whether or not the areas detected in the first detection step are detected as a unified area; and an output step of outputting the detection result of the second detection step when it is determined in the determination step that the detected objects are united, and outputting the detection result of the first detection step when it is determined in the determination step that the detected objects are not united. [Explanation of symbols]

[0078] 100: CPU 110: ROM 111: Detection unit 112: Area processing unit 113: Map generation unit 114: Loss calculation unit 115: Update unit 120: Input unit 130: RAM 140: Storage unit 150: Display unit 160: Communication unit 811: Post-processing unit 812: Determination unit

Claims

1. A detection means for detecting a region of a detection target object from an input image; a region processing means for integrating a plurality of adjacent correct answer regions of the detection target in the input image into one correct answer region; a loss calculation means for calculating a loss based on the area detected by the detection means and a correct area which is an output target of the detection means and which has been processed by the area processing means; and updating means for updating a parameter used by said detecting means for detecting an object based on the calculated loss.

2. 2. The information processing apparatus according to claim 1, wherein said region processing means does not merge two adjacent correct regions when a difference in size between the two correct regions is equal to or greater than a predetermined threshold value.

3. 3. The information processing device according to claim 2, wherein the area processing means merges two adjacent correct answer areas into one correct answer area when a vertical overlap rate of the two adjacent correct answer areas is equal to or greater than a first threshold value and a horizontal overlap rate of the two adjacent correct answer areas is equal to or greater than a second threshold value.

4. 2. The information processing device according to claim 1, wherein the loss calculation means calculates the loss by comparing a detection map generated based on the area detected by the detection means with a correct map generated based on the correct area processed by the area processing means.

5. 5. The information processing apparatus according to claim 4, wherein the detection map and the correct answer map each include at least information regarding the probability of the presence of a detection target at a corresponding pixel position and a width and height of an area.

6. a first detection means for detecting a plurality of adjacent detection target objects from an input image as a single region; and second detection means for detecting a detection target object from an enlarged image relating to the area detected by the first detection means.

7. a first detection means for detecting a plurality of adjacent detection target objects from an input image as a single region; a second detection means for detecting the object to be detected from the input image without merging the object; a determination means for determining whether or not the area detected by the first detection means is a unified area; an output means for outputting the detection result by the second detection means when the determination means determines that the detected objects are integrated and detected together, and for outputting the detection result by the first detection means when the determination means determines that the detected objects are not integrated and detected together.

8. 8. The information processing apparatus according to claim 7, wherein the determining means makes a determination using information on the probability of whether or not the signals are merged, which is output from the detecting means.

9. 7. The information processing apparatus according to claim 6, further comprising the first detection means obtained by learning using the information processing apparatus according to claim 1.

10. An information processing method executed by an information processing device, a detection step of detecting an area of ​​a detection target object from an input image by a detection means; a region processing step of merging a plurality of adjacent correct answer regions of the detection target in the input image into one correct answer region; a loss calculation step of calculating a loss based on the area detected in the detection step and a correct area that is an output target of the detection means and that is processed in the area processing step; and updating a parameter used by the detection means for detecting the detection target based on the calculated loss.

11. An information processing method executed by an information processing device, a first detection step of detecting a plurality of adjacent detection target objects from an input image as a single region; a second detection step of detecting a detection target object from an enlarged image relating to the area detected in the first detection step.

12. An information processing method executed by an information processing device, a first detection step of detecting a plurality of adjacent detection target objects from an input image as a single region; a second detection step of detecting the detection target object from the input image without merging the object; a determination step of determining whether or not the regions detected in the first detection step are detected as a unified region; an output process for outputting the detection result of the second detection process when it is determined in the determination process that the detected objects are united, and outputting the detection result of the first detection process when it is determined in the determination process that the detected objects are not united.

13. The computer of the information processing device a detection step of detecting an area of ​​a detection target object from an input image by a detection means; a region processing step of merging a plurality of adjacent correct answer regions of the detection target in the input image into one correct answer region; a loss calculation step of calculating a loss based on the area detected in the detection step and a correct area that is an output target of the detection means and that is processed in the area processing step; and an updating step of updating parameters used by the detection means for detecting the detection target based on the calculated loss.

14. The computer of the information processing device a first detection step of detecting a plurality of adjacent detection target objects from an input image as a single region; and a second detection step of detecting a detection target object from an enlarged image relating to the area detected in the first detection step.

15. The computer of the information processing device a first detection step of detecting a plurality of adjacent detection target objects from an input image as a single region; a second detection step of detecting the detection target object from the input image without merging the object; a determination step of determining whether or not the areas detected in the first detection step are detected as a unified area; a program for executing an output step of outputting the detection result of the second detection step when it is determined in the judgment step that the detected objects are united, and an output step of outputting the detection result of the first detection step when it is determined in the judgment step that the detected objects are not united.

Images