Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To provide a technology for detecting a subject desired by a user by additional learning while maintaining detection performance before performing the additional learning.SOLUTION: The information processing apparatus includes a storage unit configured to store a fixed model which is an unchangeable recognition model trained to detect an object of a predetermined category and a custom model which is a customizable recognition model trained to detect an object of the same category as the fixed model, a setting unit configured to set an integration method of integrating a detection result using the fixed model for the input image and a detection result using the custom model for the input image, and an integration unit configured to acquire an integrated detection result by integrating the detection results based on the integration method.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] Object detection is used to detect the area of ​​a specific object in an image. For example, face detection is used to detect the area of ​​a person's face in an image with a person as the subject. Based on the results of face detection, facial recognition and autofocus processing during photography are performed.

[0003] In recent years, a technology for training a recognition model using a neural network has been developed as an object detection technology. Non-Patent Document 1 discloses a method for detecting an object by training a neural network to output key points indicating the position of the object to be detected as a heat map.

[0004] Furthermore, a trained neural network may be additionally trained based on data obtained at the operation site. Patent Document 1 discloses a method of training an inspection device using a neural network, performing additional training using additional data collected during operation of the inspection device, and updating the neural network. As shown in Patent Document 1, if there is an object that is difficult to detect at the operation site, the recognition accuracy of that object can be improved by collecting image data of that object and performing additional training.

[0005] For example, if the detection accuracy of a particular person in face detection seems low, it is expected that the person will be detected with high accuracy by additionally training the recognition model using images of that person. Specifically, camera manufacturers train subject detectors (recognition models) for autofocus, incorporate them into camera products, and sell and provide them, and it is conceivable that users may additionally train those detectors according to their own preferences. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7271306 [Non-patent literature]

[0007] [Non-Patent Document 1] CenterNet: Keypoint Triplets for Object Detection, Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, Qi Tian; ICCV2019, pp.6569-6578. Summary of the Invention [Problem to be solved by the invention]

[0008] However, when a recognition model is updated through additional learning, it may no longer be possible to recognize something that was successful before the additional learning. For example, in face detection, additional learning of a specific person may cause the detection of a different person that was previously detectable to become unstable.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that enables a user to detect a desired subject through additional learning while maintaining the detection performance before additional learning. [Means for solving the problem]

[0010] To achieve the above object, an information processing device according to the present invention comprises: An information processing device that detects a subject from an input image, a storage means for storing a fixed model, which is an unchangeable recognition model trained to detect objects of a predetermined category, and a custom model, which is a customizable recognition model trained to detect objects of the same category as the fixed model; a setting means for setting an integration method for integrating the detection result using the fixed model for the input image and the detection result using the custom model for the input image; an integration means for integrating the detection results based on the integration method to obtain an integrated detection result; The present invention is characterized by comprising: [Effects of the Invention]

[0011] According to the present invention, it is possible to additionally detect a subject desired by a user through additional learning while maintaining the detection performance before additional learning is performed. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram of an information processing device according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram of a recognition model and detection processing according to the first embodiment. [Figure 3] 1 is a flowchart showing the overall processing flow according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of the processing of a learning setting unit according to the first embodiment. [Figure 5] 10 is a flowchart showing the procedure of a process executed by an erroneous detection verification unit according to the first embodiment. [Figure 6] 5 is a flowchart showing the procedure of processing executed by a custom model registration unit according to the first embodiment. [Figure 7] FIG. 3 is an explanatory diagram showing the procedure of processing executed by an integration method setting unit according to the first embodiment. [Figure 8] 4 is a flowchart showing the procedure of processing executed by a detection unit according to the first embodiment. [Figure 9] 5 is a flowchart showing the procedure of processing executed by a detection result integration unit according to the first embodiment. [Figure 10] FIG. 4 is an explanatory diagram of an integrated detection result according to the first embodiment. [Figure 11] 5 is a flowchart showing the procedure of processing executed by a display control unit according to the first embodiment. [Figure 12] FIG. 10 is an explanatory diagram of a multitask recognition model according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] (Embodiment 1) A first embodiment of the present invention will be described below with reference to the drawings. The following embodiment describes a recognition model that performs object detection to detect subjects of a predetermined category. A category is a classification of detection targets. For example, a person's face area or an animal's whole body area is a category of detection targets. A separate recognition model is trained for each category of detection target. This embodiment also describes an example in which a user performs additional training on a pre-trained recognition model using additional data related to a subject desired by the user. For example, this is an example in which a user who purchases a camera performs additional training on a recognition model originally included in a camera sold by a camera manufacturer in order to improve the detection accuracy of the user's desired subject. Note that this embodiment is described as one example of implementing the present invention, and the present invention is not limited to this example.

[0015] <Configuration of information processing device> 1 is a configuration diagram of an information processing device according to this embodiment. A CPU 101 controls the entire information processing device. A first memory 103 and a second memory 104 are storage units that store control programs and various data that perform processing according to this embodiment. Here, the first memory 103 is described as mainly storing control programs, and the second memory 104 is described as mainly storing various data, but this is not limited to this.

[0016] The input unit 105 is composed of a keyboard, mouse, touch panel, etc., and accepts input from a user. The display unit 106 is composed of a display device such as a liquid crystal display, and can display processing results to a user. The communication unit 107 can communicate with an external device to send and receive data. The above components are connected via a computer bus 102. The information processing device according to this embodiment can be implemented as a computer that includes each of the processing units described below as a program. Furthermore, some of the above components may be provided in separate computers, and the computers may communicate with each other via the communication units 107 to perform processing. For example, a processing unit related to learning and evaluating a recognition model may be provided in a cloud-based computer, and a detection unit and display unit that use the recognition model may be provided on an edge device such as a camera or smartphone.

[0017] A fixed model 120 is stored in advance in the memory 104. This is a recognition model trained to detect objects of a predetermined category, and is trained using, for example, a neural network. The recognition model is trained so that it can detect areas of objects of a predetermined category by inputting an image into the recognition model.

[0018] 2(a) and 2(b) are diagrams illustrating an example of the operation of a recognition model that performs object detection. As shown in Fig. 2(a), recognition model 201 is trained to output an object likelihood map 203 and an object size map 204 when an input image 202 is input to recognition model 201. Recognition model 201 is, for example, a neural network.

[0019] The object likelihood map 203 is a map that represents the likelihood that an object of a predetermined category is estimated to exist at each position on the image. On the object likelihood map 203, independent regions (blobs) whose map values ​​are equal to or greater than a predetermined value are extracted, and the position within each region where the map value is maximum is calculated. The center of the object is assumed to be at that position. The map value of the object likelihood map 203 at that center position is used as the detection score for that detection.

[0020] The subject size map 204 is a map in which values ​​estimating the size of the subject for each position on the image are output as map values. The map value of the subject size map 204 corresponding to the position of the subject center calculated as described above is read and used as the size of the subject. The subject size can be expressed in any way, but in this embodiment, for simplicity of explanation, the subject size is assumed to be a square rectangle, and the subject size map 204 is assumed to be a map for estimating the side lengths of that square.

[0021] In this case, the detection result for one subject is expressed as a bounding box, which is a pair of the subject's center coordinates and size value. Since there can be multiple subjects in one image, the detection result for one image is a list of bounding boxes, as shown in the table of detection results 205 in Figure 2(b). id is the identifier of the subject detected in the image, cx is the x-coordinate of the subject's center, cy is the y-coordinate of the subject's center, size is the subject size, and score is the detection score. This example shows an example in which two subjects are detected. The detection result for id 1 is an area with center coordinates (cx1, cy1), size size1, and a detection score of score1. The detection result for id 2 is an area with center coordinates (cx2, cy2), size size2, and a detection score of score2.

[0022] Note that the above example is just one example of a method for performing object detection, and is not limited to this. For example, the object size may be expressed as the long and short sides of a rectangle surrounding the object, and learning may be performed to output two maps, a long side size map and a short side size map, as the object size map 204. Furthermore, instead of learning to output the area where the object exists as an object likelihood map or an object size map, a learning method may be used to directly infer the value of the object's bounding box, for example.

[0023] The memory 104 in FIG. 1 stores positive example training data 121 and negative example training data 122. These are data prepared in advance as data for training a recognition model. For example, the training data used when training the fixed model 120 may be used. In this embodiment, a recognition model that performs object detection is used as an example, so training data including an image of a subject to be detected and the value of a bounding box representing the area to be detected is prepared. The negative example training data 122 is data that collects cases that are likely to be erroneously detected in the category to be detected. These training data are used when training the detection of a predetermined category. Different data is prepared for each category to be detected, but for simplicity, this embodiment illustrates the data for one category being stored.

[0024] Additionally, memory 104 stores positive example additional training data 123 and negative example additional training data 124. These additional training data store training data related to a desired subject for which the user wishes to perform additional training in the detection of a predetermined category. Here, negative example additional training data 124 is not essential and may be empty. This is because when the user performs additional training on cases for which the user wishes to improve detection accuracy, the number of example patterns that are likely to be erroneously detected in the predetermined category does not increase, and therefore it is often sufficient to use negative example training data 122 that has been prepared in advance for training. It is assumed that these additional training data are created in advance by the user and stored in memory 104.

[0025] <Processing> Fig. 3 is a flowchart illustrating the overall flow of processing according to the first embodiment. In S301, the learning setting unit 110 sets up additional learning based on a user operation. Here, Fig. 4 is a diagram illustrating a user interface (UI) provided by the learning setting unit 100. The UI is displayed on the display unit 106, and the user can set it up via the input unit 105.

[0026] In FIG. 4, 402 to 405 are sections for inputting numerical values ​​specifying data weights. In 402, a data weight for the positive example teacher data 121 is set. In 403, a data weight for the negative example teacher data 122 is set. In 404, a data weight for the positive example additional teacher data 123 is set. In 405, a data weight for the negative example additional teacher data 124 is set. The detection category selection menu 406 is a pull-down menu, and one of the detection categories (in the illustrated example, human face, whole human body, animal face, whole animal body) in the detection category menu 407 is selected and set. When the decision button 408 is pressed, the learning setting unit 110 saves the value set in the UI in the memory 104. Regarding the weights set in 402 to 405, each set value may be divided by the sum of the values ​​set in 402 to 405 and saved as a data weight ratio. The respective set weight values ​​are saved in memory 104 as positive example teacher data weight 125, negative example additional teacher data weight 126, positive example additional teacher data weight 127, and negative example additional teacher data weight 128. The category set in detection category selection menu 406 is saved in detection category 129 in memory 104. When decision button 408 is pressed, processing by learning setting unit 110 ends, and S301 is terminated.

[0027] The processing of the learning setting unit 110 is not limited to the above-mentioned settings, and other settings related to learning may also be set. For example, it may be possible to set the learning rate, data augmentation, etc.

[0028] In S302, the learning unit 111 performs additional learning. The learning unit 111 generates a neural network using, as an initial value, a fixed model 120 corresponding to the detection category set in the detection category 129, and performs additional learning on the neural network. As the training data, positive example training data 121, negative example training data 122, positive example additional training data 123, and negative example additional training data 124 in the memory 104 are used. Here, the training data also uses data corresponding to the detection category set in the detection category 129. In addition, as weights for the training data, positive example training data weight 126, negative example training data weight 126, positive example additional training data weight 127, and negative example additional training data weight 128 in the memory 104 are used.

[0029] The learning progress status may be displayed to the user so that the user can understand the learning status. The additional positive case training data 123 provided by the user may be divided into training data and validation data, and the detection accuracy for the validation data may be displayed. This allows the user to proceed with learning while checking the detection accuracy for the desired subject. Similarly, the false detection rate may also be displayed to the user while proceeding with learning. The learning process is similar to a commonly used method, and a detailed description thereof will be omitted here. The additionally trained neural network is stored as a custom model 130 in memory 104.

[0030] Next, in S303, the false detection verification unit 112 evaluates the custom model 130. Model evaluation may be performed from any perspective; however, here, a detailed description will be given of a case where false detection is evaluated using the false detection verification unit 112. FIG. 5 is a flowchart showing the procedure of processing executed by the false detection verification unit 112 according to this embodiment. Data for false detection evaluation is prepared in advance as false detection evaluation data 131 in the memory 104. Data for false detection evaluation is prepared in advance for each category set in the detection category 129. Since each detection category has images that are prone to false detection, the false detection evaluation data is, for example, evaluation data that collects such data. However, since data containing positive examples also contains background elements that may cause false detection, the data containing positive examples may be used as the false detection evaluation data as is. The false detection evaluation data 131 may be configured so that the user can add more.

[0031] In S501, the misdetection verification unit 112 performs detection processing on the misdetection evaluation data 131 using the fixed model 120, calculates the misdetection rate of the fixed model 120, and stores it in the fixed model misdetection rate 132 in the memory 104.

[0032] In S502, the false positive verification unit 112 performs a detection process on the false positive evaluation data 131 using the custom model 130, calculates the false positive rate of the custom model 130, and stores it in the custom model false positive rate 133 in the memory 104.

[0033] In S503, the misdetection verification unit 112 calculates a misdetection index as an evaluation value for the custom model 130 and stores it in the misdetection index 134 in the memory 104. The calculation formula for the misdetection index 134 represents the relative difference between the misdetection rate of the custom model and the misdetection rate of the fixed model. For example, it can be calculated using the following formula 1. This uses the ratio of the misdetection rate of the custom model to the misdetection rate of the fixed model as the misdetection index.

[0034] False positive index = Custom model false positive rate / Fixed model false positive rate … (1) Formula 1 makes it possible to index the degree of false positives of a custom model without presenting the false positive rate of the fixed model itself. In other words, if the function of the false positive verification unit 112 is placed on the cloud, it becomes possible to conceal the false positive rate of the fixed model from the user.

[0035] Since the false positive rate of a fixed model may be a trade secret of the camera manufacturer, it is effective to keep it secret. This false positive index may be calculated for the model being trained during additional training and presented to the user during training. This allows the user to understand whether the training is progressing well.

[0036] This completes the process of S303 in Fig. 3. The processes from S301 to S303 are processes for additional learning, and may be performed by, for example, a computer on the cloud rather than by a device that performs detection processing, which will be described later.

[0037] In S304, the custom model registration unit 113 registers the additionally trained custom model 130 as a valid recognition model for detection. Note that one or more custom models may be registered for one detection category.

[0038] FIG. 6 is a flowchart showing the procedure of processing executed by the custom model registration unit 113 according to this embodiment. In S601, the custom model registration determination unit 114 determines whether or not to register the custom model 130 as valid based on the verification result of the false detection verification unit 112. The registration determination unit 114 determines whether or not to register the custom model 130, for example, by determining whether or not the false detection indicator 134 in the memory 104 is lower than a predetermined value. The registration determination unit 114 determines that registration is possible when the false detection indicator 134 is lower than the predetermined value. Note that the processing performed by the registration determination unit 114 here is merely an example; various evaluations, such as an evaluation of the accuracy of correct detection, may also be performed, and the registration determination may be made based on the results. If this step is yes, proceed to S602. On the other hand, if this step is no, proceed to S603.

[0039] In S602, the custom model registration unit 113 activates the custom model by turning on the custom model activation flag 135 in the memory 104. Then, the process ends.

[0040] In S603, the custom model registration unit 113 notifies the user that the false positive index is high and asks the user to confirm whether or not to register the custom model as valid. For example, a confirmation dialog may be presented to the user, and the user's intention may be confirmed by having the user press the OK button (or the registration button) or the cancel button in the dialog. If this step is yes, proceed to S604. On the other hand, if this step is no, proceed to S605.

[0041] In S604, the custom model registration unit 113 records information indicating that the user has registered the custom model after confirming that the false detection index is high, and proceeds to S602 to activate the custom model. In S605, the custom model registration unit 113 disables the custom model. That is, it turns off the custom model activation flag 135 in the memory 104. Note that if additional learning of the custom model has been performed in advance on another device, the additionally trained custom model may be acquired from the other device via the communication unit 107 only when the custom model is activated in S602.

[0042] This concludes the description of the processing of the custom model registration unit 113 in S304 of FIG.

[0043] Subsequently, in S305, the integration method setting unit 115 sets an integration method for the detection results. Here, Fig. 7 is a diagram for explaining the operation of the integration method setting unit 115 according to this embodiment.

[0044] Reference numeral 701 denotes an example of a UI screen presented to the user by the integration method setting unit 115. For example, the UI may be displayed on the camera's display, allowing the user to change the settings using the camera's setting buttons, etc. Alternatively, the settings may be made on a computer or the like, and the settings may be saved in the camera.

[0045] Reference numeral 702 denotes a detection category selection menu, which is a pull-down menu similar to the detection category selection menu 406 shown in Fig. 4. The user selects a category to be detected from the detection category selection menu 702.

[0046] An image selection button 703 is displayed. When the image selection button 703 is pressed, a dialog box (not shown) for image selection is displayed, allowing the user to select an image. The selected image is subjected to subject detection by the detection unit 116, which will be described later, and the detection results are displayed on the screen 704 by the display control unit 118. When this processing is performed on the camera, the image may be selected from images stored in the camera's memory, or an image captured live by the camera's capture unit may be used.

[0047] A slider bar 705 is used to set the weight of the custom model. The user adjusts the slider bar 705 while checking the detection results displayed on the screen 704 for the desired image. For example, detection results using the fixed model are displayed as red detection frames, and detection results using the custom model are displayed as yellow detection frames. Manipulating the position of the slider bar 705 changes the number of yellow detection frames indicating detection results using the custom model. For example, increasing the weight increases the number of yellow detection frames displayed due to erroneous detections. The user manipulates the slider bar 705 so that the custom model detects the subject correctly without any erroneous detections. The value set using the slider bar 705 is stored in the integration result calculation parameter 136 in the memory 104. This parameter is used by the detection result integration unit 117 (described later) for the process of integrating the detection results from the fixed model and the detection results from the custom model. Details of this parameter will be described later.

[0048] A decision button 706 is provided. When the decision button 706 is pressed, the processing of the integration method setting unit 115 ends.

[0049] Subsequently, in S306, the detection unit 116 performs a detection process. One example of the detection process is a process of detecting a subject using a recognition model for images sequentially acquired by a camera or video. The process of the detection unit 116 will be described in detail later.

[0050] In S307, the display control unit 118 displays the detection result obtained in S306. This is processing for displaying the result detected by the detection unit 116 on the display unit 106 or the like. Details of the processing by the display control unit 118 will be described later.

[0051] In S308, the detection unit 116 determines whether the input of images to be detected has been completed. In cameras and video cameras, images to be detected are typically input continuously, and the detection process is typically repeated. If the input of images to be detected has been completed, the process ends. On the other hand, if the input has not been completed, the process returns to S306, and the detection and display of the detection results are repeated.

[0052] In the above description, for simplicity, the processing related to additional learning and the processing related to detection are described together as a series of processes, but this is not limited to this. The processing from S301 to S303 may be performed by a cloud computer, the processing from S304 to S305 may be performed on the camera, and the processing from S306 onwards may be repeated each time a photograph is taken on the camera. For example, if the detection results are used for autofocus (AF), the processing from S306 to S307 may be repeated while the user half-presses the shutter button on the camera. This concludes the overall processing flow of this embodiment.

[0053] Next, a detailed description will be given of the process performed by the detection unit 116 in 306 in Fig. 3. Fig. 8 is a flowchart showing the procedure of the process performed by the detection unit 116 according to this embodiment.

[0054] In S801, the detection unit 116 acquires an input image and stores it in the input image 137 in the memory 104. The input image 137 may be acquired by capturing an image using a camera's imaging unit (not shown), by specifying an image stored in advance in the memory, or by acquiring it from an external device via the communication unit 107.

[0055] In S802, detection unit 116 inputs input image 137 into fixed model 120 in memory 104 to detect an object. As described with reference to Fig. 2, when an image is input into the recognition model, an object likelihood map and an object size map are obtained. The obtained maps are stored in fixed model object likelihood map 138 and fixed model object size map 139 in memory 104.

[0056] In S803, the detection unit 116 determines whether the custom model is valid. This can be done by checking whether the custom model validation flag 135 in the memory 104 is ON. If the custom model validation flag 135 is ON, it can be determined that the custom model is valid. If it is determined that the custom model is valid, the process proceeds to S804. On the other hand, if it is determined that the custom model is not valid, the process proceeds to S805.

[0057] In S804, detection unit 116 inputs input image 137 into custom model 130 in memory 104 to detect an object. As described with reference to Figure 2, when an image is input to the recognition model, an object likelihood map and an object size map are obtained. The obtained maps are stored in custom model object likelihood map 140 and custom model object size map 141 in memory 104.

[0058] If it is determined in S803 that the custom model is not valid, then before performing the processing of S805, the map values ​​of custom model subject likelihood map 140 and custom model subject size map 141 can be set to 0 and stored. Also, if one or more custom models are registered for a predetermined category, then the same processing as S804 can be repeated the number of times corresponding to the number of custom models. In this case, custom model subject likelihood map 140 and custom model subject size map 141 are managed separately for each custom model.

[0059] In S805, the detection result integration unit 117 integrates the detection result obtained by the fixed model 120 and the detection result obtained by the custom model 130 into one. Details of the processing by the detection result integration unit 117 will be described later. This completes the processing in FIG. 8.

[0060] Next, a detailed description will be given of the process performed by the detection result integration unit 117 in S805 of Fig. 8. Fig. 9 is a flowchart showing the procedure of the process performed by the detection result integration unit 117 according to this embodiment.

[0061] In S901, the detection result integration unit 117 integrates the fixed model subject likelihood map 138 and the custom model subject likelihood map 140 in the memory 104 into one map using, for example, the following formula 2, to calculate an integrated subject likelihood map.

[0062] Integrated object likelihood map = max (fixed model object likelihood map, custom model object likelihood map * α) ... (2) Here, the max function is a function that calculates the maximum map value at each position on the map. Furthermore, the coefficient α is a scalar value, and when the custom model enable flag in memory 104 is ON, the integrated result calculation parameter 136 in memory 104 is used as the coefficient α. When the custom model enable flag in memory 104 is OFF, the value of the coefficient α can be set to 0. The calculated integrated object likelihood map is stored in integrated object likelihood map 142 in memory 104.

[0063] Note that the calculation method of the integrated object likelihood map shown in the above formula 2 is not limited to this. For example, as another example, it is also possible to calculate it using the following formula 3.

[0064] Integrated subject likelihood map = Fixed model subject likelihood map + (Custom model subject likelihood map * β)…(3) Equation 3 indicates that an integrated object likelihood map is obtained by calculating a weighted sum of the fixed-model object likelihood map and the custom-model object likelihood map at each position on the map. The coefficient β is the weight value of the weighted sum. For objects that would not be detected in a detection process using only the fixed model due to insufficient map values ​​in the object likelihood map, adding the map values ​​in the custom-model object likelihood map enables detection. The setting of coefficient β varies the behavior of positive and false detections due to the custom model. The coefficient β can be set by operating the slider bar 705 shown in FIG. 7. Furthermore, the UI screen shown in FIG. 7 may display an option to select whether to use Equation 2 or Equation 3, allowing the user to select.

[0065] In this embodiment, the calculation method of the integrated subject likelihood map is arbitrary, but both of the calculation methods of the integrated subject likelihood map using the above-mentioned formula 2 and formula 3 are methods that guarantee that a subject detected only by the fixed model 120 will always be detected. This is because, when the methods of formula 2 and formula 3 are used, the map value at each position of the integrated subject likelihood map 142 will never be smaller than the map value of the fixed model subject likelihood map 138.

[0066] This configuration makes it possible to improve the detection performance for a desired subject additionally learned by the user without reducing the detection performance of the fixed model 120. When the fixed model 120 is a recognition model provided by the camera manufacturer, the detection performance intended by the camera manufacturer is guaranteed, and the detection performance can only be improved by user customization.

[0067] However, when integrating using this method, if there are too many false positives in the custom model, the integrated detection results will also contain many false positives. To avoid this, in the custom model evaluation process in S303 of Fig. 3, evaluation is performed not only on the performance of the correct detection side, but also by using the false positive verification unit 112, with particular attention paid to the performance of the false positive side.

[0068] Furthermore, in Equation 3, the coefficient β may be configured to be negative. In this case, detection will be suppressed in areas where the map value of the custom model object likelihood map 140 is high. The user can perform additional learning by using data of objects that the user does not want detected as positive case additional training data 123. This configuration makes it possible to use the custom model 130 to avoid erroneous detection of specific objects by the fixed model 120. In this case, detection by the fixed model 120 will be reduced, which will be a result that is in line with the user's intention.

[0069] Next, in S902, the detection result integration unit 117 calculates the object center position of the object region using the integrated object likelihood map 142. As explained with reference to FIG. 2, independent regions (blobs) whose map values ​​are equal to or greater than a predetermined value are extracted from the object likelihood map output by the recognition model, and the coordinates of the maximum value within each independent region are set as the coordinates of the object center position. Zero to multiple object center positions can be extracted in this way. From S903 onwards, processing is performed for each of the extracted object center positions.

[0070] In S903, the detection result integration unit 117 determines the recognition model that contributed to the detection of one of the subject center positions calculated in S902. The detection result integration unit 117 compares the map value of the fixed model subject likelihood map 138 at the subject center position with the map value of the custom model subject likelihood map 140. Then, based on the comparison result, the detection result integration unit 117 determines the model with the larger map value as the contributing model at that subject center position. For example, if the map value of the fixed model subject likelihood map 138 at the subject center position is larger than the map value of the custom model subject likelihood map 140, the contributing model is determined to be the fixed model.

[0071] In S904, the detection result integration unit 117 determines a detection score, and sets the map value at the object center position on the object likelihood map output by the contribution model determined in S903 as the detection score.

[0072] In S905, the detection result integration unit 117 determines the size of the subject. The map value at the coordinates of the subject center position is read from the size map of the determined contributing model, and this is set as the size value of the subject size. For example, if the contributing model is a fixed model, the map value at the coordinates of the subject center position in the fixed model subject size map 139 is set as the size value of the subject size.

[0073] In S906, the detection result integration unit 117 adds a set of values ​​including the coordinate values ​​of the subject center position, the subject size value, and the detection score value calculated by these processes, as well as an identification ID indicating the type of contributing model, to the integrated detection result 143 in the memory 104 and stores the set. The integrated detection result 143 includes information such as that shown in table 1001 in FIG. 10. A term for model_id indicating the contributing model has been added to the information shown in the detection result 205 in FIG. 2. For example, a model_id of 0 may indicate detection using a fixed model, and a model_id of 1 may indicate detection using a custom model. In the example of FIG. 10, two integrated detection results are stored.

[0074] In S907, the detection result integration unit 117 determines whether or not the processing has been completed for all of the one or more independent regions (blobs) extracted in S902. If the result of this step is no, the processing returns to S903 and is repeated. On the other hand, if the result of this step is yes, the series of processes ends.

[0075] In the above description, an example of integrating detection results in which each recognition model outputs a likelihood map and a size map has been described, but this is not limiting. For example, a recognition model trained to directly infer parameters of the bounding box of a detected subject may be used. In this case, the bounding boxes output by the fixed model and the custom model may be integrated into one based on the degree of overlap between their respective regions. Bounding boxes that overlap by a certain percentage or more may be integrated into one by averaging the values ​​of their positions, sizes, and detection scores. When determining the contributing model, the model with the greatest degree of overlap with the integrated bounding box may be determined as the contributing model. This concludes the description of the processing performed by the detection result integration unit 117.

[0076] Next, a detailed description will be given of the processing of the display control unit 118 that performs the detection result display processing in S307 of Fig. 3. This is processing that displays the integrated detection result 143 in the memory 104 on the display unit 106. The display unit 106 may be a display connected to a computer, a display attached to a camera, a display unit of a smartphone, or the like.

[0077] The display control unit 118 may display the input image 137 and superimpose the bounding box represented by the integrated detection result 143 as a rectangular frame on top of the input image 137. At this time, the display frame may be drawn differently based on the model_id of the integrated detection result 143. For example, a different color or line type may be used based on the model_id. This allows the user to easily understand which recognition model was used to detect each detection result. In other words, the user can easily understand how registering a custom model has improved detection performance compared to using only fixed models.

[0078] Furthermore, if the number of false positives increases due to the registration of a custom model, the user can easily determine whether the false positives are due to the fixed model or the custom model. For example, since the user can determine that the number of false positives has increased due to the registration of a custom model while using the camera, the user can choose to unregister the custom model. To unregister a custom model, the user simply performs a process to turn off the custom model activation flag 135 in memory 104.

[0079] 11 is a flowchart showing the procedure of processing executed by the display control unit 118. In S1101, the display control unit 118 displays the input image 137 in the memory 104 on the display unit 106. In S1102, the display control unit 118 acquires one detection result from the integrated detection results 143 in the memory 104. For example, the results may be acquired in the order of id. In S1103, the display control unit 118 determines drawing parameters based on the contribution model represented by model_id of the one detection result acquired in S1102. The drawing parameters here include the color and line type to be drawn.

[0080] For example, if the drawing parameter is color, it may be drawn in red if model_id is 0, and drawn in green if model_id is 1. Alternatively, if the drawing parameter is line type, it may be drawn in a different rectangular frame line type depending on model_id. Any method may be used as long as the drawing representation is distinguished based on the contributing model.

[0081] In S1104, the display control unit 118 superimposes and displays the bounding box of the detection result on the display unit 106 in accordance with the drawing parameters (for example, color, line type, etc.) determined in S1103.

[0082] In S1105, the display control unit 118 determines whether or not all results of the integrated detection result 143 have been displayed. If yes in this step, the process ends. On the other hand, if no in this step, the process returns to S1102.

[0083] As described above, in this embodiment, two models are provided: a fixed model, which is an unchangeable recognition model trained to detect subjects of a predetermined category, and a custom model, which is a customizable recognition model trained to detect subjects of the predetermined category.

[0084] Then, a method for integrating the detection results using the fixed model and the detection results using the custom model (integration of the detection results, not integration of models) is set. The weights can be set on the UI screen using the slider bar described above, or they can be received and set from another device. The detection results are then integrated based on the set integration method, and the integrated detection results are acquired and displayed.

[0085] According to this embodiment, since the detection results obtained from each model are integrated rather than the fixed model and the custom model being integrated, it is possible to detect additional subjects desired by the user through additional learning while maintaining the detection performance before additional learning. Therefore, the user can freely customize the camera while ensuring the detection performance provided by the camera manufacturer.

[0086] In this embodiment, the configuration in which each processing unit and memory are located in a single information processing device has been described, but this is not limiting. For example, the processing units and memory related to learning may be located in a cloud-based computer, and the processing units related to setting the integration method and detecting and displaying may be located in the camera. By configuring in this manner, it is possible to keep the details of the learning process and the data prepared in advance, such as the positive example training data 121 and negative example training data 122, confidential from the user.

[0087] Furthermore, camera manufacturers may adjust the processing within the camera, hoping for accuracy in the detection results of the fixed model 120 for a specific category. On the other hand, a user may want to additionally learn a desired subject in a human face detection category to make it easier to autofocus on the desired subject. In such a case, the human face detection results are actually used in processes other than autofocus within the camera, and the registration of a custom model may change the detection accuracy, resulting in processing that the camera manufacturer did not intend.

[0088] On the other hand, in a predetermined process implemented by a camera manufacturer, the detection process may be performed with the custom model validation flag 135 temporarily turned OFF. This makes it possible to obtain the detection results that the camera manufacturer previously expected. According to the configuration of this embodiment, by changing the custom model validation flag 135 in the memory 104, it is possible to easily switch between using and not using a custom model for detection.

[0089] (Embodiment 2) In the first embodiment, an example was described in which one recognition model detects objects of one category. Also, the fixed model 120 and the custom model 130 were described as being treated individually as recognition models such as the recognition model 201 in Fig. 2. In the second embodiment, an example will be described in which one recognition model is a multi-task model that detects objects of multiple categories. Furthermore, an example will be described in which the fixed model and the custom model are parts of a single multi-task model.

[0090] 12 is a diagram illustrating a recognition model that is a multitask model according to this embodiment. Multitask recognition model 1201 is trained to detect subjects of two categories, a first category and a second category, using a single recognition model. For example, a person's face is set as the first category, and the person's eye of the face is set as the second category, and the single recognition model is trained to simultaneously detect a person's face and a person's eye.

[0091] Of course, this is not a limitation, and categories may be learned to detect the whole body of a person and the whole body of an animal. The number of categories is not limited to two, and may be multiple. Furthermore, it may be possible to register multiple custom models for one category. An input image 1202 is input to the multitask recognition model 1201.

[0092] The multi-task recognition model 1201 is composed of a shared layer 1203 and fixed and custom models for each category. The fixed and custom models for each category output an object likelihood map and an object size map, respectively. In the example of Fig. 12, a first category fixed model 1204, a first category custom model 1205, a second category fixed model 1206, and a second category custom model 1207 are provided.

[0093] The first category fixed model 1204 outputs a first category fixed model subject likelihood map 1208 and a first category fixed model size map 1209. The first category custom model 1205 outputs a first category custom model subject likelihood map 1210 and a first category custom model size map 1211. The second category fixed model 1206 outputs a second category fixed model subject likelihood map 1212 and a second category fixed model size map 1213. The second category custom model 1207 outputs a second category custom model subject likelihood map 1214 and a second category custom model size map 1215.

[0094] The shared layer 1203 is trained simultaneously with the training of the fixed model and is stored in memory in advance. The shared layer 1203 may also be further subdivided and partially shared. Such an example is common in multitask models using neural networks, and therefore a detailed description thereof will be omitted here.

[0095] In this embodiment, the learning unit 111 additionally learns only the custom model portion corresponding to the detection category 129 set by the learning setting unit 110. For example, if the category indicated by the detection category 129 is the first category, only the first category custom model 1205 is learned. Of course, this process may be repeated to additionally learn custom models for multiple categories.

[0096] Each processing unit described in the first embodiment performs processing separately for each category. The memory required for processing may also be managed separately for each category. For example, the custom model registration unit 113 only needs to register custom models related to the category indicated by the detection category 129, and therefore only needs to perform registration processing for the custom model portion of the recognition model 1201 for that category. The display control unit 118 may display the detection results separately for each category.

[0097] This configuration allows for simultaneous detection of multiple categories with less memory and computational overhead than maintaining a separate model for each category to be detected. Furthermore, because the scope of additional custom model training is limited, custom model training is stable and the memory and computational overhead required for training can be reduced. Furthermore, registering a custom model for one category does not affect the detection results for other categories.

[0098] The disclosure of this specification includes the following information processing device, control method for an information processing device, and program. (Item 1) An information processing device that detects a subject from an input image, a storage means for storing a fixed model, which is an unchangeable recognition model trained to detect objects of a predetermined category, and a custom model, which is a customizable recognition model trained to detect objects of the same category as the fixed model; a setting means for setting an integration method for integrating the detection result using the fixed model and the detection result using the custom model; an integration means for integrating the detection results for the input image based on the integration method to obtain an integrated detection result; An information processing device comprising: (Item 2) 2. The information processing device according to item 1, further comprising: a learning unit that learns the custom model by additional learning using the fixed model as an initial value. (Item 3) further comprising a learning setting means for setting the learning means; 3. The information processing device according to item 2, wherein the settings include weighting settings for positive example training data that includes data of the predetermined category and negative example training data that does not include data of the predetermined category. (Item 4) further comprising a display control means for displaying the integrated detection result on a display means; The information processing device described in any one of items 1 to 3, characterized in that the display control means displays the integrated detection results while distinguishing between detection results obtained using the fixed model and detection results obtained using the custom model. (Item 5) 5. The information processing device according to any one of items 1 to 4, wherein the integration method is a method that does not reduce the detection results of the fixed model. (Item 6) 6. The information processing device according to any one of items 1 to 5, wherein the integration method is a method of adding the detection result of the custom model to the detection result of the fixed model. (Item 7) The information processing device described in any one of items 1 to 6 is characterized in that the integration means extracts one or more independent regions from the subject likelihood maps output by the fixed model and the custom model, respectively, where the map value is greater than or equal to a predetermined value, and obtains the coordinates with the maximum value within each independent region as the subject center position. (Item 8) the setting means sets a weight when integrating a detection result using the fixed model and a detection result using the custom model; The information processing device described in any one of items 1 to 7, characterized in that the integration means obtains as the integrated detection result a weighted sum of the detection result using the fixed model and the detection result using the custom model based on the weight set by the setting means. (Item 9) The information processing device described in item 8 is characterized in that the setting means presents a screen to the user for accepting input of the weight, and sets the weight input by the user as a weight when integrating the detection result using the fixed model and the detection result using the custom model. (Item 10) 10. The information processing device according to item 9, wherein the screen includes a slider bar for receiving input of the weight. (Item 11) 11. The information processing device according to any one of items 8 to 10, wherein the weight is a weight of a detection result using the custom model relative to a detection result using the fixed model. (Item 12) Further, a verification means for performing a false positive evaluation of the custom model is provided, The information processing device described in any one of items 1 to 11, characterized in that the verification means obtains an error detection index indicating the difference between the error detection rate of the custom model and the error detection rate of the fixed model as an evaluation value of the custom model, and performs the error detection evaluation based on the evaluation value. (Item 13) Item 13. The information processing device according to item 12, further comprising a determination unit that determines whether or not the custom model can be registered based on the verification result of the verification unit. (Item 14) the custom model is a recognition model additionally trained with a subject that the user does not want to detect, 5. The information processing device according to any one of items 1 to 4, wherein the integration method is a method for reducing the detection results of the fixed model. (Item 15) the fixed model includes a plurality of fixed models respectively corresponding to a plurality of categories; 15. The information processing device according to any one of items 1 to 14, wherein the custom model includes a plurality of custom models respectively corresponding to the plurality of categories. (Item 16) A control method for an information processing device that detects a subject from an input image, comprising: a storage step of storing in a storage means a fixed model, which is an unchangeable recognition model trained to detect objects of a predetermined category, and a custom model, which is a customizable recognition model trained to detect objects of the same category as the fixed model; a setting step of setting an integration method for integrating the detection results using the fixed model and the detection results using the custom model; an integration step of integrating the detection results for the input image based on the integration method to obtain an integrated detection result; 1. A method for controlling an information processing device, comprising: (Item 17) A program for causing a computer to function as the information processing device according to any one of items 1 to 15.

[0099] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0100] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0101] 101, CPU, 102: computer bus, 103: first memory, 104: second memory, 105: input unit, 106: display unit, 107: communication unit

Claims

1. An information processing device that detects a subject from an input image, a storage means for storing a fixed model, which is an unchangeable recognition model trained to detect objects of a predetermined category, and a custom model, which is a customizable recognition model trained to detect objects of the same category as the fixed model; a setting means for setting an integration method for integrating the detection result using the fixed model and the detection result using the custom model; an integration means for integrating the detection results for the input image based on the integration method to obtain an integrated detection result; An information processing device comprising:

2. The information processing apparatus according to claim 1 , further comprising: a learning unit that learns the custom model by additional learning using the fixed model as an initial value.

3. further comprising a learning setting means for setting the learning means; 3. The information processing apparatus according to claim 2, wherein the setting includes a weighting setting for positive example training data that includes data of the predetermined category and negative example training data that does not include data of the predetermined category.

4. further comprising a display control means for displaying the integrated detection result on a display means; 2. The information processing apparatus according to claim 1, wherein the display control means displays the integrated detection results while distinguishing between detection results obtained using the fixed model and detection results obtained using the custom model.

5. The information processing apparatus according to claim 1 , wherein the integration method is a method that does not reduce the detection results of the fixed model.

6. The information processing apparatus according to claim 1 , wherein the integration method is a method of adding the detection result of the custom model to the detection result of the fixed model.

7. The information processing device described in claim 1, characterized in that the integration means extracts one or more independent areas where the map value is greater than or equal to a predetermined value from the subject likelihood maps output by the fixed model and the custom model, and obtains the coordinates with the maximum value within each independent area as the subject center position.

8. the setting means sets a weight when integrating a detection result using the fixed model and a detection result using the custom model; 2. The information processing apparatus according to claim 1, wherein the integration means acquires, as the integrated detection result, a weighted sum of the detection result using the fixed model and the detection result using the custom model based on the weights set by the setting means.

9. The information processing device according to claim 8, characterized in that the setting means presents a screen to the user for accepting input of the weight, and sets the weight input by the user as a weight when integrating detection results using the fixed model and detection results using the custom model.

10. 10. The information processing apparatus according to claim 9, wherein the screen includes a slider bar for receiving input of the weight.

11. The information processing apparatus according to claim 8 , wherein the weight is a weight of the detection result using the custom model relative to the detection result using the fixed model.

12. Further, a verification means for performing a false positive evaluation of the custom model is provided, The information processing device according to claim 1, characterized in that the verification means obtains an error detection index indicating the difference between the error detection rate of the custom model and the error detection rate of the fixed model as an evaluation value of the custom model, and performs the error detection evaluation based on the evaluation value.

13. 13. The information processing apparatus according to claim 12, further comprising a determination unit that determines whether or not the custom model can be registered based on a result of the verification by the verification unit.

14. the custom model is a recognition model additionally trained with a subject that the user does not want to detect, The information processing apparatus according to claim 1 , wherein the integration method is a method for reducing the detection results of the fixed model.

15. the fixed model includes a plurality of fixed models respectively corresponding to a plurality of categories; The information processing apparatus according to claim 1 , wherein the custom model includes a plurality of custom models corresponding to the plurality of categories, respectively.

16. A control method for an information processing device that detects a subject from an input image, comprising: a storage step of storing in a storage means a fixed model, which is an unchangeable recognition model trained to detect objects of a predetermined category, and a custom model, which is a customizable recognition model trained to detect objects of the same category as the fixed model; a setting step of setting an integration method for integrating the detection results using the fixed model and the detection results using the custom model; an integration step of integrating the detection results for the input image based on the integration method to obtain an integrated detection result; 1. A method for controlling an information processing device, comprising:

17. A program for causing a computer to function as the information processing device according to any one of claims 1 to 15.

Images