Learning device, learning method, and learning program

Abstract

To efficiently improve the detection accuracy of object detection models. [Solution] The learning device comprises an acquisition unit, an extraction unit, a generation unit, and a learning unit. The acquisition unit acquires erroneous location information from the detection results, which the object detection model has misdetected, based on a comparison between the location information of the detection result detected from the training image by an object detection model for detecting a predetermined object and the true location information of the predetermined object included as the correct label in the training image. The extraction unit extracts a partial image from the training image that is the region indicated by the erroneous location information and is the region of the erroneous object that the object detection model has incorrectly inferred to be in the same class as the predetermined object. The generation unit generates a learning image by pasting the partial image as an incorrect image onto a predetermined image different from the training image. The learning unit uses the learning image to train an object detection model that detects a predetermined object from an input image.

Description

Technical Field

[0001] The present invention relates to a learning device, a learning method, and a learning program.

Background Art

[0002] Techniques for generating learning data used for training an object detection model that detects an object from an input image have been proposed. For example, techniques for efficiently generating learning data by image synthesis and techniques for generating natural images that can be used as learning data have been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Means for Solving the Problems

[0004] The learning device according to the present application includes: an acquisition unit that acquires error position information that is misdetected by the object detection model from the position information of the detection result detected by the object detection model for detecting a predetermined object and the true position information of the predetermined object included as the correct label in the teacher image; an extraction unit that extracts a partial image from the teacher image, which is a region indicated by the error position information and is a region of an error object that is erroneously inferred by the object detection model to be of the same class as the predetermined object; a generation unit that generates a learning image by pasting the partial image as an incorrect image onto a predetermined image different from the teacher image; and a learning unit that learns an object detection model that detects the predetermined object from an input image using the learning image.

Brief Description of the Drawings

[0005] [Figure 1] Figure 1 is a diagram illustrating the prerequisite technology. [Figure 2] Figure 2 shows an example of a false positive in an AI model. [Figure 3] Figure 3 shows an example of the configuration of an information processing system according to the embodiment. [Figure 4] Figure 4 shows an example of the configuration of a learning device according to an embodiment. [Figure 5] Figure 5 shows an example of the learning device in operation. [Figure 6] Figure 6 shows an example of the difference calculation process according to the embodiment. [Figure 7] Figure 7 shows an example of a method for pasting partial images. [Figure 8] Figure 8 is a hardware configuration diagram showing an example of a computer that implements the functions of the learning device according to this embodiment. [Modes for carrying out the invention]

[0006] Embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0007] The one or more embodiments (including examples, modifications, and applications) described below can each be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in appropriate combination with at least some of the other embodiments. These embodiments may contain novel features that differ from each other. Therefore, these embodiments may contribute to solving different objectives or problems and may produce different effects.

[0008] Furthermore, the proposed technology in this application relates to the training of an object detection model that detects a predetermined object from an input image.

[0009] Furthermore, the task of the object detection model is to detect a given object from an input image, specifically, to predict the location information (coordinate information) of the given object in the input image. More specifically, the task of the object detection model is to simultaneously estimate the coordinates (localization) and type (classification) of the object contained in the input image, and then, through post-processing, output the detection result as a list containing information about the bounding box (rectangular region) (e.g., the x-coordinate of the box's center, the y-coordinate of the box's center, the box's width, the box's height) and the confidence score of the detection result.

[0010] Therefore, in the following embodiments, the positional information detected by the object detection model from the image refers to the positional information of a rectangular region (bounding box) in the image. Furthermore, in the following embodiments, the object detection model is assumed to be an AI model specialized in detecting only one class of objects (e.g., dog class), but the proposed technology in this application is also applicable to AI models capable of detecting multiple classes of objects (e.g., dog class, person class, car class).

[0011] <Embodiment> [1. Introduction] (Prerequisite technology) Challenges in building object detection models include the lack of large-scale labeled datasets and the need for data collection and annotation that can handle various environments. Therefore, methods for efficiently generating training data, such as the conventional techniques described above, have been proposed.

[0012] For example, there is a cut-and-paste method that generates training images to be used as training data for an object detection model by extracting the region of the object to be detected from a captured image of the object and pasting the image onto a specific background image.

[0013] Using FIG. 1, the Cut&Paste method, which is the prior art of the proposed method, will be described. FIG. 1 is a diagram for explaining the prior art. The prior art is divided into a Cut step and a Paste step. The Cut step is shown in FIG. 1(a). On the other hand, the Paste step is shown in FIG. 1(b).

[0014] In the Cut step, from the original image containing the detection target, a detected target image from which the region of the detection target (the dog in the example of FIG. 1(a)) is extracted is cut out.

[0015] In the Paste step, for example, by image synthesis in which the detected target image cut out in the Cut step is pasted onto a randomly acquired background image, a composite image in which the detected target image is superimposed on the background image is generated. Then, as shown in FIG. 1(b), what associates the composite image with the correct label (annotation information AN) based on the detected target image becomes the learning image LIM1 (learning data). Note that the background image used is not limited to images of the same scene and may be images of different scenes.

[0016] In this way, according to the Cut&Paste method, the learning image LIM1 can be efficiently mass-produced, and the set G1 of the learning images LIM1 can be efficiently obtained. Also, as shown in FIG. 1, the learning image LIM1 is given a correct label in which the position information PT of the detection target in the learning image LIM1 and the class information of the detection target are associated as annotation information AN. Here, the position information PT refers to the position coordinates of the bounding box, that is, the rectangular region AR, on the learning image LIM1, which is the true position information tPT.

[0017] Also, as an AI model learned based on the set G1 (dataset) of the learning images LIM1, the first object detection model M1 can be generated. The first object detection model M1 may be a learned AI model learned based on a pre-trained model in order to detect a predetermined object PO (for example, an object belonging to the dog class) from an input image.

[0018] (Background and Problem) Next, the background regarding the first object detection model M1 will be described. For example, the first object detection model M1 can be used for a task of counting a predetermined object PO by detecting the predetermined object PO that has entered the video captured by a surveillance camera. At this time, the detection accuracy of the first object detection model M1 does not necessarily have to be 100%, but since it directly affects the accuracy degradation of the task of the application destination, it is required to avoid false detections as much as possible.

[0019] Here, the false detection by the first object detection model M1 will be described using FIG. 2. FIG. 2 is a diagram showing an example of false detection of an AI model. In FIG. 2, one captured image obtained by shooting with a surveillance camera is used as an input image IN, and the first object detection model M1 performs inference, and the state of the detection result in which a predetermined object PO (in the example of FIG. 2, an object belonging to the dog class) is detected for the input image IN is shown.

[0020] The detection result is the position information PT of the predetermined object PO on the input image IN. In the example of FIG. 2, the rectangular area AR indicated by the position information PT is superimposed on the input image IN. The position information PT here is the position information rPT of the detection result.

[0021] Also, according to the example of FIG. 2, the rectangular area AR1 surrounds the predetermined object PO (an object belonging to the dog class) included in the input image IN, and can be said to be a correct detection result. The rectangular area AR2 also surrounds the predetermined object PO (an object belonging to the dog class) included in the input image IN, and can be said to be a correct detection result. The rectangular area AR3 also surrounds the predetermined object PO (an object belonging to the dog class) included in the input image IN, and can be said to be a correct detection result.

[0022] On the other hand, the rectangular area AR4 surrounds another object AO (that is, an object that does not belong to the dog class) belonging to a class different from the predetermined object PO among the objects included in the input image IN, rather than the predetermined object PO, and can be said to be an incorrect detection result. The other object AO looks and has a shape similar to the predetermined object PO.

[0023] Thus, the first object detection model M1 may misdetect objects that closely resemble the correct object. In such cases, one possible approach is to improve accuracy by having a human determine which of the detection results are false positives, preparing new training data with the falsely detected objects as incorrect (negative samples), and retraining the first object detection model M1.

[0024] However, this method has the problem of being time-consuming, as it requires manually creating training data that includes negative samples every time a false positive occurs. Furthermore, even after going through the trouble of retraining, the first object detection model M1 may start falsely detecting new objects, meaning that the cumbersome work does not disappear.

[0025] (Proposed method) The proposed method in this application was developed in view of the above-mentioned problems, and is an approach that automates the generation of training data, which was previously done manually, and also automates the improvement of the accuracy of the object detection model by repeatedly training the object detection model. Specifically, in the proposed method, a series of operations are automatically executed, including the step of automatically extracting negative samples based on images that the object detection model (for example, the first object detection model M1) has misdetected using a rule-based method, the step of automatically generating training data using a cut-and-paste method, and the step of applying the training data to the task. This series of operations is looped until the accuracy of the object detection model meets the criteria. Furthermore, according to this proposed method, it is possible to improve communication quality and operational efficiency through the use of AI, and since it will become an innovative technological foundation in the telecommunications business, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation." The proposed method will be explained in detail below.

[0026] [2. System Configuration] The configuration of the information processing system 1 will be explained using Figure 3. Figure 3 is a diagram showing an example configuration of the information processing system 1 according to an embodiment. As shown in Figure 3, the information processing system 1 includes an imaging system 2, a detection device 60, and a learning device 100. The imaging system 2, the detection device 60, and the learning device 100 are connected to each other via a predetermined communication network (network N) by wired or wireless means. Note that the information processing system 1 shown in Figure 3 may include multiple imaging systems 2, multiple detection devices 60, and multiple learning devices 100.

[0027] As shown in Figure 3, the imaging system 2 may consist of an imaging device 10, a display control device 11, and a display device 12.

[0028] The imaging device 10 may be an imaging means (camera) installed at any location with a fixed field of view. Furthermore, for example, the imaging device 10 may be installed for security purposes or for the purpose of counting a predetermined object PO. Also, the imaging device 10 may be, for example, an AI camera.

[0029] For example, the display control device 11 superimposes the object detection result information onto the captured image acquired by the imaging device 10, and controls the display device 12 to display the superimposed captured image.

[0030] The display device 12 has a screen that uses, for example, liquid crystal, electroluminescence (EL), or a cathode ray tube (CRT). The display device 12 may support 4K or 8K resolution, or it may be formed by multiple display devices 12. The display device 12 displays the captured image that is controlled to be displayed by the display control device 11.

[0031] The detection device 60 inputs the captured image into the object detection model generated by the learning device 100, causing the object detection model to perform inference to detect a predetermined object PO from the captured image. In this way, the detection device 60 corresponds to a detector that performs object detection using the object detection model.

[0032] For example, the imaging device 10 performs continuous shooting, and when it acquires an image, it uploads the acquired image to the detection device 60. When the detection device 60 receives the uploaded image, it uses an object detection model to perform inference processing to detect a predetermined object PO from the image it has just acquired. Specifically, the detection device 60 inputs the image into the object detection model, causing it to output the position information PT of the predetermined object PO in the image. The detection device 60 may also transmit the position information PT detected from the image to the display control device 11. The display control device 11 draws a rectangular region AR indicating the position information PT of the predetermined object PO on the image it has just acquired. The display control device 11 also controls the display device 12 to display the image including the rectangular region AR. The display device 12 displays the image including the rectangular region AR in accordance with the control by the display control device 11. Through this series of processes, the user can, for example, check the detection results by the object detection model via the display device 12. Furthermore, if the imaging device 10 is fixed in a specific location (i.e., if the object detection model is operated with a camera with a fixed field of view), the learning device 100 can generate the training image LIM1 more effectively by using the background image of that field of view.

[0033] The learning device 100 performs information processing related to the proposed technology of this application. Specifically, the learning device 100 obtains erroneous location information mPT from the detected location information rPT detected by the first object detection model M1 for detecting a predetermined object PO from the training image TD, based on a comparison between the detection result location information rPT detected by the first object detection model M1 from the training image TD and the true location information tPT of the predetermined object PO included as the correct label in the training image TD. The learning device 100 then extracts a partial image pIM from the training image TD, which is the region indicated by the erroneous location information mPT and is the region of the erroneous object that the first object detection model M1 mistakenly inferred to be of the same class as the predetermined object PO. Furthermore, the learning device 100 generates a training image by pasting the partial image pIM as an incorrect image onto a predetermined image different from the training image TD, and uses the training image to train an object detection model that detects the predetermined object PO from the input image.

[0034] For example, the learning device 100 may generate a trained second object detection model M2 by retraining the first object detection model M1 based on training images. Alternatively, the learning device 100 may generate an Xth object detection model Mx with improved detection accuracy by repeating the training process. Furthermore, the Xth object detection model Mx may be deployed to the detection device 60 as an operational model.

[0035] The learning device 100 may be implemented as either a local server or a cloud server with a built-in learning function (AI software).

[0036] [3. Configuration of the learning device] The learning device 100 according to the embodiment will be described using Figure 4. Figure 4 is a diagram showing an example of the configuration of the learning device 100 according to the embodiment. As shown in Figure 4, the learning device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0037] <Communications Department 110> The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). For example, the communication unit 110 transmits and receives information with the imaging system 2 and the detection device 60.

[0038] <Storage section 120> The storage unit 120 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120 may store, for example, data and programs related to information processing according to the embodiment. As shown in Figure 4, the storage unit 120 may have a training image data storage unit 121 and a learning image data storage unit 122.

[0039] <Control Unit 130> The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs (for example, the generation program according to the embodiment) stored in the memory device inside the learning device 100 using RAM as the working area. Alternatively, the control unit 130 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0040] As shown in Figure 4, the control unit 130 includes an acquisition unit 131, a detection unit 132, a calculation unit 133, an extraction unit 134, a generation unit 135, and a learning unit 136, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 4, and other configurations are also possible as long as they perform the information processing described later. Also, the connection relationships of the various processing units in the control unit 130 are not limited to the connection relationships shown in Figure 4, and other connection relationships are also possible.

[0041] <Acquisition part 131> The acquisition unit 131 acquires information necessary for the information processing according to the embodiment. For example, the acquisition unit 131 acquires error location information mPT. The acquisition unit 131 also acquires training images TD. It acquires a dataset DS composed of training images TD. The dataset DS may be an open dataset that is publicly available for AI development and model training. The dataset DS may be pre-stored in the training image data storage unit 121.

[0042] <Detection unit 132> The detection unit 132 may use an object detection model to detect the position information PT of a predetermined object PO in the input image. For example, the detection unit 132 may input the input image to the object detection model and obtain the position information PT output from the object detection model as the detection result.

[0043] <Calculation Unit 133> The calculation unit 133 executes the difference calculation process according to the embodiment. Specifically, the calculation unit 133 calculates the erroneous location information mPT that the first object detection model M1 misdetected from the detected location information rPT, which is included as the correct label (annotation information AN) in the training image TD, based on a comparison between the location information rPT detected by the first object detection model M1 for detecting a predetermined object PO from the training image TD and the true location information tPT of the predetermined object PO. As a result, the acquisition unit 131 acquires the erroneous location information mPT. In the following embodiment, as an example of comparing the detected location information rPT with the true location information tPT, the process of calculating the difference between the detected location information rPT and the true location information tPT will be described. However, the calculation unit 133 does not necessarily need to calculate the difference. For example, the calculation unit 133 may calculate the difference in the number of pixels by comparing the image of the part indicated by the detected location information rPT with the image of the part indicated by the true location information tPT.

[0044] As explained in Figure 1, the first object detection model M1 is a pre-trained AI model trained using a training image LIM1, which is based on a composite image in which an image of an object containing a predetermined object PO as the detection target is pasted onto a background image as the correct label.

[0045] <Extraction part 134> The extraction unit 134 performs the extraction process according to the embodiment. The extraction unit 134 extracts a partial image pIM from the training image TD, which is a rectangular region AR indicated by the error position information mPT, and which surrounds another object AO that the first object detection model M1 has incorrectly inferred to be of the same class as a predetermined object PO as an error object.

[0046] <Generation section 135> The generation unit 135 generates a training image LIM2 by attaching a partial image pIM as an incorrect image to a predetermined image different from the training image TD. For example, the partial image pIM may be attached to the composite image used to generate the first object detection model M1, which is a predetermined image different from the training image TD. The partial image pIM as an incorrect image can be understood as a negative sample. Specifically, the process of attaching a partial image pIM as an incorrect image means attaching it without adding annotation information AN.

[0047] More specifically, the generation unit 135 may attach partial image pIMs to the annotated training image LIM1 used to generate the first object detection model M1. In other words, the generation unit 135 may attach partial image pIMs to the training image LIM1 included in the dataset G1, which is the dataset used to generate the first object detection model M1. The training image LIM1, training image LIM2, and the background image from which training image LIM1 was derived may be stored in the training image data storage unit 122.

[0048] Furthermore, the generation unit 135 may paste the partial image pIM in a state that has been processed to a state different from that at the time of extraction. For example, the generation unit 135 may paste the partial image pIM in a state that has been changed to a state different from that at the time of extraction by the extraction unit 134, such as size, color, and orientation. In addition, to enable such processing, a size range in which the size of the partial image pIM can be changed, a chromaticity range in which the color of the partial image pIM can be changed, and an angle range in which the orientation of the partial image pIM can be changed may be pre-set for the learning device 100.

[0049] <Learning Section 136> The learning unit 136 uses training images LIM2 to train an object detection model that detects a predetermined object PO from an input image. Specifically, the learning unit 136 generates a trained object detection model as an AI model trained on a set of training images LIM2 (dataset) G2.

[0050] For example, the learning unit 136 may perform retraining on the first object detection model M1 based on the training image LIM2 to generate a second object detection model M2 in which detection accuracy has been improved by learning not to mistakenly detect other objects AO as being of the same class as a predetermined object PO.

[0051] As another example, the learning unit 136 may perform training based on the training image LIM2 on the pre-trained model used to train the first object detection model M1, thereby training it not to mistakenly detect other objects AO as being in the same class as a predetermined object PO, and generating a second object detection model M2 with improved detection accuracy.

[0052] As mentioned above, the learning device 100 may generate an Xth object detection model Mx with improved detection accuracy by repeating the learning process. In this example, the object detection model may be retrained using the training image LIM2, and the final Xth object detection model Mx to be deployed may be generated through a retraining loop.

[0053] Through this iterative learning process, the acquisition unit 131 acquires error location information mPT each time it is learned, the extraction unit 134 extracts a partial image pIM based on the newly acquired error location information mPT, and the generation unit 135 generates a training image LIM2 based on the newly extracted partial image pIM. Then, the learning unit 136 retrains the object detection model using the newly generated training image LIM2 to generate the final Xth object detection model Mx.

[0054] [4. Examples of the learning device's operation] Figure 5 shows an example of the operation of the learning device 100. Figure 5 shows the operation procedure of the learning device 100 in information processing according to the embodiment. From Figure 5 onward, the first object detection model M1 will be described as an AI model that detects "dogs" as the target object. That is, the first object detection model M1 will be an AI model that has been trained to detect a predetermined object PO belonging to the dog class from an input image.

[0055] Furthermore, in the information processing according to the embodiment, as described above, a loop occurs in which learning is repeated using the training image LIM2, and Figure 5 shows the first loop, i.e., the initial learning process.

[0056] First, the acquisition unit 131 acquires a dataset DS consisting of training images TD (step S11). Figure 5(a) shows an example of a training image DT used as input. In the example in Figure 5(a), the training image TD is already assigned annotation information AN, which is the correct label that associates the true position information tPT of a given object PO with the class information of the given object PO.

[0057] For example, annotation information AN1, AN2, and AN3 each contain the true position information PT of a given object PO on the training image TD, and class information classifying the given object PO as a "dog". Note that the training image TD also contains other objects AO (dog-shaped robot) that resemble the given object PO, but since these other objects are not "dogs", annotation information AN is not assigned to them.

[0058] The detection unit 132 inputs the dataset DS into the first object detection model M1 to perform inference for detecting a predetermined object PO (step S12). In this way, the detection unit 132 can improve the efficiency of learning by using the first object detection model M1, which is a trained AI model, in the information processing according to the embodiment. On the other hand, if efficiency is not a consideration, a pre-trained model may be used instead of the first object detection model M1.

[0059] Returning to the explanation, the first object detection model M1 performs inference in accordance with the control of the detection unit 132 (step S13) and outputs a detection result (step S14). The detection result may include position information rPT, which is the position information PT of a predetermined object PO on the training image TD, and a confidence score SC. The confidence score SC here is an index value of the likelihood that the predetermined object PO is enclosed by the rectangular region AR indicated by the position information rPT of the detection result, and the higher the value, the higher the probability that the predetermined object PO is located within the rectangular region AR.

[0060] As a result, the detection unit 132 can obtain a training image TD' to which the detection results have been applied. Figure 5(b) shows an example of a training image TD' to which the detection results have been applied. In the example in Figure 5(b), the training image TD' is associated with rectangular regions AR1, AR2, AR3, and AR4 as positional information rPT.

[0061] However, at this point, it is unclear whether the first object detection model M1 is misidentifying other objects AO (i.e., objects not belonging to the dog class) that belong to a different class than the given object PO as belonging to the dog class. Specifically, it is unclear whether any of the rectangular regions AR (rectangular region AR1, rectangular region AR2, rectangular region AR3, rectangular region AR4) indicated by the detected position information rPT is the misidentified erroneous position information mPT.

[0062] Therefore, the calculation unit 133 performs a difference calculation process to compare the training image TD and the training image TD' and calculate the difference (step S15). Specifically, the calculation unit 133 calculates the difference between the true position information tPT contained in the training image TD and the detected position information rPT.

[0063] If the acquisition unit 131 calculates a difference between the true location information tPT and the detected location information rPT, it acquires the erroneous location information mPT, which was incorrectly detected by the first object detection model M1, from the detected location information rPT, based on that difference (step S16).

[0064] Here, we will explain how to obtain error location information mPT by difference calculation processing using Figure 6. Figure 6 is a diagram showing an example of difference calculation processing according to the embodiment.

[0065] Figure 6 shows an example in which the difference between the detected position information rPT and the true position information tPT is calculated by subtracting the training image TD from the training image TD'. In this example, the calculation unit 133 calculates the difference by subtracting the information of the true position information tPT from the information of the position information rPT.

[0066] As an example, the calculation unit 133 may assign a predetermined value (for example, "10") to each image portion within each rectangular region AR (rectangular regions AR1 to AR4) indicated by the position information rPT in the training image TD'. Similarly, the calculation unit 133 may assign a predetermined value (for example, "10") to each image portion within each rectangular region AR indicated by the true position information tPT contained in the annotation information AN (annotation information AN1 to AN3) in the training image TD'.

[0067] In this state, the calculation unit 133 subtracts the teacher image TD from the teacher image TD'. In this case, in the example of Figure 6, only "10" assigned to the image portion within the rectangular region AR4 remains as a positive value. Therefore, in the example of Figure 6, the calculation unit 133 can calculate the position information rPT corresponding to the rectangular region AR4 as a difference from the detected position information rPT. As a result, the acquisition unit 131 acquires the position information rPT corresponding to the rectangular region AR4 as erroneous position information mPT.

[0068] In this way, the learning device 100 can dynamically determine false detections. In the example shown in Figure 6, the learning device 100 can dynamically determine that the first object detection model M1 may incorrectly infer that another object AO, which is similar to a given object PO, belongs to the same dog class as the given object PO.

[0069] Furthermore, the calculation unit 133 may perform the reverse calculation by subtracting the teacher image TD' from the teacher image TD after assigning predetermined values ​​as described above. In this case, only "10," which is assigned to the image portion within the rectangular region AR4, will remain as a negative value.

[0070] Returning to Figure 5, the extraction unit 134 extracts a partial image pIM from the training image TD' based on the rectangular region AR indicated by the error location information mPT, which surrounds another object AO that the first object detection model M1 has incorrectly inferred to be the same dog class as a predetermined object PO (step S16). As shown in Figure 5, the extraction unit 134 may extract a partial image pIM from the training image TD' within the rectangular region AR indicated by the error location information mPT.

[0071] Next, the generation unit 135 performs superposition synthesis, generating training images LIM2 using partial images pIM as incorrect images (step S17). For example, the generation unit 135 may attach partial images pIM as incorrect images to the annotated training images LIM1 used to generate the first object detection model M1. For example, the generation unit 135 may attach partial images pIM to training images LIM1 included in the dataset G1 used to generate the first object detection model M1. As a result, Figure 5 shows an example in which the generation unit 135 has obtained the dataset G2 of training images LIM2.

[0072] In this way, the generation unit 135 can improve the efficiency of learning by reusing the training image LIM1 that has already been used for learning. On the other hand, if efficiency is not a consideration, for example, the background image used in the Cut & Paste method in Figure 1 may be used.

[0073] The generation unit 135 may paste the partial image pIM in a state that has been processed to a state different from that at the time of extraction. For example, the generation unit 135 may paste the partial image pIM in a state that has been changed to a state different from that at the time of extraction by the extraction unit 134, such as size, color, orientation, etc.

[0074] Furthermore, Figure 5 shows an example where one partial image pIM is extracted, but multiple partial image pIMs may also be extracted. For example, if multiple training images TD are input in a single inference, or if multiple other objects AO are incorrectly inferred to belong to the dog class, multiple partial image pIMs may be extracted. In such cases where multiple partial image pIMs are extracted, the generation unit 135 may attach each partial image pIM to a separate training image LIM1, or it may attach them so that each partial image pIM coexists in a single training image LIM1.

[0075] For example, suppose a difference in the detection accuracy of the generated object detection model occurs when each partial image pIM is attached to a separate training image LIM1 during the learning loop, compared to when each partial image pIM is attached to a single training image LIM1 so that they coexist. In such a case, the learning device 100 may adopt the attachment method that allows for greater accuracy improvement in subsequent learning.

[0076] Furthermore, pasting a partial image pIM as an incorrect image means pasting it as is without adding annotation information AN. Specifically, the generation unit 135 may paste the partial image pIM as is without providing a rectangular region AR based on the true position information tPT. This allows the AI ​​model to learn that the object indicated by the partial image pIM does not belong to the dog class.

[0077] Returning to the explanation, finally, the learning unit 136 learns an object detection model that detects a predetermined object PO from the input image using the training image LIM2 (step S18). For example, the learning unit 136 may generate a second object detection model M2 with improved detection accuracy by having the first object detection model M1 retrained based on the training image LIM2, thereby preventing the false detection of other objects AO as belonging to the same class as the predetermined object PO. This allows the learning unit 136 to efficiently improve detection accuracy.

[0078] On the other hand, if efficiency is not a consideration, the learning unit 136 may generate a second object detection model M2 by performing training based on the training image LIM2 on the pre-trained model used to train the first object detection model M1.

[0079] Up to this point, using Figure 5, we have explained an example in which a second object detection model M2 is generated by training the first object detection model M1 as the first training loop. However, this series of training steps S11 to S18 may be repeated a predetermined number of times.

[0080] For example, in the second loop iteration, steps S11 to S18 may be executed using the second object detection model M2 instead of the first object detection model M1, thereby generating a third object detection model M3 with improved accuracy. Similarly, in the third loop iteration, steps S11 to S18 may be executed using the third object detection model M3 instead of the second object detection model M2, thereby generating a fourth object detection model M4 with improved accuracy.

[0081] Thus, the learning device 100 may generate an improved Xth object detection model Mx by looping through a series of learning steps S11 to S18. The number of loops is not limited and may be specified, for example, by the user operating the object detection model.

[0082] Furthermore, the same dataset does not need to be used in each loop. For example, the training images TD included in the dataset DS may be divided into multiple sets, and the detection unit 132 may input different groups of dataset DS into the object detection model in step S11 of each loop. Similarly, the training images LIM1 included in the set G1 may also be divided into multiple sets, and the training unit 136 may train the object detection model with different sets of training images LIM1 in step S18 of each loop.

[0083] [5. Prioritizing use based on confidence score] As explained in Figure 5, in each loop, the detection result output by the object detection model (the first object detection model M1 in the first loop) includes a confidence score SC in addition to the location information rPT. Therefore, the acquisition unit 131 may further acquire the confidence score SC. Furthermore, if multiple partial image pIMs are extracted, the generation unit 135 may paste the partial image pIMs whose confidence score SC satisfies predetermined conditions from among the multiple partial image pIMs. For example, partial image pIMs with higher confidence scores SC may be pasted preferentially. This point will be explained using Figure 7.

[0084] Figure 7 shows an example of a method for pasting partial images (pIMs). Figure 7 shows an example in which partial image pIM1, partial image pIM2, and partial image pIM3 are extracted by the extraction unit 134. Partial images pIM1, pIM2, and pIM3 may be extracted from one training image TD at any number of loop iterations, or they may be extracted from multiple training images TD at any number of loop iterations.

[0085] Furthermore, as shown in the example in Figure 7, the confidence score SC of the location information rPT used to extract partial image pIM1 was "0.8", the confidence score SC of the location information rPT used to extract partial image pIM2 was "0.6", and the confidence score SC of the location information rPT used to extract partial image pIM3 was "0.3". In this example, the confidence score SC corresponding to partial image pIM1 is the highest. From this, it is suggested that the object detection model used in this detection (the first object detection model M1 in the first loop) is most likely to mistakenly infer that the object shown in partial image pIM1 is in the same class as the given object PO, among the objects shown in partial image pIM1, partial image pIM2, and partial image pIM3, respectively. Therefore, it is thought that improving accuracy will be possible by training the object detection model to focus more on the fact that objects that it is more likely to misdetect are not in the same class as the given object PO.

[0086] Therefore, as shown in Figure 7, the generation unit 135 may prioritize pasting partial image pIM1 over partial image pIM2 and partial image pIM3. Prioritized pasting, as used here, includes patterns such as pasting a larger number of partial image pIM1 compared to partial image pIM2 and pIM3, pasting partial image pIM2 and pIM3 as well, but pasting more than a certain number of partial image pIM1, or excluding partial image pIM2 and pIM3 and pasting only partial image pIM1. Such pasting methods may be executed in each loop.

[0087] [6. Hardware Configuration] The learning device 100 according to the embodiment may be implemented by a computer 1000 having a configuration such as that shown in Figure 8. Figure 8 is a hardware configuration diagram showing an example of a computer that implements the functions of the learning device 100 according to the embodiment. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0088] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, controlling various components. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0089] The HDD1400 stores programs executed by the CPU1100, as well as data used by such programs. The communication interface1500 receives data from other devices via a predetermined communication network and sends it to the CPU1100, and transmits data generated by the CPU1100 to other devices via the predetermined communication network.

[0090] The CPU 1100 controls output devices such as displays and input devices such as keyboards via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0091] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0092] For example, when the computer 1000 functions as a learning device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0093] [7. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0094] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0095] Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

[0096] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, including the embodiments described in the section on the present invention. [Explanation of symbols]

[0097] 1. Information Processing System 2. Imaging System 10 Imaging device 11 Display control device 12 Display device 60 Detection device 100 Learning Devices 130 Control Unit 131 Acquisition Department 132 Detection unit 133 Calculation Section 134 Extraction part 135 Generation part 136 Learning Department

Claims

1. An acquisition unit acquires erroneous location information that the object detection model misdetected from the detection result, based on a comparison between the location information of the detection result detected from a training image by an object detection model for detecting a predetermined object and the true location information of the predetermined object included as a correct label in the training image. An extraction unit extracts a partial image from the aforementioned training image that is a region indicated by the error location information and is a region of an error object that the object detection model has incorrectly inferred to be of the same class as the predetermined object. A generation unit generates a training image by pasting the partial image as an incorrect image onto a predetermined image different from the aforementioned training image, A learning unit that learns an object detection model for detecting a predetermined object from an input image using the aforementioned training images, A learning device equipped with the following features.

2. The object detection model is a trained object detection model that has been trained based on a composite image in which the image of the predetermined object is attached as the correct label, The generation unit further pastes the partial image onto the composite image. The learning device according to claim 1.

3. The generation unit pastes the partial image in a state that has been processed to a state different from that at the time of extraction. The learning device according to claim 1.

4. If multiple partial images are extracted, the generation unit either pastes the multiple different partial images onto one predetermined image, or pastes each of the multiple different partial images individually onto a different predetermined image. The learning device according to claim 1.

5. The object detection model is retrained using the training images. The acquisition unit acquires the error location information each time learning is performed. The extraction unit extracts the partial image based on the erroneous position information acquired again. The generation unit generates the training image based on the extracted partial image again. The learning unit retrains the object detection model using the newly generated training images. The learning device according to claim 1.

6. The acquisition unit further acquires a confidence score, which is an index value of the likelihood that the predetermined object is located within the area indicated by the location information of the detection result, each time it performs learning. The generation unit generates the training image by pasting the partial image whose confidence score satisfies predetermined conditions onto the predetermined image. The learning device according to claim 5.

7. The generation unit preferentially attaches to the predetermined image the partial images with higher confidence scores among the partial images. The learning device according to claim 6.

8. A learning method performed by a learning device, An acquisition step to acquire erroneous location information that the object detection model misdetected from the detection result location information, based on the difference between the location information of the detection result detected from the training image by the object detection model for detecting a predetermined object and the true location information of the predetermined object, which is included as the correct label in the training image; Extraction step of extracting a partial image from the training image that is a region indicated by the error location information and is a region of an error object that the object detection model has incorrectly inferred to be of the same class as the predetermined object, A generation step of generating a training image by pasting the partial image as an incorrect image onto a predetermined image different from the aforementioned training image, A learning step in which an object detection model is trained to detect a predetermined object from an input image using the aforementioned training images, Learning methods that include this.

9. An acquisition procedure for acquiring erroneous location information that was misdetected by the object detection model from the detection results, based on the difference between the location information of the detection result detected by the object detection model from a training image and the true location information of the predetermined object included as the correct label in the training image. Extraction procedure for extracting a partial image from the aforementioned training image that is a region indicated by the error location information and is a region of an error object that the object detection model incorrectly inferred to be of the same class as the predetermined object; A generation procedure for generating training images by pasting the partial image as an incorrect image onto a predetermined image different from the aforementioned training image, A learning procedure for training an object detection model that detects a predetermined object from an input image using the aforementioned training images, A learning program to get a computer to execute something.

Images