Learning device, learning method, and learning program

Abstract

To efficiently improve the detection accuracy of an object detection model. [Solution] The learning device includes an acquisition unit, an extraction unit, a generation unit, and a learning unit. The acquisition unit acquires erroneous position information from the position information of the detection result, which is erroneously detected by the object detection model, based on a comparison between position information of the detection result detected from a teacher image by an object detection model for detecting a predetermined object and true position information of the predetermined object included as a correct label in the teacher image. The extraction unit extracts a partial image from the teacher image, which is an area indicated by the erroneous position information and is an area of ​​an erroneous object that the object detection model erroneously inferred to be in the same class as the predetermined object. The generation unit generates learning images by pasting the partial image as an incorrect image into a predetermined image different from the teacher image. The learning unit uses the learning images to learn an object detection model that detects the predetermined object from an input image.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, and a learning program. [Background technology]

[0002] Methods for generating training data to be used in training an object detection model that detects objects from input images have been proposed. For example, methods for efficiently generating training data by image synthesis and methods for generating natural images that can be used as training data have been proposed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-149086 [Patent Document 2] Japanese Patent Application Publication No. 2023-028298 Summary of the Invention [Means for solving the problem]

[0004] The learning device of the present application includes an acquisition unit that acquires erroneous position information from the position information of the detection result, which is detected by an object detection model for detecting a specified object from a teacher image, based on a comparison between the position information of the detection result detected by the object detection model and the true position information of the specified object included as a correct label in the teacher image; an extraction unit that extracts a partial image from the teacher image, which is an area indicated by the erroneous position information and is an area of ​​an erroneous object that the object detection model has erroneously inferred to be in the same class as the specified object; a generation unit that generates a learning image by pasting the partial image as an incorrect image to a specified image different from the teacher image; and a learning unit that uses the learning image to learn an object detection model that detects the specified object from an input image. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a diagram illustrating the underlying technology. [Figure 2] FIG. 2 is a diagram illustrating an example of a false positive in an AI model. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 5] FIG. 5 illustrates an example of the operation of the learning device. [Figure 6] FIG. 6 is a diagram illustrating an example of the difference calculation process according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a partial image pasting technique. [Figure 8] FIG. 8 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the learning device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0006] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0007] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0008] Furthermore, the proposed technology according to the present application relates to learning of an object detection model that detects a predetermined object from an input image.

[0009] The task of the object detection model is to detect a predetermined object from an input image, specifically, to predict the position information (coordinate information) of the predetermined object in the input image. More specifically, the task of the object detection model is to simultaneously estimate the coordinates (localization) and type (classification) of the object contained in the input image, and perform post-processing to output a list of information on the bounding box (rectangular area) (for example, the x-coordinate of the center of the box, the y-coordinate of the center of the box, the width of the box, and the height of the box), a confidence score of the detection result, etc.

[0010] Therefore, in the following embodiments, the position information detected from an image by an object detection model indicates the position information of a rectangular area (bounding box) in the image. Also, in the following embodiments, the object detection model is assumed to be an AI model specialized in detecting objects of only one class (e.g., dog class), but the proposed technology of the present application is also applicable to AI models capable of detecting objects of multiple classes (e.g., dog class, person class, car class).

[0011] <Embodiment> 1. Introduction (Prerequisite technology) Challenges in building object detection models include the lack of large-scale labeled datasets and the need to collect and annotate data for various environments. For these reasons, methods for efficiently generating training data, such as the conventional techniques mentioned above, have been proposed.

[0012] For example, there is a Cut & Paste method in which a detection object image, in which the area of ​​the detection object is extracted from a photographed image containing the detection object, is pasted onto a specific background image to generate a learning image to be used as learning data for an object detection model.

[0013] The Cut & Paste method, which is the underlying technology of the proposed method, is explained using Figure 1. Figure 1 is a diagram explaining the underlying technology. The underlying technology is divided into a Cut step and a Paste step, with Figure 1(a) showing the Cut step, and Figure 1(b) showing the Paste step.

[0014] In the Cut step, a detection object image in which the region of the detection object (a dog in the example of FIG. 1(a)) is extracted is cut out from the original image including the detection object.

[0015] In the Paste step, for example, a composite image is generated in which the detection object image cut out in the Cut step is pasted onto a randomly acquired background image. Then, as shown in FIG. 1(b), the composite image is associated with a correct label (annotation information AN) based on the detection object image, resulting in a training image LIM1 (training data). The background images used are not limited to images of the same scene, but may be images of different scenes.

[0016] In this way, the Cut & Paste method enables efficient mass production of training images LIM1, and efficient acquisition of a set G1 of training images LIM1. Furthermore, as shown in Fig. 1, the training images LIM1 are assigned annotation information AN, which is a correct label that associates position information PT of a detected object in the training image LIM1 with class information of the detected object. The position information PT here refers to the position coordinates of a bounding box, i.e., a rectangular area AR, on the training image LIM1, and is the true position information tPT.

[0017] Furthermore, a first object detection model M1 can be generated as an AI model trained based on a set G1 (dataset) of training images LIM1. The first object detection model M1 may be a trained AI model trained based on a pre-trained model to detect a predetermined object PO (e.g., an object belonging to a dog class) from an input image.

[0018] (Background / Issues) Next, the background of the first object detection model M1 will be explained. For example, the first object detection model M1 can be used in a task of counting predetermined objects PO by detecting them in an image captured by a surveillance camera. In this case, the detection accuracy of the first object detection model M1 does not necessarily need to be 100%, but it is required to avoid false detection as much as possible because it directly leads to a decrease in the accuracy of the task to which it is applied.

[0019] Here, false detection by the first object detection model M1 will be explained using Fig. 2. Fig. 2 is a diagram showing an example of false detection by an AI model. Fig. 2 shows a detection result in which a predetermined object PO (an object belonging to a dog class in the example of Fig. 2) is detected from an input image IN, which is an image captured by a surveillance camera, by performing inference using the first object detection model M1.

[0020] The detection result is position information PT of the predetermined object PO on the input image IN, and in the example of Fig. 2, a rectangular area AR indicated by the position information PT is superimposed on the input image IN. The position information PT here refers to position information rPT of the detection result.

[0021] 2, rectangular area AR1 surrounds a predetermined object PO (an object belonging to the dog class) included in the input image IN, and can be considered a correct detection result. Rectangular area AR2 also surrounds a predetermined object PO (an object belonging to the dog class) included in the input image IN, and can be considered a correct detection result. Rectangular area AR3 also surrounds a predetermined object PO (an object belonging to the dog class) included in the input image IN, and can be considered a correct detection result.

[0022] On the other hand, rectangular area AR4 does not enclose the predetermined object PO (an object belonging to the dog class) among the objects contained in the input image IN, but rather encloses another object AO (i.e., an object not belonging to the dog class) that belongs to a different class from the predetermined object PO, and can be said to be an incorrect detection result. The other object AO is similar in appearance and shape to the predetermined object PO.

[0023] In this way, the first object detection model M1 may erroneously detect an object that closely resembles a correct object. In such cases, one possible approach is to have a human determine which of the detection results are erroneous detections, prepare new training data in which the erroneously detected objects are treated as incorrect answers (negative samples), and retrain the first object detection model M1 with the new training data, thereby improving accuracy.

[0024] However, this method has the problem that it is necessary to manually create training data including negative samples every time a false positive occurs, which is time-consuming. Moreover, even if the first object detection model M1 is retrained through the effort, it may still falsely detect a new object, which is a problem that the tedious work remains.

[0025] (Proposed method) The proposed method, developed in consideration of the above-mentioned problems, automates the manual process of generating training data and automatically improves the accuracy of the object detection model by repeatedly training the object detection model. Specifically, the proposed method automatically executes a series of steps: automatically extracting negative samples based on rules from images misdetected by the object detection model (e.g., the first object detection model M1), automatically generating training data using a cut-and-paste method, and applying the training data to a task. This series of steps is repeated until the accuracy of the object detection model meets the required standards. Furthermore, the proposed method utilizes AI to improve communication quality and streamline operations, providing an innovative technological foundation for the telecommunications industry and contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience." The following describes the proposed method in detail.

[0026] [2. System Configuration] The configuration of the information processing system 1 will be described using Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. As shown in Fig. 3, the information processing system 1 includes an imaging system 2, a detection device 60, and a learning device 100. The imaging system 2, the detection device 60, and the learning device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication. Note that the information processing system 1 shown in Fig. 3 may include multiple imaging systems 2, multiple detection devices 60, and multiple learning devices 100.

[0027] As shown in FIG. 3, the imaging system 2 may be configured with an imaging device 10, a display control device 11, and a display device 12.

[0028] The imaging device 10 may be an imaging means (camera) installed at any location with a fixed angle of view. For example, the imaging device 10 may be installed for security purposes or for the purpose of counting predetermined objects PO. The imaging device 10 may be, for example, an AI camera.

[0029] For example, the display control device 11 superimposes information on the object detection result on the captured image acquired by the imaging device 10, and controls the display device 12 to display the superimposed captured image.

[0030] The display device 12 has a screen using, for example, a liquid crystal display, an electro-luminescence (EL), a cathode ray tube (CRT), etc. The display device 12 may be compatible with 4K or 8K, or may be formed by a plurality of display devices 12. The display device 12 displays the captured image controlled to be displayed by the display control device 11.

[0031] The detection device 60 inputs a captured image into the object detection model generated by the learning device 100, causing the object detection model to perform inference for detecting a predetermined object PO from the captured image. In this way, the detection device 60 corresponds to a detector that performs object detection using the object detection model.

[0032] For example, the imaging device 10 takes continuous photographs, and when a photographed image is acquired, uploads the acquired photographed image to the detection device 60. When the detection device 60 accepts the upload of the photographed image, it uses an object detection model to perform inference processing to detect a predetermined object PO from the currently acquired photographed image. Specifically, the detection device 60 inputs the photographed image into the object detection model, thereby outputting position information PT of the predetermined object PO in the photographed image. The detection device 60 may also transmit the position information PT detected from the photographed image to the display control device 11. The display control device 11 renders a rectangular area AR indicating the position information PT of the predetermined object PO for the currently acquired photographed image. The display control device 11 also controls the display device 12 to display the photographed image including the rectangular area AR. The display device 12 displays the photographed image including the rectangular area AR in accordance with the control of the display control device 11. Through this series of processing, the user can confirm the detection results using the object detection model, for example, via the display device 12. In addition, when the imaging device 10 is fixed in a specific location (i.e., when the object detection model is operated using a camera with a fixed angle of view), the learning device 100 can more effectively generate the learning image LIM1 by using a background image with that angle of view.

[0033] The learning device 100 performs information processing related to the proposed technology of the present application. Specifically, the learning device 100 compares position information rPT of the detection result detected from a teacher image TD by a first object detection model M1 for detecting a predetermined object PO with true position information tPT of the predetermined object PO included as a correct label in the teacher image TD, and acquires erroneous position information mPT of the detection result position information rPT that is erroneously detected by the first object detection model M1. The learning device 100 then extracts a partial image pIM from the teacher image TD, which is a region indicated by the erroneous position information mPT and is a region of an erroneous object that the first object detection model M1 erroneously inferred to be in the same class as the predetermined object PO. Furthermore, the learning device 100 generates training images by pasting the partial image pIM as an incorrect image to a predetermined image other than the teacher image TD, and uses the training images to train an object detection model that detects the predetermined object PO from an input image.

[0034] For example, the learning device 100 may generate a trained second object detection model M2 by retraining the first object detection model M1 based on training images. Furthermore, the learning device 100 may generate an Xth object detection model Mx with improved detection accuracy by repeating the learning process. Furthermore, the Xth object detection model Mx may be deployed to the detection device 60 as an operational model.

[0035] The learning device 100 may be implemented as either a local server or a cloud server with a built-in learning function (AI software).

[0036] 3. Configuration of the learning device The learning device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the learning device 100 according to the embodiment. As shown in Fig. 4, the learning device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0037] <Communication Unit 110> The communication unit 110 is realized by, for example, a network interface card (NIC), etc. For example, the communication unit 110 transmits and receives information to and from the imaging system 2 and the detection device 60.

[0038] <Storage section 120> The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 may store, for example, data and programs related to the information processing according to the embodiment. As shown in FIG. 4, the storage unit 120 may include a teacher image data storage unit 121 and a learning image data storage unit 122.

[0039] <Control unit 130> The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing various programs (e.g., the generation program according to the embodiment) stored in a storage device inside the learning device 100 using RAM as a work area. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0040] As shown in Fig. 4, control unit 130 has an acquisition unit 131, a detection unit 132, a calculation unit 133, an extraction unit 134, a generation unit 135, and a learning unit 136, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 130 is not limited to the configuration shown in Fig. 4, and may have other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between each processing unit included in control unit 130 is not limited to the connection relationship shown in Fig. 4, and may be other connection relationships.

[0041] <Acquisition part 131> The acquisition unit 131 acquires information necessary for the information processing according to the embodiment. For example, the acquisition unit 131 acquires error position information mPT. The acquisition unit 131 also acquires teacher images TD. The acquisition unit 131 acquires a dataset DS composed of the teacher images TD. The dataset DS may be an open dataset that is publicly available for AI development and model training. The dataset DS may be stored in advance in the teacher image data storage unit 121.

[0042] <Detection unit 132> The detection unit 132 may use an object detection model to detect position information PT of a predetermined object PO in an input image. For example, the detection unit 132 may input the input image to the object detection model, and obtain position information PT output from the object detection model as a detection result.

[0043] <Calculation unit 133> The calculation unit 133 executes a difference calculation process according to the embodiment. Specifically, the calculation unit 133 calculates erroneous position information mPT, which is erroneously detected by the first object detection model M1, from the position information rPT of the detection result, based on a comparison between the position information rPT detected from the teacher image TD by the first object detection model M1 for detecting the predetermined object PO and the true position information tPT of the predetermined object PO included as a correct label (annotation information AN) in the teacher image TD. As a result, the acquisition unit 131 acquires the erroneous position information mPT. In the following embodiment, as an example of the comparison between the position information rPT of the detection result and the true position information tPT, a process of calculating a difference between the position information rPT of the detection result and the true position information tPT will be described. However, the calculation unit 133 does not necessarily need to calculate the difference. For example, the calculation unit 133 may calculate the difference in the number of pixels by comparing an image of a portion indicated by the position information rPT of the detection result with an image of a portion indicated by the true position information tPT.

[0044] As explained in Figure 1, the first object detection model M1 is a trained AI model trained using a training image LIM1 based on a composite image in which a detection object image including a specified object PO as a detection target is pasted onto a background image as a correct label.

[0045] <Extraction part 134> The extraction unit 134 executes extraction processing according to the embodiment. The extraction unit 134 extracts a partial image pIM from the teacher image TD, which is a rectangular area AR indicated by the error position information mPT and which surrounds, as an error object, another object AO that the first object detection model M1 erroneously inferred to be in the same class as the predetermined object PO.

[0046] <Generation section 135> The generation unit 135 generates a learning image LIM2 by pasting the partial image pIM as an incorrect image to a predetermined image different from the teacher image TD. For example, the partial image pIM may be pasted to a synthetic image used to generate the first object detection model M1 as a predetermined image different from the teacher image TD. The partial image pIM as an incorrect image can be interpreted as a negative sample. Note that the process of pasting the partial image pIM as an incorrect image specifically means the process of pasting the partial image pIM without adding annotation information AN.

[0047] More specifically, the generation unit 135 may attach the partial image pIM to the annotated training image LIM1 used to generate the first object detection model M1. That is, the generation unit 135 may attach the partial image pIM to the training image LIM1 included in the set G1, which is the data set used to generate the first object detection model M1. The training image LIM1, the training image LIM2, and the background image that is the basis for the training image LIM1 may be stored in the training image data storage unit 122.

[0048] The generation unit 135 may paste the partial image pIM in a state where the partial image pIM has been processed in a different manner from when it was extracted. For example, the generation unit 135 may paste the partial image pIM in a state where the size, color, orientation, etc. of the partial image pIM have been changed in a different manner from when it was extracted by the extraction unit 134. To enable such processing, a size range in which the size of the partial image pIM can be changed, a chromaticity range in which the color of the partial image pIM can be changed, and an angle range in which the orientation of the partial image pIM can be changed may be set in advance for the learning device 100.

[0049] <Study Section 136> The learning unit 136 uses the learning images LIM2 to learn an object detection model that detects a predetermined object PO from an input image. Specifically, the learning unit 136 generates a trained object detection model as an AI model trained based on a set G2 (dataset) of the learning images LIM2.

[0050] For example, the learning unit 136 may perform re-learning on the first object detection model M1 based on the training image LIM2, thereby learning to prevent other objects AO from being mistakenly detected as objects of the same class as the specified object PO, thereby generating a second object detection model M2 with improved detection accuracy.

[0051] As another example, the learning unit 136 may perform learning based on the learning image LIM2 on the pre-trained model used to train the first object detection model M1, thereby learning to prevent other objects AO from being mistakenly detected as objects of the same class as the specified object PO, thereby generating a second object detection model M2 with improved detection accuracy.

[0052] As described above, the learning device 100 may repeat learning to generate an Xth object detection model Mx with improved detection accuracy. According to this example, the object detection model may be retrained using the training images LIM2, and a final Xth object detection model Mx to be deployed may be generated through a retraining loop.

[0053] By repeating this learning process, the acquisition unit 131 acquires error position information mPT each time learning is performed, the extraction unit 134 extracts a partial image pIM based on the re-acquired error position information mPT, and the generation unit 135 generates a training image LIM2 based on the re-extracted partial image pIM. The learning unit 136 then re-trains the object detection model using the re-generated training image LIM2, thereby generating a final Xth object detection model Mx.

[0054] [4. Example of learning device operation] Fig. 5 is a diagram showing an example of the operation of the learning device 100. Fig. 5 shows the operation procedure of the learning device 100 in information processing according to the embodiment. In Fig. 5 and subsequent figures, the first object detection model M1 will be described as an AI model that sets a "dog" as a detection target. In other words, the first object detection model M1 is an AI model that has been trained to detect a predetermined object PO belonging to a dog class from an input image.

[0055] Furthermore, in the information processing according to the embodiment, as described above, a loop occurs in which learning is repeated using the learning image LIM2, and FIG. 5 shows the first loop, that is, the scene of the first learning.

[0056] First, the acquisition unit 131 acquires a dataset DS composed of teacher images TD (step S11). Fig. 5(a) shows an example of the teacher images DT used for input. According to the example of Fig. 5(a), annotation information AN, which is a correct label that associates true position information tPT of a predetermined object PO with class information of the predetermined object PO, has already been assigned to the teacher image TD.

[0057] For example, each of the annotation information AN1, AN2, and AN3 includes true position information PT of the predetermined object PO on the teacher image TD and class information that classifies the predetermined object PO as a "dog." Note that the teacher image TD also includes another object AO (a dog-shaped robot) that resembles the predetermined object PO, but the other object is not a "dog," so no annotation information AN is assigned to it.

[0058] The detection unit 132 inputs the dataset DS into the first object detection model M1, causing it to perform inference to detect the predetermined object PO (step S12). In this way, the detection unit 132 can improve the efficiency of learning by using the first object detection model M1, which is a trained AI model, for the information processing according to the embodiment. On the other hand, in a situation where efficiency is not a consideration, a pre-trained model may be used instead of the first object detection model M1.

[0059] Returning to the explanation, the first object detection model M1 executes inference under the control of the detection unit 132 (step S13) and outputs a detection result (step S14). The detection result may include position information rPT, which is position information PT of the predetermined object PO on the teacher image TD, and a reliability score SC. The reliability score SC here is an index value indicating the likelihood that the predetermined object PO is enclosed by the rectangular area AR indicated by the position information rPT of the detection result, and the higher the numerical value, the higher the probability that the predetermined object PO exists within the rectangular area AR.

[0060] As a result, the detection unit 132 can obtain a teacher image TD' in which the detection result is applied to the teacher image TD. Fig. 5(b) shows an example of the teacher image TD' to which the detection result is applied. According to the example of Fig. 5(b), rectangular areas AR1, AR2, AR3, and AR4 are associated with the teacher image TD' as position information rPT.

[0061] However, at this point, it is unclear whether the first object detection model M1 has erroneously detected another object AO (i.e., an object not belonging to the dog class) that belongs to a class different from the predetermined object PO as an object belonging to the dog class. Specifically, it is unclear which of the rectangular areas AR (rectangular areas AR1, AR2, AR3, and AR4) indicated by the position information rPT of the detection result is the erroneous position information mPT that has been erroneously detected.

[0062] Therefore, the calculation unit 133 executes a difference calculation process to compare the teacher image TD with the teacher image TD' and calculate the difference (step S15). Specifically, the calculation unit 133 calculates the difference between the true position information tPT included in the teacher image TD and the position information rPT of the detection result.

[0063] When a difference is calculated between the true position information tPT and the position information rPT of the detection result, the acquisition unit 131 acquires, based on the difference, erroneous position information mPT from the position information rPT of the detection result that was incorrectly detected by the first object detection model M1 (step S16).

[0064] Here, a method for acquiring the error position information mPT by the difference calculation process will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the difference calculation process according to the embodiment.

[0065] 6 shows an example in which the difference between the position information rPT of the detection result and the true position information tPT is calculated by subtracting the teacher image TD from the teacher image TD'. According to this example, the calculation unit 133 calculates the difference by subtracting the information of the true position information tPT from the information of the position information rPT.

[0066] As an example, the calculation unit 133 may assign a predetermined value (e.g., "10") to an image portion within each rectangular area AR (rectangular areas AR1 to AR4) indicated by the position information rPT in the teacher image TD'. The calculation unit 133 may also assign a similar predetermined value (e.g., "10") to an image portion within each rectangular area AR indicated by the true position information tPT included in the annotation information AN (annotation information AN1 to AN3) in the teacher image TD.

[0067] In this state, the calculation unit 133 subtracts the teacher image TD from the teacher image TD'. In this case, in the example of FIG. 6, only the "10" assigned to the image portion within the rectangular area AR4 remains as a positive value. Therefore, in the example of FIG. 6, the calculation unit 133 can calculate the position information rPT corresponding to the rectangular area AR4 from the position information rPT of the detection result as the difference. Furthermore, as a result, the acquisition unit 131 acquires the position information rPT corresponding to the rectangular area AR4 as the error position information mPT.

[0068] In this way, learning device 100 can dynamically determine false detection. In the example of Fig. 6, learning device 100 can dynamically determine that first object detection model M1 may erroneously infer that another object AO that resembles predetermined object PO is an object that belongs to the same dog class as predetermined object PO.

[0069] The calculation unit 133 may assign predetermined values ​​as described above, and then perform the reverse calculation, i.e., subtract the teacher image TD' from the teacher image TD. In this case, only the value "10" assigned to the image portion within the rectangular area AR4 remains as a negative value.

[0070] 5, the extraction unit 134 extracts a partial image pIM from the teacher image TD' based on a rectangular area AR indicated by the error position information mPT that surrounds another object AO that the first object detection model M1 erroneously inferred to be in the same dog class as the predetermined object PO as an erroneous object (step S16). As shown in FIG. 5, the extraction unit 134 may extract a partial image pIM from the teacher image TD' within the rectangular area AR indicated by the error position information mPT.

[0071] Next, the generation unit 135 performs superimposition synthesis, using the partial image pIM as an incorrect image to generate a training image LIM2 (step S17). For example, the generation unit 135 may paste the partial image pIM as an incorrect image onto the annotated training image LIM1 used to generate the first object detection model M1. For example, the generation unit 135 may paste the partial image pIM onto the training image LIM1 included in the set G1, which is the data set used to generate the first object detection model M1. As a result, FIG. 5 shows an example in which the generation unit 135 obtains a set G2 of training images LIM2.

[0072] In this way, the generation unit 135 can improve the efficiency of learning by reusing the learning image LIM1 that has already been used for learning. On the other hand, if efficiency is not a consideration, for example, the background image used in the Cut & Paste method of FIG. 1 may be used.

[0073] The generating unit 135 may paste the partial image pIM in a state where the partial image pIM has been processed in a state different from that at the time of extraction. For example, the generating unit 135 may paste the partial image pIM in a state where the size, color, orientation, etc. of the partial image pIM has been changed in a state different from that at the time of extraction by the extracting unit 134.

[0074] 5 shows an example in which one partial image pIM is extracted, but multiple partial images pIM may be extracted. For example, if multiple teacher images TD are input in one inference, or if multiple other objects AO are erroneously inferred to be objects belonging to the dog class, multiple partial images pIM may be extracted. In this way, if multiple partial images pIM are extracted, the generation unit 135 may paste each partial image pIM onto a separate training image LIM1, or may paste each partial image pIM so that they coexist in a single training image LIM1.

[0075] For example, if a difference occurs in the detection accuracy of the generated object detection model between pasting each partial image pIM to a separate learning image LIM1 during a learning loop and pasting each partial image pIM so that they coexist in a single learning image LIM1, the learning device 100 may adopt the pasting method that results in greater accuracy in subsequent learning.

[0076] Also, pasting the partial image pIM as an incorrect image means pasting it as is without adding annotation information AN. Specifically, the generation unit 135 may paste the partial image pIM as is without providing a rectangular area AR based on the true position information tPT in the partial image pIM. This allows the AI ​​model to learn that the object indicated by the partial image pIM does not belong to the dog class.

[0077] Returning to the explanation, finally, the learning unit 136 uses the training images LIM2 to train an object detection model that detects the predetermined object PO from the input image (step S18). For example, the learning unit 136 may cause the first object detection model M1 to re-train based on the training images LIM2, thereby generating a second object detection model M2 that has been trained to not erroneously detect another object AO as an object of the same class as the predetermined object PO and has improved detection accuracy. This allows the learning unit 136 to efficiently improve detection accuracy.

[0078] On the other hand, in a situation where efficiency is not a consideration, the learning unit 136 may generate the second object detection model M2 by performing learning based on the training image LIM2 on the pre-trained model used to train the first object detection model M1.

[0079] Up to this point, an example has been described in which the second object detection model M2 is generated by learning the first object detection model M1 as the first learning loop, using Fig. 5. However, such a series of learning steps S11 to S18 may be looped a predetermined number of times.

[0080] For example, in the second loop, steps S11 to S18 may be performed using the second object detection model M2 instead of the first object detection model M1, thereby generating a third object detection model M3 with improved accuracy. Also, in the third loop, steps S11 to S18 may be performed using the third object detection model M3, rather than the second object detection model M2, thereby generating a fourth object detection model M4 with improved accuracy.

[0081] In this way, the learning device 100 may generate an Xth object detection model Mx with improved accuracy by looping a series of learning steps S11 to S18. Note that the number of loops is not limited and may be specified by, for example, a user who operates the object detection model.

[0082] Furthermore, it is not necessary to use the same dataset in each loop. For example, the training images TD included in the dataset DS may be divided into multiple sets, and the detection unit 132 may input different groups of datasets DS to the object detection model in step S11 for each loop. Furthermore, the training images LIM1 included in the set G1 may also be divided into multiple sets, and the training unit 136 may train the object detection model using different sets of training images LIM1 in step S18 for each loop.

[0083] [5. Priority use based on reliability score] As described in FIG. 5, in each loop, the detection result output by the object detection model (the first object detection model M1 in the first loop) includes a confidence score SC in addition to the position information rPT. Therefore, the acquisition unit 131 may further acquire a confidence score SC. Furthermore, for example, when a plurality of partial images pIM are extracted, the generation unit 135 may paste the partial image pIM whose confidence score SC satisfies a predetermined condition from among the plurality of partial images pIM. For example, among the partial images pIM, the partial image pIM with a higher confidence score SC may be pasted preferentially. This point will be described with reference to FIG. 7.

[0084] Fig. 7 is a diagram showing an example of a technique for pasting partial images pIM. Fig. 7 shows an example in which partial images pIM1, pIM2, and pIM3 are extracted by the extraction unit 134. The partial images pIM1, pIM2, and pIM3 may be extracted from one teacher image TD at any number of times in the loop, or may be extracted among multiple teacher images TD at any number of times in the loop.

[0085] 7 shows an example in which the reliability score SC of the position information rPT that led to the extraction of partial image pIM1 was "0.8," the reliability score SC of the position information rPT that led to the extraction of partial image pIM2 was "0.6," and the reliability score SC of the position information rPT that led to the extraction of partial image pIM3 was "0.3." In this example, the reliability score SC corresponding to partial image pIM1 is the highest. This suggests that the object detection model used in this detection (the first object detection model M1 in the first loop) is most likely to erroneously infer that the object represented by partial image pIM1 is of the same class as the predetermined object PO, among the objects represented by partial images pIM1, pIM2, and pIM3. Therefore, it is believed that accuracy can be improved by having the object detection model focus on learning that objects that are more likely to be erroneously detected are not of the same class as the predetermined object PO.

[0086] 7, the generation unit 135 may paste the partial image pIM1 preferentially over the partial images pIM2 and pIM3. The term "pasting preferentially" as used herein includes patterns such as pasting a larger number of partial images pIM1 compared to the partial images pIM2 and pIM3, pasting partial images pIM2 and pIM3 but pasting a number of partial images pIM1 equal to or greater than a reference value, or pasting only partial images pIM1 while excluding partial images pIM2 and pIM3. Such pasting methods may be executed in each loop.

[0087] [6. Hardware Configuration] The learning device 100 according to the embodiment may be realized, for example, by a computer 1000 configured as shown in Fig. 8. Fig. 8 is a hardware configuration diagram showing an example of a computer that realizes the functions of the learning device 100 according to the embodiment. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0088] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0089] The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0090] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. The CPU 1100 also outputs generated data to the output device via the input / output interface 1600.

[0091] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0092] For example, when computer 1000 functions as learning device 100 according to the embodiment, CPU 1100 of computer 1000 executes programs loaded onto RAM 1200 to realize the functions of control unit 130. CPU 1100 of computer 1000 reads and executes these programs from recording medium 1800, but as another example, these programs may be acquired from another device via a predetermined communication network.

[0093] [7. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0094] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0095] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0096] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0097] 1. Information Processing Systems 2. Imaging system 10. Imaging device 11 Display control device 12 Display device 60 Detection Device 100 Learning Device 130 Control Unit 131 Acquisition Department 132 Detector 133 Calculation Unit 134 Extraction part 135 Generation part 136 Learning Department

Claims

1. an acquisition unit that acquires, based on a comparison between position information of a detection result detected from a teacher image by an object detection model for detecting a predetermined object and true position information of the predetermined object included as a correct label in the teacher image, erroneous position information of the detection result that is erroneously detected by the object detection model; an extraction unit that extracts a partial image of a region of the teacher image indicated by the error position information, the partial image being a region of an error object that the object detection model has erroneously inferred to be in the same class as the predetermined object; a generation unit that generates a learning image by pasting the partial image as an incorrect image onto a predetermined image different from the teacher image; a learning unit that uses the learning images to learn an object detection model that detects the predetermined object from an input image; A learning device comprising:

2. the object detection model is a trained object detection model trained based on a synthetic image to which an image of the predetermined object is attached as a correct answer label, as the predetermined image; the generation unit further pastes the partial image onto the composite image. The learning device according to claim 1 .

3. the generation unit pastes the partial image in a state where the partial image has been processed in a different manner from when it was extracted. The learning device according to claim 1 .

4. When a plurality of partial images are extracted, the generation unit pastes the plurality of different partial images onto one of the predetermined images, or pastes the plurality of different partial images onto different predetermined images individually. The learning device according to claim 1 .

5. the object detection model is retrained using the training images; the acquiring unit acquires the error position information each time learning is performed, the extraction unit extracts the partial image based on the error position information acquired again; the generation unit generates the learning image based on the re-extracted partial image; The learning unit re-learns the object detection model using the re-generated learning images. The learning device according to claim 1 .

6. the acquisition unit further acquires a reliability score, which is an index value of likelihood that the predetermined object is present within an area indicated by the position information of the detection result, each time learning is performed; the generation unit generates the learning image by pasting, onto the predetermined image, a partial image whose reliability score satisfies a predetermined condition among the partial images. The learning device according to claim 5 .

7. the generation unit preferentially pastes, among the partial images, a partial image having a higher reliability score onto the predetermined image. The learning device according to claim 6.

8. A learning method executed by a learning device, comprising: an acquisition step of acquiring erroneous position information of the detection result, which is erroneously detected by the object detection model, based on a difference between position information of the detection result detected from a teacher image by an object detection model for detecting a predetermined object and true position information of the predetermined object included as a correct answer label in the teacher image; an extraction step of extracting a partial image of the teacher image, which is a region indicated by the error position information and is a region of an error object that the object detection model has erroneously inferred to be in the same class as the predetermined object; a generating step of generating a learning image by pasting the partial image as an incorrect image onto a predetermined image different from the teacher image; a learning step of learning an object detection model for detecting the predetermined object from an input image using the learning image; Learning methods including.

9. an acquisition step of acquiring erroneous position information of the detection result, which is erroneously detected by the object detection model, based on a difference between position information of the detection result detected from a teacher image by an object detection model for detecting a predetermined object and true position information of the predetermined object included as a correct answer label in the teacher image; an extraction step of extracting a partial image of the teacher image, which is a region indicated by the error position information and is a region of an error object that the object detection model has erroneously inferred to be in the same class as the predetermined object; a generation step of generating a learning image by pasting the partial image as an incorrect image onto a predetermined image different from the teacher image; a learning procedure for learning an object detection model that detects the predetermined object from an input image using the learning images; A learning program that allows a computer to execute the above.

Images