Learning data generation device, learning data generation method, and learning data generation program

Abstract

This enables the generation of effective image data to improve detection accuracy without human intervention. [Solution] The image conversion unit 21 receives image data, which is training data 31, as input to the image conversion model 41 that converts image data, and uses this training data to train an object detection model that detects objects with the attributes of the target object from the image data. The image conversion model 41 then converts the image data to obtain a converted image. The training data addition unit 22 adds the converted image obtained by the image conversion unit 21 to the training data 31.

Description

Technical Field

[0001] The present disclosure relates to a technique for generating learning data for an object detection model.

Background Art

[0002] There is an object detection model that detects an object of an attribute to be detected from image data. In order to improve the detection accuracy of the object detection model, it is necessary to make the object detection model learn various image data as learning data. The creation of learning data is often done manually, resulting in a large workload. Also, when learning dangerous situations etc., image data of dangerous situations is required, but it is difficult for a person to actually reproduce dangerous situations, and learning data cannot be easily created.

[0003] Patent Document 1 describes that when there is a shortage of teacher images, new teacher images are generated by performing processes such as enlargement, reduction, movement, and synthesis on the teacher images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] With the method described in Patent Document 1, basically only image data similar to the original teacher image can be generated as new teacher images. Therefore, it has been difficult to generate effective image data for improving detection accuracy. An object of the present disclosure is to enable the generation of effective image data for improving detection accuracy without relying on manual labor.

Means for Solving the Problems

[0006] The learning data generation device according to the present disclosure is An image conversion unit receives image data, which is training data for an object detection model, as input to an image conversion model that converts image data, and obtains a converted image obtained by the image conversion model. A training data addition unit adds the converted image acquired by the image conversion unit, in which the confidence level of object detection by the object detection model is less than the evaluation threshold, to the training data. Equipped with, The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The image conversion unit inputs the image data and attribute information indicating a specified attribute to the text generation model, instructs the model to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, obtains the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputs the obtained explanatory text to the document conversion model, instructs the model to change other parts without changing the specified attribute and the field of view, obtains the converted text converted by the document conversion model, inputs the converted text to the image generation model, and obtains the image data generated by the image generation model as the converted image. If at least some of the converted images are not added to the training data, the training data addition unit causes the image conversion unit to generate the missing number of converted images from the specified number. [Effects of the Invention]

[0007] In this disclosure, image data, which serves as training data, is input to an image transformation model, and the transformed image data, which has been transformed by the image transformation model, is added to the training data. This makes it possible to generate image data that has undergone various modifications, not only simple processing such as enlargement, reduction, movement, and compositing, but also modifications such as changing the background, changing surrounding objects, and changing the way light hits the image. Therefore, it is possible to generate image data that is effective in improving detection accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] Configuration diagram of the learning data generation device 10 according to Embodiment 1. [Figure 2] A flowchart of the processing of the learning data generation device 10 according to Embodiment 1. [Figure 3] An explanatory diagram of the input to the image conversion model 41 according to Embodiment 1. [Figure 4] Configuration diagram of the learning data generation device 10 according to Embodiment 2. [Figure 5] A flowchart of the processing of the learning data generation device 10 according to Embodiment 2. [Figure 6] An explanatory diagram of the input to the text generation model 42 according to Embodiment 2. [Figure 7] An explanatory diagram of the output from the text generation model 42 according to Embodiment 2. [Figure 8] An explanatory diagram of the input to the document conversion model 43 according to Embodiment 2. [Figure 9] Configuration diagram of the learning data generation device 10 according to Embodiment 3. [Figure 10] Flowchart of the learning data addition process according to Embodiment 3. [Figure 11] An explanatory diagram of the input to the attribute determination model 46 according to Embodiment 3. [Figure 12] Configuration diagram of the learning data generation device 10 according to Embodiment 4. [Figure 13] A flowchart of the processing of the learning data generation device 10 according to Embodiment 4.

Mode for Carrying Out the Invention

[0009] Embodiment 1. ***Description of Configuration*** Referring to FIG. 1, the configuration of the learning data generation device 10 according to Embodiment 1 will be described. The learning data generation device 10 is a computer. The learning data generation device 10 includes hardware of a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via signal lines and controls these other hardware.

[0010] The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit. Specific examples of the processor 11 are a CPU, a DSP, and a GPU. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0011] The memory 12 is a storage device that temporarily stores data. Specific examples of the memory 12 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory.

[0012] The storage 13 is a storage device that stores data. Specific examples of the storage 13 are SSD. SSD is an abbreviation for Solid State Drive. Also, the storage 13 may be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. SD is an abbreviation for Secure Digital. DVD is an abbreviation for Digital Versatile Disk.

[0013] The communication interface 14 is an interface for communicating with an external device. As a specific example, the communication interface 14 is a port for Ethernet (registered trademark), USB, or HDMI (registered trademark). USB is the abbreviation for Universal Serial Bus. HDMI is the abbreviation for High-Definition Multimedia Interface.

[0014] The learning data generation device 10 includes, as functional components, an image conversion unit 21 and a learning data addition unit 22. The functions of each functional component of the learning data generation device 10 are realized by software. The storage 13 stores a program for realizing the functions of each functional component of the learning data generation device 10. This program is read into the memory 12 by the processor 11 and executed by the processor 11. Thereby, the functions of each functional component of the learning data generation device 10 are realized. The storage 13 stores learning data 31. The learning data 31 is data for training an object detection model. The learning data 31 includes a plurality of image data. The object detection model is a learning model for detecting an object of an attribute to be detected from the image data. The object detection model is, for example, a model configured using deep learning.

[0015] The learning data generation device 10 is connected to an image conversion model 41 via the communication interface 14. Image transformation model 41 is a so-called generative AI that transforms images. AI stands for Artificial Intelligence. Image transformation model 41 may be constructed using algorithms such as BERT and GPT. BERT stands for Bidirectional Encoder Representations from Transformers. GPT stands for Generative Pretrained Transformer. The learning model 112 may be constructed by combining multiple algorithms, including these algorithms.

[0016] Here, it is assumed that the image conversion model 41 is connected via the communication interface 14. In other words, the image conversion model 41 is assumed to be outside the training data generation device 10. However, the image conversion model 41 may also be inside the training data generation device 10.

[0017] In Figure 1, only one processor 11 was shown. However, there may be multiple processors 11, and multiple processors 11 may work together to execute programs that implement each function.

[0018] ***Explanation of operation*** The operation of the learning data generation device 10 according to Embodiment 1 will be explained with reference to Figures 2 and 3. The operation procedure of the learning data generation device 10 according to Embodiment 1 corresponds to the learning data generation method according to Embodiment 1. Furthermore, the program that realizes the operation of the learning data generation device 10 according to Embodiment 1 corresponds to the learning data generation program according to Embodiment 1.

[0019] Referring to Figure 2, the processing of the learning data generation device 10 according to Embodiment 1 will be explained. (Step S11: Input Processing) The image conversion unit 21 extracts one image data from the training data 31. The image conversion unit 21 inputs the extracted image data and attribute information indicating the specified attribute, which is the attribute to be detected by the object detection model, to the image conversion model 41. If there are multiple attributes to be detected by the object detection model, the image conversion unit 21 inputs the attribute information specified for the extracted image data from among the multiple attributes. At this time, the image conversion unit 21 inputs a prompt to the image conversion model 41 instructing the image data conversion rules. The image data extracted in step S11 should preferably be an image in which a specific attribute could not be detected or an image in which a different attribute was detected during the pre-training model evaluation phase of the object detection model.

[0020] For example, as shown in Figure 3, the image conversion unit 21 inputs a prompt to the image conversion model 41 specifying the image data conversion rules, along with the image data and attribute information. The conversion rules specify which elements will be changed randomly and which will never be changed. The elements that can be randomly changed include clothing type, gender, and background. Other elements such as lighting and the number of people may also be specified depending on the situation. The elements that are absolutely unchangeable are attribute information and field of view information. Information related to attribute information includes not only the attribute information itself, but also anything associated with it. For example, if the attribute information is a white cane, then anything associated with the attribute information would be that the white cane is being held in the right hand. In the prompt in Figure 3, it is simply specified as "information related to ${attribute information}", but it can also be specified more specifically as "${attribute information} and the hand holding ${attribute information}". Here, ${attribute information} represents the attribute information specified in the input information (in the above example, "white cane"). In other words, "${attribute information} and the hand holding ${attribute information}" can be read as "white cane and the hand holding the white cane". Other ${***} are similarly replaced with the information specified as *** in the input information.

[0021] If the image data to be created is already determined, the image conversion unit 21 may prompt for detailed instructions on the image data to be created.

[0022] (Step S12: Image conversion process) The image conversion model 41 generates a specified number of converted images by converting the input image data in step S21. The image conversion model 41 then outputs the specified number of converted images. The specified number is, for example, about 10% of the number of image data in the training data 31. In this process, the image conversion model 41, following the prompt instructions, randomly modifies the attributes specified to be changed for each converted image. The image conversion model 41 also leaves the attributes specified as never being changed unchanged. In other words, the generated converted image retains the specified attributes without modification. Attributes not specified as either randomly modifiable or never being changed can be arbitrarily modified. The image conversion unit 21 acquires a specified number of converted images output by the image conversion model 41.

[0023] (Step S13: Adding training data) The training data addition unit 22 adds a specified number of converted images acquired by the image conversion unit 21 in step S12 to the training data 31. The training data addition process shown in Figure 2 is executed for each attribute that needs to be added to the training data.

[0024] ***Effects of Embodiment 1*** As described above, the learning data generation device 10 according to Embodiment 1 inputs image data, which is learning data, to the image conversion model 41 and adds the converted image obtained by the image conversion model 41 to the learning data 31. This makes it possible to generate image data with various modifications, such as changes to the background, changes to surrounding objects, and changes to how light hits the image. Therefore, it is possible to generate image data that is effective in improving detection accuracy.

[0025] The learning data generation device 10 according to Embodiment 1 is capable of creating learning data that is difficult to collect, such as in dangerous situations. Furthermore, the learning data generation device 10 according to Embodiment 1 is capable of increasing the variety of learning data that is difficult to collect.

[0026] Embodiment 2. Embodiment 2 differs from Embodiment 1 in that it generates a descriptive text for the image data, converts the descriptive text, and generates the image data from the converted descriptive text to produce the converted image. Embodiment 2 will explain this difference, and will omit explanations of the points that are the same.

[0027] ***Explanation of the structure*** Referring to Figure 4, the configuration of the learning data generation device 10 according to Embodiment 2 will be described. The image conversion model 41 includes a text generation model 42, a document conversion model 43, and an image generation model 44. The text generation model 42 is a model that generates descriptive text for image data. The document conversion model 43 is a model that converts descriptive text. The image generation model 44 is a model that generates image data from descriptive text.

[0028] ***Explanation of operation*** Referring to Figure 5, the processing of the learning data generation device 10 according to Embodiment 2 will be explained. (Step S21: Image input processing) The image conversion unit 21 extracts one image data from the training data 31. The image conversion unit 21 inputs the extracted image data and attribute information indicating the specified attribute, which is the attribute to be detected by the object detection model, to the text generation model 42. If there are multiple attributes to be detected by the object detection model, the image conversion unit 21 inputs attribute information indicating one or more of the multiple attributes. At this time, the image conversion unit 21 inputs a prompt to the text generation model 42 that instructs the rules for generating the explanatory text.

[0029] For example, as shown in Figure 6, the image conversion unit 21 inputs a prompt to the text generation model 42 that specifies the rules for generating the descriptive text, along with the image data and attribute information. The rules for generating the descriptive text specify that the field of view information and attribute information must always be described.

[0030] (Step S22: Description generation process) The text generation model 42 generates and outputs a descriptive text describing the image data input in step S21. At this time, as shown in Figure 7, the text generation model 42 generates a descriptive text for the entire image data, including explanations of the field of view information and attribute information, in accordance with the instructions given by the prompt. The image conversion unit 21 then acquires the descriptive text output by the text generation model 42.

[0031] (Step S23: Inputting the description) The image conversion unit 21 inputs the descriptive text acquired in step S22 and attribute information indicating the specified attribute, which is the attribute of the object detection model's detection target, to the document conversion model 43. At this time, the image conversion unit 21 inputs a prompt to the image conversion model 41 instructing the conversion rules for the descriptive text. For example, as shown in Figure 8, the image conversion unit 21 inputs a prompt to the document conversion model 43 specifying the conversion rules for the description, along with the description and attribute information. In the conversion rules, as in Figure 3, some items are specified to be changed randomly, while others are specified to never be changed. Also, in Figure 8, the description is converted to English. This is because English descriptions tend to result in higher accuracy in generating images from the description.

[0032] (Step S24: Description text conversion process) The document conversion model 43 generates and outputs a converted text by converting the explanatory text input in step S23. The image conversion unit 21 then acquires the converted text output by the document conversion model 43. In this process, the document conversion model 43 follows the prompt instructions and randomly modifies elements designated as to be changed for each converted sentence. It also leaves elements designated as never to be changed unchanged. Elements not designated as either randomly changed or never changed can be arbitrarily modified. Furthermore, the document conversion model 43 converts the explanatory text into English. Note that the converted sentence is generated as a prompt for image generation.

[0033] (Step S25: Inputting the converted text) The image conversion unit 21 inputs the converted text (image generation prompt) obtained in step S24 to the image generation model 44. At this time, the image conversion unit 21 inputs a prompt to the image generation model 44 instructing it to generate a specified number of image data representing the converted text.

[0034] (Step S26: Image generation process) The image generation model 44 generates a specified number of image data representing the converted sentence input in step S25. The image generation model 44 generates image data that represents the converted sentence while randomly modifying each image data. Then, the image generation model 44 outputs the specified number of converted images. The image conversion unit 21 acquires a specified number of converted images output by the image generation model 44.

[0035] (Step S27: Adding training data) The training data addition unit 22 adds a specified number of converted images acquired by the image conversion unit 21 in step S26 to the training data 31.

[0036] ***Effects of Embodiment 2*** As described above, the learning data generation device 10 according to Embodiment 2 generates a descriptive text for image data, converts the descriptive text, and generates image data from the converted descriptive text, thereby generating a converted image. By generating a converted text by converting the descriptive text, the converted content can be clarified.

[0037] Furthermore, the learning data generation device 10 according to Embodiment 1 can utilize separate generating AIs for the text generation model 42, the document conversion model 43, and the image generation model 44. This makes it possible to specialize the document conversion model 43 in language processing, and the text generation model 42 and the image generation model 44 in image processing. As a result, it is possible to increase processing speed and accuracy.

[0038] ***Other configurations*** <Example 1> In Embodiment 2, one converted sentence is generated, and a specified number of converted images are generated from this one converted sentence. Alternatively, a specified number of converted sentences may be generated, and one converted image may be generated from each converted sentence. Furthermore, M converted sentences may be generated, and N converted images may be generated from each converted sentence. In this case, M and N are integers of 1 or greater, and M × N = the specified number.

[0039] Embodiment 3. Embodiment 3 differs from Embodiments 1 and 2 in that it determines whether or not to add the converted image to the training data 31. Embodiment 3 explains this difference, while omitting explanations of the same points.

[0040] ***Explanation of the structure*** Referring to Figure 9, the configuration of the learning data generation device 10 according to Embodiment 3 will be described. The learning data generation device 10 is connected to the object detection model 45 and the attribute determination model 46 via a communication interface 14. The object detection model 45 is a model that detects objects with the attributes to be detected from image data, and it is a model that learns from the training data 31. The attribute determination model 46 is a model that determines whether or not an image data contains an object with a specified attribute. Similar to the image transformation model 41, the attribute determination model 46 is a so-called generative AI. That is the case.

[0041] ***Explanation of operation*** Referring to Figure 10, the learning data addition process according to Embodiment 3 (step S13 in Figure 2, step S27 in Figure 5) will be explained. During the training data addition process, the following operations are performed on each transformed image.

[0042] (Step S31: First input processing) The training data addition unit 22 inputs the transformed image of the target and the attribute information used as input when generating the transformed image to the object detection model 45. The attribute information used as input when generating the transformed image is the attribute information input in step S11 in Figure 2 or step S21 in Figure 5.

[0043] (Step S32: Object detection process) The object detection model 45 detects objects with the attributes indicated by the attribute information input in step S31 from the transformed image of the target input in step S31. The object detection model 45 outputs the detection result and the confidence level of the detection. The training data addition unit 22 acquires the detection results and confidence levels output by the object detection model 45.

[0044] (Step S33: Confidence level determination process) The training data addition unit 22 determines whether the confidence level obtained in step S32 is above a threshold. If the confidence level is above the threshold, the training data addition unit 22 does not add the target converted image to the training data 31 and terminates processing for the target converted image. On the other hand, if the confidence level is below the threshold, the training data addition unit 22 proceeds to step S34.

[0045] (Step S34: Second input processing) The training data addition unit 22 inputs the target converted image and the attribute information used as input when generating the converted image to the attribute determination model 46. At this time, as shown in Figure 11, the training data addition unit 22 inputs a prompt to the attribute determination model 46 instructing it to determine whether or not an object with the attributes indicated by the input attribute information exists in the input converted image.

[0046] (Step S35: Attribute determination process) The attribute determination model 46 determines whether or not an object with the attribute indicated by the attribute information input in step S31 exists in the converted image of the target input in step S34. The attribute determination model 46 outputs the determination result. The training data addition unit 22 acquires the judgment results output by the attribute judgment model 46.

[0047] (Step S36: Result determination process) If the determination result obtained in step S35 indicates that an object exists in the target converted image, the training data addition unit 22 proceeds to step S37. On the other hand, if the determination result obtained in step S35 indicates that an object does not exist in the target converted image, the training data addition unit 22 does not add the target converted image to the training data 31 and terminates processing of the target converted image.

[0048] (Step S37: Additional processing) The training data addition unit 22 adds the target converted image to the training data 31.

[0049] ***Effects of Embodiment 3*** The training data generation device 10 according to Embodiment 3 uses an object detection model 45 and an attribute determination model 46 to determine whether or not to add the converted image to the training data 31. This prevents the addition of converted images that would degrade detection accuracy or converted images that do not require training to the training data 31. Specifically, in step S33, the decision prevents overfitting of the object detection model by not adding objects that can be detected by the current object detection model to the training data 31. Furthermore, in step S36, the decision prevents irrelevant images from being included in the training data 31 by not adding images that do not possess the attribute information desired for training in the converted image.

[0050] ***Other configurations*** <Modification 2> In Embodiment 3, only the converted images that could be registered using the object detection model 45 and the attribute determination model 46 were added to the training data 31. Therefore, even if a specified number of converted images were generated, it was not guaranteed that the specified number of converted images would be added to the training data 31. Therefore, if at least some of the converted images are not added to the training data 31, the training data addition unit 22 may return the processing to the image conversion unit 21 and have it regenerate the missing number of converted images. The missing number is the value obtained by subtracting the number of converted images added to the training data 31 from the specified number of images. In other words, the training data addition unit 22 may set the missing number as the new specified number of images and have the processing restart from step S11 in Figure 2 or step S21 in Figure 5.

[0051] Embodiment 4. Embodiment 4 differs from Embodiment 3 in that it includes training of the object detection model 45. Embodiment 4 explains this difference, while omitting explanations of the same points.

[0052] ***Explanation of the structure*** Referring to Figure 12, the configuration of the learning data generation device 10 according to Embodiment 4 will be described. The learning data generation device 10 differs from the learning data generation device 10 shown in Figure 9 in that it includes a learning unit 23 as a functional component. The learning unit 23, like the image conversion unit 21 and the learning data addition unit 22, is implemented by software.

[0053] ***Explanation of operation*** Referring to Figure 13, the processing of the learning data generation device 10 according to Embodiment 4 will be explained. (Step S41: Generation and addition process) The image conversion unit 21 and the training data addition unit 22 add the converted image to the training data 31 by the process described in Embodiment 3.

[0054] (Step S42: Learning process) The learning unit 23 uses the training data 31 to which the converted images were added in step S41 to train the object detection model 45.

[0055] (Step S43: Model evaluation process) The learning unit 23 evaluates the object detection model 45 that was trained in step S42. The evaluation of the object detection model 45 can be performed using existing technologies. If the evaluation obtained in step S43 is lower than the previous evaluation, the learning unit 23 proceeds to step S44. If the evaluation obtained in step S43 is higher than the previous evaluation but does not meet the standard, the learning unit 23 returns to step S41. If this occurs, the process will be terminated.

[0056] (Step S44: Data Deletion Process) The learning unit 23 removes the converted image added in step S41 from the training data 31. The learning unit 23 also returns the object detection model 45 to its state before training in step S42. Then, the learning unit 23 returns to step S41.

[0057] ***Effects of Embodiment 4*** As described above, the learning data generation device 10 according to Embodiment 4 repeatedly adds learning data 31 and trains the object detection model 45. This makes it possible to improve the detection accuracy of the object detection model 45 without manual intervention.

[0058] ***Other configurations*** <Variation 3> In the embodiments described above, each functional component was implemented in software. However, in Modification 3, each functional component may be implemented in hardware. The differences between this Modification 3 and the embodiments described above will be explained.

[0059] When each functional component is implemented in hardware, the learning data generation device 10 includes electronic circuits instead of the processor 11, memory 12, and storage 13. The electronic circuits are dedicated circuits that implement the functions of each functional component, as well as the functions of the memory 12 and storage 13.

[0060] Electronic circuits can include single circuits, complex circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be implemented in a single electronic circuit, or it may be implemented by distributing each functional component across multiple electronic circuits.

[0061] <Modification 4> As a fourth variation, some of the functional components may be implemented in hardware, while others may be implemented in software.

[0062] The processor 11, memory 12, storage 13, and electronic circuitry are collectively referred to as the processing circuit. In other words, the function of each functional component is realized by the processing circuit.

[0063] Furthermore, the term "part" in the above explanation may be replaced with "circuit," "process," "procedure," "processing," or "processing circuit."

[0064] The various aspects of this disclosure are summarized below as an appendix. (Note 1) An image conversion unit receives image data, which is training data for an object detection model, as input to an image conversion model that converts image data, and obtains a converted image obtained by the image conversion model. A learning data addition unit adds the converted image acquired by the image conversion unit to the learning data. A learning data generation device equipped with the following features. (Note 2) The image conversion unit instructs the image conversion model to maintain the state in which the image data includes the specified attribute, and converts the image data. The learning data generation device described in Appendix 1. (Note 3) The image conversion unit instructs the image conversion model to convert the image data by modifying other parts of the image data while leaving the parts relating to the specified attributes unchanged. A learning data generation device as described in Appendix 1 or 2. (Note 4) The image conversion model includes a text generation model that generates a descriptive text for image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The image conversion unit inputs the image data to the text generation model to obtain a description of the image data generated by the text generation model, inputs the acquired description to the document conversion model to obtain a converted text converted by the document conversion model, inputs the converted text converted to the image generation model to obtain the image data generated by the image generation model as the converted image. A learning data generation device as described in Appendix 1 or 2. (Note 5) The learning data generation device described in Appendix 4, wherein the image conversion unit instructs the document conversion model to convert the explanatory text by changing other parts without changing the part relating to the specified attribute in the explanatory text. (Note 6) The aforementioned learning data generation device further, A data determination unit inputs the converted image acquired by the image conversion unit to the object detection model and determines whether the confidence level of object detection by the object detection model is equal to or greater than an evaluation threshold. Equipped with, The training data addition unit adds the converted image to the training data when the data determination unit determines that the confidence level is less than the evaluation threshold. A learning data generation device as described in any one of the items 1 to 5 in the appendix. (Note 7) The data determination unit, when the confidence level is less than the evaluation threshold, inputs the converted image and the specified attribute to an attribute determination model that determines whether or not an object with a specific attribute exists in the image data, and obtains a determination result of whether or not an object with the specified attribute exists in the converted image. The training data addition unit adds the converted image to the training data when the data determination unit obtains a determination result that an object with the specified attribute exists in the converted image. The learning data generation device described in Appendix 6. (Note 8) A computer inputs image data, which is training data to be used to train an object detection model, into an image transformation model that transforms image data, and obtains a transformed image obtained by the image transformation model. A method for generating training data in which a computer adds the converted image to the training data. (Note 9) Image transformation processing involves inputting image data, which is training data for training an object detection model, to an image transformation model that transforms image data, and obtaining a transformed image obtained by the image transformation model. A training data addition process is performed to add the converted image obtained by the image conversion process to the training data. A training data generation program that makes a computer function as a training data generation device.

[0065] The embodiments and variations of this disclosure have been described above. Some of these embodiments and variations may be implemented in combination. Alternatively, some or all of them may be implemented in part. However, this disclosure is not limited to the embodiments and variations described above, and various modifications are possible as needed. [Explanation of symbols]

[0066] 10 Training data generation device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Image conversion unit, 22 Training data addition unit, 23 Training unit, 31 Training data, 41 Image conversion model, 42 Text generation model, 43 Document conversion model, 44 Image generation model, 45 Object detection model, 46 Attribute determination model.

Claims

1. An image conversion unit receives image data, which is training data for an object detection model, as input to an image conversion model that converts image data, and obtains a converted image obtained by the image conversion model. A training data addition unit adds the converted image acquired by the image conversion unit, in which the confidence level of object detection by the object detection model is less than the evaluation threshold, to the training data. Equipped with, The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The image conversion unit inputs the image data and attribute information indicating a specified attribute to the text generation model, instructs the model to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, obtains the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputs the obtained explanatory text to the document conversion model, instructs the model to change other parts without changing the specified attribute and the field of view, obtains the converted text converted by the document conversion model, inputs the converted text to the image generation model, and obtains the image data generated by the image generation model as the converted image. The aforementioned training data addition unit is a training data generation device that, if at least some of the converted images are not added to the training data, causes the image conversion unit to generate the missing number of converted images from the specified number.

2. An image conversion unit receives image data, which is training data for an object detection model, as input to an image conversion model that converts image data, and obtains a converted image obtained by the image conversion model. A training data addition unit adds the converted image acquired by the image conversion unit, in which the confidence level of object detection by the object detection model is less than the evaluation threshold, to the training data. The learning unit learns using the training data to which the converted image has been added. Equipped with, The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The image conversion unit inputs the image data and attribute information indicating a specified attribute to the text generation model, instructs the model to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, obtains the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputs the obtained explanatory text to the document conversion model, instructs the model to change other parts without changing the specified attribute and the field of view, obtains the converted text converted by the document conversion model, inputs the converted text to the image generation model, and obtains the image data generated by the image generation model as the converted image. The learning unit is a learning data generation device that evaluates the object detection model and deletes the added converted image if the evaluation value is less than the previous evaluation value.

3. The learning unit evaluates the object detection model. If the evaluation value is greater than or equal to the previous evaluation value but less than the threshold value, it adds the converted image again and continues learning. If the evaluation value is greater than or equal to the previous evaluation value and greater than or equal to the threshold value, it terminates the learning process. The learning data generation device according to claim 2.

4. The aforementioned learning data generation device further, A data determination unit inputs the converted image acquired by the image conversion unit to the object detection model and determines whether the confidence level of object detection by the object detection model is equal to or greater than an evaluation threshold. Equipped with, The training data addition unit adds the converted image to the training data when the data determination unit determines that the confidence level is less than the evaluation threshold. A learning data generation device according to any one of claims 1 to 3.

5. A computer inputs image data, which is training data to be used to train an object detection model, into an image transformation model that transforms image data, and obtains a transformed image obtained by the image transformation model. The computer adds the transformed image, in which the confidence level of object detection by the object detection model is below the evaluation threshold, to the training data. The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The computer inputs the image data and attribute information indicating a specified attribute to the text generation model, and instructs it to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, thereby obtaining the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputting the obtained explanatory text to the document conversion model, and instructing it to change other parts without changing the specified attribute and the field of view, thereby obtaining the converted text converted by the document conversion model, inputting the converted text to the image generation model, thereby obtaining the image data generated by the image generation model as the converted image. A method for generating training data in which a computer generates the number of converted images that are missing from a specified number of converted images if at least some of the converted images have not been added to the training data.

6. Image transformation processing involves inputting image data, which is training data for training an object detection model, to an image transformation model that transforms image data, and obtaining a transformed image obtained by the image transformation model. A training data addition process is performed to add the converted image obtained by the image conversion process, in which the confidence level of object detection by the object detection model is less than the evaluation threshold, to the training data. The computer is used as a learning data generation device to perform this task. The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The image conversion process inputs the image data and attribute information indicating a specified attribute to the text generation model, instructs it to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, obtains the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputs the obtained explanatory text to the document conversion model, instructs it to change other parts without changing the specified attribute and the field of view, obtains the converted text converted by the document conversion model, inputs the converted text to the image generation model, and obtains the image data generated by the image generation model as the converted image. The aforementioned training data addition process is a training data generation program that, if at least some of the converted images are not added to the training data, generates the missing number of converted images from the specified number using the image conversion process.

7. A computer inputs image data, which is training data to be used to train an object detection model, into an image transformation model that transforms image data, and obtains a transformed image obtained by the image transformation model. The computer adds the transformed image, in which the confidence level of object detection by the object detection model is below the evaluation threshold, to the training data. The computer learns using the training data to which the converted image has been added. The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The computer inputs the image data and attribute information indicating a specified attribute to the text generation model, and instructs it to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, thereby obtaining the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputting the obtained explanatory text to the document conversion model, and instructing it to change other parts without changing the specified attribute and the field of view, thereby obtaining the converted text converted by the document conversion model, inputting the converted text to the image generation model, thereby obtaining the image data generated by the image generation model as the converted image. A method for generating training data in which a computer evaluates the object detection model and deletes the added converted images if the evaluation value is less than the previous evaluation value.

8. Image transformation processing involves inputting image data, which is training data for training an object detection model, to an image transformation model that transforms image data, and obtaining a transformed image obtained by the image transformation model. A training data addition process is performed to add the converted image obtained by the image conversion process, in which the confidence level of object detection by the object detection model is less than the evaluation threshold, to the training data. The learning process involves learning using the training data to which the converted image has been added. The computer is used as a learning data generation device to perform this task. The image conversion model includes a text generation model that generates a descriptive text for the image data, a document conversion model that converts the descriptive text, and an image generation model that generates image data from the descriptive text. The image conversion process inputs the image data and attribute information indicating a specified attribute to the text generation model, instructs it to generate an explanatory text that includes an explanation of the specified attribute indicated by the attribute information contained in the image data and an explanation of the field of view of the image data, obtains the explanatory text of the image data that includes the explanation of the specified attribute and the explanation of the field of view generated by the text generation model, inputs the obtained explanatory text to the document conversion model, instructs it to change other parts without changing the specified attribute and the field of view, obtains the converted text converted by the document conversion model, inputs the converted text to the image generation model, and obtains the image data generated by the image generation model as the converted image. The aforementioned learning process is a learning data generation program that evaluates the object detection model and deletes the added converted images if the evaluation value is less than the previous evaluation value.

Images