Training data generation device, training data generation method, and training data generation program

Abstract

To generate effective image data for improving detection accuracy without relying on human intervention. [Solution] An image conversion unit (21) inputs image data, which is training data (31), to an image conversion model (41) that converts image data, and acquires a converted image in which the image data is converted by the image conversion model (41). A training data addition unit (22) adds the converted image acquired by the image conversion unit (21) to the training data (31).

Description

[Technical Field]

[0001] The present disclosure relates to a technique for generating training data for an object detection model. [Background technology]

[0002] There are object detection models that detect objects with target attributes from image data. To improve the detection accuracy of an object detection model, it is necessary to train the object detection model using a variety of image data as training data. Creating training data is often done manually, which is a heavy workload. Furthermore, when training a model to detect dangerous situations, image data of dangerous situations is required, but it is difficult for people to actually recreate dangerous situations, and so creating training data is not easy.

[0003] Patent Document 1 describes that when there are insufficient teacher images, new teacher images are generated by processing the teacher images, such as enlarging, reducing, moving, and combining. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-169672 Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 basically only allows image data similar to the original training image to be generated as a new training image, making it difficult to generate effective image data to improve detection accuracy. The present disclosure aims to enable the generation of effective image data for improving detection accuracy without relying on human intervention. [Means for solving the problem]

[0006] The training data generation device according to the present disclosure includes: an image conversion unit that inputs image data, which is training data to be trained by an object detection model, into an image conversion model that converts image data, and acquires a converted image in which the image data is converted by the image conversion model; a training data adding unit that adds, to the training data, the converted image acquired by the image conversion unit and for which the degree of certainty that the object has been detected by the object detection model is less than an evaluation threshold; Equipped with the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, the image conversion unit inputs the image data and attribute information indicating a designated attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the designated attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby acquiring the explanatory text of the image data including an explanation of the designated attribute and an explanation of the angle of view generated by the sentence generation model, inputs the acquired explanatory text to the document conversion model, and instructs the document conversion model to leave the designated attribute and the angle of view unchanged and to change other parts, thereby acquiring a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby acquiring the image data generated by the image generation model as the converted image, When at least some of the converted images are not added to the learning data, the learning data adding unit causes the image converting unit to generate the missing number of converted images that are less than the designated number. [Effects of the Invention]

[0007] In the present disclosure, image data serving as training data is input to an image transformation model, and the transformed image obtained by transforming the image data using the image transformation model is added to the training data. This makes it possible to generate image data that has undergone various modifications, including not only simple processing such as enlarging, reducing, moving, and compositing, but also modifications to the background, surrounding objects, and lighting conditions. This makes it possible to generate image data that is effective in improving detection accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a training data generation device 10 according to a first embodiment. [Figure 2] 3 is a flowchart of processing by the training data generation device 10 according to the first embodiment. [Figure 3] FIG. 3 is an explanatory diagram of input to an image transformation model 41 according to the first embodiment. [Figure 4] FIG. 10 is a configuration diagram of a training data generation device 10 according to a second embodiment. [Figure 5] 10 is a flowchart of processing by the training data generation device 10 according to the second embodiment. [Figure 6] FIG. 10 is an explanatory diagram of input to a sentence generation model 42 according to the second embodiment. [Figure 7] FIG. 10 is an explanatory diagram of an output from a sentence generation model 42 according to the second embodiment. [Figure 8] FIG. 10 is an explanatory diagram of input to a document transformation model 43 according to the second embodiment. [Figure 9] FIG. 10 is a configuration diagram of a training data generation device 10 according to a third embodiment. [Figure 10] 11 is a flowchart of a learning data addition process according to the third embodiment. [Figure 11] FIG. 11 is an explanatory diagram of input to an attribute determination model 46 according to the third embodiment. [Figure 12] FIG. 10 is a configuration diagram of a training data generation device 10 according to a fourth embodiment. [Figure 13] 10 is a flowchart of processing by the training data generation device 10 according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiment 1 ***Configuration Description*** The configuration of a training data generation device 10 according to the first embodiment will be described with reference to FIG. The training data generation device 10 is a computer. The training data generation device 10 includes the following hardware components: a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware components via signal lines and controls the other hardware components.

[0010] The processor 11 is an IC that performs processing. IC stands for Integrated Circuit. Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0011] The memory 12 is a storage device that temporarily stores data. Specific examples of the memory 12 include SRAM and DRAM. SRAM stands for Static Random Access Memory. DRAM stands for Dynamic Random Access Memory.

[0012] The storage 13 is a storage device that stores data. A specific example of the storage 13 is an SSD. SSD is an abbreviation for Solid State Drive. The storage 13 may also be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD is an abbreviation for Secure Digital. DVD is an abbreviation for Digital Versatile Disk.

[0013] The communication interface 14 is an interface for communicating with external devices. Specific examples of the communication interface 14 include Ethernet (registered trademark), USB, and HDMI (registered trademark) ports. USB stands for Universal Serial Bus. HDMI stands for High-Definition Multimedia Interface.

[0014] The training data generation device 10 includes, as functional components, an image conversion unit 21 and a training data addition unit 22. The functions of the functional components of the training data generation device 10 are realized by software. The storage 13 stores a program that realizes the functions of each functional component of the training data generation device 10. The program is read into the memory 12 by the processor 11. The program is executed by the processor 11. This allows the functions of the functional components of the training data generation device 10 to be realized. The storage 13 stores training data 31. The training data 31 is data for training an object detection model. The training data 31 includes a plurality of image data. The object detection model is a training model that detects an object with a detection target attribute from the image data. The object detection model is, for example, a model configured using deep learning.

[0015] The training data generation device 10 is connected to an image transformation model 41 via a communication interface 14 . The image transformation model 41 is a so-called generative AI that transforms images. AI is an abbreviation for Artificial Intelligence. The image transformation model 41 may be configured using algorithms such as BERT and GPT, as specific examples. BERT is an abbreviation for Bidirectional Encoder Representations from Transformers. GPT is an abbreviation for Generative Pretrained Transformer. The learning model 112 may be configured by combining multiple algorithms including these algorithms.

[0016] Here, the image transformation model 41 is connected via the communication interface 14. In other words, the image transformation model 41 is assumed to be external to the training data generation device 10. However, the image transformation model 41 may also be internal to the training data generation device 10.

[0017] 1 shows only one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functions.

[0018] ***Explanation of Operation*** The operation of the training data generation device 10 according to the first embodiment will be described with reference to FIGS. The operation procedure of the training data generation device 10 according to the first embodiment corresponds to the training data generation method according to the first embodiment. Moreover, the program that realizes the operation of the training data generation device 10 according to the first embodiment corresponds to the training data generation program according to the first embodiment.

[0019] The processing of the training data generation device 10 according to the first embodiment will be described with reference to FIG. (Step S11: Input processing) The image conversion unit 21 extracts one image data from the learning data 31. The image conversion unit 21 inputs the extracted image data and attribute information indicating a designated attribute, which is an attribute of the detection target of the object detection model, to the image conversion model 41. If there are multiple attributes of the detection target of the object detection model, the image conversion unit 21 inputs attribute information designated for the extracted image data from among the multiple attributes. At this time, the image conversion unit 21 inputs a prompt to the image conversion model 41 indicating the conversion rule for the image data. The piece of image data extracted in step S11 is preferably an image in which a specific attribute was not detected or a different attribute was detected in the pre-learning model evaluation stage of the object detection model.

[0020] 3, the image conversion unit 21 inputs a prompt specifying the conversion rule for the image data, together with the image data and attribute information, to the image conversion model 41. The conversion rule specifies what should be changed randomly and what should never be changed. Items that can be randomly changed include the type of clothing, gender, background, etc. Other items such as the way the light hits the person, the number of people, etc. may also be specified depending on the situation. As items that should never be changed, information relating to attribute information and information relating to the angle of view are specified. Information about attribute information includes not only the attribute information itself but also things that accompany the attribute information. For example, if the attribute information is a white cane, things that accompany the attribute information include holding the white cane in the right hand. In the prompt in Figure 3, "information about ${attribute information}" is simply specified, but it is also possible to specify more specifically, such as "${attribute information} and the hand holding ${attribute information}." Here, ${attribute information} represents the attribute information specified in the input information ("white cane" in the above example). In other words, "${attribute information} and the hand holding ${attribute information}" is read as "white cane and the hand holding the white cane." Other ${***} are similarly replaced with the information specified as *** in the input information.

[0021] When the image data to be created has been determined, the image conversion unit 21 may specify the image data to be created in detail as a prompt.

[0022] (Step S12: Image conversion processing) The image transformation model 41 generates a specified number of transformed images by transforming the image data input in step S21. Then, the image transformation model 41 outputs the specified number of transformed images. The specified number is, for example, about 10% of the number of image data in the training data 31. At this time, the image transformation model 41 follows the instructions in the prompt and randomly changes those designated to be changed randomly for each transformed image. Also, the image transformation model 41 does not change those designated to never be changed. In other words, a transformed image is generated in which the designated attributes remain unchanged. Those that are not designated to be changed randomly or never be changed can be changed as desired. The image transformation unit 21 acquires the designated number of transformed images output by the image transformation model 41.

[0023] (Step S13: Adding learning data) The learning data adding unit 22 adds the designated number of converted images acquired by the image converting unit 21 in step S12 to the learning data 31. The learning data addition process shown in FIG. 2 is executed for each piece of attribute information that needs to be added to the learning data.

[0024] ***Effects of the First Embodiment*** As described above, the training data generation device 10 according to the first embodiment inputs image data, which is training data, to the image transformation model 41, and adds the converted image obtained by converting the image data by the image transformation model 41 to the training data 31. This makes it possible to generate image data with various modifications, such as changes to the background, surrounding objects, and lighting conditions. This makes it possible to generate image data that is effective in improving detection accuracy.

[0025] The training data generation device 10 according to the first embodiment can create training data that is difficult to collect, such as data for dangerous situations. Furthermore, the training data generation device 10 according to the first embodiment can increase the variety of training data that is difficult to collect.

[0026] Embodiment 2 The second embodiment differs from the first embodiment in that a converted image is generated by generating a description of the image data, converting the description, and generating image data from the converted description. In the second embodiment, this difference will be explained, and the explanation of the same points will be omitted.

[0027] ***Configuration Description*** The configuration of a training data generation device 10 according to the second embodiment will be described with reference to FIG. The image conversion model 41 includes a sentence generation model 42 , a document conversion model 43 , and an image generation model 44 . The text generation model 42 is a model that generates explanatory text for image data. The document conversion model 43 is a model that converts explanatory text. The image generation model 44 is a model that generates image data from explanatory text.

[0028] ***Explanation of Operation*** Referring to FIG. 5, the processing of the training data generation device 10 according to the second embodiment will be described. (Step S21: Image input processing) The image conversion unit 21 extracts one image data from the training data 31. The image conversion unit 21 inputs the extracted image data and attribute information indicating a designated attribute, which is an attribute of the detection target of the object detection model, to the sentence generation model 42. If there are multiple attributes of the detection target of the object detection model, the image conversion unit 21 inputs attribute information indicating one or more of the multiple attributes. At this time, the image conversion unit 21 inputs a prompt indicating the generation rules of the explanatory sentence to the sentence generation model 42.

[0029] 6, the image conversion unit 21 inputs a prompt specifying a rule for generating an explanatory sentence together with the image data and attribute information to the sentence generation model 42. The rule for generating an explanatory sentence specifies that the information on the angle of view and the attribute information must be explained.

[0030] (Step S22: Explanation generation process) The sentence generation model 42 generates and outputs an explanatory text that explains the image data input in step S21. At this time, as shown in Fig. 7, the sentence generation model 42 generates an explanatory text for the entire image data, including an explanation of the angle of view information and attribute information, in accordance with the instructions in the prompt. Then, the image conversion unit 21 acquires the explanatory text output by the sentence generation model 42.

[0031] (Step S23: Explanation input process) The image conversion unit 21 inputs the explanatory text acquired in step S22 and attribute information indicating the designated attribute, which is the attribute of the detection target of the object detection model, to the document conversion model 43. At this time, the image conversion unit 21 inputs a prompt to the image conversion model 41 instructing the conversion rule of the explanatory text. For example, as shown in Fig. 8, the image conversion unit 21 inputs a prompt specifying the conversion rules for the explanatory text along with the explanatory text and attribute information to the document conversion model 43. As in Fig. 3, the conversion rules specify what should be changed randomly and what should never be changed. Also, in Fig. 8, the explanatory text is converted into English. This is because there is a tendency for English explanatory text to have higher accuracy in generating an image from the explanatory text.

[0032] (Step S24: Explanation conversion process) The document conversion model 43 generates and outputs a converted sentence by converting the explanatory sentence input in step S23. Then, the image conversion unit 21 acquires the converted sentence output by the document conversion model 43. At this time, the document conversion model 43 follows the instructions in the prompt and randomly changes each converted sentence if it is specified to be changed randomly. The document conversion model 43 also does not change any sentences that are specified to never be changed. Any sentences that are not specified to be changed randomly or never be changed can be changed as desired. The document conversion model 43 also converts the explanatory sentence into English. Here, the converted sentences are generated as prompts for image generation.

[0033] (Step S25: Converted sentence input process) The image conversion unit 21 inputs the converted sentence (prompt for image generation) acquired in step S24 to the image generation model 44. At this time, the image conversion unit 21 inputs a prompt to the image generation model 44 to instruct it to generate a specified number of image data representing the converted sentence.

[0034] (Step S26: Image generation process) The image generation model 44 generates a specified number of image data representing the converted sentence input in step S25. The image generation model 44 generates image data by randomly modifying each image data while representing the converted sentence. The image generation model 44 then outputs the specified number of converted images. The image conversion unit 21 acquires the designated number of converted images output by the image generation model 44.

[0035] (Step S27: Learning data addition process) The learning data adding unit 22 adds the designated number of converted images acquired by the image converting unit 21 in step S26 to the learning data 31.

[0036] ***Effects of the Second Embodiment*** As described above, the training data generation device 10 according to the second embodiment generates a description of image data, converts the description, and generates image data from the converted description to generate a converted image. By generating a converted description by converting the description, the converted content can be clarified.

[0037] Furthermore, the training data generation device 10 according to the first embodiment can use separate generation AIs for the sentence generation model 42, the document conversion model 43, and the image generation model 44. This makes it possible to use the document conversion model 43 as a model specialized for language processing, and the sentence generation model 42 and the image generation model 44 as models specialized for image processing. As a result, it is possible to increase the processing speed and processing accuracy.

[0038] ***Other Configurations*** <Variation 1> In the second embodiment, one converted sentence is generated, and a specified number of converted images are generated from the converted sentence. A specified number of converted sentences may be generated, and one converted image may be generated from one converted sentence. Alternatively, M converted sentences may be generated, and N converted images may be generated from one converted sentence. In this case, M and N are integers greater than or equal to 1, and M x N = the specified number.

[0039] Embodiment 3 The third embodiment differs from the first and second embodiments in that it determines whether or not to add a converted image to the training data 31. In the third embodiment, this difference will be explained, and explanation of the same points will be omitted.

[0040] ***Configuration Description*** The configuration of a training data generation device 10 according to the third embodiment will be described with reference to FIG. The training data generation device 10 is connected to an object detection model 45 and an attribute determination model 46 via a communication interface 14. The object detection model 45 is a model that detects an object with a detection target attribute from image data, and is a model that learns the learning data 31. The attribute determination model 46 is a model that determines whether or not an object with a specified attribute is included in image data. Similar to the image conversion model 41, the attribute determination model 46 is a model that determines whether or not an object with a specified attribute is included in image data. is.

[0041] ***Explanation of Operation*** The learning data addition process (step S13 in FIG. 2, step S27 in FIG. 5) according to the third embodiment will be described with reference to FIG. In the learning data addition process, the following process is executed for each transformed image.

[0042] (Step S31: First input process) The learning data adding unit 22 inputs the transformed image of the target and the attribute information input when generating the transformed image to the object detection model 45. The attribute information input when generating the transformed image is the attribute information input in step S11 of FIG. 2 or step S21 of FIG. 5.

[0043] (Step S32: Object detection process) The object detection model 45 detects an object having attributes indicated by the attribute information input in step S31 from the transformed image of the object input in step S31. The object detection model 45 outputs the detection result and the degree of certainty of the detection. The learning data adding unit 22 acquires the detection results and confidence levels output by the object detection model 45.

[0044] (Step S33: Confidence Determination Process) The learning data adding unit 22 determines whether the confidence level acquired in step S32 is equal to or greater than a threshold value. If the confidence level is equal to or greater than the threshold, the training data adding unit 22 does not add the target transformed image to the training data 31 and ends the processing for the target transformed image. On the other hand, if the confidence level is less than the threshold, the training data adding unit 22 proceeds to step S34.

[0045] (Step S34: Second input process) The learning data adding unit 22 inputs the target transformed image and the attribute information input when generating the transformed image to the attribute determination model 46. At this time, as shown in Fig. 11 , the learning data adding unit 22 inputs a prompt to the attribute determination model 46 to instruct it to determine whether or not an object having an attribute indicated by the input attribute information exists in the input transformed image.

[0046] (Step S35: Attribute determination process) The attribute determination model 46 determines whether or not an object having the attribute indicated by the attribute information input in step S31 exists in the converted image of the target input in step S34. The attribute determination model 46 outputs the determination result. The learning data adding unit 22 acquires the determination result output by the attribute determination model 46 .

[0047] (Step S36: Result determination process) If the determination result obtained in step S35 indicates that an object exists in the target transformed image, the learning data adding unit 22 proceeds to step S37. On the other hand, if the determination result obtained in step S35 indicates that an object does not exist in the target transformed image, the learning data adding unit 22 does not add the target transformed image to the learning data 31, and ends the processing for the target transformed image.

[0048] (Step S37: Additional processing) The training data adding unit 22 adds the target transformed image to the training data 31.

[0049] ***Effects of the Third Embodiment*** The training data generation device 10 according to the third embodiment uses the object detection model 45 and the attribute determination model 46 to determine whether or not to add a converted image to the training data 31. This makes it possible to prevent a converted image that would degrade detection accuracy or a converted image that does not require training from being added to the training data 31. Specifically, in the determination in step S33, overlearning of the object detection model is prevented by not adding to the training data 31 images that can be detected by the current object detection model. In addition, in the determination in step S36, converted images that do not have the attribute information that is to be learned are not added to the training data 31, thereby preventing irrelevant images from being included in the training data 31.

[0050] ***Other Configurations*** <Variation 2> In the third embodiment, only the converted images that can be registered using the object detection model 45 and the attribute determination model 46 are added to the learning data 31. Therefore, even if the specified number of converted images are generated, the specified number of converted images are not necessarily added to the learning data 31. Therefore, if at least some of the converted images have not been added to the learning data 31, the learning data adding unit 22 may return the process to the image conversion unit 21 to regenerate the missing number of converted images. The missing number is a value obtained by subtracting the number of converted images added to the learning data 31 from the designated number. In other words, the learning data adding unit 22 may set the missing number as a new designated number and restart the process from step S11 in FIG. 2 or step S21 in FIG. 5.

[0051] Embodiment 4 The fourth embodiment differs from the third embodiment in that it also performs the learning of the object detection model 45. In the fourth embodiment, this difference will be explained, and explanation of the same points will be omitted.

[0052] ***Configuration Description*** The configuration of a training data generation device 10 according to the fourth embodiment will be described with reference to FIG. 9 in that the training data generation device 10 includes a learning unit 23 as a functional component. The learning unit 23, like the image conversion unit 21 and the training data addition unit 22, is realized by software.

[0053] ***Explanation of Operation*** Referring to FIG. 13, the processing of the training data generation device 10 according to the fourth embodiment will be described. (Step S41: Generation and addition process) The image conversion unit 21 and the learning data addition unit 22 add the converted image to the learning data 31 by the processing described in the third embodiment.

[0054] (Step S42: Learning process) The learning unit 23 causes the object detection model 45 to learn using the learning data 31 to which the converted image has been added in step S41.

[0055] (Step S43: Model evaluation process) The learning unit 23 evaluates the object detection model 45 learned in step S42. The object detection model 45 can be evaluated using existing technology. If the evaluation obtained in step S43 is lower than the previous evaluation, the learning unit 23 advances the process to step S44. If the evaluation obtained in step S43 is higher than the previous evaluation but does not reach the standard, the learning unit 23 returns the process to step S41. If the evaluation obtained in step S43 is higher than the previous evaluation but does not reach the standard, the learning unit 23 If so, the process ends.

[0056] (Step S44: Data deletion process) The learning unit 23 deletes the converted image added in step S41 from the learning data 31. The learning unit 23 also returns the object detection model 45 to the state before learning in step S42. Then, the learning unit 23 returns the process to step S41.

[0057] ***Effects of the Fourth Embodiment*** As described above, the training data generation device 10 according to the fourth embodiment repeatedly adds training data 31 and trains the object detection model 45. This makes it possible to improve the detection accuracy of the object detection model 45 without manual intervention.

[0058] ***Other Configurations*** <Variation 3> In the above-described embodiment, each functional component is realized by software. However, as a third modification, each functional component may be realized by hardware. The following describes the differences between this third modification and the above-described embodiment.

[0059] When each functional component is realized by hardware, the training data generation device 10 includes an electronic circuit instead of the processor 11, the memory 12, and the storage 13. The electronic circuit is a dedicated circuit for realizing the functions of each functional component, the memory 12, and the storage 13.

[0060] Possible electronic circuits include single circuits, composite circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by one electronic circuit, or each functional component may be realized by distributing it among a plurality of electronic circuits.

[0061] <Variation 4> As a fourth modification, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0062] The processor 11, memory 12, storage 13, and electronic circuitry are collectively referred to as a processing circuit. In other words, the functions of the functional components are realized by the processing circuit.

[0063] Furthermore, the term "unit" in the above description may be read as a "circuit," "step," "procedure," "process," or "processing circuit."

[0064] Various aspects of the present disclosure are summarized below as appendices. (Appendix 1) an image conversion unit that inputs image data, which is training data to be trained by an object detection model, into an image conversion model that converts image data, and acquires a converted image in which the image data is converted by the image conversion model; a learning data adding unit that adds the converted image acquired by the image conversion unit to the learning data; A training data generation device comprising: (Appendix 2) The image conversion unit converts the image data by instructing the image conversion model to maintain a state in which the image data includes a designated attribute. 2. A training data generation device according to claim 1. (Appendix 3) The image conversion unit converts the image data by instructing the image conversion model not to change a portion of the image data related to a specified attribute but to change other portions. 3. The training data generation device according to claim 1 or 2. (Appendix 4) the image conversion model includes a sentence generation model that generates a description of image data, a document conversion model that converts the description, and an image generation model that generates image data from the description; The image conversion unit inputs the image data to the sentence generation model, acquires an explanatory sentence for the image data generated by the sentence generation model, inputs the acquired explanatory sentence to the document conversion model, acquires a converted sentence converted by the document conversion model, and inputs the converted sentence to the image generation model, acquires the image data generated by the image generation model as the converted image. 3. The training data generation device according to claim 1 or 2. (Appendix 5) 5. The training data generation device according to claim 4, wherein the image conversion unit converts the explanatory text by instructing the document conversion model to leave the portion of the explanatory text related to a specified attribute unchanged and to change other portions. (Appendix 6) The training data generation device further comprises: a data determination unit that inputs the converted image acquired by the image conversion unit to the object detection model and determines whether or not the degree of certainty that the object has been detected by the object detection model is equal to or greater than an evaluation threshold; Equipped with The learning data adding unit adds the converted image to the learning data when the data determining unit determines that the confidence level is less than the evaluation threshold. 6. The training data generation device according to any one of appendices 1 to 5. (Appendix 7) the data determination unit inputs the transformed image and the designated attribute to an attribute determination model that determines whether an object with a specific attribute exists in image data when the certainty factor is less than the evaluation threshold, and obtains a determination result of whether an object with the designated attribute exists in the transformed image; The learning data adding unit adds the transformed image to the learning data when the data determining unit obtains a determination result that an object having the specified attribute exists in the transformed image. 7. The training data generation device according to claim 6. (Appendix 8) A computer inputs image data, which is training data to be trained by an object detection model, into an image transformation model that transforms image data, and obtains a transformed image in which the image data is transformed by the image transformation model; A training data generation method in which a computer adds the transformed image to the training data. (Appendix 9) an image conversion process in which image data, which is training data to be trained by an object detection model, is input to an image conversion model that converts image data, and a converted image is obtained by converting the image data using the image conversion model; a learning data addition process of adding the transformed image acquired by the image transformation process to the learning data; A learning data generation program that causes a computer to function as a learning data generation device that performs the above.

[0065] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Also, one or more of them may be implemented partially. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as needed. [Explanation of symbols]

[0066] 10 Training data generation device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Image conversion unit, 22 Training data addition unit, 23 Training unit, 31 Training data, 41 Image conversion model, 42 Text generation model, 43 Document conversion model, 44 Image generation model, 45 Object detection model, 46 Attribute determination model.

Claims

1. an image conversion unit that inputs image data, which is training data to be trained by an object detection model, into an image conversion model that converts image data, and acquires a converted image in which the image data is converted by the image conversion model; a training data adding unit that adds, to the training data, the converted image acquired by the image conversion unit and for which the degree of certainty that the object has been detected by the object detection model is less than an evaluation threshold; Equipped with the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, the image conversion unit inputs the image data and attribute information indicating a designated attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the designated attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby acquiring the explanatory text of the image data including an explanation of the designated attribute and an explanation of the angle of view generated by the sentence generation model, inputs the acquired explanatory text to the document conversion model, and instructs the document conversion model to leave the designated attribute and the angle of view unchanged and to change other parts, thereby acquiring a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby acquiring the image data generated by the image generation model as the converted image, The learning data generation device, wherein if at least some of the transformed images are not added to the learning data, the learning data addition unit causes the image conversion unit to acquire the missing number of transformed images that are lacking from the specified number.

2. an image conversion unit that inputs image data, which is training data to be trained by an object detection model, into an image conversion model that converts image data, and acquires a converted image in which the image data is converted by the image conversion model; a training data adding unit that adds the converted image, acquired by the image conversion unit and having a degree of certainty that the object has been detected by the object detection model, less than an evaluation threshold to the training data; a learning unit that learns using learning data to which the converted image has been added; Equipped with the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, the image conversion unit inputs the image data and attribute information indicating a designated attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the designated attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby acquiring the explanatory text of the image data including an explanation of the designated attribute and an explanation of the angle of view generated by the sentence generation model, inputs the acquired explanatory text to the document conversion model, and instructs the document conversion model to leave the designated attribute and the angle of view unchanged and to change other parts, thereby acquiring a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby acquiring the image data generated by the image generation model as the converted image, The learning unit evaluates the object detection model, and deletes the added converted image if the evaluation value is less than the previous evaluation value.

3. The learning unit evaluates the object detection model, and when the evaluation value is equal to or greater than the previous evaluation value and less than a reference value, adds a converted image again and performs learning, and when the evaluation value is equal to or greater than the previous evaluation value and greater than a reference value, ends learning. The training data generating device according to claim 2 .

4. The training data generation device further comprises: a data determination unit that inputs the converted image acquired by the image conversion unit to the object detection model and determines whether or not the degree of certainty that the object has been detected by the object detection model is equal to or greater than an evaluation threshold; Equipped with The learning data adding unit adds the converted image to the learning data when the data determining unit determines that the confidence level is less than the evaluation threshold. The training data generating device according to any one of claims 1 to 3.

5. A computer inputs image data, which is training data to be trained by an object detection model, into an image transformation model that transforms image data, and obtains a transformed image in which the image data is transformed by the image transformation model; The computer adds the transformed image, for which the confidence level of the object detected by the object detection model is less than an evaluation threshold, to the training data; the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, a computer inputs the image data and attribute information indicating a specified attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the specified attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby obtaining an explanatory text for the image data including an explanation of the specified attribute and an explanation of the angle of view generated by the sentence generation model, inputs the obtained explanatory text to the document conversion model, and instructs the document conversion model to leave the specified attribute and the angle of view unchanged and to change other parts, thereby obtaining a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby obtaining the image data generated by the image generation model as the converted image, A learning data generation method, in which a computer acquires the missing number of transformed images from a specified number if at least some transformed images have not been added to the learning data.

6. an image conversion process in which image data, which is training data to be trained by an object detection model, is input to an image conversion model that converts image data, and a converted image is obtained by converting the image data using the image conversion model; a training data addition process of adding the transformed image, which is acquired by the image transformation process and has a confidence level of an object detected by the object detection model that is less than an evaluation threshold, to the training data; causing a computer to function as a learning data generation device that performs the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, the image conversion process inputs the image data and attribute information indicating a specified attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the specified attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby acquiring the explanatory text of the image data including an explanation of the specified attribute and an explanation of the angle of view generated by the sentence generation model, inputs the acquired explanatory text to the document conversion model, and instructs the document conversion model to leave the specified attribute and the angle of view unchanged and to change other parts, thereby acquiring a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby acquiring the image data generated by the image generation model as the converted image, The learning data addition process is a learning data generation program that, if at least some of the converted images are not added to the learning data, acquires the missing number of converted images that are lacking from the specified number through the image conversion process.

7. A computer inputs image data, which is training data to be trained by an object detection model, into an image transformation model that transforms image data, and obtains a transformed image in which the image data is transformed by the image transformation model; The computer adds the transformed image, for which the confidence level of the object detected by the object detection model is less than an evaluation threshold, to the training data; The computer learns using the training data to which the transformed image has been added; the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, a computer inputs the image data and attribute information indicating a specified attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the specified attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby obtaining an explanatory text for the image data including an explanation of the specified attribute and an explanation of the angle of view generated by the sentence generation model, inputs the obtained explanatory text to the document conversion model, and instructs the document conversion model to leave the specified attribute and the angle of view unchanged and to change other parts, thereby obtaining a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby obtaining the image data generated by the image generation model as the converted image, A learning data generation method in which a computer evaluates the object detection model and deletes the added transformed image if the evaluation value is less than the previous evaluation value.

8. an image conversion process in which image data, which is training data to be trained by an object detection model, is input to an image conversion model that converts image data, and a converted image is obtained by converting the image data using the image conversion model; a training data addition process of adding, to the training data, the transformed image acquired by the image transformation process, for which the confidence level of the object detected by the object detection model is less than an evaluation threshold; a learning process for learning using the learning data to which the converted image has been added; causing a computer to function as a learning data generation device that performs the image conversion model includes a sentence generation model that generates a description of the image data, a document conversion model that converts the description, and an image generation model that generates image data from the description, the image conversion process inputs the image data and attribute information indicating a specified attribute to the sentence generation model, and instructs the sentence generation model to generate an explanatory text including an explanation of the specified attribute indicated by the attribute information included in the image data and an explanation of the angle of view of the image data, thereby acquiring the explanatory text of the image data including an explanation of the specified attribute and an explanation of the angle of view generated by the sentence generation model, inputs the acquired explanatory text to the document conversion model, and instructs the document conversion model to leave the specified attribute and the angle of view unchanged and to change other parts, thereby acquiring a converted text converted by the document conversion model, and inputs the converted converted text to the image generation model, thereby acquiring the image data generated by the image generation model as the converted image, The learning process is a learning data generation program that evaluates the object detection model and deletes the added converted image if the evaluation value is less than the previous evaluation value.

Images