Data generation device and data generation method

Abstract

Collect high-quality training data at low cost. [Solution] A data generation device for generating training data for an image recognition model that outputs recognition results of an input image, comprising a computer having a computing unit that executes a program and a storage device accessible by the computing unit, wherein the computing unit acquires scene specification information that specifies environmental elements constituting a scene and objects included in the scene, calculates reality constraints that indicate the conditions on the image that the training data must follow based on a traffic situation description that describes the traffic situation in a predetermined format and camera settings which are camera setting information, inputs the scene specification information to the image generation model, generates candidate training data images with the camera as the viewpoint, determines a reality consistency score that indicates the consistency between the generated candidate training data images and the reality constraints, and decides whether or not to update the training data specification information that specifies the training data according to the reality consistency score.

Description

Technical Field

[0001] The present invention relates to a data generation device and a data generation method, and particularly relates to a technology for generating machine learning data for an AI model for autonomous driving.

Background Art

[0002] With the development of AI (Artificial Intelligence) technology, the realization of autonomous driving of automobiles is approaching. Currently, the autonomous driving technology in operation is autonomous driving with a person on board, and in places where the system has difficulty coping, it is switched to manual operation for safety. Autonomous driving without a person on board is limited operation in sparsely populated areas and events, and an expansion of autonomous driving without a person on board is desired. In expanding autonomous driving without a person on board, it is necessary to improve the AI model that can handle situations (corner cases) that occur only rarely.

[0003] The input to the AI model for autonomous driving accurately recognizes the distance and direction between the host vehicle and surrounding vehicles, people, structures, etc. from the data acquired by cameras and three-dimensional sensors attached to the vehicle.

[0004] The AI model learns using learning data in which examples and correct answers are paired. It is known that the accuracy of the AI model improves as the number of learning data increases, but it requires a large cost to collect data on corner cases in the actual environment. Current general in-vehicle AI models recognize vehicles, people, and bicycles, but do not recognize corner cases (for example, obstacles such as debris on the road), and preventing accidents through accurate recognition of such obstacles is an issue.

[0005] To address the aforementioned challenges, Patent Document 1 discloses a method for collecting corner case training data using three-dimensional computer simulation. Patent Document 1 describes a 3D virtual space generation and rendering means for generating and rendering objects in a 3D virtual space using 3D computer graphics; an AI recognition target information generation and rendering means for generating and rendering information about recognition targets of an AI model in the 3D virtual space using 3D computer graphics; a 3D virtual space scene generation and rendering means for generating and rendering various scenes by changing the settings of at least one of the 3D virtual space and the recognition target using 3D computer graphics; a CG image including the recognition target generated by the 3D virtual space scene generation and rendering means; and information about the recognition target generated by the AI ​​recognition target information generation and rendering means for machine learning of the AI ​​model. A machine learning system is disclosed comprising: machine learning data generation means for generating necessary training data; AI model machine learning means for training the AI ​​model using the training data generated by the machine learning data generation means; and AI trained model execution means for executing processing by the AI ​​model using the trained model trained by the AI ​​model machine learning means. The AI ​​recognition target / information generation and drawing means is characterized by having: a 3D spatial information generation function for generating 3D spatial information of feature points of the recognition target in the 3D virtual space; a 3D spatial information placement and drawing function for placing and drawing the 3D spatial information of the recognition target at a specified position in the 3D virtual space by referring to the 3D spatial information; and a 2D image position information generation function for generating position information on a 2D image of the recognition target drawn by the 3D spatial information placement and drawing function.

[0006] Furthermore, Non-Patent Document 1 discloses a method for generating an arbitrary image from a string of characters. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2023-146041 [Non-patent literature]

[0008] [Non-Patent Document 1] Tim Brooks, et al., "Video generation models as world simulators", [online], February 15, 2024, [February 29, 2024], Internet<URL: https: / / openAI.com / research / video-generation-models-as-world-simulators> [Overview of the project] [Problems that the invention aims to solve]

[0009] The invention described in Patent Document 1 allows for the reproduction of corner cases in a computer-calculated three-dimensional space, and enables the theoretical calculation of "two-dimensional images" and "distances to various objects" from any position in the three-dimensional space, thereby enabling the collection of low-cost training data. Furthermore, since roads and the surrounding environment can be easily modified in computer simulations, a wide variety of training data can be collected. While the invention described in Patent Document 1 enables the collection of low-cost and diverse training data, the resolution of the training data remains a challenge. To reproduce a real environment with high resolution in three-dimensional space, complex three-dimensional models of vehicles, people, and the surrounding environment are necessary. However, the cost of creating complex three-dimensional models is high, necessitating their simplification. The technology described in Non-Patent Document 1 enables the low-cost creation of training data images that closely resemble the real environment of a corner case from text-based instructions (prompts). While the technology described in Non-Patent Document 1 can create training data with lower cost and higher resolution than computer simulations, its quality remains a challenge. Image generation using prompts makes it difficult to give accurate instructions to the AI ​​model, and inconsistencies, such as distorted roads, are likely to occur in the generated images.

[0010] The present invention aims to provide a data generation device and a data generation method for collecting high-quality training data at low cost. [Means for solving the problem]

[0011] A typical example of the invention disclosed in this application is as follows: A data generation device for generating training data for an image recognition model that outputs recognition results of an input image, comprising a computer having a computing unit for executing a program and a storage device accessible by the computing unit, wherein the computing unit acquires scene specification information that specifies environmental elements constituting a scene and objects included in the scene, calculates reality constraints that indicate the conditions on the image that the training data must follow based on a traffic situation description that describes the traffic situation in a predetermined format and camera settings which are camera setting information, inputs the scene specification information to the image generation model, generates candidate training data images with the camera as the viewpoint, determines a reality consistency score that indicates the consistency between the generated candidate training data images and the reality constraints, and determines whether or not to update the training data specification information that specifies the training data according to the reality consistency score. [Effects of the Invention]

[0012] According to one aspect of the present invention, the quality of training data images can be improved. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0013] [Figure 1] This is a configuration diagram of a data generation device according to the first embodiment of the present invention. [Figure 2] This is a flowchart of the process performed by the data generation device according to the first embodiment of the present invention. [Figure 3] This figure shows the process performed by the reality constraint calculation unit according to the first embodiment of the present invention. [Figure 4] This figure shows the process performed by the reality consistency score determination unit according to the first embodiment of the present invention. [Figure 5] It is a diagram showing a prompt sentence update unit according to the first embodiment of the present invention. [Figure 6] It is a flowchart of the process executed by the prompt sentence update unit according to the first embodiment of the present invention. [Figure 7] It is a diagram showing an example of a user interface screen output by the data generation device according to the first embodiment of the present invention. [Figure 8] It is a block diagram showing the physical configuration of a computer constituting the data generation device according to the first embodiment of the present invention. [Figure 9] It is a diagram showing an example of a prompt generated by the data generation device according to the first embodiment of the present invention. [Figure 10] It is a configuration diagram of a data generation device according to the second embodiment of the present invention. [Figure 11] It is a flowchart of the process executed by the data generation device according to the second embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0014] The following embodiments are merely examples for explaining the present invention, and the present invention is not limited to these embodiments. It is obvious that various modifications and alterations are possible within the scope of the technical idea of the present invention, other than the components, means, or combinations thereof described in each of the above embodiments.

[0015] <First Embodiment> FIG. 1 is a configuration diagram of a data generation device 100 according to the first embodiment.

[0016] The data generation device 100 includes a scene specification unit 101, a learning data candidate generation unit 103, a domain-specific knowledge specification unit 108, a real-world constraint calculation unit 111, a real-world consistency score determination unit 107, a data selection unit 106, and a prompt sentence update unit 102.

[0017] The scene specification unit 101 sets the scene specification information for the generated image. The scene specification information represents the scene to be generated and the location, type, quantity, behavior, etc. of objects in natural language or a combination of natural language and images. Scene specification information in natural language is also called a prompt. The scene specification information includes, for example, information (first specification information) that constitutes the scene, such as road structure, time of day, and streetscape. Furthermore, the scene specification information may also include object information (second specification information) that specifies the number, type, and behavior of objects. Specifically, the scene specification information may include, for example, scenes such as "urban environment," "highway," and "general road," and object types such as "vehicle" and "pedestrian." Furthermore, the scene specification information may include, for example, the location of objects such as "preceding vehicle" and "pedestrian on a crosswalk." Furthermore, the scene specification information may include, for example, the quantity of objects such as "two preceding vehicles" and "five pedestrians," and the behavior of objects such as "pedestrian crossing a crosswalk." The above scene specification information is illustrative and not limited to these examples.

[0018] The training data candidate generation unit 103 receives scene specification information set by the scene specification unit 101, inputs the received scene specification information into a large-scale AI model (hereinafter also called an image generation model), obtains candidate images for training data (hereinafter also called training data candidate images) and correct values, and outputs them to the data selection unit 106 or the reality consistency score determination unit 107. The training data candidate generation unit 103 may have a large-scale AI model internally, or it may use a large-scale AI model provided outside the data generation device 100.

[0019] The image generation unit 104 generates candidate images for training data based on the scene specification information set by the scene specification unit 101. The image generation unit 104 can generate images of the driving environment from the perspective of a camera installed on the vehicle. The candidate images for training data generated by the image generation unit 104 may be a single image, images from multiple viewpoints assuming multiple cameras mounted on the vehicle, or multiple images that change over time. Furthermore, the candidate images for training data generated by the image generation unit 104 may be images from various cameras according to the scene specification information, such as panoramic images or fisheye images. The image generation unit 104 can use an image generation model (e.g., stable diffusion) that generates images having features described in the input text.

[0020] The ground truth value generation unit 105 takes the candidate training data images generated by the image generation unit 104 as input and generates ground truth values ​​for the candidate training data images. For example, the ground truth values ​​may be obtained by recognizing the environment of the generated image using an environment recognition AI model that can derive ground truth values ​​with high accuracy, such as a machine learning model that is more multilayered, has more dimensions and a larger number of parameters than the target in-vehicle environment recognition AI model, or a machine learning model that has been trained on more data than the target in-vehicle environment recognition AI. Alternatively, the image generation and the predicted ground truth values ​​may be output simultaneously using a diffusion model or the like. By using an environment recognition AI model that can derive ground truth values ​​with high accuracy, the position of the object on the image (high IOU, which is the overlap rate between the ground truth value and the bounding box), the distance to the object (the difference between the distance from the camera associated with the pixel and the ground truth value), and the type of object can be accurately derived.

[0021] The ground truth generation unit 105 outputs an object class representing the type of object in the image, such as a roadside, white lines, vehicles, or animals, and position information indicating the position of the object in the image. The ground truth generation unit 105 may also calculate other ground truth values ​​in the environmental recognition AI model, such as parallax / depth information and drivable area information.

[0022] The domain-specific knowledge specification unit 108 retrieves the traffic situation description 109 and the camera settings 110. Details of the traffic situation description 109 and the camera settings 110 will be described later.

[0023] The reality constraint calculation unit 111 obtains the traffic situation description 109 and camera settings 110 from the domain-specific knowledge specification unit 108 and calculates the reality constraints. The reality constraints refer to the conditions on the image that the training data 112 must follow. The processes performed by the reality constraint calculation unit 111 will be described later with reference to Figure 3.

[0024] The reality consistency score determination unit 107 receives the candidate training data images and correct values ​​output from the candidate training data generation unit 103, as well as the reality constraints calculated by the reality constraint calculation unit 111. It then calculates the difference between the generated candidate training data images and the reality constraints as the reality consistency score. The processes executed by the reality consistency score determination unit 107 will be described later with reference to Figure 4.

[0025] The data selection unit 106 takes the reality consistency score as input and decides whether to save the candidate training data images to memory as training data 112 or discard them as unsuitable data, and then saves or discards them according to the decision.

[0026] The prompt text update unit 102 updates the prompt text used for scene specification so that the reality consistency score of the training data candidate images is improved. The processes executed by the prompt text update unit 102 will be described later with reference to Figures 5 and 6.

[0027] Figure 8 is a block diagram showing the physical configuration of the computer constituting the data generation device 100 according to the first embodiment.

[0028] The data generation device 100 in this embodiment is composed of a computer having a processor (CPU) 1, memory 2, auxiliary storage device 3, and communication interface 4. The data generation device 100 may also have an input interface 5 and an output interface 6.

[0029] Processor 1 is an arithmetic unit that executes programs stored in memory 2. By executing various programs, Processor 1 realizes the various functional units of the data generation device 100 (for example, the scene specification unit 101, the prompt text update unit 102, the training data candidate generation unit 103, the data selection unit 106, the reality consistency score determination unit 107, the domain-specific knowledge specification unit 108, the reality constraint calculation unit 111, etc.). Note that some of the processing performed by Processor 1 when executing programs may be executed by other arithmetic units (for example, hardware such as a GPU, ASIC, FPGA, etc.).

[0030] Memory 2 is a storage device that includes ROM, a non-volatile memory element, and RAM, a volatile memory element. ROM stores immutable programs (e.g., BIOS). RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by processor 1 and data used during program execution.

[0031] The auxiliary storage device 3 is, for example, a high-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device 3 also stores data used by the processor 1 when executing a program and the program that the processor 1 executes. In other words, the program is read from the auxiliary storage device 3, loaded into memory 2, and executed by the processor 1, thereby realizing each function of the data generation device 100.

[0032] Communication interface 4 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0033] Input interface 5 is an interface to which input devices such as a keyboard 7 and a mouse 8 are connected and to receive input from the operator. Output interface 6 is an interface to which output devices such as a display device 9 and a printer (not shown) are connected and to output the program execution results in a format that can be viewed by the operator.

[0034] The program executed by processor 1 is provided to data generation device 100 via removable media (such as CD-ROM or flash memory) or a network, and stored in non-volatile auxiliary storage device 3, which is a non-temporary storage medium. For this reason, it is preferable for data generation device 100 to have an interface for reading data from removable media.

[0035] The data generation device 100 is a computer system that operates on a single physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer built on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple units may be combined and operate on a single physical or logical computer.

[0036] Figure 2 is a flowchart of the process performed by the data generation device 100 according to the first embodiment.

[0037] First, in step S202, the scene specification unit 101 sets the scene specification information (prompt). The scene specification unit 101 sets the scene specification information based, for example, on natural language input received from the user via the input interface 5. In addition to natural language, it is also possible to set the scene specification information using an image input by the user via the input interface 5. Scene specification information in natural language may be, for example, "In an urban environment, there are two preceding vehicles and five pedestrians are crossing a crosswalk." When adding an image as scene specification information, the changes from the image are specified in natural language. For example, if there are two preceding vehicles in the image, specifying "There are three preceding vehicles" in natural language will cause the scene specification unit 101 to generate a scene with one more preceding vehicle added while maintaining the style of the input image.

[0038] In step S203, the domain-specific knowledge specification unit 108 obtains the traffic situation description 109 and camera settings 110. The traffic situation description 109 and camera settings 110 may be received from the user via the input interface 5, or they may be stored in memory 2 beforehand. Here, the traffic situation specified by the scene specification information and the traffic situation description 109 may differ. Also, if the scene specification information is set in natural language and images, and there is no difference in the number of objects or the shape of roads and white lines between the scene specified by natural language and the scene specified by images, the input image may be converted into a traffic situation description 109 to create the traffic situation description 109.

[0039] In step S204, the reality constraint calculation unit 111 calculates the reality constraints.

[0040] In step S205, the reality consistency score determination unit 107 generates candidate training data images and correct values ​​using the training data candidate generation unit 103.

[0041] In step S206, the reality consistency score determination unit 107 calculates a reality consistency score between the training data candidate images and the reality constraints.

[0042] In step S207, the data selection unit 106 determines whether to adopt a candidate image as training data 112 or discard it, based on whether the reality consistency score between the candidate image and the reality constraints is below a predetermined threshold value. If the reality consistency score is lower than the predetermined threshold value, the data selection unit 106 discards the candidate image, and in step S208, the prompt text update unit 102 updates the prompt. On the other hand, if the reality consistency score generated by the reality consistency score determination unit 107 exceeds the predetermined threshold value, the data selection unit 106 saves the candidate image as training data 112 in memory in step S209. The training data 112 saved in memory can be used, for example, as training data for an image recognition model used in an image processing device mounted on a vehicle (not shown). By training an image recognition model using the training data 112, an image recognition model can be created that, for example, outputs recognition results of input images acquired by an in-vehicle camera. The training of the image recognition model using the training data 112 will not be explained here, but it may be performed inside the data generation device 100 or on an external information processing device.

[0043] Figure 3 shows the process performed by the reality constraint calculation unit 111 according to the first embodiment.

[0044] The reality constraint calculation unit 111 has a 3D model engine and calculates reality constraints based on the traffic situation description 109 and camera settings 110 obtained from the domain-specific knowledge specification unit 108.

[0045] The traffic situation description 109 is represented by the overview information 301, which describes the traffic situation specified in the domain-specific knowledge specification unit 108. The overview information 301 is a format of information also called BEV (Bird's Eye View), and is provided as an overview view from above, showing the surroundings of the vehicle. It is an all-directional representation showing the class, position, and position of objects. For example, the overview information 301 is input to the reality constraint calculation unit 111 in XML format or CSV format.

[0046] Camera settings 110 are camera setting information specified by the domain-specific knowledge specification unit 108, and include information such as the camera mounting position, focal length, type, and distortion, and are input to the reality constraint calculation unit 111 in a file format similar to the overhead view information 301.

[0047] The reality constraint calculation unit 111 calculates reality constraints 304 based on the traffic situation description 109 and camera settings 110, which are based on the overhead information 301. The calculated reality constraints 304 are represented by a simple diagram of lines and objects in the space described in the traffic situation description 109, and are the coordinates, size, and class information of objects when the space is photographed and rendered according to the camera settings 110. The object classes may be, for example, white lines, roadsides, and others. Other objects may include vehicles, people, motorcycles, animals, furniture, and any other objects that have fallen onto the road.

[0048] The reality constraint calculation unit 111 calculates the field of view of the image, the coordinates of the road edge and white lines within that field of view, and the position coordinates and size of other objects as reality constraints 304, and can output the reality constraints 304 as visual image information.

[0049] The reality constraint calculation unit 111 generates terrain in a three-dimensional environment from information on road edges, white lines, and intersections in the traffic situation description 109, for example, using a simulator, and places boxes in the environment using the position and size information of objects. It places a virtual camera in the three-dimensional environment using the vehicle's position and camera settings 110 in the traffic situation description 109, and renders the scene in the three-dimensional environment from the viewpoint of the placed virtual camera to generate an image. The same environment recognition AI model as the ground truth generation unit 105 is applied to the image generated by rendering to estimate the position coordinates and sizes of white lines, road edges, and other objects in the image, and outputs visual image information.

[0050] Figure 4 shows the process performed by the reality consistency score determination unit 107 according to the first embodiment.

[0051] The reality consistency score determination unit 107 includes an object position score calculation unit 404 that calculates an object position score using the correct values ​​403 generated by the correct value generation unit 105, and an object class score calculation unit 405 that calculates an object class score using the correct values ​​403. The reality consistency score determination unit 107 determines the reality consistency score based on the calculated object position score and object class score. The reality consistency score may be the sum of the object position score and object class score or a weighted average. The method for calculating the reality consistency score is not limited to this.

[0052] The object position score calculation unit 404 calculates the error between the object positions included in the ground truth value 403 and the object positions specified by the real-world constraints 304 as the object position score. The object position score calculation unit 404 calculates a conversion parameter to match the generated ground truth value 403 with the real-world constraints 304 positions, for example using the least squares method, and calculates the error ratio between the generated ground truth value 403 and the object positions in the image under the real-world constraints 304. The error ratio is expressed, for example, by subtracting the overlap rate (IOU) of the bounding boxes of the two from 1. Alternatively, the IOU may be calculated by the proportion of pixels that are white lines in the image of the generated ground truth value 403 that are also white lines in the real-world constraints 304.

[0053] In the ground truth generation unit 105, the environment recognition unit 402 estimates the object position and class from the training data candidate images 401 generated by the image generation unit 104 using an environment recognition AI model, and converts the training data candidate images 401 into ground truth values ​​403 of image representations that conform to the reality constraints 304.

[0054] The object position score calculation unit 404 may calculate the percentage of pixel error for the white line class, the percentage of pixel error for the drivable area class, and the percentage of position error for other object position detection, normalize each class error to information between 0 and 1, and then calculate the average.

[0055] The object class score calculation unit 405 calculates the object class score as the difference between the number of objects of each class in the image generated by the image generation unit 104 and the number of objects of each class specified by the reality constraint 304.

[0056] Figure 5 shows the prompt text update unit 102 according to the first embodiment.

[0057] The prompt text update unit 102 includes a scene specification update prompt creation unit 501 and an image generation prompt generation unit 502. The prompt text update unit 102 receives scene specification information and reality consistency score used in the generation of the previous training data candidate images by the image generation unit 104. The scene specification update prompt creation unit 501 receives the scene specification information and reality consistency score used in the generation of the previous training data candidate images and creates a scene specification update prompt.

[0058] The image generation prompt generation unit 502 can utilize machine learning models such as generative AI trained to solve general-purpose problems, input scene specification update prompts to a large-scale AI model (language model), acquire updated scene specification information, and output it. For example, the image generation prompt generation unit 502 outputs updated scene specification information that improves the reasons for consistency degradation in order to increase the reality consistency score. The image generation prompt generation unit 502 may have a large-scale AI model internally, or it may utilize a large-scale AI model provided outside the data generation device 100.

[0059] The scene specification update prompt creation unit 501 integrates the scene specification information used in the previous image generation, the reality consistency score, a sentence explaining the reason for the deterioration of object positions, and a sentence instructing the creation of an image generation prompt that reduces object class errors, and creates it as a scene specification update prompt. It has a deterioration reason generation function that inputs the generated prompt into a large-scale AI model to generate the reason for the deterioration of object positions.

[0060] Figure 6 is a flowchart of the processes executed by the prompt text update unit 102 and the scene specification unit 101 according to the first embodiment.

[0061] Steps S602 to S606 describe the processing flow in the scene specification update prompt creation unit 501 within the prompt text update unit 102. In step S602, the scene specification information and reality consistency score used to generate the previous training data candidate images are obtained from the image generation unit 104 and added to the scene specification update prompt. In step S603, the prompt text update unit 102 determines whether the object class score is 1 or greater. If it is determined in step S603 that the object class score is 1 or greater, in step S604, the prompt text update unit 102 adds a class error suppression prompt to the scene specification update prompt.

[0062] An example of a class error suppression prompt might be: "A class score of 0 indicates that the object has been created correctly. A class score of 0 or greater indicates that an extra object has been added or that the object class has not been created. Based on this, explain the reason for the degraded class score, such as currently having one car missing. Furthermore, generate a prompt that will reduce the class error."

[0063] In step S605, the prompt text update unit 102 determines whether the object position score is above a predetermined threshold. If it is determined in step S605 that the object position score is above a predetermined threshold, in step S606, the large-scale AI model is used to generate quantitative error causes for the object position based on the generated training data candidate images, reality constraints, and object position score, and an object position error suppression prompt is added to the scene specification update prompt to suppress the error.

[0064] An example of an object position error suppression prompt might be: "Based on the training data candidate images, reality constraints, and object position score, quantitatively tell us whether there are errors in each item, such as the left-right error of the object, the object's scale error, and defects in the road structure, for example, if the road is shifted approximately 0.4m to the left. Furthermore, generate prompts that reduce the object position error."

[0065] On the other hand, if it is determined in step S605 that the object position score is smaller than a predetermined threshold, the object position error suppression prompt is not set, and the process proceeds to step S607.

[0066] In step S607, the image generation prompt generation unit 502 obtains a prompt for updating the scene specification, provides the prompt to the large-scale AI model to generate scene specification information, and outputs it to the scene specification unit 101. An example of the final updated scene specification information is "Delete one vehicle in the front right, slide the road to the right (approximately 0.4m)".

[0067] The scene specification information updated by the prompt text update unit 102 is input again to the training data candidate generation unit 103 along with the training data candidate image generated previously. The training data candidate generation unit 103 then generates a new training data candidate image that matches the scene specification information from the input scene specification information and the training data candidate image generated previously, and outputs it to the reality consistency score determination unit 107. The reality consistency score determination unit 107 calculates a reality consistency score that represents the consistency between the newly generated training data candidate image and the scene specification information, and outputs it to the prompt text update unit 102. In this way, the data generation device 100 of this embodiment repeatedly generates images that match the scene specification information while updating the scene specification information.

[0068] Figure 9 shows an example of a scene specification update prompt input to the large-scale AI model by the image generation prompt generation unit 502. The scene specification update prompt consists of the previous image generation prompt, the previous training data candidate image, reality constraints, object class score, object position score, and instructions to the large-scale AI model. In addition to these, the scene specification update prompt may also include information on previous image generation prompts and their rejection rates. The processing shown in Figure 6 is executed by the data generation device 100, for example, by the prompt shown in Figure 9. The format and language of the prompt may be other than those shown in Figure 9. That is, according to the prompt shown in Figure 9, the previous prompt, training data candidate image, object class score, and object position score are provided, and a prompt is generated that explains the reason for degradation as an instruction, and reduces the object class error and object position error. This makes it possible to generate training data with high consistency (high quality) with real-world driving scenes by giving accurate instructions to the image generation unit 104. Furthermore, by calculating a reality consistency score that indicates consistency with the reality constraints 304, which are constraints that the images generated as training data must adhere to, it is possible to appropriately evaluate whether or not inconsistencies occur in the generated images, such as distortion of roads.

[0069] Figure 7 shows an example of a user interface screen 701 output by the data generation device 100 according to the first embodiment.

[0070] The user interface screen 701 includes an image display area, a discrimination threshold input area 702, and a consistency check result display area 703. The image display area displays candidate training data images 401, reality constraints 304, and correct values ​​403, visualizing the generated data. The consistency check result display area 703 changes its display content depending on the input discrimination threshold. The consistency check result display area 703 includes the identification information of the scenario, the identification information of the generated image, the consistency check result, the identification information of the generative model used to generate the image, and the prompts used to generate the image.

[0071] As described above, according to the embodiments of the present invention, in the generation of training data images by prompts, a score indicating whether the generated images are realistic is calculated, the prompts are modified based on the calculated score, and the training data images are regenerated, thereby improving the quality of the training data images.

[0072] <Second Embodiment> Next, a second embodiment will be described using Figures 10 and 11. Note that the common points with the first embodiment will not be explained.

[0073] Figure 10 is a configuration diagram of the data generation device 200 according to the second embodiment.

[0074] The data generation device 200 according to the second embodiment differs from the data generation device 100 according to the first embodiment in that it has a camera constraint calculation unit 801 and a learning data candidate generation adjustment unit 802.

[0075] In the data generation device 100 according to the second embodiment, if the camera settings of the images used during training of the image generation model used in the training data candidate generation unit 103 differ from the camera settings 110 specified in the domain-specific knowledge specification unit 108, even if the real-world consistency score exceeds the standard value, the real-world consistency based on the camera settings (hereinafter also referred to as the camera constraint score) may not be satisfied. Camera settings include the camera's focal length, resolution, pose, etc. Real-world consistency based on camera settings (camera constraint score) indicates the degree to which the training data satisfies the constraints related to the camera settings. For example, when generating an image pair with a stereo camera as the viewpoint as training data 112, the camera settings of the stereo camera include the field of view and resolution of the right and left cameras respectively, the relative position and orientation between the right and left cameras, and the baseline length. In this case, the image pair generated by the training data candidate generation unit 103 must satisfy the real-world consistency based on the camera settings of the stereo camera. That is, the camera constraint score of the training data must be equal to or greater than the standard value. However, if the camera settings of the images used during training of the image generation model used in the training data candidate generation unit 103 differ from the camera settings 110 specified in the domain-specific knowledge specification unit 108, and pixels in one image do not match corresponding pixels located horizontally in the other image, then real-world consistency based on camera settings cannot be satisfied. In such cases, where the camera settings of the images used during training of the image generation model used in the training data candidate generation unit 103 differ from the camera settings 110 specified in the domain-specific knowledge specification unit 108, the image generation model is adjusted so that the camera constraint score of the generated training data 112 satisfies the standard value, thereby generating training data 112 that satisfies real-world consistency based on camera settings.

[0076] In step S209 of Figure 2, after the training data 112 is stored in memory, the camera constraint calculation unit 801 calculates the camera constraint score for each training data 112 based on the camera settings of each training data 112. For example, the camera constraint calculation unit 801 generates warping data (a third training data image) by performing a geometric transformation on one of the two training data images, the first training data image, using the camera settings. Warping data can be generated, for example, using epipolar geometry through a pinhole camera model or depth estimation. Alternatively, if the image generation model used in the training data candidate generation unit 103 is a free-viewpoint generation model, the warping data may be generated using point cloud data generated using Multi-View Stereo technology from the image group used during training of the free-viewpoint generation model. Next, the camera constraint calculation unit 801 can calculate the camera constraint score by finding the difference between the generated warping data (a third training data image) and the other second training data image. The camera constraint score can be obtained, for example, from the squared error of the difference between the warping data and the corresponding training data 112. However, the method for generating the warping data and the method for calculating the camera constraint score are not limited to this. If the camera constraint calculation unit 801 calculates the camera constraint score in this way and it falls below a reference value, it instructs the training data candidate generation adjustment unit 802 to adjust the training data candidate generation unit 103. If the camera constraint score exceeds the reference value, the camera constraint calculation unit 801 may terminate the process without updating the training data 112.

[0077] The training data candidate generation adjustment unit 802 adjusts the output of the image generation model used by the training data candidate generation unit 103 so that the camera constraint score of the training data 112 exceeds a reference value. One way to adjust the output of the image generation model is to update the parameters of the image generation model. The parameters to be updated are one or more parameters that can be adjusted during inference of the image generation model, and are also called hyperparameters. Parameter updates can be performed using known deep learning methods. The training data candidate generation unit 103 generates training data candidates using the image generation model whose parameters have been updated by the training data candidate generation adjustment unit 802, and the data selection unit 106 saves the generated training data candidates in memory as training data 112. The training data candidate generation adjustment unit 802 may repeatedly update the parameters of the image generation model until the camera constraint score of the training data 112 exceeds a reference value, and if the camera constraint score of the training data 112 exceeds a reference value, the data selection unit 106 may save the images generated by the image generation model with the updated parameters set in memory as training data 112. Alternatively, the baseline value of the camera constraint score may be changed sequentially, and the above process may be repeated.

[0078] Figure 11 shows the processing flow according to the second embodiment. Steps S202 to S209 are the same as in the first embodiment and therefore will not be described.

[0079] In step 901, the camera constraint calculation unit 801 calculates the camera constraint score for each piece of training data 112 based on the camera settings of that piece of training data 112, after the training data 112 has been saved.

[0080] In step 902, the camera constraint calculation unit 801 determines whether the calculated camera constraint score is below a threshold value. If the camera constraint score is above the threshold value, the training data 112 is saved to memory and the flow ends. If the camera constraint score is below the threshold value, the process proceeds to step S903.

[0081] In step 903, the parameters of the image generation model used in the training data candidate generation unit 103 are updated.

[0082] In step S904, the training data candidate generation unit 103 generates training data candidates again using the image generation model after the parameter update.

[0083] In step S905, the data selection unit 106 stores the generated training data candidates in memory as training data 112. Alternatively, after generating training data candidates in step S904, the camera constraint calculation unit 801 may calculate the camera constraint score and re-determine whether the camera constraint score of the training data candidates generated by the image generation model after parameter updates meets the criteria value.

[0084] As described above, according to the second embodiment of the present invention, the image generation model is updated so that the training data generated by the image generation model satisfies the actual camera settings, thereby improving the quality of the training data images.

[0085] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0086] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0087] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other storage media such as IC cards, SD cards, and DVDs.

[0088] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0089] 100, 200 data generation device 101 Scene designation section 102 Prompt text update section 103 Training Data Candidate Generation Unit 104 Image generation unit 105 Correct Value Generation Unit 106 Data Sorting Unit 107 Reality Consistency Score Determination Unit 108 Domain-Specific Knowledge Designation Section 109 Traffic Conditions Description 110 Camera Settings 111 Reality Constraint Calculation Unit 112 training data 301 Overview Information 304 Reality Constraints 401 Generated Image 402 Environmental recognition department 403 Correct answer 404 Object Position Score Calculation Unit 405 Object Class Score Calculation Unit 501 Image generation prompt generation unit 701 User Interface Screen

Claims

1. A data generation device that generates training data for an image recognition model that outputs recognition results of an input image, It consists of a computer having an arithmetic unit for executing programs and a storage device accessible by the arithmetic unit, The aforementioned computing device is The scene specification information, which specifies the environmental elements that make up the scene and the objects included in the scene, is obtained. Based on a traffic situation description in a predetermined format and camera settings, which are camera setting information, the system calculates real-world constraints that indicate the image conditions that the training data must follow. The aforementioned scene specification information is input to the image generation model to generate candidate training data images with the camera as the viewpoint. A reality consistency score is determined that indicates the consistency between the generated training data candidate images and the reality constraints. A data generation device that determines whether or not to update the learning data specification information that specifies the learning data according to the reality consistency score.

2. A data generation device according to claim 1, The aforementioned computing device is A data generation device that stores candidate images of training data whose reality consistency score is equal to or greater than a standard value as training data in the storage device.

3. A data generation device according to claim 1, The aforementioned computing device is When it is decided to update the aforementioned training data specification information, Based on the aforementioned reality consistency score, the specified training data information is updated. The updated training data specification information is input to the image generation model to generate new training data candidate images with the camera as the viewpoint. Determine the new reality consistency score that indicates the consistency between the new candidate training data images and the reality constraints. A data generation device that stores the new candidate images for training data, whose new reality consistency score is equal to or greater than a standard value, as training data in the storage device.

4. A data generation device according to claim 1, The aforementioned computing device is A score is calculated that represents the ratio of the positional error of the object in the training data candidate image relative to the aforementioned reality constraints. An object class score is calculated that represents the ratio of the class error of each object in the candidate training data images with respect to the aforementioned reality constraints. A data generating device that determines the reality consistency score based on the object position score and the object class score.

5. A data generation device according to claim 1, The aforementioned computing device is A data generation device that inputs a scene specification update prompt to a language model instructing the generation of an image with a good reality consistency score, and obtains updated training data specification information.

6. A data generation device according to claim 5, The aforementioned computing device is Based on the aforementioned reality constraints and the object position information of the candidate images of the training data, a prompt is generated and input to the language model to explain the reason for the deterioration of the object's position, thereby generating the reason for the deterioration of the object's position. The scene specification information and a prompt instructing the language model to change the scene specification information so that the generated degradation reason is improved are input to obtain the updated scene specification information. A data generation device that updates the first designation information based on the updated learning data designation information acquired above.

7. A data generation device according to claim 6, The aforementioned computing device is A data generating device that generates prompts for a language model to generate explanations for the reasons for deterioration of at least one of the following for each object, including roads and white lines: lateral displacement, scale error, and defects in the road structure.

8. A data generation device according to claim 1, The aforementioned computing device is Based on the training data stored in the storage device, a camera constraint score is calculated, which is the degree to which the training data satisfies the constraints regarding the camera settings. A data generation device that, when the camera constraint score is below a certain threshold, adjusts the parameters of the image generation model so that the camera constraint score is above a certain threshold.

9. A data generation device according to claim 8, The aforementioned training data includes a first image and a second image, which are a pair of images viewed from a stereo camera. The aforementioned computing device is Based on the aforementioned camera settings, a third image is generated by geometrically transforming the first image. A data generation device that calculates the camera constraint score by determining the difference between the second image and the third image.

10. A data generation method in which a data generation device generates training data for an image recognition model that outputs recognition results of an input image, The data generation device is comprised of a computer having an arithmetic unit for executing a program and a storage device accessible by the arithmetic unit, and is capable of deriving answers using a language model that executes tasks according to input prompts. The aforementioned data generation method is: The calculation unit obtains scene specification information that specifies the environmental elements constituting the scene and the objects included in the scene, The calculation device performs the steps of calculating real-world constraints that indicate the conditions on the image that the learning data follows, based on a traffic situation description in a predetermined format and camera settings which are camera setting information, The calculation device inputs the scene specification information into the image generation model and generates candidate images of training data with the camera as the viewpoint, The calculation device performs the steps of determining a reality consistency score that indicates the consistency between the generated training data candidate images and the reality constraints, A data generation method comprising the step of determining whether or not the computing device updates the learning data specification information that specifies the learning data according to the reality consistency score.

Images