Data generation device, data generation method, and data generation program

Abstract

The challenge is to modify the added information more easily and with less effort. [Solution] The present invention relates to a data generation device 100 comprising: a storage unit 8 that stores data to which assignment information is assigned; a correction information input unit 1 that receives correction information which is a correction instruction for the assignment information and indicates a plurality of assignment information after the assignment information has been corrected; a data extraction unit 2 that extracts data to be corrected from the data stored in the storage unit 8, which is data that includes the assignment information to be corrected; and a correction data creation unit 6 that corrects the assignment information of the data and creates corrected data by assigning one of the items of the plurality of assignment information after the correction based on the correction information and the corrected data extracted by the data extraction unit 2.

Description

Technical Field

[0001] The present invention relates to a data generation device, a data generation method, and a data generation program related to data generation.

Background Art

[0002] In recent years, there has been an increasing need for sensing technologies that detect, track, and predict the paths of target objects by analyzing image data acquired by sensors such as cameras. In particular, in the field of autonomous driving, in order to handle complex situations, it is expected that detailed information can be recognized by cameras installed in vehicles or the environment, thereby realizing safe and secure movement that avoids accidents.

[0003] When detecting an object based on image data captured by a camera, in recent years, techniques using machine learning and deep learning that can detect objects with high accuracy are often used. In order to build a detection model capable of detecting objects by these techniques, a dataset (learning dataset) for learning is required.

[0004] In order to create a learning dataset, which is an example of data generation, it is necessary to collect and select a plurality of image data and perform a truth assignment (annotation) operation of attaching attribution information (labels) indicating the types (classes) of a plurality of objects reflected in each image and the regions (rectangles) on the image. In collecting and selecting image data, it is designed over time so that the balance of the images included in the learning dataset varies in terms of the ratio of classes, the time zone of shooting, the weather, etc.

[0005] By using the learning dataset created in this way and learning using image data as input and labels as teachers, a detection model capable of detecting objects reflected in images can be constructed.

[0006] When detecting objects in an image using the detection model created using the above procedure, the classes of objects that can be detected are limited to those defined in advance during the annotation stage. For example, if the classes of objects to be detected are defined as four types: "car," "bus," "person," and "two-wheeled vehicle," and the detection model is trained using an annotated training dataset, then the classes of objects that can be detected will also be limited to these four types.

[0007] On the other hand, as we conduct experiments and development toward realizing advanced autonomous driving, there may be a need to identify the type of object at a finer granularity (subclass) than the class definitions included in the created training dataset. For example, it is possible to further identify whether a detected "person" is an "adult" or a "child," and if it is a "child," change the autonomous driving settings to the safer side. Also, if we further identify "cars" into "golf carts" and "other cars," and a "golf cart" is detected, and a high speed is estimated in the subsequent speed estimation, it is possible to use the knowledge that golf carts move slowly to correct the speed.

[0008] Thus, if subclass information is ultimately desired, one could, for example, add processing to identify subclasses when a detection model detects an object of a certain class. However, this increases computational complexity and time. Therefore, it is desirable to construct a training dataset that includes subclasses in its class definitions so that the detection model can output subclass labels, and then retrain the detection model. Furthermore, rather than creating a new dataset, it is preferable to modify the label definitions in an existing training dataset, designed with a balance in mind as mentioned above, to include subclasses. However, manual re-annotation is time-consuming, and improvement is desired.

[0009] For example, the abstract of Patent Document 1 states that "the objective is to provide a training dataset generation system, a training server, and a training dataset generation program that can easily create training datasets for neural networks for object detection and neural networks for object recognition." Furthermore, claim 1 of the same document describes a learning dataset generation system comprising: an image input unit that inputs a seed dataset which is training data including multiple sample images and correct classification labels assigned to these sample images; an image collection unit that collects multiple related images, which are captured images taken by a camera that are similar to at least one of the multiple sample images included in the seed dataset, using an object detection NN model capable of detecting the sample images included in the seed dataset; a cluster formation unit that forms clusters corresponding to the correct classification labels using the sample images included in the seed dataset input by the image input unit; a label assignment unit that determines which of the clusters each of the multiple related images collected by the image collection unit belongs to and assigns a correct label corresponding to the cluster of this determination result to each of the related images; a correction input unit for performing correction input of the correct label assigned by the label assignment unit for each of the multiple related images; and a correction processing unit that corrects the correct label assigned to each of the multiple related images in accordance with the correction input by the correction input unit. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2020-204800 [Overview of the project] [Problems that the invention aims to solve]

[0011] Here, it is envisioned that re-annotation can be simplified and data generation more efficient by classifying which subclass an object in an existing training dataset belongs to. In such a case, multiple sample images are prepared, each containing an object of a subclass and its corresponding label, and clusters are formed according to the class of the label in the sample image. This allows for the identification of which subclass an object in the training dataset belongs to and the updating of its label.

[0012] However, proper cluster design is required for identification. For example, it is necessary to select appropriate variations and ratios of sample images to enable identification of subclasses on the cluster. In addition, specialized know-how regarding cluster formation according to the type and number of subclasses is required. Furthermore, increasing or changing the types of subclasses requires re-forming the clusters, which is time-consuming. Moreover, if the object to be identified as a subclass is a single, unique object, such as a vehicle with a custom-colored roof, it may be difficult to form a cluster that can identify the subclass.

[0013] The present invention aims to modify the assigned information more easily and with fewer steps. [Means for solving the problem]

[0014] To solve the aforementioned problems, the present invention creates modified data by assigning one of several modified assignment information and modifying the assignment information. Preferably, the assigned assignment information indicates the items to be modified, such as objects and areas included in the data, and labels (hereinafter simply referred to as labels).

[0015] A more specific aspect of the present invention is a data generation device comprising: a storage unit that stores data to which assignment information is assigned; a correction information input unit that receives correction information which is a correction instruction for the assignment information and indicates a plurality of assignment information after the assignment information has been corrected; a data extraction unit that extracts data to be corrected from the data stored in the storage unit, which is data containing the assignment information to be corrected; and a correction data creation unit that corrects the assignment information of the data and creates corrected data by assigning one of the labels of the plurality of assignment information after the correction based on the correction information and the corrected data extracted by the data extraction unit. Other aspects of the present invention will be described in the embodiments described below. [Effects of the Invention]

[0016] According to the present invention, the information to be added can be modified more easily and with less effort during data generation. [Brief explanation of the drawing]

[0017] [Figure 1A] A system configuration diagram showing a first example of a data generation system according to one embodiment of the present invention. [Figure 1B] A system configuration diagram showing a second example of a data generation system according to one embodiment of the present invention. [Figure 2] Functional block diagram of a data generation device 100 according to one embodiment of the present invention. [Figure 3] A diagram illustrating an example of a correction instruction received by a correction information input unit 1 according to one embodiment of the present invention. [Figure 4A] A diagram showing a first example of a display by the display unit 3 according to one embodiment of the present invention. [Figure 4B] A figure showing a second example of a display by the display unit 3 according to one embodiment of the present invention. [Figure 5] A functional block diagram of the auxiliary information generation unit 4 and a diagram showing examples of various types of information received by the auxiliary information receiving unit 5 according to one embodiment of the present invention. [Figure 6]A diagram showing an example of a prompt template included in the auxiliary information generation template 203 according to an embodiment of the present invention. [Figure 7] A functional block diagram of the correction data creation unit 6 according to an embodiment of the present invention. [Figure 8] A diagram for explaining the extraction process by the region extraction unit 61 according to an embodiment of the present invention and an overview of the processing using the result. [Figure 9] A diagram showing the functional blocks of the instruction generation unit 62 according to an embodiment of the present invention. [Figure 10] A diagram showing an example of the correction instruction template 204 according to an embodiment of the present invention.

Mode for Carrying Out the Invention

[0018] Embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments, and various modifications and application examples are also included within the scope of the technical concept of the present invention. Hereinafter, a specific embodiment of the present invention will be described with reference to the drawings.

[0019] [Explanation of the System Configuration of the Data Generation Device] First, a typical system configuration of this embodiment will be briefly described using FIGS. 1A to 1B.

[0020] FIG. 1A is a system configuration diagram showing a first example of the data generation system according to this embodiment. The data generation system shown in FIG. 1A includes a data generation device 100 and a server (or cloud) 110, and FIG. 1A shows their mutual relationship (the first system configuration). In this configuration, for the data generation device 100, the learned model 201 is stored in another server 110.

[0021] Examples of pre-trained models include generative AI models. Generative AI models are models that have been trained to perform tasks instructed by prompts, which are textual instructions, and multimodal data such as images and PDF data, which are inputs. Examples of generative AI models include multimodal large language models (MLLMs), large language models (LLMs), and diffusion models.

[0022] Other trained models 201 besides generative AI models include, for example, detection models and classification models. In contrast, generative AI models can include auxiliary information for performing tasks in their prompts, and by including auxiliary information for classifying objects in the prompts, they can generate data such as annotation data even for unknown objects. In this respect, generative AI models are more powerful than detection models and classification models. In this embodiment, the case where the trained model 201 is a multimodal large-scale language model (MLLM) will be described.

[0023] This configuration assumes a case where the number of parameters in the trained model 201 is enormous, making it difficult to store them in the storage device 21 of the data generation device 100. It also assumes a case where the trained model 201 is provided by another company and used via a GUI or API. In this way, the trained model 201 is stored on the server 101, and other functions and data may be provided on the data generation device 100, the server 101, or another device. Therefore, the data generation system in Figure 1A has the functions of this embodiment distributed across multiple devices.

[0024] Figure 1B is a system configuration diagram showing a second example of the data generation system according to this embodiment. In the data generation system shown in Figure 1B, the trained model 201 is stored inside the storage device 21 of the data generation device 100. This configuration assumes the use of a lightweight trained model 201. Thus, in this embodiment, the trained model 201 and other functions and data are managed solely by the data generation device 100.

[0025] Here, the data generation device 100 in Figure 1A includes a processing device 20, a storage device 21, an input device 22, a display device 23, a communication device 24, etc. The data generation device 100 in Figure 1A is mounted on a PC or server, and data generation operators can easily perform data generation using this data generation system by inputting data using a mouse or keyboard attached to the PC.

[0026] Furthermore, the processing unit 20 constituting the data generation device 100 is a central processing unit (CPU) and executes various programs stored in a storage device 21, which can be implemented as RAM or an HDD. In particular, the processing unit 20 executes the data generation program 200 among the various programs, thereby executing the processing in this embodiment. The storage device 21 is a storage medium such as an HDD and stores various programs, such as the data generation program 200, and various data for the data generation device 100 to execute processing. The various data include a trained model 201, a dataset 202, an auxiliary information generation template 203, a correction instruction template 204, and a tool 205.

[0027] Furthermore, the input device 22 can be implemented as a keyboard, mouse, or the like. The input device 22 is a device for inputting instructions from the user to the data generation device 100, which is a computer. The input device 22 accepts instructions such as program startup. The display device 23 can be implemented as a display or the like and displays the execution status and results of processing by the data generation device 100. The input device 22 and the display device 23 may be configured as an integrated unit, such as a touch panel.

[0028] Furthermore, the communication device 24 is a device that exchanges various data and commands with other devices such as the server 110 via the network. The data generation device 100 shown in Figure 1B has substantially the same configuration. The processing unit 20 realizes the functions of each part described later by executing various programs (data generation program 200), but explanations of such well-known technologies will be omitted as appropriate below. As a modification of Figure 1B, the data generation device 100 may be configured to connect to a terminal device having an input device 22 and a display device 23. In this case, the input device 22 and the display device 23 can be omitted from the data generation device 100. Furthermore, the data generation device 100 may be configured to connect to multiple terminal devices and be usable from them.

[0029] [Description of data generation device 100] Figure 2 is a functional block diagram of the data generation device 100 according to this embodiment. As shown here, the data generation device 100 of this embodiment includes a correction information input unit 1, a data extraction unit 2, a display unit 3, an auxiliary information generation unit 4, an auxiliary information receiving unit 5, a correction data creation unit 6, a transmission unit 7, and a storage unit 8. Of these units, the display unit 3 corresponds to the display device 23 in Figures 1A and 1B, and some functions of the correction information input unit 1 and the auxiliary information receiving unit 5 correspond to the input device 22. The functions of the other components are executed by the processing unit 20 in Figures 1A and 1B.

[0030] The dataset 202 in the figure is an existing training dataset consisting of multiple image data with labels. The image data is just an example; it is sufficient if the labeling information is included. The auxiliary information generation template 203 and the correction instruction template 204 are templates for prompts to the trained model 201, and the tool 205 is a set of functions that can be used in the process of generating corrected labels from the trained model 201. Their details will be described later.

[0031] The following describes each part of the data generation device 100. First, the correction information input unit 1 receives correction information from the user, which is a correction instruction that is a rule for correcting the assigned information (labels) included in the training dataset. The data extraction unit 2 extracts the data to be corrected, which is image data and label data, from the dataset 202 that matches the correction information. The display unit 3 displays various data such as image data and labels. As a result, the user can check the image data and labels.

[0032] Furthermore, the auxiliary information generation unit 4 generates auxiliary information using the trained model 201. The auxiliary information receiving unit 5 receives the auxiliary information generated by the auxiliary information generation unit 4 or auxiliary information input by the user. The auxiliary information is descriptive text of an object and / or text information of a viewpoint that identifies the object, which is used by the trained model 201 to form input prompts for correcting data based on correction instructions.

[0033] Furthermore, the correction data creation unit 6 creates corrected data by modifying the data to be corrected in the trained model 201 based on the correction information and auxiliary information. The transmission unit 7 transmits the corrected data and other data to the storage unit 8, etc. Here, transmission by the transmission unit 7 refers to the exchange of data within the data generation device 100 and is a function separate from so-called communication. For this reason, the function of the transmission unit 7 may be provided in a configuration (unit) that outputs data, such as the correction data creation unit 6.

[0034] The following sections will explain in detail the correction information input unit 1, data extraction unit 2, display unit 3, auxiliary information generation unit 4, auxiliary information reception unit 5, and correction data creation unit 6.

[0035] [Correction Information Input Section 1] Figure 3 illustrates an example of a correction instruction that can be received by the correction information input unit 1 according to this embodiment. Figure 3 is also an example of a GUI displaying the correction instruction. Furthermore, in this embodiment, for the sake of clarity, the following tasks will be used as examples for explanation. Examples of tasks include identifying the "car" class included in the assignment information (labels) of the dataset 202 into "special vehicle" and "other car (hereinafter, car)", and identifying the "person" class into "adult" and "child". However, the system is not limited to these examples, and one class may be classified into three or more subclasses, or two or more classes may be identified into multiple classes, and the system is not limited to these examples.

[0036] In the example in Figure 3, the first modification instruction 301 and the second modification instruction 302 are shown as modification instructions. First, the first modification instruction is an example instructing that the "Car" included in the label definition of the existing dataset 202 be identified as "Car" and "Special Car" as the assignment information (label) to be modified. The second modification instruction 302 is an example in which "Person" is entered to be classified as "Adult" and "Children". The assignment information to be modified can be entered, for example, by entering the class name from the modification information input unit 1 which can be implemented using a keyboard, or a list of classes can be displayed in a pull-down menu on the display unit 3 and selected by the modification information input unit 1.

[0037] Furthermore, as shown in Figure 3, the correction information input unit 1 may accept input of the class name (e.g., car) in the corrected assigned information, and input indicating whether or not the object to be identified is a unique (specific) object. In this embodiment, the special vehicle is an example of a vehicle specially designed for the PR of autonomous driving, and the correction information input unit 1 accepts input indicating that this special vehicle is a specific object. In other words, a check indicating that it is a specific object is selected.

[0038] As described above, the correction information input unit 1 allows the user to input how they want to modify the assigned information (labels) included in the dataset 202. In other words, the correction information input unit 1 accepts the input of correction information, which is a modification instruction. Of course, if the same information can be input by a method other than the GUI shown in Figure 3, this may include the GUI, entries in a configuration file, or voice input, and is not limited to these examples.

[0039] [Data extraction unit 2 and display unit 3] The data extraction unit 2 extracts the numerous image data contained in dataset 202, as well as the labels assigned to each image data, specifically those containing the label targeted for modification according to the modification instructions indicated by the modification information. For example, if "Car" is selected as the label to be modified, only the images labeled "Car" and their corresponding labels are extracted from dataset 202 as data to be modified.

[0040] Figures 4A and 4B show examples of image data and associated information (labels) displayed by the display unit 3. In this embodiment, the associated information (labels) in the image data includes the object type (class), the center position (x, y) of the object in the image data, and the width and height of the object. These center position and width and height of the object are information about the region that constitutes the image data.

[0041] First, the image data shown in Figure 4A is labeled as "vehicle to be corrected," and is therefore extracted as data to be corrected. The vehicle shown in enlargement in Figure 4A is the special vehicle in this embodiment. On the other hand, the image in Figure 4B is labeled only as "truck" and not as "vehicle," and is therefore not extracted as data to be corrected.

[0042] One of the objectives of the data generation device 100 is to enable the construction of a new dataset with modified labeling information for the existing dataset 202. For example, if the folder containing the original dataset 202 has image data and corresponding labels stored in folders such as "images" and "labels," the data generation device 100 will create a folder named "new_labels." As a result, a list of modified labeling information (labels) corresponding to the image data in "images" will be stored. The modified data creation unit 6 will then copy the label files in "labels" that correspond to the image data and labels that were not extracted as data to be modified to the "new_labels" folder. The modified data creation unit 6 will then modify the label files in "labels" that correspond to the image data and labels that were extracted as data to be modified. The transmission unit 7 will then store these modified files in the "new_labels" folder. As a result, a dataset containing modified label files corresponding to all image data can be constructed.

[0043] Based on the above, the trained model 201 selects "Car" from the attached information to be modified, which is the modification content in the modification information. Then, the trained model 201 modifies the attached information based on the selected modification content, "Car".

[0044] In the example of correction instructions shown in Figure 3, the classes of "car" and "person" were selected for correction. When two or more tasks are input together in this manner, the image data containing the labels of these classes and the labels themselves may be processed together as data to be corrected in subsequent processing, including processing in the data extraction unit 2.

[0045] In other words, the data extraction unit 2 may extract these data and then proceed to the subsequent processing of the modified data creation unit 6. Alternatively, the data extraction unit 2 may subdivide the task and first extract the car class as the data to be modified. In this case, the processing in the modified data creation unit 6 and the transmission unit 7 is executed, and the modified dataset corresponding to the car is saved in the storage unit 8. Then, the data extraction unit 2 reads the saved dataset, extracts the modified data corresponding to the person class, and performs the subsequent processing. This may create a new dataset. In this embodiment, the following explanation will be based on the latter method.

[0046] In Figure 4, the display content of the display unit 3 shows an example where one data item (image data and label) is displayed. However, the display unit 3 is not limited to this example; multiple data items may be displayed side by side. In particular, the display unit 3 may display the corrected data created by the subsequent corrected data creation unit 6 side by side to allow the user to confirm whether the instructions have been executed correctly. The display unit 3 may also display the results of running the corrected data creation unit 6 on a portion of the data to be corrected. Furthermore, the display unit 3 may be configured to display results generated from different auxiliary information side by side. In this case, by confirming the results, the user can easily determine how the results will change after modifying the auxiliary information.

[0047] Through the processing of the data extraction unit 2 described above, data corresponding to the class specified in the correction information input unit 1 can be extracted as data to be corrected.

[0048] [Auxiliary information generation unit 4 and auxiliary information receiving unit 5] The auxiliary information generation unit 4 generates auxiliary information using the trained model 201. The auxiliary information receiving unit 5 receives input of the auxiliary information generated by the auxiliary information generation unit 4 or auxiliary information from the user.

[0049] Here, auxiliary information refers to descriptive text about an object and / or text information about the viewpoint that identifies the object, which is used in the correction data creation unit 6 to form prompts that serve as input for the trained model 201 to correct the data based on the correction instructions. The details of the auxiliary information generation unit 4 and the auxiliary information receiving unit 5 will be described below.

[0050] Figure 5 shows a functional block diagram of the auxiliary information generation unit 4 according to this embodiment, and an example of various types of information that can be received by the auxiliary information receiving unit 5. As shown in Figure 5, the auxiliary information receiving unit 5 receives auxiliary information generated by the auxiliary information generation unit 4 and auxiliary information input by the user. The latter can be implemented by the input device 22 in Figures 1A and 1B. The auxiliary information receiving unit 5 may also allow the user to modify the auxiliary information generated by the auxiliary information generation unit 4. In this embodiment, the auxiliary information received includes at least one of the following: a description of the object of the modified class, and a viewpoint for identifying each class. However, other information may be added as auxiliary information, and the system is not limited to these examples.

[0051] Furthermore, as shown in Figure 5, the auxiliary information generation unit 4 in this embodiment determines whether the region to be corrected in the data to be corrected satisfies the said viewpoint. For this purpose, the auxiliary information generation unit 4 has a reference data reception unit 41, a template selection unit 42, a template entry unit 43, an auxiliary information estimation unit 44, and an auxiliary information integration unit 45, and utilizes the stored auxiliary information generation template 203. First, the reference data reception unit 41 receives data such as image data or PDF data from the user as reference data to generate auxiliary information. The template selection unit 42 selects a template for a prompt to be input into the trained model 201 from the auxiliary information generation template 203 based on the correction information from the correction information input unit 1.

[0052] Furthermore, the template entry unit 43 fills in the selected template using the modification information and constructs a prompt. The auxiliary information estimation unit 44 estimates auxiliary information using the reference data and prompt as input. The auxiliary information integration unit 45 integrates the generated auxiliary information.

[0053] Next, Figure 6 shows an example of a prompt template included in the auxiliary information generation template 203 according to this embodiment. The prompt template may be selected by the user from among several templates. In addition to existing templates, new templates may be created. Furthermore, the template may be selected by the template selection unit 42 based on predetermined rules.

[0054] The template entry unit 43 constructs prompts by specifically describing the contents of variables based on the modification information for the template shown in Figure 6. In the example in Figure 6, the template entry unit 43 inputs "Special Vehicle" for (c) and "Car" for (b) based on the modification information. As a result, prompts for generating object descriptions and identification points can be constructed. Alternatively, the display unit 3 may display the template shown in Figure 5 to the user, and the template entry unit 43 may modify it according to the user's modification instructions.

[0055] Let's return to Figure 5 for explanation. The auxiliary information estimation unit 44 can generate auxiliary information by inputting the created template and data showing special vehicles as reference data into the trained model. Note that the template used in this explanation was based on the premise that image data would be used as reference data. However, for example, PDF data such as design drawings of special vehicles may be used as reference data, and a template using PDF data of the design drawings may also be used, or it may be automatically selected based on the type of reference data. In addition, document data such as specifications or catalogs may be used as reference data or templates.

[0056] Alternatively, after the auxiliary information estimation unit 44 creates the auxiliary information, the process may return to the reference data reception unit 41 and perform the processing in the auxiliary information estimation unit 44 again. For example, auxiliary information may be generated for each of several sample image data images that previously showed a special vehicle, or auxiliary information may be generated from both the image data and the PDF data.

[0057] The auxiliary information created above is integrated by the auxiliary information integration unit 45. The integration method may involve sequentially linking the auxiliary information, or the trained model 201 may be prompted to summarize the auxiliary information and then have the model create the summary.

[0058] The auxiliary information generation unit 4 described above can generate auxiliary information using the trained model 201. When this auxiliary information is generated based on an image of an actual special vehicle as shown in Figure 4 and a prompt, the following description is output: "It has a compact design and a unique appearance. It has a small, angular cabin, a large glass window at the front for high visibility, and no doors on either side of the vehicle. It is a functional design that prioritizes work efficiency. The body is flesh-colored, and the roof and frame are distinctively dark navy blue. It is left-hand drive, and... (omitted)" In addition, from the perspective of identification, "dark navy blue roof and frame," "left-hand drive," "large glass window at the front," "angular cabin," "logo that says xx," etc. can be output.

[0059] Furthermore, it is desirable that the trained model 201 used to generate auxiliary information and the trained model used in the modified data creation unit 6 be the same model and have the same parameters. This is because if the model or parameters change, different explanations and judgments will be made even for the same object. In other words, by using the same trained model 201, auxiliary information including explanations and perspectives suitable for subclass identification in the modified data creation unit 6 can be obtained.

[0060] Furthermore, it is sometimes possible to generate auxiliary information even without reference data. For example, by creating a prompt to generate criteria for distinguishing between "adults" and "children" in image data and inputting this into the trained model 201, auxiliary information can be generated even without reference data. In this case, for example, criteria such as "height," "body proportions," and "clothing" may be output by the trained model 201.

[0061] Using the above auxiliary information generation template 203, auxiliary information can be easily generated by the template selection unit 42 and the template entry unit 43. Furthermore, the auxiliary information generation unit 4 can generate auxiliary information including descriptions and perspectives suitable for identification of subclasses in the modified data creation unit 6.

[0062] [Correction Data Creation Section 6] The modified data creation unit 6 modifies the data to be modified in the trained model 201 based on the modification information and auxiliary information, and creates modified data. Figure 7 shows a functional block diagram of the modified data creation unit 6 according to this embodiment. The modified data creation unit 6 includes a region extraction unit 61, an instruction generation unit 62, an input information generation unit 63, an agent construction unit 64, an inference unit 65, and an annotation information generation unit 66.

[0063] First, the region extraction unit 61 determines whether each of the multiple regions in the data to be corrected that have annotation information attached to them indicates or contains the annotation information to be corrected, and identifies the region to be corrected (the part of the image and the corresponding label). Then, the instruction generation unit 62 generates a prompt, which is text instructing the trained model 201 to correct the region to be corrected, based on the correction information and auxiliary information.

[0064] Furthermore, the input information generation unit 63 generates individual areas to be modified and prompts as input information. The agent construction unit 64 uses the tool 205 to construct an agent for verifying the identification aspect of the auxiliary information in the inference unit 55. The inference unit 65 outputs text related to the modification information of the added information based on the generated input information and the agent. The added information generation unit 66 generates added information based on the output text of the inference unit 65.

[0065] Figure 8 illustrates the extraction process (extraction processing) in the region extraction unit 61 and the processing using the results. As shown in Figure 8, the region extraction unit 61 receives an image and a label assigned to the image as input from the data to be corrected. The region extraction unit 61 then determines whether each of the regions of multiple objects included in the label is the label to be corrected. If it is the label to be corrected, it extracts the region image data of that region and associates that label with the label to be corrected. The region image data is an example of region information related to a region.

[0066] In the example shown in Figure 8, the region extraction unit 61 extracts two region image data images corresponding to the "Car" region because the class to be modified is "Car". The region extraction unit 61 also identifies two labels corresponding to "Car" as targets for modification. Subsequently, the region extraction unit 61 associates each of the two region images with a generation instruction prompt generated by the instruction generation unit 62. This allows the input information generation unit 63 to create two input data (input information) for input to the inference unit 65. The input information generation unit 63 then outputs each of the two input data to the inference unit 65. In response, the inference unit 65 uses the input data to create modification information for the added information and outputs the text related to this to the added information generation unit 66. The added information generation unit 66 then generates the added information based on the text. The generation of added information will be explained later.

[0067] Figure 9 is a functional block diagram of the instruction generation unit 62 according to this embodiment. As shown in Figure 9, the instruction generation unit 62 has a template selection unit 621 and a template entry unit 622, and utilizes the stored modification instruction template 204. Here, the template selection unit 621 and the template entry unit 622 of the instruction generation unit 62 are configured to perform processing in the same way as the template selection unit 42 and the template entry unit 43 of the auxiliary information generation unit 4. For this reason, a detailed explanation of these will be omitted, and a brief explanation will be given here using the modification instruction template 204 example in Figure 10.

[0068] Figure 10 shows an example of a modification instruction template 204 according to this embodiment. In the modification instruction template 204, the user may select from pre-prepared templates, or the user may edit them according to their operations or register them as new templates to improve convenience. For this purpose, as shown in Figure 10, in the example in Figure 10, "car" is entered in (a), "car" in (b), and "special vehicle" in (c) via the template entry unit 622. In addition, the description of the object received by the auxiliary information receiving unit 5 is entered in (b-description) and (c-description), and the identification points received by the auxiliary information receiving unit 5 are entered in (b-point) and (c-point).

[0069] In this embodiment, "cars" and "special vehicles" are distinguished. While "cars" are diverse in nature, "special vehicles" are a single, unique type of object. Therefore, when generating auxiliary information, the auxiliary information generation unit 4 focuses on the characteristics of special vehicles and generates auxiliary information that identifies them as cars. For this reason, (b-description)(b-point) is left blank, and the auxiliary information described in this embodiment is entered in (c-description)(c-point).

[0070] The region extraction unit 61, instruction generation unit 62, and input information generation unit 63 described above enable the construction of prompts for the trained model 201 to modify the information attached to the data to be corrected.

[0071] Now, let's return to Figure 7 and explain the agent construction unit 64. In this embodiment, an agent is a program designed to autonomously execute a specific task using a Large-Scale Language Model (LLM). The LLM agent plans to execute a task in response to an input prompt, performs an action, and then performs the next action based on the result. At this time, by registering specific functions called tools with the LLM agent, the agent can use these functions when performing an action to execute the task.

[0072] Furthermore, the task of this embodiment is to output the result of identifying which subclass the region image data to be corrected belongs to, as shown in the correction instruction template 204 in Figure 10. For this purpose, for example, an OCR function that can read characters from an image is provided to the LLM agent as a tool. In this case, for example, if the identification of auxiliary information includes character information of a logo, the LLM agent will use the OCR tool to check the logo as an action to identify which subclass the region image data to be corrected belongs to. This allows for a decision to be made considering the results of the OCR tool.

[0073] In this embodiment, OCR is used as an example of the type of tool, but the tool is not limited to this example. Various tools capable of performing image processing, language processing, etc., may be registered in tool 205, allowing the user to select multiple tools according to the task. Furthermore, the trained model 201 may be made to plan which tool to use based on the perspective of identifying auxiliary information. In this embodiment, it is not necessary to register tools, and the example is not limited to these.

[0074] Furthermore, the inference unit 65 outputs text related to the correction information of the assigned information, which is output by inputting prompt and region image data into the trained model 201 based on the agent that was actually constructed. The output format of this text can also be instructed to the trained model 201, as shown in Figure 10. In this embodiment, in JSON format, in addition to one of "car," "special vehicle," or "unknown" as the corrected label information, a value corresponding to the reason for the judgment or the confidence level of the judgment can be output.

[0075] Furthermore, the annotation information generation unit 66 generates annotation information based on the output text of the inference unit 65. In this embodiment, the modified annotation information (label) can be obtained from the label information in the JSON output. At this time, by assigning an "unknown" label to the annotation information generation unit 66, only unknown objects can be extracted and displayed on the display unit 3 to the user, who is the data generation operator, allowing them to perform annotation work.

[0076] Furthermore, the correction data creation unit 6 determines the reliability of the judgment. If the reliability is lower than a predetermined value, the correction data creation unit 6 displays this to the data generation operator using the display unit 3. As a result, the data generation operator can efficiently check correction data that is likely to contain errors by reviewing the annotations.

[0077] Furthermore, the correction data creation unit 6 outputs the reason for the judgment to the display unit 3. This means that, for example, if the user confirms that there is an error in the reason for the judgment, that is, if the system receives notification of an error from the input device 22, it may be possible to assist in correcting the perspective of identifying the auxiliary information based on that reason. In this case, the auxiliary information generation unit 4 corrects this perspective of identification. In addition, if there are many errors in the corrections, the system may have an additional function to automatically correct the perspective of identifying the auxiliary information by having a trained model analyze the reasons for the judgment of the incorrect data based on the user's check information regarding the incorrect data.

[0078] The above-described correction data creation unit (6) can generate corrected data for the correction information and the data to be corrected. Furthermore, an analysis unit may be added to collect the number of object classes included in the new dataset 202 based on the created corrected data, and to analyze the number and proportion of objects in each class. For this purpose, the analysis unit may analyze the information attached to the corrected data and detect any bias in the attached information. In this embodiment, for example, there is a possibility that the number of training data containing special vehicles may be extremely small. In such cases, the user will be notified by displaying this information on the display screen of the display unit 3, allowing the user to consider adding data containing special vehicles to the dataset 202.

[0079] With the data generation device 100 described above, a dataset with corrected assigned information can be constructed by inputting correction instructions for assigned information. Furthermore, by providing auxiliary information generation templates 203 and correction instruction templates 204, a new dataset can be generated simply by the user inputting correction information, thus streamlining data generation without requiring specialized knowledge or know-how. In addition, the auxiliary information generation unit 4 generates auxiliary information such as object descriptions and identification points using the trained model 201, and the correction data creation unit 6 then corrects the assigned information based on the trained model 201 using the auxiliary information, thereby improving the accuracy of the correction. Moreover, the auxiliary information generation unit 4 can generate auxiliary information and identification points even for a single object, and the correction data creation unit 6 can correct the assigned information based on that auxiliary information. Similarly, since it generates points for identifying objects and generates assigned information based on those points, it is less susceptible to the influence of variations in the image background. As a result, the majority of the assigned information in the data to be corrected can be automatically corrected, reducing the man-hours required for data generation.

[0080] Furthermore, the present invention is not limited to the embodiments described above. For example, it can be applied to an autonomous driving system or an autonomous driving support system using the trained model 201, or to a picking system. When applied to a picking system, it can be used to subdivide containers. It can also be applied to reclassifying products during inventory in the retail industry. [Explanation of symbols]

[0081] 1. Correction Information Input Section 2. Data Extraction Unit 3 Display section 4 Auxiliary information generation section 5. Support Information Reception Department 6. Correction Data Creation Department 7. Transmitter 20 Processing Units 21 Storage device 22 Input devices 23 Display device 24 Communication equipment 100 Data Generation Devices 110 Servers 201 Pre-trained models 202 datasets 203 Supplementary Information Generation Template 204 Revision Instruction Template 205 S

Claims

1. A storage unit that stores data to which the assigned information has been assigned, A correction information input unit that receives correction information which is a correction instruction for the aforementioned assigned information and is a correction instruction indicating multiple assigned pieces of information after the assigned information has been corrected, A data extraction unit extracts data to be modified, which is data containing the annotation information to be modified, from the data stored in the storage unit. A data generation device having a modified data creation unit that modifies the attribute information of the data and creates modified data by assigning one of the items of a plurality of attribute information after modification based on the modification information and the data to be modified extracted by the data extraction unit.

2. In the data generation apparatus according to claim 1, The aforementioned correction data creation unit is a data generation device that determines whether the region in the data to be corrected to which the assigned information is assigned indicates the assigned information to be corrected.

3. In the data generation apparatus according to claim 2, The aforementioned correction data creation unit, Identify the region to be corrected within the aforementioned data to be corrected, The domain information of the aforementioned region and the correction information are input into the trained model. A data generation device characterized in that corrected data, in which the assigned information to be corrected in the data to be corrected is corrected, is output from the trained model.

4. In the data generation apparatus according to claim 3, The trained model is a data generation device that selects a correction from the correction information for the region it has determined to be subject to correction, and corrects the information assigned to that region based on the selected correction.

5. In the data generation apparatus according to claim 3, The aforementioned data is image data. The data generation device is characterized in that the information provided is information about the type and region of an object included in the image data.

6. In the data generation device of claim 5, The data generation device is characterized in that the trained model accepts input in the form of image data and text, and outputs text information indicating the corrected data.

7. In the data generation apparatus according to any one of claims 1, It further has a receiving unit that receives supplementary information for corrections to assigned information, The data generation device is characterized in that the modified data creation unit creates modified data by modifying the assigned information based on the modified information and the auxiliary information.

8. In the data generation apparatus according to claim 7, A data generation device further comprising an auxiliary information generation unit that generates auxiliary information used to form input prompts for correcting the aforementioned data.

9. In the data generation apparatus according to claim 8, The auxiliary information includes a text for identifying the assigned information and a data generation device for identifying the assigned information.

10. In the data generation apparatus according to claim 9, The auxiliary information generation unit is a data generation device that determines whether the region to be modified in the data to be modified satisfies the said viewpoint.

11. In the data generation apparatus according to claim 3, A data generation device further comprising an analysis unit that analyzes the annotation information of the corrected data transmitted by the transmission unit and detects any bias in the annotation information.

12. In a data generation method performed by a data generation device, The memory unit stores data to which additional information has been added. The correction information input unit receives correction information which is a correction instruction for the aforementioned assigned information and is a correction instruction indicating multiple assigned pieces of information after the assigned information has been corrected. The data extraction unit extracts data to be modified from the data stored in the storage unit, which includes the annotation information to be modified. A data generation method in which a modified data creation unit modifies the attribute information of the data by assigning one of the items of the multiple attribute information after modification to the modified data based on the modification information and the data to be modified extracted by the data extraction unit, thereby creating modified data.

13. A data generation device that is a computer, A storage unit that stores data to which the assigned information has been assigned, A correction information input unit that receives correction information which is a correction instruction for the aforementioned assigned information and is a correction instruction indicating multiple assigned pieces of information after the assigned information has been corrected, A data extraction unit extracts data to be modified, which is data containing the annotation information to be modified, from the data stored in the storage unit. A data generation program that functions as a modified data creation unit, which modifies the attribute information of the data and creates modified data by assigning one of the items of the multiple attribute information after modification to the modified data based on the modification information and the data to be modified extracted by the data extraction unit.

Images