Data generation device

Abstract

It can sometimes be difficult to generate appropriate data. [Solution] The data generation device includes an extraction unit that extracts relationship information from image data corresponding to at least one of the relationships between each object detected from the image data, or between an object and an imaging device that acquired the image data, and a generation unit that uses the relationship information extracted by the extraction unit to generate data used when fine-tuning a trained model.

Description

Technical Field

[0001] The present invention relates to a data generation device, a data generation method, and a program.

Background Art

[0002] In VLM (Vision Language Model) and other large language models, fine-tuning may be performed as a fine adjustment for adapting a pre-trained model to a specific task or area.

[0003] As a document describing related techniques, for example, there is Patent Document 1. Patent Document 1 discloses using a VLM as a base model, using PEFT (Parameter Efficient Fine Tuning) as a fine-tuning method for the base model, and the like.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When performing fine-tuning using image data, there may be a case where adjustment considering the arrangement of the subject in the image data is desired. In such a case, if fine-tuning is simply performed using image data as learning data or the like, appropriate adjustment may not always be possible. Thus, there has been a problem that it may be difficult to generate appropriate data for performing fine-tuning appropriately.

[0006] Therefore, one of the objectives of this disclosure is to provide a data generation device, a data generation method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0007] To achieve this objective, the data generation device in this disclosure is An extraction unit extracts relationship information from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. A generation unit generates data used for fine-tuning a trained model using the relationship information extracted by the extraction unit, has This is the structure it takes.

[0008] Furthermore, the data generation method in this disclosure is: Information processing device, Relationship information is extracted from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. Using the extracted relationship information, data is generated to be used when fine-tuning the trained model. This is the structure it takes.

[0009] Furthermore, the program in this disclosure is In an information processing device, Relationship information is extracted from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. Using the extracted relationship information, data is generated to be used when fine-tuning the trained model. This is a program for performing the processing. [Effects of the Invention]

[0010] According to each configuration as described above, it becomes possible to generate appropriate data for performing fine tuning.

Brief Description of the Drawings

[0011] [Figure 1] It is a diagram showing an overview of the data generation system in the present disclosure. [Figure 2] It is a diagram showing a configuration example of the data generation system. [Figure 3] It is a block diagram showing a configuration example of the data generation device. [Figure 4] It is a diagram showing an example of processing when acquiring explanatory information. [Figure 5] It is a diagram showing an example of processing when extracting relational information. [Figure 6] It is a flowchart showing an operation example of the data generation device. [Figure 7] It is a block diagram showing another configuration example of the data generation device. [Figure 8] It is a diagram showing a hardware configuration example of the second data generation device in the present disclosure. [Figure 9] It is a block diagram showing a configuration example of the data generation device.

Mode for Carrying Out the Invention

[0012] [First Embodiment] A configuration example of the data generation system 100 in the present disclosure will be described with reference to FIGS. 1 to 7. FIG. 1 is a diagram showing an overview of the data generation system 100. FIG. 2 is a diagram showing a configuration example of the data generation system 100. FIG. 3 is a block diagram showing a configuration example of the data generation device 200. FIG. 4 is a diagram showing an example of processing when acquiring explanatory information. FIG. 5 is a diagram showing an example of processing when extracting relational information. FIG. 6 is a flowchart showing an operation example of the data generation device 200. FIG. 7 is a block diagram showing another configuration example of the data generation device 200. Note that in the present disclosure, the drawings may be associated with one or more embodiments.

[0013] This disclosure describes a data generation system 100 that generates training data and other data used when fine-tuning a trained model such as a Vision Language Model (VLM), as shown in Figure 1. As will be described later, the data generation system 100 acquires image data to be used when fine-tuning. Then, the training data generation system 100 extracts at least relationship information from the acquired image data by performing a predetermined extraction process on the image data. For example, the training data generation system 100 extracts relationship information from the image data by using technologies such as Visual SLAM (Simultaneous Localization and Mapping), which is a technology that performs self-localization estimation and environmental mapping according to the image data, and Bird's-eye view transformation, which converts an image viewed from an oblique angle into a bird's-eye view by performing object detection and homography transformation. Subsequently, the data generation system 100 generates fine-tuning data to be used when fine-tuning a VLM by inputting the extracted relationship information and other data into a trained model such as an LLM (Large Language Model). For example, the data generation system 100 generates training data to be used when fine-tuning a VLM as data for fine-tuning. The data generation system 100 may generate evaluation data, etc., along with or instead of the training data. For example, the data generation system 100 can generate multiple sets of fine-tuning data having a similar format by performing the above generation process using the same or similar prompts on relational information extracted from various image data.

[0014] In the present disclosure, the relational information refers to information corresponding to at least one of the relationships between each object detectable from the image data or between the object and the imaging device that acquired the image data. In other words, the relational information refers to information corresponding to the relationships between each subject in the image data, the imaging device that acquired the image data, and the like. For example, as information indicating the relationship between each object detectable from the image data, there is information corresponding to the proximity between each object. As an example, the information corresponding to the proximity between each object may indicate the distance between each object, or may indicate whether the distance between each object is closer than a predetermined value. Further, as information corresponding to the relationship between the object and the imaging device that acquired the image data, there is information indicating the speed at which the object moves or the direction in which the object moves. Note that the relational information may include information corresponding to the arrangement of objects other than those exemplified above, information corresponding to the relationships between objects, between an object and the imaging device, and the like. Further, the relational information may be associated with the image data as information separate from the image data, or the relational information may be depicted in the image data itself, such as by embedding the relational information for each object detected in the image data.

[0015] FIG. 2 shows a configuration example of the data generation system 100. Referring to FIG. 2, the data generation system 100 includes an imaging device 110 and a data generation device 200. As shown in FIG. 2, the imaging device 110 and the data generation device 200 can be connected to each other using wired or wireless communication so as to be communicable with each other.

[0016] Note that the configuration of the data generation system 100 may be other than that exemplified in FIG. 2. For example, the data generation system 100 may include a plurality of imaging devices 110. Further, the data generation system 100 may not include the imaging device 110. In this case, the data generation device 200 included in the data generation system 100 may be configured to acquire image data from an arbitrary external device.

[0017] Furthermore, this disclosure describes the case where the imaging device 110 is a drive recorder or a surveillance camera. This disclosure also provides an example of using image data acquired by the imaging device 110 to fine-tune a VLM (Visual Memory Module) used to detect predetermined situations such as accidents or near misses from arbitrary image data. In other words, this disclosure describes the case where relationship information is extracted from image data acquired by an imaging device 110, such as a drive recorder or a surveillance camera, to generate data such as training data. However, the data generation system 100 may be used in any situation other than those exemplified above. In other words, the data generation system 100 can be used in any situation where image data is used to fine-tune a VLM, not limited to the situations exemplified above. Also, the imaging device 110 is not limited to a drive recorder or the like; it may be any camera or the like.

[0018] The data generation device 200 is an information processing device that acquires image data from the imaging device 110 or other arbitrary external devices and generates data such as training data corresponding to the acquired image data. Figure 3 shows an example of the configuration of the data generation device 200. Referring to Figure 3, the data generation device 200 has as its main components an operation input unit 210, a screen display unit 220, a communication interface unit 230, a storage unit 240, and an arithmetic processing unit 250.

[0019] Figure 3 illustrates a case where the functions of the data generation device 200 are realized using a single information processing device. However, at least some of the functions of the data generation device 200 may be realized using multiple information processing devices, for example, by being implemented on the cloud. Furthermore, the data generation device 200 does not have to include some of the configurations exemplified above, such as not having an operation input unit 210 or a screen display unit 220, and may have configurations other than those exemplified above.

[0020] The operation input unit 210 consists of an operation input device such as a keyboard or mouse. The operation input unit 210 detects the operation of the operator operating the data generation device 200 and outputs it to the calculation processing unit 250.

[0021] The screen display unit 220 consists of a screen display device such as a liquid crystal display or an organic EL (electro-luminescence) display. The screen display unit 220 can display various information stored in the storage unit 240 on the screen in response to instructions from the arithmetic processing unit 250.

[0022] The communication interface unit 230 consists of data communication circuits and the like. The communication interface unit 230 performs data communication with the imaging device 110 and other arbitrary external devices connected via a communication line.

[0023] The storage unit 240 is a storage device such as a hard disk or memory. The storage unit 240 stores processing information and programs 243 necessary for various processes in the arithmetic processing unit 250. The programs 243 are read into the arithmetic processing unit 250 and executed to realize various processing functions. The programs 243 are pre-read from external devices or recording media via data input / output functions such as the communication interface unit 230 and stored in the storage unit 240. The main information stored in the storage unit 240 includes, for example, image data information 241 and data information 242.

[0024] Image data information 241 includes image data from which relationship information and other data are to be extracted. In other words, image data information 241 includes image data that can be used when performing fine tuning. Image data information 241 may also include time-series image data. In image data information 241, image data and information indicating the time when the image data was acquired may be associated. Image data information 241 can be updated in response to the image data acquisition unit 251 acquiring image data from the imaging device 110 or other arbitrary external devices.

[0025] Data information 242 contains information about various types of data, such as training data, that can be used when fine-tuning the VLM. Data information 242 can be updated when the data generation unit 255 generates data for fine-tuning, such as training data. In addition, data information 242 may associate the generated data with image data used to generate the data, or with relationship information extracted from said image data. As mentioned above, relationship information may be depicted in the image data itself.

[0026] The arithmetic processing unit 250 has an arithmetic device such as a CPU (Central Processing Unit) and its peripheral circuits. The arithmetic processing unit 250 reads and executes a program 243 from the storage unit 240, thereby realizing various processing functions by having the hardware and the program 243 work together. Examples of the main processing functions realized by the arithmetic processing unit 250 include an image data acquisition unit 251, an explanatory information acquisition unit 252, an input reception unit 253, an information extraction unit 254, a data generation unit 255, and an output unit 256.

[0027] The arithmetic processing unit 250 may have a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof, instead of the CPU described above.

[0028] The image data acquisition unit 251 acquires image data from the imaging device 110 or any other external device. For example, the image data acquisition unit 251 can acquire time-series image data from the imaging device 110, such as a drive recorder. The image data acquisition unit 251 also stores the acquired image data as image data information 241 in the storage unit 240.

[0029] The explanatory information acquisition unit 252 acquires explanatory information that explains the content of the image data acquired by the image data acquisition unit 251. For example, as shown in Figure 1, the explanatory information acquisition unit 252 acquires information corresponding to the image data, such as whether there is a person or whether a person is approaching, as explanatory information that indicates the content of the image data. The explanatory information acquisition unit 252 may acquire explanatory information for each image data, or it may acquire the same explanatory information for multiple image data.

[0030] In this disclosure, the explanatory information acquisition unit 252 acquires explanatory information in response to inputs made to the operation input unit 210 by an operator or other person operating the operation input unit 210, or to information acquired from an external device via the communication interface unit 230. Furthermore, as shown in Figure 4, the explanatory information acquisition unit 252 may acquire explanatory information from image data in response to inputting image data into an arbitrary VLM, performing object detection on the image data, or other similar actions. For example, the explanatory information acquisition unit 252 may acquire explanatory information from image data in response to inputting information indicating an object detected using an arbitrary model such as YOLO (You Only Look Once) into a trained model such as an arbitrary LLM.

[0031] The input receiving unit 253 accepts input information, which is predetermined information such as information corresponding to the correct answer. For example, as shown in Figure 1, the input receiving unit 253 accepts input information indicating that the image data has a near miss, and that the approach time, which is the time until approach, is approximately 0 to 4 seconds. The input receiving unit 253 may accept input information for each image data, or it may accept the same input for multiple image data.

[0032] In this disclosure, the input receiving unit 253 receives input information in response to an operator or other person operating the operation input unit 210 making an input to the operation input unit 210, or to information being acquired from an external device via the communication interface unit 230. The input receiving unit 253 may also receive input information using methods other than those exemplified above.

[0033] As shown in Figure 5, the information extraction unit 254 extracts relationship information from image data by utilizing technologies such as visual SLAM and bird's-eye view transformation. Furthermore, the information extraction unit 254 can extract object information, such as the names and locations of objects within the image data, by performing region segmentation processing such as object detection using an arbitrary model or semantic segmentation. For example, as shown in Figure 1, the information extraction unit 254 extracts information indicating the location of people within the image data as extracted information, including object information and relationship information. As mentioned above, the information extraction unit 254 may extract relationship information as separate information from the image data, or it may depict the extracted relationship information within the image data itself.

[0034] Specifically, for example, the information extraction unit 254 estimates its own position at predetermined intervals by confirming the positions of feature points and brightness values ​​for each image data in the time series. Then, the information extraction unit 254 calculates the speed at which the imaging device 110 moves using the self-position estimated at predetermined intervals. The information extraction unit 254 also generates a bird's-eye view diagram generated by transforming the image data to a bird's-eye view. Then, the information extraction unit 254 extracts various relationship information using the generated bird's-eye view diagram, the results of object detection, the calculated speed, etc. For example, the information extraction unit 254 uses the generated bird's-eye view diagram to extract information corresponding to the proximity between objects, such as the distance between each object in the image data and whether the distance between each object is closer than a predetermined value. The information extraction unit 254 also uses the time-series bird's-eye view diagram to extract information indicating the speed at which objects move and the direction in which objects move in the image data.

[0035] For example, as described above, the information extraction unit 254 extracts information from image data that includes at least relationship information by utilizing technologies such as visual SLAM and bird's-eye view transformation. The information extraction unit 254 may extract information from each image data in the time-series image data, or it may extract information from image data selected according to arbitrary criteria.

[0036] The data generation unit 255 generates fine-tuning data, such as training data, in response to inputting relationship information extracted by the information extraction unit 254 into a trained model such as an LLM. For example, the data generation unit 255 generates training data such as predetermined question sentences, answer sentences to questions, and sentences that explain the basis and content of the answers in more detail, corresponding to the image data, in response to the above input. In this case, the data generation unit 255 can generate training data that takes into account the relationships between each object indicated by the relationship information by receiving input such as relationship information. The data generation unit 255 also stores the generated training data as data information 242 in the storage unit 240. The data generation unit 255 may also store the generated training data as data information 242 in the storage unit 240 in association with image data, etc. As described above, the data generation unit 255 may generate evaluation data, etc., in response to the above input, either together with the training data or instead.

[0037] Specifically, as shown in Figure 1, for example, the data generation unit 255 inputs explanatory information acquired by the explanatory information acquisition unit 252, input information received by the input reception unit 253, and extracted information such as relationship information and object information extracted by the information extraction unit 254 to the LLM. The data generation unit 255 may also input image data from which relationship information has been extracted, along with the information exemplified above, to the LLM. In this case, the data generation unit 255 can input a predetermined prompt that is stored in advance to the LLM. Here, the prompt may include text that specifies the number of outputs, the number of types, the output format, etc., as shown in Figure 1. When generating data for fine tuning, such as training data, the data generation unit 255 can input similar prompts that specify the format as described above, thereby generating multiple data with the same format from various image data. As an example, in the case shown in Figure 1, the data generation unit 255 generates multiple types of training data, such as training data 1 and training data 2, in response to a certain input. The number and format of training data and other data generated by the data generation unit 255 may differ from those exemplified in Figure 1.

[0038] The LLM may be located within the data generation device 200 or it may be located externally. In other words, the data generation unit 255 may be configured to generate fine-tuning data such as training data using an LLM located outside the device.

[0039] The output unit 256 outputs fine-tuning data, such as training data, and image data contained in the data information 242 to an external device. For example, the output unit 256 can transmit fine-tuning data, such as training data, and image data contained in the data information 242 to any information processing device that performs VLM fine-tuning.

[0040] The above is an example of the configuration of the data generation device 200. Next, an example of the operation of the data generation device 200 will be described with reference to Figure 6.

[0041] Figure 6 shows an example of the operation of the data generation device 200. Referring to Figure 6, the image data acquisition unit 251 acquires image data from the imaging device 110 or other external devices (step S101).

[0042] The explanatory information acquisition unit 252 acquires explanatory information that explains the content of the image data acquired by the image data acquisition unit 251 (step S102). For example, the explanatory information acquisition unit 252 can acquire explanatory information in response to input from an operator to the operation input unit 210, or to information acquired from an external device via the communication interface unit 230.

[0043] The input receiving unit 253 accepts input information, which is predetermined information such as information corresponding to the correct answer (step S103). For example, the input receiving unit 253 can accept input information in response to input from an operator to the operation input unit 210, or to information acquired from an external device via the communication interface unit 230.

[0044] The information extraction unit 254 extracts information from the image data that includes at least relationship information (step S104). For example, the information extraction unit 254 may use techniques such as visual SLAM or bird's-eye view transformation to extract relationship information from the image data.

[0045] The data generation unit 255 generates fine-tuning data, such as training data, in response to inputting relationship information extracted by the information extraction unit 254 into a trained model such as an LLM (step S105). The data generation unit 255 also stores the generated training data as data information 242 in the storage unit 240 (step S106). The data generation unit 255 may also associate the generated fine-tuning data, such as training data, with image data and store them as data information 242 in the storage unit 240.

[0046] The above is an example of the operation of the data generation device 200. Note that Figure 6 shows an example of the operation performed by the data generation device 200, but the operation of the data generation device 200 is not limited to the example shown in Figure 6. For example, the processes in steps S101, S102, and S103 may be executed in any order, or they may be executed in parallel. Also, the processes in steps S102 and S103 may be omitted.

[0047] Thus, the data generation device 200 has an information extraction unit 254 and a data generation unit 255. With this configuration, the data generation unit 255 can generate fine-tuning data such as training data using relationship information extracted by the information extraction unit 254. As a result, fine-tuning can be performed using the generated training data and other data, as well as image data. In other words, with the above configuration, it becomes possible to generate data that enables more appropriate fine-tuning.

[0048] For example, as illustrated in this disclosure, the VLM used for detecting accidents and near misses is to be fine-tuned using image data acquired by an imaging device 110 such as a drive recorder or surveillance camera. In this case, by performing fine-tuning using data for fine-tuning, such as training data generated by the data generation device 200 described above, it becomes possible to perform adjustments and evaluations using relationship information such as the positional relationship and speed of objects. As a result, adjustments to the VLM can be made so that accidents and near misses can be detected more appropriately. This makes it possible to detect accidents and near misses more appropriately in response to inputting image data into the VLM.

[0049] Note that the configuration of the data generation device 200 is not limited to the example shown in Figure 3. For example, the data generation device 200 does not have to have at least one of the explanatory information acquisition unit 252 or the input receiving unit 253. Also, Figure 7 shows another example of the configuration of the data generation device 200. Referring to Figure 7, the arithmetic processing unit 250 of the data generation device 200 can implement a learning unit 257 in addition to the configuration exemplified in Figure 3 by reading and executing the program 243 from the storage unit 240.

[0050] The learning unit 257 performs fine tuning on any VLM stored in the memory unit 240 or held by an external device, using image data and training data contained in the data information 242. The learning unit 257 may perform fine tuning using image data and training data contained in the training data information 242 using any method.

[0051] For example, as described above, the data generation device 200 may have a function for fine-tuning the VLM. Alternatively, the data generation device 200 may be configured to only generate fine-tuning data such as training data and to store the generated fine-tuning data such as training data in an external storage device.

[0052] Furthermore, in this disclosure, the data generation unit 255 generates data for fine-tuning, such as training data, in response to inputting relationship information extracted by the information extraction unit 254 into a trained model such as an LLM. In this case, the data generation unit 255 may be configured to input information such as the proximity between each object and the speed at which the objects move, along with, or instead, a bird's-eye view generated when the information extraction unit 254 extracts relationship information or images generated by region segmentation processing into the LLM. Thus, the relationship information may include not only information such as the proximity between each object and the speed at which the objects move, but also information such as a bird's-eye view used when extracting the above information.

[0053] [Second Embodiment] Next, with reference to Figures 8 and 9, a modified example of the data generation device 200, the data generation device 300, will be described. Figure 8 is a diagram showing an example of the hardware configuration of the data generation device 300. Figure 9 is a block diagram showing an example of the configuration of the data generation device 300.

[0054] The data generation device 300 is an information processing device that generates fine-tuning data, such as training data used when performing fine-tuning on trained models such as VLM, from image data. Figure 8 shows an example of the hardware configuration of the data generation device 300. Referring to Figure 8, the data generation device 300 has the following hardware configuration as an example. ·CPU(Central Processing Unit)301(Arithmetic unit) ROM (Read Only Memory) 302 (Storage Device) • RAM (Random Access Memory) 303 (storage device) • Program group 304 loaded into RAM303 • Storage device 305 for storing the program group 304 • Drive device 306 for reading and writing to the recording medium 310 outside the information processing device. • Communication interface 307 that connects to the communication network 311 outside the information processing device. • Input / output interface 308 for data input and output. • Bus 309 connecting each component

[0055] Furthermore, the data generation device 300 can realize the functions of the extraction unit 321 and generation unit 322 shown in Figure 9 by having the CPU 301 acquire the program group 304 and execute it. The program group 304 is, for example, stored in advance in a storage device 305 or ROM 302, and the CPU 301 loads it into RAM 303 or the like and executes it as needed. Alternatively, the program group 304 may be supplied to the CPU 301 via a communication network 311, or it may be stored in advance in a recording medium 310, and the drive device 306 may read the program and supply it to the CPU 301.

[0056] Figure 8 shows an example of the hardware configuration of the data generation device 300. The hardware configuration of the data generation device 300 is not limited to the case described above. For example, the data generation device 300 may consist of only a part of the configuration described above, such as not having a drive device 306. Also, the CPU 301 may be a GPU or the like as exemplified in the first embodiment.

[0057] The extraction unit 321 extracts relationship information from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between the object and the imaging device that acquired the image data. For example, the extraction unit 321 can extract relationship information such as information indicating the speed at which the detected objects are moving from the image data, or information corresponding to the proximity between each object detected from the image data.

[0058] The generation unit 322 generates data such as training data to be used when fine-tuning the trained model, using the relationship information extracted by the extraction unit 321. For example, the generation unit 322 may generate data such as training data in response to inputting the relationship information extracted by the extraction unit 321 into a trained model such as an LLM.

[0059] Thus, the data generation device 300 has an extraction unit 321 and a generation unit 322. With this configuration, the generation unit 322 can use the relationship information extracted by the extraction unit 321 to generate data such as training data used when fine-tuning a trained model. This makes it possible to generate training data and other data that enable more appropriate fine-tuning.

[0060] The data generation device 300 described above can be realized by incorporating a predetermined program into an information processing device such as the data generation device 300. Specifically, another form of the program described herein is a program for an information processing device such as the data generation device 300 that extracts relationship information from image data corresponding to at least one of the relationships between each object detected from the image data, or between an object and an imaging device that acquired the image data, and uses the extracted relationship information to generate data used when fine-tuning a trained model.

[0061] Furthermore, the data generation method performed by the information processing device such as the data generation device 300 described above involves the information processing device such as the data generation device 300 extracting relationship information from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between the object and the imaging device that acquired the image data, and then using the extracted relationship information to generate data used for fine-tuning the trained model.

[0062] Even a program having the configuration described above, or a recording medium readable by a computer on which the program is recorded, or a data generation method, can achieve the same functions and effects as the data generation device 300 described above, and thus the objectives of this disclosure described above can be achieved.

[0063] <Note> Some or all of the above embodiments may also be described as follows. The following outlines the data generation device and other components in this disclosure. However, this disclosure is not limited to the following configurations.

[0064] (Note 1) An extraction unit extracts relationship information from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. A generation unit generates data used for fine-tuning a trained model using the relationship information extracted by the extraction unit, has Data generation device. (Note 2) The extraction unit extracts information indicating the speed at which the detected object is moving from the image data as relational information. The data generation device described in Appendix 1. (Note 3) The extraction unit extracts information corresponding to the proximity between each object detected from the image data as relationship information. A data generation device as described in Appendix 1 or Appendix 2. (Note 4) The extraction unit extracts the relationship information using at least one of the following: a bird's-eye view generated by transforming the image data to a bird's-eye view, or the result of estimating its own position according to the time-series image data. A data generation device as described in any one of the items from Appendix 1 to Appendix 3. (Note 5) The generation unit generates the data in response to inputting the relationship information extracted by the extraction unit, the image data, and the descriptive information explaining the content of the image data into a trained model for generating training data. A data generation device as described in any one of the items from Appendix 1 to Appendix 4. (Note 6) The generation unit inputs, as relational information, at least one of the following into a trained model for data generation: a bird's-eye view generated by a bird's-eye view transformation of the image data, or the result of estimating its own position according to the time-series image data. The data generation device described in Appendix 5. (Note 7) The generation unit generates the data in response to inputting a prompt to the trained model specifying the format of the output data, along with the relationship information extracted by the extraction unit. A data generation device as described in Appendix 5 or Appendix 6. (Note 8) The generation unit generates the data used when fine-tuning the VLM (Vision Language Model), a pre-trained model, in response to inputting the relationship information extracted by the extraction unit and the image data into the LLM (Large Language Model), a pre-trained model for data generation. A data generation device as described in any one of the items from Appendix 1 to Appendix 7. (Note 9) Information processing device, Relationship information is extracted from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. Using the extracted relationship information, data is generated to be used when fine-tuning the trained model. Data generation method. (Note 10) In an information processing device, Relationship information is extracted from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. Using the extracted relationship information, data is generated to be used when fine-tuning the trained model. A program to perform a process.

[0065] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are dependent on the data generation device described in Appendice 1, may also be dependent on the data generation method described in Appendice 9, the program described in Appendice 10, etc., through a similar dependency relationship. Moreover, not limited to Appendices 9 and 10, some or all of the configurations described in the appendices may also be dependent on various hardware, software, various recording means, methods, or systems for recording software, without departing from the embodiments described above.

[0066] The programs described in each of the above embodiments and appendices can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0067] Although the present disclosure has been described above with reference to the embodiments described above, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. [Explanation of symbols]

[0068] 100 Data Generation Systems 110 Imaging device 200 Data Generation Devices 210 Operation Input Section 220 Screen display section 230 Communication Interface Section 240 Storage section 241 Image data information 242 Data Information 243 Programs 250 Arithmetic Processing Unit 251 Image Data Acquisition Unit 252 Information Acquisition Section 253 Input Reception Section 254 Information extraction part 255 Data Generation Unit 256 Output section 257 Learning Department 300 Data Generation Devices 301 CPU 302 ROM 303 RAM 304 Program Groups 305 Storage device 306 Drive unit 307 Communication Interface 308 Input / Output Interfaces Bus 309 310 Recording media 311 Communication Network 321 Extraction part 322 Generation part

Claims

1. An extraction unit extracts relationship information from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. A generation unit generates data used for fine-tuning a trained model using the relationship information extracted by the extraction unit, has Data generation device.

2. The extraction unit extracts information indicating the speed at which the detected object is moving from the image data as relational information. The data generation apparatus according to claim 1.

3. The extraction unit extracts information corresponding to the proximity between each object detected from the image data as relationship information. The data generation apparatus according to claim 1.

4. The extraction unit extracts the relationship information using at least one of the following: a bird's-eye view generated by transforming the image data to a bird's-eye view, or the result of estimating its own position according to the time-series image data. The data generation apparatus according to claim 1.

5. The generation unit generates the data by inputting the relationship information extracted by the extraction unit, the image data, and the descriptive information explaining the content of the image data into a trained model for data generation. The data generation apparatus according to claim 1.

6. The generation unit inputs, as relational information, at least one of the following into a trained model for data generation: a bird's-eye view generated by a bird's-eye view transformation of the image data, or the result of estimating its own position according to the time-series image data. The data generation apparatus according to claim 5.

7. The generation unit generates the data in response to inputting a prompt to the trained model specifying the format of the output data, along with the relationship information extracted by the extraction unit. The data generation apparatus according to claim 5.

8. The generation unit generates the data used when fine-tuning the Vision Language Model (VLM), a pre-trained model, in response to inputting the relationship information extracted by the extraction unit and the image data into the Large Language Model (LLM), a pre-trained model for data generation. The data generation apparatus according to claim 1.

9. Information processing device, Relationship information is extracted from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. Using the extracted relationship information, data is generated to be used when fine-tuning the trained model. Data generation method.

10. In an information processing device, Relationship information is extracted from the image data that corresponds to at least one of the relationships between each object detected from the image data, or between an object and the imaging device that acquired the image data. Using the extracted relationship information, data is generated to be used when fine-tuning the trained model. A program to perform a process.

Images