Synthetic data generation method, apparatus and device, and readable storage medium

Abstract

The invention provides a synthetic data generation method and device, equipment and a readable storage medium, and the method comprises the steps: introducing a multi-modal large language network and a multi-modal generation network, capturing complex environment details through the powerful space imagination capability of the multi-modal generation network, and carrying out the language planning through a large language model, thereby achieving the high-speed and high-efficiency generation of synthetic data. The logic understanding and self-adaptive capability of a scene is enhanced by utilizing strong reasoning capability, more accurate prediction is provided in a complex dynamic environment, and feature information of two multi-modal networks is effectively integrated through a combined prediction fusion module of a multi-modal large language network and a multi-modal generation network. Therefore, real and logicality-rich synthetic data is generated, and a prediction scene is ensured to follow logic and also meet physical laws.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and readable storage medium for generating synthetic data. Background Art

[0002] Synthetic data is data created in the digital world that imitates real-world data, rather than being collected or measured from the real world. It simulates the data distribution and patterns in the real world and is created by computational algorithms and simulations based on generative artificial intelligence technology. It can be used as a substitute for real-world data to train machine learning models and test the performance of algorithms in different scenarios. The existing synthetic data generation methods are difficult to achieve long-term prediction of future scenarios that conform to the laws of physical space, and are difficult to adapt to complex and changing environmental details, resulting in the lack of realism and poor generalization of the generated synthetic data. Summary of the invention

[0003] In view of this, in order to solve the above technical problems, the present application provides a synthetic data generation method, device, equipment and readable storage medium.

[0004] Specifically, the present application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a synthetic data generation method is provided, which is applied to an initialized synthetic data generation network; the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module; the method includes:

[0006] Get the data to be processed corresponding to the task generated by the user;

[0007] Inputting the encoded features of the data to be processed into the multimodal generation network and the multimodal large language network respectively; the initialization is achieved based on the network parameters obtained by pre-training the synthetic data generation network and matching the user-specified generation task;

[0008] The joint prediction fusion module is used to fuse the output features of the multimodal generation network and the multimodal large language network, and synthetic data is generated according to the output fusion features.

[0009] Optionally, the method further comprises:

[0010] In the case where the data to be processed includes user-set requirements, detecting whether the generated synthetic data meets the user-set requirements;

[0011] In the case that the synthesized data does not meet the user-set requirements, the description information of the user-set requirements is adjusted, and the step of generating the synthesized data according to the data to be processed is repeated until the generated synthesized data meets the user-set requirements or reaches a threshold of the number of repeated executions.

[0012] Optionally, after obtaining the to-be-processed data corresponding to the user-specified generation task, the method further includes:

[0013] Performing input modality encoding processing on the data to be processed to obtain encoding features of the data to be processed;

[0014] In the case where the data to be processed includes non-text data, converting the encoded features into feature representations that match the text modality, and adjusting the feature dimension of the feature representation to be the same as the input dimension of the multimodal large language network;

[0015] The feature representation matched with the text modality is input into the multimodal large language network.

[0016] Optionally, when the data to be processed includes scene data of multiple different perspectives, inputting the encoded features of the data to be processed into the multimodal generation network includes:

[0017] For the coding features corresponding to the scene data of each perspective, using the multimodal generation network, the coding features of each perspective are used as a first query vector, and the connection of the coding features of two adjacent perspectives is used as a first key vector and a first value vector, and the first query vector, the first key vector and the first value vector are linearly transformed through a cross-perspective weight matrix to obtain a cross-perspective attention output of each perspective; the scene data includes image data or description data of the same scene;

[0018] According to the cross-view attention output of each view, the output features of the multimodal generation network are constructed.

[0019] Optionally, when the data to be processed includes a multi-frame scene data sequence, inputting the encoded features of the data to be processed into the multimodal generation network includes:

[0020] For the coding features of each frame of scene data, using the multimodal generation network, the coding features of the target frame are used as the second query vector, the connection of the coding features of the first frame in the multi-frame scene data sequence and the next frame of the target frame is used as the second key vector, and the connection of the coding features of the first frame and the previous frame of the target frame is used as the second value vector; the target frame refers to any frame in the multi-frame scene data sequence;

[0021] Performing a linear transformation on the second query vector, the second key vector, and the second value vector through a cross-frame weight matrix to obtain a cross-frame attention output of the target frame;

[0022] According to the cross-frame attention output of each frame of scene data, the output features of the multimodal generation network are constructed.

[0023] Optionally, when the data to be processed includes image type data, inputting the encoding features of the data to be processed into the multimodal generation network includes:

[0024] Inputting the data to be processed together with the encoding features into the multimodal generation network;

[0025] Using the multimodal generative network, obtaining first image features of a first subregion where each object instance in the data to be processed is located, and adding a learnable category hint of the object instance to the first image features to obtain second image features;

[0026] Acquire a third image feature of a second sub-region that does not contain any object instance in the data to be processed, and add a learnable empty instance prompt to the third image feature to obtain a fourth image feature;

[0027] Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene controllable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0028] The self-attention outputs of the first sub-region and the second sub-region are fused with output features of other feature extraction layers of the multimodal generation network to construct output features of the multimodal generation network.

[0029] Optionally, the using the joint prediction fusion module to fuse the output features of the multimodal generation network and the multimodal large language network includes:

[0030] Using the joint prediction fusion module, the output features of the multimodal generation network and the multimodal large language network are respectively standardized to obtain a first output feature of the multimodal generation network and a second output feature of the multimodal large language network;

[0031] The first output feature and the second output feature are fused by using a fusion weight parameter to obtain the fused feature.

[0032] Optionally, after the normalization process, the method further comprises a cross attention process step:

[0033] Using the first output feature as a third query vector, and using the second output feature as a third key vector and a third value vector;

[0034] Applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector to obtain a cross-attention output of the second output feature;

[0035] The fusing the first output feature with the second output feature by using a fusion weight parameter includes:

[0036] The cross-attention output of the first output feature and the second output feature is fused through the fusion weight parameter.

[0037] Optionally, the synthetic data generation network further includes a decoder; and generating synthetic data according to the fusion features outputted by the decoder includes:

[0038] Determining a target decoder for implementing the task specified by the user from among a plurality of decoders preset for implementing different tasks;

[0039] The target decoder is used to decode the fused features to obtain the synthesized data.

[0040] Optionally, the network parameters obtained by pre-training the synthetic data generation network correspond one-to-one to the predefined generation task; the method further comprises the step of training the network parameters:

[0041] Using scene data from multiple different perspectives, the multi-view consistency training is performed on the multimodal generation network until the training converges to obtain a first network; the scene data at least includes image data or description data of the same scene;

[0042] Using a multi-frame scene data sequence, performing time consistency training on the first network until the training converges to obtain a trained multimodal generation network, and freezing the network parameters of the multimodal generation network;

[0043] For a plurality of predefined generation tasks, the multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the training sample data corresponding to the generation tasks, and the extracted features are input into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges, so as to obtain and store the network parameters of the synthetic data generation network that meet the generation tasks after training.

[0044] According to a second aspect of an embodiment of the present application, a synthetic data generation device is provided, which is applied to an initialized synthetic data generation network; the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module; the device includes:

[0045] The module for obtaining data to be processed is used to obtain the data to be processed corresponding to the task generated by the user;

[0046] A coding feature input module, used to input the coding features of the data to be processed into the multimodal generation network and the multimodal large language network respectively; the initialization is achieved based on the network parameters obtained by pre-training the synthetic data generation network and matching the user-specified generation task;

[0047] The feature fusion and synthetic data output module is used to use the joint prediction fusion module to fuse the output features of the multimodal generation network and the multimodal large language network, and generate synthetic data according to the output fusion features.

[0048] Optionally, the device further comprises:

[0049] In the case where the data to be processed includes user-set requirements, detecting whether the generated synthetic data meets the user-set requirements;

[0050] In the case that the synthesized data does not meet the user-set requirements, the description information of the user-set requirements is adjusted, and the step of generating the synthesized data according to the data to be processed is repeated until the generated synthesized data meets the user-set requirements or reaches a threshold of the number of repeated executions.

[0051] Optionally, after obtaining the to-be-processed data corresponding to the user-specified generation task, the device further includes:

[0052] Performing input modality encoding processing on the data to be processed to obtain encoding features of the data to be processed;

[0053] In the case where the data to be processed includes non-text data, converting the encoded features into feature representations that match the text modality, and adjusting the feature dimension of the feature representation to be the same as the input dimension of the multimodal large language network;

[0054] The feature representation matched with the text modality is input into the multimodal large language network.

[0055] Optionally, when the data to be processed includes scene data of multiple different perspectives, the coding feature input module is specifically used to:

[0056] For the coding features corresponding to the scene data of each perspective, using the multimodal generation network, the coding features of each perspective are used as a first query vector, and the connection of the coding features of two adjacent perspectives is used as a first key vector and a first value vector, and the first query vector, the first key vector and the first value vector are linearly transformed through a cross-perspective weight matrix to obtain a cross-perspective attention output of each perspective; the scene data includes image data or description data of the same scene;

[0057] According to the cross-view attention output of each view, the output features of the multimodal generation network are constructed.

[0058] Optionally, when the data to be processed includes a multi-frame scene data sequence, the coding feature input module is specifically used to:

[0059] For the coding features of each frame of scene data, using the multimodal generation network, the coding features of the target frame are used as the second query vector, the connection of the coding features of the first frame in the multi-frame scene data sequence and the next frame of the target frame is used as the second key vector, and the connection of the coding features of the first frame and the previous frame of the target frame is used as the second value vector; the target frame refers to any frame in the multi-frame scene data sequence;

[0060] Performing a linear transformation on the second query vector, the second key vector, and the second value vector through a cross-frame weight matrix to obtain a cross-frame attention output of the target frame;

[0061] According to the cross-frame attention output of each frame of scene data, the output features of the multimodal generation network are constructed.

[0062] Optionally, when the data to be processed includes image type data, the coding feature input module is specifically used to:

[0063] Inputting the data to be processed together with the encoding features into the multimodal generation network;

[0064] Using the multimodal generative network, obtaining first image features of a first subregion where each object instance in the data to be processed is located, and adding a learnable category hint of the object instance to the first image features to obtain second image features;

[0065] Acquire a third image feature of a second sub-region that does not contain any object instance in the data to be processed, and add a learnable empty instance prompt to the third image feature to obtain a fourth image feature;

[0066] Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene controllable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0067] The self-attention outputs of the first sub-region and the second sub-region are fused with output features of other feature extraction layers of the multimodal generation network to construct output features of the multimodal generation network.

[0068] Optionally, the feature fusion and synthesis data output module is specifically used for:

[0069] Using the joint prediction fusion module, the output features of the multimodal generation network and the multimodal large language network are respectively standardized to obtain a first output feature of the multimodal generation network and a second output feature of the multimodal large language network;

[0070] The first output feature and the second output feature are fused by using a fusion weight parameter to obtain the fused feature.

[0071] Optionally, after the normalization process, the apparatus further comprises a cross attention process step:

[0072] Using the first output feature as a third query vector, and using the second output feature as a third key vector and a third value vector;

[0073] Applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector to obtain a cross-attention output of the second output feature;

[0074] The fusing the first output feature with the second output feature by using a fusion weight parameter includes:

[0075] The cross-attention output of the first output feature and the second output feature is fused through the fusion weight parameter.

[0076] Optionally, the synthetic data generation network further includes a decoder; and the feature fusion and synthetic data output module is specifically used for:

[0077] Determining a target decoder for implementing the task specified by the user from among a plurality of decoders preset for implementing different tasks;

[0078] The target decoder is used to decode the fused features to obtain the synthesized data.

[0079] Optionally, the network parameters obtained by pre-training the synthetic data generation network correspond one-to-one to the predefined generation task; the device further comprises a step of training the network parameters:

[0080] Using scene data from multiple different perspectives, the multi-view consistency training is performed on the multimodal generation network until the training converges to obtain a first network; the scene data at least includes image data or description data of the same scene;

[0081] Using a multi-frame scene data sequence, performing time consistency training on the first network until the training converges to obtain a trained multimodal generation network, and freezing the network parameters of the multimodal generation network;

[0082] For a plurality of predefined generation tasks, the multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the training sample data corresponding to the generation tasks, and the extracted features are input into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges, so as to obtain and store the network parameters of the synthetic data generation network that meet the generation tasks after training.

[0083] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned synthetic data generation method by calling the computer program.

[0084] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned synthetic data generating method is implemented.

[0085] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0086] In the technical solution provided by the present application, a multimodal large language network and a multimodal generative network are introduced, the powerful spatial imagination ability of the multimodal generative network is used to capture complex environmental details, and a large language model is used for language planning. The powerful reasoning ability is used to enhance the logical understanding and adaptability of the scene, thereby providing more accurate predictions in complex dynamic environments. The joint prediction fusion module of the multimodal large language network and the multimodal generative network effectively integrates the feature information of the two multimodal networks, thereby generating synthetic data that is both realistic and logical, ensuring that the prediction scenario follows both logic and satisfies the laws of physics. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a schematic diagram of a synthetic data generation method flow chart shown in an exemplary embodiment of the present application;

[0088] Figure 2 is a structural schematic diagram of a synthetic data generation network shown in an exemplary embodiment of the present application;

[0089] Figure 3 It is a feature processing flow chart of introducing a cross-view attention mechanism in a multimodal generation network shown in an exemplary embodiment of the present application;

[0090] Figure 4 It is a feature processing flow chart of introducing a cross-frame attention mechanism in a multimodal generation network shown in an exemplary embodiment of the present application;

[0091] Figure 5 It is a feature processing flow chart of introducing a scene-controllable attention mechanism in a multimodal generation network shown in an exemplary embodiment of the present application;

[0092] Fig. 6A is a flowchart of steps for performing feature fusion in a joint prediction module as shown in an exemplary embodiment of the present application;

[0093] Figure 6B is a flowchart of another step of performing feature fusion by a joint prediction module shown in an exemplary embodiment of the present application;

[0094] Figure 7 is a flowchart of pre-training steps of a synthetic data generation network shown in an exemplary embodiment of the present application;

[0095] Fig. 8A This is an example of synthetic data generation shown in an exemplary embodiment of the present application;

[0096] Figure 8B is another example of synthetic data generation shown in an exemplary embodiment of the present application;

[0097] Fig. 9 is a structural schematic diagram of a synthetic data generating device shown in an exemplary embodiment of the present application;

[0098] Fig.10 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0099] Synthetic data is data created in the digital world that imitates real-world data, rather than being collected or measured from the real world. It simulates the data distribution and patterns in the real world and is created by computational algorithms and simulations based on generative artificial intelligence technology. It has the same mathematical properties as the actual data on which it is based but does not contain the same information. It can mathematically or statistically reflect the properties of real-world data and can be used as a substitute for real-world data to train machine learning models, test the performance of algorithms in different scenarios, and provide large amounts of data while protecting personal privacy. The generation of synthetic data can be based on statistical models, deep learning techniques (such as generative adversarial networks (GANs), or other computational methods to ensure that the generated data has similar properties to real data.

[0100] At present, the development of autonomous driving and embodied intelligence relies on large computing power and big data, which are the data sets required for training and testing. At present, this data set is usually obtained in the following two ways: one is real data collected through real physical environments, and the other is synthetic data generated by virtual environments built using data twin simulation technology or machine learning. Real data can reflect the actual traffic environment, but because effective scenarios such as long-tail scenarios or challenging scenarios are difficult to capture, the amount of available data in real data is often insufficient relative to the amount of data required for development, and the cost of collecting such data is high and often limited by actual conditions, making it difficult to popularize widely. In contrast, the synthetic data generated by data twin simulation technology has effectively alleviated the problem of insufficient real data to a certain extent.

[0101] The existing data synthesis algorithms have achieved certain results in scene image synthesis tasks, but when faced with complex generation tasks such as video synthesis, scene generalization, and future scene prediction, there are still problems such as difficulty in achieving long-term future scene prediction that conforms to the laws of physical space, and difficulty in effectively responding to dynamic changes and new rules in the real world. Long-term scenes involve complex causal relationships, requiring algorithms to not only have strong time series analysis capabilities, but also perform accurate counterfactual reasoning to ensure the logical coherence and physical feasibility of the predicted scenes. The existence of the above problems leads to the lack of realism and poor generalization of synthetic data generated using the existing synthetic data generation methods.

[0102] Therefore, the present application provides a method for generating synthetic data, which is applied to a synthetic data generation network that has been initialized, and the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module. The multimodal generation network is a neural network that can process and generate data from different modalities such as text, images, audio, etc., and can generate data from one modality of data or multiple modalities of data at the same time by learning the intrinsic connection between different modal data. The multimodal large language network refers to a large-scale pre-trained language model that can understand and process natural language text and interact with other modalities (such as images and videos). It has a strong text understanding ability, can extract information from other modalities, and perform cross-modal fusion and reasoning.

[0103] In this embodiment, the multimodal generation network and the multimodal large language network respectively receive input information from the input layer and perform feature processing respectively, and the outputs of the two networks are input into the joint prediction fusion module, which is used to fuse the output features of the multimodal generation network and the multimodal large language network, and the obtained fused features are passed to the subsequent layers of the synthetic data generation network, such as the decoder module, for further feature processing to generate synthetic data.

[0104] The initialization process of the synthetic data generation network refers to the process of setting or assigning network parameters in the synthetic data generation network before the inference begins. In this embodiment, the network parameters used in the initialization process refer to the network parameters obtained by pre-training the synthetic data generation network and matching the user-specified generation task. That is, the synthetic data generation network is task-specifically fine-tuned for different synthetic data generation tasks in advance to obtain the network parameters obtained by training the synthetic data generation network suitable for each generation task, so as to directly obtain the network parameters matching the user-specified generation task in the inference stage to initialize the synthetic data generation network. For example, for task 1, a set of parameters A corresponds to the task 1, and for task 2, a set of parameters B corresponds to the task 2. When the user specifies the task as task 1, the parameter A is used to initialize the synthetic data generation network.

[0105] Regarding the pre-training process of the synthetic data generation network, it can be implemented through a three-stage training method. The first two stages of the three-stage method train the multimodal generation network, and the third stage trains the joint prediction fusion module of the multimodal generation network and the large language network. During the third stage training process, the network parameters of the trained multimodal generation network and the large language network are in a frozen state. The three-stage training can include: performing cross-perspective consistency training and spatiotemporal consistency training on the multimodal generation network in turn to obtain a trained multimodal generation network, and training the joint prediction module of the trained multimodal generation network and the large language network. The three-stage method conforms to the difficulty of the training task from simple to difficult, so that the multimodal generation network sequentially learns cross-perspective consistency, spatiotemporal consistency, and joint prediction generalization capabilities. Among them, the training of the multimodal generation network can also include scene controllability training after performing cross-perspective consistency training and spatiotemporal consistency training in turn, so as to obtain a trained multimodal generation network.

[0106] Based on the above initialized synthetic data generation network, see Figure 1 As shown, the synthetic data generation method provided by the present application may at least include the following steps:

[0107] S101, obtaining the to-be-processed data corresponding to the task generated by the user;

[0108] The user-specified generation task may include but is not limited to reusing existing data, or generalizing the existing data to generate possible future scene data, or generating specific new scene data based on existing historical frame data, such as generating future frame scenes, long-tail scenes, and challenging scenes.

[0109] Among them, when the existing data set is of high quality and has been annotated, data reuse usually saves more time and cost than new data production. Data annotation includes but is not limited to 2D, 3D boxes, semantic segmentation, text labels, etc. that have completed the detection task. When the existing data set is large enough and can cover most application scenarios, reusing and appropriately adjusting the existing data can quickly obtain a large number of synthetic data sets suitable for vertical intelligence fields. Generalizing the existing data can make up for the shortcomings of existing data collection, expand the data, and provide more possibilities for downstream tasks. Generating specific new scene data based on existing historical frame data mainly refers to the generation of long-tail scenes and challenging scenes. Such scenes are difficult to collect in real life or the stock is small, which is not enough to meet the model training needs. For example, a vehicle on fire is encountered during driving, wild animals walking on urban roads, etc. The problem of insufficient real data can be made up by constructing such scene data through a synthetic data generation network.

[0110] The data to be processed is used to provide various information necessary for generating synthetic data, which may include input information required for generating synthetic data, or a combination of the input information and user-set requirements. For example, the data to be processed may include user-set requirements, control signal information, image information, point cloud information, sequence information, geometric layout information, etc. Optionally, the above information or a combination of several information are used as the data to be processed.

[0111] Specifically, for user-set requirements, encoding can be performed according to the user's specific description and requirements, such as the requirements for specific functions, preferences for certain scenarios, etc. By encoding such information into feature sequences, the synthetic data generation network can fully understand the input and perform effective data generation. Optionally, through language, keyboard selection or text, it is finally converted into a text encoding embedded in the model. The user-set requirements may include text descriptions combined according to view information, weather, light, elements, etc. For example, "driving in the rain", "remove 50% of traffic participants", "the front of the red vehicle on the left is facing outward", "this is a driving scene, please change the front car to a truck and drive in the snow", "according to the provided historical frame scene, generate a front car deceleration, self-vehicle lane change scene", "please generate a front multi-vehicle collision scene", etc. The above exemplary examples show reference examples of user-set requirements, and this embodiment is not limited to the above input forms.

[0112] Control signal information refers to the signals executed at the bottom layer of embodied intelligent terminals such as vehicles or robots, such as speed, steering wheel angle, etc.; image information refers to the frame to be edited or the historical reference view frame from the database as the image condition; point cloud information refers to the point cloud data information collected by the lidar at a certain moment from the database as the point cloud condition; sequence information refers to image information or point cloud information in the time series of historical frames, which is data collected in real time at a certain frequency or stored in the cloud; geometric layout information refers to the annotation information or true value information corresponding to the image input, which may include 2D / 3D bounding boxes, high-precision maps, bird's-eye view segmentation maps, semantic segmentation, instance segmentation maps, occupancy information and other information.

[0113] For example, if the user specifies that the generation task is to generate an image based on a text description, the data to be processed may include the text description; for another example, if the user specifies that the generation task is to generate a scene generalization image based on an existing scene image, the data to be processed may include the existing scene image such as “a traffic scene image directly in front of a vehicle”, and may also include the user's setting requirements for scene generalization such as “generating generalized images under different weather conditions”.

[0114] S102, inputting the encoding features of the data to be processed into the multimodal generation network and the multimodal large language network respectively;

[0115] For the acquired data to be processed, the data to be processed can be converted into coding features that can be understood by the model through input modality encoding processing or other feature extraction methods, thereby converting data of different modalities into a unified numerical expression, so as to facilitate subsequent processing by the multimodal generation network and the multimodal large language network.

[0116] In the encoding processing stage of the data to be processed, it can include cleaning and normalizing the input data to be processed to remove existing noise, errors or inconsistent data. For example, preprocessing the image information, such as adjusting the resolution, contrast enhancement, color correction, etc., to ensure the consistency and availability of image quality; performing data simplification, denoising and coordinate calibration on the point cloud information to make the point cloud data more accurate and clear; sorting, segmenting or extracting key nodes for the sequence information to better meet the subsequent data generation needs; standardizing and formatting the geometric layout information so that it can be better integrated with the synthetic data generation network. Alternatively, it can also include classifying and labeling the data so that the synthetic data generation network can more efficiently identify and process different types of data. For example, different images are labeled according to categories, or sequence information is classified according to specific attributes. Through the above preprocessing operations, the efficiency and quality of batch production of synthetic data can be improved, and potential errors and deviations can be reduced.

[0117] The coded features are respectively sent to two different network models: a multimodal generative network and a multimodal large language network. The multimodal generative network is used for visual planning, and its powerful spatial imagination ability is used to capture complex environmental details. The large language network is used for language planning, and its powerful reasoning ability is used to enhance the logical understanding and adaptability of the scene. In this embodiment, the multimodal generative network and the multimodal large language network further process the received coded features based on the network parameters assigned by the initialization processing, accurately extract complex dynamic environmental features and scene features, and fully explore the spatiotemporal physical evolution laws and potential associations of the scenes in the data to be processed.

[0118] S103, using the joint prediction fusion module to fuse the output features of the multimodal generation network and the multimodal large language network, and generating synthetic data according to the output fusion features.

[0119] The joint prediction fusion module is used to fuse the output features from the multimodal generative network and the multimodal large language network. This module can understand and integrate information from different networks to generate rich and coordinated fusion features. Before fusion, the received features from different sources can be preprocessed to ensure their consistency in dimension, scale or data distribution. Normalization, feature dimensionality reduction or feature selection can be used according to the actual situation.

[0120] The feature fusion strategy of the joint prediction fusion module may include but is limited to direct concatenation, attention mechanism, gating mechanism, and feature conversion and alignment. Among them, direct concatenation is used to directly concatenate feature vectors from different networks into a longer vector to retain all original information; the attention mechanism can dynamically adjust the importance of different features based on the weight matrix learned from task requirements and data characteristics, so that the model can focus more on features that are useful for the current synthesis task; the gating mechanism is used to control the information flow between different feature streams, and can determine which features should be retained and which should be suppressed through learning; feature conversion and alignment is used to convert features through learnable conversion matrices or network layers to make them more aligned or compatible between different modalities.

[0121] The fused features output by the joint prediction module can be further optimized by using additional neural network layers such as fully connected layers, convolutional layers, etc. to extract higher-level features or perform feature selection to reduce redundant information. For generalization or prediction of new scene data, a context modeling mechanism can be set to perform joint modeling based on the fused features and their associated context to capture the relationship and dependency between different data points.

[0122] Based on the fusion feature, one or more decoders can be used to generate synthetic data. The decoder can be designed for a specific modality, such as an image decoder, a text decoder, etc., or a general multimodal decoder. In this embodiment, different decoders can be set for different tasks. When it is necessary to decode the fusion feature to generate synthetic data, the target decoder for implementing the user-specified generation task can be determined from a plurality of decoders pre-set for implementing different tasks, and the fusion feature is decoded using the target decoder to obtain the synthetic data. Among them, the decoder is used to gradually convert the fusion feature into the original form of synthetic data such as pixel values, audio waveforms, text characters, etc. The output synthetic data type depends on the user-specified generation task type and the user-set requirements. The output synthetic data can be new scene image data, video data, specific execution action data, text description, etc., or a combination of the above data.

[0123] In the process of generating synthetic data based on the fusion features output by the joint prediction fusion module, the decoder can be set to perform batch generation, that is, to generate multiple sets of synthetic data at one time to meet different needs or perform large-scale data analysis.

[0124] In the disclosed embodiment, by integrating a multimodal generation network, a multimodal large language network and a joint prediction fusion module, the spatial imagination ability of the multimodal generation network is used to capture environmental details, while the powerful reasoning ability of the multimodal large language network is used to enhance logical coherence, and the feature information of the two multimodal networks is effectively integrated through the joint prediction fusion module, thereby generating synthetic data that is both real and logical. Among them, the multimodal generation network can capture complex visual details and ensure that the generated synthetic data is highly realistic visually. Its powerful spatial imagination ability enables the predicted scene to strictly follow the laws of physical space, avoiding the common physical logic inconsistency problems in traditional methods. The introduction of the multimodal large language network enables the generation of synthetic data to be reasoned based on language planning, thereby enhancing the logical coherence of the entire prediction sequence. Combined with joint prediction modeling, the system's adaptability to new scenes is improved. When facing a complex and changing environment, it can also generate prediction scenes that are both logical and meet the laws of physics.

[0125] In some embodiments, in order to further ensure the accuracy and practicality of the data, to obtain synthetic data that meets the expectations and requirements of the user, and to improve the quality and efficiency of data production, after obtaining the synthetic data generated according to the data to be processed, when the data to be processed includes user-set requirements, the above-mentioned step S103 may also include a production quality review step, that is, detecting whether the generated synthetic data meets the user-set requirements; when the synthetic data does not meet the user-set requirements, adjusting the description information of the user-set requirements, and repeating the step of generating synthetic data according to the data to be processed until the generated synthetic data meets the user-set requirements or reaches a threshold of the number of repeated executions.

[0126] The production quality audit is divided into intelligent algorithm audit and manual precision audit. The audit method actually adopted by the intelligent algorithm audit is dynamically determined according to the type of synthetic data to be audited. For example, in the production of image data, the generated images can be audited in terms of resolution, color accuracy, image clarity, etc.; for the production of sequence data such as video data, the time series consistency and whether it meets the laws of spatial physical evolution can be checked.

[0127] During the production quality review process, data that meets the user's set requirements will be screened out and stored in the cloud data platform in a timely manner to achieve efficient data management and utilization. For data that does not meet the user's set requirements, the description information of the user's set requirements can be dynamically adjusted through prompt engineering and machine learning. The description method is changed to more accurately describe the user's set requirements while maintaining the same core meaning, and the feature encoding is re-performed based on the updated user's set requirements. The obtained encoded features are respectively input into the multimodal large language network and the multimodal generation network to continue to generate synthetic data until the generated synthetic data meets the user's set requirements or the number of regeneration iterations reaches the set threshold.

[0128] Through this production quality audit method, the production quality and efficiency of the synthetic data finally obtained can be improved. For example, in practical applications, when faced with a large amount of initial data, through the processing flow of the present invention, it is possible to accurately screen and store data that meets the requirements, while continuously optimizing data that does not meet the requirements, thereby obtaining a high-quality synthetic data set that meets the comprehensive needs of users.

[0129] In some embodiments, based on the limitations of large language models in processing multimodal data, in order to enable the large language model to effectively process and generate synthetic content based on non-text data, after obtaining the data to be processed corresponding to the user-specified generation task, the method further includes a feature modality alignment step, which can be implemented by the following steps a1-a3:

[0130] a1, performing input modality encoding processing on the data to be processed to obtain encoding features of the data to be processed;

[0131] The input modality encoding processing is performed on the data to be processed corresponding to the user-specified generation task, which is used to convert the original data such as text, image or other types of data into encoding features that can be understood by the model, thereby converting the data of different modalities into a unified numerical expression, so as to facilitate the subsequent processing by the multimodal generation network and the multimodal large language network. Among them, the input modality encoding processing of the data to be processed can be implemented by the input modality encoding module that has been trained in the relevant technology, and this application does not limit this.

[0132] a2, when the data to be processed includes non-text data, converting the encoded features into feature representations matching the text modality, and adjusting the feature dimension of the feature representation to be the same as the input dimension of the multimodal large language network;

[0133] Since large language models are essentially trained and work based on text data, they cannot directly understand and process data in non-text formats. Therefore, when it is recognized that the data to be processed contains non-text data, the encoded features are converted into feature representations that match the text modality. Deep learning techniques such as modality alignment modules, multi-layer perceptron networks, or multi-layer attention mechanisms can be used to convert the feature representation of non-text data into a feature space similar to the text modality, thereby achieving alignment between modalities.

[0134] At the same time, in order to ensure that the converted feature representation can be correctly received and processed by the multimodal large language network, the feature dimension of the feature representation is further adjusted to keep it consistent with the input dimension of the multimodal large language network.

[0135] a3, inputting the feature representation matching the text modality into the multimodal large language network.

[0136] The feature representation that has been modally aligned and dimensionally adjusted and matches the text modality is input into the multimodal large language network, so that the large language model can collaborate with the multimodal generation network to generate richer, more diverse and user-friendly synthetic content based on multimodal input data.

[0137] The above-mentioned characteristic mode alignment step can be implemented by the network structure layer including the synthetic data network, that is, Figure 2 The structural diagram of the synthetic data generation network shown in the figure, in addition to the multimodal generation network, the multimodal large language network and the joint prediction module, the synthetic data generation network can also include an input modality encoding module and a feature modality alignment module. Among them, the input modality encoding module is used to perform input modality encoding processing on the data to be processed to obtain the encoding features of the data to be processed; the feature modality alignment module is used to receive the encoding features from the input modality encoding module, and convert the encoding features into feature representations that match the text modality, and adjust the feature dimensions of the feature representations to be the same as the input dimensions of the multimodal large language network, and the adjusted feature representations are input into the multimodal large language network for processing.

[0138] In the disclosed embodiment, by introducing the modality alignment module, the alignment of non-text data and text modality at the feature level is achieved, which solves the problem that the large language model cannot directly process non-text data. At the same time, by adjusting the feature dimension, the compatibility of the data and the effectiveness of the model are ensured. In addition, through modality alignment and feature conversion, complex non-text data is converted into a unified text modality feature representation, which reduces the number of tokens processed by the large language network, simplifies the processing flow of the model, reduces the computing power loss and reasoning time of the model operation, and improves the efficiency of data processing. In addition, the design of the multimodal large language network combined with the modality alignment module enables the model to process more diverse input data, enhances the versatility and scalability of the model, and provides users with a more flexible and convenient data generation method to meet the diverse needs in different scenarios.

[0139] In some embodiments, in the case where the data to be processed includes scene data from multiple different perspectives, in order to make full use of the complementary information between multiple perspectives and enhance the richness and accuracy of the generated content, this embodiment introduces a cross-perspective consistency attention layer in the multimodal generation network to focus on the information interaction and fusion between cross-perspectives while considering the information extraction within a single perspective. Based on the cross-perspective consistency attention layer, Figure 3 As shown in the flowchart, the aforementioned step S102 of inputting the encoded features of the data to be processed into the multimodal generation network may include the following steps:

[0140] S301, for the coding features corresponding to the scene data of each perspective, using the multimodal generation network, taking the coding features of each perspective as a first query vector, and connecting the coding features of two adjacent perspectives as a first key vector and a first value vector;

[0141] When the current view for which the cross-view attention output is to be calculated has an adjacent view, the connection value of the encoding feature of the current view and the encoding feature of the adjacent view can be used as the key vector and the value vector, or the encoding feature of the adjacent view can be directly used as the key vector and the value vector.

[0142] S302, performing a linear transformation on the first query vector, the first key vector and the first value vector through a cross-view weight matrix to obtain a cross-view attention output for each view; the scene data includes image data or description data of the same scene;

[0143] The cross-view weight matrix refers to a parameter matrix that can be updated during the training process, which is used to linearly transform the encoded features from different views in the cross-view attention mechanism. This matrix can learn the correlation and importance of the encoded features between different views to better integrate cross-view information.

[0144] The cross-view attention output refers to the attention-weighted feature representation of each view calculated by the cross-view attention mechanism. In the attention mechanism, the network dynamically assigns attention weights based on the correlation between the encoding features of the current view (query vector) and the encoding features of other views (key vector and value vector), and obtains the cross-view attention output by weighted summation. The cross-view attention output contains complementary information between the current view and the adjacent view, which facilitates the generation of more accurate and coherent predicted images.

[0145] Assuming that there are scene data from three perspectives, such as left perspective, middle perspective, and right perspective, first extract the encoding features of each perspective. Next, for the middle perspective, the encoding features of the middle perspective are used as the query vector, and the connection values ​​of the encoding features of the adjacent left and right perspectives are used as the key vector and value vector; then, the query vector, key vector, and value vector features are linearly transformed by the above cross-perspective weight matrix, and the attention weights are calculated. Finally, the cross-perspective attention output of the middle perspective is obtained by weighted summation. The cross-perspective attention output of the middle perspective integrates the information of the left and right perspectives, making the predicted image of the middle perspective more accurate and coherent.

[0146] For the cross-view attention mechanism described in step S302 above, its calculation process can also be described by the following formula:

[0147]

[0148] Among them, [,] represents the feature connection operation, As the cross-view weight matrix in the cross-view attention mechanism, it is used to linearly transform the query vector, key vector, and value vector; d is the dimension of the key vector and query vector, which is used to scale the dot product result to prevent the gradient from disappearing; refers to the encoding feature of the i-th view, and is the encoded features of the two adjacent views of the i-th view; the softmax function is used to normalize the attention weight. For the i-th view, the attention weight is calculated by the softamx function and multiplied by the value vector of the adjacent view to obtain the weighted feature representation of the i-th view as the cross-view attention output.

[0149] S303: construct output features of the multimodal generation network according to the cross-view attention output of each view.

[0150] The cross-view attention output of each view integrates the feature representation of the current view and the adjacent view information, capturing the intrinsic connection and complementary information of the scene between different viewpoints. After obtaining the cross-view attention output of each view, the cross-view attention output of each view can be passed to the subsequent processing layer to be complementary and fused with the output features of other feature extraction layers of the multimodal generation network to jointly construct the final output features of the multimodal generation network.

[0151] In the disclosed embodiments, a cross-view attention mechanism is introduced to enable the multimodal generation network to make full use of the complementary information in the multi-view scene data, construct a unique query vector for each view, and use the information of its adjacent views as key vectors and value vectors, so that each view can pay attention to and integrate the complementary information of other views while generating its own features, effectively integrating information from different viewpoints, generating richer and more comprehensive output features, and improving the generalization of synthetic data generation.

[0152] In some embodiments, when the data to be processed includes a multi-frame scene data sequence, based on the fact that each frame of data contains state information of the scene at different time points, in order to capture the temporal dependency and dynamic change information between frames, this embodiment introduces a cross-frame attention mechanism in the multimodal generation network to focus on the feature expression within a single frame and capture the transmission and fusion of information between frames, thereby providing a more comprehensive and coherent input for the multimodal generation network. Based on the cross-frame consistency attention layer, Figure 4 As shown, the step S102 of inputting the encoded features of the data to be processed into the multimodal generation network can be implemented in the following manner:

[0153] S401, for the coding features of each frame of scene data, using the multimodal generation network, taking the coding features of the target frame as the second query vector, taking the connection of the coding features of the first frame in the multi-frame scene data sequence and the next frame of the target frame as the second key vector, and taking the connection of the coding features of the first frame and the previous frame of the target frame as the second value vector; the target frame refers to any frame in the multi-frame scene data sequence;

[0154] In the attention mechanism, the query vector refers to the vector used to query relevant information, the key vector is the vector that is compared with the query vector to determine which information is important, and the value vector contains the vector of the information actually to be retrieved. In this embodiment, the query vector represents the encoding features of the target frame, which is used to retrieve relevant information from other frames; the key vector is formed by connecting the encoding features of the next frame of the target frame and the first frame of the sequence, and contains future and initial scene information related to the target frame; the value vector is formed by connecting the encoding features of the previous frame of the target frame and the first frame of the sequence, and provides information about scene changes before the target frame.

[0155] Assume that the input sequence has three frames of images A (first frame), B (target frame), and C (next frame of the target frame). When processing the target frame B, the encoded features of B are used as the query vector, the key vector is formed by connecting the encoded features of A and C, and the value vector is formed by connecting the encoded features of the previous frame of A and B. When the target frame is the first frame of the input sequence, the previous frame of the target frame can be calculated by using the target frame itself or a zero vector; similarly, when the target frame is the last frame of the input sequence, the next frame of the target frame can be calculated by using the target frame itself or a zero vector.

[0156] S402, performing a linear transformation on the second query vector, the second key vector, and the second value vector through a cross-frame weight matrix to obtain a cross-frame attention output of the target frame;

[0157] The cross-frame weight matrix refers to the trainable parameter matrix used in the attention mechanism to perform a linear transformation on the query vector, key vector, and value vector in order to calculate the attention weights and the final attention output. This linear transformation maps the input vector to another vector space through matrix multiplication. The cross-frame attention output is obtained after the attention mechanism is processed and contains the relevant information retrieved from the key vector and value vector according to the query vector.

[0158] In this embodiment, the cross-frame learnable matrix performs a linear transformation on the second query vector, the second key vector, and the second value vector to calculate an attention weight matrix, which determines the degree of attention that should be given to other frames when generating a predicted image of the target frame. The cross-frame attention output of the target frame is obtained by weighted summing the value vectors, which includes relevant information from other frames, thereby generating a more coherent and accurate predicted image.

[0159] The calculation process of the cross-view attention mechanism described in this step can also be described by the following formula:

[0160]

[0161] Among them, [,] represents the feature connection operation, As the cross-frame weight matrix in the cross-frame attention mechanism, it is used to linearly transform the query vector, key vector, and value vector; d is the dimension of the key vector and query vector, which is used to scale the dot product result to prevent the gradient from disappearing; refers to the potential features of the j-th frame scene data, They are respectively the potential features of the first frame scene data, the previous frame of the j-th frame, and the next frame scene data in the input sequence; the softmax function is used to normalize the attention weight. For the j-th frame scene data, the attention weight is calculated by the softamx function, multiplied by the value vector composed of the first frame and the next frame scene data, to obtain the weighted feature representation of the j-th frame scene data as the cross-frame attention output.

[0162] S403, constructing output features of the multimodal generation network according to the cross-frame attention output of each frame of scene data.

[0163] In this step, the cross-frame attention output based on each frame of scene data is combined with the characteristics of the target frame itself and the enhanced feature representation from related information of other frames. After obtaining the cross-view attention output of each perspective, the cross-view attention output of each perspective can be passed to the subsequent processing layer, and complementarily fused with the output features of other feature extraction layers of the multimodal generation network to jointly construct the final output features of the multimodal generation network, thereby enhancing the spatiotemporal evolution laws and scene motion trajectory characteristics extracted by the multimodal generation network, so as to achieve accurate prediction of dynamic scenes.

[0164] In the disclosed embodiment, by introducing the information of the frames before and after the target frame as the key vector and the value vector, and introducing the cross-frame attention mechanism, the multimodal generation network can take into account a wider context when generating the target frame features, and capture the temporal dependency and dynamic change information between frames, and fuse the feature information at different time points, so as to generate more coherent and natural output features.

[0165] In some embodiments, when the data to be processed includes image type data, in order to achieve a more refined and controllable multimodal generation process, this embodiment introduces a scene controllable attention mechanism in the multimodal generation network, focusing on the detailed feature extraction of each object instance in the image, and enhancing the network's understanding and control capabilities of the scene content through learnable category prompts and empty instance prompts. By dividing the image into regions and processing them separately, it ensures that the generated content can flexibly adapt to and integrate background information while maintaining clear object instance features.

[0166] Based on the scene controllable attention mechanism, such as Figure 5 As shown, the step S102 of inputting the encoded features of the data to be processed into the multimodal generation network can be implemented in the following manner:

[0167] S501, inputting the data to be processed and the encoding features into the multimodal generation network;

[0168] S502, using the multimodal generation network, obtaining a first image feature of a first sub-region where each object instance in the data to be processed is located, and adding a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0169] The object instance refers to the physical object described in the data to be processed, such as a pedestrian, a car or a traffic light, and each object instance is a separate instance of the same type of object, which is identified by a target or object detection algorithm. The first sub-region refers to the image region associated with each object instance, which can be determined based on the bounding box of the object instance, such as directly taking the region where the bounding box is located as the first sub-region where the object instance is located.

[0170] The learnable category hint refers to information that is dynamically adjusted and optimized during the model training process. The category hint is used to represent the feature representation of the category of the object instance, and is used to guide the model to better understand and distinguish object instances of different categories. The category hint usually exists in the form of a vector, each vector represents a category and contains information or attributes unique to the category. The category hint is continuously iterated and updated during the training process to better combine with the image features, and is shared among all the attention layers of the model, ensuring a consistent understanding of the object instances at different levels.

[0171] Identify each object instance contained in the image type data included in the data to be processed, and locate the first sub-region where the object instance is located; for each first sub-region, extract its corresponding first image feature, and innovatively add a learnable category hint to each object instance. The category hint serves as additional information input to guide the network to more accurately understand and represent the category features of the object instance, thereby generating a more accurate second image feature.

[0172] Combining the category hint of each object instance with its corresponding first image feature can be achieved through a simple splicing operation. The first image feature of the first sub-region where each object instance is located is added with the category hint of the corresponding object instance, thereby forming the second image feature of the first sub-region. The second image feature not only contains the image information of the object instance, but also incorporates the semantic information at the category level of the object instance, so that the model can take into account both the image features and category information of the object instance, which is convenient for precise control of each object instance during the synthetic data generation process.

[0173] S503, obtaining a third image feature of a second sub-region that does not contain any object instance in the data to be processed, and adding a learnable empty instance prompt to the third image feature to obtain a fourth image feature;

[0174] Focusing on the second sub-region in the image that does not contain any object instance, such as the background part, extracting its corresponding third image feature, and adding a learnable empty instance prompt to it, the empty instance prompt is used to guide the model to distinguish and generate background information. The fourth image feature obtained based on this method not only contains the visual information of the second sub-region, but also contains additional prompt information indicating that it does not contain an object instance.

[0175] S504, performing linear transformation on the second image feature and the fourth image feature respectively according to the scene controllable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0176] In the self-attention mechanism, the preconfigured weight matrix includes W Q , W K and W V A matrix is ​​used to convert the input feature vector into a query vector, a key vector, and a value vector, and the weight matrix is ​​learned by the back propagation algorithm during the model training process. The linear transformation of the second image feature and the fourth image feature according to the pre-configured weight matrix can be implemented by a pre-configured linear projection layer, which is used to convert the input data into a query, key, and value vector by applying a linear transformation. For example, the linear projection layer can be expressed as Then, performing a linear transformation on the second image feature to obtain the target query, target key and target value can be expressed as:

[0177]

[0178] Among them, r (i) represents the first image feature of the first subregion of the i-th object instance of category k, r (i) =a (i) +e k represents the second image feature after adding category hints, represents the linear projection layer shared across object instances. Based on this, the self-attention output of the first sub-region obtained by applying the self-attention mechanism can be expressed as:

[0179]

[0180] Similarly, The self-attention output of the second sub-region obtained by applying the self-attention mechanism can be expressed as:

[0181]

[0182] The second and fourth image features are linearly transformed using a scene-controllable weight matrix, and the self-attention mechanism is applied to enhance the feature interaction between regions within the image, so that the network can dynamically adjust the weights of different regions according to scene requirements, achieving more flexible scene control.

[0183] S505: Fusing the self-attention outputs of the first sub-region and the second sub-region with output features of other feature extraction layers of the multimodal generation network to construct output features of the multimodal generation network.

[0184] The self-attention outputs of the first sub-region and the second sub-region are fused with the output features of other feature extraction layers of the multimodal generative network to construct the final output features, which integrates multiple information sources in the image and ensures the comprehensiveness and consistency of the generated content.

[0185] In the embodiments of the present disclosure, by introducing learnable category hints, empty instance hints, and scene controllable weight matrices, the network can focus more on feature extraction and representation of each region. The application of the self-attention mechanism further enhances the feature interaction and fusion between regions, thereby achieving fine control of the scene content during the image generation process, enabling the network to generate diverse scene layouts and object configurations according to different needs.

[0186] In some embodiments, in order to optimize and enhance the feature representation of the instance box of the object instance (i.e., the bounding box of the object instance), while ensuring the consistency and controllability of the instance box in the process of generating synthetic data, this embodiment adds a step of optimizing the features of the instance box by using a gated self-attention mechanism on the basis of adding a category prompt to the area where the object instance is located. The optimization of the features of the instance box by using a gated self-attention mechanism can be achieved by the following steps b1-b3:

[0187] b1, obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector;

[0188] The bounding box of the object instance is a rectangular box used to locate the position of an object instance in the image. It can be defined by four coordinate values ​​to accurately indicate the position and size of the object. Obtaining the bounding box of the object instance is essentially obtaining the position information of the bounding box. Each object instance is classified into a specific category, such as "dog", "cat", "car", etc. The category information is used to distinguish different object instances.

[0189] Hidden vectors are usually used in computer vision and deep learning to store additional information about data, which is information that needs to be specially encoded during the reasoning process. In this embodiment, the hidden vector is used to store additional information of each object instance, such as the category of the instance and the location information of the bounding box or other related features, to capture the structure and relationship of the instance boxes of different object instances, so as to achieve fine control over specific instances when generating synthetic data.

[0190] For example, the position information can be converted into a first representation vector by Fourier embedding or position encoding, machine learning, etc., and the category of the instance object can be converted to generate the second representation vector by text embedding representation or pre-trained models such as CLIP (Contrastive Language-Image Pre-training). After obtaining the second representation vector and the second representation vector, the two vector representations are concatenated using a [,] connection operation to obtain a combined vector. Regarding the use of a nonlinear transformation of a neural network to convert the combined vector into a hidden vector, a multi-layer perceptron can be used to map the combined vector to generate a fixed-length vector representation as the hidden vector.

[0191] b2, feature-connecting the hidden vector with the image features of the image type data included in the data to be processed to obtain a first feature representation, linearly transforming the first feature representation according to the weight matrix, and applying the self-attention mechanism to obtain the self-attention output of the first feature representation, and applying a visual tag selection operation and a gating mechanism to the self-attention output of the first feature representation to generate the merged image feature; the visual tag selection operation is used to select the tag that is most relevant to the task requirement from the self-attention output of the first feature representation.

[0192] The gating mechanism can adjust the contribution of the output result of the visual mark selection operation to the initial image feature through learnable gating parameters and hyperbolic tangent functions, that is, the above-mentioned application of the visual mark selection operation and the gating mechanism to the self-attention output of the first feature representation can be implemented as follows: applying the visual mark selection operation to the self-attention output of the first feature representation to obtain a second feature representation; applying the gating mechanism to the second feature representation, and adding the corresponding gating mechanism output to the initial image feature to generate the merged image feature.

[0193] Based on the above hidden vector and gating mechanism implementation, the process of constructing the merged image features according to the hidden vector and the initial image features can be expressed by the formula:

[0194]

[0195] Wherein, x on the right side of the formula represents the initial image feature; η is a learnable parameter used to adjust the degree of fusion between the initial image feature x and the output result of TS(); tanh() represents the hyperbolic tangent function; TS() represents the visual marker selection operation; () represents the self-attention mechanism; [,] represents the feature vector connection operation; H b Represents a hidden vector. When using fourier() embedding, text embedding representation, and multi-layer perceptron to obtain a hidden vector, the process of obtaining the hidden vector can be expressed as:

[0196]

[0197] Among them, MLP() represents the mapping process of the multi-layer perceptron, and the bounding box Can be a 2D box or a 3D box, bounding box type Where N is the number of instance boxes of the object instance.

[0198] To better understand the above hidden vectors and merged image features, we will take the input scenario of "the image in front of the vehicle, including a traffic light and a pedestrian" as an example. b It refers to the hidden vector containing the position information and category information of the bounding box of traffic lights and pedestrians; the initial image feature X on the right side of the formula represents the image feature encoding of the input scene, which contains the visual information of the background area and all object instances; TS() is used to select the visual mark that is most relevant to the current task (such as optimizing the features of pedestrians or traffic lights) from the output of the self-attention mechanism. For example, if the current task is to optimize the position of pedestrians in the scene, then TS() will be more inclined to select marks associated with pedestrian features; X, which is finally output by the gated self-attention mechanism, is a scene graph feature encoding for the entire scene that contains the optimization results of traffic lights and pedestrian features, which more accurately expresses the relative position of pedestrians in the scene and the state of traffic lights, ensuring that the position, category and fusion of all instance boxes in the scene with the background are better optimized, thereby ensuring the controllability and accuracy of details of the generated scene.

[0199] In the case of an instance box containing multiple object instances, each instance box has a corresponding hidden vector. In this step, the hidden vectors of all instance boxes are first integrated together, and then the integration result is spliced ​​with the initial image feature. The weight matrix used for linear transformation in this embodiment is the same as the weight matrix used for linear transformation in the aforementioned step 504, and the same linear transformation is used.

[0200] b3. Construct output features of the multimodal generation network according to the self-attention outputs of the first sub-region and the second sub-region, the merged image features, and output features of other feature extraction layers of the multimodal generation network.

[0201] In the disclosed embodiments, the bounding box of the object instance is encoded together with the category information into a hidden vector, so that different object instances can be identified and distinguished when processing different object instances. At the same time, the self-attention mechanism is used to focus on the overall layout and contextual information in the initial image features, thereby enhancing the richness of the feature representation of each object instance. The gating mechanism is combined to dynamically adjust the attention to different object instance features, thereby achieving precise control of the instance frame, and maintaining consistency between the categories of the object instances and the layout and position of the instance frames in the subsequent feature integration and decoding process.

[0202] In some embodiments, the output features of the multimodal generation network and the multimodal large language network are fused using the joint prediction fusion module as described in step S103 above, such as Fig. 6A As shown, this embodiment uses weighted fusion to achieve output feature fusion of different networks:

[0203] S601, using the joint prediction fusion module, respectively standardizing the output features of the multimodal generation network and the multimodal large language network to obtain a first output feature of the multimodal generation network and a second output feature of the multimodal large language network;

[0204] This embodiment provides a joint prediction fusion module to make a fusion decision on the output features of the multimodal generation network and the output features of the multimodal large language network, and finally outputs the synthetic data expected by the user through decoders defined by different tasks. In order to ensure that the output features of the two networks are comparable on a unified scale, the output features of the two networks need to be standardized or normalized, which can be achieved by subtracting the mean and dividing by the standard deviation, that is:

[0205]

[0206] Among them, O mwm Represents the output features of the multimodal generation network without normalization, O' mwm is the corresponding first output feature, O ml Represents the output features of the multimodal large language network without normalization, O' mlm The corresponding second output features, μ, σ, correspond to the mean and standard deviation of the output features of the multimodal generation network and the multimodal large language network, respectively.

[0207] S602: Fusing the first output feature and the second output feature by using a fusion weight parameter to obtain the fused feature.

[0208] The fusion process can be expressed by the following formula:

[0209] F final =β·O' mwm +(1-β)·O' mlm

[0210] Among them, β is a learnable weight parameter used to balance the importance of the first output feature of the multimodal generation network output and the second output feature of the multimodal large language network output.

[0211] In some embodiments, in order to improve the processing capability and fusion effect of the multimodal large language network for complex information, this embodiment further introduces a cross attention mechanism after the output features of the two networks are standardized and applied to the second output feature of the multimodal large language network. That is, after the standardization process, Figure 6B As shown, the method also includes a cross attention processing step:

[0212] S603, using the first output feature as a third query vector, and using the second output feature as a third key vector and a third value vector, and applying a cross attention mechanism to the third query vector, the third key vector, and the third value vector to obtain a cross attention output of the second output feature;

[0213] That is, the cross-attention output obtained by applying the cross-attention mechanism to the second output feature of the multimodal large language network can be expressed by the following formula:

[0214]

[0215] Among them, O' mwm is the first output feature of the multimodal generation network, O' mlm It is the second output feature of the multimodal large language network. Adding the cross attention mechanism can further improve the interaction and fusion effect of multimodal features, ensuring that each feature position of the multimodal generative network obtains a normalized attention weight distribution, pointing to different parts of the first extracted feature output by the large language network.

[0216] The second output feature based on the output of the multimodal large language network is further aggregated into a cross-attention output. The step S602 in the above embodiment fuses the first output feature with the second output feature through a fusion weight parameter, and can be updated to step S604: fusing the cross-attention output of the first output feature with the second output feature through the fusion weight parameter.

[0217] In the disclosed embodiment, by introducing the cross-attention mechanism, the second output features of the multimodal large language network can more deeply understand and integrate into the first output features of the multimodal generation network, capture more detailed and subtle relationships between features, and promote the interaction and fusion between different modal features, so that the fused features output by the joint prediction fusion module incorporate the interaction and dependency between features, and can generate richer, more diverse and actual scene data, and the generated synthetic data can more accurately reflect the intrinsic connection and mutual influence between multiple modal information.

[0218] In some embodiments, combined with the description of the initialization of the synthetic data generation network described in the above embodiments, it can be seen that the network parameters obtained by pre-training the synthetic data generation network in the embodiments of the present application correspond one-to-one to the predefined generation tasks, that is, based on the different processing requirements of different synthetic data generation tasks, the synthetic data generation network is pre-trained for tasks, and the network parameters adapted to each synthetic data generation task and used to implement the generation task are obtained and stored. Figure 7 As shown in the flowchart of the steps, the synthetic data generation network method is pre-trained to obtain network parameters that match the set generation task can be obtained by the following three-stage training method:

[0219] S701, using scene data of multiple different perspectives, performing multi-perspective consistency training on the multimodal generative network until the training converges, to obtain a first network; the scene data at least includes image data or description data of the same scene;

[0220] Scene data refers to information that describes or represents a specific environment or scene. For example, in a smart home scenario, scene data may include images of indoor environments, such as images of living rooms and bedrooms. In this embodiment, the scene data may include image data or description data. Among them, image data refers to two-dimensional or three-dimensional visual information captured by an image sensor (such as a camera), stored in the form of a pixel matrix, and contains visual features such as color, brightness, and texture. Description data refers to text information that describes an object, event, or scene in natural language or other forms. For example, description data may include a text describing the content of the scene, such as a scene description of "a black and white dog chasing a frisbee in the park", and an image of the scene captured by a camera or other image acquisition device can be used as image data of the scene.

[0221] Scene data from multiple different perspectives refers to static scene information of the same scene captured from multiple different perspectives at the same time. This scene information is used to describe the scene content such as objects, layout, relationships, and scene status in the scene at a specific time point. The scene data from multiple different perspectives can be collected by multiple different sensors or image acquisition devices. Each perspective provides different side information of the scene, which can be used to extract higher-level semantic and structural information, such as object category, position, orientation, relationship, and other scene space geometric layout information, to help the multimodal generation network better understand the scene.

[0222] Cross-view consistency is used to describe the consistency of the same object across viewpoints captured by multiple cameras arranged on embodied carriers such as vehicles or robots. Cross-view consistency training refers to improving the generalization and robustness of the network by ensuring that the network can generate consistent outputs at different viewpoints when training a multimodal generative network. Through cross-view consistency training, the data generated by the network can have consistent semantics and structural information at different viewpoints, reducing ambiguity and errors caused by changes in viewpoints, thereby improving the network's ability to understand and process complex scenes, enabling it to better adapt to the changing environment in practical applications. For example, in an autonomous driving scenario, image data from different cameras in the front, back, left, and right of the vehicle can be used to train a multimodal generative network to ensure that no matter where the vehicle is, the network can accurately identify and generate corresponding road, pedestrian and other obstacle information.

[0223] Regarding cross-view consistency training, scene data from multiple viewpoints can be collected as sample data, and a first loss function can be defined during the cross-view consistency training process. The first loss function includes at least a single view loss and a cross-view consistency loss. The first loss function is used to evaluate the consistency between the generated data and the sample data of the multimodal generation network at different viewpoints, and an optimization algorithm such as stochastic gradient descent or its variants is used to back-propagate the gradient according to the first loss function to update the network parameters. Among them, the single view loss is used to measure the difference between the generated predicted image and the first training sample in each single view; the cross-view consistency loss is used to measure the consistency of the generated results between different viewpoints, which can be achieved by comparing the similarity or difference between the predicted images of different viewpoints.

[0224] In addition, during the cross-view consistency training process, the cross-view attention mechanism described in the aforementioned embodiment can be introduced, and the cross-view attention output of the scene data of each view is calculated in the same way, and based on the cross-view attention output of each view, a predicted image of the corresponding view is generated, and the first loss function value is calculated based on the predicted image, and the network parameters of the multimodal generation network are adjusted, including the cross-view weight matrix of the cross-view attention mechanism.

[0225] S702, using a multi-frame scene data sequence, performing time consistency training on the first network until the training converges, obtaining a trained multimodal generation network, and freezing network parameters of the multimodal generation network;

[0226] A multi-frame scene data sequence refers to a data set composed of multiple continuous video frames, which is used to describe the changes of a specific scene over time. It can be used to analyze the dynamic characteristics of the scene and perform spatiotemporal modeling. There is temporal correlation and spatial consistency between multi-frame scene data, and it contains spatiotemporal information about the scene, so that the entire sequence can fully display the state or change of the scene. For example, in autonomous driving, the continuous road video captured by the on-board camera can be used as a multi-frame scene data sequence. Each frame represents the scene of the same road section at different time points. The road video frames composed of multiple frames of images jointly depict the changes of road conditions, traffic flow, etc. over time.

[0227] Spatiotemporal consistency is used to maintain the consistency of generated data in time and space, that is, changes in time frames in the scene will not change the dynamic changes of a single entity, and the semantics in a single-frame scene space will not change. Spatiotemporal consistency training refers to using multi-frame scene data sequences as training samples, training the network to learn the temporal dependencies and dynamic change rules in the scene, and learning to accurately capture the association information between frames, so that the network maintains temporal and spatial coherence and consistency when generating new frames or scenes. For example, in the task of generating future scene images, spatiotemporal consistency training can ensure that future video frames generated by the network are logically and visually coherent with previous frames in content. For example, if a pedestrian is walking forward, the next frame predicted by the network should show the pedestrian continuing to move forward, rather than suddenly appearing in other locations or performing unrelated actions.

[0228] Regarding spatiotemporal consistency training, a video frame sequence or dynamic video of the same scene changing over time can be collected as a training sample, and a second loss function is defined in the spatiotemporal consistency training process. The second loss function includes at least pixel-level reconstruction loss, cross-view consistency loss, and motion consistency loss, which are used to evaluate the difference between the multi-frame predicted image sequence generated by the first network and the real multi-frame image sequence corresponding to the second training sample. By minimizing the third loss function, the parameters of the first network can be adjusted to make the prediction result closer to the actual situation. Among them, the reconstruction loss is used to evaluate the difference between the output generated by the network and the real data, and the spatiotemporal consistency loss is used to promote the consistency and coherence of the generated results at different time steps and perspectives. The discontinuity in time and the inconsistency between perspectives can be penalized by designing a specific loss function. In each training iteration, the multi-frame data sequence is input into the first network, and the network parameters are updated by back propagation using gradient descent or its variant algorithm to minimize the loss value.

[0229] In addition, during the spatiotemporal consistency training process, the cross-frame attention mechanism described in the aforementioned embodiment can be introduced, and the cross-view attention output of each frame of scene data in the multi-frame scene data sequence is calculated using the cross-frame attention calculation method recorded in the above embodiment. Based on the cross-view attention output of each frame of scene data, a predicted image sequence is generated, and the first loss function value is calculated based on the predicted image sequence, and the network parameters of the multimodal generation network are adjusted, including the cross-frame weight matrix of the cross-frame attention mechanism.

[0230] Spatiotemporal consistency training is to further train the multimodal generative network that has completed cross-view consistency training. Therefore, when the spatiotemporal consistency training converges, the multimodal generative network that has completed the training has the ability to make the generated data consistent with cross-view consistency and spatiotemporal consistency. In order to better integrate the cross-view consistency and spatiotemporal consistency capabilities, during the spatiotemporal consistency training process, the training data can include video frame sequences from multiple different perspectives in the same time period. The video frames of each perspective at each time point are aligned in the time and space dimensions to adaptively fine-tune the learnable network parameters for cross-view consistency.

[0231] In this embodiment, when the multimodal generation network obtained after training enters the third stage of training the joint prediction fusion module, the trained network parameters of the multimodal generation network will be maintained without network parameter update during the training process.

[0232] S703, for a plurality of predefined generation tasks, the multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the training sample data corresponding to the generation tasks, and the extracted features are input into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges, so as to obtain and store the network parameters of the synthetic data generation network that meet the generation tasks after training.

[0233] In this embodiment, the multimodal large language network can be pre-trained by any method used to train the multimodal large language network in the relevant technology, and the network parameters of the trained multimodal large language network are in a frozen state during the training of the joint prediction fusion module, that is, the network parameters are kept unchanged together with the trained multimodal generation network described in step S102.

[0234] The training of the joint prediction fusion module aims to train the module to effectively fuse features from different network sources to generate synthetic data that meets the synthetic data generation task. During the training process, the joint prediction fusion module can fuse features from different network sources through simple splicing, weighted summation, attention mechanism, etc. The fused features are used to generate synthetic data. The network parameters of the joint prediction fusion module can be adjusted by calculating and minimizing the loss function of the predicted output result and the actual result corresponding to the sample demand data or the predefined evaluation standard, or by combining self-supervision with reinforcement learning.

[0235] For example, the generation task is to predict a scene image where a collision accident occurs based on an existing scene image. The training sample data may include a traffic scene image with at least two vehicles and text information describing the collision accident (such as a scene image in which a blue car collides with a white car in the prediction image). The training sample data will be simultaneously input into the trained multimodal generation network and the multimodal large language network. The output features of the two networks are further input into the joint prediction fusion module. The predicted image is output based on the fusion feature decoding process output by the module. The pixel-level loss between the predicted image and the real image corresponding to the sample demand data (i.e., the real scene image in which the blue car collides with the white car in the image) can be evaluated, or the semantic loss between the predicted image and the text information describing the collision accident in the sample demand data is evaluated, and other losses such as the motion consistency loss of the predicted image and the traffic scene image are combined, and the network parameters of the joint prediction fusion module are updated by back propagation to minimize the loss value.

[0236] The above embodiment gives the whole process of the synthetic data generation network from initialization to customized training for a specific generation task. First, through multi-view consistency training, scene data from multiple different perspectives are used to enhance the multimodal generation network's ability to understand and process data from different perspectives of the same scene, ensuring that the network can capture the key information of the scene from multiple perspectives. Then, through time consistency training, a multi-frame scene data sequence is used to train the network to understand the time evolution and dynamic changes in the scene, further improving the network's ability to process complex scene data. After obtaining the trained multimodal generation network, its network parameters are frozen to maintain its stability in subsequent training. Subsequently, for a variety of pre-defined generation tasks, the frozen multimodal generation network and the pre-trained multimodal large language network are used to extract features from the training sample data corresponding to the generation task, respectively. The two networks extract key information from the training sample data, and input their respective output features into the joint prediction fusion module. By training the joint prediction fusion module, its parameters are optimized to better fuse features from different modalities and generate synthetic data that meets the requirements of the generation task.

[0237] In the disclosed embodiment, by performing targeted customized training for the generation task, the synthetic data generation network can more accurately understand the requirements of the generation task and generate synthetic data that meets the requirements, thereby improving the practicality and effectiveness of the network. The above-mentioned multi-view consistency training and time consistency training enhance the network's ability to understand and process complex scene data, making the generated synthetic data more realistic and accurate. At the same time, the introduction of the joint prediction fusion module further optimizes the fusion effect between different modal features and improves the overall quality of the synthetic data. Moreover, through a training process that gradually increases the difficulty of the task in three stages, from cross-view consistency training to spatiotemporal consistency training, and then to joint prediction fusion training combined with a multimodal large language network, the network is fully learned at each stage, thereby gradually enhancing the ability of the synthetic data generation network, avoiding the problem of overfitting or training problems that may be caused by one-step training.

[0238] In some embodiments, after further introducing spatiotemporal consistency training on the basis of the multimodal generative network (i.e., the first network) that has been trained for cross-view consistency through the above step S702, in order to achieve scene generation controllability, the input of set conditional data is used to guide the model to generate data output under specific conditions. Scene controllability training can be introduced on the basis of spatiotemporal consistency training, so that the network can pay attention to the image style or layout desired by the user, and learn how to accurately extract semantic features aligned with the fused text or image. For this scene controllability training, a third loss function is introduced in the training process, including at least pixel-level image reconstruction loss, cross-view consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss.

[0239] Among them, style loss is used to keep the generated image consistent with the artistic style or visual atmosphere implied by the given text description, and uses the feature statistics of the pre-trained network to measure the similarity between the generated image and the style implied by the text description. Semantic consistency loss ensures that the content of the generated image is semantically consistent with the input text description. This is achieved through additional semantic labels or matching losses between the layout map and the generated image, or using distance metrics in cross-modal embedding space. Cycle consistency loss ensures cycle consistency in the process from image to text and then to image by allowing the model to generate text from images and then generate images from text, thereby improving the quality of the generated image and its match with the text description.

[0240] Furthermore, we combine the above loss functions to construct a composite loss function to balance the authenticity of the image, the correctness of the content, the conformity of the style, and the consistency with the text description. By adjusting the weights of different loss terms, we can guide the model to focus on specific goals or features when generating images.

[0241]

[0242] in, They respectively represent reconstruction loss, perspective consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss. α, β, γ, ε, ∈, ζ are hyperparameters used to adjust the influence of each loss item on the third loss function.

[0243] Based on the scene-controllable attention mechanism described in the aforementioned embodiments, the same scene-controllable attention calculation method can be used to perform scene-controllable attention output calculation on the training samples of scene controllability training, and perform a fourth loss function calculation based on the predicted image and the real image obtained based on the scene attention output. The loss function is minimized to adjust the network parameters of the multimodal generation network, including the scene-controllable weight matrix of the scene-controllable attention mechanism.

[0244] In this embodiment, the multimodal generation network undergoes cross-view consistency training, spatiotemporal consistency training, and scene controllability training in sequence, and adjusts the network parameters of the multimodal generation network through continuous iteration to minimize the total loss value of the third loss function, so that the network gradually learns how to generate diversified images or videos that meet user requirements and maintain cross-view consistency and spatiotemporal consistency based on the input set condition data.

[0245] In some embodiments, for the above step S703, the extracted features are jointly input into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges. The network parameters of the joint prediction module can be self-evolved by introducing reinforcement learning and interacting with the environment. In this way, the synthetic data generation network optimizes the behavior of the joint prediction fusion module by predicting future states, evaluating actions, and updating strategies through environmental feedback.

[0246] Based on the reinforcement learning, combined with the feature normalization and fusion weight parameters of the joint prediction fusion module described in the above embodiment, after the extracted features extracted by the pre-trained multimodal large language network and the trained multimodal generative network are input into the joint prediction module, the network parameters of the joint prediction fusion module can be updated in the following manner:

[0247] The extracted features extracted by the multimodal large language network and the multimodal generative network are respectively standardized to obtain a first extracted feature of the multimodal large language network and a second extracted feature of the trained multimodal generative network; the first extracted feature and the second extracted feature are fused through a learnable weight parameter to obtain a fused feature; and according to the fused feature, reinforcement learning is applied to the joint prediction fusion module to adjust the network parameters of the joint prediction fusion module.

[0248] After obtaining the fusion features, the synthetic data generation network W is based on the current state s t 、Current action a t And the fusion feature F final , try to predict the next state s through some form of neural network architecture such as recurrent neural network, Mamba, etc. t+1 and expected reward t+1 , fusion feature F final As the input to the agent decision to enhance its understanding and prediction ability of the environment, it can be expressed as:

[0249]

[0250] The current action refers to any decision or behavior taken by the agent given the current state and fusion features. It is not limited to control signals in the traditional physical world (such as vehicle speed and direction control), but can also include but is not limited to the prediction of video frames, the generation of future image sequences, etc. In the scenario of video prediction or future frame generation, the current action may represent a certain strategy, algorithm selection, parameter adjustment, or activation mode change of a certain layer in the network when generating future frames. The current action is used to optimize the agent's ability to predict the environment, that is, to improve the accuracy and quality of generating future frames. Then the current action can be regarded as the agent's current state s at time t. t And the fusion feature F fina A specific instance of a set of decisions or choices made.

[0251] The current state is the agent's perception or understanding of the environment at a specific point in time (t), and contains all the information the agent needs to make decisions. In this scenario, the current state can include but is not limited to the current fused features, past states (such as predictions or actual observations at previous time steps), the agent's internal state (such as network parameters, hidden layer activations, etc.), and any environmental factors that may affect future predictions. In the context of video prediction or future frame generation, the state may also contain historical information about the video sequence, detailed features of the current frame, and any external factors relevant to predicting future frames. The current action and state together define how the agent interacts with the environment and determine how the agent optimizes its prediction performance through learning.

[0252] Select action a t According to the current strategy π(a t |s t ,F final ), the strategy can be adjusted based on the fusion features and the current state; it should be noted that the action does not only refer to the signal that can be executed by the embodied carrier such as the vehicle or robot, but also includes the future evolution law in the preset world scene, that is, the signal of the future video image frame.

[0253] The next state refers to the new state of the environment after the agent performs the current action. In the context of the joint prediction fusion module, this new state may include future video frames, images, or other forms of data predicted by the synthetic data generation network W based on the current state, current action, and fused features. The predicted data represents the possible state of the environment at a certain point in the future.

[0254] The expected reward is the immediate feedback that the agent gets from the environment after performing an action, which is used to measure the quality of the action. In the scenario of the joint prediction fusion module, the expected reward may be calculated based on a variety of factors, such as the similarity between the predicted future frame and the actual observed frame, the reduction in prediction error, the quality of the generated data, etc. The expected reward is a key signal in reinforcement learning, guiding the agent to learn how to choose actions to maximize the long-term cumulative reward. During training, the agent tries to find a policy that maximizes the cumulative reward it receives after performing a series of actions.

[0255] L = UpdatePolicy((s t ,a t ,r t+1 ,s t+1 ),π)

[0256] Among them, UpdatePolicy is an update operation for the policy, which optimizes the world model by interacting with the world in real time. The performance of the current policy is evaluated by calculating the cumulative reward or loss function, and the policy is updated using reinforcement learning algorithms (such as Q learning, policy gradient methods, etc.) to select better actions in the future. Since the joint prediction fusion module is a key part of generating the next state and expected reward, its parameters are also adjusted as the policy is updated. This can be achieved through the back-propagation algorithm, where the gradient of the loss function with respect to the prediction error and reward is used to update the network weights. The above process will be repeated until the agent learns an effective strategy that maximizes the cumulative reward it obtains after performing a series of actions. At the same time, the network parameters of the joint prediction fusion module will be gradually optimized to improve prediction accuracy and overall performance.

[0257] Through the above steps, this embodiment can model the synthetic data generation network based on the feature fusion of the multimodal generation model and the multimodal large language model, combining reinforcement learning and self-supervised learning paradigms, and realize self-evolutionary learning that interacts with the environment. It can predict future scenes based on user input or scenes that have occurred, and conform to the logical laws of the physical world. Specifically, this embodiment not only utilizes the infinite spatial imagination of the world by the multimodal generation model, but also utilizes the cognitive ability of the multimodal large language network for universal things in the world, and through continuous interaction with the environment to optimize the strategy, while improving the network's prediction ability, it prevents the network from generating scenes and operations that do not conform to the physical world and logic.

[0258] Next, in order to enable those skilled in the art to better understand the synthetic data generation method provided by the present application, the synthetic data generation method is illustrated by taking some specific synthetic data generation processes in the field of autonomous driving as an example.

[0259] Taking the existing dataset reuse generation task as an example, the network parameters of the synthetic data generation network corresponding to the existing dataset reuse generation task are obtained in advance, and the synthetic data generation network is initialized using the network. Fig. 8A The scene image data Figure A1 is shown, and the geometric layout information corresponding to the scene is shown in Figure A2. When using the above-mentioned synthetic data generation method to generate synthetic data, Figures A1 and A2 can be used together as the data to be processed corresponding to the user-specified generation task, or, Figure A1 and a text description describing the modification of the control instance orientation can be used together as the data to be processed, and the data to be processed is input into the initialized synthetic data generation network to obtain Figure A3 in which the instance orientation is modified, that is, the vehicle orientation of Figures A1 and A3 is changed.

[0260] Similarly, if Figure 8B The example figures given, Figures B1 and B2 respectively represent the image data and geometric layout information of the illustrated scene. By inputting them into the initialized synthetic data generation network, we obtain Figure B3 in which the instance orientation of the synthetic data generation network output is modified, that is, the orientation of the illustrated bus changes before and after the modification.

[0261] Corresponding to the embodiment of the aforementioned synthetic data generation method, see Fig. 9 As shown, the present application also provides an embodiment of a synthetic data generation device, which is applied to an initialized synthetic data generation network; the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module; the device includes:

[0262] The to-be-processed data acquisition module 901 is used to acquire the to-be-processed data corresponding to the task generated by the user;

[0263] The coding feature input module 902 is used to input the coding features of the data to be processed into the multimodal generation network and the multimodal large language network respectively; the initialization is achieved based on the network parameters obtained by pre-training the synthetic data generation network and matching the user-specified generation task;

[0264] The feature fusion and synthetic data output module 903 is used to use the joint prediction fusion module to fuse the output features of the multimodal generation network and the multimodal large language network, and generate synthetic data according to the output fusion features.

[0265] In some embodiments, the apparatus further comprises:

[0266] In the case where the data to be processed includes user-set requirements, detecting whether the generated synthetic data meets the user-set requirements;

[0267] In the case that the synthesized data does not meet the user-set requirements, the description information of the user-set requirements is adjusted, and the step of generating the synthesized data according to the data to be processed is repeated until the generated synthesized data meets the user-set requirements or reaches a threshold of the number of repeated executions.

[0268] In some embodiments, after obtaining the to-be-processed data corresponding to the user-specified generation task, the device further includes:

[0269] Performing input modality encoding processing on the data to be processed to obtain encoding features of the data to be processed;

[0270] In the case where the data to be processed includes non-text data, converting the encoded features into feature representations that match the text modality, and adjusting the feature dimension of the feature representation to be the same as the input dimension of the multimodal large language network;

[0271] The feature representation matched with the text modality is input into the multimodal large language network.

[0272] In some embodiments, when the data to be processed includes scene data of multiple different perspectives, the coding feature input module is specifically used to:

[0273] For the coding features corresponding to the scene data of each perspective, using the multimodal generation network, the coding features of each perspective are used as a first query vector, and the connection of the coding features of two adjacent perspectives is used as a first key vector and a first value vector, and the first query vector, the first key vector and the first value vector are linearly transformed through a cross-perspective weight matrix to obtain a cross-perspective attention output of each perspective; the scene data includes image data or description data of the same scene;

[0274] According to the cross-view attention output of each view, the output features of the multimodal generation network are constructed.

[0275] In some embodiments, when the data to be processed includes a multi-frame scene data sequence, the coding feature input module is specifically used to:

[0276] For the coding features of each frame of scene data, using the multimodal generation network, the coding features of the target frame are used as the second query vector, the connection of the coding features of the first frame in the multi-frame scene data sequence and the next frame of the target frame is used as the second key vector, and the connection of the coding features of the first frame and the previous frame of the target frame is used as the second value vector; the target frame refers to any frame in the multi-frame scene data sequence;

[0277] Performing a linear transformation on the second query vector, the second key vector, and the second value vector through a cross-frame weight matrix to obtain a cross-frame attention output of the target frame;

[0278] According to the cross-frame attention output of each frame of scene data, the output features of the multimodal generation network are constructed.

[0279] In some embodiments, when the data to be processed includes image type data, the encoding feature input module is specifically used to:

[0280] Inputting the data to be processed together with the encoding features into the multimodal generation network;

[0281] Using the multimodal generative network, obtaining first image features of a first subregion where each object instance in the data to be processed is located, and adding a learnable category hint of the object instance to the first image features to obtain second image features;

[0282] Acquire a third image feature of a second sub-region that does not contain any object instance in the data to be processed, and add a learnable empty instance prompt to the third image feature to obtain a fourth image feature;

[0283] Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene controllable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0284] The self-attention outputs of the first sub-region and the second sub-region are fused with output features of other feature extraction layers of the multimodal generation network to construct output features of the multimodal generation network.

[0285] In some embodiments, the feature fusion and synthesis data output module is specifically used to:

[0286] Using the joint prediction fusion module, the output features of the multimodal generation network and the multimodal large language network are respectively standardized to obtain a first output feature of the multimodal generation network and a second output feature of the multimodal large language network;

[0287] The first output feature and the second output feature are fused by using a fusion weight parameter to obtain the fused feature.

[0288] In some embodiments, after the normalization process, the apparatus further comprises a cross attention process step:

[0289] Using the first output feature as a third query vector, and using the second output feature as a third key vector and a third value vector;

[0290] Applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector to obtain a cross-attention output of the second output feature;

[0291] The fusing the first output feature with the second output feature by using a fusion weight parameter includes:

[0292] The cross-attention output of the first output feature and the second output feature is fused through the fusion weight parameter.

[0293] In some embodiments, the synthetic data generation network further includes a decoder; the feature fusion and synthetic data output module is specifically used for:

[0294] Determining a target decoder for implementing the task specified by the user from among a plurality of decoders preset for implementing different tasks;

[0295] The target decoder is used to decode the fused features to obtain the synthesized data.

[0296] In some embodiments, the network parameters obtained by pre-training the synthetic data generation network correspond one-to-one to the predefined generation task; the device also includes the step of training the network parameters:

[0297] Using scene data from multiple different perspectives, the multi-view consistency training is performed on the multimodal generation network until the training converges to obtain a first network; the scene data at least includes image data or description data of the same scene;

[0298] Using a multi-frame scene data sequence, performing time consistency training on the first network until the training converges to obtain a trained multimodal generation network, and freezing the network parameters of the multimodal generation network;

[0299] For a plurality of predefined generation tasks, the multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the training sample data corresponding to the generation tasks, and the extracted features are input into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges, so as to obtain and store the network parameters of the synthetic data generation network that meet the generation tasks after training.

[0300] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0301] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.10As shown, the electronic device 1000 includes at least one processor 1001, a memory 1002 and a bus 1003, and the at least one processor 1001 is electrically connected to the memory 1002; the memory 1002 is configured to store at least one computer-executable instruction, and the processor 1001 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any synthetic data generation method provided in any embodiment or any optional implementation manner in the present application.

[0302] Furthermore, the processor 1001 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0303] An embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any synthetic data generation method provided in any embodiment or any optional implementation manner in the present application when executed by a processor.

[0304] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0305] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating synthetic data, characterized in that: Applied to the initialized synthetic data generation network; the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module; the method includes: Get the data to be processed corresponding to the task generated by the user; Inputting the encoded features of the data to be processed into the multimodal generation network and the multimodal large language network respectively; the initialization is achieved based on the network parameters obtained by pre-training the synthetic data generation network and matching the user-specified generation task; The joint prediction fusion module is used to fuse the output features of the multimodal generation network and the multimodal large language network, and synthetic data is generated according to the output fusion features.

2. The method according to claim 1, characterized in that The method further comprises: In the case where the data to be processed includes user-set requirements, detecting whether the generated synthetic data meets the user-set requirements; In the case that the synthesized data does not meet the user-set requirements, the description information of the user-set requirements is adjusted, and the step of generating the synthesized data according to the data to be processed is repeated until the generated synthesized data meets the user-set requirements or reaches a threshold of the number of repeated executions.

3. The method according to claim 1, characterized in that: After obtaining the to-be-processed data corresponding to the user-specified generation task, the method further includes: Performing input modality encoding processing on the data to be processed to obtain encoding features of the data to be processed; In the case where the data to be processed includes non-text data, converting the encoded features into feature representations that match the text modality, and adjusting the feature dimension of the feature representation to be the same as the input dimension of the multimodal large language network; The feature representation matched with the text modality is input into the multimodal large language network.

4. The method according to claim 1, characterized in that In the case where the data to be processed includes scene data of multiple different perspectives, inputting the encoded features of the data to be processed into the multimodal generation network includes: For the coding features corresponding to the scene data of each perspective, using the multimodal generation network, the coding features of each perspective are used as a first query vector, and the connection of the coding features of two adjacent perspectives is used as a first key vector and a first value vector, and the first query vector, the first key vector and the first value vector are linearly transformed through a cross-perspective weight matrix to obtain a cross-perspective attention output of each perspective; the scene data includes image data or description data of the same scene; According to the cross-view attention output of each view, the output features of the multimodal generation network are constructed.

5. The method according to claim 1, characterized in that In the case where the data to be processed includes a multi-frame scene data sequence, the step of inputting the encoded features of the data to be processed into the multimodal generation network comprises: For the coding features of each frame of scene data, using the multimodal generation network, the coding features of the target frame are used as the second query vector, the connection of the coding features of the first frame in the multi-frame scene data sequence and the next frame of the target frame is used as the second key vector, and the connection of the coding features of the first frame and the previous frame of the target frame is used as the second value vector; the target frame refers to any frame in the multi-frame scene data sequence; Performing a linear transformation on the second query vector, the second key vector, and the second value vector through a cross-frame weight matrix to obtain a cross-frame attention output of the target frame; According to the cross-frame attention output of each frame of scene data, the output features of the multimodal generation network are constructed.

6. The method according to claim 1, characterized in that In the case where the data to be processed includes image type data, the step of inputting the encoding features of the data to be processed into the multimodal generation network comprises: Inputting the data to be processed together with the encoding features into the multimodal generation network; Using the multimodal generative network, obtaining first image features of a first subregion where each object instance in the data to be processed is located, and adding a learnable category hint of the object instance to the first image features to obtain second image features; Acquire a third image feature of a second sub-region that does not contain any object instance in the data to be processed, and add a learnable empty instance prompt to the third image feature to obtain a fourth image feature; Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene controllable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region; The self-attention outputs of the first sub-region and the second sub-region are fused with output features of other feature extraction layers of the multimodal generation network to construct output features of the multimodal generation network.

7. The method according to claim 1, characterized in that The step of fusing the output features of the multimodal generation network and the multimodal large language network using the joint prediction fusion module includes: Using the joint prediction fusion module, the output features of the multimodal generation network and the multimodal large language network are respectively standardized to obtain a first output feature of the multimodal generation network and a second output feature of the multimodal large language network; The first output feature and the second output feature are fused by using a fusion weight parameter to obtain the fused feature.

8. The method according to claim 7, characterized in that After the normalization process, the method further comprises a cross attention process step: Using the first output feature as a third query vector, and using the second output feature as a third key vector and a third value vector; Applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector to obtain a cross-attention output of the second output feature; The fusing the first output feature with the second output feature by using a fusion weight parameter includes: The cross-attention output of the first output feature and the second output feature is fused through the fusion weight parameter.

9. The method according to claim 1, characterized in that: The synthetic data generation network also includes a decoder; the generation of synthetic data according to the fusion features output by the decoder includes: Determining a target decoder for implementing the task specified by the user from among a plurality of decoders preset for implementing different tasks; The target decoder is used to decode the fused features to obtain the synthesized data.

10. The method according to claim 1, characterized in that The network parameters obtained by pre-training the synthetic data generation network correspond one-to-one to the predefined generation task; the method also includes the step of training the network parameters: Using scene data from multiple different perspectives, the multi-view consistency training is performed on the multimodal generation network until the training converges to obtain a first network; the scene data at least includes image data or description data of the same scene; Using a multi-frame scene data sequence, performing time consistency training on the first network until the training converges to obtain a trained multimodal generation network, and freezing the network parameters of the multimodal generation network; For a plurality of predefined generation tasks, the multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the training sample data corresponding to the generation tasks, and the extracted features are input into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges, so as to obtain and store the network parameters of the synthetic data generation network that meet the generation tasks after training.

11. A synthetic data generating device, characterized in that: Applied to the initialized synthetic data generation network; the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module; the device includes: The module for obtaining data to be processed is used to obtain the data to be processed corresponding to the task generated by the user; A coding feature input module, used to input the coding features of the data to be processed into the multimodal generation network and the multimodal large language network respectively; the initialization is achieved based on the network parameters obtained by pre-training the synthetic data generation network and matching the user-specified generation task; The feature fusion and synthetic data output module is used to use the joint prediction fusion module to fuse the output features of the multimodal generation network and the multimodal large language network, and generate synthetic data according to the output fusion features.

12. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-10.

13. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images