Training method and device of synthetic data generation network, equipment and medium

Abstract

The invention provides a training method, device and equipment for a synthetic data generation network and a medium, and the method comprises the steps: carrying out the training process of gradually increasing the task difficulty in three stages from the cross-view consistency training to the space-time consistency training, and then carrying out the combined prediction fusion training in combination with a multi-modal large language network. The network is enabled to fully learn at each stage, so that the capability of the synthetic data generation network is gradually enhanced, the problem of overfitting or training possibly caused by one-step training is avoided, and the synthetic data generation network can fuse multi-modal information such as images and texts. More real and diversified synthetic data with dynamic prediction capability are generated, and the training efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a training method, device, equipment and medium for a synthetic data generation network. Background Art

[0002] At present, the training data required for the development of the autonomous driving field and embodied intelligence industry usually comes from real data collected from real physical environments and synthetic data of virtual environments generated by data twin simulation technology or machine learning. Among them, the collection cost of real data is high and is limited by actual conditions. It is difficult to effectively collect challenging scenarios, long-tail distributions or rare scenarios. The synthetic data generated by data twin simulation technology or machine learning can make up for the problem of insufficient real data to a certain extent.

[0003] However, the neural networks currently used to generate synthetic data are usually trained end-to-end. The synthetic data generated by the neural networks trained in this way have problems such as insufficient realism of the generated scenes, poor diversity and generalization. For example, it is difficult to accurately simulate extreme events such as collision accidents and extreme weather conditions. Summary of the invention

[0004] In view of this, in order to solve the above technical problems, the present application provides a training method, device, equipment and medium for a synthetic data generation network.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a training method for a synthetic data generation network is provided, wherein the synthetic data generation network comprises at least a multimodal generation network, a multimodal large language network, and a joint prediction fusion module, wherein the joint prediction fusion module is used to fuse output features of the multimodal generation network and the multimodal large language network; the method comprises:

[0007] Using scene data from multiple different perspectives, the multimodal generative network is trained for cross-perspective consistency until the training converges to obtain a first network; the scene data includes image data or description data of the same scene;

[0008] Using a multi-frame scene data sequence, performing spatiotemporal consistency training on the first network until the training converges to obtain a trained multimodal generation network, and freezing the network parameters of the multimodal generation network;

[0009] According to the sample demand data corresponding to different synthetic data generation tasks, the trained multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the sample demand data respectively, and the extracted features are input into the joint prediction fusion module together, and the joint prediction fusion module is trained until the training converges to obtain a trained synthetic data generation network.

[0010] Optionally, the using scene data of multiple different perspectives to perform cross-perspective consistency training on the multimodal generation network until the training converges to obtain a first network includes:

[0011] Acquire first training samples, each of which includes at least single-frame scene data of the same scene collected from multiple perspectives;

[0012] Inputting the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network;

[0013] According to the predicted images of the multiple corresponding perspectives and the corresponding first training samples, a first loss function is calculated, and the network parameters of the multimodal generation network are adjusted by minimizing the first loss function; the first loss function includes at least a loss within a single perspective and a consistency loss across perspectives.

[0014] Optionally, the step of inputting the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding viewing angles generated by the multimodal generation network includes:

[0015] Utilizing the multimodal generative network, respectively extracting potential features corresponding to the single-frame scene data collected from each viewing angle;

[0016] The latent feature of each view is used as the first query vector, and the connection of the latent features of its two adjacent views is used as the first key vector and the first value vector. The cross-view attention output of each view is obtained by performing linear transformation through the cross-view learnable matrix;

[0017] According to the cross-view attention output of each view, the predicted image of the corresponding view is generated.

[0018] Optionally, generating a predicted image of a corresponding perspective according to the cross-perspective attention output of each perspective includes:

[0019] According to the predicted images of the i-1 perspectives that have been generated, the cross-perspective joint distribution conditional probability of the i-th perspective is obtained; i is a positive integer greater than 1;

[0020] According to the predicted image of the i-1 perspective and the cross-perspective attention output of the i-th perspective, combined with the cross-perspective joint distribution conditional probability, the predicted image of the i-th perspective is generated.

[0021] Optionally, before inputting the first training sample into the multimodal generation network, the method further includes:

[0022] Obtaining single-view scene data and inputting it into the multimodal generation network for image generation task training, and adjusting the network parameters of the multimodal generation network by minimizing the second loss function;

[0023] When the single-view scene data is image type data, the second loss function includes at least pixel-level reconstruction loss; when the single-view scene data is text type data, the second loss function includes at least semantic loss.

[0024] Optionally, the using of a multi-frame scene data sequence to perform spatiotemporal consistency training on the first network until the training converges to obtain a trained multimodal generation network includes:

[0025] Acquire second training samples, each second training sample at least comprising an input sequence consisting of multiple frames of scene data of the same scene that continuously changes over time;

[0026] Inputting the second training sample into the first network, and obtaining a multi-frame predicted image sequence generated by the first network and conforming to the task type based on the task type set for each second training sample;

[0027] According to the multi-frame predicted image sequence and the multi-frame real image sequence corresponding to the second training sample, a third loss function is calculated, and the network parameters of the first network are adjusted by minimizing the third loss function; the third loss function includes at least pixel-level reconstruction loss, cross-view consistency loss and motion consistency loss.

[0028] Optionally, obtaining a multi-frame predicted image sequence generated by the first network and conforming to the task type includes:

[0029] Using the first network, respectively extracting potential features of each frame of scene data in the input sequence;

[0030] The latent features of the target frame scene data are used as a second query vector, the connection of the first frame scene data in the input sequence and the latent features of the next frame of the target frame scene data is used as a second key vector, and the connection of the first frame scene data and the latent features of the previous frame of the target frame scene data is used as a second value vector; the target frame scene data refers to any frame scene data in the input sequence;

[0031] Linearly transforming the second query vector, the second key vector, and the second value vector through a cross-frame learnable matrix to obtain a cross-frame attention output of the target frame scene data;

[0032] The multi-frame predicted image sequence is generated according to the cross-frame attention output of each frame of scene data in the input sequence.

[0033] Optionally, generating the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence includes:

[0034] In the case of generating a future multi-frame prediction image sequence according to the second training sample, obtaining an inter-frame joint distribution conditional probability of a j-th prediction image to be generated according to the input sequence and the j-1-th prediction image that has been generated; wherein j is a positive integer greater than 1;

[0035] According to the cross-frame attention output of each frame of scene data and combined with the inter-frame joint distribution conditional probability, the j-th frame prediction image is generated in an autoregressive manner.

[0036] Optionally, after completing the spatiotemporal consistency training of the first network, the method further includes:

[0037] Acquire setting condition data for guiding the output of the image generation task of the second network; the second network refers to a network obtained by completing the spatiotemporal consistency training of the first network; the setting condition data includes any one of text data describing the setting scene, or text and image combined data;

[0038] The second network is trained for controllability of image generation scenarios using the set condition data, and the network parameters of the second network are adjusted by minimizing a fourth loss function to obtain the trained multimodal generation network; the loss terms of the fourth loss function include at least pixel-level image reconstruction loss, cross-view consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss.

[0039] Optionally, the using the set condition data to perform image generation scene controllability training on the second network includes:

[0040] extracting initial image features corresponding to the set condition data using the second network;

[0041] Acquire a first image feature of a first subregion where each object instance in the initial image feature is located, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0042] Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0043] Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene-controllable and learnable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0044] Obtain a predicted scene image generated by the second network based on the self-attention output of the first sub-region and the second sub-region, compare the predicted scene image with the set scene corresponding to the set condition data to calculate a fourth loss function, and minimize the fourth loss function to adjust the network parameters of the second network to complete the controllability training of the image generation scene.

[0045] Optionally, the extracting features are inputted into the joint prediction fusion module, and the joint prediction fusion module is trained until the training converges, including:

[0046] Standardizing the extracted features extracted by the multimodal large language network and the multimodal generative network respectively to obtain a first extracted feature of the multimodal large language network and a second extracted feature of the trained multimodal generative network;

[0047] Fusing the first extracted feature with the second extracted feature through a learnable weight parameter to obtain a fused feature;

[0048] According to the fusion features, reinforcement learning is applied to the joint prediction fusion module to adjust network parameters of the joint prediction fusion module.

[0049] Optionally, after the extracted features are respectively standardized, the method further includes:

[0050] Using the first extracted features as a third key vector and a third value vector, and using the second extracted features as a third query vector;

[0051] Obtaining a cross-attention output of the first extracted features by applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector;

[0052] The fusing the first extracted feature with the second extracted feature by using a learnable weight parameter comprises:

[0053] The cross-attention output is fused with the second extracted features through a learnable weight parameter.

[0054] According to a second aspect of an embodiment of the present application, a training device for a synthetic data generation network is provided, wherein the synthetic data generation network comprises at least a multimodal generation network, a multimodal large language network, and a joint prediction fusion module; the joint prediction fusion module is used to fuse the output features of the multimodal generation network and the multimodal large language network; the device comprises:

[0055] A first training module is used to perform cross-view consistency training on the multimodal generative network using scene data of multiple different viewpoints until the training converges to obtain a first network; the scene data at least includes description data of geometric layout information of the same scene;

[0056] A second training module is used to perform spatiotemporal consistency training on the first network using a multi-frame scene data sequence until the training converges, thereby obtaining a trained multimodal generation network, and freezing network parameters of the multimodal generation network;

[0057] The third training module is used to obtain sample demand data corresponding to different synthetic data generation task requirements, and use the trained multimodal generation network and the pre-trained multimodal large language network in a frozen state to extract features from the sample demand data respectively, and input the extracted features into the joint prediction fusion module, perform feature fusion training on the joint prediction fusion module until the training converges, and obtain a trained synthetic data generation network.

[0058] Optionally, the first training module is specifically used for:

[0059] Acquire first training samples, each of which includes at least single-frame scene data of the same scene collected from multiple perspectives;

[0060] Inputting the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network;

[0061] According to the predicted images of the multiple corresponding perspectives and the corresponding first training samples, a first loss function is calculated, and the network parameters of the multimodal generation network are adjusted by minimizing the first loss function; the first loss function includes at least a loss within a single perspective and a consistency loss across perspectives.

[0062] Optionally, when the first training module is used to input the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network, the first training module includes:

[0063] Utilizing the multimodal generative network, respectively extracting potential features corresponding to the single-frame scene data collected from each viewing angle;

[0064] The latent feature of each view is used as the first query vector, and the connection of the latent features of its two adjacent views is used as the first key vector and the first value vector. The cross-view attention output of each view is obtained by performing linear transformation through the cross-view learnable matrix;

[0065] According to the cross-view attention output of each view, the predicted image of the corresponding view is generated.

[0066] Optionally, the first training module, when used to generate a predicted image of a corresponding perspective according to the cross-perspective attention output of each perspective, includes:

[0067] According to the predicted images of the i-1 perspectives that have been generated, the cross-perspective joint distribution conditional probability of the i-th perspective is obtained; i is a positive integer greater than 1;

[0068] According to the predicted image of the i-1 perspective and the cross-perspective attention output of the i-th perspective, combined with the cross-perspective joint distribution conditional probability, the predicted image of the i-th perspective is generated.

[0069] Optionally, before the first training module inputs the first training sample into the multimodal generation network, the device further includes:

[0070] Obtaining single-view scene data and inputting it into the multimodal generation network for image generation task training, and adjusting the network parameters of the multimodal generation network by minimizing the second loss function;

[0071] When the single-view scene data is image type data, the second loss function includes at least pixel-level reconstruction loss; when the single-view scene data is text type data, the second loss function includes at least semantic loss.

[0072] Optionally, the second training module is specifically used for:

[0073] Acquire second training samples, each second training sample at least comprising an input sequence consisting of multiple frames of scene data of the same scene that continuously changes over time;

[0074] Inputting the second training sample into the first network, and obtaining a multi-frame predicted image sequence generated by the first network and conforming to the task type based on the task type set for each second training sample;

[0075] According to the multi-frame predicted image sequence and the multi-frame real image sequence corresponding to the second training sample, a third loss function is calculated, and the network parameters of the first network are adjusted by minimizing the third loss function; the third loss function includes at least pixel-level reconstruction loss, cross-view consistency loss and motion consistency loss.

[0076] Optionally, when the second training module is used to obtain a multi-frame predicted image sequence generated by the first network and conforming to the task type, it includes:

[0077] Using the first network, respectively extracting potential features of each frame of scene data in the input sequence;

[0078] The latent features of the target frame scene data are used as a second query vector, the connection of the first frame scene data in the input sequence and the latent features of the next frame of the target frame scene data is used as a second key vector, and the connection of the first frame scene data and the latent features of the previous frame of the target frame scene data is used as a second value vector; the target frame scene data refers to any frame scene data in the input sequence;

[0079] Linearly transforming the second query vector, the second key vector, and the second value vector through a cross-frame learnable matrix to obtain a cross-frame attention output of the target frame scene data;

[0080] The multi-frame predicted image sequence is generated according to the cross-frame attention output of each frame of scene data in the input sequence.

[0081] Optionally, the second training module is used to generate the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence, including:

[0082] In the case of generating a future multi-frame prediction image sequence according to the second training sample, obtaining an inter-frame joint distribution conditional probability of a j-th prediction image to be generated according to the input sequence and the j-1-th prediction image that has been generated; wherein j is a positive integer greater than 1;

[0083] According to the cross-frame attention output of each frame of scene data and combined with the inter-frame joint distribution conditional probability, the j-th frame prediction image is generated in an autoregressive manner.

[0084] Optionally, after completing the spatiotemporal consistency training of the first network, the apparatus further includes:

[0085] A scene controllable training module is used to obtain set condition data for guiding the output of an image generation task of a second network; the second network refers to a network obtained by completing spatiotemporal consistency training on the first network; the set condition data includes text data describing a set scene, or any one of text and image combination data; the second network is trained for scene controllability of image generation using the set condition data, and the network parameters of the second network are adjusted by minimizing a fourth loss function to obtain the trained multimodal generation network; the loss terms of the fourth loss function include at least pixel-level image reconstruction loss, cross-view consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss.

[0086] Optionally, when the scene controllability training module is used to perform image generation scene controllability training on the second network using the set condition data, it includes:

[0087] extracting initial image features corresponding to the set condition data using the second network;

[0088] Acquire a first image feature of a first subregion where each object instance in the initial image feature is located, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0089] Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0090] Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene-controllable and learnable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0091] Obtain a predicted scene image generated by the second network based on the self-attention output of the first sub-region and the second sub-region, compare the predicted scene image with the set scene corresponding to the set condition data to calculate a fourth loss function, and minimize the fourth loss function to adjust the network parameters of the second network to complete the controllability training of the image generation scene.

[0092] Optionally, the third training module is specifically used for:

[0093] Standardizing the extracted features extracted by the multimodal large language network and the multimodal generative network respectively to obtain a first extracted feature of the multimodal large language network and a second extracted feature of the trained multimodal generative network;

[0094] Fusing the first extracted feature with the second extracted feature through a learnable weight parameter to obtain a fused feature;

[0095] According to the fusion features, reinforcement learning is applied to the joint prediction fusion module to adjust network parameters of the joint prediction fusion module.

[0096] Optionally, after the second training module performs standardization processing on the extracted features respectively, the device further comprises:

[0097] Using the first extracted features as a third key vector and a third value vector, and using the second extracted features as a third query vector;

[0098] Obtaining a cross-attention output of the first extracted features by applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector;

[0099] The fusing the first extracted feature with the second extracted feature by using a learnable weight parameter comprises:

[0100] The cross-attention output is fused with the second extracted features through a learnable weight parameter.

[0101] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned training method for the synthetic data generation network by calling the computer program.

[0102] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the training method of the synthetic data generation network is implemented.

[0103] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0104] This embodiment proposes a phased and modular training method for a synthetic data generation network. Through a three-stage training process that gradually increases the difficulty of the task, from cross-view consistency training to spatiotemporal consistency training, and then to joint prediction fusion training combined with a multimodal large language network, the network is fully learned at each stage, thereby gradually enhancing the ability of the synthetic data generation network, avoiding the problem of overfitting or training problems that may be caused by one-step training. The multimodal generation network can generate synthetic data with multi-view consistency and time continuity through cross-view and spatiotemporal consistency training, and enhances the ability of the synthetic data generation network to handle dynamic multi-view scenes. Combined with the training of the multimodal large language network through the joint prediction fusion module, the synthetic data generation network can fuse multimodal information such as images and texts, improve the network's ability to predict future scenes, and enable the network to generate more realistic, diverse and dynamically predictable synthetic data, thereby improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1A is a schematic diagram of a structure of a synthetic data generation network shown in an exemplary embodiment of the present application;

[0106] Figure 1B It is a flowchart of a training method for a synthetic data generation network shown in an exemplary embodiment of the present application;

[0107] Figure 2A is a schematic diagram of the structure of another synthetic data generation network shown in an exemplary embodiment of the present application;

[0108] Figure 2B is a flowchart of a step for implementing cross-view consistency training shown in an exemplary embodiment of the present application;

[0109] Figure 3A is a schematic diagram of the structure of another synthetic data generation network shown in an exemplary embodiment of the present application;

[0110] Figure 3B is a flowchart of a step for implementing spatiotemporal consistency training shown in an exemplary embodiment of the present application;

[0111] Figure 4 It is a structural schematic diagram of a training device for a synthetic data generation network shown in an exemplary embodiment of the present application;

[0112] Figure 5 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0113] Synthetic data is data created in the digital world that imitates real-world data, rather than being collected or measured from the real world. It simulates the data distribution and patterns in the real world and is created by computational algorithms and simulations based on generative artificial intelligence technology. It has the same mathematical properties as the actual data on which it is based but does not contain the same information. It can mathematically or statistically reflect the properties of real-world data and can be used as a substitute for real-world data to train machine learning models, test the performance of algorithms in different scenarios, and provide large amounts of data while protecting personal privacy. The generation of synthetic data can be based on statistical models, deep learning techniques (such as generative adversarial networks (GANs), or other computational methods to ensure that the generated data has similar properties to real data.

[0114] At present, the development of autonomous driving and embodied intelligence relies on large computing power and big data, which are the data sets required for training and testing. At present, this data set is usually obtained in the following two ways: one is real data collected through real physical environments, and the other is synthetic data generated by virtual environments built using data twin simulation technology or machine learning. Real data can reflect the actual traffic environment, but because effective scenarios such as long-tail scenarios or challenging scenarios are difficult to capture, the amount of available data in real data is often insufficient relative to the amount of data required for development, and the cost of collecting such data is high and often limited by actual conditions, making it difficult to popularize widely. In contrast, the synthetic data generated by data twin simulation technology has effectively alleviated the problem of insufficient real data to a certain extent.

[0115] The current neural network used to generate synthetic data usually adopts an end-to-end training method. The synthetic data generated by the neural network trained by this training method has the problems of insufficient realism of the generated scene, poor diversity and generalization, for example, it is difficult to accurately simulate extreme events such as collision accidents and extreme weather conditions. Therefore, the present application provides a training method for a synthetic data generation network, which uses a three-stage training method. It uses an existing pre-trained multimodal generative model and a large language model base. In the first stage of training, through the pre-training of multi-view image generation tasks, the attention mechanism is integrated into the multi-view task, promoting cross-view feature fusion and complementary information utilization, and strengthening the model's understanding of the scene structure; in the second stage of training, the training is deepened to end-to-end joint optimization, which not only consolidates the image generation quality, but also introduces a spatiotemporal attention layer to capture dynamic timing information, ensuring the consistency and spatiotemporal continuity of the generated sequence under multiple perspectives; in the third stage of training, it is built on the results of the first two stages, focusing on the synergy of the multimodal generation model and the large language model, aiming to give the model the ability to predict future scenes.

[0116] The training method for the synthetic data generation network provided in the present application can be performed by any electronic device that supports running software or neural networks. The method is suitable for network training scenarios that generate training data required for development and research in the field of autonomous driving or other embodied intelligence fields such as intelligent robots.

[0117] See also Figure 1A The structural block diagram of the synthetic data generation network shown in the figure includes at least a multimodal generation network, a multimodal large language network and a joint prediction fusion module. The multimodal generation network is a neural network that can process and generate data from different modalities such as text, images, audio, etc. By learning the intrinsic connection between different modal data, it can generate data from one modality to another modality, or generate data from multiple modalities at the same time. The multimodal large language network refers to a large-scale pre-trained language model that can understand and process natural language text and interact with other modalities (such as images and videos). It has a strong text understanding ability, can extract information from other modalities, and perform cross-modal fusion and reasoning.

[0118] As shown in the figure, the multimodal generation network and the multimodal large language network respectively receive input information from the input layer and perform feature processing respectively. The outputs of the two networks are input into the joint prediction fusion module, which is used to fuse the output features of the multimodal generation network and the multimodal large language network. The obtained fused features are passed to the subsequent layers of the synthetic data generation network, such as the decoder module, for further feature processing to generate synthetic data.

[0119] It is understandable that the synthetic data generation network may also include other network structures required to generate synthetic data, such as Figure 1A The input modality encoding module, modality feature alignment module and output modality decoding module are shown as examples. The input modality encoding module is used to perform feature encoding processing on the input information input to the synthetic data generation network. The encoded features are respectively input to the multimodal generation network and the modality feature alignment module. The modality feature alignment module is used to convert the received features into features that match the text modality. The output modality decoding module is used to decode the received fusion features to generate synthetic data. In the present application, the synthetic data generation network can adopt any other network structure adapted to the multimodal generation network and the multimodal large language network to generate synthetic data, and the present application does not limit this.

[0120] Based on the multimodal generation network, multimodal large language network and joint prediction fusion module included in the synthetic data generation network, see Figure 1B As shown, the training method of the synthetic data generation network provided in the present application may at least include the following steps:

[0121] S101, using scene data of multiple different perspectives, performing cross-perspective consistency training on the multimodal generative network until the training converges, to obtain a first network; the scene data includes image data or description data of the same scene;

[0122] Scene data refers to information that describes or represents a specific environment or scene. For example, in a smart home scenario, scene data may include images of indoor environments, such as images of living rooms and bedrooms. In this embodiment, the scene data may include image data or description data. Among them, image data refers to two-dimensional or three-dimensional visual information captured by an image sensor (such as a camera), stored in the form of a pixel matrix, and contains visual features such as color, brightness, and texture. Description data refers to text information that describes an object, event, or scene in natural language or other forms. For example, description data may include a text describing the content of the scene, such as a scene description of "a black and white dog chasing a frisbee in the park", and an image of the scene captured by a camera or other image acquisition device can be used as image data of the scene.

[0123] Scene data from multiple different perspectives refers to static scene information of the same scene captured from multiple different perspectives at the same time. This scene information is used to describe the scene content such as objects, layout, relationships, and scene status in the scene at a specific time point. The scene data from multiple different perspectives can be collected by multiple different sensors or image acquisition devices. Each perspective provides different side information of the scene, which can be used to extract higher-level semantic and structural information, such as object category, position, orientation, relationship, and other scene space geometric layout information, to help the multimodal generation network better understand the scene.

[0124] Cross-view consistency is used to describe the consistency of the same object across viewpoints captured by multiple cameras arranged on embodied carriers such as vehicles or robots. Cross-view consistency training refers to improving the generalization and robustness of the network by ensuring that the network can generate consistent outputs at different viewpoints when training a multimodal generative network. Through cross-view consistency training, the data generated by the network can have consistent semantics and structural information at different viewpoints, reducing ambiguity and errors caused by changes in viewpoints, thereby improving the network's ability to understand and process complex scenes, enabling it to better adapt to the changing environment in practical applications. For example, in an autonomous driving scenario, image data from different cameras in the front, back, left, and right of the vehicle can be used to train a multimodal generative network to ensure that no matter where the vehicle is, the network can accurately identify and generate corresponding road, pedestrian and other obstacle information.

[0125] Regarding cross-view consistency training, we can collect scene data pairs from multiple viewpoints as sample data, introduce a cross-view consistency loss function in the cross-view consistency training process, evaluate the consistency between the generated data and the sample data of the multimodal generative network at different viewpoints, and use optimization algorithms such as stochastic gradient descent or its variants to back-propagate the gradient and update the network parameters according to the loss function. In addition, we can also use the complementarity of multi-view data to design a cross-view feature fusion mechanism to integrate and optimize feature information from different viewpoints, such as designing a reasonable cross-attention to fuse feature information from different viewpoints, and learn consistent feature representations from scene data from different viewpoints.

[0126] S102, using a multi-frame scene data sequence, performing spatiotemporal consistency training on the first network until the training converges, obtaining a trained multimodal generation network, and freezing network parameters of the multimodal generation network;

[0127] A multi-frame scene data sequence refers to a data set composed of multiple continuous video frames, which is used to describe the changes of a specific scene over time. It can be used to analyze the dynamic characteristics of the scene and perform spatiotemporal modeling. There is temporal correlation and spatial consistency between multi-frame scene data, and it contains spatiotemporal information about the scene, so that the entire sequence can fully display the state or change of the scene. For example, in autonomous driving, the continuous road video captured by the on-board camera can be used as a multi-frame scene data sequence. Each frame represents the scene of the same road section at different time points. The road video frames composed of multiple frames of images jointly depict the changes of road conditions, traffic flow, etc. over time.

[0128] Spatiotemporal consistency is used to maintain the consistency of generated data in time and space, that is, changes in time frames in the scene will not change the dynamic changes of a single entity, and the semantics in a single-frame scene space will not change. Spatiotemporal consistency training refers to using multi-frame scene data sequences as training samples, training the network to learn the temporal dependencies and dynamic change rules in the scene, and learning to accurately capture the association information between frames, so that the network maintains temporal and spatial coherence and consistency when generating new frames or scenes. For example, in the task of generating future scene images, spatiotemporal consistency training can ensure that future video frames generated by the network are logically and visually coherent with previous frames in content. For example, if a pedestrian is walking forward, the next frame predicted by the network should show the pedestrian continuing to move forward, rather than suddenly appearing in other locations or performing unrelated actions.

[0129] Regarding spatiotemporal consistency training, we can collect video frame sequences or dynamic videos of the same scene changing over time as training samples, and introduce reconstruction loss and spatiotemporal consistency loss as loss functions during spatiotemporal consistency training. The reconstruction loss is used to evaluate the difference between the output generated by the network and the real data, and the spatiotemporal consistency loss is used to promote the consistency and coherence of the generated results at different time steps and perspectives. We can design a specific loss function to penalize the discontinuity in time and the inconsistency between perspectives. In each training iteration, a multi-frame data sequence is input into the first network, and the network parameters are updated by backpropagation using gradient descent or its variant algorithm to minimize the loss value.

[0130] In addition, the complementarity of multi-view data can be used to design a cross-view feature fusion mechanism to integrate and optimize feature information from different viewpoints, such as designing a reasonable cross-attention to fuse feature information from different viewpoints and learn consistent feature representations from scene data from different viewpoints. In addition, based on the first network, a layer for simultaneously processing the temporal and spatial dimensions, or an attention mechanism that focuses on the temporal and spatial related information of the inter-frame changes of a multi-frame scene data sequence can be introduced to accurately capture the temporal and spatial consistency.

[0131] Spatiotemporal consistency training is to further train the multimodal generative network that has completed cross-view consistency training. Therefore, when the spatiotemporal consistency training converges, the multimodal generative network that has completed the training has the ability to make the generated data consistent with cross-view consistency and spatiotemporal consistency. In order to better integrate the cross-view consistency and spatiotemporal consistency capabilities, during the spatiotemporal consistency training process, the training data can include video frame sequences from multiple different perspectives in the same time period. The video frames of each perspective at each time point are aligned in the time and space dimensions to adaptively fine-tune the learnable network parameters for cross-view consistency.

[0132] Network parameter freezing means that during the training or reasoning process, part or all of the network parameters are kept unchanged and no longer updated or adjusted, in order to avoid unnecessary interference to the trained network part, or to use a trained network module as a fixed feature extractor. In this embodiment, when the multimodal generation network obtained after training enters the third stage of training the joint prediction fusion module, the trained network parameters of the multimodal generation network will be kept unchanged during the training process, and no network parameter update will occur.

[0133] S103, according to the sample demand data corresponding to different synthetic data generation tasks, use the trained multimodal generation network and the pre-trained, frozen multimodal large language network to extract features from the sample demand data respectively, and input the extracted features into the joint prediction fusion module, train the joint prediction fusion module until the training converges, and obtain a trained synthetic data generation network.

[0134] The synthetic data generation task refers to the use of computer algorithms and technologies to generate non-real data sets with certain specific statistical characteristics or patterns for training, testing or verification of machine learning models to simulate real-world data sets. For example, in the field of autonomous driving, it is necessary to generate a large amount of simulated traffic scene data, including vehicles, pedestrians, road signs, etc. These data, as synthetic data, can help train autonomous driving systems to identify and respond to various complex situations. Synthetic data generation tasks can include a variety of task processing types, such as generation tasks that reuse existing data sets, generalization tasks that generalize scenes based on existing scene data to obtain batch synthetic data, prediction tasks that predict scene data that is difficult to collect in the real world based on existing scene data, and prediction scenarios in prediction tasks such as long-tail distribution, rare or extreme scene data (such as collision accidents, extreme weather conditions, wild animals on the road ahead, etc.).

[0135] Sample demand data refers to input data prepared according to the requirements of the synthetic data generation task, which contains the basic information or constraints required to generate synthetic data. For example, in the above generalization task, the sample demand data may at least include an existing scene data set as the basis for scene generalization, such as a collected scene image containing a car, and may also include text data used to guide scene generalization processing, such as "change the weather conditions of the scene", "replace the blue car in the scene with a white car", etc.

[0136] The joint prediction fusion module is used to fuse features from different networks or models, and make predictions based on the fused features to generate synthetic data. In this embodiment, the module can integrate the output features of the multimodal large language network and the multimodal generation network to improve the prediction ability and robustness of the synthetic data generation network.

[0137] In this embodiment, the multimodal large language network can be pre-trained by any method used to train the multimodal large language network in the relevant technology, and the network parameters of the trained multimodal large language network are in a frozen state during the training of the joint prediction fusion module, that is, the network parameters are kept unchanged together with the trained multimodal generation network described in step S102.

[0138] Based on the trained multimodal generative network and multimodal large language network, the obtained sample demand data and / or the corresponding feature vectors are respectively input into the two trained networks, and the features processed and output by the two networks are input into the joint prediction fusion module, so as to train the joint prediction module with the sample demand data, thereby adjusting the network parameters of the joint prediction module. The training of the joint prediction fusion module aims to train the module to effectively fuse the features from different network sources to generate synthetic data that meets the synthetic data generation task. During the training process, the joint prediction fusion module can fuse the features from different network sources through simple splicing, weighted summation, attention mechanism, etc. The fused features are used for synthetic data generation. The network parameters of the joint prediction fusion module can be adjusted by calculating and minimizing the loss function of the predicted output result and the real result corresponding to the sample demand data or the predefined evaluation standard, or by combining self-supervision with reinforcement learning.

[0139] For example, the generation task is to predict a scene image in which a collision accident occurs based on an existing scene image. The sample demand data may include a traffic scene image with at least two vehicles and text information describing the collision accident (such as a scene image in which a blue car collides with a white car in the prediction image). The sample demand data will be simultaneously input into the trained multimodal generation network and the multimodal large language network. The output features of the two networks are further input into the joint prediction fusion module. The predicted image is output based on the decoding process of the fused features output by the module. The pixel-level loss between the predicted image and the real image corresponding to the sample demand data (i.e., the real scene image in which the blue car collides with the white car in the image) is evaluated, or the semantic loss between the predicted image and the text information describing the collision accident in the sample demand data is evaluated, and other losses such as the motion consistency loss of the predicted image and the traffic scene image are combined, and the network parameters of the joint prediction fusion module are updated by back propagation to minimize the loss value.

[0140] In the disclosed embodiment, the embodiment proposes a phased and modular training method for a synthetic data generation network, through a three-stage training process that gradually increases the difficulty of the task, from cross-perspective consistency training to spatiotemporal consistency training, and then to joint prediction fusion training combined with a multimodal large language network, so that the network can fully learn at each stage, thereby gradually enhancing the ability of the synthetic data generation network, avoiding the problem of overfitting or training problems that may be caused by one-step training. The multimodal generation network can generate synthetic data with multi-perspective consistency and time continuity through cross-perspective and spatiotemporal consistency training, and enhances the ability of the synthetic data generation network to handle dynamic multi-perspective scenes. Combined with the training of the multimodal large language network through the joint prediction fusion module, the synthetic data generation network can fuse multimodal information such as images and texts, improve the network's prediction ability for future scenes, enable the network to generate more realistic, diverse and dynamically predictable synthetic data, improve training efficiency, and ensure the stability and generalization ability of the network on different tasks.

[0141] In addition, different stages require different types and degrees of computing resources. The multi-stage training strategy can flexibly allocate resources according to actual needs, effectively reducing training costs, achieving modularity and maintainability, facilitating troubleshooting, performance tuning, and updating or replacing specific parts, and improving the flexibility and scalability of the network; and, from simple feature learning to complex logical reasoning and application of physical laws, the network is trained on multiple different but related tasks, and can learn more extensive and abstract features, improving its generalization ability in new environments or unseen tasks.

[0142] In some embodiments, in order to more clearly explain the cross-view consistency training described in step S101 above, this embodiment provides an exemplary training process for performing cross-view consistency training on the multimodal generative network using scene data from multiple different viewpoints as described in step S101 above until the training converges to obtain a first network. The cross-view consistency training can be implemented by the following steps a1-a3:

[0143] a1, obtaining first training samples, each of which includes at least single-frame scene data of the same scene collected from multiple perspectives;

[0144] The first training sample refers to the sample data in the data set used for cross-view consistency training, and each sample contains at least a single frame of scene data of the same scene collected from multiple different viewpoints. These data can be images, video frames, or other forms of multimodal data such as depth maps, point clouds, description data for describing scene geometric layout information, etc. It is understandable that the "first" in the first training sample has no specific meaning and is only used to distinguish other sample data.

[0145] For example, taking the synthetic data generation task in an autonomous driving scenario as an example, each first training sample may include multiple perspective images of an intersection, and the perspective images may include the intersection scene at the same time captured by cameras in front, on the side, behind, and above the vehicle. Multiple perspective images together constitute a first training sample.

[0146] a2, inputting the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network;

[0147] The multimodal generative network processes the input first training sample using a preset network structure, learns the complex patterns in the first training sample, and generates output data related to it but not identical to it. In this embodiment, the multimodal generative network is used to generate a predicted image of the same scene observed from the same perspective as the first training sample.

[0148] a3. Calculate a first loss function based on the predicted images of the multiple corresponding perspectives and the corresponding first training samples, and adjust the network parameters of the multimodal generation network by minimizing the first loss function; the first loss function at least includes the loss within a single perspective and the consistency loss across perspectives.

[0149] The first loss function is a function used to evaluate the difference between the predicted image generated by the multimodal generative network and the real image. In cross-view consistency training, the first loss function not only considers the difference between the predicted image and the real image in a single view, i.e., the intra-view loss, but also considers the consistency of the predicted images between different views, i.e., the cross-view consistency loss. The first loss function is a weighted sum of the intra-view loss and the cross-view consistency loss, and the parameters of the multimodal generative network are adjusted by minimizing the first loss function.

[0150] The intra-single-view loss is used to measure the difference between the generated predicted image and the first training sample in each single view, which can be achieved by comparing the pixel difference, feature difference or semantic difference between the generated image and the actual image corresponding to the generated image and from the same view; the cross-view consistency loss is used to measure the consistency of the generated results between different views, which can be achieved by comparing the similarity or difference between the predicted images of different views. For example, the cross-view consistency loss function can be used to ensure that the images generated from different views are consistent in key elements and structures, such as measuring the cross-view consistency loss by calculating the similarity matrix between the generated results from different views, or extracting key features from the generated results from different views and performing feature matching, calculating the consistency loss based on the matching results, or converting the generated results from different views to the same view through a projection transformation, and then calculating the difference between them.

[0151] In addition, other auxiliary losses can be added to further guide the training process according to specific task requirements. For example, for tasks that require the simultaneous generation of images and semantic segmentation results, semantic segmentation losses can be added to ensure the consistency between the generated images and the semantic segmentation results. In order to improve the generalization and robustness of the model, it is also possible to consider adding regularization terms to the loss function and introducing data augmentation strategies to reduce overfitting and improve the network's performance on unseen data.

[0152] In the case where the first training sample includes complete scene images of the same scene from multiple perspectives, the intra-single-perspective loss can be calculated by measuring the pixel loss between the predicted image of the same perspective and the complete scene image in the sample; in the case where the first training sample includes images (such as depth maps, semantic segmentation maps) or text descriptions of geometric layout information of the same scene from multiple perspectives, the first training sample can also include real complete scene images of the corresponding perspectives to calculate the intra-single-perspective loss, or the loss function can be calculated by measuring the consistency or difference between the structured information of the generated predicted image and the structured information of the scene in the first training sample through loss functions such as semantic loss and depth loss.

[0153] Based on the cross-view consistency training process described in the above steps, taking the capture of static scene images at a single time point by acquisition devices installed at different positions on the vehicle (such as a front camera, a side view camera, and a rear view camera) as an example, multiple images of the same scene at different viewpoints acquired at the same time constitute a first training sample. The training process of the multimodal generative network may include the following steps: (1) Network initialization: Initializing the parameters of the multimodal generative network. The initialized parameters may be randomly generated or obtained based on a pre-trained network. (2) Forward propagation: Inputting multiple first training samples into the multimodal generative network in batches, and obtaining the predicted image or other output form such as a feature vector representation of the corresponding viewpoint through the forward propagation calculation of the multimodal generative network. (3) Backward propagation: Calculating the first loss function, and performing backpropagation according to the loss function value to update the network parameters to minimize the loss function value. (4) Iterative training: Repeating the forward propagation and backpropagation processes until the training converges or reaches a preset number of training rounds.

[0154] In the disclosed embodiment, by simultaneously utilizing scene data from multiple different perspectives, and designing a loss within a single perspective and a consistency loss across perspectives. The first loss function guides the multimodal generative network to improve the accuracy of single-perspective image generation during training, and enhances the consistency of predicted images between different perspectives, that is, minimizes the consistency loss across perspectives. Through this training mechanism, the network is prompted to learn to extract and integrate key information from multiple perspectives, capture the common structural features and intrinsic associations between different perspectives, and enable the network to maintain the consistency and complementarity of information between different perspectives when generating predicted images. In addition, temporarily ignoring the temporal attention layer at the beginning of training and focusing on the structural constraints in the learning scene helps the network to establish basic spatial perception and feature extraction capabilities more quickly. Through this cross-perspective consistency training, the network can generate high-quality images more robustly when facing different perspectives, lighting conditions or occlusions.

[0155] In some embodiments, multi-view consistency is used in a multimodal generative network to maintain the consistency of scene data obtained from different camera perspectives after processing. For example, in the application scenario of an autonomous vehicle, there may be multiple cameras installed around the vehicle, such as 6, 7, or 10, to form a multi-view monitoring system. The images captured by multiple different cameras will be fused into a unified bird's-eye view to help the vehicle understand and predict the surrounding environment. In order to ensure that the scene data captured from different perspectives can still maintain the consistency of object information after being processed by the multimodal generative network, see Figure 2A As shown, this embodiment introduces a cross-view consistency attention layer in the multimodal generation network, applies a cross-view attention mechanism, and enhances the multimodal generation network's learning of the consistency of object performance under different viewpoints by comparing the feature vectors of the central view and the adjacent viewpoints.

[0156] Based on the cross-view consistency attention layer, when the multimodal generative network is trained for cross-view consistency, as Figure 2B As shown, the inputting of the first training sample into the multimodal generation network to obtain the predicted images of multiple corresponding perspectives generated by the multimodal generation network as described in the above embodiment can be achieved in the following manner:

[0157] S201, using the multimodal generation network, respectively extracting potential features corresponding to single-frame scene data collected from each perspective;

[0158] Latent features refer to the high-level, abstract feature representations extracted after nonlinear transformation of the input data through a series of network layers such as convolutional layers and pooling layers in the neural network, which can capture the essential information of the input data.

[0159] In this embodiment, for the single-frame scene data of the same scene collected from multiple perspectives included in the first training sample, low-level features such as edges, textures, and shapes of each perspective image can be extracted through structures such as convolutional layers, and then higher-level semantic features such as object categories and positional relationships can be further extracted through deeper network layers, and all the extracted features are collectively used as potential features corresponding to the perspective.

[0160] S202, taking the potential feature of each view as the first query vector, and taking the connection of the potential features of two adjacent views as the first key vector and the first value vector, and performing linear transformation through the cross-view learnable matrix to obtain the cross-view attention output of each view;

[0161] The cross-view learnable matrix refers to a parameter matrix that can be updated during the training process, which is used to linearly transform the potential features from different views in the cross-view attention mechanism. This matrix can learn the correlation and importance of potential features between different views to better integrate cross-view information.

[0162] The cross-view attention output refers to the attention-weighted feature representation of each view calculated by the cross-view attention mechanism. In the attention mechanism, the network dynamically assigns attention weights based on the correlation between the potential features of the current view (the first query vector) and the potential features of other views (the first key vector and the first value vector), and obtains the cross-view attention output by weighted summation. The cross-view attention output contains complementary information between the current view and the adjacent view, which facilitates the generation of more accurate and coherent predicted images.

[0163] Assuming that there are scene data from three perspectives, such as the left perspective, the middle perspective, and the right perspective, first extract the potential features of each perspective. Next, for the middle perspective, the potential features of the middle perspective are used as the first query vector, and the connection value of the potential features of the left and right perspectives adjacent to the middle perspective is obtained, and the connection value is used as the first key vector and the first value vector; then, the first query vector, the first key vector, and the first value vector features are linearly transformed by the above-mentioned cross-perspective learnable matrix, and the attention weights are calculated. Finally, the cross-perspective attention output of the middle perspective is obtained by weighted summation. The cross-perspective attention output of the middle perspective integrates the information of the left and right perspectives, making the predicted image of the middle perspective more accurate and coherent. In the case where the current perspective to be calculated for the cross-perspective attention output has an adjacent perspective, the connection value of the potential feature of the current perspective and the potential feature of the adjacent perspective can be used as the key vector and the value vector.

[0164] For the cross-view attention mechanism described in step S202 above, its calculation process can also be described by the following formula:

[0165]

[0166] Among them, [,] represents the feature connection operation, As a cross-view learnable matrix in the cross-view attention mechanism, it is used to linearly transform the first query vector, the first key vector, and the first value vector; d is the dimension of the first key vector and the first query vector, which is used to scale the dot product result to prevent the gradient from disappearing; refers to the potential features of the i-th perspective, and are the potential features of the two adjacent views of the i-th view; the softmax function is used to normalize the attention weight. For the i-th view, the attention weight is calculated by the softamx function and multiplied by the value vector of the adjacent view to obtain the weighted feature representation of the i-th view as the cross-view attention output.

[0167] S203: Generate a predicted image of the corresponding perspective based on the cross-perspective attention output of each perspective.

[0168] The cross-view attention output of each view integrates the feature representation of the current view and the adjacent view information, capturing the intrinsic connection and complementary information of the scene between different views. After obtaining the cross-view attention output of each view, the predicted image of the corresponding view can be output through decoding processing and image generation and other processing steps.

[0169] In the disclosed embodiment, this embodiment introduces a cross-view attention mechanism in the cross-view consistency training process, by respectively extracting the potential features corresponding to the single-frame scene data collected from each view, and then using the cross-view attention mechanism to interact the potential features of each view with the potential features of the adjacent view, the network's ability to integrate cross-view information is enhanced, so that the cross-view attention output of each view can capture the intrinsic connection and complementary information between different viewpoints, reduce the information missing or misleading that may be caused by a single view, and make the generated predicted image more coherent and accurate in content.

[0170] In some embodiments, based on the need to improve the generation capability of the multimodal generation network in cross-view scenarios and ensure that the generated predicted images have high consistency and coherence between different perspectives, in order to more accurately capture and utilize the potential correlation information between different perspectives, this embodiment further introduces a cross-view joint distribution to generate a predicted image after obtaining the cross-view attention output of each perspective. Based on the cross-view joint distribution, the generation of the predicted image of the corresponding perspective according to the cross-view attention output of each perspective described in the aforementioned step S203 can be achieved in the following way:

[0171] According to the predicted images of the i-1 perspectives that have been generated, the conditional probability of the cross-perspective joint distribution of the i-th perspective is obtained; the i is a positive integer greater than 1; according to the predicted images of the i-1 perspectives and the cross-perspective attention output of the i-th perspective, combined with the cross-perspective joint distribution conditional probability, the predicted image of the i-th perspective is generated.

[0172] That is, after the predicted images of the first i-1 perspectives have been generated, the cross-perspective joint distribution conditional probability of the i-th perspective can be calculated based on the potential correlation between the generated predicted images and the current i-th perspective. The conditional probability reflects the possible distribution of the i-th perspective image content given the information of the first i-1 perspectives. Next, the network will use the cross-perspective attention output of the i-th perspective that integrates the information of the current perspective and the adjacent perspectives, and the cross-perspective joint distribution conditional probability calculated above as the basis for generating the predicted image of the i-th perspective. By combining the feature information in the cross-perspective attention output with the structural information in the cross-perspective joint distribution conditional probability, the i-th perspective predicted image that conforms to multi-perspective consistency is generated.

[0173] Assume x K represents the kth perspective, and the cross-perspective joint distribution can be expressed by the following formula:

[0174] p(x 1 ,...,x K )=p(x 1 )p(x 2 |x 1 )…p(x K |x 1 ,…,x K-1 )

[0175] Regarding the cross-view joint conditional probability distribution, the conditional probability relationship can be implicitly captured by a pre-trained neural network layer. During the training phase, the neural network layer learns how to extract useful information from the predicted image of the i-1 view and predict the image of the i-th view based on this information. In the process of combining the predicted image of the i-1 view with the cross-view attention output of the i-th view and the cross-view joint distribution conditional probability, it is also possible to learn how to fuse the three types of information, namely the predicted image of the i-1 view, the cross-view attention output of the i-th view, and the cross-view joint distribution conditional probability, through a network structure such as an attention mechanism and a feature fusion layer to predict the image of the i-th view. The cross-view joint distribution conditional probability does not need to be explicitly calculated.

[0176] In the disclosed embodiment, through cross-view joint distribution modeling, when generating the predicted image of each view, the predicted image of the previously generated view and the cross-view attention output between the current view and the adjacent view can be fully utilized, thereby ensuring that the generated predicted image has a high degree of consistency and coherence between different viewpoints. At the same time, the introduction of cross-view joint distribution also further improves the generation capability and robustness of the multimodal generation network in complex scenes.

[0177] In some embodiments, in order to improve the performance of the multimodal generation network when processing cross-modal tasks, especially to more accurately generate the target modality output when facing a single-view input, before inputting the first training sample into the multimodal generation network, the method may also include a single-view training step, based on the scene data of a single view, for the multimodal generation network. This pre-training step is intended to enhance the network's ability to understand single-modal input, so that it can perform more robustly and accurately in subsequent cross-modal generation tasks. This step may include the following implementation process:

[0178] Obtaining single-view scene data and inputting it into the multimodal generation network for image generation task training, and adjusting the network parameters of the multimodal generation network by minimizing the second loss function;

[0179] When the single-view scene data is image type data, the second loss function includes at least pixel-level reconstruction loss; when the single-view scene data is text type data, the second loss function includes at least semantic loss.

[0180] Single-view scene data refers to image data or description data about a set scene collected from a single viewpoint, containing detailed information about the scene content from a certain viewpoint. In image-to-image generation tasks, single-view scene data refers to images from a single viewpoint; in tasks of generating images based on text descriptions, single-view data includes text descriptions of the scene.

[0181] In order to evaluate the accuracy of the network generation results and guide the optimization of network parameters, a second loss function is defined to evaluate the difference between the network generation results and the real input. In different modalities, the composition of the second loss function can be flexibly adjusted. When the single-view scene data is image type data, a pixel-level reconstruction loss, such as Mean Squared Error (MSE) or Structural Similarity Index (SSIM), can be used to directly compare the difference between the generated image and the real image at the pixel level to achieve the minimum visual difference between the network-generated image and the input image. When the single-view scene data is text type data, the semantic consistency between the generated image and the text description can be evaluated by semantic loss, not just the superficial visual similarity. The semantic loss can be calculated by a pre-trained semantic embedding model, or by designing a specific contrast loss to ensure the matching degree between the generated content and the text description at the semantic level.

[0182] In the disclosed embodiment, before performing cross-view consistency training, the single-view scene data is first used to calculate the single-view loss of the multimodal generation network, so that in the subsequent cross-view consistency training using multi-view scene data, the learned single-view image understanding and generation knowledge can be transferred to the understanding and generation of multi-view scene data, thereby improving the initialization quality and subsequent training efficiency of the multimodal generation network.

[0183] In some embodiments, for the multi-frame scene data sequence described in step S102, the first network is trained for spatiotemporal consistency until the training converges to obtain a trained multimodal generative network, and the training process may at least include the following steps b1-b3:

[0184] b1, obtaining second training samples, each of which at least includes an input sequence consisting of multiple frames of scene data of the same scene that continuously changes over time;

[0185] The second training samples include a set of samples used to train the first network, i.e., a multimodal generative network trained for cross-view consistency, each sample containing multiple frames of scene data of the same scene that continuously changes over time, and the scene data may be video frames, radar data, depth images, etc. The input sequence refers to a sequence of data frames at multiple consecutive time points, which is used to simulate the change of a scene over time.

[0186] For example, each second training sample can be a sequence of 10 consecutive video frames, which shows the process of pedestrians walking on the street; or, the second training sample can also be a sequence of 10 image frames with a certain time interval, where the frames satisfy a strict time sequence and the changes in the spatial geometric layout of adjacent frames conform to physical laws.

[0187] b2, inputting the second training sample into the first network, and obtaining a multi-frame predicted image sequence generated by the first network and conforming to the task type based on the task type set for each second training sample;

[0188] The task type refers to a specific task that the training process expects to perform based on the second training sample, such as including but not limited to reconstruction of video frames or inter-frame interpolation, wherein the reconstruction of video frames is used to generate past frames or future frames, and the inter-frame interpolation is used to insert intermediate frames between adjacent frames with a time interval to make the sequence more coherent and smooth.

[0189] The multi-frame predicted image sequence is generated by the first network based on the input sequence and the set task type, and can include at least any one of the reconstruction of the past frame sequence, the intermediate frame prediction sequence, and the future frame prediction sequence.

[0190] Taking the above sequence of 10 consecutive video frames showing the process of pedestrians walking on the street as an example, if the set task type is future frame prediction, the first network will predict the 11th frame and subsequent video frames of the pedestrians as the prediction results based on the input sequence.

[0191] b3. Calculate a third loss function based on the multi-frame predicted image sequence and the multi-frame real image sequence corresponding to the second training sample, and adjust the network parameters of the first network by minimizing the third loss function; the third loss function includes at least pixel-level reconstruction loss, cross-view consistency loss, and motion consistency loss.

[0192] The third loss function is used to evaluate the difference between the multi-frame predicted image sequence generated by the first network and the real multi-frame image sequence corresponding to the second training sample. By minimizing the third loss function, the parameters of the first network can be adjusted to make the prediction result closer to the actual situation. Based on the fact that the spatiotemporal consistency training is further trained on the basis of the first network, in this embodiment, the third loss function not only includes motion consistency loss and reconstruction loss, but also includes cross-perspective consistency loss for fine-tuning the parameters learned by cross-perspective consistency training, so as to better integrate the cross-perspective consistency and spatiotemporal consistency learned by the network. The third loss function is the weighted sum of each loss term, and the influence of each loss term on the third loss function can be adjusted by the weight parameter. For situations that do not involve multi-perspective or motion consistency, the corresponding loss term and its weight can be set to 0. Among them, the motion consistency loss is used to evaluate whether the motion trajectory of moving objects (such as pedestrians and vehicles) in the predicted image is consistent with the actual situation, ensuring the continuity and rationality of the predicted image in the time and space dimensions.

[0193] For example, for the above sequence of 10 consecutive video frames, which shows an example of a pedestrian walking on the street, after the first network predicts the image sequence of the pedestrian's future 11-20 frames based on the input sequence, the third loss function can be calculated by comparing the predicted 11-20 frames with the 11-20 frames in the real scene frame by frame, and then the network parameters of the first network are updated by back-propagation gradient to minimize the third loss function.

[0194] In the disclosed embodiment, a training method is used to gradually increase the complexity and difficulty of tasks through staged training. In the cross-view consistency training stage, the first network is trained to understand and generate a consistent representation of the same scene from different viewpoints. Subsequently, in the spatiotemporal consistency training stage, a multi-frame scene data sequence is used to further train the first network to capture features of the scene that continuously change over time, and generate a multi-frame predicted image sequence that conforms to spatiotemporal consistency by designing a third loss function that comprehensively considers pixel-level reconstruction loss, cross-view consistency loss, and motion consistency loss. The progressive training enables the multimodal generation network to gradually establish a deep understanding of scene, viewpoint, and time changes, effectively improving the training efficiency and stability of the multimodal generation network, while enhancing the generalization ability and multi-task learning ability of the network.

[0195] In some embodiments, in order for the multimodal generative network to better understand the intrinsic connection between frames in the input sequence and improve the temporal and spatial consistency of the generated image sequence, see Figure 3AAs shown, this embodiment introduces a cross-frame consistency attention layer based on the first network trained for cross-view consistency, which is used to capture the mutual connection between frames when the scene changes continuously, so that the network can take into account the information of other frames when generating each frame, thereby generating a more coherent and natural image sequence.

[0196] Based on the cross-frame consistency attention layer, when the first network obtained through cross-view consistency training is trained for spatiotemporal consistency, such as Figure 3B As shown, the multi-frame predicted image sequence generated by the first network and conforming to the task type described in the above embodiment can be achieved in the following way:

[0197] S301, using the first network, respectively extracting potential features of each frame of scene data in the input sequence;

[0198] The first network refers to a multimodal generative network that has been trained for cross-view consistency and has the ability to generate consistent scene representations from different viewpoints. In this embodiment, for a multi-frame scene data sequence of the same scene included in the second training sample, the first network can be used to extract low-level features such as edges, textures, and shapes of each frame of scene data through structures such as convolutional layers, and then further extract higher-level semantic features such as object categories, positional relationships, etc. through deeper network layers, and all the features extracted are used together as potential features of the frame of scene data.

[0199] For example, assuming that the input sequence is a video clip containing several consecutive frames of images, the first network processes each frame of the image one by one and extracts the potential features of each frame of the image through its encoder part. The potential features may include key information such as edges, textures, shapes, etc. in the image.

[0200] S302, taking the latent features of the target frame scene data as a second query vector, taking the connection of the first frame scene data in the input sequence and the latent features of the next frame of the target frame scene data as a second key vector, and taking the connection of the first frame scene data and the latent features of the previous frame of the target frame scene data as a second value vector; the target frame scene data refers to any frame scene data in the input sequence;

[0201] In the attention mechanism, the query vector refers to a vector used to query relevant information, the key vector is a vector that is compared with the query vector to determine which information is important, and the value vector contains a vector of information that is actually to be retrieved. In this embodiment, the second query vector represents the latent features of the target frame, which is used to retrieve relevant information from other frames; the second key vector is formed by connecting the latent features of the next frame of the target frame and the first frame of the sequence, and contains future and initial scene information related to the target frame; the second value vector is formed by connecting the latent features of the previous frame of the target frame and the first frame of the sequence, and provides information about scene changes before the target frame.

[0202] Assume that the input sequence has three frames of images A (first frame), B (target frame), and C (next frame of the target frame). When processing the target frame B, the latent features of B are used as the second query vector, the second key vector is formed by connecting the latent features of the first frame A of the sequence and the latent features of the next frame C of the target frame B, and the second value vector is formed by connecting the latent features of the first frame A and the latent features of the previous frame A of the target frame B. When the target frame is the first frame of the input sequence, the previous frame of the target frame can be calculated by using the target frame itself or a zero vector; similarly, when the target frame is the last frame of the input sequence, the next frame of the target frame can be calculated by using the target frame itself or a zero vector.

[0203] S303, performing a linear transformation on the second query vector, the second key vector, and the second value vector through a cross-frame learnable matrix to obtain a cross-frame attention output of the target frame scene data;

[0204] The cross-frame learnable matrix refers to the trainable parameter matrix used in the attention mechanism to linearly transform the second query vector, the second key vector, and the second value vector in order to calculate the attention weights and the final attention output. The linear transformation maps the input vector to another vector space through matrix multiplication. The cross-frame attention output is obtained after the attention mechanism is processed, and contains the relevant information retrieved from the second key vector and the second value vector according to the second query vector.

[0205] In this embodiment, the cross-frame learnable matrix performs a linear transformation on the second query vector and the second key vector to calculate an attention weight matrix, which determines the degree of attention that should be given to other frames when generating a predicted image of the target frame. The cross-frame attention output of the target frame is obtained by weighted summation of the second value vector, which includes relevant information from other frames, thereby generating a more coherent and accurate predicted image.

[0206] For the cross-view attention mechanism described in steps S302 and S303 above, its calculation process can also be described by the following formula:

[0207]

[0208] Among them, [,] represents the feature connection operation, As a cross-frame learnable matrix in the cross-frame attention mechanism, it is used to linearly transform the second query vector, the second key vector, and the second value vector; d is the dimension of the second key vector and the second query vector, which is used to scale the dot product result to prevent the gradient from disappearing; refers to the potential features of the j-th frame scene data, They are respectively the potential features of the first frame scene data, the previous frame of the j-th frame, and the next frame scene data in the input sequence; the softmax function is used to normalize the attention weight. For the j-th frame scene data, the attention weight is calculated by the softamx function, multiplied by the second value vector composed of the first frame and the next frame scene data, to obtain the weighted feature representation of the j-th frame scene data as the cross-frame attention output.

[0209] S304, generating the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence.

[0210] In this step, the cross-frame attention output based on each frame of scene data combines the characteristics of the target frame itself and the enhanced feature representation from related information of other frames. The enhanced feature representation can be further sent to the subsequent parts of the network, such as the decoder or generator, to generate the predicted image of the target frame using the enhanced strong features. Since each target frame has been processed by the cross-frame consistency attention layer, the generated predicted image sequence will show higher consistency in time and space, and the predicted images between adjacent frames will be more coherent. By performing the above process on each frame in the input sequence, a complete multi-frame predicted image sequence can be obtained, which reflects the appearance of objects in the scene and captures their changes and motion trajectories over time, thereby achieving accurate prediction of dynamic scenes.

[0211] In the disclosed embodiments, a cross-frame consistency attention layer is introduced to enhance the performance of a multimodal generative network in spatiotemporal consistency training. The attention layer utilizes the potential feature relationship between adjacent frames in the input sequence to generate a cross-frame attention output for a target frame that combines global and local information, so that the network can take into account the contextual information of other frames when generating a predicted image for each frame. By explicitly modeling the potential feature relationship between frames, the generated predicted image sequence is more consistent in time and space, reducing the sense of discontinuity caused by independently processing each frame, and improving the smoothness of the overall video or dynamic scene, thereby generating a more coherent and accurate multi-frame predicted image sequence.

[0212] In some embodiments, based on the need for in-depth understanding of dynamic scenes, in order to improve the coherence and accuracy of multi-frame predicted image sequences, this embodiment further introduces cross-frame joint distribution to generate multi-frame predicted image sequences after obtaining the cross-frame attention output of each frame of scene data in the input sequence. Based on the cross-frame joint distribution, the generation of the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence described in step S304 can be achieved in the following way:

[0213] When generating a future multi-frame prediction image sequence based on the second training sample, the inter-frame joint distribution conditional probability of the j-th frame prediction image to be generated is obtained according to the input sequence and the j-1 frame prediction image that has been generated; j is a positive integer greater than 1; based on the cross-frame attention output of each frame of scene data, combined with the inter-frame joint distribution conditional probability, the j-th frame prediction image is generated in an autoregressive manner.

[0214] In video prediction or dynamic scene generation tasks, relying solely on the cross-frame attention output of a single frame may not be enough to capture complex spatiotemporal dynamic changes. Therefore, this embodiment introduces a cross-frame joint distribution to better model the dependency between frames and generate a more coherent and accurate predicted image sequence.

[0215] In the autoregressive generation process, when the predicted images of the previous j-1 frames have been generated according to the input sequence, it is necessary to calculate the conditional probability of generating the predicted image of the j-th frame. This conditional probability can be obtained based on the inter-frame joint distribution of the input sequence and the generated j-1 frame predicted image. The conditional probability model (such as conditional generative adversarial network, conditional variational autoencoder, etc.) can be used to estimate the inter-frame joint distribution conditional probability. The model takes into account the overall information of the input sequence and the local information of the generated predicted image sequence, thereby outputting a distribution that can guide the generation of the j-th frame.

[0216] After obtaining the cross-frame attention output of each frame of scene data, it is combined with the conditional probability of the joint distribution between frames, so that when generating the j-th frame prediction image, the contextual information from other frames and the coherence of the generated prediction image sequence are comprehensively considered. The combination can be to use the cross-frame attention output as one of the inputs of the conditional probability model, or to perform some form of weighted fusion of the two. Based on the combined information, the j-th frame prediction image is generated in an autoregressive manner. In the autoregressive generation process, the generation of each frame depends on all previous frames, including frames in the input sequence and frames in the generated prediction image sequence, so as to ensure that the generated prediction image sequence is coherent in time and space.

[0217] For example, assuming that the input sequence contains three frames of scene data (frame 0, frame 1, and frame 2), and the goal is to generate a predicted image sequence of the next three frames (frame 3, frame 4, and frame 5), then the processing steps in the above manner are as follows:

[0218] Step 1: First, use the cross-frame attention mechanism to process the input sequence and obtain the cross-frame attention output of each frame of scene data; Step 2: When generating the predicted image of the 4th frame, the conditional probability model can be used to calculate the inter-frame joint distribution conditional probability of the 4th frame based on the input sequence (frame 0, frame 1, frame 2) and the generated predicted image of the 3rd frame; Step 3: Combine the cross-frame attention output with the inter-frame joint distribution conditional probability to obtain an enhanced feature representation; Step 4: Based on the enhanced feature representation, generate the predicted image of the 4th frame in an autoregressive manner. Generate the predicted image of the 5th frame by repeating the above steps 2-4, and consider the information of all previous frames each time.

[0219] The above inter-frame joint distribution modeling can be expressed by the following formula:

[0220]

[0221] Among them, x future represents the future frame generated based on the input video frame, x past Represents the input sequence of video frames.

[0222] Regarding the conditional probability of joint distribution between frames, the conditional probability relationship can also be implicitly captured by a pre-trained neural network layer. During the training phase, the neural network layer learns how to extract useful information from all the generated image frames before the predicted frame, and predicts the predicted image of the jth frame based on this information. In the process of combining the conditional probability of joint distribution between frames, it is also possible to learn how to fuse the three types of information, namely, the existing j-1 frame image, the cross-frame attention output of the jth frame, and the conditional probability of joint distribution between frames, through a network structure such as an attention mechanism and a feature fusion layer to predict the jth frame image. The conditional probability of joint distribution between frames does not need to be explicitly calculated.

[0223] In the disclosed embodiment, on the basis of a multimodal image prediction framework based on a cross-frame attention mechanism, inter-frame joint modeling is further introduced, and the cross-frame attention mechanism is used to process each frame of scene data in the input sequence to capture the long-distance dependency between frames, and in the process of generating future multi-frame predicted image sequences, it relies on the cross-frame attention output and the already generated predicted image sequence and input sequence at the same time, effectively utilizing the temporal continuity, generating predicted images frame by frame in an autoregressive manner, ensuring that each frame of the generated image fully considers all the previous information, thereby maintaining the overall coherence of the predicted image sequence, enabling the network to effectively deal with complex external environmental changes such as occlusion and high-speed motion of objects, and avoiding the problem of element inconsistency caused by too long time intervals.

[0224] Through the above-mentioned embodiment, the aforementioned steps S101 and S102 are used to train the multimodal generation network in the synthetic data generation network. In the cross-perspective consistency training stage, multi-perspective structured scene data is input to provide the static or quasi-static part of the scene and the visual performance of the scene at different perspectives. In the spatiotemporal consistency training, multiple scene frames including a time series are provided. The input increases the time dimension, so that the player can learn the evolution and dynamic changes of the scene over time. In addition, the loss function of the cross-perspective consistency training stage is used to evaluate the generation quality of a single time point or frame and the consistency between different perspectives, and the spatiotemporal consistency training is extended to combine cross-perspective consistency with spatiotemporal continuity, so that the network-generated data can simultaneously meet the perspective coherence and the temporal dynamics and continuous changes in the scene.

[0225] In some embodiments, scene controllability refers to meeting the user's production data needs, and adjusting and changing the environmental variables, dynamic elements, layout conditions, etc. in the scene without changing the overall consistency, in order to increase data diversity and generalization. In order to further enhance the flexibility and practicality of the multimodal generation network, this embodiment introduces a scene controllability training step after completing the spatiotemporal consistency training of the first network. After the scene controllability training is completed until convergence, a fully trained multimodal generation network is obtained. While giving the network the ability to maintain overall spatiotemporal consistency, it can flexibly adjust the environmental variables, dynamic elements or layout conditions in the generated scene according to the conditions set by the user, thereby increasing the diversity and generalization of the generated data and achieving effective data reuse. The scene controllability training step can be implemented through the following steps:

[0226] Acquire setting condition data for guiding the output of the image generation task of the second network; the second network refers to a network obtained by completing the spatiotemporal consistency training of the first network; the setting condition data includes any one of text data describing the setting scene, or text and image combined data;

[0227] The second network is trained for controllability of image generation scenarios using the set condition data, and the network parameters of the second network are adjusted by minimizing a fourth loss function to obtain the trained multimodal generation network; the loss terms of the fourth loss function include at least pixel-level image reconstruction loss, cross-view consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss.

[0228] In order to achieve controllability of scene generation, the input of set conditional data is used to guide the model to generate data output under specific conditions. On the basis of the first network trained with spatiotemporal consistency, scene controllability training is introduced so that the network can focus on the image style or layout that the user wants and learn how to accurately extract semantic features that are aligned with the fused text or image. To this end, in addition to the losses that need to be considered in the spatiotemporal consistency training process, scene controllability needs to consider style loss, semantic consistency loss, and cycle consistency loss.

[0229] Among them, style loss is used to keep the generated image consistent with the artistic style or visual atmosphere implied by the given text description. The feature statistics of the pre-trained network are also used to measure the similarity between the style implied by the generated image and the text description. Semantic consistency loss ensures that the content of the generated image is semantically consistent with the input text description. This is achieved through additional semantic labels or matching losses between the layout map and the generated image, or using distance metrics in cross-modal embedding space. Cycle consistency loss ensures cycle consistency in the process from image to text and then to image by allowing the model to generate text from images and then generate images from text, thereby improving the quality of the generated image and its match with the text description.

[0230] Furthermore, we combine the above loss functions to construct a composite loss function to balance the authenticity of the image, the correctness of the content, the conformity of the style, and the consistency with the text description. By adjusting the weights of different loss terms, we can guide the model to focus on specific goals or features when generating images.

[0231]

[0232] in, They respectively represent reconstruction loss, perspective consistency loss, motion consistency loss, adversarial loss, style loss, semantic consistency loss, and cycle consistency loss. α, β, γ, δ, ε, ∈, ζ are hyperparameters used to adjust the influence of each loss item on the fourth loss function.

[0233] In the above-mentioned scene controllable training process, the set condition data is used to provide clear generation guidance for the network, which can be text data describing the set scene, such as sentences and keywords describing the scene, or a combination of text and image data, such as a structural diagram of the scene geometric layout plus corresponding text descriptions, so that the generated image or video can be adjusted according to the user's intention. The scene controllability training is used to train the second network to learn how to adjust its internal generation mechanism according to the input set condition data to generate images or videos that meet the user's expectations. The training process is guided by introducing the above-mentioned pre-defined fourth loss function containing multiple loss terms, and the network parameters of the second network are adjusted by continuous iteration to minimize the total loss value of the fourth loss function, so that the network gradually learns how to generate diversified images or videos that meet user requirements and maintain cross-perspective consistency and spatiotemporal consistency according to the input set condition data.

[0234] For example, if you are required to generate an image describing a scene of "children playing happily on a summer beach", in scene controllability training, users can enter a descriptive text as the set condition data, and specify some additional conditions, such as sunny weather conditions, beach layouts including sand castles and parasols, etc. After training, the multimodal generative network can generate a scene image that contains both a vivid picture of children playing on the beach and the characteristics of sunny summer weather based on the set condition data. By adjusting different parameters in the set condition data, users can generate a variety of videos with different styles, layouts, and dynamic changes, thereby achieving effective reuse and diversified generation of data.

[0235] The use of the set condition data to train the second network on the controllability of image generation scenarios as described in the above embodiment can be achieved through the following steps c1-c5:

[0236] c1, using the second network to extract initial image features corresponding to the set condition data;

[0237] The initial image features corresponding to the set condition data may be obtained through a transformer-based architecture Vision Transformer, or other image feature extraction methods such as a mixed use of traditional convolution and Transformer architectures.

[0238] c2, obtaining a first image feature of a first sub-region where each object instance is located in the initial image feature, and adding a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0239] The object instance refers to the physical object described in the set condition data, such as a pedestrian, a car or a traffic light, and each object instance is a separate instance of the same type of object, which is identified by a target or object detection algorithm. The first sub-region refers to the image region associated with each object instance, which can be determined based on the bounding box of the object instance, such as directly using the region where the bounding box is located as the first sub-region where the object instance is located.

[0240] The first image feature of the first sub-region refers to a feature vector used to describe the region where the object instance is located. Based on the input being the initial image feature, the first image feature of the first sub-region where the object instance is located is extracted from the initial image feature, that is, the local features of a specific object instance are separated from the global features of the image, and the first image feature of the first sub-region where each object instance is located can be cropped from the global initial image feature according to the segmentation mask of the object instance in the initial image feature. This process can be implemented by matrix multiplication, where the segmentation mask of the object instance is multiplied by the global feature map, and only the feature information of the corresponding object is retained, while the background features are set to zero or ignored, thereby realizing the individual extraction of the features of each object instance to form the first image feature.

[0241] Among them, regarding the segmentation mask of the object instance in the initial image feature, the segmentation mask can be obtained by performing object detection and segmentation on the input image, that is, a target detection model (such as a fast region convolutional neural network, a neural network YOLO) can be used to perform object detection on the input image to identify each object instance and its position in the image, and then a segmentation mask for each object instance is generated through a semantic segmentation or instance segmentation model such as a mask region convolutional neural network, a deep learning semantic segmentation or a U-shaped network.

[0242] The learnable category hint refers to information that is dynamically adjusted and optimized during the model training process. The category hint is used to represent the feature representation of the category of the object instance, and is used to guide the model to better understand and distinguish object instances of different categories. The category hint usually exists in the form of a vector, each vector represents a category and contains information or attributes unique to the category. The category hint is continuously iterated and updated during the training process to better combine with the image features, and is shared among all the attention layers of the model, ensuring a consistent understanding of the object instances at different levels.

[0243] Combining the category hint of each object instance with its corresponding first image feature can be achieved through a simple splicing operation. The first image feature of the first sub-region where each object instance is located is added with the category hint of the corresponding object instance, thereby forming the second image feature of the first sub-region. The second image feature not only contains the image information of the object instance, but also incorporates the semantic information at the category level of the object instance, so that the model can take into account both the image features and category information of the object instance, which is convenient for precise control of each object instance during the synthetic data generation process.

[0244] Suppose there is an image containing a cat and a dog, identify the location and category of the cat and dog, and assign corresponding instance hints e to the cat and dog. cat and e dog , the image feature r in the area where the cat is located cat Add e cat ,, similarly, for the image feature r of the area where the dog is located dog Will add e dog , generating the corresponding second image features.

[0245] c3, obtaining a third image feature of the second sub-region that does not contain any object instance in the initial image feature, and adding a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0246] Corresponding to the first sub-region where the object instance described in the aforementioned step c3 is located, the second sub-region refers to a region that does not contain any object instance, which may be a background region on an image containing an object instance, or may be an image region where no object instance exists in the entire image.

[0247] After the third image feature of the second sub-region is separated from the initial image feature, in order to improve the attention layer's ability to understand and generate background information, a learnable empty instance hint is added to the third image feature. The empty instance hint is a special vector that indicates that the current processing is a background region that does not contain an object instance. It will be learned and optimized during the training process to better distinguish and generate background information. The fourth image feature obtained in this way not only contains the visual information of the second sub-region, but also contains additional hint information indicating that it does not contain an object instance.

[0248] c4, performing linear transformation on the second image feature and the fourth image feature respectively according to the scene-controllable and learnable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region;

[0249] In the self-attention mechanism, the preconfigured weight matrix includes W Q , WK and W V A matrix is ​​used to convert the input feature vector into a query vector, a key vector, and a value vector, and the weight matrix is ​​learned by the back propagation algorithm during the model training process. The linear transformation of the second image feature and the fourth image feature according to the pre-configured weight matrix can be implemented by a pre-configured linear projection layer, which is used to convert the input data into a query, key, and value vector by applying a linear transformation. For example, the linear projection layer can be expressed as Then performing a linear transformation on the second image feature to obtain the target query, target key and target value can be expressed as:

[0250]

[0251] Among them, r (i) represents the first image feature of the first subregion of the i-th object instance of category k, r (i) =a (i) +e k represents the second image feature after adding category hints, represents the linear projection layer shared across object instances. Based on this, the self-attention output of the first sub-region obtained by applying the self-attention mechanism can be expressed as:

[0252]

[0253] Similarly, The self-attention output of the second sub-region obtained by applying the self-attention mechanism can be expressed as:

[0254]

[0255] c5, obtaining the predicted scene image generated by the second network based on the self-attention output of the first sub-region and the second sub-region, comparing the predicted scene image with the set scene corresponding to the set condition data to calculate a fourth loss function, and minimizing the fourth loss function to adjust the network parameters of the second network to complete the controllability training of the image generation scene.

[0256] In order to optimize and enhance the feature representation of the instance box of the object instance (i.e., the bounding box of the object instance) and ensure the consistency and controllability of the instance box in the process of generating synthetic data, this embodiment adds a step of optimizing the features of the instance box by using a gated self-attention mechanism on the basis of adding a category prompt to the area where the object instance is located. The optimization of the features of the instance box by using a gated self-attention mechanism can be achieved by the following steps d1-d3:

[0257] d1, obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector;

[0258] The bounding box of the object instance is a rectangular box used to locate the position of an object instance in the image. It can be defined by four coordinate values ​​to accurately indicate the position and size of the object. Obtaining the bounding box of the object instance is essentially obtaining the position information of the bounding box. Each object instance is classified into a specific category, such as "dog", "cat", "car", etc. The category information is used to distinguish different object instances.

[0259] Hidden vectors are usually used in computer vision and deep learning to store additional information about data, which is information that needs to be specially encoded during the reasoning process. In this embodiment, the hidden vector is used to store additional information of each object instance, such as the category of the instance and the location information of the bounding box or other related features, to capture the structure and relationship of the instance boxes of different object instances, so as to achieve fine control over specific instances when generating synthetic data.

[0260] For example, the position information can be converted into a first representation vector by Fourier embedding or position encoding, machine learning, etc., and the category of the instance object can be converted to generate the second representation vector by text embedding representation or pre-trained models such as CLIP (Contrastive Language-Image Pre-training). After obtaining the second representation vector and the second representation vector, the two vector representations are concatenated using a [,] connection operation to obtain a combined vector. Regarding the use of a nonlinear transformation of a neural network to convert the combined vector into a hidden vector, a multi-layer perceptron can be used to map the combined vector to generate a fixed-length vector representation as the hidden vector.

[0261] d2, feature-connecting the hidden vector with the initial image feature to obtain a first feature representation, linearly transforming the first feature representation according to the weight matrix, applying the self-attention mechanism to obtain the self-attention output of the first feature representation, and applying a visual tag selection operation and a gating mechanism to the self-attention output of the first feature representation to generate the merged image feature; the visual tag selection operation is used to select the tag that is most relevant to the task requirement from the self-attention output of the first feature representation.

[0262] The gating mechanism can adjust the contribution of the output result of the visual marker selection operation to the initial image feature through learnable gating parameters and hyperbolic tangent functions, that is, the above-mentioned application of the visual marker selection operation and the gating mechanism to the self-attention output of the first feature representation can be implemented as follows: applying the visual marker selection operation to the self-attention output of the first feature representation to obtain a second feature representation; applying the gating mechanism to the second feature representation, and adding the corresponding gating mechanism output to the initial image feature to generate the merged image feature.

[0263] Based on the above hidden vector and gating mechanism implementation, the process of constructing the merged image features according to the hidden vector and the initial image features can be expressed by the formula:

[0264]

[0265] Wherein, x on the right side of the formula represents the initial image feature; η is a learnable parameter used to adjust the degree of fusion between the initial image feature x and the output result of TS(); tanh() represents the hyperbolic tangent function; TS() represents the visual marker selection operation; () represents the self-attention mechanism; [,] represents the feature vector concatenation operation; H b Represents a hidden vector. When using fourier() embedding, text embedding representation, and multi-layer perceptron to obtain a hidden vector, the process of obtaining the hidden vector can be expressed as:

[0266]

[0267] Among them, MLP() represents the mapping process of the multi-layer perceptron, and the bounding box Can be a 2D box or a 3D box, bounding box type Where N is the number of instance boxes of the object instance.

[0268] To better understand the above hidden vectors and merged image features, we will take the input scenario of "the image in front of the vehicle, including a traffic light and a pedestrian" as an example. bIt refers to the hidden vector containing the position information and category information of the bounding box of traffic lights and pedestrians; the initial image feature X on the right side of the formula represents the image feature encoding of the input scene, which contains the visual information of the background area and all object instances; TS() is used to select the visual mark that is most relevant to the current task (such as optimizing the features of pedestrians or traffic lights) from the output of the self-attention mechanism. For example, if the current task is to optimize the position of pedestrians in the scene, then TS() will be more inclined to select marks associated with pedestrian features; X, which is finally output by the gated self-attention mechanism, is a scene graph feature encoding for the entire scene that contains the optimization results of traffic lights and pedestrian features, which more accurately expresses the relative position of pedestrians in the scene and the state of traffic lights, ensuring that the position, category and fusion of all instance boxes in the scene with the background are better optimized, thereby ensuring the controllability and accuracy of details of the generated scene.

[0269] In the case of an instance box containing multiple object instances, each instance box has a corresponding hidden vector. In this step, the hidden vectors of all instance boxes are first integrated together, and then the integration result is spliced ​​with the initial image feature. The weight matrix used for linear transformation in this embodiment is the same as the weight matrix used for linear transformation in the aforementioned step c4, and the same linear transformation is used.

[0270] d3, obtain the predicted scene image generated by the second network according to the self-attention output of the first sub-region and the second sub-region, compare the predicted scene image with the set scene corresponding to the set condition data to calculate the fourth loss function, and minimize the fourth loss function to adjust the network parameters of the second network to complete the controllability training of the image generation scene.

[0271] In the disclosed embodiments, the bounding box of the object instance is encoded together with the category information into a hidden vector, so that different object instances can be identified and distinguished when processing different object instances. At the same time, the self-attention mechanism is used to focus on the overall layout and contextual information in the initial image features, thereby enhancing the richness of the feature representation of each object instance. The gating mechanism is combined to dynamically adjust the attention to different object instance features, thereby achieving precise control of the instance frame, and maintaining consistency between the categories of the object instances and the layout and position of the instance frames in the subsequent feature integration and decoding process.

[0272] After completing the above-mentioned scene controllability training based on the set condition data, in order to improve the scene controllability of the second network in image generation, the following enhancement steps can be continued: for object instances of different categories in the set condition data, the category prompts of the object instances can be dynamically adjusted (such as changing color, shape or position), and for the case where the predicted image data output by the second network does not match the category prompts of the adjusted object examples, a penalty term is added to the fourth loss function to enhance the network's sensitivity to condition changes; alternatively, a plurality of real scene images containing different object instances and layouts can be selected as training samples, and the set condition data can be modified by gradually changing the object instances or the overall layout in the real scene images to train the second network to accurately generate intermediate scene images after changing the object instances or the overall layout, and maintain the continuity and logic between the scenes before and after the change.

[0273] In some embodiments, for the training of the joint prediction fusion module described in the aforementioned step S103, the network parameters of the joint prediction fusion module can be adjusted by minimizing the loss function between the predicted value and the true value, using back propagation or other methods; or, the network parameter adjustment can also be achieved by introducing reinforcement learning, and self-evolutionary learning through interaction with the environment. In this way, the synthetic data generation network optimizes the behavior of the joint prediction fusion module by predicting future states, evaluating actions, and updating strategies through environmental feedback. That is, after the extracted features extracted by the pre-trained multimodal large language network and the trained multimodal generation network are input into the joint prediction module, the training of the joint prediction module can be achieved through the following steps e1-e3:

[0274] e1, respectively standardizing the extracted features extracted by the multimodal large language network and the multimodal generative network to obtain the first extracted features of the multimodal large language network and the second extracted features of the trained multimodal generative network;

[0275] This embodiment provides a joint prediction fusion module to make a fusion decision on the output features of the multimodal generation network and the output features of the multimodal large language network, and finally outputs the synthetic data expected by the user through decoders defined by different tasks. In order to ensure that the output features of the two networks are comparable on a unified scale, the output features of the two networks need to be standardized or normalized, which can be achieved by subtracting the mean and dividing by the standard deviation, that is:

[0276]

[0277] Among them, O mwm Represents the output features of the multimodal generation network, O' mwm is the corresponding second extracted feature, O mlm Represents the output features of the multimodal large language network, O'mlm The corresponding first extracted features, μ, σ, correspond to the mean and standard deviation of the output features of the multimodal generative network and the multimodal large language network, respectively.

[0278] e2, fusing the first extracted feature with the second extracted feature through a learnable weight parameter to obtain a fused feature;

[0279] The fusion process can be expressed by the following formula:

[0280] F final =β·O' mwm +(1-β)·O' mlm

[0281] Among them, β is a learnable weight parameter used to balance the importance of the second extracted features output by the multimodal generative network and the first extracted features output by the multimodal large language network.

[0282] e3. Apply reinforcement learning to the joint prediction fusion module to adjust network parameters of the joint prediction fusion module according to the fusion features.

[0283] Reinforcement learning frameworks usually involve the interaction between agents and environments. Agents affect the state of the environment by performing actions and receive rewards or penalties as feedback from these interactions. After obtaining the fusion features, the synthetic data generation network W is based on the current state s t 、Current action a t And the fusion feature F final , try to predict the next state s through some form of neural network architecture such as recurrent neural network, Mamba, etc. t+1 and expected reward t+1 , fusion feature F final As the input to the agent decision to enhance its understanding and prediction ability of the environment, it can be expressed as:

[0284]

[0285] The current action refers to any decision or behavior taken by the agent given the current state and fusion features. It is not limited to control signals in the traditional physical world (such as vehicle speed and direction control), but can also include but is not limited to the prediction of video frames, the generation of future image sequences, etc. In the scenario of video prediction or future frame generation, the current action may represent a certain strategy, algorithm selection, parameter adjustment, or activation mode change of a certain layer in the network when generating future frames. The current action is used to optimize the agent's ability to predict the environment, that is, to improve the accuracy and quality of generating future frames. Then the current action can be regarded as the agent's current state s at time t. t And the fusion feature F finalA specific instance of a set of decisions or choices made.

[0286] The current state is the agent's perception or understanding of the environment at a specific point in time (t), and contains all the information the agent needs to make decisions. In this scenario, the current state can include but is not limited to the current fused features, past states (such as predictions or actual observations at previous time steps), the agent's internal state (such as network parameters, hidden layer activations, etc.), and any environmental factors that may affect future predictions. In the context of video prediction or future frame generation, the state may also contain historical information about the video sequence, detailed features of the current frame, and any external factors relevant to predicting future frames. The current action and state together define how the agent interacts with the environment and determine how the agent optimizes its prediction performance through learning.

[0287] Select action a t According to the current strategy π(a t |s t ,F final ), the strategy can be adjusted based on the fusion features and the current state; it should be noted that the action does not only refer to the signal that can be executed by the embodied carrier such as the vehicle or robot, but also includes the future evolution law in the preset world scene, that is, the signal of the future video image frame.

[0288] The next state refers to the new state of the environment after the agent performs the current action. In the context of the joint prediction fusion module, this new state may include future video frames, images, or other forms of data predicted by the synthetic data generation network W based on the current state, current action, and fused features. The predicted data represents the possible state of the environment at a certain point in the future.

[0289] The expected reward is the immediate feedback that the agent gets from the environment after performing an action, which is used to measure the quality of the action. In the scenario of the joint prediction fusion module, the expected reward may be calculated based on a variety of factors, such as the similarity between the predicted future frame and the actual observed frame, the reduction in prediction error, the quality of the generated data, etc. The expected reward is a key signal in reinforcement learning, guiding the agent to learn how to choose actions to maximize the long-term cumulative reward. During training, the agent tries to find a policy that maximizes the cumulative reward it receives after performing a series of actions.

[0290] L = UpdatePolicy((s t ,a t ,r t+1 ,s t+1 ),π)

[0291] Among them, UpdatePolicy is an update operation for the policy, which optimizes the world model by interacting with the world in real time. The performance of the current policy is evaluated by calculating the cumulative reward or loss function, and the policy is updated using reinforcement learning algorithms (such as Q learning, policy gradient methods, etc.) to select better actions in the future. Since the joint prediction fusion module is a key part of generating the next state and expected reward, its parameters are also adjusted as the policy is updated. This can be achieved through the back-propagation algorithm, where the gradient of the loss function with respect to the prediction error and reward is used to update the network weights. The above process will be repeated until the agent learns an effective strategy that maximizes the cumulative reward it obtains after performing a series of actions. At the same time, the network parameters of the joint prediction fusion module will be gradually optimized to improve prediction accuracy and overall performance.

[0292] Through the above steps, this embodiment can model the synthetic data generation network based on the feature fusion of the multimodal generation model and the multimodal large language model, combining reinforcement learning and self-supervised learning paradigms, and realize self-evolutionary learning that interacts with the environment. It can predict future scenes based on user input or scenes that have occurred, and conform to the logical laws of the physical world. Specifically, this embodiment not only utilizes the infinite spatial imagination of the world by the multimodal generation model, but also utilizes the cognitive ability of the multimodal large language network for universal things in the world, and through continuous interaction with the environment to optimize the strategy, while improving the network's prediction ability, it prevents the network from generating scenes and operations that do not conform to the physical world and logic.

[0293] In the field of machine learning and deep learning, the adjustment of model parameters usually involves a variety of methods, including gradient descent through loss functions and policy optimization through reinforcement learning. Parameter adjustment through loss functions is the most common and basic parameter adjustment method in deep learning. During the training process, the model calculates the loss function based on the difference between the input information and the predicted output. Through the back-propagation algorithm, the gradient of the loss function is propagated back to each layer of the model, and the model parameters are updated according to the gradient to minimize the loss function. Introducing reinforcement learning for policy update is a method of learning the optimal strategy through the interaction between the agent and the environment. In the context of machine learning, the environment can be the model itself or the data set processed by the model, and the agent is the algorithm or system used to select model parameters. In reinforcement learning, the agent will choose an action based on the current state, then observe the impact of this action on the environment, and update the strategy based on the reward obtained.

[0294] In practical applications, the choice of adjustment method depends on the specific application scenario and requirements: when there is a large amount of labeled data and you want to quickly optimize the model performance, you can use the loss function to adjust the parameters; when the labeled data is limited, or you want to explore a better combination of model parameters through trial and error and reward mechanisms, you can introduce reinforcement learning to update the strategy. In addition, you can also consider combining the two methods to take full advantage of their respective advantages. For example, in the early stages of model training, you can use the loss function adjustment method to quickly optimize the model performance; in the later stages of training, you can introduce reinforcement learning to further explore a better combination of parameters.

[0295] In some embodiments, in order to improve the processing capability and fusion effect of the multimodal large language network for complex information, this embodiment further introduces a cross attention layer after normalizing the output features of the two networks respectively, and the above embodiment may also include the following steps:

[0296] Using the first extracted feature as a third key vector and a third value vector, and using the second extracted feature as a third query vector; obtaining a cross-attention output of the first extracted feature by applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector;

[0297] That is, the first extracted feature of the multimodal large language network is applied with the cross attention mechanism to obtain the cross attention output which can be expressed by the following formula:

[0298]

[0299] Among them, O' mwm The second extracted feature of the multimodal generation network, O mlm The output feature represented by O' mlm It is the first extracted feature of the multimodal large language network. Adding the cross attention mechanism can further improve the interaction and fusion effect of multimodal features, ensuring that each feature position of the multimodal generative network obtains a normalized attention weight distribution, pointing to different parts of the first extracted feature output by the large language network.

[0300] The first extracted features based on the output of the multimodal large language network are further aggregated into cross-attention outputs. The fusing of the first extracted features with the second extracted features through learnable weight parameters described in the aforementioned embodiment can be updated to: fusing the cross-attention output with the second extracted features through learnable weight parameters.

[0301] Corresponding to the embodiment of the training method of the aforementioned synthetic data generation network, see Figure 4As shown, the present application also provides an embodiment of a training device for a synthetic data generation network, wherein the synthetic data generation network at least includes a multimodal generation network, a multimodal large language network, and a joint prediction fusion module; the joint prediction fusion module is used to fuse the output features of the multimodal generation network and the multimodal large language network; the device includes:

[0302] A first training module 401 is used to perform cross-view consistency training on the multimodal generative network using scene data of multiple different viewpoints until the training converges to obtain a first network; the scene data at least includes description data of geometric layout information of the same scene;

[0303] A second training module 402 is used to perform spatiotemporal consistency training on the first network using a multi-frame scene data sequence until the training converges, thereby obtaining a trained multimodal generation network, and freezing network parameters of the multimodal generation network;

[0304] The third training module 403 is used to obtain sample demand data corresponding to different synthetic data generation task requirements, and use the trained multimodal generation network and the pre-trained multimodal large language network in a frozen state to extract features from the sample demand data respectively, and input the extracted features into the joint prediction fusion module, perform feature fusion training on the joint prediction fusion module until the training converges, and obtain a trained synthetic data generation network.

[0305] In some embodiments, the first training module is specifically used to: obtain first training samples, each first training sample includes at least a single frame of scene data of the same scene collected from multiple perspectives; input the first training samples into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network; calculate a first loss function based on the predicted images of the plurality of corresponding perspectives and the corresponding first training samples, and adjust the network parameters of the multimodal generation network by minimizing the first loss function; the first loss function includes at least a loss within a single perspective and a consistency loss across perspectives.

[0306] In some embodiments, the first training module, when used to input the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network, includes:

[0307] The multimodal generative network is used to extract the potential features corresponding to the single-frame scene data collected from each perspective respectively; the potential features of each perspective are used as the first query vector, and the connection of the potential features of its two adjacent perspectives is used as the first key vector and the first value vector, and a cross-perspective attention output of each perspective is obtained by performing a linear transformation through a cross-perspective learnable matrix; according to the cross-perspective attention output of each perspective, a predicted image of the corresponding perspective is generated.

[0308] In some embodiments, the first training module, when used to generate a predicted image of a corresponding perspective based on the cross-perspective attention output of each perspective, includes: obtaining the cross-perspective joint distribution conditional probability of the ith perspective based on the predicted images of the i-1 perspectives that have been generated; the i is a positive integer greater than 1; and generating the predicted image of the ith perspective based on the predicted images of the i-1 perspectives and the cross-perspective attention output of the ith perspective, combined with the cross-perspective joint distribution conditional probability.

[0309] In some embodiments, before the first training module inputs the first training sample into the multimodal generation network, the device also includes: acquiring single-view scene data, and inputting it into the multimodal generation network for image generation task training, and adjusting the network parameters of the multimodal generation network by minimizing a second loss function; when the single-view scene data is image type data, the second loss function at least includes pixel-level reconstruction loss; when the single-view scene data is text type data, the second loss function at least includes semantic loss.

[0310] In some embodiments, the second training module is specifically used for:

[0311] Acquire second training samples, each second training sample at least including an input sequence consisting of multiple frames of scene data of the same scene that changes continuously over time; input the second training samples into the first network, and based on the task type set for each second training sample, obtain a multi-frame predicted image sequence generated by the first network that meets the task type; calculate a third loss function based on the multi-frame predicted image sequence and the multi-frame real image sequence corresponding to the second training sample, and adjust the network parameters of the first network by minimizing the third loss function; the third loss function at least includes pixel-level reconstruction loss, cross-view consistency loss, and motion consistency loss.

[0312] In some embodiments, when the second training module is used to obtain a multi-frame predicted image sequence generated by the first network that meets the task type, the second training module includes:

[0313] Utilize the first network to extract the potential features of each frame of scene data in the input sequence respectively; use the potential features of the target frame scene data as the second query vector, use the connection of the potential features of the first frame scene data in the input sequence and the next frame of the target frame scene data as the second key vector, and use the connection of the potential features of the first frame scene data and the previous frame of the target frame scene data as the second value vector; the target frame scene data refers to any frame of scene data in the input sequence; linearly transform the second query vector, the second key vector and the second value vector through a cross-frame learnable matrix to obtain the cross-frame attention output of the target frame scene data; generate the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence.

[0314] In some embodiments, the second training module is used to generate the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence, including:

[0315] In the case of generating a future multi-frame prediction image sequence based on the second training sample, the inter-frame joint distribution conditional probability of the j-th frame prediction image to be generated is obtained according to the input sequence and the j-1 frame prediction image that has been generated; j is a positive integer greater than 1; according to the cross-frame attention output of each frame of scene data, combined with the inter-frame joint distribution conditional probability, the j-th frame prediction image is generated in an autoregressive manner.

[0316] In some embodiments, after completing the spatiotemporal consistency training of the first network, the apparatus further comprises:

[0317] A scene controllable training module is used to obtain set condition data for guiding the output of an image generation task of a second network; the second network refers to a network obtained by completing spatiotemporal consistency training on the first network; the set condition data includes text data describing a set scene, or any one of text and image combination data; the second network is trained for scene controllability of image generation using the set condition data, and the network parameters of the second network are adjusted by minimizing a fourth loss function to obtain the trained multimodal generation network; the loss terms of the fourth loss function include at least pixel-level image reconstruction loss, cross-view consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss.

[0318] In some embodiments, when the scene controllability training module is used to perform image generation scene controllability training on the second network using the set condition data, it includes:

[0319] The second network is used to extract the initial image features corresponding to the set condition data; the first image features of the first sub-region where each object instance is located in the initial image features are obtained, and the learnable category prompt of the object instance is added to the first image features to obtain the second image features; the third image features of the second sub-region that does not contain any object instance in the initial image features are obtained, and the learnable empty instance prompt is added to the third image features to obtain the fourth image features; the second image features and the fourth image features are linearly transformed according to the scene controllable and learnable weight matrix, and the self-attention mechanism is applied to obtain the self-attention output of the first sub-region and the self-attention output of the second sub-region; the predicted scene image generated by the second network according to the self-attention output of the first sub-region and the second sub-region is obtained, the predicted scene image is compared with the set scene corresponding to the set condition data to calculate the fourth loss function, and the network parameters of the second network are adjusted to minimize the fourth loss function to complete the controllability training of the image generation scene.

[0320] In some embodiments, the third training module is specifically used to:

[0321] The extracted features extracted by the multimodal large language network and the multimodal generative network are respectively standardized to obtain a first extracted feature of the multimodal large language network and a second extracted feature of the trained multimodal generative network; the first extracted feature and the second extracted feature are fused through a learnable weight parameter to obtain a fused feature; and according to the fused feature, reinforcement learning is applied to the joint prediction fusion module to adjust the network parameters of the joint prediction fusion module.

[0322] In some embodiments, after the second training module performs standardization processing on the extracted features respectively, the device further comprises:

[0323] Using the first extracted feature as a third key vector and a third value vector, and using the second extracted feature as a third query vector; obtaining a cross-attention output of the first extracted feature by applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector;

[0324] The fusing the first extracted feature with the second extracted feature through a learnable weight parameter includes: fusing the cross-attention output with the second extracted feature through a learnable weight parameter.

[0325] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0326] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Figure 5 As shown, the electronic device 500 includes at least one processor 501, a memory 502 and a bus 503, and the at least one processor 501 is electrically connected to the memory 502; the memory 502 is configured to store at least one computer-executable instruction, and the processor 501 is configured to execute the at least one computer-executable instruction, thereby executing the steps of any method for training a synthetic data generation network provided in any embodiment or any optional implementation manner in the present application.

[0327] Furthermore, the processor 501 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0328] An embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any method for training a synthetic data generation network provided in any embodiment or any optional implementation manner of the present application when executed by a processor.

[0329] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0330] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for training a synthetic data generation network, characterized in that: The synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module, wherein the joint prediction fusion module is used to fuse the output features of the multimodal generation network and the multimodal large language network; the method includes: Using scene data from multiple different perspectives, the multimodal generative network is trained for cross-perspective consistency until the training converges to obtain a first network; the scene data includes image data or description data of the same scene; Using a multi-frame scene data sequence, performing spatiotemporal consistency training on the first network until the training converges to obtain a trained multimodal generation network, and freezing the network parameters of the multimodal generation network; According to the sample demand data corresponding to different synthetic data generation tasks, the trained multimodal generation network and the pre-trained, frozen multimodal large language network are used to extract features of the sample demand data respectively, and the extracted features are input into the joint prediction fusion module together, and the joint prediction fusion module is trained until the training converges to obtain a trained synthetic data generation network.

2. The method according to claim 1, characterized in that The method uses scene data from multiple different perspectives to perform cross-perspective consistency training on the multimodal generation network until the training converges to obtain a first network, including: Acquire first training samples, each of which includes at least single-frame scene data of the same scene collected from multiple perspectives; Inputting the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding perspectives generated by the multimodal generation network; According to the predicted images of the multiple corresponding perspectives and the corresponding first training samples, a first loss function is calculated, and the network parameters of the multimodal generation network are adjusted by minimizing the first loss function; the first loss function includes at least a loss within a single perspective and a consistency loss across perspectives.

3. The method according to claim 2, characterized in that The step of inputting the first training sample into the multimodal generation network to obtain a plurality of predicted images of corresponding viewing angles generated by the multimodal generation network includes: Utilizing the multimodal generative network, respectively extracting potential features corresponding to the single-frame scene data collected from each viewing angle; The latent feature of each view is used as the first query vector, and the connection of the latent features of its two adjacent views is used as the first key vector and the first value vector. The cross-view attention output of each view is obtained by performing linear transformation through the cross-view learnable matrix; According to the cross-view attention output of each view, the predicted image of the corresponding view is generated.

4. The method according to claim 3, characterized in that The step of generating a predicted image of a corresponding perspective according to the cross-perspective attention output of each perspective includes: According to the predicted images of the i-1 perspectives that have been generated, the cross-perspective joint distribution conditional probability of the i-th perspective is obtained; i is a positive integer greater than 1; According to the predicted image of the i-1 perspective and the cross-perspective attention output of the i-th perspective, combined with the cross-perspective joint distribution conditional probability, the predicted image of the i-th perspective is generated.

5. The method according to claim 2, characterized in that: Before inputting the first training sample into the multimodal generation network, the method further includes: Obtaining single-view scene data and inputting it into the multimodal generation network for image generation task training, and adjusting the network parameters of the multimodal generation network by minimizing the second loss function; When the single-view scene data is image type data, the second loss function includes at least pixel-level reconstruction loss; when the single-view scene data is text type data, the second loss function includes at least semantic loss.

6. The method according to claim 1, characterized in that The method of using a multi-frame scene data sequence to perform spatiotemporal consistency training on the first network until the training converges to obtain a trained multimodal generation network includes: Acquire second training samples, each second training sample at least comprising an input sequence consisting of multiple frames of scene data of the same scene that continuously changes over time; Inputting the second training sample into the first network, and obtaining a multi-frame predicted image sequence generated by the first network and conforming to the task type based on the task type set for each second training sample; According to the multi-frame predicted image sequence and the multi-frame real image sequence corresponding to the second training sample, a third loss function is calculated, and the network parameters of the first network are adjusted by minimizing the third loss function; the third loss function includes at least pixel-level reconstruction loss, cross-view consistency loss and motion consistency loss.

7. The method according to claim 6, characterized in that The obtaining of a multi-frame predicted image sequence generated by the first network and conforming to the task type includes: Using the first network, respectively extracting potential features of each frame of scene data in the input sequence; The latent features of the target frame scene data are used as a second query vector, the connection of the first frame scene data in the input sequence and the latent features of the next frame of the target frame scene data is used as a second key vector, and the connection of the first frame scene data and the latent features of the previous frame of the target frame scene data is used as a second value vector; the target frame scene data refers to any frame scene data in the input sequence; Linearly transforming the second query vector, the second key vector, and the second value vector through a cross-frame learnable matrix to obtain a cross-frame attention output of the target frame scene data; The multi-frame predicted image sequence is generated according to the cross-frame attention output of each frame of scene data in the input sequence.

8. The method according to claim 7, characterized in that The step of generating the multi-frame predicted image sequence according to the cross-frame attention output of each frame of scene data in the input sequence comprises: In the case of generating a future multi-frame prediction image sequence according to the second training sample, obtaining an inter-frame joint distribution conditional probability of a j-th prediction image to be generated according to the input sequence and the j-1-th prediction image that has been generated; wherein j is a positive integer greater than 1; According to the cross-frame attention output of each frame of scene data and combined with the inter-frame joint distribution conditional probability, the j-th frame prediction image is generated in an autoregressive manner.

9. The method according to claim 1, characterized in that: After completing the spatiotemporal consistency training of the first network, the method further includes: Acquire setting condition data for guiding the output of the image generation task of the second network; the second network refers to a network obtained by completing the spatiotemporal consistency training of the first network; the setting condition data includes at least one of text data describing the setting scene, or text and image combined data; The second network is trained for controllability of image generation scenarios using the set condition data, and the network parameters of the second network are adjusted by minimizing a fourth loss function to obtain the trained multimodal generation network; the loss terms of the fourth loss function include at least pixel-level image reconstruction loss, cross-view consistency loss, motion consistency loss, style loss, semantic consistency loss, and cycle consistency loss.

10. The method according to claim 9, characterized in that The using the set condition data to perform image generation scene controllability training on the second network includes: extracting initial image features corresponding to the set condition data using the second network; Acquire a first image feature of a first subregion where each object instance in the initial image feature is located, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature; Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature; Performing linear transformation on the second image feature and the fourth image feature respectively according to the scene-controllable and learnable weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region and a self-attention output of the second sub-region; Obtain a predicted scene image generated by the second network based on the self-attention output of the first sub-region and the second sub-region, compare the predicted scene image with the set scene corresponding to the set condition data to calculate a fourth loss function, and minimize the fourth loss function to adjust the network parameters of the second network to complete the controllability training of the image generation scene.

11. The method according to claim 1, characterized in that: The step of inputting the extracted features into the joint prediction fusion module and training the joint prediction fusion module until the training converges includes: Standardizing the extracted features extracted by the multimodal large language network and the multimodal generative network respectively to obtain a first extracted feature of the multimodal large language network and a second extracted feature of the trained multimodal generative network; Fusing the first extracted feature with the second extracted feature through a learnable weight parameter to obtain a fused feature; According to the fusion features, reinforcement learning is applied to the joint prediction fusion module to adjust network parameters of the joint prediction fusion module.

12. The method according to claim 11, characterized in that After the extracted features are respectively standardized, the method further comprises: Using the first extracted features as a third key vector and a third value vector, and using the second extracted features as a third query vector; Obtaining a cross-attention output of the first extracted features by applying a cross-attention mechanism to the third query vector, the third key vector, and the third value vector; The fusing the first extracted feature with the second extracted feature by using a learnable weight parameter comprises: The cross-attention output is fused with the second extracted feature through a learnable weight parameter.

13. A synthetic data generation network training device, characterized in that: The synthetic data generation network at least includes a multimodal generation network, a multimodal large language network and a joint prediction fusion module; The joint prediction fusion module is used to fuse the output features of the multimodal generation network and the multimodal large language network; The device comprises: A first training module is used to perform cross-view consistency training on the multimodal generative network using scene data of multiple different viewpoints until the training converges to obtain a first network; the scene data at least includes description data of geometric layout information of the same scene; A second training module is used to perform spatiotemporal consistency training on the first network using a multi-frame scene data sequence until the training converges, thereby obtaining a trained multimodal generation network, and freezing network parameters of the multimodal generation network; The third training module is used to obtain sample demand data corresponding to different synthetic data generation task requirements, and use the trained multimodal generation network and the pre-trained multimodal large language network in a frozen state to extract features from the sample demand data respectively, and input the extracted features into the joint prediction fusion module, perform feature fusion training on the joint prediction fusion module until the training converges, and obtain a trained synthetic data generation network.

14. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-12.

15. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images