Method and device for training model, and method for generating driving scene video

Abstract

The embodiment of the disclosure discloses a processing method and a training method and device of a model for generating a driving scene video, wherein the method comprises: acquiring multi-modal scene data; based on the multi-modal scene data, generating scene two-dimensional features corresponding to each view angle respectively and scene environment prediction information of a future preset frame number; based on the scene two-dimensional features corresponding to each view angle respectively and the scene environment prediction information, generating a driving scene video by using a video generation model. The embodiment of the disclosure uses the scene environment prediction information as a conditional signal to constrain the generated driving scene video, so that the generated driving scene video is more consistent with the conditions such as the road type, the traffic rules and the vehicle behavior of the driving scene, thereby helping to improve the accuracy and reliability of the driving scene video.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a processing method, a model training method, and an apparatus for generating driving scene videos. Background Technology

[0002] With the continuous development of Artificial Intelligence (AI) technology, AI-generated data technology has made significant progress. From initial text generation to current text-to-image, text-to-video, and text-to-3D generation technologies, AI's generative capabilities are constantly expanding, laying a solid foundation for more natural and intuitive interaction between humans and computers. Compared to traditional AI generation work, autonomous driving data generation faces more complex challenges. In the field of autonomous driving, generated data not only needs to meet the conversion requirements from text to images, videos, or 3D scenes, but also must comply with stringent driving scenario simulation requirements. Driving scenario simulation requirements can include complex conditions such as various weather conditions, road types, traffic rules, and vehicle behavior to provide high-quality training data for optimizing the performance of autonomous driving systems.

[0003] In related technologies, driving scene videos are typically generated based on Video-LDM (Video-Latent Diffusion Models). This involves using a diffusion model to generate driving scene videos from the first frame of the driving scene image, providing diverse training data for autonomous driving systems. Video-LDM video generation methods usually utilize existing autonomous driving video data to train a model capable of generating driving scene videos. However, the generated driving scene videos may not meet the simulation requirements of real-world driving scenarios. Summary of the Invention

[0004] This disclosure provides a processing method, a model training method, and an apparatus for generating driving scene videos, so as to improve the accuracy and reliability of the generated driving scene videos.

[0005] One aspect of this disclosure provides a processing method for generating driving scene videos, including:

[0006] Acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of the following: scene bird's-eye view, scene environment description text data, and vehicle control signals; each frame of scene image data includes at least one scene image corresponding to each viewpoint;

[0007] Based on the multimodal scene data, generate two-dimensional scene features corresponding to each viewpoint and scene environment prediction information for a preset number of future frames;

[0008] Based on the two-dimensional features of the scene corresponding to each of the aforementioned viewpoints and the scene environment prediction information, a driving scene video is generated using a video generation model.

[0009] Another aspect of this disclosure provides a method for training a generative model for generating driving scene videos, comprising:

[0010] Acquire training sample data; the training sample data includes multiple first training samples and multiple second training samples;

[0011] Based on each of the first training samples, the scene environment prediction model to be trained is trained in the first stage to obtain the scene environment prediction model.

[0012] Based on the scene environment prediction model and the video generation model to be trained obtained in the first stage of training, the generation model to be trained is determined.

[0013] The second training phase is performed on the generator model to be trained based on each of the second training samples to obtain a generator model for generating driving scene videos; wherein, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generator model to be trained.

[0014] In another aspect of this disclosure, a processing apparatus for generating driving scene videos is provided, comprising:

[0015] A multimodal data acquisition module is used to acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of the following: scene bird's-eye view, scene environment description text data, and vehicle control signals; each frame of scene image data includes at least one scene image corresponding to each viewpoint;

[0016] The first processing module is used to generate scene two-dimensional features corresponding to each of the aforementioned viewpoints and scene environment prediction information for a preset number of future frames based on the multimodal scene data.

[0017] The second processing module is used to generate driving scene videos based on the two-dimensional features of the scene corresponding to each of the aforementioned viewpoints and the scene environment prediction information, using a video generation model.

[0018] In another aspect of this disclosure, a training apparatus for a generative model used to generate driving scene videos is provided, comprising:

[0019] The sample acquisition module is used to acquire training sample data; the training sample data includes multiple first training samples and multiple second training samples.

[0020] The first training module is used to perform a first-stage training on the scene environment prediction model to be trained based on each of the first training samples, so as to obtain the scene environment prediction model.

[0021] The second training module is used to determine the generation model to be trained based on the scene environment prediction model and the video generation model to be trained obtained in the first stage of training; and to perform a second stage of training on the generation model to be trained based on each of the second training samples to obtain a generation model for generating driving scene videos; wherein, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generation model to be trained.

[0022] In another aspect of the disclosed embodiments, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the methods described in any of the above embodiments of the present disclosure.

[0023] In another aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0024] Memory, used to store computer programs;

[0025] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the method described in any of the above embodiments of the present disclosure.

[0026] In another aspect of the present disclosure, a computer program product is provided, including computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method described in any of the above embodiments of the present disclosure.

[0027] The processing method, model training method, and apparatus for generating driving scene videos provided in this disclosure can integrate multimodal scene data to generate two-dimensional scene features from various perspectives, as well as structured scene environment prediction information for a preset number of future frames. This allows the scene environment prediction information for a preset number of future frames to be used as a conditional signal for generating the driving scene video. Based on the two-dimensional scene features from various perspectives, diffusion iterations are performed to generate the driving scene video. Because the scene environment prediction information is used as a conditional signal to constrain the generated driving scene video, the generated driving scene video is more consistent with the road type, traffic rules, vehicle behavior, and other conditions of the driving scene, thereby helping to improve the accuracy and reliability of the driving scene video.

[0028] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0029] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0030] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0031] Figure 1 This is an exemplary application scenario of the processing method for generating driving scene videos provided in this disclosure;

[0032] Figure 2 This is a flowchart illustrating a method for generating driving scene videos provided by some exemplary embodiments of this disclosure;

[0033] Figure 3 This is a schematic diagram of the process for generating two-dimensional scene features corresponding to each viewpoint, provided by some exemplary embodiments of this disclosure;

[0034] Figure 4 This is a schematic diagram of the encoding principle of a multimodal encoder provided in an exemplary embodiment of this disclosure;

[0035] Figure 5 This is a schematic diagram of the process for generating scene environment prediction information for a preset number of future frames, provided by some exemplary embodiments of this disclosure;

[0036] Figure 6 This is a flowchart illustrating step 220 provided in some optional embodiments of this disclosure;

[0037] Figure 7 This is a flowchart illustrating the process of predicting scene environment prediction information provided by some exemplary embodiments of this disclosure;

[0038] Figure 8 This is a schematic diagram of the network structure of a scene environment prediction model provided in an exemplary embodiment of this disclosure;

[0039] Figure 9 This is a schematic diagram of the process for generating scene environment prediction information for a preset number of future frames, provided by some other exemplary embodiments of this disclosure;

[0040] Figure 10 This is a flowchart illustrating a processing method for generating driving scene videos provided by some other exemplary embodiments of this disclosure;

[0041] Figure 11 This is a flowchart illustrating a method for generating driving scene videos provided in some exemplary embodiments of this disclosure;

[0042] Figure 12 This is a schematic diagram of the network structure of a generative model provided in an exemplary embodiment of this disclosure;

[0043] Figure 13 This is a flowchart illustrating a training method for a generative model used to generate driving scene videos, provided by some exemplary embodiments of this disclosure.

[0044] Figure 14 This is a flowchart illustrating a training method for a generative model used to generate driving scene videos, provided by some other exemplary embodiments of this disclosure.

[0045] Figure 15 This is a schematic diagram of the first-stage training process provided by some exemplary embodiments of this disclosure;

[0046] Figure 16 This is a schematic diagram of the second-stage training process provided by some exemplary embodiments of this disclosure;

[0047] Figure 17 This is a schematic diagram illustrating the training principle of a generative model to be trained provided in an exemplary embodiment of this disclosure;

[0048] Figure 18 This is a schematic diagram of the structure of a processing apparatus for generating driving scene videos provided in some exemplary embodiments of this disclosure;

[0049] Figure 19 This is a schematic diagram of the structure of a training apparatus for generating a generative model for generating driving scene videos, provided by some exemplary embodiments of this disclosure.

[0050] Figure 20 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0051] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0052] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0053] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0054] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0055] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0056] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0057] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0058] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0059] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0061] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0062] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0063] This disclosure outlines

[0064] In developing this disclosure, the inventors discovered that in the field of autonomous driving, driving scene videos are typically generated based on Video-LDM (Video-Latent Diffusion Models). This involves using a diffusion model to generate driving scene videos from the first frame of a driving scene image, providing diverse training data for autonomous driving systems. Video-LDM video generation methods usually utilize existing autonomous driving video data to train a model capable of generating driving scene videos. However, the generated driving scene videos may not meet the simulation requirements of real-world driving scenarios.

[0065] Exemplary Overview

[0066] Figure 1 This is an exemplary application scenario of the processing method for generating driving scene videos provided in this disclosure.

[0067] like Figure 1As shown, in the field of autonomous driving, the processing method for generating driving scene videos disclosed herein can generate two-dimensional scene features and scene environment prediction information for a preset number of future frames based on multimodal scene data and a pre-trained scene environment prediction model. Multimodal scene data can include scene data from various modalities such as multi-view video data, multi-view image data, scene bird's-eye views, scene environment description text, and vehicle control signals. Alternatively, multimodal scene data can include a portion of the aforementioned scene data, such as multi-view images or multi-view videos, and at least one of scene bird's-eye views, scene environment description text, and vehicle control signals; the specific details are not limited. The two-dimensional scene features for each viewpoint can include the scene features of that viewpoint. The scene environment prediction information for a preset number of future frames can include road environment prediction information and surrounding object motion information for a preset number of future frames. The road environment prediction information can include road map information such as lane lines, curbs, and traffic signs. The surrounding object motion information can include the motion trajectory information of surrounding objects, or the position, direction, speed, acceleration, and angular velocity of surrounding objects in each future frame, which can be specifically set according to actual needs. Furthermore, based on the two-dimensional features of the scene corresponding to each perspective and the scene environment prediction information, a video generation model is used to generate driving scene videos. The scene environment prediction information is used as a condition for generating driving scene videos, making the generated driving scene videos more in line with the simulation requirements of road types, traffic rules and vehicle behavior in driving scenarios, thereby improving the accuracy and reliability of driving scene videos.

[0068] Exemplary methods

[0069] Figure 2 This is a flowchart illustrating a method for generating driving scene videos provided by some exemplary embodiments of this disclosure. For example... Figure 2 As shown, the method includes the following steps:

[0070] Step 210: Obtain multimodal scene data.

[0071] The multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and also includes at least one of the following: scene bird's-eye view, scene environment description text data, and vehicle control signals; each frame of scene image data includes at least one scene image corresponding to each viewpoint.

[0072] In some optional embodiments, at least one frame of scene image data may include one or more frames of image data, or video data, or a combination of image and video data. A scene bird's-eye view refers to a view from the perspective of a BEV (Browser Electric Vehicle), which can unify the characteristics of the vehicle from different perspectives into a single viewpoint.

[0073] In some optional embodiments, at least one frame may include the Tth frame of the vehicle data acquisition (e.g., the current frame) or the (n+1)th frame from the Tnth frame to the Tth frame, where n is a positive integer. For example, at least one frame may include the current frame and n historical frames, used to generate driving scene video of the future M frames of the current frame, without any specific limitation.

[0074] In some optional embodiments, the scene environment description text data (Text) is scene environment information described through text, such as weather information, road information, and other environmental information. Examples include "rain" and "Drive in the countryside." Vehicle control signals (Command) are command signals that control the vehicle's movement, such as signals indicating vehicle speed, steering, etc.

[0075] In some alternative embodiments, vehicle control signals can be presented as text data.

[0076] Step 220: Based on multimodal scene data, generate two-dimensional scene features corresponding to each viewpoint and scene environment prediction information for a preset number of future frames.

[0077] The preset number of frames can be any number, such as 1 frame, 2 frames, 5 frames, 10 frames, etc., without any specific limitation. Scene environment prediction information can include road environment prediction information and motion information of objects around the vehicle. Road environment prediction information can include road map information such as lane lines, curbs, and traffic signs. The motion information of surrounding objects can include the motion trajectory information of surrounding objects, or the position, direction, speed, acceleration, angular velocity, etc. of surrounding objects in each future frame, without any specific limitation.

[0078] In some optional embodiments, multimodal features can be generated based on multimodal scene data, and then scene two-dimensional features corresponding to each viewpoint and scene environment prediction information for a preset number of future frames can be generated based on the multimodal features.

[0079] Step 230: Based on the two-dimensional features of the scene and the scene environment prediction information corresponding to each viewpoint, a driving scene video is generated using a video generation model.

[0080] The video generation model can be implemented based on any feasible model, such as a video diffusion model. It uses the two-dimensional features of the scene corresponding to each viewpoint as the initial signal, and the scene environment prediction information for a preset number of future frames as the conditional signal for generating the driving scene video. Based on the initial signal, it continuously diffuses and iterates to generate a driving scene video that conforms to the scene environment conditions for the preset number of future frames.

[0081] The processing method for generating driving scene videos provided in this disclosure can integrate multimodal scene data to generate two-dimensional scene features from each perspective and structured scene environment prediction information for a preset number of future frames. This allows the scene environment prediction information for a preset number of future frames to be used as a conditional signal for generating the driving scene video. Based on the two-dimensional scene features from each perspective, diffusion iteration is performed to generate the driving scene video. Since the scene environment prediction information is used as a conditional signal to constrain the generated driving scene video, the generated driving scene video is more consistent with the road type, traffic rules, vehicle behavior, and other conditions of the driving scene, thereby helping to improve the accuracy and reliability of the driving scene video.

[0082] In some alternative embodiments, Figure 3 This is a schematic diagram of the process for generating two-dimensional scene features corresponding to each viewpoint, provided by some exemplary embodiments of this disclosure, such as... Figure 3 As shown, step 220 generates two-dimensional scene features corresponding to each viewpoint based on multimodal scene data, including:

[0083] Step 2210: Encode the multimodal scene data using a multimodal encoder to obtain multimodal features.

[0084] Among them, the multimodal encoder can jointly encode data of various modalities in multimodal scene data to obtain multimodal features.

[0085] In some optional embodiments, the multimodal encoder can be a Transformer-based encoder. The encoding process may include: converting scene data from each modality into one-dimensional token information; combining the token information of the scene data from each modality to obtain a combination result; performing positional encoding on the combination result to obtain a first encoding result; performing Transformer encoding on the position-encoded first encoding result to obtain a second encoding result; and using the second encoding result as a multimodal feature. The Transformer encoding of the first encoding result can be implemented based on a Transformer encoder, which includes an attention layer that allows for attention interactions between multimodal data to better capture the correlation information between multimodal scene data, thereby obtaining more effective multimodal features. The specific structure of the multimodal encoder is not limited.

[0086] For example, Figure 4 This is a schematic diagram illustrating the encoding principle of a multimodal encoder provided in an exemplary embodiment of this disclosure. For example... Figure 4As shown, the multimodal scene data includes multi-view video data, multi-view image data, scene bird's-eye view (BEV), scene environment description text data, and vehicle control signals (Command). The multimodal encoder first preprocesses the multimodal scene data, such as performing patch decomposition on two-dimensional signals like video, images, and VEV to obtain one-dimensional token information. It then tokenizes the Text and Command data and converts them into one-dimensional token information through a linear mapping layer. Next, all tokens are combined, and a position encoder is used to encode the positions of the tokens from each modality. Position encoding helps capture the positional relationships between different tokens in subsequent encoding processes, thus improving encoding quality. The position encoding result is used as the first encoding result and input to the Transformer Encoder. The Transformer Encoder encodes the first encoding result to better capture the correlation information between the multimodal scene data, thereby obtaining more effective multimodal features.

[0087] Step 2220: Extract the visual features corresponding to the scene image data from the multimodal features.

[0088] Visual features can be extracted from multimodal features based on the positional encoding order. The Transformer encoder encodes the positional encoding results primarily to enable interaction between multimodal features. For example, for the positional encoding result of any modality, information from other modalities is fused into that modality, and vice versa. In the resulting multimodal features, the feature order of each modality can remain unchanged. Based on this, visual features corresponding to the scene image data can be extracted from the multimodal features. Similarly, features from other modalities can also be extracted.

[0089] Step 2230: Transform the visual features into a three-dimensional grid coordinate system to obtain the three-dimensional spatial features corresponding to the visual features.

[0090] In some alternative embodiments, the three-dimensional mesh coordinate system may be a three-dimensional mesh coordinate system centered on the vehicle, such as a three-dimensional mesh coordinate system centered on the rear axle center of the vehicle in frame T (e.g., the current frame).

[0091] In some optional embodiments, the transformation of visual features to a 3D mesh coordinate system can be implemented in any feasible manner. For example, it can be based on implicit depth information using a Transformer, or on explicit depth estimation projection, i.e., using LSS (Lift Splat Shoot) to transform visual features to a 3D mesh coordinate system and obtain the corresponding 3D spatial features. Specifically, the transformation can be based on the Lift operation in LSS. The Lift operation process can include feature extraction and depth estimation, outer product operation, and grid sampling. Feature extraction is used to further extract features from the visual features to obtain target features; depth estimation is used to estimate the depth features of the visual features to obtain the depth features corresponding to the target features; the outer product operation constructs a comprehensive feature from the target features and the depth features; and then grid sampling projects the comprehensive feature onto the corresponding grid to obtain the 3D spatial features in the 3D mesh coordinate system. The specific principles of LSS will not be elaborated further.

[0092] Step 2240: Based on the three-dimensional spatial features, determine the two-dimensional scene features corresponding to each viewpoint.

[0093] This involves converting three-dimensional spatial features into two-dimensional spaces from various perspectives to obtain the scene's two-dimensional features corresponding to each perspective.

[0094] In some optional embodiments, step 2240, which determines the scene's two-dimensional features corresponding to each viewpoint based on three-dimensional spatial features, includes:

[0095] By using the neural radiation field rendering method, two-dimensional scene features corresponding to each viewpoint are obtained from three-dimensional spatial features.

[0096] Among them, Neural Radiance Field (NeRF) is a 3D reconstruction technology based on neural networks. It models a scene as a continuous 5D radiance field and stores it implicitly in the neural network. It can input two-dimensional features from multiple angles and obtain a neural radiance field model through training. Based on this model, the two-dimensional features of the scene from any viewpoint can be rendered.

[0097] This embodiment obtains multimodal features by encoding multimodal scene data, which helps to better capture the correlation information between multimodal scene data, thereby obtaining more effective multimodal features. Then, three-dimensional spatial features are obtained based on the multimodal features, and two-dimensional scene features from each perspective are determined based on the three-dimensional spatial features. This helps to provide effective initial signals for generating driving scene videos, and facilitates the generation of accurate and effective driving scene videos through diffusion iteration.

[0098] In some alternative embodiments, Figure 5 This is a schematic diagram of a process for generating scene environment prediction information for a preset number of future frames, provided by some exemplary embodiments of this disclosure. Figure 5 As shown, after generating the three-dimensional spatial features in step 2230, the scene environment prediction information for a preset number of future frames based on multimodal scene data in step 220 includes:

[0099] Step 2250: Based on the three-dimensional spatial features, predict the scene environment prediction information.

[0100] Among them, 3D spatial features are features in a 3D spatial coordinate system centered on the vehicle. They include 3D spatial features corresponding to multimodal scene data, and based on this, scene environment prediction information for future frames can be predicted. For example, predicting road map information and motion information of objects around the vehicle in future frames.

[0101] In some alternative embodiments, scene environment prediction information can be predicted based on any implementable prediction network, for example, scene environment prediction information can be predicted based on a Transformer prediction network.

[0102] In some optional embodiments, the three-dimensional spatial features can be further spatiotemporally encoded to further learn the spatiotemporal relationships between the three-dimensional spatial features, which helps to further improve the accuracy and effectiveness of scene environment prediction information.

[0103] This embodiment uses three-dimensional spatial features to predict scene environment information for future frames, which is then used as conditional information for generating driving scene videos. This helps to make the generated driving scene videos more consistent with the simulation conditions of driving scenarios.

[0104] In some alternative embodiments, Figure 6 This is a flowchart illustrating step 220 provided in some optional embodiments of this disclosure. For example... Figure 6 As shown, step 220 includes steps 2210 to 2250 above. Steps 2240 and 2250 can be executed in any order, either in parallel or sequentially.

[0105] In some alternative embodiments, Figure 7 This is a flowchart illustrating the process of predicting scene environment prediction information provided by some exemplary embodiments of this disclosure, such as... Figure 7 As shown, step 2250, which predicts scene environment prediction information based on three-dimensional spatial features, includes:

[0106] Step 22510: Use a spatiotemporal encoder to perform spatiotemporal encoding on the three-dimensional spatial features to obtain the encoded target three-dimensional spatial features.

[0107] Among them, the spatiotemporal encoder can be an encoder based on a self-attention mechanism. The spatiotemporal encoder can be used to perform intentional interaction on three-dimensional spatial features to better capture the temporal and spatial relationships of the three-dimensional spatial features. By considering the temporal information of different frames, the obtained target three-dimensional spatial features help to decode more accurate road environment prediction information and surrounding object motion information for a preset number of future frames.

[0108] Step 22520: Using the first decoder, decode the three-dimensional spatial features of the target to obtain road environment prediction information for a preset number of future frames.

[0109] The first decoder can be any series of decoders based on Transformer, or any other implementable decoder, as long as it can decode the road environment prediction information of future frames. No specific limitation is imposed.

[0110] Step 22530: Use the second decoder to decode the three-dimensional spatial features of the target to obtain the motion information of surrounding objects in the next preset number of frames.

[0111] The second decoder can be any series of decoders based on Transformer, or any other implementable decoder, as long as it can decode the motion information of surrounding objects in future frames; there is no specific limitation. Surrounding objects refer to other objects around the vehicle. For example, surrounding objects can include other vehicles, pedestrians, cyclists, and other dynamic objects around the vehicle.

[0112] Step 22540: Use the road environment prediction information and the motion information of surrounding objects as the scene environment prediction information.

[0113] Since the road environment prediction information and the motion information of surrounding objects are obtained based on the decoder, they have a certain data structure. Therefore, the obtained scene environment prediction information is structured scene environment information, which helps to serve as conditional information for generating driving scene videos. This constrains the generated driving scene videos to meet the conditional information, thereby enabling the generated driving scene videos to meet the simulation conditions of the driving scene.

[0114] In some optional embodiments, a spatiotemporal encoder, a first decoder, and a second decoder that predict scene environment prediction information based on three-dimensional spatial features can form a scene environment prediction model. Figure 8 This is a schematic diagram of the network structure of a scene environment prediction model provided in an exemplary embodiment of this disclosure. For example... Figure 8As shown, multi-view image features refer to the visual features extracted from multimodal features. T represents the T-th frame, Ti represents the Ti-th frame, i = 0, 1, 2, ..., N, where N is a positive integer. The multi-view image features are converted into three-dimensional spatial features via LSS. The first decoder (Map decoder) decodes the three-dimensional spatial features (Spatial Temporal Voxel) to obtain road environment prediction information for the next preset number of frames. The second decoder (Motion decoder) decodes the three-dimensional spatial features to obtain motion information of surrounding objects for the next preset number of frames. In practical applications, a spatiotemporal encoder can be included before the Map decoder and Motion decoder to perform spatiotemporal encoding on the Spatial Temporal Voxel, obtaining the encoded target three-dimensional spatial features. Then, the Map decoder decodes the target three-dimensional spatial features to obtain road environment prediction information for the next preset number of frames, and the Motion decoder decodes the target three-dimensional spatial features to obtain motion information of surrounding objects for the next preset number of frames.

[0115] In some alternative embodiments, the scene environment prediction model may also include a multimodal encoder and a module for generating two-dimensional features of the scene.

[0116] This embodiment uses a spatiotemporal encoder to spatiotemporally encode 3D spatial features, which helps to better capture the spatiotemporal relationships of these features. This makes the obtained target 3D spatial features more conducive to decoding road environment prediction information and surrounding object motion information for a predetermined number of future frames, improving the accuracy and effectiveness of the predicted road environment prediction information and surrounding object motion information. Furthermore, multi-task inference using road environment prediction information and surrounding object motion information helps to comprehensively consider the interrelationships between different tasks, thereby improving the accuracy and robustness of the prediction results.

[0117] In some alternative embodiments, Figure 9 This is a schematic diagram of a process for generating scene environment prediction information for a preset number of future frames, provided by some other exemplary embodiments of this disclosure, such as... Figure 9 As shown, when the multimodal scene data includes a scene bird's-eye view, after obtaining the multimodal features in step 2210, step 220 generates scene environment prediction information for a preset number of future frames based on the multimodal scene data, including:

[0118] Step 22a0: Extract the bird's-eye view features corresponding to the scene bird's-eye view from the multimodal features.

[0119] Among them, the extraction of bird's-eye view features can be achieved based on the sequence of multimodal features, which is similar to the extraction of visual features mentioned above, and will not be repeated here.

[0120] Step 22b0: Based on the features of the bird's-eye view, predict the scene environment prediction information.

[0121] Among them, the bird's-eye view features more directly include the scene environment features corresponding to the scene image data and the state of objects around the vehicle in at least one frame of the scene image data, such as position, direction, speed, angular velocity, etc. Based on the bird's-eye view features, the scene environment prediction information of future frames can be better predicted.

[0122] In some alternative embodiments, the bird's-eye view features can be decoded based on a third decoder to obtain scene environment prediction information. The third decoder can be any implementable decoder, such as a series of decoders based on Transformer.

[0123] This embodiment predicts scene environment information by using bird's-eye view features, which helps to further improve the accuracy and effectiveness of scene environment prediction information.

[0124] In some optional embodiments, the structured scene environment prediction information for future frames can be predicted by combining bird's-eye view features with the aforementioned three-dimensional spatial features (or target three-dimensional spatial features) to further improve the accuracy and effectiveness of the scene environment prediction information.

[0125] In some optional embodiments, the multimodal scene data includes at least one frame of scene image data, scene environment description text data, and vehicle control signals. Figure 10 This is a flowchart illustrating a processing method for generating driving scene videos provided by other exemplary embodiments of this disclosure. For example... Figure 10 As shown, the method in this embodiment of the disclosure further includes:

[0126] Step 310: Extract text features corresponding to the scene environment description text data and command features corresponding to the vehicle control signals from the multimodal features.

[0127] The extraction of text features and command features can be found in the aforementioned extraction of visual features, and will not be repeated here.

[0128] Step 230 uses a video generation model to generate driving scene videos based on the scene's two-dimensional features and scene environment prediction information corresponding to each viewpoint, including:

[0129] Step 2310: Based on the scene 2D features, scene environment prediction information, text features and command features corresponding to each viewpoint, a driving scene video is generated using a video generation model.

[0130] In this method, text features, command features, and scene environment prediction information are used together as conditional information for generating driving scene videos. This ensures that the generated driving scene videos conform to the road environment conditions, motion conditions of surrounding objects, and weather and road conditions described in the text data, and that the vehicle's motion conforms to the vehicle behavior conditions of the vehicle control signals. This further improves the effectiveness and reliability of the generated driving scene videos, enabling them to simulate more realistic driving scenarios and thus more effectively optimize the performance of autonomous driving systems.

[0131] In some alternative embodiments, Figure 11 This is a flowchart illustrating a method for generating driving scene videos provided in some exemplary embodiments of this disclosure. For example... Figure 11 As shown, the multimodal scene data includes at least one frame of scene image data, scene environment description text data, and vehicle control signals. Steps 2210 to 2250 of step 220, combined with steps 310 and 2310, enable editing of the generated driving scene video based on text features and vehicle control signals. This allows the generated driving scene video to further conform to the weather conditions, vehicle behavior, and other conditions described in the text, thereby further enhancing the richness of the driving scene video.

[0132] In some optional embodiments, the multimodal scene data may include at least one frame of scene image data, a bird's-eye view of the scene corresponding to at least one frame of image data, scene environment description text data, and vehicle control signals. Figure 10 Step 220 shown may also include Figure 3 Steps 2210 to 2240 and Figure 9 22a0 to 22b0.

[0133] In some optional embodiments, the above steps of the processing method for generating driving scene videos according to the embodiments of this disclosure can be implemented based on a complete generative model for generating driving scene videos, which can be pre-trained. For example, Figure 12 This is a schematic diagram of the network structure of a generative model provided in an exemplary embodiment of this disclosure. For example... Figure 12 As shown, "Driving World Model" represents the generated model; however, it can also be represented by other names, not limited to those shown in the figure. Multimodal scene data can be found in [reference needed]. Figure 4The details are omitted here. The generative model includes a multimodal encoder, an LSS transformation module (multi-view geometric constraints), a viewpoint rendering module, a future frame structured prediction module, and a video generation module (Video Diffusion). Multimodal scene data is processed by the multimodal encoder to obtain multimodal features. Visual features (multi-view image features) are extracted from these multimodal features. The LSS transformation module converts the visual features into 3D spatial features based on LSS. These 3D spatial features are then rendered by viewpoints to obtain 2D scene features for each viewpoint. These 2D scene features serve as the initial signal for the video generation module. The future frame structured prediction module processes the 3D spatial features based on a scene environment prediction model to obtain structured scene environment prediction information (i.e., structured conditions). Text conditions and / or command conditions are extracted from the multimodal features. Video Diffusion utilizes the video generation model to continuously iterate and diffuse based on the initial signal (2D scene features for each viewpoint) and the constraints of each condition, achieving functions such as driving scene video generation / prediction and video editing. For example, under structured conditions, basic driving scene videos can be generated. Under both structured and text conditions, environmental editing, such as weather conditions, can be applied to these basic driving scene videos to generate driving scene videos under different weather conditions (referred to as video editing). Video editing can refer to generating driving scene videos based on text feature constraints, such as generating driving scene videos under different weather conditions. Motion prediction and 3D perception can also be tasks achievable by the generative model in this embodiment. Motion prediction refers to predicting the future trajectories of other objects around the vehicle, while 3D perception refers to perceiving the 3D environment around the vehicle in future frames. Specific task types can be set according to actual needs.

[0134] In some optional embodiments, the network structure of the video generation model can be based on an LDM (Latent Space Diffusion Model). For example, a temporal layer (also known as a temporal sequence layer) can be introduced into the LDM to achieve latent temporal alignment. Based on a pre-trained image generation model (a model for generating images from text), for UNet, a temporal layer is added. The temporal layer can include 3D convolutional layers and temporal attention layers, extending the two-dimensional VAE (Variational Autoencoders) in the image generation model. A temporal layer is added to the decoder part, the spatial layer parameters are fixed, and fine-tuning training is performed on video data to convert the image generator into a video generator. Then, it is jointly trained with the scene environment prediction model of this disclosure to obtain the generation model of this disclosure for generating driving scene videos. The specific network structure of the video generation model can be set according to actual needs, and will not be elaborated here. The specific process of the generative model may include: generating latent key frames, interpolating frames based on the diffusion model, retaining adjacent latent key frames as the two ends of the interpolated frame segment, initializing the frames to be inserted in the middle with noise, generating video through the decoder, and then increasing the resolution of the video based on the super-resolution module.

[0135] In some alternative embodiments, the network structure of the generative model may also be implemented using other possible networks, and the specific network structure of the generative model is not limited.

[0136] The embodiments described above can be implemented individually or in any combination without conflict, and can be set according to actual needs.

[0137] Any of the methods for generating driving scene videos provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the methods for generating driving scene videos provided in this disclosure can be executed by a processor, such as by a processor executing any of the methods for generating driving scene videos mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0138] Figure 13 This is a flowchart illustrating a training method for a generative model used to generate driving scene videos, provided by some exemplary embodiments of this disclosure. For example... Figure 13 As shown, the method includes the following steps:

[0139] Step 410: Obtain training sample data; the training sample data includes multiple first training samples and multiple second training samples.

[0140] In some optional embodiments, each first training sample may include at least one frame of training scene image data corresponding to two-dimensional features of the training scene, and ground truth information of the training scene environment for a future preset number of frames corresponding to the training scene image data.

[0141] In some optional embodiments, each first training sample may include training multimodal scene data and ground truth information of the training scene environment for a preset number of future frames. Specifically, this can be determined based on the network structure of the scene environment prediction model to be trained. For example, if the network structure of the scene environment prediction model to be trained includes the aforementioned spatiotemporal encoder, first decoder, and second decoder, then the first training sample may include training scene two-dimensional features corresponding to at least one frame of training scene image data, and ground truth information of the training scene environment for a preset number of future frames corresponding to the training scene image data. If the network structure further includes a multimodal encoder and a module for generating training scene two-dimensional features, then the first training sample may include training multimodal scene data and ground truth information of the training scene environment for a preset number of future frames.

[0142] In some optional embodiments, each second training sample includes training multimodal scene data and training driving scene ground truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of training scene bird's-eye view, training scene environment description text data, and training vehicle control signals; each frame of training scene image data includes training scene image from at least one perspective.

[0143] In some optional embodiments, for each first training sample, the determination of the training scene two-dimensional features corresponding to the training scene image data included in the first training sample is consistent with the scene two-dimensional features in the model inference stage of the aforementioned embodiments, that is, it can be determined based on the training visual features in the training multimodal features generated from the training multimodal data, which will not be elaborated here. The training scene environment ground truth information may include the training road environment ground truth information and the training surrounding object motion information for a preset number of future frames. The training scene environment ground truth information can be obtained by pre-annotation, and the annotation method may include manual annotation, semi-automatic annotation, automatic annotation, etc., which are not specifically limited. The training scene environment ground truth information is used to supervise the training scene environment prediction information obtained by the training scene environment prediction model to promote the convergence of the model.

[0144] In some optional embodiments, for each second training sample, the training multimodal scene data included in the second training sample is consistent with the multimodal scene data in the model inference stage of the aforementioned embodiments, and will not be repeated here. The training driving scene ground truth video can be a real video of a driving scene obtained in advance. In practical applications, the training multimodal scene data can be obtained based on the training driving scene ground truth video. For example, a large number of driving scene videos can be obtained in advance as training driving scene ground truth videos. For any training driving scene ground truth video, the first n frames of the training driving scene video are used as the training scene image data corresponding to the training driving scene video, where n is a positive integer. Based on the training scene image data, data of other modalities corresponding to the training scene image data are obtained, and the training scene image data and the data of other modalities are used as training multimodal scene data. Consistent with the inference stage of the aforementioned embodiments, the data of other modalities may include at least one of the following: training scene bird's-eye view, training scene environment description text data, and training vehicle control signals. Similarly, consistent with the scene image data of the aforementioned embodiments, each frame of training scene image data may include training scene images from at least one perspective.

[0145] Step 420: Based on each first training sample, perform the first stage of training on the scene environment prediction model to be trained, and obtain the scene environment prediction model.

[0146] The network structure of the scene environment prediction model to be trained can be found in the aforementioned embodiments. For example, the network structure of the scene environment prediction model to be trained includes a spatiotemporal encoder, a first decoder, and a second decoder. In some optional embodiments, in addition to including the spatiotemporal encoder, the first decoder, and the second decoder, the network structure of the scene environment prediction model to be trained may also include a multimodal encoder and a module for generating two-dimensional features of the training scene (e.g., including the LSS conversion module and the viewpoint rendering module in the aforementioned embodiments). In this case, the first training sample may include training multimodal scene data and training training scene environment ground truth information, which can be specifically set according to actual needs.

[0147] In some optional embodiments, during the first stage of training, the training scene environment prediction information for a preset number of frames can be predicted based on the two-dimensional features of the training scene from each viewpoint. This predicted information is then compared with the ground truth information of the training scene environment. The model loss is calculated according to a preset loss function, and the update gradient of the model network parameters is determined based on the model loss. The model network parameters are then updated to obtain the updated scene environment prediction model to be trained. The updated scene environment prediction model is trained using the above process until the training termination condition is met, thus obtaining the scene environment prediction model. The training termination condition may include, for example, model convergence, or the number of iterations reaching a first preset number, etc., and is not specifically limited. The preset loss function can be any implementable loss function, such as the cross-entropy loss function or the mean squared error (MSE) loss function.

[0148] In some optional embodiments, if the first training sample includes training multimodal scene data, and if the training multimodal scene data includes a bird's-eye view of the training scene, then the training scene environment prediction information for a preset number of future frames can also be predicted based on the bird's-eye view of the training scene. Specific details are not limited.

[0149] In some alternative embodiments, the multiple first training samples can be divided into multiple batches, with each iteration using the first training samples from one of the batches.

[0150] Step 430: Based on the scene environment prediction model and the video generation model to be trained obtained in the first stage of training, determine the generation model to be trained.

[0151] In the first stage of training, after obtaining the scene environment prediction model, the scene environment prediction model and the video generation model to be trained are used together to determine the generation model to be trained. In the second stage of training, the prediction results of the scene environment prediction model can be used as the structured conditions of the video generation model to be trained, and the scene environment prediction model and the video generation model to be trained are jointly trained.

[0152] In some optional embodiments, if the scene environment prediction model includes a spatiotemporal encoder, a first decoder, and a second decoder, then the network structure of the generator model to be trained may include, in addition to the scene environment prediction model and the video generation model to be trained, a multimodal encoder and a module for generating three-dimensional spatial features and two-dimensional scene features. That is, the network structure of the generator model to be trained can generate training driving scene prediction videos based on training multimodal scene data; details will not be elaborated further.

[0153] In some optional embodiments, the network structure of the video generation model to be trained can refer to the video generation model of any of the foregoing embodiments, and will not be repeated here.

[0154] Step 440: Based on each second training sample, perform a second stage of training on the generator model to be trained to obtain a generator model for generating driving scene videos.

[0155] Among them, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generation model to be trained, which is used to constrain the generated training driving scene prediction video so that the driving scene prediction video can meet the conditions of the structured training scene environment prediction information output by the scene environment prediction model.

[0156] In some optional embodiments, the training multimodal scene data of the second training sample can be processed based on the generative model to be trained to generate a training driving scene prediction video. The training driving scene prediction video is compared with the ground truth video of the training driving scene of the second training sample. Based on a preset loss function, the model loss is calculated, and then the gradient of the network parameter update of the generative model to be trained is calculated based on the model loss. The network parameters of the generative model to be trained are updated based on the updated gradient to obtain the updated generative model to be trained. The updated generative model to be trained continues to be trained according to the above process until the termination condition of the second stage of training is met, and the generative model is obtained. The termination condition of the second stage of training may include model convergence, the number of iterations reaching a second preset number, etc. The loss function of the second stage of training can be any loss function that can be implemented by the video generation model, and there is no specific limitation.

[0157] In some optional embodiments, during the second stage of training, the training video generation model can be trained first based on the structured conditions output by the scene environment prediction model. After the training video generation model converges, the scene environment prediction model can be further optimized using a large number of training driving scene ground truth videos and training driving scene prediction videos generated by the training video generation model. Then, structured conditions are generated based on the optimized scene environment prediction model to further optimize the video generation model. This process is repeated, and through iterative training, the generalization ability and adaptability of the generation model are continuously improved, enabling it to effectively predict structured data (scene environment prediction information) in various scenarios, which helps to achieve a great unification of visual generation and understanding.

[0158] In some alternative embodiments, the multiple second training samples can be divided into multiple batches, with each iteration using one of the batches of second training samples.

[0159] This embodiment, through two-stage training, can effectively improve the performance of the generative model, enabling the model-generated driving scene videos to better meet the simulation requirements of driving scenarios, thereby improving the accuracy and effectiveness of driving scene videos.

[0160] Figure 14 This is a flowchart illustrating a training method for a generative model used to generate videos of driving scenarios, provided by some other exemplary embodiments of this disclosure.

[0161] In some optional embodiments, each first training sample includes training multimodal scene data and ground truth information of the training scene environment for a preset number of future frames; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, training scene environment description text data, and training vehicle control signals; each frame of training scene image data includes a training scene image from at least one viewpoint; such as Figure 14 As shown, step 420 involves training the scene environment prediction model based on each first training sample in the first stage to obtain the scene environment prediction model, including:

[0162] Step 4210: Perform steps 42110 to 42150 using the scene environment prediction model to be trained, until the first stage of training of the scene environment prediction model is completed, and obtain the scene environment prediction model (see...). Figure 15 ):

[0163] Step 42110: For any first training sample, based on the training multimodal scene data in the first training sample, determine the two-dimensional features of the first training scene corresponding to each viewpoint.

[0164] Step 42120: Use the scene environment prediction model to predict the two-dimensional features of the first training scene corresponding to each viewpoint to obtain the first training scene environment prediction information for the next preset number of frames.

[0165] It should be noted that the specific operating principles of steps 42110 and 42120 are consistent with those of generating two-dimensional scene features and predicting scene environment prediction information from various perspectives in the aforementioned embodiments, and will not be repeated here.

[0166] Step 42130: Based on the first training scene environment prediction information and the training scene environment ground truth information in the first training sample, determine the first loss corresponding to the first training sample.

[0167] The first loss can be determined based on the first loss function, which can be the cross-entropy loss function, the mean squared error loss function, etc., without any specific limitation.

[0168] Step 42140: Determine the first model loss based on the first loss corresponding to each first training sample.

[0169] The first model loss can be achieved by summing or averaging the first losses of each first training sample, without any specific limitation.

[0170] Step 42150: Update the network parameters of the scene environment prediction model to be trained based on the first model loss to obtain the updated scene environment prediction model to be trained.

[0171] Specifically, the update gradient of the network parameters can be determined based on the loss of the first model, and the network parameters can be updated based on the update gradient to obtain the updated prediction model of the scene environment to be trained.

[0172] In some alternative embodiments, the updated gradient can be obtained based on any implementable gradient descent algorithm, such as stochastic gradient descent, learning rate adaptive gradient descent, etc.

[0173] After obtaining the updated scene environment prediction model to be trained, the updated scene environment prediction model to be trained can be used as the above-mentioned scene environment prediction model to be trained. Steps 42120 and subsequent steps are repeated until the first stage of training is completed, and the updated scene environment prediction model to be trained is obtained.

[0174] In some alternative embodiments, the first stage of training can be based on a small number of initial training samples, which helps reduce the amount of annotation work. In the second stage, the scene environment prediction model is continuously optimized based on a large number of driving scene videos.

[0175] In some optional embodiments, each second training sample includes training multimodal scene data and a training driving scene ground truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, training scene environment description text data, and training vehicle control signals; each frame of training scene image data includes a training scene image from at least one viewpoint. Step 440, which involves performing a second stage of training on the generative model to be trained based on each second training sample to obtain a generative model for generating driving scene videos, includes:

[0176] Step 4410: Perform steps 44110 to 44150 using the generator model to be trained until the second stage of training of the generator model is completed, thus obtaining the generator model used to generate driving scene videos (see...). Figure 16 ):

[0177] Step 44110: For any second training sample, based on the training multimodal scene data in the second training sample, generate the second training scene two-dimensional features corresponding to each viewpoint and the second training scene environment prediction information for the next preset number of frames.

[0178] The generation principle of the second training scene two-dimensional features and the second training scene environment prediction information of the future preset frame number is the same as that of the scene two-dimensional features and the scene environment prediction information of the future preset frame number in the aforementioned embodiment, and will not be repeated here.

[0179] Step 44120: Generate a training driving scene prediction video based on the two-dimensional features of the second training scene and the environment prediction information of the second training scene corresponding to each viewpoint.

[0180] The generation principle of the training driving scene prediction video is the same as that of the driving scene video in the previous embodiment, and will not be repeated here.

[0181] Step 44130: Based on the training driving scene prediction video and the training driving scene ground truth video in the second training sample, determine the second loss corresponding to the second training sample.

[0182] The second loss can be determined based on the second loss function, which may include loss functions related to the video generation model, without any specific limitation.

[0183] Step 44140: Determine the second model loss based on the second loss corresponding to each second training sample.

[0184] The second model loss can be determined by summing or averaging the second losses of each second training sample.

[0185] Step 44150: Based on the second model loss, update the network parameters of the generator model to be trained to obtain the updated generator model to be trained.

[0186] Specifically, the update gradient of the network parameters can be determined based on the second model loss, and then the network parameters can be updated based on the update gradient to optimize the model to be trained.

[0187] In some optional embodiments, step 44110, based on the training multimodal scene data in the second training sample, generates two-dimensional features of the second training scene corresponding to each viewpoint and environment prediction information of the second training scene for a preset number of future frames, including:

[0188] The training multimodal scene data in the second training sample is encoded using a multimodal encoder to obtain training multimodal features; training visual features corresponding to the training scene image data are extracted from the training multimodal features; the training visual features are transformed into a three-dimensional grid coordinate system to obtain training three-dimensional spatial features corresponding to the training visual features; based on the training three-dimensional spatial features, the second training scene two-dimensional features corresponding to each viewpoint are determined; based on the training three-dimensional spatial features, the second training scene environment prediction information is predicted; or, if the second training sample includes a bird's-eye view of the training scene, the training bird's-eye view features corresponding to the training scene bird's-eye view are extracted from the training multimodal features, and the second training scene environment prediction information is predicted based on the training bird's-eye view features.

[0189] The specific operational principles of each step in predicting the environment prediction information of the second training scenario can be found in the steps of predicting the environment prediction information of the aforementioned embodiment, and will not be repeated here.

[0190] In some alternative embodiments, Figure 17 This is a schematic diagram illustrating the training principle of a generative model to be trained, provided in an exemplary embodiment of this disclosure. For example... Figure 17 As shown, the network structure of the generative model to be trained is as follows: Figure 12 The structure shown (partial structures not shown) represents the visual features in the multi-view image features. In the first training phase, the scene environment prediction model is trained based on a small amount of labeled data, generating a first-stage gradient. This first-stage gradient is backpropagated to update the network parameters, resulting in an updated scene environment prediction model. This iterative training continues until the first phase ends, yielding the scene environment prediction model. In the second training phase, the scene environment prediction model predicts second training scene environment prediction information based on the training multi-modal data of the second training samples. This second training scene environment prediction information serves as the structured information condition for Video Diffusion. Under the constraint of this structured information condition, Video Diffusion generates training driving scene prediction videos based on the training scene 2D features, text features, and command features generated from the training multi-modal features. Supervised training is then performed based on the ground truth videos of the training driving scenes corresponding to the training multi-modal features (i.e., the unlabeled video data in the figure), generating a second-stage gradient. This backpropagation updates the overall network parameters of the generator model, continuously optimizing the generator model until the second phase ends. The resulting updated generator model can then be used as the generator model.

[0191] This embodiment uses two-stage training to effectively improve the model's convergence speed, as well as its generalization ability and adaptability.

[0192] The embodiments described above can be implemented individually or in any combination without conflict, and can be set according to actual needs.

[0193] The training method for any of the generative models for generating driving scene videos provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, the training method for any of the generative models for generating driving scene videos provided in this disclosure can be executed by a processor, such as by a processor executing any of the training methods for generating driving scene videos mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated upon below.

[0194] Exemplary device

[0195] Figure 18 This is a schematic diagram of the structure of a processing apparatus for generating driving scene videos provided in some exemplary embodiments of this disclosure. The apparatus of this embodiment can be used to implement corresponding processing method embodiments for generating driving scene videos of this disclosure, such as... Figure 18 The device shown includes: a multimodal data acquisition module 510, a first processing module 520, and a second processing module 530.

[0196] The multimodal data acquisition module 510 is used to acquire multimodal scene data. The multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and also includes at least one of the following: scene bird's-eye view, scene environment description text data, and vehicle control signals. Each frame of scene image data includes at least one scene image corresponding to each viewpoint.

[0197] The first processing module 520 is used to generate scene two-dimensional features corresponding to each viewpoint and scene environment prediction information for a preset number of future frames based on multimodal scene data.

[0198] The second processing module 530 is used to generate driving scene videos based on the scene two-dimensional features and scene environment prediction information corresponding to each viewpoint, using a video generation model.

[0199] In some optional embodiments, the first processing module 520 is specifically used for:

[0200] Multimodal scene data is encoded using a multimodal encoder to obtain multimodal features; visual features corresponding to scene image data are extracted from the multimodal features; the visual features are transformed into a three-dimensional grid coordinate system to obtain the three-dimensional spatial features corresponding to the visual features; based on the three-dimensional spatial features, the two-dimensional scene features corresponding to each viewpoint are determined.

[0201] In some optional embodiments, the first processing module 520 is specifically used for:

[0202] Based on three-dimensional spatial features, predict scene environment information.

[0203] In some optional embodiments, the first processing module 520 is specifically used for:

[0204] A spatiotemporal encoder is used to perform spatiotemporal encoding on the three-dimensional spatial features to obtain the encoded target three-dimensional spatial features; a first decoder is used to decode the target three-dimensional spatial features to obtain road environment prediction information for a preset number of future frames; a second decoder is used to decode the target three-dimensional spatial features to obtain motion information of surrounding objects for a preset number of future frames; the road environment prediction information and the motion information of surrounding objects are used as scene environment prediction information.

[0205] In some optional embodiments, the first processing module 520 is specifically used for:

[0206] When multimodal scene data includes a bird's-eye view of the scene, bird's-eye view features corresponding to the scene bird's-eye view are extracted from the multimodal features; based on the bird's-eye view features, scene environment prediction information is predicted.

[0207] In some optional embodiments, the multimodal scene data includes at least one frame of scene image data, scene environment description text data, and vehicle control signals. The first processing module 520 is further configured to:

[0208] Extract text features corresponding to scene environment description text data and command features corresponding to vehicle control signals from multimodal features.

[0209] The second processing module 530 is specifically used for:

[0210] Based on the scene's two-dimensional features, scene environment prediction information, text features, and command features corresponding to each perspective, a driving scene video is generated using a video generation model.

[0211] It should be noted that the specific implementation of the processing device for generating driving scene videos in this disclosure is similar to the specific implementation of the processing method for generating driving scene videos in this disclosure. For details, please refer to the processing method for generating driving scene videos section. To reduce redundancy, it will not be described again here.

[0212] Figure 19 This is a schematic diagram of the structure of a training apparatus for generating a generative model for generating driving scene videos, provided by some exemplary embodiments of this disclosure. The apparatus of this embodiment can be used to implement corresponding training method embodiments of the generative model for generating driving scene videos disclosed herein, such as... Figure 19The apparatus shown includes: a sample acquisition module 610, a first training module 620, and a second training module 630.

[0213] The sample acquisition module 610 is used to acquire training sample data; the training sample data includes multiple first training samples and multiple second training samples.

[0214] The first training module 620 is used to perform the first stage of training on the scene environment prediction model to be trained based on each first training sample, so as to obtain the scene environment prediction model.

[0215] The second training module 630 is used to determine the generation model to be trained based on the scene environment prediction model and the video generation model to be trained obtained in the first stage of training; and to perform a second stage of training on the generation model to be trained based on each second training sample to obtain a generation model for generating driving scene videos; wherein, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generation model to be trained.

[0216] In some optional embodiments, each first training sample includes training multimodal scene data and ground truth information of the training scene environment for a preset number of future frames; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, training scene environment description text data, and training vehicle control signals; each frame of training scene image data includes a training scene image from at least one viewpoint; the first training module 620 is specifically used for:

[0217] The following operations are performed on the scene environment prediction model to be trained until the first stage of training of the scene environment prediction model is completed, thereby obtaining the scene environment prediction model:

[0218] For any first training sample, based on the training multimodal scene data in the first training sample, determine the two-dimensional features of the first training scene corresponding to each viewpoint; use the scene environment prediction model to be trained to predict the two-dimensional features of the first training scene corresponding to each viewpoint to obtain the first training scene environment prediction information for a preset number of frames in the future; based on the first training scene environment prediction information and the ground truth information of the training scene environment in the first training sample, determine the first loss corresponding to the first training sample; based on the first loss corresponding to each first training sample, determine the first model loss; based on the first model loss, update the network parameters of the scene environment prediction model to be trained to obtain the updated scene environment prediction model to be trained.

[0219] In some optional embodiments, each second training sample includes training multimodal scene data and a training driving scene ground truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, training scene environment description text data, and training vehicle control signals; each frame of training scene image data includes a training scene image from at least one viewpoint. The second training module 630 is specifically used for:

[0220] The following operations are performed on the generator model to be trained until the second stage of training of the generator model is completed, resulting in a generator model for generating driving scene videos:

[0221] For any second training sample, based on the training multimodal scene data in the second training sample, generate two-dimensional features of the second training scene corresponding to each viewpoint and prediction information of the second training scene environment for a preset number of future frames; based on the two-dimensional features of the second training scene corresponding to each viewpoint and the prediction information of the second training scene environment, generate a training driving scene prediction video; based on the training driving scene prediction video and the ground truth video of the training driving scene in the second training sample, determine the second loss corresponding to the second training sample; based on the second loss corresponding to each second training sample, determine the second model loss.

[0222] Based on the second model loss, the network parameters of the generative model to be trained are updated to obtain the updated generative model to be trained.

[0223] In some optional embodiments, the second training module 630 is specifically used for:

[0224] The training multimodal scene data in the second training sample is encoded using a multimodal encoder to obtain training multimodal features; training visual features corresponding to the training scene image data are extracted from the training multimodal features; the training visual features are transformed into a three-dimensional grid coordinate system to obtain training three-dimensional spatial features corresponding to the training visual features; based on the training three-dimensional spatial features, the second training scene two-dimensional features corresponding to each viewpoint are determined; based on the training three-dimensional spatial features, the second training scene environment prediction information is predicted; or, if the second training sample includes a bird's-eye view of the training scene, the training bird's-eye view features corresponding to the training scene bird's-eye view are extracted from the training multimodal features, and the second training scene environment prediction information is predicted based on the training bird's-eye view features.

[0225] It should be noted that the specific implementation of the apparatus in this disclosure is similar to the specific implementation of the corresponding method in this disclosure. For details, please refer to the corresponding method section. In order to reduce redundancy, it will not be described again here.

[0226] In addition, embodiments of this disclosure also provide an electronic device, including:

[0227] Memory, used to store computer programs;

[0228] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the processing method for generating driving scene videos as described in any of the above embodiments of the present disclosure.

[0229] Figure 20 This is a schematic diagram illustrating the structure of one application embodiment of the electronic device disclosed herein. Figure 20 As shown, the electronic device includes one or more processors and memory.

[0230] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0231] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the methods of the various embodiments of this disclosure described above and / or other desired functions.

[0232] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0233] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0234] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0235] Of course, for the sake of simplicity, Figure 20 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0236] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0237] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0238] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the foregoing portion of this specification.

[0239] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0240] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0241] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0242] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0243] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0244] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0245] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0246] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0247] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating driving scene videos, characterized in that, include: Acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of the following: scene bird's-eye view, scene environment description text data, and vehicle control signals; each frame of scene image data includes at least one scene image corresponding to each viewpoint; Based on the multimodal scene data, generate two-dimensional scene features corresponding to each viewpoint and scene environment prediction information for a preset number of future frames; Based on the two-dimensional scene features corresponding to each of the aforementioned viewpoints and the scene environment prediction information, a driving scene video is generated using a video generation model. Specifically, the two-dimensional scene features corresponding to each of the aforementioned viewpoints are used as the initial signal, and the scene environment prediction information for a preset number of future frames is used as the condition signal for generating the driving scene video. Based on the initial signal, a diffusion iteration is performed to generate the driving scene video that meets the scene environment conditions for a preset number of future frames. The video generation model is a video diffusion model. The step of generating scene two-dimensional features and scene environment prediction information for a preset number of future frames based on the multimodal scene data includes: The multimodal scene data is encoded using a multimodal encoder to obtain multimodal features; Extract visual features corresponding to the scene image data from the multimodal features; The visual features are transformed into a three-dimensional grid coordinate system to obtain the three-dimensional spatial features corresponding to the visual features. Based on the three-dimensional spatial features, determine the two-dimensional scene features corresponding to each of the aforementioned viewpoints; Based on the aforementioned three-dimensional spatial features, predict the scene environment prediction information; or... When the multimodal scene data includes the scene bird's-eye view, the bird's-eye view features corresponding to the scene bird's-eye view are extracted from the multimodal features; Based on the features of the bird's-eye view, predict the scene environment information.

2. The method according to claim 1, characterized in that, The prediction of scene environment prediction information based on the three-dimensional spatial features includes: The three-dimensional spatial features are spatiotemporally encoded using a spatiotemporal encoder to obtain the encoded target three-dimensional spatial features. The first decoder is used to decode the three-dimensional spatial features of the target to obtain the road environment prediction information for the next preset number of frames; Using a second decoder, the three-dimensional spatial features of the target are decoded to obtain the motion information of surrounding objects in the next preset number of frames; The road environment prediction information and the surrounding object motion information are used as the scene environment prediction information.

3. The method according to claim 1, characterized in that, The multimodal scene data includes at least one frame of scene image data, the scene environment description text data, and the vehicle control signal; The method further includes: Extract the text features corresponding to the scene environment description text data and the command features corresponding to the vehicle control signals from the multimodal features; The process of generating driving scene videos using a video generation model based on the two-dimensional features of the scene corresponding to each of the aforementioned viewpoints and the scene environment prediction information includes: Based on the scene 2D features corresponding to each of the aforementioned viewpoints, the scene environment prediction information, the text features, and the command features, the driving scene video is generated using the video generation model.

4. A training method for a generative model used to generate driving scene videos, characterized in that, include: Obtain training sample data; The training sample data includes multiple first training samples and multiple second training samples; Each of the first training samples includes training multimodal scene data and ground truth information of the training scene environment for a preset number of future frames; The training multimodal scene data includes at least one frame of training scene image data, as well as at least one of the following: a bird's-eye view of the training scene, text data describing the training scene environment, and training vehicle control signals. Each frame of training scene image data includes training scene images from at least one viewpoint; each second training sample includes training multimodal scene data and training driving scene ground truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of training scene bird's-eye view, training scene environment description text data, and training vehicle control signals. Each frame of training scene image data includes training scene images from at least one viewpoint; Based on each of the first training samples, the scene environment prediction model to be trained is trained in the first stage to obtain the scene environment prediction model. Based on the scene environment prediction model and the video generation model to be trained obtained in the first stage of training, the generation model to be trained is determined. The second training model is trained in the second stage based on each of the second training samples to obtain a generation model for generating driving scene videos; wherein, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generation model to be trained, and is used to constrain the generated training driving scene prediction video so that the driving scene prediction video can meet the conditions of the structured training scene environment prediction information output by the scene environment prediction model; the generation model is a video diffusion model. The second stage of training the generator model to be trained based on each of the second training samples to obtain a generator model for generating driving scene videos includes: The following operations are performed on the generator model to be trained until the second stage of training of the generator model is completed, resulting in a generator model for generating driving scene videos: For any second training sample, based on the training multimodal scene data in the second training sample, generate two-dimensional features of the second training scene corresponding to each of the aforementioned viewpoints and prediction information of the second training scene environment for a preset number of future frames. Based on the two-dimensional features of the second training scene corresponding to each of the aforementioned viewpoints and the environment prediction information of the second training scene, a training driving scene prediction video is generated. Based on the predicted training driving scene video and the ground truth training driving scene video in the second training sample, the second loss corresponding to the second training sample is determined. The second model loss is determined based on the second loss corresponding to each of the second training samples. Based on the second model loss, the network parameters of the generator model to be trained are updated to obtain the updated generator model to be trained. The step of generating two-dimensional features of the second training scene and second training scene environment prediction information for a preset number of future frames based on the training multimodal scene data in the second training sample includes: The training multimodal scene data in the second training sample is encoded using a multimodal encoder to obtain training multimodal features; Extract training visual features corresponding to the training scene image data from the training multimodal features; The training visual features are transformed into a three-dimensional grid coordinate system to obtain the training three-dimensional spatial features corresponding to the training visual features. Based on the training three-dimensional spatial features, determine the second training scene two-dimensional features corresponding to each of the aforementioned viewpoints; Based on the aforementioned training 3D spatial features, predict the environment information of the second training scene; or... If the second training sample includes a bird's-eye view of the training scene, the training bird's-eye view features corresponding to the training scene bird's-eye view are extracted from the training multimodal features, and the environment prediction information of the second training scene is predicted based on the training bird's-eye view features.

5. The method according to claim 4, characterized in that, The first stage of training the scene environment prediction model based on each of the first training samples to obtain the scene environment prediction model includes: The following operations are performed on the scene environment prediction model to be trained until the first stage of training of the scene environment prediction model is completed, thereby obtaining the scene environment prediction model: For any of the first training samples, based on the training multimodal scene data in the first training sample, determine the two-dimensional features of the first training scene corresponding to each viewpoint; The two-dimensional features of the first training scene corresponding to each viewpoint are predicted using the scene environment prediction model to be trained, so as to obtain the first training scene environment prediction information for a preset number of frames in the future. Based on the first training scene environment prediction information and the training scene environment ground truth information in the first training sample, the first loss corresponding to the first training sample is determined. The first model loss is determined based on the first loss corresponding to each of the first training samples. The network parameters of the scene environment prediction model to be trained are updated based on the first model loss to obtain the updated scene environment prediction model to be trained.

6. A processing apparatus for generating driving scene videos, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of the following: scene bird's-eye view, scene environment description text data, and vehicle control signals; each frame of scene image data includes at least one scene image corresponding to each viewpoint; The first processing module is used to generate scene two-dimensional features corresponding to each of the aforementioned viewpoints and scene environment prediction information for a preset number of future frames based on the multimodal scene data. The second processing module is used to generate a driving scene video based on the scene two-dimensional features corresponding to each of the aforementioned viewpoints and the scene environment prediction information, using a video generation model. Specifically, the scene two-dimensional features corresponding to each of the aforementioned viewpoints are used as the initial signal, and the scene environment prediction information for a preset number of future frames is used as the condition signal for generating the driving scene video. Based on the initial signal, the process iterates to generate the driving scene video that meets the scene environment conditions for the preset number of future frames. The video generation model is a video diffusion model. The first processing module is specifically used for: The multimodal scene data is encoded using a multimodal encoder to obtain multimodal features; Extract visual features corresponding to the scene image data from the multimodal features; The visual features are transformed into a three-dimensional grid coordinate system to obtain the three-dimensional spatial features corresponding to the visual features. Based on the three-dimensional spatial features, determine the two-dimensional scene features corresponding to each of the aforementioned viewpoints; Based on the aforementioned three-dimensional spatial features, predict the scene environment prediction information; or... When the multimodal scene data includes the scene bird's-eye view, the bird's-eye view features corresponding to the scene bird's-eye view are extracted from the multimodal features; Based on the features of the bird's-eye view, predict the scene environment information.

7. A training device for a generative model used to generate driving scene videos, characterized in that, include: The sample acquisition module is used to acquire training sample data; The training sample data includes multiple first training samples and multiple second training samples; Each of the first training samples includes training multimodal scene data and ground truth information of the training scene environment for a preset number of future frames; The training multimodal scene data includes at least one frame of training scene image data, as well as at least one of the following: a bird's-eye view of the training scene, text data describing the training scene environment, and training vehicle control signals. Each frame of training scene image data includes training scene images from at least one viewpoint; each second training sample includes training multimodal scene data and training driving scene ground truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of training scene bird's-eye view, training scene environment description text data, and training vehicle control signals. Each frame of training scene image data includes training scene images from at least one viewpoint; The first training module is used to perform a first-stage training on the scene environment prediction model to be trained based on each of the first training samples, so as to obtain the scene environment prediction model. The second training module is used to determine the generation model to be trained based on the scene environment prediction model and the video generation model to be trained obtained in the first stage of training; and to perform a second stage of training on the generation model to be trained based on each of the second training samples to obtain a generation model for generating driving scene videos; wherein, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generation model to be trained, and is used to constrain the generated training driving scene prediction video so that the driving scene prediction video can meet the conditions of the structured training scene environment prediction information output by the scene environment prediction model; the generation model is a video diffusion model. Specifically, the second training module is used for: The following operations are performed on the generator model to be trained until the second stage of training of the generator model is completed, resulting in a generator model for generating driving scene videos: For any second training sample, based on the training multimodal scene data in the second training sample, generate two-dimensional features of the second training scene corresponding to each of the aforementioned viewpoints and prediction information of the second training scene environment for a preset number of future frames. Based on the two-dimensional features of the second training scene corresponding to each of the aforementioned viewpoints and the environment prediction information of the second training scene, a training driving scene prediction video is generated. Based on the predicted training driving scene video and the ground truth training driving scene video in the second training sample, the second loss corresponding to the second training sample is determined. The second model loss is determined based on the second loss corresponding to each of the second training samples. Based on the second model loss, the network parameters of the generator model to be trained are updated to obtain the updated generator model to be trained. The second training module is specifically used for: The training multimodal scene data in the second training sample is encoded using a multimodal encoder to obtain training multimodal features; Extract training visual features corresponding to the training scene image data from the training multimodal features; The training visual features are transformed into a three-dimensional grid coordinate system to obtain the training three-dimensional spatial features corresponding to the training visual features. Based on the training three-dimensional spatial features, determine the second training scene two-dimensional features corresponding to each of the aforementioned viewpoints; Based on the aforementioned training 3D spatial features, predict the environment information of the second training scene; or... If the second training sample includes a bird's-eye view of the training scene, the training bird's-eye view features corresponding to the training scene bird's-eye view are extracted from the training multimodal features, and the environment prediction information of the second training scene is predicted based on the training bird's-eye view features.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the processing method for generating driving scene videos as described in any one of claims 1-3, or implements the training method for a generative model for generating driving scene videos as described in any one of claims 4-5.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the processing method for generating driving scene videos as described in any one of claims 1-3, or the training method for the generative model for generating driving scene videos as described in any one of claims 4-5.

Images