Processing method for generating driving scene video, model training method and device

Abstract

The embodiment of the invention discloses a processing method for generating a driving scene video, and a model training method and device, and the method comprises the steps: obtaining multi-modal scene data; based on the multi-modal scene data, generating scene two-dimensional features corresponding to each view angle and scene environment prediction information of a future preset frame number; and generating a driving scene video by using a video generation model based on the scene two-dimensional features and the scene environment prediction information corresponding to each view angle. According to the embodiment of the invention, the scene environment prediction information is used as the condition signal for constraining the generated driving scene video, so that the generated driving scene video better conforms to conditions such as road types, traffic rules and vehicle behaviors of the driving scene, and the accuracy and reliability of the driving scene video can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a processing method for generating a driving scene video, a model training method and a device. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, AI generation technology has made great progress. From the initial text generation to the current text-to-image, text-to-video, and text-to-3D generation technologies, AI's generation capabilities have been continuously expanded, laying a solid foundation for more natural and intuitive interaction between people and computers. Compared with traditional AI generation work, autonomous driving data generation faces more complex challenges. In the field of autonomous driving, generated data not only needs to meet the conversion of text to image, video or 3D scene, but also meet strict driving scene simulation requirements. Driving scene simulation requirements can include, for example, various weather conditions, road types, traffic rules, vehicle behavior and other complex conditions, in order to provide high-quality training data for optimizing the performance of autonomous driving systems.

[0003] In the related art, driving scene videos are usually generated based on Video-LDM (Video-Latent Diffusion Models), that is, based on the diffusion model, the driving scene video is generated from the first frame of the driving scene image to provide diversified training data for the autonomous driving system. The video generation method of Video-LDM usually uses the existing autonomous driving video data to train the model so that the model can generate driving scene videos, but the generated driving scene videos may not meet the simulation requirements of the driving scene. Summary of the invention

[0004] The embodiments of the present disclosure provide a processing method for generating a driving scene video, a model training method and a device to improve the accuracy and reliability of the generated driving scene video.

[0005] One aspect of an embodiment of the present disclosure provides a method for generating a driving scene video, including:

[0006] Acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of a bird's-eye view of the scene, text data describing the scene environment, and a vehicle control signal; each frame of scene image data includes at least one scene image corresponding to each viewing angle;

[0007] Based on the multimodal scene data, generating scene two-dimensional features corresponding to each of the perspectives and scene environment prediction information of a preset number of frames in the future;

[0008] Based on the two-dimensional features of the scene corresponding to each of the perspectives and the scene environment prediction information, a driving scene video is generated using a video generation model.

[0009] Another aspect of the embodiments of the present disclosure provides a training method for a generation model for generating a driving scene video, comprising:

[0010] Acquire training sample data; the training sample data includes a plurality of first training samples and a plurality of second training samples;

[0011] Performing a first-stage training on a scene environment prediction model to be trained based on each of the first training samples to obtain a scene environment prediction model;

[0012] Determine the generation model to be trained based on the scene environment prediction model obtained from the first stage of training and the generation model of the video to be trained;

[0013] The generation model to be trained is trained in the second stage based on each of the second training samples to obtain a generation model for generating a driving scene video; wherein the training scene environment prediction information predicted by the scene environment prediction model is used as condition information of the generation model of the video to be trained.

[0014] In another aspect of the embodiments of the present disclosure, a processing device for generating a driving scene video is provided, comprising:

[0015] A multimodal data acquisition module, used to acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of a bird's-eye view of the scene, text data describing the scene environment, and a vehicle control signal; each frame of scene image data includes at least one scene image corresponding to each viewing angle;

[0016] A first processing module, configured to generate, based on the multimodal scene data, scene two-dimensional features corresponding to each of the viewing angles and scene environment prediction information of a preset number of frames in the future;

[0017] The second processing module is used to generate a driving scene video using a video generation model based on the two-dimensional features of the scene corresponding to each of the perspectives and the scene environment prediction information.

[0018] Another aspect of the embodiments of the present disclosure provides a training device for generating a generation model for a driving scene video, including:

[0019] A sample acquisition module, used to acquire training sample data; the training sample data includes a plurality of first training samples and a plurality of second training samples;

[0020] A first training module, used for performing a first stage of training on a scene environment prediction model to be trained based on each of the first training samples to obtain a scene environment prediction model;

[0021] The second training module is used to determine the generation model to be trained based on the scene environment prediction model obtained in the first stage of training and the generation model of the video to be trained; the second stage of training is performed on the generation model to be trained based on each of the second training samples to obtain a generation model for generating a driving scene video; wherein the training scene environment prediction information predicted by the scene environment prediction model is used as conditional information of the generation model of the video to be trained.

[0022] According to another aspect of the disclosed embodiment, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in any of the above embodiments of the present disclosure is implemented.

[0023] Another aspect of the embodiments of the present disclosure provides an electronic device, the electronic device comprising:

[0024] Memory for storing computer programs;

[0025] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method described in any of the above embodiments of the present disclosure is implemented.

[0026] Another aspect of the embodiments of the present disclosure provides a computer program product, including computer program instructions, characterized in that when the computer program instructions are executed by a processor, the method described in any of the above embodiments of the present disclosure is implemented.

[0027] The processing method for generating driving scene videos, the model training method and the device provided by the present disclosure can integrate multimodal scene data to generate two-dimensional scene features of each perspective and structured scene environment prediction information of a preset number of frames in the future, so that the scene environment prediction information of the preset number of frames in the future can be used as a conditional signal for generating a driving scene video, and diffusion iteration is performed on the basis of the two-dimensional scene features of each perspective to generate a driving scene video. Since the scene environment prediction information is used as a conditional signal to constrain the generated driving scene video, the generated driving scene video is more in line with the road type, traffic rules, vehicle behavior and other conditions of the driving scene, thereby helping to improve the accuracy and reliability of the driving scene video.

[0028] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0030] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0031] Figure 1 This is an exemplary application scenario of the processing method for generating driving scene videos provided by the present disclosure;

[0032] Figure 2 is a flowchart of a processing method for generating a driving scene video provided by some exemplary embodiments of the present disclosure;

[0033] Figure 3 is a schematic diagram of a process of generating two-dimensional features of a scene corresponding to each perspective provided by some exemplary embodiments of the present disclosure;

[0034] Figure 4 is a schematic diagram of the encoding principle of a multimodal encoder provided by an exemplary embodiment of the present disclosure;

[0035] Figure 5 is a schematic diagram of a process for generating scene environment prediction information of a preset number of frames in the future provided by some exemplary embodiments of the present disclosure;

[0036] Figure 6 is a flowchart of step 220 provided in some optional embodiments of the present disclosure;

[0037] Figure 7 is a schematic diagram of a process for predicting scene environment prediction information provided by some exemplary embodiments of the present disclosure;

[0038] Figure 8 is a schematic diagram of a network structure of a scene environment prediction model provided by an exemplary embodiment of the present disclosure;

[0039] Fig. 9 is a schematic diagram of a process for generating scene environment prediction information of a preset number of frames in the future provided by other exemplary embodiments of the present disclosure;

[0040] Fig.10 is a flowchart of a processing method for generating a driving scene video provided by some other exemplary embodiments of the present disclosure;

[0041] Fig.11 is a flowchart of a processing method for generating a driving scene video provided by some further exemplary embodiments of the present disclosure;

[0042] Fig.12 is a schematic diagram of a network structure of a generation model provided by an exemplary embodiment of the present disclosure;

[0043] Fig.13 is a flowchart of a training method for a generative model for generating a driving scene video provided by some exemplary embodiments of the present disclosure;

[0044] Fig.14 is a flowchart of a training method for a generation model for generating a driving scene video provided by some other exemplary embodiments of the present disclosure;

[0045] Fig.15 is a schematic diagram of a training process of the first stage provided by some exemplary embodiments of the present disclosure;

[0046] Fig.16 is a schematic diagram of the training process of the second stage provided by some exemplary embodiments of the present disclosure;

[0047] Fig.17 is a schematic diagram of the training principle of a generative model to be trained provided by an exemplary embodiment of the present disclosure;

[0048] Fig.18 is a structural schematic diagram of a processing device for generating a driving scene video provided by some exemplary embodiments of the present disclosure;

[0049] Fig.19 is a schematic diagram of a structure of a training device for generating a generation model for a driving scene video provided by some exemplary embodiments of the present disclosure;

[0050] Fig. 20 It is a structural schematic diagram of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0051] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0052] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0053] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0054] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0055] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0056] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0057] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0058] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0059] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0060] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0061] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0062] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0063] SUMMARY OF THE DISCLOSURE

[0064] In the process of realizing the present disclosure, the inventors found that in the field of autonomous driving, driving scene videos are usually generated based on Video-LDM (Video-Latent Diffusion Models), that is, based on the diffusion model, the driving scene video is generated from the first frame of the driving scene image to provide diversified training data for the autonomous driving system. The video generation method of Video-LDM usually uses the existing autonomous driving video data to train the model so that the model can generate driving scene videos, but the generated driving scene videos may not meet the simulation requirements of the driving scene.

[0065] Exemplary Overview

[0066] Figure 1 This is an exemplary application scenario of the processing method for generating driving scene videos provided by the present disclosure.

[0067] like Figure 1As shown, in the field of autonomous driving, the processing method for generating driving scene videos disclosed in the present invention can be used to generate scene two-dimensional features and scene environment prediction information of a preset number of frames in the future based on multimodal scene data and using a pre-trained scene environment prediction model. The multimodal scene data may include scene data of modalities such as multi-view video data, multi-view image data, scene bird's-eye view, scene environment description text, and vehicle control signals, or the multimodal scene data may include a part of the scene data of the aforementioned modalities, for example, the multimodal scene data may include multi-view images or multi-view videos, and also include at least one of a scene bird's-eye view, a scene environment description text, and a vehicle control signal, without specific limitation. The scene two-dimensional features of each perspective may include the scene features of the perspective, and the scene environment prediction information of a preset number of frames in the future may include road environment prediction information and surrounding object motion information of a preset number of frames in the future. The road environment prediction information may include road map information such as lane lines, curbs, and traffic signs, and the surrounding object motion information may include motion trajectory information of surrounding objects, or may include the position, direction, speed, acceleration, angular velocity, etc. of surrounding objects in each future frame, which may be set according to actual needs. Then, based on the two-dimensional features of the scene corresponding to each perspective and the scene environment prediction information, a driving scene video is generated using a video generation model, wherein the scene environment prediction information is used as a condition for generating a driving scene video, so that the generated driving scene video is more in line with the simulation requirements of the driving scene such as road type, traffic rules and vehicle behavior, thereby improving the accuracy and reliability of the driving scene video.

[0068] Exemplary Methods

[0069] Figure 2 FIG. 1 is a flow chart of a processing method for generating a driving scene video provided by some exemplary embodiments of the present disclosure. Figure 2 As shown, the method comprises the following steps:

[0070] Step 210: Acquire multimodal scene data.

[0071] Among them, the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of a bird's-eye view of the scene, text data describing the scene environment, and a vehicle control signal; each frame of scene image data includes a scene image corresponding to at least one perspective.

[0072] In some optional embodiments, at least one frame of scene image data may include one or more frames of image data, or include video data, or include image data and video data. The scene bird's-eye view refers to the view from the perspective of BEV (bird's-eye-view). Based on the bird's-eye view, the features of the vehicle from different perspectives can be unified and expressed under the same perspective.

[0073] In some optional embodiments, at least one frame may include the Tth frame (e.g., the current frame) of the vehicle collected data or include the n+1th frame from the Tnth frame to the Tth frame, where n is a positive integer. For example, at least one frame includes the current frame and the nth frame of the historical frame, which is used to generate the future M frames of the driving scene video of the current frame, and is not specifically limited.

[0074] In some optional embodiments, the scene environment description text data (Text) is scene environment information described by text, for example, text describing weather information, road information and other environmental information, such as "raining", "Drive in the countryside", etc. The vehicle control signal (Command) is a command signal for controlling the driving of the vehicle, such as a signal indicating the vehicle speed, steering, etc.

[0075] In some optional embodiments, the vehicle control signal may be a type of text data.

[0076] Step 220, based on the multimodal scene data, generate the scene two-dimensional features corresponding to each perspective and the scene environment prediction information of the future preset frame number.

[0077] The future preset frame number may be any frame number, such as 1 frame, 2 frames, 5 frames, 10 frames, etc., without specific limitation. The scene environment prediction information may include road environment prediction information and vehicle surrounding object motion information. The road environment prediction information may include road map information such as lane lines, curbs, and traffic signs, and the surrounding object motion information may include motion trajectory information of the surrounding objects, or include the position, direction, speed, acceleration, angular velocity, etc. of the surrounding objects in each future frame, without specific limitation.

[0078] In some optional embodiments, multimodal features may be generated based on multimodal scene data, and then two-dimensional scene features corresponding to each perspective and scene environment prediction information for a preset number of frames in the future may be generated based on the multimodal features.

[0079] Step 230 , based on the two-dimensional features of the scene corresponding to each perspective and the scene environment prediction information, a driving scene video is generated using a video generation model.

[0080] The video generation model can be implemented based on any feasible model, for example, it can be implemented based on a video diffusion model. The two-dimensional features of the scene corresponding to each perspective are used as the initial signal, and the scene environment prediction information of the future preset frame number is used as the conditional signal for generating the driving scene video, so that on the basis of the initial signal, the driving scene video that meets the scene environment conditions of the future preset frame number is continuously diffused and iterated to generate.

[0081] The processing method for generating a driving scene video provided by the embodiment of the present disclosure can integrate multimodal scene data to generate two-dimensional scene features of each perspective and structured scene environment prediction information of a preset number of frames in the future, so that the scene environment prediction information of the preset number of frames in the future can be used as a conditional signal for generating a driving scene video, and diffusion iteration is performed on the basis of the two-dimensional scene features of each perspective to generate a driving scene video. Since the scene environment prediction information is used as a conditional signal to constrain the generated driving scene video, the generated driving scene video is more in line with the road type, traffic rules, vehicle behavior and other conditions of the driving scene, thereby helping to improve the accuracy and reliability of the driving scene video.

[0082] In some optional embodiments, Figure 3 is a schematic diagram of a process for generating two-dimensional features of a scene corresponding to each viewing angle provided by some exemplary embodiments of the present disclosure, such as Figure 3 As shown, in step 220, based on the multimodal scene data, generating the scene two-dimensional features corresponding to each perspective respectively includes:

[0083] Step 2210: Encode the multimodal scene data using a multimodal encoder to obtain multimodal features.

[0084] Among them, the multimodal encoder can jointly encode data of various modes in the multimodal scene data to obtain multimodal features.

[0085] In some optional embodiments, the multimodal encoder may be a Transformer-based encoder, and the encoding process may include: converting the scene data of each modality into one-dimensional Token information, and then combining the Token information of the scene data of each modality to obtain a combination result, position-encoding the combination result to obtain a first encoding result, Transformer encoding the first encoding result after position encoding to obtain a second encoding result, and using the second encoding result as a multimodal feature. Transformer encoding of the first encoding result may be implemented based on a Transformer Encoder (Transformer Encoder), and the Transformer Encoder includes an attention layer, which can perform attention interaction between multimodalities to better capture the correlation information between multimodal scene data, thereby obtaining more effective multimodal features. The specific structure of the multimodal encoder is not limited.

[0086] For example, Figure 4 FIG. 1 is a schematic diagram of the encoding principle of a multimodal encoder provided by an exemplary embodiment of the present disclosure. Figure 4As shown, the multimodal scene data includes multi-view scene video data (Multi View Video), multi-view scene image data (Multi view Image), scene bird's-eye view (BEV), scene environment description text data (Text) and vehicle control signal (Command). The multimodal encoder first preprocesses the multimodal scene data, such as performing Patch decomposition on two-dimensional signals such as video, image, VEV, obtaining one-dimensional Token information, tokenizing Text and Command, and converting them into one-dimensional Token information through a linear mapping layer. After all the tokens are combined, the position encoder is used to position encode the tokens of each mode. Position encoding helps to capture the position relationship of different tokens in the subsequent encoding process, thereby improving the encoding quality. The position encoding result is input into the Transformer Encoder as the first encoding result. The first encoding result is encoded by the Transformer Encoder to better capture the correlation information between the multimodal scene data, thereby obtaining more effective multimodal features (Multi-modal Features).

[0087] Step 2220, extracting visual features corresponding to the scene image data from the multimodal features.

[0088] Among them, visual features can be extracted from multimodal features based on the order of position coding. The Transformer encoder encodes the position coding results mainly to realize the interaction between multimodalities. For example, for the position coding results of any modality, the information of other modalities is fused into this modality, and the information of this modality is also fused into other modalities. In the obtained multimodal features, the feature order of each modality can remain unchanged. Based on this, the visual features corresponding to the scene image data can be extracted from the multimodal features. Similarly, the features of other modalities can also be extracted.

[0089] Step 2230, converting the visual features into a three-dimensional grid coordinate system to obtain three-dimensional spatial features corresponding to the visual features.

[0090] In some optional embodiments, the three-dimensional grid coordinate system may be a grid coordinate system in a three-dimensional space centered on the vehicle, for example, a three-dimensional grid coordinate system centered on the rear axle center of the vehicle in the Tth frame (eg, the current frame).

[0091] In some optional embodiments, the conversion of visual features to a three-dimensional grid coordinate system can be implemented based on any feasible method, for example, it can be converted based on implicit depth information based on a Transformer, or it can be converted based on a displayed depth estimation projection method, that is, based on LSS (Lift Splat Shoot), the visual features are converted to a three-dimensional grid coordinate system to obtain the three-dimensional spatial features corresponding to the visual features. Specifically, the visual features can be converted to a three-dimensional grid coordinate system based on the Lift operation in LSS. The Lift operation process may include feature extraction and depth estimation, outer product operation and grid sampling. Among them, feature extraction is used to further extract features of visual features to obtain target features, depth estimation is used to estimate the depth features of visual features, obtain the depth features corresponding to the target features, and construct the target features and depth features into a comprehensive feature through the outer product operation, and then project the comprehensive features to the corresponding grid through GridSampling to obtain the three-dimensional spatial features under the three-dimensional grid coordinate system. The specific principles of LSS are not repeated here.

[0092] Step 2240: Determine the two-dimensional features of the scene corresponding to each perspective based on the three-dimensional spatial features.

[0093] Among them, the three-dimensional space features can be converted into the two-dimensional space of each perspective to obtain the two-dimensional features of the scene corresponding to each perspective.

[0094] In some optional embodiments, determining the two-dimensional features of the scene corresponding to each perspective based on the three-dimensional space features in step 2240 includes:

[0095] By using the neural radiation field rendering method, the two-dimensional features of the scene corresponding to each perspective are obtained from the three-dimensional space features.

[0096] Among them, Neural Radiance Field (NeRF) is a 3D reconstruction technology based on neural networks. It models the scene as a continuous 5D radiation field and implicitly stores it in the neural network. It can input two-dimensional features from multiple angles and obtain a neural radiation field model through training. Based on this model, the two-dimensional features of the scene at any perspective can be rendered.

[0097] This embodiment obtains multimodal features by encoding multimodal scene data, which helps to better capture the correlation information between multimodal scene data, thereby obtaining more effective multimodal features, and then obtains three-dimensional spatial features based on the multimodal features. Based on the three-dimensional spatial features, the two-dimensional features of the scene at each perspective are determined, which helps to provide an effective initial signal for generating a driving scene video, and facilitates the generation of accurate and effective driving scene videos through diffusion iteration.

[0098] In some optional embodiments, Figure 5 is a schematic diagram of a process for generating scene environment prediction information of a preset number of frames in the future provided by some exemplary embodiments of the present disclosure, such as Figure 5 As shown, after the three-dimensional space features are generated in step 2230, the scene environment prediction information of the future preset number of frames is generated based on the multimodal scene data in step 220, including:

[0099] Step 2250, predicting scene environment prediction information based on three-dimensional spatial features.

[0100] Among them, the three-dimensional spatial feature is a feature in the three-dimensional spatial coordinate system centered on the vehicle, which contains the three-dimensional spatial features corresponding to the multimodal scene data, based on which the scene environment prediction information of the future frame can be predicted, such as the road map information of the future frame and the motion information of the objects around the vehicle.

[0101] In some optional embodiments, the scene environment prediction information may be predicted based on any implementable prediction network, for example, the scene environment prediction information may be predicted based on a Transformer prediction network.

[0102] In some optional embodiments, the three-dimensional spatial features may be further spatiotemporally encoded to further learn the spatiotemporal relationship between the three-dimensional spatial features, which helps to further improve the accuracy and effectiveness of the scene environment prediction information.

[0103] This embodiment predicts scene environment prediction information of future frames through three-dimensional spatial features, which is used as condition information for generating driving scene videos, helping to make the generated driving scene videos more consistent with the simulation conditions of the driving scene.

[0104] In some optional embodiments, Figure 6 FIG. 2 is a flow chart of step 220 provided in some optional embodiments of the present disclosure. Figure 6 As shown, step 220 includes the above-mentioned steps 2210 to 2250, wherein step 2240 and step 2250 are executed in no particular order and can be executed in parallel or one after the other.

[0105] In some optional embodiments, Figure 7 is a flowchart of predicting scene environment prediction information provided by some exemplary embodiments of the present disclosure, such as Figure 7 As shown, the step 2250 of predicting scene environment prediction information based on the three-dimensional space features includes:

[0106] Step 22510, use the space-time encoder to perform space-time encoding processing on the three-dimensional spatial features to obtain the encoded target three-dimensional spatial features.

[0107] The spatiotemporal encoder may be an encoder based on a self-attention mechanism. The spatiotemporal encoder may be used to perform attention interaction on three-dimensional spatial features to better capture the temporal and spatial relationship of three-dimensional spatial features. By considering the temporal information of different frames, the obtained target three-dimensional spatial features are helpful to decode more accurate road environment prediction information and surrounding object motion information of a preset number of future frames.

[0108] Step 22520, using the first decoder, decodes the target three-dimensional spatial features to obtain road environment prediction information for a preset number of frames in the future.

[0109] The first decoder may be any series of decoders based on Transformer, or any other implementable decoder, as long as it can decode the road environment prediction information of the future frame, without any specific limitation.

[0110] Step 22530, using the second decoder, decodes the target three-dimensional spatial features to obtain motion information of surrounding objects for a preset number of frames in the future.

[0111] The second decoder can be any series of decoders based on Transformer, or any other implementable decoder, as long as it can decode the motion information of surrounding objects in future frames, without specific limitation. Surrounding objects refer to other objects around the vehicle. For example, surrounding objects may include other vehicles, pedestrians, cyclists and other dynamic objects around the vehicle.

[0112] Step 22540, using the road environment prediction information and the surrounding object motion information as scene environment prediction information.

[0113] Among them, since the road environment prediction information and the surrounding object motion information are obtained based on decoder decoding, they have a certain data structure. Therefore, the obtained scene environment prediction information is structured scene environment information, which is helpful as conditional information for generating driving scene videos, constraining the generated driving scene videos to satisfy the conditional information, thereby making the generated driving scene videos meet the simulation conditions of the driving scenes.

[0114] In some optional embodiments, the spatiotemporal encoder, the first decoder, and the second decoder that predict the scene environment prediction information based on the three-dimensional spatial features may form a scene environment prediction model. Figure 8 Schematic diagram of the network structure of the scene environment prediction model provided by an exemplary embodiment of the present disclosure. Figure 8As shown, multi-view image features refer to visual features extracted from multimodal features, T represents the Tth frame, Ti represents the Tith frame, i=0, 1, 2, ..., N, N is a positive integer, and the multi-view image features are converted into three-dimensional spatial features by LSS, and the first decoder (Map decoder) decodes the three-dimensional spatial features (Spatial Temporal Voxel, spatiotemporal voxel) to obtain road environment prediction information for a preset number of frames in the future. The second decoder (Motion decoder) decodes the three-dimensional spatial features to obtain motion information of surrounding objects for a preset number of frames in the future. In practical applications, a spatiotemporal encoder can also be included before the Map decoder and the Motion decoder, which is used to perform spatiotemporal encoding on the Spatial Temporal Voxel to obtain the encoded target three-dimensional spatial features, then the Map decoder decodes the target three-dimensional spatial features to obtain road environment prediction information for a preset number of frames in the future, and the Motion decoder decodes the target three-dimensional spatial features to obtain motion information of surrounding objects for a preset number of frames in the future.

[0115] In some optional embodiments, the scene environment prediction model may also include a multimodal encoder and a module for generating scene two-dimensional features.

[0116] This embodiment performs spatiotemporal encoding of three-dimensional spatial features through a spatiotemporal encoder, which helps to better capture the spatiotemporal relationship of three-dimensional spatial features, so that the obtained target three-dimensional spatial features are more helpful in decoding the road environment prediction information and surrounding object motion information of the future preset frame number, and improve the accuracy and effectiveness of the predicted road environment prediction information and surrounding object motion information. In addition, through multi-task reasoning of road environment prediction information and surrounding object motion information, it is helpful to comprehensively consider the relationship between different tasks, thereby improving the accuracy and robustness of the prediction results.

[0117] In some optional embodiments, Fig. 9 is a schematic diagram of a process for generating scene environment prediction information of a preset number of frames in the future provided by other exemplary embodiments of the present disclosure, such as Fig. 9 As shown, in the case where the multimodal scene data includes a bird's-eye view of the scene, after obtaining the multimodal features in step 2210, the scene environment prediction information of a preset number of frames in the future is generated based on the multimodal scene data in step 220, including:

[0118] Step 22a0, extracting bird's-eye view features corresponding to the scene bird's-eye view from the multimodal features.

[0119] Among them, the extraction of bird's-eye view features can be implemented based on the order of multimodal features, which is similar to the aforementioned extraction of visual features and will not be repeated here.

[0120] Step 22b0, predicting scene environment prediction information based on the bird's-eye view features.

[0121] Among them, the bird's-eye view features more directly include the scene environment features corresponding to the scene image data and the states of objects around the vehicle in at least one frame corresponding to the scene image data, such as position, direction, speed, angular velocity, etc. Based on the bird's-eye view features, the scene environment prediction information of future frames can be better predicted.

[0122] In some optional embodiments, the bird's-eye view features may be decoded based on a third decoder to obtain scene environment prediction information. The third decoder may be any implementable decoder, such as a series decoder based on a Transformer.

[0123] This embodiment predicts scene environment prediction information by using bird's-eye view features, which helps to further improve the accuracy and effectiveness of scene environment prediction information.

[0124] In some optional embodiments, the bird's-eye view features and the aforementioned three-dimensional spatial features (or target three-dimensional spatial features) may be combined to predict the structured scene environment prediction information of future frames to further improve the accuracy and effectiveness of the scene environment prediction information.

[0125] In some optional embodiments, the multimodal scene data includes at least one frame of scene image data, scene environment description text data, and a vehicle control signal. Fig.10 FIG. 1 is a flow chart of a processing method for generating a driving scene video provided by some other exemplary embodiments of the present disclosure. Fig.10 As shown, the method of the embodiment of the present disclosure also includes:

[0126] Step 310: extracting text features corresponding to the scene environment description text data and command features corresponding to the vehicle control signal from the multimodal features.

[0127] The extraction of text features and command features may refer to the extraction of visual features described above, and will not be described in detail here.

[0128] Step 230 generates a driving scene video using a video generation model based on the scene two-dimensional features and scene environment prediction information corresponding to each perspective, including:

[0129] Step 2310, based on the scene two-dimensional features, scene environment prediction information, text features and command features corresponding to each perspective, a video generation model is used to generate a driving scene video.

[0130] Among them, text features, command features and scene environment prediction information are used together as conditional information for generating driving scene videos, so that the generated driving scene videos can meet the road environment conditions of the surrounding environment prediction information, the motion conditions of surrounding objects, and the weather conditions, road conditions, etc. described by the text data, and the movement of the vehicle meets the vehicle behavior conditions of the vehicle control signal, thereby further improving the effectiveness and reliability of the generated driving scene videos, so that the generated driving scene videos can be used to simulate more realistic driving scenes, so that they can be more effectively used for performance optimization of vehicle automatic driving systems.

[0131] In some optional embodiments, Fig.11 FIG. 1 is a flow chart of a processing method for generating a driving scene video provided by some exemplary embodiments of the present disclosure. Fig.11 As shown, the multimodal scene data includes at least one frame of scene image data, scene environment description text data and vehicle control signals. Steps 2210 to 2250 of step 220 are combined with steps 310 and 2310 to implement the editing function of the generated driving scene video based on the text features and the vehicle control signal, so that the generated driving scene video can further meet the weather conditions, vehicle behavior and other conditions described in the text, and further improve the richness of the driving scene video.

[0132] In some optional embodiments, the multimodal scene data may include at least one frame of scene image data, a scene bird's-eye view corresponding to at least one frame of image data, scene environment description text data, and a vehicle control signal. Fig.10 Step 220 shown may also include Figure 3 Steps 2210 to 2240 and Fig. 9 22a0 to 22b0.

[0133] In some optional embodiments, the above steps of the processing method for generating driving scene videos in the embodiment of the present disclosure can be implemented based on a complete generation model for generating driving scene videos, and the generation model can be obtained by pre-training. Fig.12 Schematic diagram of the network structure of the generation model provided by an exemplary embodiment of the present disclosure. Fig.12 As shown in the figure, Driving World Model represents the generative model. Of course, the generative model can also be represented by other names, not limited to the names in the figure. Multimodal scene data can be found in Figure 4, which will not be elaborated here. The generation model includes a multimodal encoder, an LSS conversion module (multi-view geometric constraints), a perspective rendering module, a future frame structured prediction module, and a video generation module (Video Diffusion). The multimodal scene data is processed by the multimodal encoder to obtain multimodal features, and visual features (multi-view image features) are extracted from the multimodal features. The LSS conversion module converts the visual features into three-dimensional space features based on LSS. The three-dimensional space features are rendered by perspective to obtain the scene two-dimensional features of each perspective. The scene two-dimensional features of each perspective are used as the initial signal of the video generation module. The future frame structured prediction module processes the three-dimensional space features based on the scene environment prediction model to obtain structured scene environment prediction information (i.e., structured Conditions), and extracts text features (Text Conditions) and / or command features (Command Conditions) from the multimodal features. Video Diffusion uses the video generation model to continuously perform diffusion iterations based on the constraints of each Conditions on the basis of the initial signal (the scene two-dimensional features of each perspective) to achieve driving scene video generation / prediction, video editing and other functions. For example, under the constraints of structured conditions (Structured Conditions), basic driving scene videos can be generated. Under the constraints of structured conditions and text conditions (Text Conditions), the basic driving scene videos can be edited for weather and other environments to generate driving scene videos under different weather conditions (referred to as video editing). Video editing can refer to generating driving scene videos of corresponding conditions based on the constraints of text features, such as generating driving scene videos under different weather conditions. Motion prediction and 3D perception can also be tasks that can be achieved by the generation model of the embodiment of the present disclosure. Motion prediction refers to predicting the future motion trajectory of other objects around the vehicle, and 3D perception is the perception of the 3D environment around the vehicle in future frames. The specific task type can be set according to actual needs.

[0134] In some optional embodiments, the network structure of the video generation model can be a model based on LDM (Latent space Diffusion Model). For example, a temporal layer (also called a time series layer) can be introduced in LDM to achieve alignment on the latent time series. Based on the pre-trained image generation model (a model for generating images from text), for Unet, a temporal layer is added, and the temporal layer can include a 3D convolutional layer and a temporal attention layer. The two-dimensional VAE (Variational autoencoders) in the image generation model is expanded, and a temporal layer is added to the decoder part. The spatial layer parameters are fixed, and fine-tuning training is performed on the video data to achieve the conversion of the image generator into a video generator. Then, combined with the scene environment prediction model of the present invention for joint training, the generation model of the present invention for generating driving scene videos is obtained. The specific network structure of the video generation model can be set according to actual needs, which will not be repeated here. The specific process of the generation model may include: generating latent key frames, inserting frames based on the diffusion model, retaining adjacent latent key frames as both ends of the inserted frame fragment, initializing the middle frame to be inserted with noise, and then generating a video through a decoder, and then increasing the resolution of the video based on the super-resolution module.

[0135] In some optional embodiments, the network structure of the generation model may also be implemented using other possible networks, and the specific network structure of the generation model is not limited.

[0136] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict, and may be specifically configured according to actual needs.

[0137] Any processing method for generating a driving scene video provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any processing method for generating a driving scene video provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any processing method for generating a driving scene video mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0138] Fig.13 is a flowchart of a training method for generating a generation model for a driving scene video provided by some exemplary embodiments of the present disclosure. Fig.13 As shown, the method comprises the following steps:

[0139] Step 410: Acquire training sample data; the training sample data includes a plurality of first training samples and a plurality of second training samples.

[0140] In some optional embodiments, each first training sample may include two-dimensional features of a training scene corresponding to at least one frame of training scene image data, and true value information of a future preset number of training scene environments corresponding to the training scene image data.

[0141] In some optional embodiments, each first training sample may include training multimodal scene data, and true value information of the training scene environment for a preset number of frames in the future. It can be specifically determined according to the network structure of the scene environment prediction model to be trained. For example, if the network structure of the scene environment prediction model to be trained includes the above-mentioned spatiotemporal encoder, the first decoder and the second decoder, the first training sample may include the two-dimensional features of the training scene corresponding to at least one frame of training scene image data, and the true value information of the training scene environment for a preset number of frames in the future corresponding to the training scene image data. If the network structure also includes a multimodal encoder and a module for generating two-dimensional features of the training scene, the first training sample may include training multimodal scene data, and true value information of the training scene environment for a preset number of frames in the future.

[0142] In some optional embodiments, each second training sample includes training multimodal scene data, and a training driving scene ground-truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, training scene environment description text data, and a training vehicle control signal; each frame of training scene image data includes a training scene image from at least one perspective.

[0143] In some optional embodiments, for each first training sample, the determination of the two-dimensional features of the training scene corresponding to the training scene image data included in the first training sample is consistent with the scene two-dimensional features in the model reasoning stage of the aforementioned embodiment, that is, it can be determined based on the training visual features in the training multimodal features generated based on the training multimodal data, which will not be elaborated here. The training scene environment truth information may include the training road environment truth information and the training surrounding object motion information of a preset number of frames in the future. The training scene environment truth information may be obtained by pre-annotation, and the annotation method may include manual annotation, semi-automatic annotation, automatic annotation, etc., which is not specifically limited. The training scene environment truth information is used to supervise the training scene environment prediction information obtained by the training scene environment prediction model to promote the convergence of the model.

[0144] In some optional embodiments, for each second training sample, the training multimodal scene data included in the second training sample is consistent with the multimodal scene data in the model reasoning stage of the aforementioned embodiment, which will not be repeated here. The training driving scene truth video can be a real video of the driving scene obtained in advance. In practical applications, the training multimodal scene data can be obtained based on the training driving scene truth video. For example, a large number of driving scene videos can be obtained in advance as training driving scene truth videos. For any training driving scene truth video, the first n frames of the training driving scene video are used as the training scene image data corresponding to the training driving scene video, n is a positive integer, based on the training scene image data, and the data of other modalities corresponding to the training scene image data are obtained, and the training scene image data and the data of other modalities are used as training multimodal scene data. Consistent with the reasoning stage of the aforementioned embodiment, the data of other modalities may include at least one of a training scene bird's-eye view, a training scene environment description text data, and a training vehicle control signal. Similarly, consistent with the scene image data of the aforementioned embodiment, each frame of training scene image data may include a training scene image of at least one perspective.

[0145] Step 420 , performing a first stage of training on the scene environment prediction model to be trained based on each first training sample to obtain a scene environment prediction model.

[0146] Among them, the network structure of the scene environment prediction model to be trained can refer to the aforementioned embodiment. For example, the network structure of the scene environment prediction model to be trained includes a spatiotemporal encoder, a first decoder and a second decoder. In some optional embodiments, on the basis of including the spatiotemporal encoder, the first decoder and the second decoder, the network structure of the scene environment prediction model to be trained can also include a multimodal encoder and a module for generating two-dimensional features of the training scene (for example, including the LSS conversion module and the perspective rendering module of the aforementioned embodiment). In this case, the first training sample can include training multimodal scene data and training training scene environment true value information, which can be specifically set according to actual needs.

[0147] In some optional embodiments, in the first stage of the training process, the training scene environment prediction information of the future preset number of frames can be predicted based on the two-dimensional features of the training scene at each perspective, the training scene environment prediction information is compared with the training scene environment true value information, the model loss is calculated according to the preset loss function, the update gradient of the model network parameters is determined based on the model loss, the model network parameters are updated, and the updated scene environment prediction model to be trained is obtained. Continue to train the updated scene environment prediction model to be trained according to the above process until the training end condition is reached to obtain the scene environment prediction model. The training end condition may include, for example, model convergence, the number of iterations reaching the first preset number of times, etc., which are not specifically limited. The preset loss function can adopt any implementable loss function, such as a cross entropy loss function, a mean square error (MSE) loss function, etc.

[0148] In some optional embodiments, when the first training sample includes training multimodal scene data, if the training multimodal scene data includes a training scene bird's-eye view, training scene environment prediction information for a preset number of frames in the future may also be predicted based on the training scene bird's-eye view.

[0149] In some optional embodiments, the plurality of first training samples may be divided into a plurality of batches, and each iteration uses the first training samples of one batch.

[0150] Step 430, based on the scene environment prediction model obtained by the first stage training and the video generation model to be trained, determine the generation model to be trained.

[0151] Among them, after the scene environment prediction model is obtained in the first stage of training, the generation model to be trained is determined based on the scene environment prediction model and the video generation model to be trained, so that in the second stage of training, the prediction result of the scene environment prediction model can be used as the structured condition of the video generation model to be trained, and the scene environment prediction model and the video generation model to be trained are jointly trained.

[0152] In some optional embodiments, if the scene environment prediction model includes a spatiotemporal encoder, a first decoder, and a second decoder, the network structure of the generated model to be trained may include a multimodal encoder and a module for generating three-dimensional spatial features and two-dimensional scene features in addition to the scene environment prediction model and the generated model to be trained. That is, the network structure of the generated model to be trained can generate a training driving scene prediction video based on the training multimodal scene data, and the details are not repeated here.

[0153] In some optional embodiments, the network structure of the video generation model to be trained can refer to the video generation model of any of the aforementioned embodiments, which will not be repeated here.

[0154] Step 440 , performing a second stage of training on the generation model to be trained based on each second training sample, to obtain a generation model for generating a driving scene video.

[0155] Among them, the training scene environment prediction information predicted by the scene environment prediction model is used as the condition information of the video generation model to be trained, and is used to constrain the generated training driving scene prediction video so that the driving scene prediction video can meet the conditions of the structured training scene environment prediction information output by the scene environment prediction model.

[0156] In some optional embodiments, the training multimodal scene data of the second training sample can be processed based on the generative model to be trained to generate a training driving scene prediction video, the training driving scene prediction video is compared with the training driving scene true value video of the second training sample, and the model loss is calculated based on a preset loss function, and then the network parameter update gradient of the generative model to be trained is calculated based on the model loss, and the network parameters of the generative model to be trained are updated based on the updated gradient to obtain the updated generative model to be trained, and the updated generative model to be trained is continued to be trained according to the above process until the end condition of the second stage training is reached to obtain the generative model. The end condition of the second stage training may include model convergence, the number of iterations reaching the second preset number, etc. The loss function of the second stage training can adopt a loss function that can be implemented by the video generation model, and is not specifically limited.

[0157] In some optional embodiments, during the second stage of training, the to-be-trained video generation model can be first trained based on the structured conditions output by the scene environment prediction model. After the to-be-trained video generation model converges, the scene environment prediction model can be further optimized using a large number of true training driving scene videos and training driving scene prediction videos generated by the to-be-trained video generation model. Structured conditions are then generated based on the optimized scene environment prediction model to continue optimizing the video generation model, and so on. Through iterative training, the generalization ability and adaptability of the generation model are continuously improved, so that it can effectively predict structured data (scene environment prediction information) in various scenarios, which helps to achieve the grand unification of visual generation and understanding.

[0158] In some optional embodiments, the plurality of second training samples may be divided into a plurality of batches, and each iteration uses the second training samples from one batch.

[0159] Through two-stage training, this embodiment can effectively improve the performance of the generated model, so that the driving scene video generated by the model can better meet the simulation requirements of the driving scene, thereby improving the accuracy and effectiveness of the driving scene video.

[0160] Fig.14 It is a flowchart of a training method for a generative model for generating driving scene videos provided by other exemplary embodiments of the present disclosure.

[0161] In some optional embodiments, each first training sample includes training multimodal scene data and future preset number of frames of training scene environment truth information; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, a training scene environment description text data, and a training vehicle control signal; each frame of training scene image data includes a training scene image of at least one perspective; Fig.14 As shown, step 420 performs a first phase of training on the scene environment prediction model to be trained based on each first training sample to obtain the scene environment prediction model, including:

[0162] Step 4210, performing the following operations from step 42110 to step 42150 through the scene environment prediction model to be trained, until the first stage training of the scene environment prediction model to be trained is completed, and the scene environment prediction model is obtained (see Fig.15 ):

[0163] Step 42110: For any first training sample, based on the training multimodal scene data in the first training sample, determine the first training scene two-dimensional features corresponding to each perspective.

[0164] Step 42120, using the to-be-trained scene environment prediction model to predict the first training scene two-dimensional features corresponding to each perspective, to obtain first training scene environment prediction information for a preset number of frames in the future.

[0165] It should be noted that the specific operating principles of step 42110 and step 42120 are consistent with the generation of scene two-dimensional features of each perspective and the predicted scene environment prediction information in the aforementioned embodiment, and will not be repeated here.

[0166] Step 42130, based on the first training scene environment prediction information and the training scene environment true value information in the first training sample, determine the first loss corresponding to the first training sample.

[0167] Among them, the first loss can be determined based on a first loss function, and the first loss function can be a cross entropy loss function, a mean square error loss function, etc., which is not specifically limited.

[0168] Step 42140: Determine the first model loss based on the first losses corresponding to each first training sample.

[0169] The first model loss can be achieved by summing or averaging the first losses of each first training sample, and the specific details are not limited.

[0170] Step 42150: Update the network parameters of the scene environment prediction model to be trained based on the first model loss to obtain an updated scene environment prediction model to be trained.

[0171] Among them, the update gradient of the network parameters can be determined based on the first model loss, and the network parameters can be updated based on the update gradient to obtain an updated scene environment prediction model to be trained.

[0172] In some optional embodiments, the update gradient can be obtained based on any implementable gradient descent algorithm, and the gradient descent algorithm may include, for example, a stochastic gradient descent algorithm, a learning rate adaptive gradient descent algorithm, and the like.

[0173] After obtaining the updated scene environment prediction model to be trained, the updated scene environment prediction model to be trained can be used as the above-mentioned scene environment prediction model to be trained, and step 42120 and subsequent steps are repeated until the first stage of training is completed to obtain the updated scene environment prediction model to be trained.

[0174] In some optional embodiments, the first stage of training can be based on a small number of first training samples, which helps to reduce the workload of annotation. In the second stage, the scene environment prediction model is continuously optimized based on a large number of driving scene videos.

[0175] In some optional embodiments, each second training sample includes training multimodal scene data and a training driving scene true value video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, a training scene environment description text data, and a training vehicle control signal; each frame of training scene image data includes a training scene image from at least one perspective. Step 440 of performing a second stage of training the generative model to be trained based on each second training sample to obtain a generative model for generating a driving scene video includes:

[0176] Step 4410, performing the following operations from step 44110 to step 44150 through the generated model to be trained until the second stage of training of the generated model to be trained is completed, and a generated model for generating a driving scene video is obtained (see Fig.16 ):

[0177] Step 44110, for any second training sample, based on the training multimodal scene data in the second training sample, generate the second training scene two-dimensional features corresponding to each perspective and the second training scene environment prediction information of the future preset frame number.

[0178] Among them, the generation principle of the second training scene two-dimensional features and the second training scene environment prediction information of the future preset frame number is consistent with the scene two-dimensional features and the scene environment prediction information of the future preset frame number in the aforementioned embodiment, and will not be repeated here.

[0179] Step 44120, generating a training driving scene prediction video based on the second training scene two-dimensional features corresponding to each perspective and the second training scene environment prediction information.

[0180] Among them, the generation principle of the training driving scene prediction video is consistent with the driving scene video of the aforementioned embodiment, and will not be repeated here.

[0181] Step 44130, based on the training driving scene prediction video and the training driving scene true value video in the second training sample, determine the second loss corresponding to the second training sample.

[0182] Among them, the second loss can be determined according to a second loss function, and the second loss function may include a loss function related to a video generation model, which is not specifically limited.

[0183] Step 44140, determining the second model loss based on the second losses corresponding to each second training sample.

[0184] The second model loss may be determined by summing or averaging the second losses of each second training sample.

[0185] Step 44150: Based on the second model loss, the network parameters of the to-be-trained generation model are updated to obtain an updated to-be-trained generation model.

[0186] Among them, the update gradient of the network parameters can be determined based on the second model loss, and then the network parameters are updated based on the update gradient to achieve the optimization of the generated model to be trained.

[0187] In some optional embodiments, step 44110 generates the second training scene two-dimensional features corresponding to each perspective and the second training scene environment prediction information of the future preset number of frames based on the training multimodal scene data in the second training sample, including:

[0188] The training multimodal scene data in the second training sample is encoded using a multimodal encoder to obtain training multimodal features; training visual features corresponding to the training scene image data are extracted from the training multimodal features; the training visual features are converted into a three-dimensional grid coordinate system to obtain training three-dimensional spatial features corresponding to the training visual features; based on the training three-dimensional spatial features, two-dimensional features of the second training scene corresponding to each perspective are determined; based on the training three-dimensional spatial features, environmental prediction information of the second training scene is predicted; or, in the case where the second training sample includes a bird's-eye view of the training scene, training bird's-eye view features corresponding to the bird's-eye view of the training scene are extracted from the training multimodal features, and based on the training bird's-eye view features, environmental prediction information of the second training scene is predicted.

[0189] Among them, the specific operating principles of each step of predicting the second training scene environment prediction information can be found in the steps of predicting the scene environment prediction information in the aforementioned embodiment, and will not be repeated here.

[0190] In some optional embodiments, Fig.17 FIG. 1 is a schematic diagram of the training principle of a training generation model provided by an exemplary embodiment of the present disclosure. Fig.17 As shown, the network structure of the generated model to be trained is Fig.12 The structure shown (part of the structure is not shown), the multi-view image feature represents the visual feature in the multimodal feature. In the first stage training process, the scene environment prediction model to be trained is trained based on a small amount of annotated data, a first-stage gradient is generated, and the first-stage gradient is back-propagated to update the network parameters to obtain the updated scene environment prediction model to be trained, and the scene environment prediction model is obtained through iterative training until the first stage training is completed. In the second stage training process, the scene environment prediction model predicts the second training scene environment prediction information based on the training multimodal data of the second training sample, and the second training scene environment prediction information is used as the structured information condition of Video Diffusion. Under the constraint of the structured information condition, Video Diffusion generates a training driving scene prediction video based on the training scene two-dimensional features, text features, and command features generated by the training multimodal features, and supervised training is performed based on the training driving scene true value video corresponding to the training multimodal features (i.e., the video data without annotations in the figure), and a second-stage gradient is generated, and back-propagated to achieve the update of the overall network parameters of the training generation model to be trained, and the training generation model to be trained is continuously optimized until the second stage training is completed. The updated generation model to be trained can be used as the generation model.

[0191] This embodiment can effectively improve the model convergence speed through two-stage training, and can improve the generalization ability and adaptability of the model.

[0192] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict, and may be specifically configured according to actual needs.

[0193] Any training method for generating a generation model for a driving scene video provided in the embodiments of the present disclosure can be performed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any training method for generating a generation model for a driving scene video provided in the embodiments of the present disclosure can be performed by a processor, such as the processor executing any training method for generating a generation model for a driving scene video mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be repeated below.

[0194] Exemplary Devices

[0195] Fig.18 is a schematic diagram of a structure of a processing device for generating a driving scene video provided by some exemplary embodiments of the present disclosure. The device of this embodiment can be used to implement the corresponding processing method for generating a driving scene video of the present disclosure, such as Fig.18 The device shown includes: a multimodal data acquisition module 510 , a first processing module 520 and a second processing module 530 .

[0196] The multimodal data acquisition module 510 is used to acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of a bird's-eye view of the scene, text data describing the scene environment, and a vehicle control signal; each frame of scene image data includes a scene image corresponding to at least one perspective.

[0197] The first processing module 520 is used to generate two-dimensional scene features corresponding to each perspective and scene environment prediction information of a preset number of frames in the future based on the multimodal scene data.

[0198] The second processing module 530 is used to generate a driving scene video using a video generation model based on the scene two-dimensional features and scene environment prediction information corresponding to each perspective.

[0199] In some optional embodiments, the first processing module 520 is specifically used for:

[0200] The multimodal scene data is encoded using a multimodal encoder to obtain multimodal features; the visual features corresponding to the scene image data are extracted from the multimodal features; the visual features are converted into a three-dimensional grid coordinate system to obtain the three-dimensional spatial features corresponding to the visual features; based on the three-dimensional spatial features, the two-dimensional features of the scene corresponding to each perspective are determined.

[0201] In some optional embodiments, the first processing module 520 is specifically used for:

[0202] Based on three-dimensional spatial features, predict scene environment prediction information.

[0203] In some optional embodiments, the first processing module 520 is specifically used for:

[0204] A spatiotemporal encoder is used to perform spatiotemporal encoding processing on the three-dimensional spatial features to obtain encoded target three-dimensional spatial features; a first decoder is used to decode the target three-dimensional spatial features to obtain road environment prediction information for a preset number of frames in the future; a second decoder is used to decode the target three-dimensional spatial features to obtain motion information of surrounding objects for a preset number of frames in the future; the road environment prediction information and the surrounding object motion information are used as scene environment prediction information.

[0205] In some optional embodiments, the first processing module 520 is specifically used for:

[0206] In the case where the multimodal scene data includes a bird's-eye view of the scene, bird's-eye view features corresponding to the scene bird's-eye view are extracted from the multimodal features; and based on the bird's-eye view features, scene environment prediction information is predicted.

[0207] In some optional embodiments, the multimodal scene data includes at least one frame of scene image data, scene environment description text data and vehicle control signals. The first processing module 520 is also used for:

[0208] The text features corresponding to the scene environment description text data and the command features corresponding to the vehicle control signal are extracted from the multimodal features.

[0209] The second processing module 530 is specifically used for:

[0210] Based on the two-dimensional scene features, scene environment prediction information, text features and command features corresponding to each perspective, a video generation model is used to generate a driving scene video.

[0211] It should be noted that the specific implementation of the processing device for generating a driving scene video in the embodiment of the present disclosure is similar to the specific implementation of the processing method for generating a driving scene video in the embodiment of the present disclosure. Please refer to the part of the processing method for generating a driving scene video for details. In order to reduce redundancy, it will not be repeated here.

[0212] Fig.19 is a schematic diagram of a structure of a training device for generating a generative model for generating a driving scene video provided by some exemplary embodiments of the present disclosure. The device of this embodiment can be used to implement the corresponding training method embodiment for generating a generative model for generating a driving scene video of the present disclosure, such as Fig.19The device shown includes: a sample acquisition module 610 , a first training module 620 and a second training module 630 .

[0213] The sample acquisition module 610 is used to acquire training sample data; the training sample data includes a plurality of first training samples and a plurality of second training samples.

[0214] The first training module 620 is used to perform a first stage of training on the scene environment prediction model to be trained based on each first training sample to obtain the scene environment prediction model.

[0215] The second training module 630 is used to determine the generation model to be trained based on the scene environment prediction model obtained in the first stage of training and the generation model of the video to be trained; perform the second stage of training on the generation model to be trained based on each second training sample to obtain a generation model for generating a driving scene video; wherein the training scene environment prediction information predicted by the scene environment prediction model is used as conditional information of the generation model of the video to be trained.

[0216] In some optional embodiments, each first training sample includes training multimodal scene data and future preset number of frames of training scene environment truth information; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, a training scene environment description text data, and a training vehicle control signal; each frame of training scene image data includes a training scene image of at least one perspective; the first training module 620 is specifically used to:

[0217] The following operations are performed through the scene environment prediction model to be trained until the first stage of training of the scene environment prediction model to be trained is completed to obtain the scene environment prediction model:

[0218] For any first training sample, based on the training multimodal scene data in the first training sample, determine the first training scene two-dimensional features corresponding to each perspective; use the to-be-trained scene environment prediction model to predict the first training scene two-dimensional features corresponding to each perspective, and obtain first training scene environment prediction information for a preset number of frames in the future; based on the first training scene environment prediction information and the training scene environment true value information in the first training sample, determine the first loss corresponding to the first training sample; based on the first losses corresponding to each first training sample, determine the first model loss; based on the first model loss, update the network parameters of the to-be-trained scene environment prediction model, and obtain an updated to-be-trained scene environment prediction model.

[0219] In some optional embodiments, each second training sample includes training multimodal scene data and a training driving scene ground truth video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, a training scene environment description text data, and a training vehicle control signal; each frame of training scene image data includes a training scene image of at least one perspective. The second training module 630 is specifically used to:

[0220] The following operations are performed through the generative model to be trained until the second stage of training of the generative model to be trained is completed, and a generative model for generating driving scene videos is obtained:

[0221] For any second training sample, based on the training multimodal scene data in the second training sample, generate the second training scene two-dimensional features corresponding to each perspective and the second training scene environment prediction information of the future preset number of frames; based on the second training scene two-dimensional features corresponding to each perspective and the second training scene environment prediction information, generate a training driving scene prediction video; based on the training driving scene prediction video and the training driving scene true value video in the second training sample, determine the second loss corresponding to the second training sample; determine the second model loss based on the second losses corresponding to each second training sample;

[0222] Based on the second model loss, the network parameters of the to-be-trained generation model are updated to obtain an updated to-be-trained generation model.

[0223] In some optional embodiments, the second training module 630 is specifically used for:

[0224] The training multimodal scene data in the second training sample is encoded using a multimodal encoder to obtain training multimodal features; training visual features corresponding to the training scene image data are extracted from the training multimodal features; the training visual features are converted into a three-dimensional grid coordinate system to obtain training three-dimensional spatial features corresponding to the training visual features; based on the training three-dimensional spatial features, two-dimensional features of the second training scene corresponding to each perspective are determined; based on the training three-dimensional spatial features, environmental prediction information of the second training scene is predicted; or, in the case where the second training sample includes a bird's-eye view of the training scene, training bird's-eye view features corresponding to the bird's-eye view of the training scene are extracted from the training multimodal features, and based on the training bird's-eye view features, environmental prediction information of the second training scene is predicted.

[0225] It should be noted that the specific implementation of the device in the embodiment of the present disclosure is similar to the specific implementation of the corresponding method in the embodiment of the present disclosure. Please refer to the corresponding method part for details. In order to reduce redundancy, it will not be repeated here.

[0226] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0227] Memory for storing computer programs;

[0228] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the processing method for generating a driving scene video described in any of the above embodiments of the present disclosure is implemented.

[0229] Fig. 20 FIG. 1 is a schematic diagram of a structure of an application embodiment of the electronic device disclosed in the present invention. Fig. 20 As shown, the electronic device includes one or more processors and memory.

[0230] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0231] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0232] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0233] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0234] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0235] Of course, to simplify, Fig. 20 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0236] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method according to various embodiments of the present disclosure described in the above part of this specification.

[0237] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0238] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above part of this specification.

[0239] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0240] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0241] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0242] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0243] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0244] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0245] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0246] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0247] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating a driving scene video, characterized in that: include: Acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of a bird's-eye view of the scene, text data describing the scene environment, and a vehicle control signal; each frame of scene image data includes at least one scene image corresponding to each viewing angle; Based on the multimodal scene data, generating scene two-dimensional features corresponding to each of the perspectives and scene environment prediction information of a preset number of frames in the future; Based on the two-dimensional features of the scene corresponding to each of the perspectives and the scene environment prediction information, a driving scene video is generated using a video generation model.

2. The method according to claim 1, characterized in that The step of generating two-dimensional scene features corresponding to each of the perspectives based on the multimodal scene data includes: Encoding the multimodal scene data using a multimodal encoder to obtain multimodal features; Extracting visual features corresponding to scene image data from the multimodal features; Converting the visual feature into a three-dimensional grid coordinate system to obtain a three-dimensional spatial feature corresponding to the visual feature; Based on the three-dimensional space features, the two-dimensional features of the scene corresponding to each of the perspectives are determined.

3. The method according to claim 2, characterized in that Based on the multimodal scene data, scene environment prediction information of a preset number of frames in the future is generated, including: Based on the three-dimensional space features, the scene environment prediction information is predicted.

4. The method according to claim 3, characterized in that The predicting the scene environment prediction information based on the three-dimensional space feature includes: Performing spatiotemporal encoding processing on the three-dimensional spatial features using a spatiotemporal encoder to obtain encoded target three-dimensional spatial features; Decoding the target three-dimensional spatial features using a first decoder to obtain road environment prediction information of a preset number of frames in the future; Decoding the target three-dimensional spatial features using a second decoder to obtain motion information of surrounding objects for the future preset number of frames; The road environment prediction information and the surrounding object motion information are used as the scene environment prediction information.

5. The method according to claim 2, characterized in that: Based on the multimodal scene data, scene environment prediction information of a preset number of frames in the future is generated, including: In a case where the multimodal scene data includes the bird's-eye view of the scene, extracting bird's-eye view features corresponding to the bird's-eye view of the scene from the multimodal features; Based on the bird's-eye view features, the scene environment prediction information is predicted.

6. The method according to any one of claims 2 to 5, characterized in that: The multimodal scene data includes the at least one frame of scene image data, the scene environment description text data and the vehicle control signal; The method further comprises: Extracting text features corresponding to the scene environment description text data and command features corresponding to the vehicle control signal from the multimodal features; The step of generating a driving scene video by using a video generation model based on the scene two-dimensional features and the scene environment prediction information respectively corresponding to each of the perspectives includes: The driving scene video is generated using the video generation model based on the scene two-dimensional features, the scene environment prediction information, the text features and the command features respectively corresponding to each of the perspectives.

7. A method for training a generative model for generating driving scene videos, characterized in that: include: Obtain training sample data; The training sample data includes a plurality of first training samples and a plurality of second training samples; Performing a first-stage training on a scene environment prediction model to be trained based on each of the first training samples to obtain a scene environment prediction model; Determine the generation model to be trained based on the scene environment prediction model obtained from the first stage of training and the generation model of the video to be trained; The generation model to be trained is trained in the second stage based on each of the second training samples to obtain a generation model for generating a driving scene video; wherein the training scene environment prediction information predicted by the scene environment prediction model is used as condition information of the generation model of the video to be trained.

8. The method according to claim 7, characterized in that Each of the first training samples includes training multimodal scene data and true value information of training scene environment for a future preset number of frames; The training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, training scene environment description text data, and a training vehicle control signal; Each frame of training scene image data includes a training scene image of at least one perspective; The first stage of training the scene environment prediction model to be trained based on each of the first training samples to obtain the scene environment prediction model includes: The following operations are performed by the scene environment prediction model to be trained until the first stage training of the scene environment prediction model to be trained is completed to obtain the scene environment prediction model: For any of the first training samples, determining first training scene two-dimensional features corresponding to each perspective respectively based on the training multimodal scene data in the first training sample; Using the to-be-trained scene environment prediction model to predict the first training scene two-dimensional features corresponding to each of the perspectives, to obtain first training scene environment prediction information for a preset number of frames in the future; Determine a first loss corresponding to the first training sample based on the first training scene environment prediction information and the training scene environment true value information in the first training sample; Determine a first model loss based on the first losses respectively corresponding to the first training samples; Based on the first model loss, the network parameters of the scene environment prediction model to be trained are updated to obtain an updated scene environment prediction model to be trained.

9. The method according to claim 7 or 8, characterized in that: Each of the second training samples includes training multimodal scene data and a training driving scene true value video corresponding to the training multimodal scene data; the training multimodal scene data includes at least one frame of training scene image data, and at least one of a training scene bird's-eye view, a training scene environment description text data, and a training vehicle control signal; Each frame of training scene image data includes a training scene image of at least one perspective; The step of performing a second phase of training on the to-be-trained generation model based on each of the second training samples to obtain a generation model for generating a driving scene video includes: The following operations are performed by the generation model to be trained until the second stage of training of the generation model to be trained is completed, and a generation model for generating a driving scene video is obtained: For any second training sample, based on the training multimodal scene data in the second training sample, generate the second training scene two-dimensional features corresponding to each of the perspectives and the second training scene environment prediction information of the future preset number of frames; Generate a training driving scene prediction video based on the second training scene two-dimensional features corresponding to each of the perspectives and the second training scene environment prediction information; Determining a second loss corresponding to the second training sample based on the training driving scene prediction video and the training driving scene true value video in the second training sample; Determine a second model loss based on the second losses respectively corresponding to each of the second training samples; Based on the second model loss, the network parameters of the to-be-trained generation model are updated to obtain an updated to-be-trained generation model.

10. The method according to claim 9, characterized in that The generating, based on the training multimodal scene data in the second training sample, the second training scene two-dimensional features corresponding to each of the viewing angles and the second training scene environment prediction information of a preset number of frames in the future includes: Encoding the training multimodal scene data in the second training sample using a multimodal encoder to obtain training multimodal features; Extracting training visual features corresponding to the training scene image data from the training multimodal features; Converting the training visual features into a three-dimensional grid coordinate system to obtain training three-dimensional spatial features corresponding to the training visual features; Based on the training three-dimensional space features, determining the second training scene two-dimensional features corresponding to each of the perspectives; Based on the training three-dimensional space features, predicting the second training scene environment prediction information; or, In the case where the second training sample includes a training scene bird's-eye view, a training bird's-eye view feature corresponding to the training scene bird's-eye view is extracted from the training multimodal features, and based on the training bird's-eye view feature, the second training scene environment prediction information is predicted.

11. A processing device for generating a driving scene video, characterized in that: include: A multimodal data acquisition module, used to acquire multimodal scene data; the multimodal scene data includes at least one frame of scene image data of the scene where the vehicle is located, and the multimodal scene data also includes at least one of a bird's-eye view of the scene, text data describing the scene environment, and a vehicle control signal; each frame of scene image data includes at least one scene image corresponding to each viewing angle; A first processing module, configured to generate, based on the multimodal scene data, scene two-dimensional features corresponding to each of the viewing angles and scene environment prediction information of a preset number of frames in the future; The second processing module is used to generate a driving scene video using a video generation model based on the two-dimensional features of the scene corresponding to each of the perspectives and the scene environment prediction information.

12. A training device for generating a generative model for generating a driving scene video, characterized in that: include: A sample acquisition module is used to obtain training sample data; The training sample data includes a plurality of first training samples and a plurality of second training samples; A first training module, used for performing a first stage of training on a scene environment prediction model to be trained based on each of the first training samples to obtain a scene environment prediction model; The second training module is used to determine the generation model to be trained based on the scene environment prediction model obtained in the first stage of training and the generation model of the video to be trained; the second stage of training is performed on the generation model to be trained based on each of the second training samples to obtain a generation model for generating a driving scene video; wherein the training scene environment prediction information predicted by the scene environment prediction model is used as conditional information of the generation model of the video to be trained.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to execute a computer program stored in the memory, and when the computer program is executed, implements the processing method for generating a driving scene video described in any one of claims 1 to 6, or implements the training method for generating a generation model for generating a driving scene video described in any one of claims 7 to 10.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the processing method for generating a driving scene video described in any one of claims 1 to 6 is implemented, or the training method for generating a generation model for generating a driving scene video described in any one of claims 7 to 10 is implemented.

Images