Video generation method and device, electronic equipment and storage medium

Abstract

The invention relates to a video generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a multi-mode video generation control condition; wherein the multi-mode video generation control condition comprises a geometric control condition and a text condition; performing compression processing on the random noise by using a space compression encoder in the video generation model to generate potential space features; a multi-modal condition encoder in the video generation model is used for encoding the geometric control conditions and the text conditions to obtain multi-modal control parameters; and inputting the potential spatial features and the multi-modal control parameters into a video generation module in a video generation model for processing to obtain a target video. According to the embodiment of the invention, the control precision of the video generation process can be improved, and the quality and scene diversity of the generated video are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving and video generation, and particularly relates to a video generation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Video generation technology is an important research direction in the current field of artificial intelligence and computer vision, aiming to automatically generate dynamic image content that meets specific semantics or scenarios through algorithms. With the development of deep learning, video generation methods have gradually developed from simple image sequence splicing to complex modeling processes that integrate multi-modal information, and are widely used in virtual reality, autonomous driving simulation, film production and other fields.

[0003] In the field of autonomous driving, autonomous driving systems need to operate safely and stably in complex real-world scenarios, and the generated autonomous driving videos need to contain rich visual information to provide decision-making basis for the stable operation of autonomous driving systems under various road conditions, traffic rules, weather conditions and unexpected scenarios. However, related technologies lack sufficient editing control granularity for generated content, limiting scene diversity, and the generated videos contain limited information and relatively single content. SUMMARY

[0004] The present application provides a video generation method, device, electronic equipment and storage medium, which can improve the control accuracy of the video generation process and improve the quality and scene diversity of the generated video.

[0005] The present application provides a video generation method, device, electronic equipment and storage medium, which can improve the control accuracy of the video generation process and improve the quality and scene diversity of the generated video. The method comprises: obtaining multi-modal video generation control conditions; wherein the multi-modal video generation control conditions include geometric control conditions and text conditions; the geometric control conditions are used to describe the spatial position, motion trajectory and geometric relationship of objects in the video; the text conditions are used to describe the global semantic features of the video; using a spatial compression encoder in the video generation model to compress the random noise and generate latent space features; using a multi-modal condition encoder in the video generation model to encode the geometric control conditions and the text conditions respectively to obtain multi-modal control parameters; inputting the latent space features and the multi-modal control parameters into a video generation module in the video generation model for processing to obtain a target video.

[0006] According to the above technical means, firstly, the geometric control condition and the text condition are decoupled and independently modeled and coded, thereby realizing high controllability of the video content; secondly, the spatial compression encoder is introduced to compress the random noise and generate the latent space feature, thereby effectively reducing the calculation resource consumption and improving the processing efficiency; finally, the latent space feature and the coded multi-modal control parameter are input into the video generation module, so that the video generation module calculates in the latent space based on the multi-modal control parameter to generate the target video, thereby improving the quality and controllability of the generated video.

[0007] In some embodiments, the video generation module includes a first processing branch, a second processing branch, and a decoder; the latent space feature and the multi-modal control parameter are input into the video generation module in the video generation model for processing to obtain the target video, including: dividing the multi-modal control parameter into a first control parameter group and a second control parameter group based on the video generation requirement; processing the first control parameter group and the second control parameter group by using the first processing branch to obtain first video parameters; processing the first control parameter group and the latent space feature by using the second processing branch, and fusing the first video parameters into the second processing branch to obtain second video parameters; decoding the second video parameters by using the decoder to obtain the target video.

[0008] According to the above technical means, by dividing the multi-modal control parameter into two groups and inputting them into two processing branches for processing, the differential processing of different types of control information can be realized, and the adaptability of the model to complex scenes can be improved; at the same time, the output results of the first processing branch are fused into the second processing branch, which helps to enhance the information transmission and collaborative optimization in the video generation process, and further improves the quality and continuity of the generated video.

[0009] In some embodiments, the first processing branch includes M first processing units, the first processing unit includes a first space-time diffusion module and a second space-time diffusion module; the second processing branch includes N second processing units, the second processing unit includes a multi-view space-time diffusion module and a third space-time diffusion module; M and N are positive integers, and M is less than N; the first video parameters include the output results of the first space-time diffusion module and the second space-time diffusion module in the M first processing units; fusing the first video parameters into the second processing branch, including: fusing the output results of the first space-time diffusion module of the i-th first processing unit in the first processing branch and the output results of the multi-view space-time diffusion module of the i-th second processing unit in the second processing branch, and inputting them into the third space-time diffusion module of the i-th second processing unit. The output result of the second space-time diffusion module of the i-th first processing unit is added to the output end of the third space-time diffusion module of the i-th second processing unit.

[0010] According to the above technical means, by arranging multiple processing units containing first and second space-time diffusion modules in the first processing branch, the dynamic changes of the video content can be gradually refined in the time dimension; the second processing branch enhances the modeling capability of object motion and spatial layout under multiple perspectives through the combination of the multi-perspective space-time diffusion module and the third space-time diffusion module; in addition, by fusing the output result of the first processing branch with the intermediate result of the second processing branch, cross-branch information sharing and complementation can be realized, thereby improving the overall consistency and detail expressiveness of the video generation.

[0011] In some embodiments, the geometric control conditions include camera pose parameters, ego vehicle trajectory information, three-dimensional bounding boxes, and map information; the multi-modal condition encoder includes a camera pose encoder, an ego vehicle trajectory encoder, a three-dimensional bounding box encoder, a map encoder, and a text encoder; the multi-modal condition encoder in the video generation model is used to encode and process the geometric control conditions and the text condition respectively to obtain multi-modal control parameters, including: The camera pose parameters are encoded and processed by the camera pose encoder to obtain camera pose encoding; The ego vehicle trajectory information is encoded and processed by the ego vehicle trajectory encoder to obtain ego vehicle trajectory encoding; The three-dimensional bounding boxes are encoded and processed by the three-dimensional bounding box encoder to obtain three-dimensional bounding box encoding; The map information is encoded and processed by the map encoder to obtain road global features; The text condition is encoded and processed by the text encoder to obtain text encoding.

[0012] According to the above technical means, by designing special encoders for different types of geometric control conditions, such as camera pose encoders and ego vehicle trajectory encoders, the corresponding feature information can be more accurately extracted and represented, providing high-quality control signals for subsequent video generation.

[0013] In some embodiments, the camera pose encoder includes a Fourier transform layer and a multi-layer perception layer; the camera pose encoder is used to encode and process the camera pose parameters to obtain camera pose encoding, including: The camera pose parameters are Fourier transformed by the Fourier transform layer to obtain first pose parameters; The first pose parameters are encoded by the multi-layer perception layer to obtain camera pose encoding.

[0014] According to the above technical means, by adopting the combination of the Fourier transform layer and the multi-layer perception layer to encode the camera pose parameters, richer geometric features can be extracted, and these geometric features can be converted into compact vector representations, which is beneficial to improve the understanding ability of the video generation model for spatial transformation, thereby improving the geometric consistency and spatio-temporal stability in the video generation process.

[0015] In some embodiments, the ego vehicle trajectory encoder includes a Fourier transform layer, a self-attention layer, a multi-layer perception layer, and a pooling layer; the ego vehicle trajectory information is encoded by using the ego vehicle trajectory encoder to obtain an ego vehicle trajectory code, including: The Fourier transform layer is used to perform Fourier transform on the ego vehicle trajectory information to obtain first ego vehicle trajectory parameters; The self-attention layer is used to perform rotation position encoding on the first ego vehicle trajectory parameters to obtain second ego vehicle trajectory parameters; The multi-layer perception layer is used to encode the second ego vehicle trajectory parameters to obtain initial ego vehicle trajectory codes; The pooling layer is used to reduce the dimension of the initial ego vehicle trajectory codes to obtain the ego vehicle trajectory codes.

[0016] According to the above technical means, by combining Fourier transform and self-attention mechanism, the long-term dependence of the ego vehicle trajectory in time and space can be effectively captured; by using the pooling operation to reduce the dimension of high-dimensional features, the redundancy information is reduced, and the encoding efficiency is improved; the final obtained ego vehicle trajectory code has strong generalization ability and robustness, and can better guide the rationality and continuity of vehicle behavior in the video generation process.

[0017] In some embodiments, the three-dimensional bounding box includes a category parameter and a position parameter; the three-dimensional bounding box encoder includes a Fourier transform layer, a first multi-layer perception layer, a text encoder, a pooling layer, and a second multi-layer perception layer; the three-dimensional bounding box is encoded by using the three-dimensional bounding box encoder to obtain a three-dimensional bounding box code, including: The Fourier transform layer is used to perform Fourier transform on the position parameter to obtain first position parameters; The first multi-layer perception layer is used to encode the first position parameters to obtain position embedding parameters; The text encoder is used to extract the category parameter to obtain a word vector of the category parameter; The pooling layer is used to reduce the dimension of the word vector to obtain semantic embedding parameters; The second multi-layer perception layer is used to compress the position embedding parameters and the semantic embedding parameters to obtain the three-dimensional bounding box code.

[0018] According to the above technical means, by performing Fourier transform and text encoding processing on the position parameter and the category parameter respectively, and fusing through the multi-layer perception, the spatial attribute and semantic information of the object can be reserved at the same time, a more comprehensive and accurate three-dimensional bounding box representation is formed, the addition of the pooling operation helps to reduce the feature dimension and improve the calculation efficiency, and finally obtained three-dimensional bounding box code can not only describe the position and shape of the object, but also reflect the category information, which provides stronger control ability for video generation.

[0019] The embodiment of the application provides a video generation device, the device comprises: The acquisition module is configured to acquire a multi-modal video generation control condition; wherein the multi-modal video generation control condition comprises a geometric control condition and a text condition; the geometric control condition is used to describe the spatial position, motion trajectory and geometric relationship of an object in a video; and the text condition is used to describe the global semantic features of the video. The compression module is configured to perform compression processing on random noise by using a spatial compression encoder in the video generation model to generate latent space features. The encoding module is configured to perform encoding processing on the geometric control condition and the text condition respectively by using a multi-modal condition encoder in the video generation model to obtain multi-modal control parameters. The generation module is configured to input the latent space features and the multi-modal control parameters into a video generation module in the video generation model for processing to obtain a target video.

[0020] The embodiment of the application provides an electronic device, comprising a processor and a memory, the memory stores a computer program which can run on the processor, and the processor implements the steps in the above method when executing the computer program.

[0021] The embodiment of the application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0022] The embodiment of the application provides a computer program product, comprising a computer program or instructions, and the computer program or instructions are executed by a processor to implement the steps in the above method.

[0023] The application provides a vehicle, which comprises the above electronic device.

[0024] The beneficial effects of the application are: (1) The embodiment of the application decouples the geometric control condition and the text condition, and independently models and encodes them respectively, so as to realize the control granularity of the video content and improve the quality and controllability of the generated video.

[0025] (2) The embodiment of the present application can realize differentiated processing of different types of control information by dividing the multi-modal control parameters into two groups and inputting them into two processing branches for processing, thereby improving the adaptability of the model to complex scenes. Meanwhile, the output result of the first processing branch is fused into the second processing branch, which helps to enhance information transmission and collaborative optimization in the video generation process, further improving the quality and coherence of the generated video.

[0026] (3) The embodiment of the present application can gradually refine the dynamic changes of video content in the time dimension by setting multiple processing units containing the first and second space-time diffusion modules in the first processing branch. The second processing branch enhances the modeling capability of object motion and spatial layout under multi-view through the combination of the multi-view space-time diffusion module and the third space-time diffusion module. In addition, by fusing the output result of the first processing branch with the intermediate result of the second processing branch, cross-branch information sharing and complementation can be realized, thereby improving the overall consistency and detail expressiveness of video generation.

[0027] (4) The embodiment of the present application designs special encoders for different types of geometric control conditions, such as camera pose encoders and ego trajectory encoders, which can more accurately extract and represent the corresponding feature information, providing high-quality control signals for subsequent video generation.

[0028] (5) The embodiment of the present application introduces a spatial compression encoder to compress and process random noise, generating latent space features, effectively reducing the consumption of computing resources and improving processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 An implementation flow diagram of a video generation method provided by the embodiment of the present application; Figure 2 A structure diagram of a video generation model provided by the embodiment of the present application; Figure 3 A schematic diagram of 3D bounding boxes under different perspectives provided by the embodiment of the present application; Figure 4 An aerial view provided by the embodiment of the present application; Figure 5 A structure diagram of a map encoder provided by the embodiment of the present application; Figure 6 A structure diagram of an ego trajectory encoder provided by the embodiment of the present application; Figure 7 A structure diagram of a three-dimensional bounding box encoder provided by the embodiment of the present application; Figure 8 A structure diagram of a video generation module provided by the embodiment of the present application; Figure 9 A multi-view spatio-temporal diffusion module and a structural schematic diagram of the spatio-temporal diffusion module provided for an embodiment of the present application; Figure 10 A structural schematic diagram of a video generation device provided for an embodiment of the present application; Figure 11 A hardware entity schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0030] The advantages and effects of the present application can be easily understood by those skilled in the art from the content disclosed in the specification. The present application can also be implemented or applied by means of other different specific embodiments, and various modifications or changes can be made to the details in the specification based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for the purpose of illustrating the present application, but not for limiting the protection scope of the present application.

[0031] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the diagrams, but not drawn according to the number, shape and size of the components in actual implementation. The actual implementation of each component may be randomly changed in shape, number and proportion, and the layout pattern of the components may also be more complex.

[0032] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0033] In the following description, the terms "first", "second", "third" are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first", "second", "third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0035] In related technologies, the automatic driving video generation method usually relies on a single visual feature or simple convolution operation, which is difficult to achieve precise control of object position, road topology and global semantics, and has problems such as cross-view jitter and non-key object blur.

[0036] The application provides a video generation method, which first decouples the geometric control condition and the text condition, and independently models and encodes them respectively, so as to realize the controllability of the video content; secondly, the spatial compression encoder is introduced to compress the random noise and generate latent space features, so as to reduce the consumption of computing resources and improve the processing efficiency of video generation; finally, the multi-modal control parameters and the latent space features are input into the video generation module to generate a target video, so as to realize controllable and high-resolution video generation and improve the information diversity and video quality of the automatic driving video.

[0037] In the following, the technical solutions in the embodiments of the application will be described clearly and completely with reference to the drawings in the embodiments of the application.

[0038] Figure 1 An implementation flowchart of the video generation method provided by the embodiments of the application is shown in FIG. 1, which includes steps 101 to 103. Figure 1 As shown in FIG. 1, the method includes steps 101 to 103, wherein: Step 101: Obtain multi-modal video generation control conditions.

[0039] The video generation method provided by the embodiments of the application can be applied to a video generation system, which can be a computer program or instructions running on an electronic device.

[0040] The multi-modal video generation control condition refers to an input control condition for controlling multi-dimensional information in the video generation process, including a geometric control condition and a text condition.

[0041] The geometric control condition is used to describe the spatial position, motion trajectory and geometric relationship of objects in the video. For example, the geometric control condition can include camera pose parameters, ego vehicle trajectory information, three-dimensional bounding boxes and map information. For example, the camera pose parameters are used to represent the attitude of the camera under different viewing angles, including rotation information and translation information, so as to ensure that the generated video can present consistent visual effects from multiple viewing angles. The ego vehicle trajectory information is used to describe the path of the vehicle moving in the video. The three-dimensional bounding box is used to describe the position, size and direction of each object in the video in three-dimensional space, so as to accurately restore the appearance of the object. The map information provides the road structure information of the entire scene, such as lane lines, pedestrian crossings and parking areas, which helps to generate video content that conforms to traffic rules.

[0042] Text conditions are used to describe the global semantic features of the video, such as weather, environmental style, traffic rules, etc. Illustratively, the text condition can be input in the form of natural language, and converted into an embedding vector understandable by the video generation model through a text encoder. The text condition not only affects the background atmosphere of the scene, but also guides the video generation model to generate specific types of objects and behavior patterns. For example, a congested road section may mean more vehicles and pedestrians appear in the picture, while rain may cause changes in visual effects such as wet road surface and light reflection.

[0043] It can be understood that the geometric control condition in the multi-modal video generation control condition provides precise spatial layout information, and the text condition supplements the overall atmosphere and environmental details of the vehicle driving scene. When generating a video, the geometric control condition and the text condition can be independently set according to specific needs, improving the control granularity of video generation.

[0044] The video generation system can extract multi-modal video generation control conditions based on user input, or extract multi-modal video generation control conditions based on collected driving information of the autonomous vehicle. The multi-modal video generation control condition determines the specific content of the finally generated video. For example, in an autonomous driving scenario, a user may want to generate a video containing specific vehicle positions, road structures, and weather conditions. The video generation system can obtain geometric control conditions such as camera pose, ego trajectory, 3D bounding box, and map information corresponding to the generated autonomous driving video based on user input, and text conditions such as rainy night and urban trunk road representing the vehicle driving environment.

[0045] In some embodiments, the process of obtaining multi-modal video generation control conditions can include data acquisition, preprocessing, and format processing stages. In the data acquisition stage, the video generation system can obtain raw information from sensors, user input, or other external devices; in the preprocessing stage, the raw data is processed for cleaning, normalization, and standardization to adapt the raw data to the input requirements of the video learning model; in the formatting stage, various conditions are uniformly encoded into a format suitable for processing by the video generation model, and the subsequent processing process proceeds smoothly.

[0046] Step 102, compressing the random noise using a spatial compression encoder in the video generation model to generate latent space features.

[0047] In the embodiments of the present application, the spatial compression encoder in the video generation model can adopt a three-dimensional variational autoencoder (3DVAE) for compression processing of the input random noise. The 3DVAE is an encoder-decoder structure based on a three-dimensional convolutional neural network, which can model video data in both time and space dimensions. Compared with the traditional 2DVAE which only processes single-frame images, the 3DVAE can capture the spatio-temporal correlation in the video sequence, thereby generating more coherent and natural video content.

[0048] Illustratively, the 3DVAE first maps the input high-dimensional random noise to a low-dimensional latent space through the encoder part. The low-dimensional latent space retains the main features of the original data and removes redundant information, making the subsequent generation process more efficient. The embodiments of the present application complete the main calculation process in the compressed low-dimensional latent space, and finally restore the latent space features in the video generation model to a video frame sequence using the decoder part, and ensure that the generation of each frame only depends on the previous frame information through the causal convolution design, avoiding the interference of future information. The causal convolution design can effectively prevent jitter artifacts and motion discontinuity problems in the generation process.

[0049] Further, in order to improve the video generation effect, the spatial compression encoder in the embodiments of the present application can adopt a specific compression ratio, for example, time T x space W x H = 4 x 8 x 8, to balance the expression ability of the video generation model and the consumption of computing resources. In addition, the spatial compression encoder in the embodiments of the present application also has the ability of geometric anchoring, that is, it can constrain the trajectory of the latent space according to the geometric control condition in the multi-modal video generation control condition during the video generation process, to ensure that the generated target video conforms to the laws of the physical world. For example, when generating a vehicle turning scene, the decoder of the spatial compression encoder will automatically adjust the motion trajectory in the latent space, so that the motion trajectory in the latent space conforms to the principle of centrifugal force and inertia, thereby generating more realistic pictures.

[0050] Step 103, encoding the geometric control condition and the text condition respectively by using the multi-modal condition encoder in the video generation model to obtain multi-modal control parameters.

[0051] The multi-modal condition encoder in the embodiments of the present application is used to convert the geometric control condition and the text condition in the multi-modal video generation control condition into embedded vectors that can be processed by the model. It can be understood that different types of conditions have different data forms and semantic meanings, and the present application encodes the geometric control condition and the text condition respectively by using special encoding strategies through the multi-modal condition encoder to ensure the reliability of the encoding results.

[0052] As an example, for the geometric control conditions in the multi-modal video generation control conditions, such as camera pose parameters, ego vehicle trajectory information, 3D bounding boxes, and map information, the multi-modal condition encoder can adopt multiple encoding methods. For example, both the camera pose parameters and the ego vehicle trajectory information are encoded using Fourier embedding and multi-layer perceptron (MLP) to capture the high-frequency variation characteristics of the camera pose and the ego vehicle trajectory; the 3D bounding box adopts a dual-stream fusion mechanism to process the class label and the geometric information respectively, and compresses the information of the 3D bounding box into a unified representation through MLP; and the map information, such as the bird's eye view map (BevMap), extracts the road global feature through down-sampling convolution, and maps the road global feature into the latent space.

[0053] For the text conditions in the multi-modal video generation control conditions, the multi-modal condition encoder can adopt Text-to-Text Transfer Transformer (T5) as the text encoder. T5 is a powerful natural language processing model that can convert input text into high-quality word vector embedding. By using Text-to-Text Transfer Transformer (T5) as the text encoder, the text conditions are effectively integrated into the video generation process, thereby achieving precise control over the overall atmosphere and details of the scene in the generated target video.

[0054] It can be understood that the multi-modal condition encoder in the embodiments of the present application can adopt a modular design to encode and process different types of conditions, thereby uniformly converting multiple types of input data into a format recognizable by the model while preserving the key information of the input data, and ensuring the controllability and consistency of the generated target video.

[0055] Step 104, inputting the latent space features and the multi-modal control parameters into a video generation module in the video generation model for processing to obtain a target video.

[0056] Referring to Figure 2 , a video generation model provided by an embodiment of the present application is shown, such as Figure 2As shown, the video generation model in the embodiments of the present application includes a spatial compression encoder, a multi-modal conditional encoder, and a video generation module. The multi-modal conditional encoder is used to perform encoding operations on different types of conditions respectively, and convert these heterogeneous data into feature representations suitable for processing by the video generation module. In order to increase the diversity of the generated video, random noise is input to the spatial compression encoder, such as a 3DVAE, and the random noise is compressed by the 3DAVE to generate latent space features. In order to further reduce resource consumption in the real-time generation process, the latent space noise is divided into smaller noise blocks, and after adding position encoding, it is input to the video generation module. The video generation module generates a target video based on the multi-modal control parameters output by the multi-modal conditional encoder and the latent space features generated by the spatial compression encoder, and outputs the target video.

[0057] The video generation method provided by the embodiments of the present application decouples the geometric control conditions and the text conditions of the generated video, encodes the geometric control conditions and the text conditions by the multi-modal conditional encoder in the video generation model, introduces random noise, compresses the high-dimensional random noise into low-dimensional latent space features by the spatial compression encoder, and inputs the latent space features and the encoded multi-modal control parameters to the video generation module. The video generation module calculates based on the multi-modal control parameters in the latent space to generate a target video, thereby improving the quality and controllability of the generated video.

[0058] In some embodiments, the geometric control conditions include camera pose parameters, ego vehicle trajectory information, three-dimensional bounding boxes, and map information; and the multi-modal conditional encoder includes a camera pose encoder, an ego vehicle trajectory encoder, a three-dimensional bounding box encoder, a map encoder, and a text encoder.

[0059] The camera pose parameters are used to describe the position and attitude of the camera in space, and usually include a rotation matrix and a translation vector, which are used to determine the shooting angle. Exemplarily, in the field of autonomous driving, the camera angle usually includes six angles: left front, front, right front, right rear, rear, and left rear. Therefore, the camera pose parameters can be represented as: (1) wherein, is the camera pose (6x3x7), is the camera intrinsic matrix (6x3x3), is the rotation matrix (6x3x1), is the translation vector (6x3x1).

[0060] The ego vehicle trajectory information represents the motion path of the vehicle in the scene, including the position and direction of each frame. Exemplarily, the ego vehicle trajectory information can be represented by a rotation matrix and a translation vector The configuration is specifically shown in formula (2): (2) The three-dimensional bounding box is used to describe the spatial position and size of the object, so as to accurately present the position, size, pose and category of the object in three-dimensional space in the generated video, so as to conform to the actual situation of the object in the physical world. Exemplarily, the three-dimensional bounding box in the embodiment of the present application can contain the category information and the bounding box of each object, which can be specifically represented as (category, bounding box), wherein the bounding box is defined by 8 three-dimensional corner coordinates, and can be specifically represented as:

[0061] wherein, The bounding box is composed of 8 three-dimensional coordinate points; The category, in the embodiment of the present application, can use 10 categories of car, truck, construction vehicle, bus, trailer, barrier, bicycle, motorcycle, pedestrian and traffic cone. Referring to Figure 3 , a 3D bounding box under different camera perspectives is shown, as shown in Figure 3 , the 3D bounding box identifies the position information of the object in the real physical world. In addition, Figure 3 Each bounding box in the embodiment corresponds to a category.

[0062] The map information describes the structure of the road, such as lane line, drivable area, etc. Exemplarily, the map information in the embodiment of the present application can adopt a bird's eye view (BevMap). Referring to Figure 4 , a visual BevMap is shown. The BevMap can include 8 different road structure categories: {drivable area, crosswalk, sidewalk, stop line, parking area, road divider, lane divider, barrier / road closure}, and the BevMap is encoded using one-hot code, which can be specifically represented as: (6) wherein, 8, indicating 8 road structures; is the pixel size, for example, The value of each of the above can be 400.

[0063] It can be understood that the above geometric control conditions jointly determine the spatial layout and motion trajectory of the object in the video.

[0064] In addition, in the embodiments of the present application, the text condition L is used to control the global environmental attributes (weather, time) and the implied details (such as vehicle color, background building style).

[0065] Based on the above various conditions, for an automatic driving scene, the multi-modal video generation control condition in the embodiments of the present application may be expressed as: (7) wherein t represents the current frame number, and T is the total frame number of the target video to be generated.

[0066] The multi-modal condition encoder in the embodiments of the present application includes multiple encoders, each of which is responsible for processing a specific type of input and converting the specific type of input into a unified feature representation for subsequent fusion use. For example, the camera pose encoder converts the camera pose parameters into an embedding vector suitable for model understanding, and the text encoder converts the environmental attributes described in natural language into a semantic vector.

[0067] By designing a dedicated encoder for each type of control condition, the embodiments of the present application can independently model different types of control conditions, thereby avoiding mutual interference between these control conditions and facilitating the improvement of encoding efficiency and video generation quality.

[0068] Step 103 encodes and processes the geometric control condition and the text condition respectively by using the multi-modal condition encoder in the video generation model, to obtain multi-modal control parameters, including steps S11 to S15, wherein: Step S11, the camera pose parameters are encoded and processed by using the camera pose encoder to obtain camera pose encoding.

[0069] The camera pose encoder in the embodiments of the present application is a deep learning model, which usually adopts Fourier embedding and multi-layer perceptron (MLP) structure, and is used to convert the camera pose parameters into a high-dimensional feature vector to preserve the spatial information of the camera parameters. Through camera pose encoding, the video generation model can more accurately understand the spatial relationship under different perspectives, thereby maintaining the perspective consistency when generating the video.

[0070] Step S12, the ego vehicle trajectory information is encoded and processed by using the ego vehicle trajectory encoder to obtain ego vehicle trajectory encoding.

[0071] The self-vehicle trajectory encoder in the embodiments of the present application can also adopt Fourier embedding and MLP structure to extract the motion features of the vehicle in the scene. In addition, the self-vehicle trajectory information in the embodiments of the present application contains time sequence information, and the self-vehicle trajectory encoder can also introduce a self-attention mechanism to capture the dependency between self-vehicle trajectories at different times. Through self-vehicle trajectory encoding, the video generation model can effectively model the dynamic behavior of the vehicle, ensuring that the generated video is coherent and reasonable in time sequence.

[0072] In step S13, the three-dimensional bounding box is encoded by using a three-dimensional bounding box encoder to obtain three-dimensional bounding box encoding.

[0073] The three-dimensional bounding box encoder in the embodiments of the present application adopts a dual-stream structure to process the category information and the geometric information of the object respectively. The category information can be obtained by a CLIP text encoder. The geometric information, such as the bounding box, can be encoded by Fourier embedding and MLP. The information of the two parts is finally fused into a unified feature vector, i.e., the three-dimensional bounding box encoding in the embodiments of the present application. Through three-dimensional bounding box encoding, the video generation model can accurately control the position, size and category of the object in the video to meet the requirements of the actual driving scene.

[0074] In step S14, the map information is encoded by using a map encoder to obtain road global features.

[0075] The map encoder in the embodiments of the present application can adopt a convolutional neural network (CNN) to downsample and extract features of the map information, such as BevMap, to obtain the global topological information of the scene. The road global features generated by the map encoder can help the video generation model understand the road structure, such as lane distribution, stop line, pedestrian crossing, etc., which is conducive to generating a video that meets the real road rules and improves the realism of the video.

[0076] For example, the map information, such as BevMap, in the embodiments of the present application can adopt a two-dimensional grid format, and the map encoder is used to compress the map information in space and time in the latent space. Referring to Figure 5 , a map encoder is shown, as shown in Figure 5 The map encoder in the embodiments of the present application can be 2 groups of down-sampling convolution, one of which is a spatial convolution group with a convolution kernel size of 1x8x8 for spatial compression of the map information, and the other is a temporal convolution group with a convolution kernel size of 4x1x1 for temporal compression of the map information. By compressing the map information in space and time, road global features can be extracted.

[0077] In step S15, the text condition is encoded by using a text encoder to obtain text encoding.

[0078] The text encoder in the embodiments of the present application can adopt a Text-to-Text Transfer Transformer (T5) model to convert natural language descriptions (such as rainy night, congested road section) into semantic vectors, that is, text encoding in the embodiments of the present application. The text encoding output by the text encoder can guide the video generation model to generate video content conforming to the specified semantic style. Through text encoding, the video generation model can generate videos with specific environmental atmosphere and details, thereby enhancing the diversity and controllability of video generation.

[0079] In the embodiments of the present application, the geometric control condition and the text condition are decoupled and encoded separately, which can more flexibly control the video generation process. Moreover, different encoders are used to encode different types of control conditions, which realizes independent modeling of different types of control conditions, thereby avoiding mutual interference between these control conditions, which is conducive to improving the encoding efficiency and the accuracy and diversity of video generation, and meeting the data demand of complex scenes in the training process of the automatic driving system.

[0080] In some embodiments, the camera pose encoder includes a Fourier transform layer and a multi-layer perception layer. Step S11 encodes the camera pose parameters using the camera pose encoder to obtain camera pose encoding, including steps S111 to S112, wherein: Step S111, Fourier transform the camera pose parameters using the Fourier transform layer to obtain first pose parameters.

[0081] In the embodiments of the present application, the camera pose parameters can be represented as wherein, is the camera pose (6x3x7), is the camera intrinsic matrix (6x3x3), is the rotation matrix (6x3x1), is the translation vector (6x3x1). In the embodiments of the present application, the camera pose parameters can be encoded using a multi-layer perception.

[0082] Further, since contains values from a sine or cosine function, and a three-dimensional offset, in order for the video generation model to effectively interpret these high-frequency changes, the Fourier transform layer is used to perform Fourier transform processing on the camera pose parameters before the multi-layer perception MLP is used to process the camera pose parameters.

[0083] It should be noted that the Fourier transform layer is used to convert the original input data (such as time series or spatial coordinates) into a frequency domain representation. Through the Fourier transform layer, the high-frequency and low-frequency components of the input data can be extracted, which helps the model better capture the periodicity, directionality and structured information in the data. In the embodiments of the present application, the camera pose parameters include rotation information and translation information, which have complex geometric properties and may be difficult to be effectively modeled by the neural network directly. Therefore, in the embodiments of the present application, before the camera pose parameters are input into the multi-layer perceptron layer, the Fourier transform layer is applied to map the original camera pose parameters to the frequency domain space, thereby enhancing the learnability of the camera pose parameters. For example, for a 6-view camera pose matrix, the Fourier transform layer can decompose the camera pose parameters into multiple frequency domain components, which facilitates the subsequent multi-layer perceptron layer to extract key spatial transformation features in the first pose parameters decomposed by the Fourier transform layer.

[0084] By using the above multi-view feature fusion method based on the transformation matrix, the sensitivity of the deep learning model to complex geometric transformations can be improved, and the spatial relationship under different views can be more accurately expressed, thereby improving the geometric consistency of the generated video.

[0085] In step S112, the first pose parameter is encoded by using the multi-layer perceptron layer to obtain camera pose encoding.

[0086] The multi-layer perceptron layer is a neural network structure composed of multiple fully connected layers, which is used for non-linear transformation and feature extraction of input data. In the embodiments of the present application, the first pose parameter after the Fourier transform layer is input into the multi-layer perceptron layer to further extract higher-level abstract features and encode the first pose parameter into a compact vector form, i.e., camera pose encoding. Exemplarily, the specific encoding process of the camera pose parameter in the embodiments of the present application is as follows: (8) wherein, is the camera pose encoding, is the camera pose parameter, represents the first pose parameter, represents the encoding processing of the first pose parameter by using the multi-layer perceptron layer.

[0087] After the above processing, the camera pose encoding in the embodiment of the application can retain the key information of the original camera pose, reduce the dimension redundancy, and facilitate improving the processing efficiency of the video generation model. For example, the first pose parameter processed by the Fourier transform layer can contain multiple frequency components, and the multi-layer perception layer can gradually extract and integrate the frequency components through a nonlinear activation function (such as ReLU, GELU, etc.), and finally output a low-dimensional but semantically rich camera pose encoding, thereby realizing effective compression and representation of complex geometric features in the camera pose parameter, and reducing the computational overhead of the video generation model, and improving the video generation efficiency and quality.

[0088] In the embodiment of the application, by adopting the combination of the Fourier transform layer and the multi-layer perception layer to encode and process the camera pose parameter, more rich geometric features can be extracted, and these geometric features can be converted into compact vector representation, which is conducive to improving the understanding ability of the video generation model to spatial transformation, thereby improving the geometric consistency and spatio-temporal stability in the video generation process.

[0089] In some embodiments, referring to Figure 6 , a self-vehicle trajectory encoder provided by the embodiment of the application is shown, as shown in Figure 6 , the self-vehicle trajectory encoder includes a Fourier transform layer, a self-attention layer, a multi-layer perception layer, and a pooling layer. Further, a normalization layer can be added at the input end of the self-attention layer and the multi-layer perception layer, respectively, for normalizing the input data. A fully connected layer can also be added at the output end of the Fourier transform layer, for globally combining the output results of the Fourier transform layer through the fully connected manner.

[0090] In some embodiments, step S12: encoding and processing the self-vehicle trajectory information by using the self-vehicle trajectory encoder to obtain the self-vehicle trajectory encoding, includes steps S121 to S124, wherein: Step S121: performing Fourier transform on the self-vehicle trajectory information by using the Fourier transform layer to obtain first self-vehicle trajectory parameters.

[0091] It can be understood that the Fourier transform layer is a mathematical method for converting time domain signals into frequency domain representation, which can extract the frequency components in the signals, thereby better capturing the periodic or high-frequency change characteristics in the self-vehicle trajectory information. In the embodiment of the application, the Fourier transform layer is used to preliminarily process the self-vehicle trajectory information, and map the self-vehicle trajectory information to the frequency domain space, so that the trajectory features can be more efficiently extracted in the subsequent processing process.

[0092] In the embodiment of the present application, the Fourier transform layer can effectively remove noise in the trajectory information of the ego vehicle, and the Fourier transform layer can also enhance the learnability of key motion patterns, so that the video generation model can better understand the dynamic behavior of the vehicle, such as turning, accelerating, decelerating, etc., which is beneficial to improve the robustness and generalization ability of trajectory coding.

[0093] In step S122, a rotation position coding is performed on the first ego vehicle trajectory parameter by using the self-attention layer to obtain a second ego vehicle trajectory parameter.

[0094] It should be noted that the self-attention layer is a neural network structure that can pay attention to the dependency between different positions when processing sequence data, thereby capturing long-distance context information.

[0095] In the embodiment of the present application, the self-attention layer is used to perform rotation position coding on the first ego vehicle trajectory parameter processed by the Fourier transform layer, so as to identify the potential relationship between the time points in the first ego vehicle trajectory parameter. The rotation position coding is an improved position coding method, which can better preserve relative position information and adapt to nonlinear changes in the first ego vehicle trajectory parameter compared with traditional sine / cosine position coding. The combination of rotation position coding and self-attention layer in the embodiment of the present application can more accurately capture the motion trend and direction change of the vehicle between different frames, avoid misjudgment caused by local noise or abnormal points, thereby improving the accuracy of trajectory prediction and the quality of overall trajectory modeling.

[0096] In step S123, the second ego vehicle trajectory parameter is encoded by using a multi-layer perception layer to obtain an initial ego vehicle trajectory encoding.

[0097] The multi-layer perception layer (MLP) is a feedforward neural network composed of multiple fully connected layers, which can perform nonlinear transformation and abstract expression on input features. In the embodiment of the present application, the multi-layer perception layer is used to further extract and compress the second ego vehicle trajectory parameter processed by the self-attention layer, and generate a compact but rich semantic high-dimensional trajectory encoding representation, which is the initial ego vehicle trajectory encoding in the embodiment of the present application. The initial ego vehicle trajectory encoding in the embodiment of the present application can fuse the time dynamic characteristics and spatial distribution characteristics of the ego vehicle trajectory, which can be used not only for ego vehicle trajectory reconstruction, but also as input features for downstream tasks such as trajectory prediction and trajectory control, thereby improving the overall performance of the video generation model.

[0098] The embodiment of the present application introduces the multi-layer perception layer in the ego vehicle trajectory encoder, which can reduce redundant information while preserving the details of the ego vehicle trajectory, improve the calculation efficiency and memory utilization, and is suitable for large-scale automatic driving video generation tasks.

[0099] In step S124, the initial ego vehicle trajectory encoding is dimensionally reduced by using a pooling layer to obtain the ego vehicle trajectory encoding.

[0100] The pooling layer is a common downsampling operation. In a convolutional neural network, the pooling layer can reduce the spatial dimension of a feature map while retaining the main feature information. In the embodiments of the present application, a full-zero fully connected layer can be used as the pooling layer to dimensionally reduce the initial ego vehicle trajectory encoding to generate the final ego vehicle trajectory encoding.

[0101] The embodiments of the present application use the pooling layer to perform downsampling operation on the initial ego vehicle trajectory encoding to compress the initial ego vehicle trajectory encoding into a low-dimensional vector representation that still retains the key trajectory information. That is, the ego vehicle trajectory encoding in the embodiments of the present application can help the video generation model focus on the key part of the trajectory and filter out unnecessary details, thereby improving the robustness and generalization ability of the video generation model. In addition, the pooling layer can also reduce the computational resources required for subsequent processing and speed up the inference speed of the video generation model.

[0102] It should be noted that the ego vehicle trajectory encoding in the embodiments of the present application can be used to fuse with other conditions (such as camera pose, 3D bounding box, etc.) to achieve more accurate video generation and control.

[0103] In the embodiments of the present application, the Fourier transform layer, the self-attention layer, the multi-layer perception layer and the pooling layer are introduced in the ego vehicle trajectory encoding process. The Fourier transform layer is used to convert the ego vehicle trajectory information into a frequency domain representation to generate the first ego vehicle trajectory parameter, to remove noise and highlight key motion patterns. The self-attention layer is used to model the dependency between each time point in the ego vehicle trajectory in combination with the rotational position encoding to generate the second ego vehicle trajectory parameter, which enhances the understanding of the global structure of the trajectory by the ego vehicle trajectory encoder. The multi-layer perception layer is used to further abstract and compress the features of the second ego vehicle trajectory parameter to generate a high-dimensional semantic representation, i.e., the initial ego vehicle trajectory encoding. Finally, the pooling layer is used to dimensionally reduce the initial ego vehicle trajectory encoding to extract the core trajectory features to generate the ego vehicle trajectory encoding for subsequent tasks. In the embodiments of the present application, the Fourier transform layer, the self-attention layer, the multi-layer perception layer and the pooling layer cooperate with each other to efficiently encode the ego vehicle trajectory information, improve the expressiveness and robustness of the ego vehicle trajectory information, and help improve the control accuracy and video quality of video generation.

[0104] In some embodiments, the three-dimensional bounding box includes a category parameter and a position parameter. Referring to Figure 7 , a three-dimensional bounding box encoder provided by an embodiment of the present application is shown. As shown in Figure 7 , the three-dimensional bounding box encoder includes a Fourier transform layer, a first multi-layer perception layer, a text encoder, a pooling layer and a second multi-layer perception layer.

[0105] The three-dimensional bounding box in the embodiments of the present application contains geometric information and semantic information of the object in the video in three-dimensional space, is composed of 8 three-dimensional corner point coordinates, and is used to describe the spatial position and size of the object, and is specifically as shown in formula (3). As shown in formula (3), the three-dimensional bounding box contains a category parameter and a position parameter. The category parameter, as shown in formula (5) , is used to identify the type of the object, such as a car, a pedestrian, a traffic cone, etc., and the category parameter is used to provide the semantic attribute of the object described by the three-dimensional bounding box, so that the video generation model can distinguish different types of objects and assign different behavior rules to the object described by the three-dimensional bounding box; the position parameter, as shown in formula (4) , is used to describe the specific coordinate information of the object in the scene, and is the basis for modeling the physical world.

[0106] The embodiments of the present application separate the category parameter and the position parameter, which is helpful for subsequent processing of geometric information and semantic information, thereby flexibly controlling the physical changes in the video generation process and improving the expression ability and flexibility of the video generation model. For example, in the video generation process, the position of the object described by a certain type of three-dimensional bounding box can be adjusted independently, or only the appearance attribute of the object described by the three-dimensional bounding box of this type can be modified, without affecting the objects described by three-dimensional bounding boxes of other types.

[0107] In the embodiments of the present application, step S13: encoding the three-dimensional bounding box using a three-dimensional bounding box encoder to obtain three-dimensional bounding box encoding, includes steps S131 to S135, wherein: Step S131, performing Fourier transform on the position parameter using a Fourier transform layer to obtain a first position parameter.

[0108] The embodiments of the present application perform Fourier transform on the position parameter using the Fourier transform, the purpose of which is to map the position parameter to a higher-dimensional frequency domain space, thereby enhancing the expression ability of the position parameter.

[0109] It should be noted that the Fourier transform layer is usually composed of sine and cosine functions, and a set of complex frequency coefficients, i.e. the first position parameter in the embodiments of the present application, can be obtained by performing discrete Fourier transform (DFT) on the position parameter.

[0110] By using the Fourier transform layer to process the position parameter, the embodiments of the present application can capture the periodic variation characteristics and high-frequency variation characteristics of the position parameter, provide more abundant position characteristics for subsequent encoding operations, effectively enhance the modeling ability of the video generation model for the motion trajectory of the object represented by the three-dimensional bounding box, and significantly improve the prediction ability and fitting effect of the video generation model, especially in the case of complex paths or nonlinear motion patterns.

[0111] Step S132, encode the first position parameter by using the first multi-layer perception layer to obtain a position embedding parameter.

[0112] The first multi-layer perception layer receives the first position parameter after Fourier transform as input, and performs nonlinear transformation on the signal input to the first multi-layer perception layer to output the position embedding parameter.

[0113] The position embedding parameter is a low-dimensional dense representation of the position information, which can retain the key features of the first position parameter, and has stronger generalization ability and expression ability. By using the first multi-layer perception layer to encode the first position parameter, the representation of the first position parameter can be further abstracted and optimized, and the spatial distribution and dynamic change rule of the object described by the three-dimensional bounding box can be effectively captured, thereby providing a basis for generating high-quality autonomous driving videos.

[0114] Step S133, extract the category parameter by using a text encoder to obtain a word vector of the category parameter.

[0115] The text encoder is a natural language processing tool used to convert text information into numerical vector representation. In the embodiment of the application, the text encoder is used to process the category parameter, that is, the text encoder converts the category name (such as car, pedestrian) of the object described by the three-dimensional bounding box into the corresponding word vector. The word vector is a vectorized representation of the category information, which can represent the similarity between different categories in the semantic space. Through the word vector, the video generation model can understand the semantic relationship between different categories and generate video content that conforms to the actual scene accordingly.

[0116] The embodiment of the application encodes the category parameter by using the text encoder, which not only improves the understanding ability of the video generation model for semantic information, but also enhances the cross-modal fusion ability of the video generation model, so that the geometric information and semantic information of the object in the video can be better combined, which is conducive to improving the video generation quality.

[0117] Step S134, perform dimension reduction processing on the word vector by using a pooling layer to obtain a semantic embedding parameter.

[0118] The pooling layer is a common deep learning operation used to reduce the dimension of data and extract main features. In the embodiment of the application, the pooling layer is used to perform dimension reduction processing on the word vector to remove redundant information and retain key semantic features.

[0119] Wherein, the semantic embedding parameter is a word vector representation after dimension reduction, the semantic embedding parameter retains the core semantic information of the category parameter, and has a lower dimension, and can be more efficiently fused with the position embedding parameter to generate a more compact and effective three-dimensional bounding box code.

[0120] The embodiment of the application reduces the consumption of computing resources by dimension reduction processing of the word vector through the pooling layer to generate the semantic embedding parameter, and enhances the robustness of the video generation model.

[0121] In step S135, the position embedding parameter and the semantic embedding parameter are compressed by using the second multi-layer perception layer to obtain the three-dimensional bounding box code.

[0122] The second multi-layer perception layer receives the position embedding parameter and the semantic embedding parameter as input, and fuses the position embedding parameter and the semantic embedding parameter through a nonlinear transformation to generate the final three-dimensional bounding box code.

[0123] It should be noted that the three-dimensional bounding box code in the embodiment of the application is a comprehensive representation of position information and semantic information, and the three-dimensional bounding box code not only contains the position information of the object in the three-dimensional space, but also fuses the category information of the object, and can more comprehensively describe the state of the object.

[0124] As shown in Figure 7 , the three-dimensional bounding box encoder in the embodiment of the application adopts a double-flow fusion mechanism to process the category information and the geometric information of each bounding box. For example, the category label The word vector is extracted by the CLIP text encoder and the semantic embedding parameter is generated by average pooling While the 8 corner coordinates of the description box space are also subjected to Fourier embedding and MLP encoding to generate the position embedding parameter The two types of embedding parameters are compressed into a unified representation by a special MLP network, which can be specifically represented as:

[0125] The embodiment of the application can unify the information of different modalities into a low-dimensional vector by performing compression operation on the position embedding parameter and the semantic embedding parameter, that is, the three-dimensional bounding box code in the embodiment of the application can effectively extract and fuse the geometric information and the semantic information, which is conducive to improving the control accuracy of the video generation process and the video generation quality.

[0126] In some embodiments, the video generation module can adopt a double-branch structure, specifically, referring to Figure 8 , a video generation module provided by the embodiment of the application is shown, as Figure 8As shown, the video generation module includes a first processing branch, a second processing branch, and a decoder. The first processing branch and the second processing branch respectively undertake different processing tasks and work cooperatively through an information fusion mechanism, and finally the decoder converts the processing results into visual video output, thereby realizing fine control of the video generation process and improving video generation efficiency and quality.

[0127] Among them, the first processing branch is a control branch, responsible for processing local detailed information from multi-modal control parameters, such as object position, color, shape, etc., for injecting additional control signals for the second processing branch to achieve fine control of video content. The second processing branch is a basic branch, responsible for generating the basic framework of the video, focusing on global structure and semantic information, such as scene layout, weather conditions, time settings, etc. There is an information interaction relationship between the two processing branches, and the second processing branch receives part of the output of the first processing branch as supplementary information to enhance the overall generation effect. The decoder is used to convert the processed feature data into specific pixel-level image frames, thereby constructing a complete video sequence.

[0128] In the embodiments of the present application, step 104: inputting the latent space feature and the multi-modal control parameter into the video generation module in the video generation model for processing to obtain the target video, comprising steps S21 to S24, wherein: Step S21, dividing the multi-modal control parameter into a first control parameter group and a second control parameter group based on the video generation requirement.

[0129] Among them, the multi-modal control parameter is the output result of the multi-modal conditional encoder in the embodiments of the present application, such as Figure 7 As shown, the multi-modal conditional encoder can include a camera pose encoder, a self-vehicle trajectory encoder, a three-dimensional bounding box encoder, a map encoder, and a text encoder, and the multi-modal control parameter can include camera pose encoding, self-vehicle trajectory encoding, three-dimensional bounding box encoding, road global feature, and text encoding.

[0130] In order to improve processing efficiency and flexibility, the multi-modal control parameter can be divided into two groups according to the actual generation requirement: the first control parameter group and the second control parameter group. Among them, the first control parameter group can include parameters that affect the overall structure and environmental attributes of the video, such as camera pose encoding, self-vehicle trajectory encoding, three-dimensional bounding box encoding, text encoding, etc.; the second control parameter group can include video local detail parameters that need to be additionally enhanced, such as road global feature.

[0131] The embodiments of the present application divide the multi-modal control parameter into the first control parameter group and the second control parameter group, which helps to achieve more fine control of the video content in the subsequent processing process and reduces the conflict or interference between different parameters.

[0132] Step S22, processing the first control parameter group and the second control parameter group by using the first processing branch to obtain the first video parameter.

[0133] In the embodiments of the present application, the first processing branch receives the first control parameter group and the second control parameter group as input, and fuses and processes the first control parameter group and the second control parameter group through the internal neural network structure to generate a preliminary video feature representation, i.e., the first video parameter. These parameters reflect key information such as the position, appearance and dynamic changes of each object in the video.

[0134] During processing, the first processing branch can combine the information of the two control parameter groups to ensure that the generated video meets the requirements of local details and is consistent with the overall environment. For example, when generating a scene containing a vehicle turning, the first processing branch can adjust the position of the vehicle according to the 3D bounding box encoding and adjust the motion path of the vehicle according to the ego trajectory encoding, thereby ensuring that the generated video has physical rationality and visual coherence.

[0135] Step S23, processing the first control parameter group and the latent space feature by using the second processing branch, and fusing the first video parameter into the second processing branch to obtain the second video parameter.

[0136] The second processing branch in the embodiments of the present application is mainly used for processing global structure and semantic information, and receives the first control parameter group and the latent space feature as input, and further fuses the first video parameter generated by the first processing branch. The latent space feature is a low-dimensional representation obtained by compressing random noise using a spatial compression encoder, which helps the video generation module to capture the spatio-temporal correlation in the video sequence, thereby generating more coherent and natural video content.

[0137] By injecting the first video parameter into the second processing branch, the embodiments of the present application can introduce more rich detail information while maintaining global consistency, so that the second processing branch can coordinate the relationship between local and global at a higher level, thereby generating more natural and realistic video content. For example, when processing a complex urban traffic scene, the second processing branch can determine the road structure, ego trajectory and surrounding environment under different perspectives according to the camera pose encoding, 3D bounding box encoding, ego trajectory encoding and text encoding, and combine the road global features in the first video parameter to generate a video that conforms to the actual driving rules.

[0138] Step S24, decoding the second video parameter by using a decoder to obtain the target video.

[0139] The decoder is the last step in the video generation process. In the embodiments of the present application, the decoder is used to convert the second video parameters into specific pixel-level image frames, thereby constructing a complete video sequence. Typically, the decoder uses deconvolution operations or other types of inverse transformation techniques to map high-dimensional features back to low-dimensional image space.

[0140] In the embodiments of the present application, the decoder is not only responsible for image reconstruction, but also has the ability of geometric anchoring, that is, introducing physical constraints in the decoding process to ensure that the generated video conforms to the motion rules of the real world. For example, when generating a scene of a vehicle turning, the decoder can adjust the motion trajectory of the output object according to the physical dynamics model to avoid generating distorted pictures that violate the centrifugal force.

[0141] It should be noted that there is a clear cooperative relationship between the first processing branch and the second processing branch in the embodiments of the present application: the first processing branch processes local details, and then passes the first video parameters obtained after processing local details to the second processing branch, and the second processing branch integrates local information while processing global structure, thereby realizing unified expression of details and the whole. In addition, the embodiments of the present application introduce a grouping processing mechanism for multi-modal control parameters, which realizes fine control of the video generation process, is conducive to improving the spatio-temporal consistency and detail accuracy of the generated video, and enhances the video generation effect.

[0142] In some embodiments, the first processing branch includes M first processing units, each first processing unit including a first spatio-temporal diffusion module and a second spatio-temporal diffusion module; the second processing branch includes N second processing units, each second processing unit including a multi-view spatio-temporal diffusion module and a third spatio-temporal diffusion module; M and N are positive integers, and M is less than N; the first video parameters include output results of the first spatio-temporal diffusion modules and the second spatio-temporal diffusion modules in the M first processing units.

[0143] As an example, as shown in Figure 8 The video generation module in the embodiments of the present application includes a first processing branch (control branch) and a second processing branch (basic branch), wherein the second processing branch is responsible for generating the basic framework of the video, and the first processing branch is responsible for injecting additional control signals to realize fine control of the video content.

[0144] The first processing branch includes M first processing units, each first processing unit including a first spatio-temporal diffusion module and a second spatio-temporal diffusion module, and in addition to the parameters in the first control parameter group, the first processing branch additionally processes BevMap information to provide more comprehensive control signals, M = N / 2.

[0145] The second processing branch includes N second processing units, each of which includes a multi-view spatio-temporal diffusion module and a third spatio-temporal diffusion module. The multi-view spatio-temporal diffusion module is used to process spatial condition control to ensure spatial consistency between different views; and the third spatio-temporal diffusion module is used to process time angle control to ensure time coherence of the video sequence.

[0146] The video generation module in the embodiment of the present application adopts an information fusion mechanism to add the output result of the first processing branch to the second processing branch in an additive form, thereby realizing fine control of the video generation process. Through the information fusion mechanism, the video generation module can flexibly adjust scene details such as the position, color and motion trajectory of an object while maintaining the basic structure of the video.

[0147] In addition, the video generation module in the embodiment of the present application can also introduce a progressive training strategy, gradually increase the resolution and length of the video, and enable the video generation model to focus on learning different levels of features in different training stages, thereby significantly improving the training efficiency and generation quality.

[0148] In some embodiments, the step S23 of fusing the first video parameter into the second processing branch includes steps S231 to S232, wherein: The step S231 fuses the output result of the first spatio-temporal diffusion module of the i-th first processing unit in the first processing branch with the output result of the multi-view spatio-temporal diffusion module of the i-th second processing unit in the second processing branch, and inputs them into the third spatio-temporal diffusion module of the i-th second processing unit.

[0149] The first processing unit refers to a processing module located in the first processing branch (control branch), and each processing unit includes two sub-modules: a first spatio-temporal diffusion module and a second spatio-temporal diffusion module. The first spatio-temporal diffusion module is responsible for spatial condition control, and the second spatio-temporal diffusion module is responsible for time angle control. In addition to processing the same first control parameter group as the second processing branch (basic branch), the first processing branch in the embodiment of the present application additionally processes BevMap information. BevMap provides global information of a scene, and the control branch extracts inter-frame continuity and spatial consistency information by processing the BevMap information, which can provide more comprehensive control signals for the video generation process, thereby enhancing the time coherence of the video sequence. For example, when processing a vehicle turning scene, the first spatio-temporal diffusion module can identify the posture change of the vehicle in consecutive frames and maintain the smoothness of the motion trajectory.

[0150] The multi-view spatio-temporal diffusion module is a key module in the second processing branch (basic branch) for generating multi-view videos. The multi-view spatio-temporal diffusion module not only considers the spatial features of a single frame image, but also introduces semantic information of multiple views (such as front view, side view, and rear view), models the spatial relationship between different views according to the input information, and ensures the consistency of the position and posture of objects in different views. For example, when switching between the bird's eye view and the front view, the multi-view spatio-temporal diffusion module can maintain the consistency of the vehicle position, avoiding the problem of object drift or misplacement caused by view switching.

[0151] The third spatio-temporal diffusion module is a module in the second processing branch (basic branch) for integrating information from the first processing branch and the multi-view spatio-temporal diffusion module, and generating the final high-resolution video frame. The third spatio-temporal diffusion module selectively fuses multi-source information through an attention mechanism, improving the detail restoration ability and semantic accuracy of the video. For example, when synthesizing a night rain road scene, the third spatio-temporal diffusion module can combine multi-dimensional information such as lighting conditions and road reflection to generate a more realistic visual effect.

[0152] Step S232, adding the output result of the second spatio-temporal diffusion module of the i-th first processing unit to the output end of the third spatio-temporal diffusion module of the i-th second processing unit.

[0153] The second spatio-temporal diffusion module is the second sub-module in the first processing unit, mainly responsible for time angle control.

[0154] The third spatio-temporal diffusion module has received the output result of the multi-view spatio-temporal diffusion module and has completed preliminary fusion processing. The output result of the second spatio-temporal diffusion module is fused with the output result of the third spatio-temporal diffusion module again, so as to further enhance the semantic richness and detail expressiveness of the video. For example, when generating a crossroads scene during a traffic peak period, the system can ensure that information such as vehicle density, pedestrian behavior, and signal light state can be accurately presented through the fusion operation of the output result of the second spatio-temporal diffusion module and the output result of the third spatio-temporal diffusion module.

[0155] In specific implementation, the output result of the second spatio-temporal diffusion module usually contains semantic vectors, time step embeddings, etc., and the output result of the third spatio-temporal diffusion module includes image features optimized by multiple views. The output results of the second spatio-temporal diffusion module and the third spatio-temporal diffusion module can be fused by using weighted summation, splicing, or attention mechanism, etc., and the finally generated second video parameter has higher spatio-temporal consistency and semantic integrity.

[0156] In some embodiments, with reference to Figure 9, the multi-view spatio-temporal diffusion module and the information fusion process of the spatio-temporal diffusion module are shown. In the embodiment of the application, the first processing unit in the first processing branch (control branch) is composed of two spatio-temporal diffusion modules, and the second processing unit in the second processing branch (basic branch) is composed of a multi-view spatio-temporal diffusion module and a spatio-temporal diffusion module. As shown in Figure 8 , in the two branches, the control function of the camera pose, the ego trajectory, the 3D bounding box and the text condition is fused by the multi-head attention mechanism. Specifically, it can be expressed as the following formula:

[0157] , wherein, is the encoding format of the camera pose, the ego trajectory, the 3D bounding box and the text condition.

[0158] In the embodiment of the application, a fine data fusion mechanism is established between the basic branch and the control branch, which realizes the effective cooperation of local dynamic features and global semantic information in the video generation process, is conducive to improving the spatio-temporal consistency of the generated video, enhancing the matching degree of object position and action under multi-view, and generating more real and controllable automatic driving training videos.

[0159] Next, the video generation method provided by the application will be described in detail in combination with a specific embodiment.

[0160] The video generation model in the embodiment of the application includes a spatial compression encoder, a multi-modal condition encoder and a video generation module based on a diffusion self-attention model (Diffusion Transformer, DIT).

[0161] The spatial compression encoder adopts a 3DVAE, adopts a three-dimensional convolution kernel, simultaneously captures the spatial features and the motion correlation in the time dimension, and ensures that each generated frame is strictly aligned with the real world in the geometric topology. Compared with the jitter artifacts caused by the compression of the spatial dimension by the traditional 2DVAE, the causal convolution design of the 3DVAE forces the model to only rely on the historical frame information to reconstruct the current frame, fundamentally avoiding the motion discontinuity caused by the leakage of future information.

[0162] After the result of the DIT generating the latent space, the decoder of the 3DVAE is used to conditionally decode the video parameters to generate a generated video that conforms to the geometric control. For example, when the video generation model needs to generate a vehicle turning scene, the latent space trajectory is constrained by conditional decoding to conform to the physical dynamics model, avoiding the generation of distorted pictures that violate the centrifugal force.

[0163] In some embodiments, the compression ratio of the 3DVAE is 4x8x8 in time T and space W, H.

[0164] In the embodiments of the present application, the multi-modal video generation control conditions are divided into geometric control conditions and text conditions, wherein the geometric control conditions can include camera pose parameters, ego vehicle trajectory information, three-dimensional bounding boxes and map information. Based on the definition of each control condition in the embodiments of the present application, the multi-modal condition encoder in the embodiments of the present application includes a camera pose encoder, an ego vehicle trajectory encoder, a three-dimensional bounding box encoder, a map encoder and a text encoder.

[0165] wherein the camera pose encoder is configured to convert the camera pose parameters into a high-dimensional feature vector to retain the spatial information of the camera parameters. Exemplarily, for the camera pose parameters, they can be represented as wherein, is the camera pose (6x3x7), is the camera intrinsic matrix (6x3x3), is the rotation matrix (6x3x1), is the translation vector (6x3x1). In the embodiments of the present application, a multi-layer perception machine can be used to encode the camera pose parameters.

[0166] Further, since contains values from a sine function or a cosine function, and a three-dimensional offset, in order to enable the video generation model to effectively interpret these high-frequency changes, the embodiments of the present application first use a Fourier transform layer to perform Fourier transform processing on the camera pose parameters before processing the camera pose parameters using a multi-layer perception machine (MLP). Exemplarily, the specific encoding process of the camera pose parameters in the embodiments of the present application is as shown below: (8) wherein, is the camera pose encoding, is the camera pose parameter, represents the first pose parameter, represents the encoding processing of the first pose parameter using a multi-layer perception machine layer.

[0167] For the ego vehicle trajectory information, since its form is consistent with the form of the rotation matrix and the translation vector in the camera pose parameters, the embodiments of the present application also use the Fourier transform formula before encoding the ego vehicle trajectory information. The structure of the ego vehicle trajectory encoder in the embodiments of the present application is as shown in Figure 6 the self-attention layer in Figure 6 is configured to capture the position information between different video frames. The pooling layer is responsible for down-sampling processing, which is used to realize the temporal alignment of the ego vehicle trajectory encoding and the latent space features output by the 3DVAE.

[0168] For the encoding of the 3D bounding box, as Figure 7As shown, the three-dimensional bounding box encoder in the embodiment of the present application adopts a dual-stream fusion mechanism to process the class information and geometric information of each bounding box. Exemplarily, the class label The word vectors are extracted by the CLIP text encoder and are subjected to average pooling to generate semantic embedding parameters While the 8 corner coordinates of the description box space are also subjected to Fourier embedding and MLP encoding to generate position embedding parameters The two types of embedding parameters are compressed into unified representations by a special MLP network which can be specifically expressed as:

[0169] For map information such as BevMap, a two-dimensional grid format can be adopted, and a map encoder is used to compress the map information in the latent space in space and time. As Figure 5 shown, the map encoder in the embodiment of the present application can be 2 groups of down-sampling convolution, one group of which is a spatial convolution group with a convolution kernel size of 1x8x8, used for spatial compression of the map information; the other group is a temporal convolution group with a convolution kernel size of 4x1x1, used for temporal compression of the map information. By spatial and temporal compression of the map information, road global features can be extracted.

[0170] For text conditions, a Text-to-Text Transfer Transformer (T5) model can be directly used as a text encoder, which is used to convert natural language descriptions (such as rainy night, congested road section) into semantic vectors, i.e., text encoding in the embodiment of the present application.

[0171] As Figure 8 shown, the video generation module includes a first processing branch, a second processing branch, and a decoder. Among them, Figure 8 The left side indicates that different types of data need to be encoded respectively, and the purpose is to convert these heterogeneous data into feature representations suitable for model processing. In order to increase the diversity of generated videos and realize the generation function of the model according to the control conditions, random noise is input for generation. The dimension of the noise is and then the noise is compressed by the 3DVAE to generate latent space features. In order to further reduce the resource consumption in the real-time generation process of the video, the latent space noise can be divided into smaller noise blocks, and after adding position encoding, it is input into the video generation module.

[0172] The main framework of the video generation module in the embodiments of the present application adopts a double-branch structure, including a first processing branch (control branch) and a second control branch (basic branch). This design aims to process different types of information through different branches, and inject the information of the control branch into the basic branch to achieve fine control of the video generation process. The embodiments of the present application use a space-time diffusion module and introduce a multi-view space-time diffusion module for multi-view generation, which together constitute a double-branch structure. The space-time diffusion module can effectively process space-time information, and the multi-view space-time diffusion module is specifically used to process multi-view data to support multi-view video generation.

[0173] The basic branch is composed of repeated small modules, and the small modules in turn include a multi-view space-time diffusion module and a space-time diffusion module. The multi-view space-time diffusion module is responsible for spatial condition control and is used to model the spatial relationship between different views according to the input information to ensure that the generated video is consistent and reasonable in space. The space-time diffusion module is responsible for time angle control and is used to capture the temporal dynamic information in the video sequence to make the generated video have continuity in time. The number of small modules in the basic branch can be twice that of the control branch, so that only the additional information, i.e., text, bounding box, camera pose and ego trajectory information, and latent spatial features, is fused in the first half of the basic branch. In this way, the basic branch can use these additional information to build the basic framework of the video in the early stage.

[0174] The control branch is also composed of small modules, each of which includes two space-time diffusion modules, which have the same corresponding functions as the small modules in the basic branch and are responsible for spatial condition control and time angle control, respectively. In addition to processing the same part of information as the basic branch, the control branch additionally processes BevMap information. BevMap provides global information of the scene, and the control branch can provide more comprehensive control signals for the video generation process by processing BevMap information.

[0175] In the double-branch structure of the video generation module, there is information interaction between the modules of the control branch and the basic branch. Specifically, the output results of the modules in the control branch are added to the basic modules in the form of addition. By adding the output results of the control branch to the basic branch, the embodiments of the present application can inject additional control signals into the processing process of the basic branch, thereby achieving fine control of video generation.

[0176] As shown in FIG. 1, in the two branches of the embodiments of the present application, the control functions of the camera pose, the ego trajectory, the 3D bounding box and the text condition are fused through the multi-head attention mechanism, which can be expressed as the following formula: Figure 9

[0177] ​in, It is the encoding format for camera pose, vehicle trajectory, 3D bounding box, and text conditions.

[0178] In summary, the video generation method provided in this application decouples the geometric control conditions and textual conditions of the generated video. It uses a multimodal conditional encoder in the video generation model to encode the geometric control conditions and textual conditions respectively, introduces random noise, and uses a spatial compression encoder to compress the high-dimensional random noise into low-dimensional latent spatial features. The latent spatial features and the encoded multimodal control parameters are input together into the video generation module, so that the video generation module performs calculations based on the multimodal control parameters in the latent space to generate the target video, thereby improving the quality and controllability of the generated video.

[0179] Figure 10 This is a schematic diagram of a video generation device provided in an embodiment of this application. Figure 10 As shown, the video generation device 1000 includes: an acquisition module 1010, a compression module 1020, an encoding module 1030, and a generation module 1040, wherein: The acquisition module 1010 is used to acquire multimodal video generation control conditions; wherein, the multimodal video generation control conditions include geometric control conditions and text conditions; the geometric control conditions are used to describe the spatial position, motion trajectory and geometric relationship of objects in the video; the text conditions are used to describe the global semantic features of the video; Compression module 1020 is used to compress random noise using the spatial compression encoder in the video generation model to generate latent spatial features; The encoding module 1030 is used to encode the geometric control conditions and text conditions separately using the multimodal condition encoder in the video generation model to obtain multimodal control parameters; The generation module 1040 is used to input latent spatial features and multimodal control parameters into the video generation module in the video generation model for processing to obtain the target video.

[0180] In some embodiments, the video generation module includes a first processing branch, a second processing branch, and a decoder; the generation module 1040 includes: The grouping submodule is used to divide the multimodal control parameters into a first control parameter group and a second control parameter group based on video generation requirements. The first processing submodule is used to process the first control parameter group and the second control parameter group using the first processing branch to obtain the first video parameters; The second processing submodule is used to process the first control parameter group and latent spatial features using the second processing branch, and to fuse the first video parameters into the second processing branch to obtain the second video parameters; The decoding submodule is configured to decode the second video parameter by using a decoder to obtain a target video.

[0181] In some embodiments, the first processing branch includes M first processing units, the first processing unit includes a first space-time diffusion module and a second space-time diffusion module; the second processing branch includes N second processing units, the second processing unit includes a multi-view space-time diffusion module and a third space-time diffusion module; M and N are positive integers, and M is less than N; the first video parameter includes the output results of the first space-time diffusion module and the second space-time diffusion module in the M first processing units; the second processing submodule includes: The first fusion unit is configured to fuse the output result of the first space-time diffusion module of the i-th first processing unit in the first processing branch and the output result of the multi-view space-time diffusion module of the i-th second processing unit in the second processing branch, and input the fused result to the third space-time diffusion module of the i-th second processing unit. The second fusion unit is configured to add the output result of the second space-time diffusion module of the i-th first processing unit to the output end of the third space-time diffusion module of the i-th second processing unit.

[0182] In some embodiments, the geometric control condition includes camera pose parameters, ego vehicle trajectory information, three-dimensional bounding box and map information; the multi-modal condition encoder includes a camera pose encoder, an ego vehicle trajectory encoder, a three-dimensional bounding box encoder, a map encoder and a text encoder; the encoding module 1030 includes: The first encoding submodule is configured to encode the camera pose parameters by using the camera pose encoder to obtain camera pose encoding. The second encoding submodule is configured to encode the ego vehicle trajectory information by using the ego vehicle trajectory encoder to obtain ego vehicle trajectory encoding. The third encoding submodule is configured to encode the three-dimensional bounding box by using the three-dimensional bounding box encoder to obtain three-dimensional bounding box encoding. The fourth encoding submodule is configured to encode the map information by using the map encoder to obtain road global features. The fifth encoding submodule is configured to encode the text condition by using the text encoder to obtain text encoding.

[0183] In some embodiments, the camera pose encoder includes a Fourier transform layer and a multi-layer perception layer; the first encoding submodule includes: The first processing unit is configured to perform Fourier transform on the camera pose parameters by using the Fourier transform layer to obtain first pose parameters. The second processing unit is configured to encode the first pose parameters by using the multi-layer perception layer to obtain camera pose encoding.

[0184] In some embodiments, the ego vehicle trajectory encoder comprises a Fourier transform layer, a self-attention layer, a multi-layer perception layer, and a pooling layer; the second encoding submodule comprises: The third processing unit is configured to perform Fourier transform on the ego vehicle trajectory information by using the Fourier transform layer to obtain first ego vehicle trajectory parameters; The fourth processing unit is configured to perform rotation position encoding on the first ego vehicle trajectory parameters by using the self-attention layer to obtain second ego vehicle trajectory parameters; The fifth processing unit is configured to encode the second ego vehicle trajectory parameters by using the multi-layer perception layer to obtain initial ego vehicle trajectory encoding; The sixth processing unit is configured to perform dimension reduction processing on the initial ego vehicle trajectory encoding by using the pooling layer to obtain ego vehicle trajectory encoding.

[0185] In some embodiments, the three-dimensional bounding box comprises a category parameter and a position parameter; the three-dimensional bounding box encoder comprises a Fourier transform layer, a first multi-layer perception layer, a text encoder, a pooling layer, and a second multi-layer perception layer; the third encoding submodule comprises: The seventh processing unit is configured to perform Fourier transform on the position parameter by using the Fourier transform layer to obtain first position parameters; The eighth processing unit is configured to encode the first position parameters by using the first multi-layer perception layer to obtain position embedding parameters; The ninth processing unit is configured to extract the category parameter by using the text encoder to obtain a word vector of the category parameter; The tenth processing unit is configured to perform dimension reduction processing on the word vector by using the pooling layer to obtain semantic embedding parameters; The eleventh processing unit is configured to compress the position embedding parameters and the semantic embedding parameters by using the second multi-layer perception layer to obtain three-dimensional bounding box encoding.

[0186] The above description of the video generation apparatus is similar to the description of the video generation method embodiments, and has similar beneficial effects as the video generation method embodiments. For technical details not disclosed in the video generation apparatus embodiments of the present application, please refer to the description of the video generation method embodiments of the present application for understanding.

[0187] It should be noted that, in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or the part that contributes to the related art, which is stored in a storage medium, includes a number of instructions to make an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware and software combination.

[0188] The present application also provides an electronic device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the above method when executing the computer program.

[0189] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method. The computer readable storage medium can be transitory or non-transitory.

[0190] The present application also provides a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to implement part or all steps of the above method. The computer program product can be specifically implemented by hardware, software or combination thereof. The computer program product can be specifically implemented by hardware, software or combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as software development kit (Software Development Kit, SDK) and the like.

[0191] It should be noted that, Figure 11 A hardware entity schematic diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 11, which includes a processor 1101, a communication interface 1102 and a memory 1103. Figure 11 The hardware entity of the electronic device 1100 includes a processor 1101, a communication interface 1102 and a memory 1103, wherein: The processor 1101 usually controls the overall operation of the electronic device 1100.

[0192] The communication interface 1102 can enable the electronic device 1100 to communicate with other terminals or servers through a network.

[0193] The memory 1103 is configured to store instructions and applications executable by the processor 1101, and can also cache data (for example, image data, audio data, voice communication data, and video communication data) to be processed by the processor 1101 and modules in the electronic device 1100, which have been processed or are being processed. The memory 1103 can be implemented by a FLASH or a RAM. The processor 1101, the communication interface 1102, and the memory 1103 can perform data transmission through the bus 1104.

[0194] Here, the electronic device can be a car machine in a vehicle.

[0195] It should be noted that the above description of the storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application.

[0196] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process does not mean the order of execution, and the execution order of each step / process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The serial number of the above embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0197] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0198] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The above described device embodiments are merely exemplary. For example, the division of the units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the above components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0199] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place, or distributed on a plurality of network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0200] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0201] Those of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above method embodiments when executed; and the foregoing storage medium includes mobile storage devices, read-only memories, magnetic discs or optical discs, and various media that can store program codes.

[0202] Alternatively, the integrated units of the present application, if implemented in the form of software functional modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the embodiments of the method of the present application. The foregoing storage medium includes mobile storage devices, ROM, magnetic discs or optical discs, and various media that can store program codes.

[0203] The above merely provides the implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the change or replacement within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method of video generation, the method comprising: The method comprises: acquiring a multi-modal video generation control condition; wherein the multi-modal video generation control condition comprises a geometric control condition and a text condition; the geometric control condition is used to describe the spatial position, motion trajectory and geometric relationship of an object in a video; and the text condition is used to describe the global semantic features of the video; compressing random noise using a spatial compression encoder in a video generation model to generate latent space features; encoding the geometric control condition and the text condition respectively using a multi-modal condition encoder in the video generation model to obtain multi-modal control parameters; inputting the latent space features and the multi-modal control parameters into a video generation module in the video generation model for processing to obtain a target video.

2. The method of claim 1, wherein, The video generation module comprises a first processing branch, a second processing branch and a decoder; The inputting of the latent space features and the multi-modal control parameters into the video generation module in the video generation model for processing to obtain a target video comprises: dividing the multi-modal control parameters into a first control parameter group and a second control parameter group based on a video generation requirement; processing the first control parameter group and the second control parameter group using the first processing branch to obtain first video parameters; processing the first control parameter group and the latent space features using the second processing branch and fusing the first video parameters into the second processing branch to obtain second video parameters; decoding the second video parameters using the decoder to obtain a target video.

3. The method of claim 2, wherein, The first processing branch comprises M first processing units, and the first processing units comprise first and second spatio-temporal diffusion modules; the second processing branch comprises N second processing units, and the second processing units comprise multi-view spatio-temporal diffusion modules and third spatio-temporal diffusion modules; M and N are positive integers, and M is less than N; the first video parameters comprise the output results of the first and second spatio-temporal diffusion modules in the M first processing units; The fusing of the first video parameters into the second processing branch comprises: fusing the output result of the first spatio-temporal diffusion module of the i-th first processing unit in the first processing branch with the output result of the multi-view spatio-temporal diffusion module of the i-th second processing unit in the second processing branch and inputting them into the third spatio-temporal diffusion module of the i-th second processing unit; adding the output result of the second spatio-temporal diffusion module of the i-th first processing unit to the output end of the third spatio-temporal diffusion module of the i-th second processing unit.

4. The method according to any one of claims 1 to 3, characterized in that, The geometric control condition comprises camera pose parameters, ego vehicle trajectory information, three-dimensional bounding boxes and map information; The multi-modal condition encoder comprises a camera pose encoder, an ego vehicle trajectory encoder, a three-dimensional bounding box encoder, a map encoder and a text encoder; The encoding of the geometric control condition and the text condition respectively using the multi-modal condition encoder in the video generation model to obtain multi-modal control parameters comprises: The camera pose parameter is encoded by using the camera pose encoder to obtain camera pose encoding; The ego vehicle trajectory information is encoded by using the ego vehicle trajectory encoder to obtain ego vehicle trajectory encoding; The three-dimensional bounding box is encoded by using the three-dimensional bounding box encoder to obtain three-dimensional bounding box encoding; The map information is encoded by using the map encoding to obtain road global features; The text condition is encoded by using the text encoder to obtain text encoding.

5. The method of claim 4, wherein, The camera pose encoder includes a Fourier transform layer and a multi-layer perception layer; the camera pose parameter is encoded by using the camera pose encoder to obtain camera pose encoding, including: The Fourier transform layer is used to perform Fourier transform on the camera pose parameter to obtain a first pose parameter; The multi-layer perception layer is used to encode the first pose parameter to obtain the camera pose encoding.

6. The method of claim 4, wherein, The ego vehicle trajectory encoder includes a Fourier transform layer, a self-attention layer, a multi-layer perception layer and a pooling layer; the ego vehicle trajectory information is encoded by using the ego vehicle trajectory encoder to obtain ego vehicle trajectory encoding, including: The Fourier transform layer is used to perform Fourier transform on the ego vehicle trajectory information to obtain a first ego vehicle trajectory parameter; The self-attention layer is used to perform rotation position encoding on the first ego vehicle trajectory parameter to obtain a second ego vehicle trajectory parameter; The multi-layer perception layer is used to encode the second ego vehicle trajectory parameter to obtain initial ego vehicle trajectory encoding; The pooling layer is used to perform dimension reduction processing on the initial ego vehicle trajectory encoding to obtain the ego vehicle trajectory encoding.

7. The method of claim 4, wherein, The three-dimensional bounding box includes a category parameter and a position parameter; the three-dimensional bounding box encoder includes a Fourier transform layer, a first multi-layer perception layer, a text encoder, a pooling layer and a second multi-layer perception layer; the three-dimensional bounding box is encoded by using the three-dimensional bounding box encoder to obtain three-dimensional bounding box encoding, including: The Fourier transform layer is used to perform Fourier transform on the position parameter to obtain a first position parameter; The first multi-layer perception layer is used to encode the first position parameter to obtain a position embedding parameter; The text encoder is used to extract the category parameter to obtain a word vector of the category parameter; The pooling layer is used to perform dimension reduction processing on the word vector to obtain a semantic embedding parameter; The second multi-layer perception layer is used to compress the position embedding parameter and the semantic embedding parameter to obtain the three-dimensional bounding box encoding.

8. A video generating apparatus characterized by comprising: The device includes: An acquisition module is configured to acquire a multi-modal video generation control condition; wherein the multi-modal video generation control condition includes a geometric control condition and a text condition; the geometric control condition is used to describe the spatial position, motion trajectory and geometric relationship of an object in a video; the text condition is used to describe the global semantic features of the video; A compression module is configured to compress random noise by using a spatial compression encoder in a video generation model to generate latent space features; The encoding module is configured to encode the geometric control condition and the text condition respectively by using a multi-modal conditional encoder in the video generation model to obtain multi-modal control parameters; The generation module is configured to input the latent space feature and the multi-modal control parameters into a video generation module in the video generation model for processing to obtain a target video.

9. An electronic device, comprising: A processor and a memory are included, the memory stores a computer program capable of running on the processor, and the processor implements the steps in the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program implements the steps in the method of any one of claims 1 to 7 when executed by a processor.

Images