Automatic driving scene multi-view video generation method and system based on space-time consistency constraint

Abstract

The invention discloses an automatic driving scene multi-view video generation method and system based on space-time consistency constraint. According to the method, in a diffusion model generation process, three attention mechanisms are integrated: based on cross-view-angle deformable cross attention of a camera pose, geometric consistency among different view angles is realized; on the basis of cross-frame cross attention of optical flow prior, motion coherence between adjacent frames is ensured; and on the basis of the global time sequence consistency constraint of cross attention, the reasonability of long sequence generation is ensured. According to the method, the problems of geometric distortion and time sequence breakage in multi-view video generation are effectively solved, high-quality dynamic scene videos with consistent time and space can be generated, and efficient data support is provided for automatic driving simulation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, generative models, and spatiotemporal data processing, and in particular to a method and system for generating multi-view videos of autonomous driving scenarios based on spatiotemporal consistency constraints. Background Technology

[0002] The application of generative models, represented by diffusion models, in autonomous driving offers a new approach to solving perception problems in dynamic and complex environments. Generative models possess the ability to model complex data distributions, generating high-quality visual data and diverse traffic scenes, providing more comprehensive training data, and are suitable for tasks such as long-tail scene generation and high-fidelity simulation in autonomous driving. In recent years, multi-source perception and controllable generation methods for autonomous driving based on diffusion models have gained widespread attention and achieved some progress. Through implicit representation learning and probability-driven generation mechanisms of diffusion models, preliminary joint modeling and dynamic scene synthesis of multimodal data (LiDAR, vision, semantic maps, text, etc.) have been realized. However, autonomous driving scene perception and generation still face the challenge of high spatiotemporal feature alignment in multi-view video generation of dynamic scenes. In the generation of complex and dynamic autonomous driving scenes, existing diffusion models are insufficient in modeling the spatiotemporal continuity of moving targets, resulting in high difficulty in spatiotemporal feature alignment. The projection of dynamic targets under multiple views cannot satisfy the geometric constraints of three-dimensional space, and objects in different frames from the same view cannot maintain motion continuity in long sequences, leading to temporal jitter and multi-view spatial mismatch problems. Therefore, how to uncover the patterns of cross-view geometric consistency and cross-frame motion continuity to construct effective constraints and ensure the stability of dynamic scene generation for autonomous driving is a key scientific problem that this project needs to solve. This invention focuses on data generation, using a diffusion model to generate multi-view video data, and focuses on solving the cross-view and cross-frame consistency problems in the generation process. The high-quality, multi-view videos generated can provide richer and more realistic training data for perception, making up for the difficulty in collecting real samples, which is a task of great practical significance. Summary of the Invention

[0003] The purpose of this invention is to address the demand for high-fidelity dynamic scene data in autonomous driving simulation testing. During the dynamic scene generation process, a cross-view deformable cross-attention mechanism based on camera pose and a cross-frame cross-attention mechanism based on optical flow prior are used to achieve spatiotemporal consistency of generated images from different viewpoints and at different times. A global temporal consistency constraint based on the cross-attention mechanism is proposed to optimize global consistency in the time dimension and ensure the temporal rationality in dynamic scene generation. A multi-view video generation method and system for autonomous driving scenes based on spatiotemporal consistency constraints are designed.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A method for generating multi-view videos of autonomous driving scenarios based on spatiotemporal consistency constraints is proposed. In the generation process based on a diffusion model, the following three attention mechanisms are integrated to apply spatiotemporal consistency constraints:

[0006] (1) Cross-view deformable cross attention mechanism based on camera pose: The pixel coordinates on the source view image plane are transformed to the three-dimensional world coordinate system using the camera pose matrix, and then projected onto the target view image plane. Then, the cross attention between the source view and the target view feature maps is calculated to achieve cross-view geometric consistency.

[0007] (2) Cross-frame cross-attention mechanism based on optical flow prior: using the optical flow prior calculated from the images of the previous and next frames, the feature map of the current frame is aligned with the feature map of the previous frame, and the cross-attention between the two is calculated to achieve motion consistency between adjacent frames.

[0008] (3) Global temporal consistency constraint based on cross-attention mechanism: Calculate cross-frame attention by using the stitching result of the latent representation of the current frame and the image features of the historical frames to achieve motion coherence in the global temporal range.

[0009] Furthermore, the method specifically includes the following steps:

[0010] Step S1: Input past multi-frame multi-view images, use encoder ε to perform feature encoding, and use it as input z0 of the diffusion model;

[0011] Step S2: The input z0 obtained in step S1 is diffused to obtain the noise z. T ;

[0012] Step S3, the noise z obtained in step S2 T Input a 3D UNet denoising network and use at least one of text, map, or detection bounding boxes as conditional control for image generation;

[0013] Step S4, based on the cross-view deformable cross-attention mechanism of camera pose, firstly, the coordinates of the source pixel s are ( , Transforming from the image plane to the three-dimensional world coordinate system, in The corresponding depth value in the feature map ;

[0014] Step S5, convert the three-dimensional coordinates ( , , Projected onto the target camera On the image plane, where the camera pose matrix is... It contains rotation and translation information, where R is the rotation matrix and b is the translation vector;

[0015] Step S6, for the source feature map The source pixel s in the target feature map Cross-attention is calculated for the target pixel position e and its k×k neighbors, where the weight matrices for query, key, and value are respectively... , , The query, key, and value are obtained as follows: , , ;

[0016] Step S7, Sampling position ,in It is relative to the target pixel position The offset; subscript Indicates the relationship between samples within the neighborhood. Indices representing the indices of neighboring points. Represents a k×k neighborhood;

[0017] Step S8, at the sampling location Bilinear interpolation is performed on K and V to obtain the sampled key and value. , Calculate the cross-attention weights;

[0018] Step S9, Aggregate value Then perform feature updates. ,in Indicates the cross-attention weights;

[0019] Step S10: Based on the optical flow prior, a cross-frame cross-attention mechanism is used, given the previous frame image. and the current frame image The optical flow prior is calculated using the pre-trained optical flow estimation model FlowNet;

[0020] Step S11: Based on the optical flow prior obtained in step S10, the feature map of the current frame is... Aligned to the feature map of the previous frame The aligned feature map is obtained. ;

[0021] Step S12: Calculate the cross-attention between the current frame and the previous frame using optical flow information;

[0022] Step S13, perform feature update ;

[0023] Step S14: Based on the global temporal consistency constraint of the cross-attention mechanism, use the latent representation of the t-th frame. and features of the first frame image Features of the (t-1)th frame image The sum is used to calculate cross-frame attention;

[0024] Step S15, finally passed through a decoder The generated future multi-view video results are obtained from the diffusion model. The objective function of the denoising network is the same as the objective function of the conditional denoising diffusion model. At the same time, the L2 loss function is used to ensure the consistency between the generated video and the real video.

[0025] Furthermore, in step S4, the coordinate system transformation formula is:

[0026]

[0027] Among them, P ｉ This indicates the pose of the source camera.

[0028] Furthermore, in step S5, the projection formula is:

[0029]

[0030] in, , Represents the unnormalized image coordinates after projection. This represents the homogeneous components after projection. Represents the normalized plane coordinates of the projected image.

[0031] Furthermore, in step S8, the formula for calculating the cross-attention weights is:

[0032] .

[0033] Furthermore, in step S10, the formula for calculating the optical flow prior is:

[0034] .

[0035] Furthermore, in step S12, the formula for calculating cross-attention is:

[0036]

[0037] in, , , These are learnable matrices that map the input to query, key, and value features.

[0038] Furthermore, in step S14, the formula for calculating cross-frame attention is:

[0039]

[0040] in, This indicates the query features of the current frame. , These represent the key and value features of the current frame concatenated with the previous frame, respectively, with [,] indicating the feature concatenation operation. , , It is a learnable matrix that maps the input to query, key, and value features, where d is the output dimension of the query and key features.

[0041] The present invention also provides a multi-view video generation system for autonomous driving scenarios.

[0042] A multi-view video generation system for autonomous driving scenarios, used to perform the method, including:

[0043] Encoder ε is used to encode features of the input multi-frame, multi-view image;

[0044] A diffusion model is used to perform the diffusion and denoising processes.

[0045] The 3D UNet denoising network integrates a cross-view deformable cross attention module, a cross-frame cross attention module, and a global temporal consistency constraint module.

[0046] decoder It is used to output the generated future multi-view video.

[0047] Beneficial Effects: This invention incorporates camera pose information and designs a deformable cross-viewpoint cross-attention mechanism to precisely align images from different perspectives, ensuring geometric consistency between objects and the background in each viewpoint. Simultaneously, by utilizing optical flow information as prior knowledge, a cross-frame cross-attention mechanism is constructed to effectively capture the motion trajectory of dynamic targets, guaranteeing the dynamic coherence of the video in the temporal dimension. Finally, these mechanisms are integrated into a diffusion model to achieve effective interaction of multi-frame image information, generating high-quality multi-view dynamic scene videos with strong global temporal consistency and geometric alignment, providing efficient and large-scale virtual simulation for autonomous driving data. Attached Figure Description

[0048] Figure 1 It is a holistic research framework for multi-view video generation based on spatiotemporal consistency;

[0049] Figure 2 This is a schematic diagram of cross-view deformable cross attention and cross-frame cross attention. Detailed Implementation

[0050] The invention will now be further explained with reference to the accompanying drawings.

[0051] This invention presents a multi-view video generation method for autonomous driving scenarios based on spatiotemporal consistency constraints, aiming to improve the stability of video generation by studying multi-view video methods based on spatiotemporal consistency. In its specific implementation, the technical approach is designed from three aspects:

[0052] (1) Design a cross-view deformable attention mechanism based on camera pose information to effectively integrate image information from different viewpoints and ensure cross-view geometric consistency in scene generation;

[0053] (2) By using a cross-frame attention mechanism based on optical flow prior, time series information is effectively used to associate object motion with scene changes, thereby improving cross-frame motion consistency;

[0054] (3) A global temporal consistency constraint based on cross-attention mechanism is proposed. Through global temporal association, the rationality and coherence of the motion of objects and background in dynamic scene generation are guaranteed.

[0055] The operation steps of the present invention are described below with reference to embodiments.

[0056] Example 1:

[0057] This embodiment details each step of the method:

[0058] Figure 1 This invention provides an overall research framework for multi-view video generation based on spatiotemporal consistency, such as... Figure 1 As shown, this embodiment specifically includes the following steps:

[0059] Step S1: Input past multi-frame, multi-view images, use encoder ε to perform feature encoding, and use this as input z0 to the diffusion model, such as... Figure 1 As shown;

[0060] Step S2, the input z0 from step S1 is diffused to obtain noise z. T ;

[0061] Step S3, change z in S2 T Input the data into a 3D UNet denoising network, and simultaneously control image generation using conditions such as text, map, and detection boxes;

[0062] Step S4, based on the cross-view deformable cross-attention mechanism of camera pose, firstly, the coordinates of the source pixel s are ( , Transforming from the image plane to the three-dimensional world coordinate system, in The corresponding depth value in the feature map The coordinate system transformation formula is:

[0063]

[0064] Among them, P ｉ Indicates the pose of the source camera;

[0065] Step S5, set the 3D coordinates ( , , Projected onto the target camera On the image plane, the projection formula is:

[0066] ,

[0067] Camera pose matrix It contains rotation and translation information, where R is the rotation matrix and b is the translation vector;

[0068] in, , Represents the unnormalized image coordinates after projection. This represents the homogeneous components after projection. Represents the normalized plane coordinates of the projected image;

[0069] Step S6, for the source feature map The source pixel s in the target feature map Cross-attention is calculated for the target pixel position e and its k×k neighbors, where the weight matrices for query, key, and value are respectively... , , The query, key, and value are obtained as follows: , , ;

[0070] Step S7, Sampling position ,in It is relative to the target pixel position The offset is typically a small, random offset to enhance sampling flexibility; where subscript Indicates the relationship between samples within the neighborhood. Indices representing the indices of neighboring points. Represents a k×k neighborhood;

[0071] Step S8, at the sampling location Bilinear interpolation is performed on K and V to obtain the sampled key and value. , The cross-attention weights are calculated using the following formula:

[0072] ;

[0073] Step S9, Aggregate value Then perform feature updates. ;in Indicates the cross-attention weights;

[0074] Step S10: Based on the optical flow prior, a cross-frame cross-attention mechanism is used, given the previous frame image. and the current frame image We can first use the pre-trained optical flow estimation model FlowNet to calculate the optical flow prior, such as... Figure 2 As shown;

[0075] The formula for calculating the optical flow prior is as follows:

[0076]

[0077] Step S11: Based on the optical flow prior obtained in S10, the feature map of the current frame is... Aligned to the feature map of the previous frame The aligned feature map is obtained. ;

[0078] Step S12: Using optical flow information, calculate the cross-attention between the current frame and the previous frame. The calculation formula is as follows: ;

[0079] in, , , These are learnable matrices that map the input to query, key, and value features.

[0080] Step S13, perform feature update ;

[0081] Step S14: Based on the global temporal consistency constraint of the cross-attention mechanism, use the latent representation of the t-th frame. and features of the first frame image Features of the (t-1)th frame image The sum of these values ​​is used to calculate cross-frame attention, and the formula is as follows:

[0082]

[0083] in, This indicates the query features of the current frame. , These represent the key and value features of the current frame concatenated with the previous frame, respectively, with [,] indicating the feature concatenation operation. , , It is a learnable matrix that maps the input to query, key, and value features, where d is the output dimension of the query and key features;

[0084] Step S15, finally passed through a decoder The generated future multi-view video results are obtained from the diffusion model. The objective function of the denoising network is the same as the objective function of the conditional denoising diffusion model. At the same time, the L2 loss function is used to ensure the consistency between the generated video and the real video.

[0085] Example 2:

[0086] This embodiment provides a multi-view video generation system for autonomous driving scenarios, used to execute the method shown in Embodiment 1. This system instantiates each step of the method shown in Embodiment 1 into specific software modules or hardware units, and can be deployed on servers, workstations, or cloud computing platforms to efficiently generate autonomous driving simulation video data.

[0087] The system includes the following components:

[0088] Encoder ε: This module is implemented by a pre-trained neural network model. Its function is to receive the input sequence of past multi-frame, multi-view images, encode their features, and compress the high-dimensional pixel data into a low-dimensional latent feature representation, which serves as the input z0 to the subsequent diffusion model.

[0089] Diffusion Model: This module is the core generation engine of the system, responsible for performing the forward diffusion process and the backward denoising process. During inference, it mainly runs the backward denoising process, receiving the noisy latent representation z from the encoder. T And gradually remove noise to recover meaningful video content.

[0090] 3D UNet Denoising Network: This network is the specific executor of the denoising step in the diffusion model, and is a three-dimensional U-shaped convolutional neural network. Its core feature lies in the integration of three key attention modules to impose spatiotemporal consistency constraints:

[0091] Cross-view deformable cross-attention module: This module executes steps S4 to S9 as described in Example 1. It receives camera pose information, establishes a three-dimensional spatial relationship through coordinate transformation and projection, and utilizes a deformable cross-attention mechanism to achieve geometric consistency fusion of feature maps from different viewpoints.

[0092] Cross-frame attention module: This module performs steps S10 to S13 as described in Example 1. It utilizes the optical flow prior calculated by the pre-trained optical flow estimation model, aligns the features of consecutive frames, and captures and maintains the coherence of motion between frames through cross-attention.

[0093] Global temporal consistency constraint module: This module executes step S14 as described in Example 1. It uses a cross-attention mechanism to globally associate the current frame with historical frames, ensuring the temporal rationality and coherence of the long sequence generation process.

[0094] In addition, the 3D UNet network also receives conditional information such as text, map, and detection boxes, and uses them as control signals for the generation process through a cross-attention mechanism.

[0095] decoder This module corresponds to the encoder ε and is typically implemented by a pre-trained neural network decoder. Its function is to reconstruct the recovered latent features output by the 3D UNet denoising network into a high-quality, future multi-view video sequence in pixel space.

[0096] In practical deployment, the system should also include a data interface module to receive multi-view image input from the simulation disk or sensors, as well as conditional information such as text and maps. Simultaneously, a control module is responsible for scheduling the orderly operation of the above modules and executing the complete generation process.

[0097] The system's workflow corresponds exactly to the method steps described in Example 1. During runtime, the system acquires input through a data interface, then sequentially encodes features using encoder ε, performs conditional denoising and spatiotemporal consistency feature optimization using a diffusion model and a 3D UNet denoising network integrating three attention modules, and finally passes the input through decoder. The system generates final multi-view videos. It can automatically generate high-fidelity simulation videos that meet geometric and motion consistency requirements, greatly improving the efficiency and quality of autonomous driving test data generation.

[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating multi-view videos of autonomous driving scenarios based on spatiotemporal consistency constraints, characterized in that: In the diffusion-based generation process, spatiotemporal consistency constraints are imposed by integrating the following three attention mechanisms: (1) Cross-view deformable cross attention mechanism based on camera pose: The pixel coordinates on the source view image plane are transformed to the three-dimensional world coordinate system using the camera pose matrix, and then projected onto the target view image plane. Then, the cross attention between the source view and the target view feature maps is calculated to achieve cross-view geometric consistency. (2) Cross-frame cross-attention mechanism based on optical flow prior: using the optical flow prior calculated from the images of the previous and next frames, the feature map of the current frame is aligned with the feature map of the previous frame, and the cross-attention between the two is calculated to achieve motion consistency between adjacent frames. (3) Global temporal consistency constraint based on cross-attention mechanism: Calculate cross-frame attention by using the stitching result of the latent representation of the current frame and the image features of the historical frames to achieve motion coherence in the global temporal range.

2. The method according to claim 1, characterized in that: The method specifically includes the following steps: Step S1: Input past multi-frame multi-view images, use encoder ε to perform feature encoding, and use it as input z0 of the diffusion model; Step S2: The input z0 obtained in step S1 is diffused to obtain the noise z. T ; Step S3, the noise z obtained in step S2 T Input a 3D UNet denoising network and use at least one of text, map, or detection bounding boxes as conditional control for image generation; Step S4, based on the cross-view deformable cross-attention mechanism of camera pose, firstly, the coordinates of the source pixel s are ( , Transforming from the image plane to the three-dimensional world coordinate system, in The corresponding depth value in the feature map ; Step S5, convert the three-dimensional coordinates ( , , Projected onto the target camera On the image plane, where the camera pose matrix is... It contains rotation and translation information, where R is the rotation matrix and b is the translation vector; Step S6, for the source feature map The source pixel s in the target feature map Cross-attention is calculated for the target pixel position e and its k×k neighbors, where the weight matrices for query, key, and value are respectively... , , The query, key, and value are obtained as follows: , , ; Step S7, Sampling position ,in It is relative to the target pixel position The offset; subscript Indicates the relationship between samples within the neighborhood. Indices representing the indices of neighboring points. Represents a k×k neighborhood; Step S8, at the sampling location Bilinear interpolation is performed on K and V to obtain the sampled key and value. , Calculate the cross-attention weights; Step S9, Aggregate value Then perform feature updates. ,in Indicates the cross-attention weights; Step S10: Based on the optical flow prior, a cross-frame cross-attention mechanism is used, given the previous frame image. and the current frame image The optical flow prior is calculated using the pre-trained optical flow estimation model FlowNet; Step S11: Based on the optical flow prior obtained in step S10, the feature map of the current frame is... Aligned to the feature map of the previous frame The aligned feature map is obtained. ; Step S12: Calculate the cross-attention between the current frame and the previous frame using optical flow information; Step S13, perform feature update ; Step S14: Based on the global temporal consistency constraint of the cross-attention mechanism, use the latent representation of the t-th frame. and features of the first frame image Features of the (t-1)th frame image The sum is used to calculate cross-frame attention; Step S15, finally passed through a decoder The generated future multi-view video results are obtained from the diffusion model. The objective function of the denoising network is the same as the objective function of the conditional denoising diffusion model. At the same time, the L2 loss function is used to ensure the consistency between the generated video and the real video.

3. The method according to claim 2, characterized in that: In step S4, the coordinate system transformation formula is: Among them, P ｉ This indicates the pose of the source camera.

4. The method according to claim 2, characterized in that: In step S5, the projection formula is: in, , Represents the unnormalized image coordinates after projection. This represents the homogeneous components after projection. Represents the normalized plane coordinates of the projected image.

5. The method according to claim 2, characterized in that: In step S8, the formula for calculating the cross-attention weights is: 。 6. The method according to claim 2, characterized in that: In step S10, the formula for calculating the optical flow prior is: 。 7. The method according to claim 2, characterized in that: In step S12, the formula for calculating cross-attention is: in, , , These are learnable matrices that map the input to query, key, and value features.

8. The method according to claim 2, characterized in that: In step S14, the formula for calculating cross-frame attention is: in, This indicates the query features of the current frame. , These represent the key and value features of the current frame concatenated with the previous frame, respectively, and [,] denotes the feature concatenation operation. , , It is a learnable matrix that maps the input to query, key, and value features, where d is the output dimension of the query and key features.

9. A multi-view video generation system for autonomous driving scenarios, used to perform the method as described in any one of claims 1 to 8, characterized in that: include: Encoder ε is used to encode features of the input multi-frame, multi-view image; A diffusion model is used to perform the diffusion and denoising processes. The 3D UNet denoising network integrates a cross-view deformable cross attention module, a cross-frame cross attention module, and a global temporal consistency constraint module. decoder It is used to output the generated future multi-view video.

Images