A Driving Video Generation Method Based on Spatiotemporal Factorization Architecture and Hybrid Modulation

Abstract

This invention relates to the field of autonomous driving technology and provides a driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation. The method includes steps such as: obtaining a set of conditions corresponding to each time frame and each viewpoint, obtaining diffused input latent variables, and obtaining spatiotemporal fusion features, ultimately obtaining a future multi-view driving video sequence. The advantages of this application are that it unifies image information, environmental structure information, and vehicle control intent into a single generation framework, improving the model's ability to express complex driving environments; it enhances the controllability of the generation process while reducing computational complexity; and it improves the stability of the generated results in terms of spatial structure, temporal continuity, and multi-view consistency, providing more reliable scene information support for applications such as environmental prediction, behavior planning, decision support, and simulation training in autonomous driving systems.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and particularly relates to a method for generating driving videos based on a spatiotemporal factorization architecture and hybrid modulation. Background Technology

[0002] With the rapid development of autonomous driving technology, the ability of vehicles to perceive, understand, and predict the surrounding traffic environment has gradually become one of the key supporting technologies in autonomous driving systems. In practical applications, autonomous driving systems typically rely on multiple sensors to acquire environmental information. Among these, multi-camera vision systems, which involve deploying multiple cameras around the vehicle, have become the mainstream environmental perception solution due to their advantages such as low cost, rich information, and flexible deployment.

[0003] In recent years, time-series-based multi-frame visual modeling methods and generative model-based video prediction and scene generation methods have been gradually applied to the field of autonomous driving, enabling the generation and prediction of future scenes. However, existing autonomous driving scene generation and video prediction methods still have many shortcomings in terms of multi-view, multi-time-series, and multi-modal information fusion.

[0004] Existing multi-camera data processing methods mostly employ feature stitching or viewpoint-independent modeling, lacking effective cross-viewpoint feature interaction mechanisms. This leads to structural inconsistencies and semantic deviations in the generated results across different viewpoints, making it difficult to accurately depict the continuous changes in the scene and resulting in temporal instability. Regarding multimodal information fusion, multi-source data such as environmental occupancy information, depth information, and vehicle control information cannot be effectively coupled, resulting in insufficient environmental structural constraints. While diffusion models have demonstrated excellent performance in image and video generation tasks in recent years, their application in multi-view scenarios for autonomous driving remains in the exploratory stage. Existing diffusion model-based methods are generally designed and developed for single-view or single-modal data. When handling multi-view inputs and multiple constraints, they lack a unified modeling framework, making it difficult to balance computational efficiency and consistency constraints of multi-source information while ensuring generation quality.

[0005] In summary, existing technologies still have many shortcomings in handling multi-perspective information fusion, including insufficient correlation modeling and inadequate utilization of temporal dynamic information. Further improvements are needed to enhance the accuracy and stability of model predictions. Summary of the Invention

[0006] The purpose of this application is to provide a driving video generation method based on spatiotemporal factorization architecture and hybrid modulation, which aims to solve the problems of insufficient correlation modeling and inadequate utilization of temporal dynamic information in the existing technology for processing multi-view information fusion.

[0007] This application provides a method for generating driving videos based on a spatiotemporal factorization architecture and hybrid modulation, the method comprising:

[0008] The historical multi-view driving image sequence and the environmental condition information corresponding to the historical multi-view driving image sequence are aligned to obtain the condition set corresponding to each time frame and each view.

[0009] The historical multi-view driving image sequence is input into a latent variable encoder for mapping to obtain the corresponding latent space representation; the latent space representation is then subjected to noise processing to obtain diffused input latent variables.

[0010] Spatiotemporal joint modeling is performed on the diffused input latent variables to obtain spatial branch features and temporal branch features; the spatial branch features and temporal branch features are fused to obtain spatiotemporal fused features;

[0011] Based on the correlation between different camera perspectives, cross-view feature interaction processing is performed on the spatiotemporal fusion features to obtain multi-view enhanced features.

[0012] The multi-view enhancement features are used as input to a denoising network. Denoising generation is performed based on the denoising network under the constraints of the condition set to obtain the latent variable results corresponding to the target future video sequence. The latent variable results are then decoded to obtain the future multi-view driving video sequence.

[0013] Another objective of this application is to provide a driving video generation device based on a spatiotemporal factorization architecture and hybrid modulation, the device comprising:

[0014] The condition set acquisition unit is used to align the historical multi-view driving image sequence with the environmental condition information corresponding to the historical multi-view driving image sequence to obtain the condition set corresponding to each time frame and each view.

[0015] The diffusion input latent variable acquisition unit is used to input the historical multi-view driving image sequence into the latent variable encoder for mapping to obtain the corresponding latent space representation; and to add noise to the latent space representation to obtain the diffusion input latent variable.

[0016] The spatiotemporal fusion feature acquisition unit is used to perform spatiotemporal joint modeling on the diffused input latent variables to obtain spatial branch features and temporal branch features; and to fuse the spatial branch features and temporal branch features to obtain spatiotemporal fusion features.

[0017] The multi-view enhanced feature acquisition unit is used to perform cross-view feature interaction processing on the spatiotemporal fusion features based on the correlation between different camera viewpoints to obtain multi-view enhanced features.

[0018] The video sequence acquisition unit is used to take the multi-view enhancement features as input to the denoising network, perform denoising generation based on the denoising network under the constraints of the condition set, and obtain the latent variable results corresponding to the target future video sequence; and decode the latent variable results to obtain the future multi-view driving video sequence.

[0019] Another objective of this application is to provide a driving video generation system based on a spatiotemporal factorization architecture and hybrid modulation. The system includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation as described above.

[0020] The driving video generation method based on spatiotemporal factorization architecture and hybrid modulation provided in this application has the following key advantages: By constructing condition sets and aligning multiple modalities, it can integrate image information, environmental structure information, and vehicle control intent into a single generation framework, enhancing the model's ability to express complex driving environments; through latent space diffusion modeling and conditional constraint noise processing, it reduces computational complexity while enhancing the controllability of the generation process; through spatiotemporal joint modeling and cross-view feature interaction, it simultaneously models road structure, traffic participant distribution, and continuous motion trends, improving the stability of the generated results in terms of spatial structure, temporal continuity, and multi-view consistency; and through conditional constraint denoising generation and decoding recovery, it can obtain future multi-view video results that better conform to the real driving evolution laws, providing more reliable scene information support for applications such as environmental prediction, behavior planning, decision support, and simulation training in autonomous driving systems. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the steps of a driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation, provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram illustrating the video prediction effect of a driving scene provided in an embodiment of this application;

[0023] Figure 3 A structural block diagram of a driving video generation device based on spatiotemporal factorization architecture and hybrid modulation provided in an embodiment of this application;

[0024] Figure 4 This is a block diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise stated, these elements are not limited by these terms. These terms are used only to distinguish the first unit or module from another unit or module. For example, without departing from the scope of this application, the first script may be referred to as the second script, and similarly, the second script may be referred to as the first script.

[0027] Figure 1 This diagram illustrates the steps of a driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation, as provided in this application. This method targets the prediction and generation of future driving scenes in autonomous driving scenarios. It uses historical multi-view driving image sequences as the primary input, and occupancy information and vehicle motion control information as auxiliary conditions. Through several steps, including latent space diffusion modeling, spatiotemporal joint feature modeling, multi-view feature interaction, and conditional constraint denoising generation, it ultimately decodes and outputs a multi-view driving video sequence for future moments. This application can simultaneously utilize the spatial complementarity between multiple cameras, the dynamic evolution relationship between consecutive time frames, and environmental structure information, thereby improving the spatial consistency, temporal continuity, and detail realism of the generated results.

[0028] This method may specifically include the following steps:

[0029] Step S10: Align the historical multi-view driving image sequence with the environmental condition information corresponding to the historical multi-view driving image sequence to obtain a set of conditions corresponding to each time frame and each viewpoint.

[0030] In this embodiment, data from different cameras, different time frames, and different conditions are first organized into a unified data organization framework to ensure that the subsequent network can process image information and condition information from the same moment and perspective. Multiple perspectives can refer to camera views from multiple directions, such as the front camera, left front camera, right front camera, and left rear camera. Historical multi-perspective driving image sequences are sequences of multiple images captured by the vehicle at several consecutive historical moments. For example, when a vehicle is traveling at a certain speed along a city road, the most recent eight frames of images from six cameras are collected, serving as the historical observation basis for future scene generation. Environmental condition information can include at least condition frame information, occupancy semantic information, occupancy depth information, and vehicle motion control information. Occupancy semantic information refers to the semantic category description of different spatial regions in the scene, such as whether certain grid regions belong to roads, vehicles, pedestrians, green belts, or buildings. Occupancy depth information refers to the depth distribution or geometric distance information of each spatial region in the scene, used to reflect the distance of objects and spatial hierarchy. Vehicle motion control information can include the vehicle's speed, trajectory, direction angle, target point, and control commands, used to reflect the current vehicle's motion state and future control intentions. By standardizing the format, rearranging the dimensions, unfolding the information frame-by-frame and view-by-view, and copying and aligning it time-step, a set of conditions corresponding one-to-one with each time frame and view can be obtained. This allows the subsequent denoising network to simultaneously access the matching environmental structure information and control information when processing an image from a specific frame and view. For example, when a vehicle is preparing to change lanes to the right, the image content corresponding to the right-side camera, the occupancy semantics and depth information of the right-side region, and the vehicle's direction angle and trajectory information are all sent to the same spatiotemporal processing link, enabling the model to more accurately predict future changes in the right-side view.

[0031] Step S20: Input the historical multi-view driving image sequence into the latent variable encoder for mapping to obtain the corresponding latent space representation; add noise to the latent space representation to obtain the diffused input latent variable.

[0032] In this embodiment, high-dimensional image data is first compressed into a latent space, and then diffusion modeling is performed in the latent space to reduce the computational complexity of generating directly in the original image space and enhance modeling efficiency. The latent space representation refers to the compact feature representation extracted from the original image by the encoder. This representation retains the main semantic and structural information of the image, while having lower dimensionality and higher expressive efficiency compared to the original image. Based on this, this application adds noise to the latent representation in the latent space to construct diffusion input latent variables, enabling the model to learn to gradually recover the target scene from the noisy latent representation. The advantage is that the model can not only learn general denoising recovery but also guide future generated content through conditional frames, occupancy information, and motion control information. For example, if the historical input includes the current road image ahead, the vehicle's upcoming left turn direction control information, and the occupancy semantics and depth information of the left side of the road, these conditions will jointly constrain the model when constructing the diffusion input latent variables, making the model more inclined to recover the scene the vehicle should see after turning left into the new road in subsequent generation.

[0033] Step S30: Perform spatiotemporal joint modeling on the diffused input latent variables to obtain spatial branch features and temporal branch features; fuse the spatial branch features and temporal branch features to obtain spatiotemporal fusion features.

[0034] In this embodiment, the aim is to simultaneously model the spatial structural relationships within an image and the dynamic changes across time frames. Specifically, the spatial branch mainly models features within a single frame, used to depict spatial structures such as road boundaries, vehicle outlines, traffic signs, lane lines, and building edges; the temporal branch models feature changes at the same location across different time frames, used to depict dynamic processes such as vehicle movement, pedestrian movement, vehicle turning, and traffic light status changes. By fusing the spatial and temporal branch features, a spatiotemporal fusion feature that simultaneously considers spatial structure and temporal evolution can be obtained. For example, when a vehicle approaches an intersection, the positional distribution of the stop line, zebra crossing, and vehicles ahead can be seen in a single frame image; this represents spatial information. However, the changes in the speed of vehicles ahead, pedestrians moving from the roadside to the zebra crossing, and traffic signals changing from green to yellow over several consecutive frames represent temporal information. Relying solely on the spatial branch easily yields static results lacking dynamic continuity; relying solely on the temporal branch easily overlooks the spatial layout of roads and obstacles. Therefore, through spatiotemporal joint modeling, it is possible to more accurately predict which objects will be in what positions and how they will change in the future scene. The spatiotemporal factorization architecture can simultaneously capture the spatial structure information of an image and the dynamic changes across time frames, thereby improving the temporal consistency and structural stability of the generated video between consecutive frames.

[0035] Step S40: Based on the correlation between different camera viewpoints, perform cross-view feature interaction processing on the spatiotemporal fusion features to obtain multi-view enhanced features.

[0036] In this embodiment, the complementary observation capabilities of different perspectives within a multi-camera system on the same traffic environment are further utilized to enhance the model's understanding of the overall scene. Autonomous vehicles are typically equipped with cameras in multiple directions simultaneously. Although different cameras see different areas, they collectively describe the same spatiotemporal environment. Taking a vehicle turning left at an intersection as an example, the front-view camera observes the road ahead and traffic light status, the left front camera observes the target lane for the left turn and oncoming traffic, and the left rear camera may observe following vehicles and blind spots. If each perspective is modeled independently, problems such as inconsistent target positions, unreasonable restoration of occluded areas, or discontinuous boundary structures can easily arise. This embodiment addresses these issues through cross-view feature interaction processing, enabling features from a particular perspective to not only rely on its own observations but also absorb supplementary information from adjacent perspectives, thereby mitigating problems such as single-view occlusion, information loss, and unclear depth.

[0037] For example, when a vehicle is partially obscured by a large vehicle in the right front camera view, the right-side camera or the front-view camera may still retain some of the vehicle's outline and motion information. Through cross-view feature fusion, the model can more completely infer the vehicle's true position and appearance at future moments, thereby improving the consistency and reasonableness of the generated results across multiple perspectives. By introducing a multi-view interaction mechanism, the video generation quality in complex traffic scenarios can be improved.

[0038] Step S50: The multi-view enhancement features are used as input to the denoising network. Denoising generation is performed based on the denoising network under the constraints of the condition set to obtain the latent variable results corresponding to the target future video sequence. The latent variable results are decoded to obtain the future multi-view driving video sequence.

[0039] As an embodiment of this application, preferably, the multi-view enhancement features can be first projected and aligned to obtain latent space guidance features corresponding to the diffusion input latent variables in terms of spatial size and number of channels; the latent space guidance features are then fused with the diffusion input latent variables to obtain the enhanced diffusion input latent variables as input.

[0040] In this embodiment, under the constraints of a set of conditions, a denoising network is used to progressively denoise and generate latent variable results corresponding to the target future video sequence. These results are then decoded by a decoder to obtain the future multi-view driving video sequence. The guidance features can simultaneously contain spatial structure information from multiple camera perspectives, as well as semantic information supplemented across perspectives. During the progressive denoising process, the network receives noisy latent variables as input at each time step.

[0041] In this embodiment, the denoising scaling coefficients may include skip connection coefficients, network input scaling coefficients, network output scaling coefficients, and noise intensity coding coefficients. These coefficients are used to normalize and adjust the network input and output under different noise intensity conditions, thereby improving the stability of the denoising process. Specifically, the noise intensity coding coefficients encode the noise level corresponding to the current diffusion time step into a network-aware feature, enabling the denoising network to adaptively adjust its recovery strategy according to different noise stages.

[0042] In this application, occupancy information can refer to a way of regularly expressing the spatial occupancy status of the traffic environment, that is, dividing the environment into multiple spatial units and using semantic categories and depth information to describe whether there are targets such as roads, vehicles, pedestrians, and obstacles in each unit and their spatial distribution; target point can refer to the location that the vehicle plans to reach in the future, such as a turning point on the navigation path or the next key position in the planned trajectory; control command can refer to control instructions at the vehicle execution level, such as acceleration, deceleration, braking, left turn, right turn, or lane keeping.

[0043] Those skilled in the art will recognize that in multi-camera autonomous driving systems, there are clear spatial relationships between different cameras. For example, a forward-looking camera can acquire information about the road and vehicles ahead, while a side-looking camera can provide traffic information about adjacent lanes or blind spots. These different perspectives collectively describe the same traffic environment. However, the researchers in this application found that many existing methods often employ simple feature stitching or independent processing when handling multi-view data, making it difficult to fully utilize the spatial relationships between different perspectives. This affects the model's ability to understand the overall traffic scene structure, and inconsistencies in the generated results are also prone to occur between different perspectives.

[0044] Secondly, in terms of time series modeling, while some existing methods can handle multi-frame image input, they lack an effective mechanism for exchanging temporal information, making it difficult for the model to fully utilize the dynamic changes between consecutive frames. When the model generates future scenes, if it lacks effective modeling of temporal changes, it is prone to problems such as unreasonable motion changes or temporal discontinuities, thereby affecting the stability and realism of the generated results.

[0045] Furthermore, environmental structure information is crucial for understanding traffic scenarios during autonomous driving environment modeling. Recent occupancy representation methods divide the traffic environment into regular grids, representing whether each grid cell is occupied by an object and its semantic category, thus describing road structure, vehicle distribution, and the positional relationships of other traffic participants in a unified way. By combining semantic occupancy information and depth information, the spatial structure of the traffic environment can be reflected more accurately. However, in existing autonomous driving scene generation methods, occupancy information is typically used only as an auxiliary environmental representation, lacking an effective fusion mechanism with multi-view visual sequence modeling, resulting in the underutilization of environmental structure information in the scene generation process.

[0046] The advantages of this application are as follows: by constructing condition sets and aligning multimodal data, image information, environmental structure information, and vehicle control intent can be uniformly incorporated into the same generation framework, improving the model's ability to express complex driving environments; by latent space diffusion modeling and conditional constraint noise processing, the controllability of the generation process is enhanced while reducing computational complexity; by spatiotemporal joint modeling and cross-view feature interaction, road structure, traffic participant distribution, and continuous motion trends are modeled simultaneously, improving the stability of the generated results in terms of spatial structure, temporal continuity, and multi-view consistency; and by denoising generation and decoding recovery based on conditional constraints, future multi-view video results that better conform to the evolution of real driving can be obtained, providing more reliable scene information support for applications such as environmental prediction, behavior planning, decision support, and simulation training in autonomous driving systems.

[0047] In a preferred embodiment, the method for aligning historical multi-view driving image sequences with environmental condition information corresponding to the historical multi-view driving image sequences to obtain condition sets corresponding to each time frame and each viewpoint is as follows:

[0048] Acquire driving scene images of the vehicle captured by multiple cameras at consecutive historical moments to construct a historical multi-view driving image sequence. ,in:

[0049] ;

[0050] in, For the real number field, Indicates batch size, Indicates the number of camera angles. Indicates the number of time frames. Indicates the number of channels. Indicates the image height. Indicates the image width; reconstructs the historical multi-view driving image sequence into a frame-by-frame, view-by-view format. As a frame-by-frame, view-by-view image input benchmark:

[0051] ;

[0052] Acquire the historical multi-view driving image sequence The corresponding occupancy semantic information and occupancy depth information are represented as occupancy semantic tensor and occupancy depth tensor, respectively; the occupancy semantic tensor and the occupancy depth tensor are combined in the channel dimension to obtain the occupancy condition tensor. :

[0053] ;

[0054] in, This indicates the number of channels that are currently in use. Indicates the height of the occupied condition. The width of the occupancy condition is represented; the occupancy condition tensor is reconstructed into an occupancy condition representation. This is to ensure that the occupancy semantic information and occupancy depth information are mapped to the image input at each time frame and viewpoint:

[0055] ;

[0056] Obtain the conditional frame information corresponding to the historical multi-view driving image sequence. and vehicle motion control information, the vehicle motion control information including speed trajectory Direction angle Target point and control commands The condition frame information is expanded to obtain the condition frame representation corresponding to each time step. This ensures that each time step has corresponding reference image conditions; the vehicle motion control information is encoded and copied to each time frame and each viewpoint, resulting in a frame-by-frame, view-by-view format. Control condition representation aligned in the sample dimension Based on the conditional frame representation The occupancy condition indicates and the control conditions indicated This yields the condition set corresponding to each time frame and each viewpoint. :

[0057] .

[0058] In this embodiment, driving scene images of the vehicle captured by multiple cameras at consecutive historical moments are first obtained, and a multi-view image sequence including batch size, viewpoint, and time dimensions is constructed. Then the multi-view image sequence is rearranged frame by frame and view by view. This ensures that each sample corresponds to a specific combination of time frame and viewpoint. The image input benchmark is presented in frame-by-frame, viewpoint-by-view format. Used to provide a unified image sample index for subsequent latent space modeling and denoising generation. Occupancy conditional tensor It can describe the semantic categories and geometric distribution of various spatial locations in a traffic environment. The occupancy condition tensor is reconstructed in the same frame-by-frame, view-by-view manner as the image sequence to obtain the occupancy condition representation. This ensures that each frame and each viewpoint of the image input corresponds to its matching environmental structure information. (Regarding conditional frame information...) By performing operations such as temporal copying or expansion, the conditional frame representation corresponding to each time step is obtained. Conditional frame representation This provides reference image constraints for each time step, ensuring that each time step can access unified reference image information during subsequent generation processes, providing visual constraints or appearance priors. For vehicle motion control information, it is converted into a feature representation that the network can process through encoding operations, and then copied to each time frame and each viewpoint to obtain the control condition representation. This ensures that it maintains consistency with the image input in the sample dimension. Control condition representation. This can be used to characterize the vehicle's motion state and control intent in the current scenario. Finally, all the above-mentioned conditional information is integrated to construct a condition set. This condition set corresponds to the frame-by-frame, view-by-view image input in the sample dimension. One-to-one correspondence. Through this integration process, subsequent models can simultaneously acquire corresponding environmental structure information, reference image information, and vehicle motion control information when processing input from any time frame or any viewpoint, thereby achieving collaborative constraints of multimodal information.

[0059] In this embodiment, historical multi-view driving image sequences can reflect visual observation information of the vehicle's surrounding environment, while occupancy semantic information and occupancy depth information can reflect the spatial occupancy status and geometric distribution of the environment. Vehicle motion control information reflects the vehicle's future action intentions. By unifying and aligning these three types of information and constructing a condition set, the generation results of subsequent generative models can be made consistent with the environmental structure and vehicle behavior, making them more suitable for applications such as future scene generation and environmental prediction in autonomous driving scenarios.

[0060] In a preferred embodiment, the method for inputting the historical multi-view driving image sequence into a latent variable encoder for mapping to obtain the corresponding latent space representation is as follows:

[0061] The historical multi-view driving image sequence Input latent variable encoder By performing mapping, the corresponding initial representation of the latent space is obtained. :

[0062] ;

[0063] in, Indicates a latent variable encoder;

[0064] Initial representation of the latent space Introducing scaling factor Scaling is performed to obtain the latent space representation. :

[0065] .

[0066] In this embodiment of the application, the initial representation of the latent space The latent space representation reflects the latent features of the scene extracted by the encoder. This serves as the actual input used in subsequent diffusion modeling. By encoding first and then scaling, the complexity of spatial modeling of the original image can be reduced while preserving the main information of the scene, and the stability and efficiency of the subsequent generation process can be improved. This is achieved through scaling factors. The obtained latent space representation This can be used as input for subsequent diffusion and noise addition processing, enabling the model to learn the generation rules of future scenes in a lower-dimensional and more stable feature space. Compared to existing mainstream methods, this embodiment significantly reduces the computational burden of directly performing diffusion modeling on high-dimensional images, improving the accuracy of the generated results in terms of scene structure and semantic representation.

[0067] In a preferred embodiment, the method for adding noise to the latent space representation to obtain the diffused input latent variables is as follows:

[0068] Representation of latent space Sampling noise intensity The noise intensity satisfy:

[0069] ;

[0070] Right now:

[0071] ;

[0072] in, For parameters, The parameter representing the mean of the noise intensity sampling distribution. The parameter representing the standard deviation of the noise intensity sampling distribution;

[0073] Based on noise intensity Adding noise to the latent space representation yields the diffused input latent variables:

[0074] ;

[0075] in, It is Gaussian noise. This represents element-wise multiplication. This indicates the diffusion of latent input variables. It is represented by latent space.

[0076] In this embodiment, parameters The noise intensity can be sampled using the above method, where the noise follows a standard normal distribution. This ensures the noise intensity remains positive and allows different samples to participate in training under different noise levels, enhancing the model's adaptability to varying disturbance intensities. Gaussian noise It follows a standard normal distribution.

[0077] In a preferred embodiment, spatiotemporal joint modeling is performed on the diffused input latent variables to obtain spatial branch features and temporal branch features; the method for fusing the spatial branch features and temporal branch features to obtain spatiotemporal fused features is as follows:

[0078] The diffusion input latent variable Input feature map for spatiotemporal joint modeling :

[0079] ;

[0080] in, This represents the number of channels in the input feature map. Indicates the height of the input feature map. Indicates the width of the input feature map; Represents the real number field. Indicates batch size, Indicates the number of camera angles. Indicates the number of time frames;

[0081] Flattening the input feature map of each frame into a set of spatial tokens yields:

[0082] ;

[0083] in, For input feature map The spatial token sequence obtained by flattening Indicates the feature dimension of the token;

[0084] By performing correlation modeling on features at various locations within the same frame and combining this with the condition set to perform condition-guided updates, spatial branch features are obtained. ;

[0085] Reorganizing the features of the same spatial location at different time steps into a time series format yields:

[0086] ;

[0087] in, This refers to the reorganized sequence features obtained by reorganizing features from the same spatial location at different time steps;

[0088] Introducing time and location embedding , get in Add time and location embedding on the basis The time series features obtained afterwards :

[0089] ;

[0090] For full-time modeling, time branch features are obtained. :

[0091] ;

[0092] The spatial branching feature and time branch features Weighted fusion is performed to obtain spatiotemporal fusion features. :

[0093] ;

[0094] in, Represents the fusion coefficient. This represents the function for modeling temporal attention.

[0095] In this embodiment, a spatial token refers to representing different spatial locations in a single-frame feature map as feature units that can be processed by the network, facilitating subsequent modeling of the relationships between different locations within the same frame. For example, in a driving image frame, lane line regions, vehicle regions, pedestrian regions, and road boundary regions correspond to different spatial locations. These locations, after being flattened, form a set of spatial tokens, which the network can use to learn the spatial structural relationships between locations within the same frame. Subsequently, the features of each location within the same frame are correlated and modeled, and condition-guided updates are performed in conjunction with the aforementioned set of conditions to obtain spatial branch features. Spatial branching not only considers the relationships between different regions within an image, but also incorporates environmental structure information, reference image information, and vehicle control information through condition sets, making the single-frame feature representation more consistent with the spatial layout of real driving scenarios. Temporal location embedding can be used to distinguish features at different time steps, avoiding temporal frame confusion.

[0096] This embodiment models the features corresponding to the diffused input latent variables simultaneously along both spatial and temporal dimensions. The advantages are: spatial branching modeling fully characterizes the spatial structural relationships within the same frame, such as road boundaries, vehicle distribution, traffic signs, and pedestrian positions, thus improving the structural rationality of the generated results at the single-frame level; temporal branching modeling effectively captures the motion trends, target displacements, and scene changes between consecutive frames, improving the continuity and stability of the generated results in the temporal dimension; the introduction of temporal position embedding enables the network to distinguish the information sources at different time steps, avoiding confusion between features from different time frames during modeling; and the use of fusion coefficients... By weighted fusion of spatial and temporal branch features, the contributions of spatial and temporal information can be flexibly balanced according to task requirements, resulting in more comprehensive spatiotemporal fusion features.

[0097] In a preferred embodiment, the method for performing cross-view feature interaction processing on the spatiotemporal fusion features based on the correlation between different camera viewpoints to obtain multi-view enhanced features is as follows:

[0098] Let the first The spatiotemporal fusion characteristics from multiple perspectives are Its adjacent view set is Then it comes from the adjacent perspective. Additional information for:

[0099] ;

[0100] in, Represents a multi-view feature interaction function; For the first The spatiotemporal fusion characteristics from multiple perspectives;

[0101] Aggregate the supplementary information from all adjacent viewpoints to obtain the first... Multi-view enhancement from each perspective :

[0102] ;

[0103] The multi-view enhancement amount Inject the first through connection mapping Spatiotemporal fusion features from multiple perspectives yield multi-view enhanced features. :

[0104] ;

[0105] in, and For trainable parameters, Indicates the first Multi-view enhancement features corresponding to each viewpoint.

[0106] In the embodiments of this application, the final multi-view enhancement features are This approach not only incorporates the spatiotemporal information of the current viewpoint but also integrates supplementary observation information from neighboring viewpoints, thus providing a more comprehensive representation of the vehicle's surrounding environment. This embodiment utilizes neighboring viewpoints to supplement the current viewpoint, mitigating information loss caused by single-view observations and improving the completeness of feature representation. Aggregating supplementary information from multiple neighboring viewpoints integrates multi-path observation results, reducing the impact of single-viewpoint errors or local noise on the current viewpoint features and improving the robustness of feature representation. By injecting enhancements into the current viewpoint features through connection mapping, information from different viewpoints is fused within a unified feature space, enhancing structural and semantic consistency between different viewpoints. The significance of this method lies in the fact that the surrounding environment of autonomous vehicles is typically observed jointly by multiple cameras, and any single viewpoint may be affected by factors such as occlusion, limited field of view, or unclear depth information. For example, in a front view image, a vehicle in front may be partially occluded by a large vehicle, but its outline and positional changes can still be observed in the left or right front viewpoints. Through cross-viewpoint feature interaction, supplementary information from these neighboring viewpoints can be introduced into the current viewpoint, thereby improving the current viewpoint features' ability to express occluded targets, boundary regions, and complex scene structures.

[0107] In a preferred embodiment, the model models causal time, allowing access to information from both the current and past moments only at the current time. For sparse causal patterns, the attention scope can be further restricted to a local time window. If the window radius is... Then the first Visible set of moments for:

[0108]

[0109] Visible sets The time characteristics are represented as follows:

[0110] .

[0111] in, This represents the attention calculation function; Indicates the first The query vector corresponding to a given time step is used to represent the information that needs to be retrieved from the historical time steps at the current time step; Represents the visible set The set of key vectors formed by the features corresponding to each time step within the time frame is used to represent the information content that can be matched at each historical moment; Represents the visible set The set of value vectors corresponding to the features at each time step within the time frame is used to provide information on the actual content involved in the weighted aggregation after matching is completed; Indicates the first The temporal feature representation obtained after sparse causal temporal attention modeling. It is a variable. In other words, it is determined by querying the vector at the current moment. With the set of key vectors within the visible historical range Perform matching, and based on the matching results, analyze the corresponding set of value vectors. We perform weighted aggregation to obtain the temporal feature representation of the current moment. .

[0112] In this embodiment, for causal time modeling, only the current moment is allowed to access information from the current moment and moments before it, so as to ensure that the time modeling process conforms to the actual temporal evolution direction and avoid the leakage of information from subsequent moments into the feature calculation of the current moment.

[0113] In a preferred embodiment, the method for obtaining the latent variable results corresponding to the target future video sequence by performing denoising generation based on a denoising network under the constraints of a set of conditions is as follows:

[0114] Construct denoising scaling factors, which include skip connection factors. Scaling factor of network prediction output Scaling factor for network input and noise intensity coding coefficients ;

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] in, Indicates noise intensity;

[0120] The latent variables of the diffusion input and the output of the denoising network are reparameterized based on the denoising scaling factor.

[0121] Based on the constraints of the denoising scaling coefficients and the condition set, the output result of the denoising network is obtained. :

[0122] ;

[0123] in, This indicates the diffusion of latent input variables. This represents a denoising network with conditional constraints. Represents a set of conditions. For mask;

[0124] Based on the output of the denoising network, progressive denoising is performed to generate the latent variables for the target's future time steps. .

[0125] In this embodiment, the denoising scaling coefficients are used to reparameterize the diffused input latent variables and the denoising network output, enabling the denoising network to maintain a stable input-output scale under different noise levels. Then, under the joint constraints of the condition set and the mask, the denoising network with conditional constraints is applied. The output of the current denoising step is calculated. The condition set provides the denoising network with reference image conditions, occupancy semantic conditions, occupancy depth conditions, and motion control conditions. The mask constrains the preset condition preservation positions, enhancing the controllability and stability of the denoising generation process. Finally, based on the output of the current denoising step, the latent variables are progressively denoised and updated until a preset termination denoising step is reached, thus obtaining the latent variable results for the target's future time. Through this process, stable latent space denoising modeling can be achieved under different noise intensities, and the latent variable representation corresponding to the target's future video sequence can be progressively recovered under the guidance of the condition set.

[0126] In a preferred embodiment, the method for decoding the latent variable results to obtain the future multi-view driving video sequence is as follows: [The method involves] decoding the latent variable results of the target at future moments. Decoding was performed to obtain a future multi-view driving video sequence. :

[0127] ;

[0128] in, Indicates decoder, This represents the scaling factor.

[0129] In this embodiment, the latent variable results are inversely scaled using a scaling factor to eliminate the scale transformation effect introduced during the latent space encoding process. The inversely scaled latent variables are then input into the decoder for decoding, resulting in a future multi-view driving video sequence. The decoder is used to restore the feature representations in the latent space to the corresponding multi-view image content, thereby outputting a future multi-view driving video sequence that can be used for subsequent scene prediction, video generation, or decision support. Through this decoding process, the latent variable results that have undergone spatiotemporal modeling and conditional constraint generation in the latent space can be effectively restored into a future driving scene video output with actual visual significance.

[0130] This application, based on the existing video generation network structure, introduces environmental occupancy information as a conditional input, integrating semantic occupancy information and depth information into the scene generation model, thereby enhancing the model's ability to understand the spatial structure of the traffic environment. Simultaneously, this application improves the feature interaction module in the network, enabling the model to uniformly model multi-camera image sequences in both temporal and spatial dimensions. By introducing a temporal feature interaction mechanism, the model can fully utilize dynamic information between consecutive frames, thereby improving the continuity and stability of the generated scene in the temporal dimension. Furthermore, this application designs a cross-view feature fusion mechanism, allowing environmental information from different camera perspectives to be shared and fused at the feature level, thereby improving the model's ability to express the overall traffic scene structure.

[0131] Compared with existing autonomous driving scenario modeling methods, this application introduces environmental occupancy information and combines it with a multi-view spatiotemporal feature fusion mechanism, enabling the model to simultaneously model the spatial structural relationships and temporal dynamic changes of the traffic environment, thereby significantly improving the stability and consistency of autonomous driving scenario generation. Simultaneously, this method can fully utilize multi-camera visual data and environmental occupancy information, enhancing the model's ability to represent complex traffic environments. This provides more reliable scenario information support for environmental prediction, behavior planning, and safety decision-making in autonomous driving systems, demonstrating significant practical value and application prospects.

[0132] In a preferred embodiment, the method for obtaining the latent variable results corresponding to the target future video sequence by performing denoising generation based on a denoising network under the constraints of the set of conditions further includes:

[0133] Constructing a noise weighting function :

[0134] ;

[0135] in, Indicates noise intensity. This represents the standard deviation parameter of the data;

[0136] Based on the noise weighting function, a main loss is constructed to constrain the consistency between the predicted latent variable and the target latent variable. :

[0137] ;

[0138] in, This represents the output of the denoising network. Representing the latent space representation, This represents element-wise multiplication;

[0139] Construct target frame difference :

[0140] ;

[0141] Constructing Predictive Frame Difference :

[0142] ;

[0143] Construct a timing consistency loss based on the target frame difference and the predicted frame difference. :

[0144] ;

[0145] in, Indicates the number of time frames;

[0146] Let the high-frequency filtering operator be A high-frequency loss function is constructed based on the high-frequency filtering operator. :

[0147] ;

[0148] A total loss function is constructed based on the main loss, time consistency loss, and high-frequency loss. :

[0149] ;

[0150] in, and These are weight hyperparameters;

[0151] Based on the total loss function, the optimizer performs backpropagation updates on the occupancy condition encoding branch parameters and the spatiotemporal fusion branch parameters until the training rounds reach the preset stopping condition.

[0152] In this embodiment, the occupancy condition encoding branch can refer to the branch unit of the model that performs related calculations on the occupancy condition tensor, etc., as mentioned above, and the spatiotemporal fusion branch can refer to the branch unit of the model used for spatiotemporal fusion calculations. The target latent variable can refer to the latent space representation obtained by encoding the real future video sequence corresponding to the historical multi-view driving image sequence in the training samples through the latent variable encoder and scaling. In this embodiment, multiple loss functions are constructed based on the output results of the denoising network during the training phase to jointly constrain the predicted latent variable and the target latent variable. Based on the total loss function, the relevant network parameters are updated by backpropagation through the optimizer, so that the denoising network and related branches gradually converge during repeated iterative training, and finally a generative model that can stably generate future multi-view driving video sequences is obtained. This step can be used to provide training objectives and parameter update basis for the aforementioned denoising generation steps. By adopting the method of joint optimization of main loss and additional loss, after the loss calculation is completed, the parameters of the generative network, the occupancy condition encoding branch, and the spatiotemporal interaction module are updated by backpropagation through the optimizer until the trained generative model is output.

[0153] In this embodiment, the noise weighting function is first constructed. This is used to weight and adjust the prediction error under different noise levels according to the current noise intensity, thereby improving the model's training stability and adaptability under various noise conditions. The main loss is constructed based on the noise weight function. Used to constrain the output of the denoising network Representation of the target latent space The consistency between these variables enables the network to learn the basic mapping relationship to recover the target latent variable from the noisy latent variable. To enhance the continuity of the generated results in the temporal dimension, a target frame difference is constructed. Difference between predicted and predicted frames Based on these two, a time-series consistency loss is constructed. This loss term ensures that the changing trends between adjacent time frames remain consistent, making the generated results more consistent with the dynamic evolution patterns in real driving scenarios and reducing jumps, discontinuities, or motion distortions in the time dimension. The preset stopping condition refers to outputting the trained generative model when the training epochs reach a preset maximum number of epochs, or when the video generation quality index on the validation set reaches a preset threshold. This is based on a high-frequency filtering operator. Constructed high-frequency loss Used to constrain the consistency between the predicted and target results in high-frequency information, improving the clarity of image edges, target contours, and detailed areas. Weight hyperparameters and This is used to adjust the contribution ratio of temporal consistency loss and high-frequency loss to the overall optimization objective. Based on the total loss function, the parameters of each module unit in the entire network, including but not limited to the parameters of the occupancy conditional encoding branch and the spatiotemporal fusion branch, are updated until a preset termination condition, such as the upper limit of the training rounds, is reached.

[0154] This application presents a systematic training and simulation verification of the aforementioned scene generation method. The training platform is built on the PyTorch framework, running on an Ubuntu system with an NVIDIA GeForce RTX 3070Ti GPU. The model is trained end-to-end on a multi-view scene dataset, with a total of 90 training rounds, 1600 training steps per round, and a total of 144,000 training steps.

[0155] Figure 2 The presentation demonstrates the prediction results of the proposed method in a real-world driving scenario. The top left image shows the input driving scene image at the current moment; the top right image shows the first predicted future frame; the bottom left image shows the middle predicted future frame; and the bottom right image shows the final predicted image generated based on the proposed method. It can be observed that the predicted frames generated by this application maintain good semantic consistency and temporal continuity with the real-world driving scene in terms of road structure, vehicle position, traffic facilities, and scene geometry, indicating that the proposed spatiotemporal modeling mechanism can effectively capture dynamic changes in the driving scene.

[0156] To verify the impact of the spatiotemporal attention enhancement structure on the generation quality and stability, this embodiment fully trains the model with the spatiotemporal attention module added, and records the loss change trend and multi-task output indicators during the training process.

[0157] The following table shows the test parameter settings:

[0158] parameter value parameter value Maximum number of training rounds 90 Batch size 8 Number of steps per round 1600 Total training steps 144000 Initial learning rate 1e-4 Final learning rate 1e-5 Optimizer AdamW Weight decay 1e-2 Diffusion loss weight λ_diff 1.0 BEV loss weight λ_bev 2.0 Trajectory loss weights λ_traj 1.5 Gradient clipping 1.0 Timing window length T 8 Number of cameras V 6

[0159] The following table shows the performance simulation test results:

[0160] Test categories Test Project Test Results Video generation FVD 87.3 Video generation SSIM 0.91 Video generation PSNR 28.6 BEV Forecast mIoU 78.4% BEV Forecast Precision 92.1% Trajectory prediction ADE 0.42m Trajectory prediction FDE 0.76m Intent prediction Classification accuracy 94.3%

[0161] The performance evaluation results from the testing phase show that the proposed method achieves superior performance across multiple metrics, including multi-view video generation quality, BEV (Bird's-Eye-View) spatial semantic consistency, and trajectory prediction accuracy. The generated results exhibit stable performance in terms of structural integrity, temporal continuity, and cross-view consistency, without significant semantic misalignment or spatial distortion, indicating that the spatiotemporal joint modeling mechanism effectively maintains scene semantic consistency. Furthermore, in the BEV prediction and trajectory estimation tasks, the model output maintains a high degree of matching with the ground truth annotations, and the prediction error remains low, demonstrating the enhanced scene structure understanding capability of the multi-task joint optimization strategy. The comprehensive test results show that the proposed multimodal controllable generation method not only ensures generation quality but also possesses excellent structural perception capabilities and planning-related information expression capabilities, demonstrating significant technical effectiveness and practical application value in autonomous driving scene modeling and controllable generation applications.

[0162] This embodiment verifies the effectiveness and stability of this application in generating controllable scenes from multiple perspectives. The training process converges smoothly, and the test results show that the model performs well in terms of generation quality, structural consistency, and trajectory prediction accuracy, demonstrating the technical advantages of spatiotemporal joint modeling and multi-task optimization mechanisms, and possessing good engineering feasibility and application value.

[0163] like Figure 3 As shown, in one embodiment, a driving video generation device based on a spatiotemporal factorization architecture and hybrid modulation is provided, which may specifically include:

[0164] The condition set acquisition unit 510 is used to align the historical multi-view driving image sequence with the environmental condition information corresponding to the historical multi-view driving image sequence to obtain a condition set corresponding to each time frame and each view.

[0165] The diffusion input latent variable acquisition unit 520 is used to input the historical multi-view driving image sequence into the latent variable encoder for mapping to obtain the corresponding latent space representation; and to add noise to the latent space representation to obtain the diffusion input latent variable.

[0166] The spatiotemporal fusion feature acquisition unit 530 is used to perform spatiotemporal joint modeling on the diffused input latent variables to obtain spatial branch features and temporal branch features; and to fuse the spatial branch features and temporal branch features to obtain spatiotemporal fusion features.

[0167] The multi-view enhanced feature acquisition unit 540 is used to perform cross-view feature interaction processing on the spatiotemporal fusion features based on the correlation between different camera viewpoints to obtain multi-view enhanced features.

[0168] The video sequence acquisition unit 550 is used to take the multi-view enhancement features as input to the denoising network, perform denoising generation based on the denoising network under the constraints of the condition set, and obtain the latent variable results corresponding to the target future video sequence; and decode the latent variable results to obtain the future multi-view driving video sequence.

[0169] In the embodiments of this application, the explanation and description of the driving video generation device based on spatiotemporal factorization architecture and hybrid modulation can be referred to the explanation and description of the corresponding method above. For the description of the driving video generation method based on spatiotemporal factorization architecture and hybrid modulation, please refer to the above text, which will not be repeated here.

[0170] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. Figure 4 As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes non-volatile storage media and internal memory.

[0171] The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When executed by a processor, this computer program enables the processor to implement a driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation. The internal memory may also store a computer program, which, when executed by a processor, enables the processor to implement a driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation.

[0172] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one embodiment, the driving video generation device based on spatiotemporal factorization architecture and hybrid modulation provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 4 The device shown operates on this device. The device's memory can store the various program modules that make up this driving video generation device based on a spatiotemporal factorization architecture and hybrid modulation, for example, Figure 3 The illustrated unit includes the condition set acquisition unit 510 and the diffusion input latent variable acquisition unit 520. The computer program, comprised of these various program modules, causes the processor to execute the steps in the driving video generation method based on spatiotemporal factorization architecture and hybrid modulation described in the various embodiments of this application.

[0174] In one embodiment, a driving video generation system based on a spatiotemporal factorization architecture and hybrid modulation is provided. The system includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the driving video generation method based on a spatiotemporal factorization architecture and hybrid modulation as described above.

[0175] It should be understood that although the steps in the flowcharts of the various embodiments of this application are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and these steps may be executed in other orders.

[0176] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for driving video generation based on spatio-temporal factorization architecture and hybrid modulation, characterized in that, The method includes: The historical multi-view driving image sequence and the environmental condition information corresponding to the historical multi-view driving image sequence are aligned to obtain the condition set corresponding to each time frame and each view. The historical multi-view driving image sequence is input into a latent variable encoder for mapping to obtain the corresponding latent space representation; the latent space representation is then subjected to noise processing to obtain diffused input latent variables. Spatiotemporal joint modeling is performed on the diffused input latent variables to obtain spatial branch features and temporal branch features; the spatial branch features and temporal branch features are fused to obtain spatiotemporal fused features; Based on the correlation between different camera perspectives, cross-view feature interaction processing is performed on the spatiotemporal fusion features to obtain multi-view enhanced features. The multi-view enhancement features are used as input to a denoising network. Denoising generation is performed based on the denoising network under the constraints of the condition set to obtain the latent variable results corresponding to the target future video sequence. The latent variable results are then decoded to obtain the future multi-view driving video sequence.

2. The driving video generation method based on spatio-temporal factorization architecture and hybrid modulation according to claim 1, characterized in that, The method for aligning historical multi-view driving image sequences with corresponding environmental condition information to obtain condition sets corresponding to each time frame and viewpoint is as follows: Acquire driving scene images of a vehicle at continuous historical time instants captured by multiple cameras, and construct a historical multi-view driving image sequence wherein: ； wherein, is a real number field, denotes a batch size, denotes a number of camera views, denotes a number of time frames, denotes a number of channels, denotes an image height, denotes an image width; reconstructing the historical multi-view driving image sequence into a frame-by-frame and view-by-view form , as a frame-by-frame and view-by-view image input reference: ； acquiring a historical multi-view driving image sequence corresponding occupancy semantic information and occupancy depth information, and are denoted as an occupancy semantic tensor and an occupancy depth tensor, respectively; the occupancy semantic tensor and the occupancy depth tensor are combined in a channel dimension to obtain an occupancy condition tensor : ； wherein, represents the number of passages of the occupancy condition, represents the height of the occupancy condition, represents the width of the occupancy condition; reconstructing the occupancy condition tensor into occupancy condition representations so that occupancy semantic information and occupancy depth information correspond to image inputs of each time frame and each view angle: ； acquire conditional frame information corresponding to the historical multi-view driving image sequence and vehicle motion control information, the vehicle motion control information including speed , trajectory , direction angle , target point and control command ; extending the conditional frame information to obtain a conditional frame representation corresponding to each time step so that each time step has a corresponding reference image condition; The vehicle motion control information is encoded and copied to each time frame and each perspective to obtain a frame-by-frame and perspective-by-perspective form Control condition representations aligned on sample dimension ; based on the condition frame representation , the occupancy condition representation , and the control condition representation , a condition set corresponding to each time frame and each view angle is obtained : 。 3. The driving video generation method based on spatio-temporal factorization architecture and hybrid modulation according to claim 1, characterized in that, The method for inputting the historical multi-view driving image sequence into a latent variable encoder for mapping to obtain the corresponding latent space representation is as follows: the historical multi-view driving image sequence input latent variable encoder performing mapping to obtain a corresponding latent space initial representation : ； wherein, represents a latent variable encoder; to the latent space initial representation introducing a scaling factor performing a scaling process to obtain a latent space representation : 。 4. The driving video generation method based on spatio-temporal factorization architecture and hybrid modulation according to claim 1, characterized in that, The method for adding noise to the latent space representation to obtain the diffused input latent variables is as follows: Subspace representation Sampling noise intensity , the noise intensity Satisfies: ； Right now: ； wherein is a parameter, denotes a mean parameter of the noise intensity sample distribution, denotes a standard deviation parameter of the noise intensity sample distribution; Based on noise intensity noising the latent space representation to obtain a diffused input latent variable: ； where, is a Gaussian noise, denotes element-wise multiplication, denotes a diffuse input latent variable, is a latent space representation.

5. The spatio-temporal factorization architecture and hybrid modulation based driving video generation method of claim 1, wherein, Spatiotemporal joint modeling is performed on the diffused input latent variables to obtain spatial branch features and temporal branch features; The method for fusing the spatial branch features and the temporal branch features to obtain the spatiotemporal fusion features is as follows: diffusion input latent variable spatio-temporal jointly modeled input feature maps : ； wherein, represents the number of channels of the input feature map, represents the height of the input feature map, represents the width of the input feature map; represents the real number field, represents the batch size, represents the number of camera perspectives, represents the number of time frames; Flattening the input feature map of each frame into a set of spatial tokens yields: ； wherein, is a flattened spatial token sequence, wherein, denotes a token feature dimension; The features of different positions in the same frame are associated and modeled, and conditional guidance update is performed in combination with the condition set, to obtain spatial branch features ; Reorganizing the features of the same spatial location at different time steps into a time series format yields: ； wherein, reorganized sequence features obtained after reorganizing the features of the same spatial location at different time steps; Introducing temporal position embeddings , on the basis of the recombinant sequence characteristics , adding temporal position embeddings , obtaining time series characteristics : ； For full time modeling, get time branch features : ； The spatial branch feature and the temporal branch feature are weighted and fused to obtain a spatio-temporal fusion feature: ； in, Represents the fusion coefficient. Indicates spatiotemporal fusion characteristics, This represents the function for modeling temporal attention.

6. The driving video generation method based on spatiotemporal factorization architecture and hybrid modulation according to claim 1, characterized in that, Based on the correlation between different camera viewpoints, the method for performing cross-view feature interaction processing on the spatiotemporal fusion features to obtain multi-view enhanced features is as follows: Let the first The spatiotemporal fusion characteristics from multiple perspectives are Its adjacent view set is Then it comes from the adjacent perspective. Additional information for: ； in, Represents a multi-view feature interaction function; For the first The spatiotemporal fusion characteristics from multiple perspectives; Aggregate the supplementary information from all adjacent viewpoints to obtain the first... Multi-view enhancement from each perspective : ； The multi-view enhancement amount Inject the first through connection mapping Spatiotemporal fusion features from multiple perspectives yield multi-view enhanced features: ； in, and For trainable parameters, Indicates the first Multi-view enhancement features corresponding to each viewpoint.

7. The driving video generation method based on spatiotemporal factorization architecture and hybrid modulation according to claim 1, characterized in that, The method for obtaining the latent variable results corresponding to the target future video sequence by performing denoising generation based on a denoising network under the constraints of the condition set is as follows: Construct denoising scaling factors, which include skip connection factors. Scaling factor of network prediction output Scaling factor for network input and noise intensity coding coefficients ; ； ； ； ； in, Indicates noise intensity; The latent variables of the diffusion input and the output of the denoising network are reparameterized based on the denoising scaling factor. Based on the constraints of the denoising scaling coefficients and the condition set, the output result of the denoising network is obtained. : ； in, This indicates the diffusion of latent input variables. This represents a denoising network with conditional constraints. Represents a set of conditions. For mask; Based on the output of the denoising network, progressive denoising is performed to generate the latent variables for the target's future time steps. .

8. The driving video generation method based on spatiotemporal factorization architecture and hybrid modulation according to claim 7, characterized in that, The method for obtaining the latent variable results corresponding to the target future video sequence by performing denoising generation based on a denoising network under the constraints of the aforementioned set of conditions further includes: Constructing a noise weighting function : ； in, Indicates noise intensity. This represents the standard deviation parameter of the data; Based on the noise weighting function, a main loss is constructed to constrain the consistency between the predicted latent variable and the target latent variable. : ； in, This represents the output of the denoising network. Representing the latent space representation, This represents element-wise multiplication; Construct target frame difference : ； Constructing Predictive Frame Difference : ； Construct a timing consistency loss based on the target frame difference and the predicted frame difference. : ； in, Indicates the number of time frames; Let the high-frequency filtering operator be A high-frequency loss function is constructed based on the high-frequency filtering operator. : ； A total loss function is constructed based on the main loss, time consistency loss, and high-frequency loss. : ； in, and These are weight hyperparameters; Based on the total loss function, the optimizer performs backpropagation updates on the occupancy condition encoding branch parameters and the spatiotemporal fusion branch parameters until a preset stopping condition is reached.

9. A driving video generation device based on spatiotemporal factorization architecture and hybrid modulation, characterized in that, The device includes: The condition set acquisition unit is used to align the historical multi-view driving image sequence with the environmental condition information corresponding to the historical multi-view driving image sequence to obtain the condition set corresponding to each time frame and each view. The diffusion input latent variable acquisition unit is used to input the historical multi-view driving image sequence into the latent variable encoder for mapping to obtain the corresponding latent space representation; and to add noise to the latent space representation to obtain the diffusion input latent variable. The spatiotemporal fusion feature acquisition unit is used to perform spatiotemporal joint modeling on the diffused input latent variables to obtain spatial branch features and temporal branch features; and to fuse the spatial branch features and temporal branch features to obtain spatiotemporal fusion features. The multi-view enhanced feature acquisition unit is used to perform cross-view feature interaction processing on the spatiotemporal fusion features based on the correlation between different camera viewpoints to obtain multi-view enhanced features. The video sequence acquisition unit is used to take the multi-view enhancement features as input to the denoising network, perform denoising generation based on the denoising network under the constraints of the condition set, and obtain the latent variable results corresponding to the target future video sequence; and decode the latent variable results to obtain the future multi-view driving video sequence.

10. A driving video generation system based on spatiotemporal factorization architecture and hybrid modulation, characterized in that, The system includes a memory and a processor. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the driving video generation method based on spatiotemporal factorization architecture and hybrid modulation as described in any one of claims 1 to 8.

Images