Autonomous driving scenario perspective simulation methods, equipment, storage media and software products

Abstract

This application discloses a method, device, storage medium, and program product for simulating the perspective of autonomous driving scenarios, relating to the technical field of autonomous driving. It involves acquiring a reference video and depth map, back-projecting them to generate a 3D point cloud; identifying the dominant structural target, replacing its point cloud with an object-level 3D model to obtain an enhanced point cloud; receiving the target camera trajectory, projecting the enhanced point cloud to obtain a projected video and an invalid mask; extracting the point cloud geometric prior and camera motion prior to form a hybrid explicit geometric prior; inputting this prior into a lightweight control branch parallel to a pre-trained video diffusion model, fusing intermediate features of the model, outputting a correction signal, and applying it to some layers of the model to guide the generation of continuous driving video under the target trajectory. This application requires no iterative optimization, maintains geometric robustness and temporal consistency under large perspective shifts, and is robust to various point cloud inputs.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to autonomous driving scene perspective simulation methods, devices, storage media and program products. Background Technology

[0002] In the field of autonomous driving, closed-loop simulation testing, world model training, and the generation of corner cases typically require synthesizing continuous driving videos of vehicles deviating from the original acquired trajectory. These synthesized scenarios demand significant changes in perspective, such as lateral lane changes, longitudinal acceleration / deceleration, and vehicle pitch or roll. Traditional solutions primarily rely on 3D reconstruction techniques such as Neural Radiation Field (NeRF) or 3D Gaussian Sputtering (3DGS) for perspective extrapolation.

[0003] However, traditional 3D reconstruction methods have significant drawbacks when faced with large viewpoint shifts. When the target viewpoint deviates significantly from the original acquisition trajectory, small errors in the reconstruction geometry are amplified dramatically, leading to distortion of the global structure. This manifests as problems such as broken lane lines, dynamic vehicle shape distortion, inconsistencies in depth maps, and temporal flickering between video frames.

[0004] To overcome the above problems, existing technologies have mainly evolved along two technical routes: The first category is iterative optimization schemes. These schemes first perform extrapolation rendering, then repair the resulting artifact regions, and repeatedly write the repair results back into the explicit 3D representation. For example, related research attempts to optimize scene robustness under large-view conditions through progressive repair. However, such schemes generally suffer from high computational costs and lengthy processing flows, and are prone to error accumulation during multiple iterations, making it difficult to support the large-scale, high-efficiency requirements of autonomous driving simulation.

[0005] The second category is feedforward generation schemes. These schemes use the projection results of point clouds or RGBD images as geometric conditions, inputting them into the diffusion or generation model. While these schemes offer fast inference speeds, their generation quality is highly dependent on the accuracy of the input geometry. If the geometric information comes from monocular or multi-view depth estimation, its inherent errors are significantly amplified during extrapolation. If relying on LiDAR point clouds, the sparsity and limited coverage of the point clouds make it difficult to guarantee stable generation quality in complex and dynamic open scenes.

[0006] In summary, the existing technology mainly suffers from the following core defects: 1. Insufficient understanding of the reliability of geometric priors: Existing solutions assume that the input (RGBD or LiDAR) point cloud is a reliable geometric constraint from the extrapolation perspective. However, in occluded boundaries, near-field dynamic vehicle regions, and regions with varying distances, this geometric information is often the most unstable, and direct use will lead to distortion of the generated results.

[0007] 2. Lack of targeted generative modeling: Existing generative models usually equate point cloud projection videos with ordinary video restoration conditions, failing to specifically model the characteristic that "geometric information, although unreliable, still has guiding value".

[0008] 3. Lack of an effective conditional injection mechanism: Existing solutions lack a lightweight injection mechanism that can combine explicit geometric priors with temporal and camera control signals in the video generation process. This makes it difficult to simultaneously ensure image rendering quality, geometric robustness, and cross-frame temporal consistency under large offset conditions.

[0009] Therefore, how to provide a simulation method that can generate videos with good geometric structure and temporal consistency in autonomous driving scenarios with large field of view displacement in a low-cost, high-efficiency and robust manner is a technical problem that urgently needs to be solved by those skilled in the art.

[0010] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0011] The main purpose of this application is to provide a method, device, storage medium and program product for simulating the perspective of autonomous driving scenarios, which aims to solve the technical problems of geometric and temporal distortion in simulation videos under large perspective shift.

[0012] To achieve the above objectives, this application proposes an autonomous driving scene perspective simulation method, which includes: Step S10: Obtain the reference video sequence and its camera parameters under the original acquisition trajectory, obtain the depth information of each frame based on the reference video sequence, and generate three-dimensional point cloud data according to the depth information and camera parameters; Step S20: Replace the 3D point cloud data of the structure-dominant target in the 3D point cloud data with the 3D point cloud data of the object-level 3D geometric model to obtain the enhanced scene 3D point cloud. Step S30: Receive the target camera trajectory, project the enhanced scene 3D point cloud onto the viewpoint corresponding to the target camera trajectory, and obtain the projected video sequence and mask information for identifying invalid regions in the projected video sequence; Step S40: Extract point cloud geometric priors based on the projected video sequence, extract camera motion priors based on the target camera trajectory and the mask information, and form a hybrid explicit geometric prior; Step S50: The hybrid explicit geometric prior is input into a control branch that runs parallel to the pre-trained video diffusion model. The control branch fuses the hybrid explicit geometric prior based on the intermediate features of the diffusion model and outputs a correction signal. The correction signal is then applied to a portion of the network layers of the diffusion model to guide the generation process and ultimately generate a continuous driving video under the target camera trajectory.

[0013] In one embodiment, the dominant target of the structure includes at least a near-field vehicle.

[0014] In one embodiment, the method of identifying the dominant structural target in step S20 includes: locating the region corresponding to the dominant structural target from the three-dimensional point cloud data based on semantic segmentation, target detection, or instance mask extraction.

[0015] In one embodiment, the object-level 3D geometric model is obtained by at least one of the following methods: image-based object-level 3D reconstruction, CAD template matching, multi-view reconstruction, or calling from a pre-built 3D asset library.

[0016] In one embodiment, the extraction method of the camera motion prior includes: constructing a geometric ray representation for each pixel based on the target camera trajectory, concatenating the geometric ray representation with the mask information, and converting it into camera motion condition features via a learnable mapping head.

[0017] In one embodiment, the pre-trained video diffusion model is a video diffusion Transformer model with initial frame conditions, noise latent variables, and time step information as the main inputs.

[0018] In one embodiment, the target camera trajectory has at least one of the following offsets relative to the original acquisition trajectory: lateral offset, longitudinal offset, rotational offset, elevation, or depression.

[0019] Furthermore, to achieve the above objectives, this application also proposes an autonomous driving scene perspective simulation device, which includes: The first module is used to acquire a reference video sequence and its camera parameters under the original acquisition trajectory, obtain depth information of each frame based on the reference video sequence, and generate three-dimensional point cloud data according to the depth information and camera parameters. The second module is used to replace the 3D point cloud data of the structure-dominant target in the 3D point cloud data with the 3D point cloud data of the object-level 3D geometric model to obtain the enhanced scene 3D point cloud. The third module is used to receive the target camera trajectory, project the enhanced scene 3D point cloud onto the viewpoint corresponding to the target camera trajectory, and obtain the projected video sequence and mask information for identifying invalid regions in the projected video sequence. The fourth module is used to extract point cloud geometric priors based on the projected video sequence, extract camera motion priors based on the target camera trajectory and the mask information, and form a hybrid explicit geometric prior. The fifth module is used to input the hybrid explicit geometric prior into a control branch that runs parallel to the pre-trained video diffusion model. The control branch fuses the hybrid explicit geometric prior based on the intermediate features of the diffusion model and outputs a correction signal. The correction signal is then applied to a portion of the network layers of the diffusion model to guide the generation process and ultimately generate a continuous driving video under the target camera trajectory.

[0020] In addition, to achieve the above objectives, this application also proposes an autonomous driving scene perspective simulation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the autonomous driving scene perspective simulation method described above.

[0021] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the autonomous driving scene perspective simulation method described above.

[0022] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the autonomous driving scene perspective simulation method described above.

[0023] One or more technical solutions proposed in this application have at least the following technical effects: By performing frame-by-frame depth estimation and backprojection, RGBD point clouds under the original trajectory are obtained, providing a basic 3D geometric representation for subsequent geometric enhancement and viewpoint transformation. This step avoids the dynamic target ghosting problem caused by multi-frame accumulation and ensures the purity of the input geometry.

[0024] For structurally dominant targets such as close-up vehicles, which are most prone to geometric distortion under large viewpoint shifts, the unstable original RGBD point cloud is replaced with a structurally complete and shape-robust object-level 3D geometric model. This significantly enhances the geometric reliability of key areas under extreme viewpoint changes, fundamentally suppressing vehicle distortion and breakage caused by depth errors in traditional solutions, and providing a solid geometric anchor point for high-quality extrapolation.

[0025] The enhanced scene point cloud is projected according to a user-defined target trajectory (such as lateral lane changes, elevation changes, etc.), and an invalid region mask is output. This step enables explicit geometric simulation for arbitrary large viewpoint shifts, while the mask information accurately identifies new regions or geometrically incomplete regions exposed due to viewpoint changes, providing crucial prior knowledge for subsequent generative models to distinguish reliable geometric regions from regions requiring repair.

[0026] The projected video extracts point cloud geometric priors, while the target trajectory and mask extract camera motion priors, forming a hybrid explicit geometric prior consisting of point cloud, occlusion, and camera motion. This prior not only includes the color-geometric correspondence after projection but also encodes the spatiotemporal constraints of camera motion, effectively solving the problem that traditional generative models rely solely on a single geometric condition and lack modeling of geometric unreliability.

[0027] A lightweight, plug-and-play control branch is employed to integrate hybrid explicit geometric priors layer by layer into the backbone network's denoising process without disrupting the priors of the pre-trained diffusion model. This mechanism enables the diffusion model to generate data following the target geometry and camera trajectory, while leveraging the powerful generative capabilities of the pre-trained model to appropriately fill in invalid regions identified by the mask. The final output continuous driving video maintains lane line continuity, vehicle shape stability, depth consistency, and cross-frame temporal smoothness even under large viewpoint shifts. Furthermore, the entire process is a feedforward, single-step generation process, requiring no iterative optimization, significantly reducing computational costs and making it suitable for large-scale autonomous driving simulation applications.

[0028] In summary, by employing a collaborative approach of geometric enhancement, explicit projection, hybrid prior extraction, and lightweight control injection, we have effectively suppressed geometric distortion and temporal flicker under large-viewpoint offset, achieving the beneficial effects of low-cost, highly robust, and feedforward simulation. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the first embodiment of the autonomous driving scene perspective simulation method of this application. Figure 2 This is a schematic diagram illustrating the effect of the Geometric Reliability Enhancement (GRE) strategy in the embodiments of this application; Figure 3 This is a schematic diagram illustrating an application scenario of the autonomous driving scene perspective simulation method proposed in this application. Figure 4 This is a schematic diagram of the module structure of the autonomous driving scene perspective simulation device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the autonomous driving scene perspective simulation method in this application embodiment.

[0032] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0033] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0034] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0035] In this embodiment, for ease of description, the processor of the autonomous driving simulation platform will be used as the execution subject in the following description.

[0036] Because existing technologies rely on pure geometric reconstruction under large-viewpoint shift conditions, structural distortion and temporal flicker are easily caused by amplified geometric errors. Feedforward generation schemes, on the other hand, are highly dependent on the quality of the input point cloud and lack specialized modeling of geometrically unreliable regions and lightweight geometric-temporal joint injection mechanisms.

[0037] This application provides a solution that enables a processor to: acquire a reference video sequence and its camera parameters under the original acquisition trajectory, estimate the depth frame by frame and back-project to generate 3D point cloud data; identify the dominant structural target in the point cloud, replace its original point cloud with the point cloud of the object-level 3D geometric model, and obtain an enhanced scene 3D point cloud; receive the target camera trajectory, project the enhanced scene point cloud onto the viewpoint corresponding to the trajectory, and obtain the projected video sequence and mask information identifying invalid regions; extract the point cloud geometric prior based on the projected video, and extract the camera motion prior based on the target camera trajectory and mask information to form a hybrid explicit geometric prior; input the hybrid prior to a lightweight control branch parallel to the pre-trained video diffusion model, which fuses the prior according to the intermediate features of the diffusion model and outputs a correction signal, which is applied to some network layers of the diffusion model to guide the generation process, and finally outputs a continuous driving video with robust geometric structure and temporal consistency under the target camera trajectory.

[0038] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as an autonomous driving scene perspective simulation device. The following description uses an autonomous driving scene perspective simulation device as an example to illustrate this embodiment and the subsequent embodiments.

[0039] Based on this, embodiments of this application provide a method for simulating the perspective of an autonomous driving scene, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the autonomous driving scene perspective simulation method of this application.

[0040] In this embodiment, the autonomous driving scene perspective simulation method includes steps S10~S50: Step S10: Obtain the reference video sequence and its camera parameters under the original acquisition trajectory, obtain the depth information of each frame based on the reference video sequence, and generate three-dimensional point cloud data based on the depth information and camera parameters; In the specific implementation, a reference video sequence under the original acquisition trajectory is first obtained. This video sequence can be recorded by the forward-facing camera of the autonomous driving acquisition vehicle, a multi-camera system, or a simulation platform, denoted as N frames video V. At the same time, the camera intrinsic parameters K and extrinsic parameters E (including rotation matrix and translation vector) corresponding to each frame are obtained. To ensure the consistency of subsequent temporal sequences, it is preferable to align the first frame of the new target trajectory with the first frame of the original trajectory.

[0041] Subsequently, a metric depth estimation is performed on each frame of the reference video. Specifically, a trained depth estimation network (such as Depth Anything V3, VGGT, etc.) can be used to output the metric depth, i.e., a depth map D with true physical scale. In alternative implementations, if the input is a stereo or multi-view system, the depth can also be obtained through stereo matching; or using depth obtained through structured light, dense reconstruction, or LiDAR-assisted correction. The depth map resolution can be the same as the video frame or it can be upsampled and aligned.

[0042] Next, based on the camera pinhole model, using the formula P=K^{-1}*(u,v,1)*D(u,v), each pixel (u,v) and its depth value D(u,v) are back-projected onto the camera coordinate system, and then transformed to a unified world coordinate system via extrinsic parameter E to obtain the RGBD point cloud of the current frame. To avoid ghosting caused by the accumulation of dynamic targets across frames, this embodiment preferably adopts a frame-by-frame independent processing method, that is, each frame generates a point cloud independently without temporal fusion. However, in scenes with a predominantly static background and few dynamic targets, multiple frame point clouds can also be combined after pose registration.

[0043] Step S20: Replace the 3D point cloud data of the structure-dominant target in the 3D point cloud data with the 3D point cloud data of the object-level 3D geometric model to obtain the enhanced scene 3D point cloud. After obtaining the initial 3D point cloud of the scene, geometric reliability enhancement is performed. First, dominant structural targets in the point cloud are identified. For each identified dominant structural target, the following sub-steps are executed: Remove all pixels / points belonging to the target mask region from the original RGBD point cloud to eliminate erroneous geometry caused by depth estimation errors.

[0044] A complete and stable object-level 3D geometric model is generated using object-level 3D reconstruction methods.

[0045] Based on the target's 3D position, orientation (given by target detection or tracking), and scale information in the world coordinate system, the generated object-level 3D model is transformed into scene coordinates and back-interpolated to the corresponding position in the original scene point cloud, thus obtaining the enhanced scene point cloud.

[0046] This step significantly improves the geometric stability of key targets such as near-field vehicles under large angular displacements, such as... Figure 2 As shown, the enhanced point cloud retains its complete shape even at extreme viewing angles.

[0047] Specifically, Figure 2 This is a schematic diagram illustrating the effect of the Geometric Reliability Enhancement (GRE) strategy in the embodiments of this application. Figure 2 The comparison between the two sets of data intuitively demonstrates the role of the GRE module in improving the geometric reliability of point clouds for critical vehicles.

[0048] The upper branch (without GRE) shows the point cloud representation obtained directly from the original RGB-D backprojection. It can be seen that due to depth estimation errors or occlusion, the point cloud of the near-field vehicle is severely distorted: holes appear in the vehicle body area, edges are broken, shapes are incomplete, and even some structures are missing. Such a point cloud, when projected with a large field-of-view offset, will inevitably lead to vehicle shape distortion and fragmentation, affecting the quality of the final generated video.

[0049] The lower branch (using GRE) shows the point cloud representation of the same vehicle region after enhancement by the GRE module. In the implementation, the system first identifies the vehicle as the dominant structural target and removes its original RGB-D point cloud; then, it uses an object-level 3D reconstruction model (such as "Amodal3R") to generate a complete and smooth object-level geometric mesh or point cloud; finally, it interpolates the object-level geometry into the scene according to the vehicle's actual position, orientation, and scale. As can be seen from the figure, the enhanced vehicle point cloud has a complete structure, continuous surface, and accurate shape, exhibiting good geometric reliability.

[0050] Figure 2 This demonstrates that the GRE module can process any structure-dominant target uniformly, independent of specific vehicle models. Through this convenient yet efficient geometry enhancement step, this embodiment fundamentally improves the geometric quality of key areas most prone to distortion under large viewing angle shifts, providing robust geometric anchors for subsequent projection and generation.

[0051] In one feasible implementation, the dominant target of the structure includes at least a near-field vehicle.

[0052] In practical implementation, in addition to near-field vehicles (other cars, trucks, buses, etc. within 30 meters of the vehicle), it can also be expanded to include other targets sensitive to viewpoint shifts, such as pedestrians, cyclists, and roadside traffic sign poles and cones. Preferably, the system filters through a preset category list, for example, extracting connected components with category IDs "car", "truck", and "bus" from the semantic segmentation results as processing objects. This controls computational overhead while ensuring geometric enhancement of the main sources of distortion under large shifts.

[0053] In one feasible implementation, the method of identifying the dominant structural target in step S20 includes: locating the region corresponding to the dominant structural target from the 3D point cloud data based on semantic segmentation, target detection, or instance mask extraction.

[0054] The method for identifying the dominant structural target in step S20 can specifically employ any one or more combinations of the following: Semantic segmentation: PSPNet, DeepLabV3+ and other models are used to predict pixel-level semantic labels frame by frame. Continuous regions with labels belonging to preset categories are extracted as target masks and then back-projected onto the 3D point cloud.

[0055] Object detection: Use detectors such as YOLO and Faster R-CNN to output two-dimensional bounding boxes, and further use instance segmentation (such as Mask R-CNN) within the boxes to obtain accurate masks.

[0056] Instance mask: Directly use the instance mask of each target output by the panoptic segmentation or instance segmentation network, suitable for multi-target fine-grained processing.

[0057] In terms of timing, to maintain the consistency of the target in consecutive frames, Kalman filtering or Hungarian matching algorithms can be combined for target tracking to ensure that the target identity does not change during 3D replacement.

[0058] In one feasible implementation, the object-level 3D geometric model is obtained by at least one of the following methods: image-based object-level 3D reconstruction, CAD template matching, multi-view reconstruction, or calling from a pre-built 3D asset library.

[0059] The object-level three-dimensional geometric model can be obtained through at least one of the following methods: Image-based object-level 3D reconstruction: This method uses models such as Amodal3R and Wonder3D as input, taking a single or multi-view image of the target and outputting a complete texture mesh or point cloud. It is suitable for online dynamic generation.

[0060] CAD template matching: A pre-built library of CAD models for common vehicle types (sedans, SUVs, trucks, etc.) is established. At runtime, based on the specific category and approximate size obtained from target detection, the model is retrieved from the library and adaptively scaled for deformation matching.

[0061] Multi-view reconstruction: If the reference video contains multiple views of the same target at different times, a high-precision model can be reconstructed offline using methods such as COLMAP and Structure of Motion (SfM).

[0062] Pre-built 3D asset library: For fixed scenes (such as simulation test sites), high-quality 3D assets modeled manually can be directly called.

[0063] Preferably, in this embodiment, a geometric library of common targets is built offline during the video preprocessing stage, and only retrieval and position alignment are performed during inference to save computing resources.

[0064] Step S30: Receive the target camera trajectory, project the enhanced scene 3D point cloud onto the viewpoint corresponding to the target camera trajectory, and obtain the projected video sequence and mask information used to identify invalid areas in the projected video sequence; The system receives the target camera trajectory {~C_j} defined by the user or simulation system, where j=1..M, and M is the frame number under the target trajectory. This trajectory, relative to the original acquisition trajectory, can include combinations of lateral offset (e.g., lane change of 3 meters, 6 meters), longitudinal offset, rotational offset (change in heading angle), and rise / sink. The enhanced scene point cloud obtained in step S20 is projected onto the image plane using the standard perspective projection formula according to the extrinsic and intrinsic parameters of the target camera for each frame, resulting in a projected video sequence ~V. Simultaneously, for each projected pixel, it is determined whether it corresponds to a valid point cloud region (i.e., a projection point exists and its depth is within a reasonable range), generating a binary mask M, where a value of 1 indicates validity, and a value of 0 indicates an invalid region due to a new area exposed by a change in viewpoint or geometric incompleteness.

[0065] In one feasible implementation, the target camera trajectory has at least one of the following offsets relative to the original acquisition trajectory: lateral offset, longitudinal offset, rotational offset, elevation, or depression.

[0066] In the specific interactive editing process, the user or simulation controller can define the following parameters: Lateral offset: Displacement along the right direction of the vehicle, such as lane change offset Δy∈[-6m,6m].

[0067] Longitudinal offset: Displacement along the direction of vehicle movement, such as displacement Δx∈[-10m,20m caused by acceleration and deceleration.

[0068] Rotational offset: The angle of rotation about the vertical axis (yaw) or roll / pitch axis.

[0069] Lifting / sinking: Displacement along the vertical direction Δz∈[-2m,2m].

[0070] These offsets can be applied individually or in combination. The system recalculates the target camera extrinsic parameters for each frame based on the offsets, and then performs the projection in step S30. Through this flexible editing method, this embodiment can generate near-infinite scene changes in viewpoint, meeting the diversity and coverage requirements of autonomous driving simulation.

[0071] Step S40: Extract point cloud geometric priors based on the projected video sequence, extract camera motion priors based on the target camera trajectory and mask information, and form a hybrid explicit geometric prior; Construct two types of priors: Point cloud geometric prior: Input the projected video ~V into a pre-trained video autoencoder (such as the encoder part of VAE) to extract compact latent space features z_pc, which encode the color-geometric correspondence after projection.

[0072] Camera motion prior: Based on the camera parameters of each frame of the target camera trajectory, a Plücker ray representation is constructed for each pixel, which is a 6-dimensional vector (d,m), where d is the ray direction, m=p×d, and p is the camera origin. This ray representation is concatenated with the occlusion mask M obtained in step S30 in the channel dimension, and then converted into camera motion conditional features z_cam through a learnable mapping head (composed of several convolutional or linear layers).

[0073] Finally, z_pc and z_cam are merged to form a hybrid explicit geometric prior consisting of point cloud geometry, occlusion, and camera motion.

[0074] In one feasible implementation, the extraction method of the camera motion prior includes: constructing a geometric ray representation for each pixel based on the target camera trajectory, concatenating the geometric ray representation with mask information, and converting it into camera motion condition features through a learnable mapping head.

[0075] The extraction of camera motion priors specifically includes: For each frame of the target trajectory, given the camera intrinsic matrix K and extrinsic parameters (rotation R, translation t), for each pixel coordinate (u, v) on the image plane, calculate the corresponding ray direction d = normalize(K^{-1}*(u, v, 1)) in the camera coordinate system. Then calculate the Plücker coordinates: d and m = (position of the camera optical center in the world coordinate system) × d (or equivalent form). Concatenate the (d, m) channels of each pixel (6 channels in total) with the single-channel mask M obtained in step S30 to obtain a 7-channel pixel-wise feature map.

[0076] The feature map is fed into a small convolutional neural network (i.e., a "learnable map head"), which contains 2-3 convolutional layers and activation functions, and outputs a feature map z_cam whose dimensions are aligned with the spatial / channel dimensions of the point cloud prior z_pc. Through end-to-end training, the map head can adaptively learn how to extract the most instructive motion conditions for video generation from ray geometry and masks.

[0077] Step S50: Input the hybrid explicit geometric prior into the control branch that runs parallel to the pre-trained video diffusion model. The control branch fuses the hybrid explicit geometric prior with the intermediate features of the diffusion model and outputs a correction signal. The correction signal is then applied to some network layers of the diffusion model to guide the generation process and finally generate a continuous driving video under the target camera trajectory.

[0078] A pre-trained video diffusion Transformer is used as the backbone generation network. This network takes the initial frame conditions, noise latent variables, and diffusion time step information as the main inputs to progressively denoise and generate videos.

[0079] This embodiment sets up a lightweight parallel control branch, called the point cloud prior control branch (PCPBranch). This branch receives partial denoised state features from the first few layers (e.g., the first 3 layers or the first half of the layers) of the backbone network, and simultaneously fuses the hybrid prior (z_pc and z_cam) obtained in step S40. Specifically, the control branch consists of several cross-attention layers or adaptive normalization layers, and its output is a correction signal (residual features). This correction signal is added back to the corresponding layer output of the backbone network layer by layer, thereby guiding the generation process to follow the target geometry and camera trajectory without destroying the original generation capability of the pre-trained model.

[0080] After T-step denoising, a continuous driving video along the target camera trajectory is output. This video can be directly used for closed-loop simulation, world model training, and perception algorithm stress testing.

[0081] In one feasible implementation, the pre-trained video diffusion model is a video diffusion Transformer model with initial frame conditions, noise latent variables, and time step information as the main inputs.

[0082] The pre-trained video diffusion model preferably uses a Transformer-based architecture, such as Wan2.1-I2V or Open-Sora. Its inputs include: Initial frame conditions: The first frame image of the target trajectory (which can be provided by the first frame of the projected video ~V, or specified separately) is encoded and used as the starting state for generating content under conditional constraints.

[0083] Noise latent variable: A noise tensor sampled from a standard Gaussian distribution with the same size as the target video, mapped to the latent space via VAE.

[0084] Time step information: The embedding vector of the current denoising step t during the diffusion process.

[0085] The model progressively denoises through multiple spatiotemporal attention blocks to generate a latent space video representation, which is then decoded by a VAE to obtain pixel-level video. In this embodiment, the control branch does not modify the weights of the pre-trained model, but only injects correction signals during forward propagation, thus maintaining the strong generative prior of the pre-trained model.

[0086] In one application scenario of the autonomous driving scene perspective simulation method of this application, such as Figure 3 As shown, the core process of this application's method consists of three stages: scene preprocessing and geometry enhancement, interactive projection, and generative simulation based on hybrid geometric priors. Specifically: Left side: Scene preprocessing and geometric reliability enhancement.

[0087] Input a reference video sequence (i.e., original video frames) under the original acquisition trajectory. Perform depth estimation on each frame to obtain a depth map, and then combine it with camera parameters for back projection to generate an initial RGB-D point cloud.

[0088] For structure-dominant targets (such as vehicles in the foreground) in the initial point cloud, a geometric reliability enhancement module (GRE, indicated by the dashed box in the figure) is used. Specifically: after identifying the target region, unreliable RGB-D points are removed from the original point cloud, and then a complete and stable object-level 3D geometry (object-level geometry) is generated using an object-level 3D reconstruction model (such as Amodal3R) or CAD template. This geometry is then transformed to the scene coordinate system and interpolated to obtain the enhanced scene point cloud.

[0089] Middle section: Interactive projection and hybrid prior extraction.

[0090] The user or simulation system defines the target camera trajectory (e.g., lateral lane change, elevation, etc.), and projects the enhanced scene point cloud according to the target trajectory to obtain the projected video sequence and the corresponding occlusion / invalid region mask.

[0091] Point cloud geometric priors are extracted from the projected video. Simultaneously, camera motion priors are extracted using a learnable embedding head based on the Plücker ray representation and mask information of the target camera trajectory. Together, these constitute a hybrid explicit geometric prior.

[0092] Right side: Generative model based on control branches.

[0093] The pre-trained video diffusion model (Video Diffusion Transformer) is used as the main input for denoising generation, taking the initial frame conditions, noise latent variables, and time step information as the main inputs.

[0094] Add a lightweight, plug-and-play point cloud control branch (PCPBranch, i.e.) Figure 3 (PCP branch in the backbone network). This branch receives some denoised state features from the backbone network and fuses them with the hybrid explicit geometric prior to output a corrected signal.

[0095] The correction signal is injected layer by layer into the first few layers of the diffusion model to guide the generation process to follow the target geometry and camera trajectory. The final output is a continuous driving simulation video under the target trajectory.

[0096] Figure 3 The arrows clearly illustrate the data flow direction: from the original video to the enhanced point cloud, then to projection and prior extraction, and finally guided by the control branch to generate the video. The entire process is feedforward-oriented, requiring no iterative optimization, highlighting the efficiency and robustness of the proposed method.

[0097] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the autonomous driving scene perspective simulation method of this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0098] This application also provides an autonomous driving scene perspective simulation device, please refer to... Figure 4 The autonomous driving scene perspective simulation device includes: The first module 10 is used to acquire a reference video sequence and its camera parameters under the original acquisition trajectory, obtain depth information of each frame based on the reference video sequence, and generate three-dimensional point cloud data according to the depth information and camera parameters. The second module 20 is used to replace the three-dimensional point cloud data of the structure-dominant target in the three-dimensional point cloud data with the three-dimensional point cloud data of the object-level three-dimensional geometric model to obtain the enhanced scene three-dimensional point cloud. The third module 30 is used to receive the target camera trajectory, project the enhanced scene 3D point cloud onto the viewpoint corresponding to the target camera trajectory, and obtain the projected video sequence and mask information for identifying invalid regions in the projected video sequence. The fourth module 40 is used to extract point cloud geometric priors based on the projected video sequence, extract camera motion priors based on the target camera trajectory and the mask information, and form a hybrid explicit geometric prior. The fifth module 50 is used to input the hybrid explicit geometric prior into a control branch that runs parallel to the pre-trained video diffusion model. The control branch fuses the hybrid explicit geometric prior based on the intermediate features of the diffusion model and outputs a correction signal. The correction signal is then applied to a portion of the network layers of the diffusion model to guide the generation process and ultimately generate a continuous driving video under the target camera trajectory.

[0099] In one embodiment, the dominant target of the structure includes at least a near-field vehicle.

[0100] In one embodiment, the method of identifying the dominant structural target includes: locating the region corresponding to the dominant structural target from the three-dimensional point cloud data based on semantic segmentation, target detection, or instance mask extraction.

[0101] In one embodiment, the object-level 3D geometric model is obtained by at least one of the following methods: image-based object-level 3D reconstruction, CAD template matching, multi-view reconstruction, or calling from a pre-built 3D asset library.

[0102] In one embodiment, the extraction method of the camera motion prior includes: constructing a geometric ray representation for each pixel based on the target camera trajectory, concatenating the geometric ray representation with the mask information, and converting it into camera motion condition features via a learnable mapping head.

[0103] In one embodiment, the pre-trained video diffusion model is a video diffusion Transformer model with initial frame conditions, noise latent variables, and time step information as the main inputs.

[0104] In one embodiment, the target camera trajectory has at least one of the following offsets relative to the original acquisition trajectory: lateral offset, longitudinal offset, rotational offset, elevation, or depression.

[0105] The autonomous driving scene perspective simulation device provided in this application, employing the autonomous driving scene perspective simulation method in the above embodiments, can solve the technical problems of geometric and temporal distortion in simulated videos under large perspective shifts. Compared with the prior art, the beneficial effects of the autonomous driving scene perspective simulation device provided in this application are the same as those of the autonomous driving scene perspective simulation method provided in the above embodiments, and other technical features in the autonomous driving scene perspective simulation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0106] This application provides an autonomous driving scene perspective simulation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the autonomous driving scene perspective simulation method in the above embodiment 1.

[0107] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an autonomous driving scene perspective simulation device suitable for implementing embodiments of this application. The autonomous driving scene perspective simulation device in this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The autonomous driving scene perspective simulation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0108] like Figure 5As shown, the autonomous driving scene perspective simulation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the autonomous driving scene perspective simulation device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the autonomous driving scene perspective simulation device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows an autonomous driving scene perspective simulation device with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0109] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0110] The autonomous driving scene perspective simulation device provided in this application, employing the autonomous driving scene perspective simulation method in the above embodiments, can solve the technical problems of geometric and temporal distortion in simulated videos under large perspective shifts. Compared with the prior art, the beneficial effects of the autonomous driving scene perspective simulation device provided in this application are the same as those of the autonomous driving scene perspective simulation method provided in the above embodiments, and other technical features in this autonomous driving scene perspective simulation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0111] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0113] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the autonomous driving scene perspective simulation method in the above embodiments.

[0114] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0115] The aforementioned computer-readable storage medium may be included in the autonomous driving scene perspective simulation device; or it may exist independently and not be assembled into the autonomous driving scene perspective simulation device.

[0116] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the autonomous driving scene perspective simulation device, cause the autonomous driving scene perspective simulation device to: acquire a reference video sequence and its camera parameters under the original acquisition trajectory; obtain depth information for each frame based on the reference video sequence; and generate 3D point cloud data based on the depth information and camera parameters; replace the 3D point cloud data of the structure-dominant target in the 3D point cloud data with the 3D point cloud data of the object-level 3D geometric model to obtain an enhanced scene 3D point cloud; receive the target camera trajectory; and project the enhanced scene 3D point cloud onto the target camera. The projected video sequence and mask information for identifying invalid regions in the projected video sequence are obtained from the viewpoint corresponding to the trajectory. Point cloud geometric priors are extracted based on the projected video sequence, and camera motion priors are extracted based on the target camera trajectory and the mask information to form a hybrid explicit geometric prior. The hybrid explicit geometric prior is input to a control branch that runs parallel to the pre-trained video diffusion model. The control branch fuses the hybrid explicit geometric prior based on the intermediate features of the diffusion model and outputs a correction signal. The correction signal is then applied to some network layers of the diffusion model to guide the generation process, ultimately generating a continuous driving video under the target camera trajectory.

[0117] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0120] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described autonomous driving scene perspective simulation method, which can solve the technical problems of geometric and temporal distortion in simulated videos under large perspective shifts. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the autonomous driving scene perspective simulation method provided in the above embodiments, and will not be repeated here.

[0121] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described autonomous driving scene perspective simulation method.

[0122] The computer program product provided in this application can solve the technical problems of geometric and temporal distortion in simulated videos under large field-of-view shifts. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the autonomous driving scene view simulation method provided in the above embodiments, and will not be repeated here.

[0123] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for simulating the perspective of an autonomous driving scenario, characterized in that, The autonomous driving scenario perspective simulation method includes: Step S10: Obtain the reference video sequence and its camera parameters under the original acquisition trajectory, obtain the depth information of each frame based on the reference video sequence, and generate three-dimensional point cloud data according to the depth information and camera parameters; Step S20: Replace the 3D point cloud data of the structure-dominant target in the 3D point cloud data with the 3D point cloud data of the object-level 3D geometric model to obtain the enhanced scene 3D point cloud. Step S30: Receive the target camera trajectory, project the enhanced scene 3D point cloud onto the viewpoint corresponding to the target camera trajectory, and obtain the projected video sequence and mask information for identifying invalid regions in the projected video sequence; Step S40: Extract point cloud geometric priors based on the projected video sequence, extract camera motion priors based on the target camera trajectory and the mask information, and form a hybrid explicit geometric prior; Step S50: The hybrid explicit geometric prior is input into a control branch that runs parallel to the pre-trained video diffusion model. The control branch fuses the hybrid explicit geometric prior based on the intermediate features of the diffusion model and outputs a correction signal. The correction signal is then applied to a portion of the network layers of the diffusion model to guide the generation process and ultimately generate a continuous driving video under the target camera trajectory.

2. The autonomous driving scene perspective simulation method as described in claim 1, characterized in that, The dominant target of the structure includes at least the near-field vehicle.

3. The autonomous driving scene perspective simulation method as described in claim 1, characterized in that, In step S20, the method of identifying the dominant structural target includes: locating the region corresponding to the dominant structural target from the three-dimensional point cloud data based on semantic segmentation, target detection, or instance mask extraction.

4. The autonomous driving scene perspective simulation method as described in claim 1, characterized in that, The object-level 3D geometric model is obtained using at least one of the following methods: image-based object-level 3D reconstruction, CAD template matching, multi-view reconstruction, or by calling from a pre-built 3D asset library.

5. The autonomous driving scene perspective simulation method as described in claim 1, characterized in that, The method for extracting the camera motion prior includes: constructing a geometric ray representation for each pixel based on the target camera trajectory, concatenating the geometric ray representation with the mask information, and converting it into camera motion condition features via a learnable mapping head.

6. The autonomous driving scene perspective simulation method as described in claim 1, characterized in that, The pre-trained video diffusion model is a video diffusion Transformer model with initial frame conditions, noise latent variables, and time step information as the main inputs.

7. The autonomous driving scene perspective simulation method as described in claim 1, characterized in that, The target camera trajectory has at least one of the following offsets relative to the original acquisition trajectory: lateral offset, longitudinal offset, rotational offset, elevation, or depression.

8. An autonomous driving scene perspective simulation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the autonomous driving scene perspective simulation method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the autonomous driving scene perspective simulation method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the autonomous driving scene perspective simulation method as described in any one of claims 1 to 7.

Images