Video generation method, system, device and medium for autonomous driving simulation

Abstract

The application relates to the field of automatic driving simulation, and particularly discloses a video generation method and system for automatic driving simulation and a computer device, which comprises the following steps: obtaining a reference frame; obtaining coding features and depth map features based on the reference frame; determining perspective dynamic features based on the coding features and the depth map features of the reference frame; and iteratively processing the perspective dynamic features based on a generation model to obtain a video segment. The scheme can generate an automatic driving simulation video with similar effects to traditional multi-step denoising and up-to-standard visual fidelity under fewer denoising steps, thereby guaranteeing the picture quality and physical scene restoration degree of the simulation video and fundamentally shortening the generation time of a single simulation video, and significantly improving the video generation efficiency of the automatic driving simulation scene.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving simulation, and specifically to a video generation method, system, device, and medium for autonomous driving simulation. Background Technology

[0002] In the field of environmental modeling for closed-loop simulation of autonomous driving, traditional simulation platforms, represented by CARLA, rely on scene mesh model rasterization rendering and physics engines to build interactive environments. While they have the advantages of fast rendering speed and strong real-time interactivity, on the one hand, their rendering effect is difficult to achieve real-world photorealistic fidelity, and there is a large domain gap between them and real sensor data. In particular, the simulation accuracy of physical effects under complex weather and extreme lighting scenarios is insufficient, and they cannot accurately match the perception requirements of real vehicles. On the other hand, scene construction is highly dependent on manual modeling, making it difficult to generate diverse real-world long-tail scenes on a large scale and automatically, which cannot meet the testing requirements of advanced autonomous driving for complex edge scenes.

[0003] The new modeling technology based on 3D Gaussian scattering (3D GS) can significantly improve visual fidelity and its rendering speed is comparable to that of traditional technologies. However, its simulation environment diversity is limited by the acquired data and it cannot generate long-tail scenes outside the data coverage, making it difficult to meet the needs of large-scale and diversified simulation.

[0004] Current video generation technologies for autonomous driving simulation modeling are mostly based on diffusion models or flow matching algorithms. During the inference stage, a large number of denoising operations (e.g., 25 to 50 steps or even more iterations) are required to ensure basic generation quality, resulting in low simulation efficiency and failing to meet the requirements of efficient and large-scale scene generation for autonomous driving closed-loop simulation. Summary of the Invention

[0005] In view of this, in order to overcome at least one aspect of the above problems, embodiments of the present invention propose a video generation method for autonomous driving simulation, comprising the following steps: Obtain the reference frame; Based on the reference frame, coded features and depth map features are obtained; The viewpoint dynamic features are determined based on the coded features of the reference frame and the depth map features. Video segments are obtained by iterating the dynamic features of the viewpoint based on the generative model.

[0006] In some embodiments, determining viewpoint dynamic features based on the coded features of the reference frame and the depth map features further includes: Obtain the camera pose of the reference frame and the camera pose of the next frame; Based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained. The coding features of each projection point are determined based on the coding features of the reference frame and the coordinate mapping relationship. The dynamic features of the viewpoint are determined based on the encoded features of each projection point.

[0007] In some embodiments, based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained, further including: Based on the depth map features and the camera pose corresponding to the reference frame, the three-dimensional point corresponding to each pixel in the reference frame is obtained, and the first coordinate correspondence between each pixel and each three-dimensional point is determined. Based on the camera pose corresponding to the next frame, the three-dimensional points are projected to obtain projection points, and the second coordinate correspondence between each three-dimensional point and each projection point is determined. The coordinate mapping relationship between each pixel and the projection point is determined based on the first coordinate correspondence and the second coordinate correspondence.

[0008] In some embodiments, obtaining coded features and depth map features based on the reference frame further includes: The reference frame is encoded using a variational autoencoder to obtain latent space features; wherein, the latent space features corresponding to each pixel in the latent space features are determined based on the coordinates of each pixel in the reference frame. Depth map features are obtained by performing depth estimation on the reference frame based on a scaled depth estimation model; wherein, the depth map features corresponding to each pixel in the depth map features are determined based on the coordinates of each pixel in the reference frame.

[0009] In some embodiments, obtaining video segments by iteratively processing the dynamic features of the viewpoint based on a generative model further includes: Noise mixing is applied to the aforementioned dynamic features of the viewpoint; Based on the generative model, the dynamic features of the viewpoint after mixing noise are iterated to obtain video segments.

[0010] In some embodiments, noise mixing of the viewpoint dynamic features further includes: Obtain the noise to be mixed and the hyperparameters for controlling the degree of noise addition; The viewpoint dynamic features are noise-mixed based on the hyperparameters and the noise to be mixed.

[0011] In some embodiments, before iterating the viewpoint dynamic features of the mixed noise based on the generative model to obtain a video segment, the method further includes: A training set is constructed using the dynamic features of the viewpoint after multiple noise mixing processes and the corresponding real video clips. The generative model is trained based on the training set.

[0012] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a video generation system for autonomous driving simulation, comprising: The acquisition module is configured to acquire reference frames; The encoding module is configured to obtain encoded features and depth map features based on the reference frame; The rendering module is configured to determine the dynamic features of the viewpoint based on the encoded features and depth map features of the reference frame. The generation module is configured to iterate through the dynamic features of the viewpoint based on the generation model to obtain video clips.

[0013] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising: At least one processor; and The memory stores a computer program that can run on the processor, which, when executing the program, performs the steps of any of the video generation methods for autonomous driving simulation as described above.

[0014] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the video generation methods for autonomous driving simulation as described above.

[0015] The present invention has one of the following beneficial technical effects: The proposed solution obtains the dynamic features of the viewpoint through reference frames, and performs a small number of iterations based on the dynamic features of the viewpoint, changing the paradigm from predicting the final result to predicting the amount of change, reducing the waste of computation in the generation process. Thus, it can generate autonomous driving simulation videos with similar visual fidelity to traditional multi-step denoising with fewer denoising steps. This not only ensures the picture quality and physical scene restoration of the simulation video, but also fundamentally shortens the generation time of a single simulation video, significantly improving the video generation efficiency of autonomous driving simulation scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a video generation method for autonomous driving simulation provided as an embodiment of the present invention; Figure 2 A schematic diagram illustrating the generation of continuous video segments provided for an embodiment of the present invention; Figure 3 A flowchart illustrating a video generation method for autonomous driving simulation provided as an embodiment of the present invention; Figure 4 A schematic diagram of a video generation system for autonomous driving simulation provided as an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0019] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0020] According to one aspect of the present invention, embodiments of the present invention propose a video generation method for autonomous driving simulation, such as... Figure 1 As shown, it may include the following steps: S1, Obtain the reference frame; S2, based on the reference frame, obtain the encoded features and depth map features; S3, determine the dynamic features of the viewpoint based on the coding features and depth map features of the reference frame; S4, Based on the generative model, iterate the dynamic features of the viewpoint to obtain video segments.

[0021] Specifically, such as Figure 2As shown, during continuous video segment generation, a single generation window (a single generated video segment) comprises multiple frames. The reference frame serves as the foundation for video generation. It can be real-world scene image data captured by image acquisition devices (such as cameras) in the actual environment, ensuring consistency between the generated video and the real environment; or it can be the last frame of the previously generated video segment, enabling coherent iteration of video generation, avoiding gaps between frames, and adapting to continuous video generation scenarios. The reference frame is then processed to obtain its encoded features and depth map features. The encoded features characterize the deep semantics and detailed texture information of the reference frame, while the depth map features characterize the spatial depth relationships of the scene within the reference frame. The collaboration of these two features ensures the accuracy of subsequent viewpoint dynamic features. Next, the reference frame is reconstructed and projected using its encoded features and depth map features, rendering the viewpoint dynamic features. Since the viewpoint dynamic features already contain most of the structural information in the final video, meaning the information from the reference frame is directly retained in subsequent frames, the number of inference steps in the generation model can be significantly reduced when using this feature as the starting point for denoising video generation, thereby improving the efficiency of video generation.

[0022] Therefore, the proposed solution obtains the dynamic features of the viewpoint through reference frames and performs a small number of iterations based on the dynamic features of the viewpoint, transforming the paradigm of predicting the final result into the paradigm of predicting the amount of change, reducing the waste of computation during the generation process. This enables the generation of autonomous driving simulation videos with similar visual fidelity to traditional multi-step denoising with fewer denoising steps. It not only ensures the picture quality and physical scene reproduction of the simulation video, but also fundamentally shortens the generation time of a single simulation video, significantly improving the video generation efficiency of autonomous driving simulation scenarios.

[0023] In some embodiments, determining viewpoint dynamic features based on the coded features of the reference frame and the depth map features further includes: Obtain the camera pose of the reference frame and the camera pose of the next frame; Based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained. The coding features of each projection point are determined based on the coding features of the reference frame and the coordinate mapping relationship. The dynamic features of the viewpoint are determined based on the encoded features of each projection point.

[0024] Specifically, such as Figure 3As shown, camera pose characterizes the spatial position and shooting angle of the camera when capturing a reference frame and the next frame. If the reference frame is real data collected from the actual environment, the camera pose parameters can be directly obtained through the camera's built-in attitude sensor (such as an IMU inertial measurement unit). If the reference frame is the last frame of the previously generated video segment, the generation model can synchronously output the camera pose corresponding to that frame and predict the camera pose of the next frame based on the viewpoint change pattern. The camera pose parameters mainly include translation and rotation matrices, which accurately describe the camera's attitude state in three-dimensional space.

[0025] The depth map features of the reference frame provide the 3D spatial depth information corresponding to each pixel in the reference frame. Combined with the camera pose of the reference frame, the pixel can be transformed from a 2D image coordinate system to a 3D world coordinate system. Then, based on the camera pose of the next frame, this 3D coordinate system is projected onto the 2D image coordinate system of the next frame, obtaining the projection point of the pixel in the 2D image coordinate system of the next frame. Simultaneously, the coordinate correspondence between each pixel and its projection point is recorded, forming a complete coordinate mapping table. Then, combining the coordinate mapping relationship, the encoded features of each pixel in the reference frame are accurately mapped to its corresponding projection point, thus obtaining the encoded features of each projection point. Combining the encoded features of each projection point yields the dynamic features of the viewpoint.

[0026] In some embodiments, based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained, further including: Based on the depth map features and the camera pose corresponding to the reference frame, the three-dimensional point corresponding to each pixel in the reference frame is obtained, and the first coordinate correspondence between each pixel and each three-dimensional point is determined. Based on the camera pose corresponding to the next frame, the three-dimensional points are projected to obtain projection points, and the second coordinate correspondence between each three-dimensional point and each projection point is determined. The coordinate mapping relationship between each pixel and the projection point is determined based on the first coordinate correspondence and the second coordinate correspondence.

[0027] Specifically, such as Figure 3 As shown, the depth map features of the reference frame provide the depth information of each pixel in the reference frame in 3D space, that is, the spatial distance of the pixel to the object in the real scene. The camera pose corresponding to the reference frame includes the extrinsic and intrinsic parameter matrices from the camera to the world, providing a pose reference for the transformation from 2D pixel coordinates to 3D spatial coordinates. Thus, by combining the camera intrinsic parameter matrix with the depth map features, the 2D image coordinates of each pixel i in the reference frame can be obtained ( , ), converted to three-dimensional world coordinates ( The point j corresponding to the given 3D world coordinates is the 3D point corresponding to pixel i. The specific formula is as follows:

[0028] in, The extrinsic parameter matrix from camera to world, This is the intrinsic parameter matrix. During the transformation process, the two-dimensional coordinates of each pixel i and the three-dimensional coordinates of its corresponding three-dimensional point j are recorded simultaneously, forming a one-to-one correspondence between the first coordinates. A coordinate mapping table between pixel i and three-dimensional point j is constructed to ensure that each pixel can be accurately associated with its corresponding three-dimensional spatial position.

[0029] The camera pose in the next frame also includes the extrinsic and intrinsic parameter matrices from the world to the camera. Based on this camera pose, a 3D-to-2D projection algorithm is used to project each 3D point j ( The three-dimensional point is projected from the three-dimensional world coordinate system to the two-dimensional image coordinate system of the next frame, resulting in the corresponding two-dimensional image point in each frame, i.e., the projection point v. , The specific formula is as follows:

[0030]

[0031] in, The extrinsic parameter matrix from the world to the camera, This is the intrinsic parameter matrix. During the transformation process, the 3D coordinates of each 3D point j and the 2D image coordinates of the corresponding projection point v are recorded one by one, forming a one-to-one correspondence in the second coordinate system, and constructing a coordinate mapping table between 3D points and projection points. This process ensures that each 3D point can be accurately mapped to its corresponding position in the 2D image coordinate system of the next frame, accurately reflecting the spatial position offset caused by the change in viewpoint from the reference frame to the next frame.

[0032] Finally, combining the first coordinate correspondence (pixel to 3D point) and the second coordinate correspondence (3D point to projection point), an indirect mapping from pixel to projection point is achieved through coordinate association fusion. That is, using the 3D point as the intermediate association carrier, each pixel is associated with its corresponding 3D point through the first correspondence, and then that 3D point is associated with its corresponding projection point through the second correspondence, thus establishing a one-to-one correspondence between each pixel and its corresponding projection point—the final coordinate mapping relationship. This coordinate mapping relationship clearly represents the specific position of each pixel in the reference frame in the next frame after a change in viewpoint, providing a precise coordinate basis for mapping the encoded features of the reference frame to the projection point to obtain dynamic viewpoint features, while ensuring the accuracy and consistency of feature mapping.

[0033] In some embodiments, obtaining coded features and depth map features based on the reference frame further includes: The reference frame is encoded using a variational autoencoder to obtain latent space features; wherein, the latent space features corresponding to each pixel in the latent space features are determined based on the coordinates of each pixel in the reference frame. Depth map features are obtained by performing depth estimation on the reference frame based on a scaled depth estimation model; wherein, the depth map features corresponding to each pixel in the depth map features are determined based on the coordinates of each pixel in the reference frame.

[0034] Specifically, such as Figure 3 As shown, conventional video generation models often do not generate directly in pixel space. Taking the latent space diffusion model as an example, it first uses a variational autoencoder to map the image to a relatively low-dimensional latent space, and then performs noise addition and denoising in the latent space. This is because for a given reference frame... It can be encoded using a variational autoencoder, thus obtaining the latent space features of the reference frame. :

[0035] The dimension of this feature is often related to the image size. For example, an image of shape [h, w, 3] can be encoded as a latent space feature of shape [h / p, w / p, c], where p is the compression ratio and c is the feature dimension. It can also be encoded using the coordinates of each pixel (…). , Using as an index, the latent space features corresponding to each pixel i are obtained. .

[0036] Latent space features can also be transformed back to the original image using the decoder corresponding to the variational autoencoder:

[0037] Simultaneously, a scaled depth estimation model can be used to estimate the depth of the reference frame. Scaled means that the estimated depth values ​​are consistent with the actual scale, rather than being relative depths (e.g., depth values ​​normalized to 0-1). Taking MapAnything as an example, the depth map is obtained by encoding the reference frame. :

[0038] It can also be obtained through the coordinates of each pixel ( , Using as an index, the depth map features corresponding to each pixel i are obtained. .

[0039] In some embodiments, obtaining video segments by iteratively processing the dynamic features of the viewpoint based on a generative model further includes: Noise mixing is applied to the aforementioned dynamic features of the viewpoint; Based on the generative model, the dynamic features of the viewpoint after mixing noise are iterated to obtain video segments.

[0040] Specifically, viewpoint dynamics describes the dynamic changes in video caused by changes in camera angle. By estimating viewpoint dynamics, information from reference frames can be directly retained in subsequent frames. However, reference frames often cannot cover all subsequent content. On one hand, new areas will be observed as the car moves forward, providing information that reference frames cannot provide; on the other hand, there are dynamic objects in the environment, or the environment itself will evolve over time. Therefore, further optimization based on viewpoint dynamics is necessary.

[0041] For the previously estimated viewpoint dynamics, an appropriate amount of noise is first added. Since depth estimation may have errors, and the viewpoint dynamics may not be completely consistent with the final result, adding noise can reduce the impact of errors on the result, allowing the model to learn from the noisy rendering result as a starting point to generate the final video.

[0042] In some embodiments, noise mixing of the viewpoint dynamic features further includes: Obtain the noise to be mixed and the hyperparameters for controlling the degree of noise addition; The viewpoint dynamic features are noise-mixed based on the hyperparameters and the noise to be mixed.

[0043] Specifically, the noise to be mixed can be random noise consistent with the dynamic feature dimension of the viewpoint, preferably Gaussian noise or uniformly distributed noise. For example, it can be generated in real time by a random noise generator to ensure the randomness and diversity of the noise; alternatively, real noise data corresponding to the video generation scene can be collected in advance, normalized, and dimensionally adjusted before being used as the noise to be mixed, making the noise more consistent with the actual generation scene and improving the realism of the video generation. Then, based on preset hyperparameters (mainly noise weight coefficients), a weighted fusion algorithm is used for noise mixing. Therefore, the noise addition process for dynamic viewpoints can be described as follows:

[0044] in It is a hyperparameter that controls the degree of noise addition. To introduce noise. Obtain the dynamic viewpoint after adding noise. Then, the generative model can be used to dynamically adjust the noisy viewpoint. Iteratively generate video clips.

[0045] In some embodiments, before iterating the viewpoint dynamic features of the mixed noise based on the generative model to obtain a video segment, the method further includes: A training set is constructed using the dynamic features of the viewpoint after multiple noise mixing processes and the corresponding real video clips. The generative model is trained based on the training set.

[0046] Specifically, for a given And the corresponding real videos A generative model can be trained to learn the following objective:

[0047]

[0048] For a time step t, the value from the previous time step is obtained by interpolation. arrive The intermediate path. For points in the intermediate path. The FM generator will take the input as input. And t, prediction That is, from arrive The direction.

[0049] After training is completed, for a given ,Right now Using FM prediction Then move in the direction of the FM output. Distance:

[0050] After multiple iterations, the final result was obtained. This refers to the final video generated.

[0051] Compared to pure noise, viewpoint dynamics already contain most of the structural information in the final video. Therefore, the number of inference steps in the generation model can be significantly reduced when generating the video, thereby improving the efficiency of video generation.

[0052] The proposed solution decouples video prediction content into viewpoint dynamic generation and video segment generation. Viewpoint dynamics describes the dynamic content caused by changes in the vehicle's position and pose during the driving scene. This means that although the viewpoint changes, the content remains essentially consistent, and good viewpoint dynamics can make the video more coherent and consistent. Since the environment itself undergoes dynamic evolution—for example, some details in the environment change over time—this is the part of the new content in the video prediction that truly differs from historical content. Therefore, predicting environmental dynamics (video segments) based on viewpoint dynamics results reduces the waste of computation during the generation process. This shifts the paradigm from predicting the final result to predicting changes, thus reducing the pressure on the model. It generates autonomous driving simulation videos with similar visual fidelity to traditional multi-step denoising with fewer denoising steps. This ensures both the image quality and physical scene reproduction of the simulation video, and fundamentally shortens the generation time of a single simulation video, significantly improving the video generation efficiency of autonomous driving simulation scenarios.

[0053] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a video generation system 300 for autonomous driving simulation, such as... Figure 4 As shown, it includes: Module 301 is configured to acquire reference frames; Encoding module 302 is configured to obtain encoded features and depth map features based on the reference frame; The rendering module 303 is configured to determine the dynamic features of the viewpoint based on the encoded features and depth map features of the reference frame. The generation module 304 is configured to iterate the dynamic features of the viewpoint based on the generation model to obtain video segments.

[0054] In some embodiments, determining viewpoint dynamic features based on the coded features of the reference frame and the depth map features further includes: Obtain the camera pose of the reference frame and the camera pose of the next frame; Based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained. The coding features of each projection point are determined based on the coding features of the reference frame and the coordinate mapping relationship. The dynamic features of the viewpoint are determined based on the encoded features of each projection point.

[0055] In some embodiments, based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained, further including: Based on the depth map features and the camera pose corresponding to the reference frame, the three-dimensional point corresponding to each pixel in the reference frame is obtained, and the first coordinate correspondence between each pixel and each three-dimensional point is determined. Based on the camera pose corresponding to the next frame, the three-dimensional points are projected to obtain projection points, and the second coordinate correspondence between each three-dimensional point and each projection point is determined. The coordinate mapping relationship between each pixel and the projection point is determined based on the first coordinate correspondence and the second coordinate correspondence.

[0056] In some embodiments, obtaining coded features and depth map features based on the reference frame further includes: The reference frame is encoded using a variational autoencoder to obtain latent space features; wherein, the latent space features corresponding to each pixel in the latent space features are determined based on the coordinates of each pixel in the reference frame. Depth map features are obtained by performing depth estimation on the reference frame based on a scaled depth estimation model; wherein, the depth map features corresponding to each pixel in the depth map features are determined based on the coordinates of each pixel in the reference frame.

[0057] In some embodiments, obtaining video segments by iteratively processing the dynamic features of the viewpoint based on a generative model further includes: Noise mixing is applied to the aforementioned dynamic features of the viewpoint; Based on the generative model, the dynamic features of the viewpoint after mixing noise are iterated to obtain video segments.

[0058] In some embodiments, noise mixing of the viewpoint dynamic features further includes: Obtain the noise to be mixed and the hyperparameters for controlling the degree of noise addition; The viewpoint dynamic features are noise-mixed based on the hyperparameters and the noise to be mixed.

[0059] In some embodiments, before iterating the viewpoint dynamic features of the mixed noise based on the generative model to obtain a video segment, the method further includes: A training set is constructed using the dynamic features of the viewpoint after multiple noise mixing processes and the corresponding real video clips. The generative model is trained based on the training set.

[0060] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 5 As shown, embodiments of the present invention also provide a computer device 501, comprising: At least one processor 520; and The memory 510 stores a computer program 511 that can run on a processor. When the processor 520 executes the program, it performs the steps of any of the video generation methods for autonomous driving simulation described above.

[0061] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the steps of any of the video generation methods for autonomous driving simulation described above.

[0062] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.

[0063] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0064] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0065] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0066] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0067] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0068] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0069] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A video generation method for autonomous driving simulation, characterized in that, Includes the following steps: Obtain the reference frame; Based on the reference frame, coded features and depth map features are obtained; The viewpoint dynamic features are determined based on the coded features of the reference frame and the depth map features. Video segments are obtained by iterating the dynamic features of the viewpoint based on the generative model.

2. The method as described in claim 1, characterized in that, Determining viewpoint dynamic features based on the coded features of the reference frame and the depth map features further includes: Obtain the camera pose of the reference frame and the camera pose of the next frame; Based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained. The coding features of each projection point are determined based on the coding features of the reference frame and the coordinate mapping relationship. The dynamic features of the viewpoint are determined based on the encoded features of each projection point.

3. The method as described in claim 2, characterized in that, Based on the depth map features, the camera pose corresponding to the reference frame, and the camera pose of the next frame, the projection point corresponding to each pixel in the reference frame and the coordinate mapping relationship between each pixel and the projection point are obtained, further including: Based on the depth map features and the camera pose corresponding to the reference frame, the three-dimensional point corresponding to each pixel in the reference frame is obtained, and the first coordinate correspondence between each pixel and each three-dimensional point is determined. Based on the camera pose corresponding to the next frame, the three-dimensional points are projected to obtain projection points, and the second coordinate correspondence between each three-dimensional point and each projection point is determined. The coordinate mapping relationship between each pixel and the projection point is determined based on the first coordinate correspondence and the second coordinate correspondence.

4. The method as described in claim 1, characterized in that, Based on the reference frame, coded features and depth map features are obtained, further including: The reference frame is encoded using a variational autoencoder to obtain latent space features; wherein, the latent space features corresponding to each pixel in the latent space features are determined based on the coordinates of each pixel in the reference frame. Depth map features are obtained by performing depth estimation on the reference frame based on a scaled depth estimation model; wherein, the depth map features corresponding to each pixel in the depth map features are determined based on the coordinates of each pixel in the reference frame.

5. The method as described in claim 1, characterized in that, Based on the generative model, the video clips are obtained by iteratively analyzing the dynamic features of the viewpoint, and further include: Noise mixing is applied to the aforementioned dynamic features of the viewpoint; Based on the generative model, the dynamic features of the viewpoint after mixing noise are iterated to obtain video segments.

6. The method as described in claim 5, characterized in that, The noise mixing of the aforementioned viewpoint dynamic features further includes: Obtain the noise to be mixed and the hyperparameters for controlling the degree of noise addition; The viewpoint dynamic features are noise-mixed based on the hyperparameters and the noise to be mixed.

7. The method as described in claim 5, characterized in that, Before iterating through the viewpoint dynamic features of the mixed noise based on the generative model to obtain the video segment, the process also includes: A training set is constructed using the dynamic features of the viewpoint after multiple noise mixing processes and the corresponding real video clips. The generative model is trained based on the training set.

8. A video generation system for autonomous driving simulation, characterized in that, include: The acquisition module is configured to acquire reference frames; The encoding module is configured to obtain encoded features and depth map features based on the reference frame; The rendering module is configured to determine the dynamic features of the viewpoint based on the encoded features and depth map features of the reference frame. The generation module is configured to iterate through the dynamic features of the viewpoint based on the generation model to obtain video clips.

9. A computer device, comprising: At least one processor; as well as A memory storing a computer program executable on the processor, characterized in that the processor executes the program by performing the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-7.

Images