Driving scene reconstruction method and device, electronic device, program product

Abstract

The method comprises the following steps: obtaining input data and output data of a video generation model at an Nth time step and input data at an N+1th time step; writing the output data of the Nth time step into a cache; determining an estimated value of output data of the N+1th time step through a mapping function; comparing fluctuation differences between the output data of the Nth time step and the estimated value; determining whether the output data of the Nth time step stored in the cache is reused at the N+1th time step to obtain output data of the N+1th time step based on whether the fluctuation differences meet preset conditions; and obtaining a first image sequence to reconstruct a driving scene based on the first image sequence. The method can improve the inference efficiency of the video generation model and reduce the demand for computing resources.

Description

Technical Field

[0001] This disclosure relates to video generation technology, and in particular to a driving scene reconstruction method and apparatus, electronic equipment, and program products. Background Technology

[0002] 4D driving scenarios are simulations of a four-dimensional dynamic driving environment created by adding a time dimension to the traditional 3D spatial dimension. They are primarily used to enhance the perception and response capabilities of autonomous driving systems. One method for reconstructing a driving scenario involves first generating a 2D driving video, and then reconstructing the dynamic scene by separating foreground dynamic objects from the static background. The reasoning process for generating the 2D driving video requires significant hardware computing resources and thus demands high computational power. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a driving scene reconstruction method and apparatus, electronic device, and program product.

[0004] According to a first aspect of the present disclosure, a driving scene reconstruction method is provided. The method includes: acquiring input data and output data of a video generation model at an Nth time step and input data at an N+1th time step; N is an integer, N≥1; writing the output data of the Nth time step into a cache; determining the output data of the N+1th time step by a mapping function based on the input data and output data of the Nth time step and the input data of the N+1th time step; the mapping function is pre-fitted based on samples of multiple input data and multiple output data of the video generation model; comparing the fluctuation difference between the output data of the Nth time step and the output data of the N+1th time step; determining whether to reuse the output data of the Nth time step stored in the cache at the N+1th time step to obtain the output data of the N+1th time step based on whether the fluctuation difference between the output data of the Nth time step and the output data of the N+1th time step meets a preset condition; obtaining a first image sequence to reconstruct a driving scene based on the first image sequence; the first image sequence includes the output data from the 1st time step to the N+1th time step.

[0005] According to a second aspect of the present disclosure, a driving scene reconstruction method is provided. The method includes: acquiring image features corresponding to a pre-acquired video or a first image sequence generated in real time; inputting the image features into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively, outputting depth information through the depth estimation network and outputting pose information through the pose estimation network; inputting the depth information and pose information into a pre-trained Gaussian prediction network, obtaining target parameters for displaying a 3D Gaussian point cloud through the Gaussian prediction network; the Gaussian prediction network is a residual network including attention calculation and convolution operation; and reconstructing the driving scene based on the target parameters and time information.

[0006] According to a third aspect of the present disclosure, a driving scene reconstruction apparatus is provided. The apparatus includes: an acquisition module, configured to acquire input data and output data of a video generation model at an Nth time step and input data at an (N+1)th time step; where N is an integer and N≥1; a caching module, configured to write the output data of the Nth time step into a cache; and a mapping module, configured to determine the output data of the (N+1)th time step using a mapping function based on the input data and output data of the Nth time step and the input data of the (N+1)th time step; the mapping function is pre-defined based on multiple input data and multiple output data of the video generation model. The sample fitting is used to obtain the following modules: a comparison module, used to compare the fluctuation difference between the output data of the Nth time step and the output data of the N+1th time step; a discrimination module, used to determine whether to reuse the output data of the Nth time step stored in the cache in the N+1th time step to obtain the output data of the N+1th time step based on whether the fluctuation difference between the output data of the Nth time step and the output data of the N+1th time step meets the preset conditions; and an output module, used to obtain a first image sequence and reconstruct the driving scene based on the first image sequence. The first image sequence includes the output data from the 1st time step to the N+1th time step.

[0007] According to a fourth aspect of the present disclosure, a driving scene reconstruction apparatus is provided. The apparatus includes: a feature fusion module for acquiring image features corresponding to a pre-acquired video or a first image sequence generated in real time; a parameter estimation module for inputting the image features into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively, outputting depth information through the depth estimation network and outputting pose information through the pose estimation network; a parameter prediction module for inputting the depth information and pose information into a pre-trained Gaussian prediction network, obtaining target parameters for displaying a 3D Gaussian point cloud through the Gaussian prediction network; the Gaussian prediction network is a residual network including attention calculation and convolution operation; and a generation module for reconstructing a driving scene based on the target parameters and time information.

[0008] Based on the driving scene reconstruction method of the first aspect provided in the above embodiments of this disclosure, a dynamic caching strategy is proposed for video scene generation by a video generation model (e.g., a video diffusion model). This strategy caches the output data obtained from inference at each current time step, for example, caching the output data of the Nth time step. Using a pre-fitted mapping function, and based on historical data such as the input data of the N+1th time step and the input data of the previous time step, the estimated value of the output data to be inferred in the inference process of the N+1th time step, which has not yet been executed, is predicted. The estimated value of the predicted output data of the N+1th time step is then compared with the output data of the Nth time step. If the difference between adjacent time steps meets a preset condition, such as a difference below a preset threshold, then the output data of the (N+1)th time step is considered to be very similar to the output data of the Nth time step. Therefore, there is no need to execute the inference process of the (N+1)th time step again; the inference process of the N+1th time step can be skipped, and the output data of the Nth time step in the cache can be directly reused as the output data of the N+1th time step. In this way, even when the output data of adjacent time steps are very similar, the output data of the previous time step can be reused, reducing the inference process of multiple time steps, thus reducing the number of inference steps, thereby reducing the computational load, lowering the demand for hardware computing resources, and improving inference efficiency.

[0009] Based on the driving scene reconstruction method provided in the second aspect of the above embodiments of this disclosure, the method first extracts image features from the video, and then predicts depth information and pose information based on the image features through a depth estimation network and a pose estimation network, respectively. The depth information and pose information are then input into a Gaussian prediction network, which obtains target parameters for displaying the 3D Gaussian point cloud. The Gaussian prediction network is a residual network that includes attention calculation and convolution operations, and can output accurate 3D Gaussian point cloud data based on the depth information and pose information. This method, through the collaborative processing mechanism of multiple networks including the depth estimation network, pose estimation network, and Gaussian prediction network, can achieve frame-by-frame dynamic prediction of depth information and pose information. Furthermore, based on the depth information and pose information, the target parameters of the 3D Gaussian point cloud for each frame are further predicted frame-by-frame, thereby combining temporal information to present a 4D driving scene. This method does not require separating dynamic foreground objects from static backgrounds to achieve dynamic scene reconstruction, maintaining extremely high spatial fidelity and avoiding a series of problems caused by dynamic-static separation mechanisms.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0013] Figure 1 This is a schematic diagram of the system architecture of one embodiment of the present disclosure.

[0014] Figure 2 This is a flowchart illustrating an embodiment of the driving scene reconstruction method provided in this disclosure.

[0015] Figure 3 This is a schematic diagram of the process of obtaining a mapping function by fitting in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0016] Figure 4 This is a schematic diagram of the process for determining reused output data in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0017] Figure 5 This is a schematic diagram of the process for determining non-reusable output data in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0018] Figure 6 This is a schematic diagram of a system framework for embedding conditional features in an alternative embodiment of the driving scene reconstruction method provided in this disclosure.

[0019] Figure 7 This is a schematic diagram of the process of embedding conditional features in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0020] Figure 8a This diagram illustrates a time step including conditional branches and unconditional branches in an alternative embodiment of the driving scene reconstruction method provided in this disclosure.

[0021] Figure 8b This is a schematic diagram of the system architecture for applying a dynamic caching strategy to conditional branches with embedded conditional features in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0022] Figure 9 This is a flowchart illustrating the application of a dynamic caching strategy in a conditional branch in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0023] Figure 10 This is a schematic diagram of the data quantization process in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0024] Figure 11 This is a schematic diagram of the process for 4D driving scene reconstruction based on video generated by a video generation model, which is an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0025] Figure 12 This is a schematic diagram of the network architecture of a Gaussian prediction network in an alternative embodiment of the driving scene reconstruction method provided in this disclosure.

[0026] Figure 13 This is a schematic diagram of a process for encoding with reference to context information in an optional embodiment of the driving scene reconstruction method provided in this disclosure.

[0027] Figure 14 This is a flowchart illustrating one embodiment of another driving scene reconstruction method provided in this disclosure.

[0028] Figure 15 This is a schematic diagram of one embodiment of the driving scene reconstruction apparatus provided in this disclosure.

[0029] Figure 16 This is a schematic diagram of an optional embodiment of the driving scene reconstruction apparatus provided in this disclosure.

[0030] Figure 17 A schematic diagram of the structure of one embodiment of a driving scene reconstruction apparatus provided in this disclosure.

[0031] Figure 18 A schematic diagram of the structure of an embodiment of the electronic device provided in this disclosure. Detailed Implementation

[0032] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0033] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0034] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0035] It should also be understood that in the embodiments of this disclosure, "a plurality of" can refer to two or more, and "at least one" can refer to one, two or more.

[0036] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0037] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0038] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0039] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0040] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0041] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0042] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0043] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0044] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0045] Application Overview

[0046] 4D driving scene generation (or synthesis) technology uses AI to construct dynamic driving environment models that include a time dimension. Primarily used in the field of autonomous driving, it can simulate real-world spatiotemporal changes and traffic flow dynamics. The 4D driving scene reconstruction method can consist of two parts: driving video generation and 4D driving scene generation based on the driving video. The 4D driving scene generation based on the driving video can be achieved using 3D Gaussian splatting technology. The core principle of 3D Gaussian splatting is to represent a 3D scene as a set of Gaussian blobs with color, transparency, and spatial attributes, and then superimpose and blend these blobs to achieve efficient rendering. Specifically, 4D driving scene generation can first use a video generation model to generate a temporally consistent driving video that conforms to 3D physical constraints, and then use 3D Gaussian splatting technology to reconstruct a 4D scene with separated dynamic foreground objects and static background. 3D physical constraints are mechanisms used to limit the degrees of freedom of motion of objects, simulating real-world physical behavior by restricting parameters such as the position and rotation of rigid bodies.

[0047] Currently, in the 4D driving scene reconstruction methods of related technologies, some video generation models produce videos with limited durations, such as around 100 frames, which cannot meet the needs of practical 4D driving scene construction. Generating longer time-series videos requires significant computational resources, necessitating higher computing power. Furthermore, the inference efficiency, or video generation efficiency, of these video generation models needs further improvement. For instance, some current models require over 600 seconds of inference time to generate a video of just over 200 frames, indicating slow response speed and insufficient inference efficiency.

[0048] In the generation of 4D driving scenes based on driving videos, the current method employs a motion-static separation mechanism. For example, it separates foreground objects from the background to achieve dynamic scene reconstruction. In the reconstructed scene, foreground objects are dynamic, while the background remains static. This does not match real-world driving scenes, as the background is not always static. Furthermore, this motion-static separation-based reconstruction method may have the following drawbacks: dynamic elements in a static background (such as pedestrians or cyclists) may exhibit motion blur artifacts; consistency is lacking. For example, due to inaccurate segmentation of foreground objects (such as pedestrians) from the background, a pedestrian's hand may be identified as static background in some frames and as foreground in others. This results in a reconstructed driving scene where the person's hand is identified as a foreground object (dynamic) in one frame and as background (static) in the next, leading to poor consistency. Additionally, limitations in computational resources prevent large-scale scene reconstruction.

[0049] In addition, driving scene reconstruction methods in related technologies may have the following limitations: they focus only on continuous video synthesis that lacks three-dimensional representation, or they are limited to static single-scene reconstruction.

[0050] In view of this, embodiments of the present disclosure provide a driving scene reconstruction method and apparatus, electronic device, and program product to solve at least one of the above problems.

[0051] Exemplary System

[0052] The driving scene reconstruction method provided in this exemplary embodiment can be applied to... Figure 1 The exemplary system architecture shown includes a driving video generation (hereinafter referred to as video generation) component and a 4D driving scene generation component.

[0053] The video generation component can include modules such as a video generation model, an indicator (or discrimination module), and a caching module. The video generation model can be, for example,... Figure 1The video diffusion model is shown. During video generation, after obtaining output data at each time step, the output data is stored in a cache. By comparing the fluctuation difference between the output data of two adjacent time steps, if the fluctuation difference meets the preset conditions, the output data of the previous time step is reused. The output data of the previous time step is read from the cache and directly used as the output data of the current time step, skipping the inference process of the current time step, thereby reducing the number of inference steps. For example, the output data of the Nth time step is stored in a cache. Using a pre-fitted mapping function, the estimated value of the output data of the (N+1)th time step is predicted based on the input data of the (N+1)th time step. If the fluctuation difference between the estimated output data of the Nth time step and the output data of the (N+1)th time step meets a preset condition (e.g., whether it is below a preset threshold), the output data of the Nth time step can be reused. The output data of the Nth time step can be read directly from the cache as the output data of the (N+1)th time step, eliminating the need to perform the inference process of the N+th time step. This reduces the number of inference steps. Reducing the number of inference steps can decrease computational load and time consumption when generating videos of the same length, thereby reducing the demand for computing resources and improving the inference efficiency of the video generation model. For example, Figure 1 As shown, in some embodiments, the video generation model can be controlled to generate videos that meet certain conditions by embedding conditions. For example, the conditions can be the motion trajectory of foreground objects, etc. The condition features are incorporated into the middle layer of the video diffusion model by embedding.

[0054] The 4D driving scene generation component can include depth estimation networks, pose estimation networks, and Gaussian prediction networks. Video generated by the video generation component can be used as input for the 4D driving scene generation component. For example, if the image sequence generated by the video generation component is not in video format, it can be decoded first to obtain a video-formatted image sequence, then encoded to obtain image features. These image features are then input into the depth estimation network and pose estimation network to obtain depth and pose information, respectively. Based on the depth and pose information and the image features, the Gaussian prediction network generates a 3D Gaussian point cloud frame by frame (i.e.,...). Figure 1 The frame-by-frame Gaussian point cloud shown in the image is used to generate a 4D driving scene (or model) by arranging multiple frames of 3D Gaussian point clouds in chronological order. This generation process does not require the separation of foreground objects and background, which can avoid motion blur artifacts in dynamic elements such as pedestrians and cyclists, thus improving consistency.

[0055] Exemplary methods

[0056] Figure 2 This is a schematic flowchart of a driving scene reconstruction method provided in an exemplary embodiment of this disclosure, combined with... Figure 1 and Figure 2 This driving scenario reconstruction method may include the following process:

[0057] Step 201: Obtain the input and output data of the video generation model at the Nth time step and the input data at the N+1th time step.

[0058] Where N is an integer, and N≥1. The video generation model can be a video diffusion model, such as a Stable Video Diffusion (SVD) model, a Video Diffusion Model (VDM), a Diffusion Transformer (DiT), etc. The following explanation primarily uses SVD as an example. SVD includes a spatio-temporal attention mechanism, which employs a spatio-temporal block attention mechanism. For example, SVD can include spatial attention layers and temporal attention layers, which can be alternated.

[0059] The Nth time step is the time step corresponding to the Nth inference process.

[0060] Before step 201, the video generation model has already performed N inference processes and is about to perform the (N+1)th inference process. The input and output data of the video generation model at the Nth time step and the input data at the (N+1)th time step are obtained. This means obtaining the input data for the current inference process and the corresponding input and output data for the previous inference process. For example, assuming N inference processes have already been performed, or N time steps of inference have been performed, and the (N+1)th time step of inference is about to be performed, then the input data I at the Nth time step is obtained. N and output data O N and the input data I at the (N+1)th time step N+1 Among them, the input data I at the (N+1)th time step N+1 It can be the output data O at the Nth time step. N The processed data, i.e., the output data of this inference process, is used as the input data for the next inference process. For example, the output data O at the Nth time step... N Transform the dimensions to match the size of the input data; this transformation can be a dimensionality reduction process.

[0061] Step 202: Write the output data of the Nth time step into the buffer.

[0062] The output data obtained from each time step from the first time step to the Nth time step can be written to the cache, or in other words, the output data obtained from each inference process from the first inference process to the Nth inference process can be written to the cache.

[0063] For example, the amount of output data stored in the cache can be determined based on the size of the cache space and the dimension of the output data. For instance, only the output data of one time step can be stored. After obtaining the output data of the Nth inference process, or the Nth time step, the output data of the Nth time step is written to the cache to overwrite the output data of the (N-1)th time step. Alternatively, the output data of several time steps can be retained in the cache; for example, the output data of the NMth to Nth time steps can be stored. M ≥ 1, and M is an integer.

[0064] Step 203: Based on the input and output data at the Nth time step and the input data at the (N+1)th time step, determine the estimated value of the output data at the (N+1)th time step using a mapping function.

[0065] The mapping function is pre-fitted based on samples of multiple input and output data from the video generation model.

[0066] It should be noted that fitting can refer to fitting a functional relationship between a first variable x obtained from the input data and a second variable y obtained from the output data. The fitted mapping function represents the mapping relationship between the first variable x and the second variable y, and y can be obtained from x. The first variable x can be obtained from the input data I, for example, from the input data of two adjacent time steps, as illustrated in the following examples. Similarly, the second variable y can be obtained from the output data O, for example, from the output data of two adjacent time steps. After obtaining the second variable y, the estimated value of the output O can be further obtained based on the relationship between y and the output O. In other embodiments, fitting can refer to fitting a functional relationship between the input data and the output data; that is, the fitted mapping function represents the mapping relationship between the input data I and the output data O, and the output data O can be directly predicted from I.

[0067] Step 204: Compare the fluctuation differences between the output data at the Nth time step and the estimated output data at the (N+1)th time step.

[0068] The output data can be either a vector or a scalar. If the output data is a vector, comparing the fluctuation difference between the estimated values ​​of the output data at time step N and time step N+1 can be done by calculating the similarity between the two vectors. For example, one or more similarity calculation methods such as Cosine Similarity, Euclidean Distance, Manhattan Distance, Chebyshev Distance, and Pearson Correlation can be used to represent the fluctuation difference. If the output data is a scalar, the absolute difference, relative difference, or standardized difference can be calculated.

[0069] Step 205: Based on whether the fluctuation difference between the output data of the Nth time step and the estimated value of the output data of the N+1th time step meets the preset conditions, determine whether to reuse the output data of the Nth time step stored in the cache in the N+1th time step to obtain the output data of the N+1th time step.

[0070] For example, the larger the numerical value used to represent the fluctuation difference between the two, the more similar they are. For instance, if similarity is used to represent the fluctuation difference, a preset condition can be met if the difference between the similarity score and 1 is less than a preset threshold. For example, if the preset threshold is 0.06, and the difference between the similarity score of the output data at the (N+)th time step and the output data at the Nth time step is 0.05, then the preset condition is met. Alternatively, the smaller the numerical value used to represent the fluctuation difference, the more similar they are. For example, if absolute difference is used to represent the difference, such as directly calculating the difference, and the difference is less than a preset threshold, then the preset condition is met.

[0071] Step 206: Obtain the first image sequence to reconstruct the driving scene based on the first image sequence.

[0072] The first image sequence includes the output data from the first time step to the (N+1)th time step.

[0073] The video is obtained after decoding the first image sequence.

[0074] Figure 2The illustrated embodiment provides a dynamic caching strategy. By using a pre-fitted mapping function, the output data of the (N+1)th time step that has not yet been executed is predicted to obtain its estimated value. The strategy compares the fluctuation differences between the output data of two adjacent time steps, for example, comparing the fluctuation difference between the estimated value of the output data of the (N+1)th time step and the output data of the Nth time step. Based on the fluctuation difference, it determines whether to reuse the output data of the Nth time step. If the fluctuation difference meets a preset condition, the output data of the Nth time step can be directly reused without executing the (N+1)th inference process, i.e., without executing the inference process of the (N+1)th time step. In the inference process, the output data of the Nth time step is directly read from the cache as the output data of the N+1th time step. This reduces the number of inference steps. For example, if the number of inference steps before adopting the above dynamic caching strategy is 300, the number of inference steps can be reduced to less than 150 after adopting the above dynamic caching strategy. The reduction in the number of inference steps can save computing resources and reduce the demand for computing resources. On the other hand, since some inference processes are skipped directly, under the premise of generating videos of the same time sequence length, adopting the above dynamic caching strategy can significantly reduce the inference time compared with not adopting the above caching strategy, thereby improving inference efficiency.

[0075] Applying the dynamic caching strategy described above, videos with 233 frames or more can be generated quickly. Experimental tests show that generating long-sequence videos with 233 frames or more can reduce inference time from 615 seconds to 309 seconds. At a frame rate of 12 FPS, 233 frames is nearly 20 seconds of video, which is considered a long-sequence video in the current field of AI-based video generation.

[0076] Furthermore, the dynamic caching strategy proposed in this method only requires a cache storage space corresponding to the output data of one time step, which can save cache storage space.

[0077] Figure 2 In the illustrated embodiment, the mapping function in step 203 can be pre-fitted based on samples of multiple input and output data. For example, in an optional embodiment, a least squares fitting method is used, such as... Figure 3 As shown, in this embodiment, the following steps may be included before step 203:

[0078] Step 301: Obtain multiple input data and corresponding multiple output data of the video generation model as multiple samples.

[0079] After determining the video generation model for generating driving videos, the input and output data of the video generation model at multiple time steps are pre-acquired as samples. For example, the input and output data of the video generation model at 100-5000 time steps are obtained as samples.

[0080] Step 302: Obtain the ratio of the difference between the input data at time step N+1 and the input data at time step N to the input data at time step N, and use it as the first variable.

[0081] For example, x represents the first variable, x = (I N+1 -I N ) / I N Among them, I N+1 I represents the input data at the (N+1)th time step. N This represents the input data at the Nth time step.

[0082] Step 303: Obtain the ratio of the difference between the output data of the (N+1)th time step and the output data of the Nth time step in multiple samples to the output data of the Nth time step, and use it as the second variable.

[0083] For example, y represents the second variable, y = (O N+1 -O N ) / O N Among them, O N+1 This represents the output data at the (N+1)th time step, O N This represents the output data at the Nth time step.

[0084] Step 304: Using the least squares method, fit multiple first variables and multiple second variables to obtain a multi-order polynomial as the mapping function.

[0085] A mapping function represents a functional relationship that maps a first variable to a second variable. For example, a fourth-order polynomial can be obtained by fitting a data using the least squares method, where the independent variable is x and the dependent variable is y.

[0086] After obtaining the mapping function, based on the input and output data at time step N and the input data at time step N+1, the estimated value of the output data at time step N+1 can be determined using the mapping function. For example, based on the input data I at time step N... N and the input data I at the (N+1)th time step N+1 According to x=(I N+1 -I N ) / I N The value of the first variable x is calculated, and this value is input into a mapping function, for example, into a fourth-order polynomial, to calculate the value of the second variable y. The output data is O. N Given that, according to y = (O N+1 -O N ) / O N Then O can be calculated. N+1This gives us an estimate of the output data at the (N+1)th time step.

[0087] Figure 3 The embodiments shown are merely examples. In other embodiments, the fitting method may also be the Random Sample Consensus (RANSAC) fitting method.

[0088] Figure 3 The illustrated embodiment uses least squares and other fitting methods to fit the relationship between multiple input and output data samples, obtaining a mapping function between the first variable x and the second variable y. Then, based on the input data at the (N+1)th time step, the first variable x is obtained. Using the first variable x and the mapping function, the second variable y corresponding to the inference process at the (N+1)th time step (which has not yet been executed) is predicted. The relationship between y and the output data at the (N+1)th time step is determined, and the output data at the (N+1)th time step can be inferred from y. This allows for prediction before the (N+1)th (i.e., the...)th time step is executed. In the case of an inference process with N+1 time steps, the output data of the N+1 time step is predicted in advance based on the input data and other data of the N+1 time step. This provides an accurate basis for determining whether to reuse the output data of the Nth time step, reducing invalid reuse caused by inaccurate discrimination of output data differences. If the output data difference is too large, reusing the output data will result in the loss of some feature information of the output data, that is, the input data of the next inference process will lose some feature information, which will affect the quality of the generated video. Accurate prediction of output data can ensure that the quality of the video generated by the video generation model does not decrease significantly.

[0089] In one alternative embodiment, based on Figure 2 The illustrated embodiment, as Figure 4 As shown, step 205 may specifically include the following process:

[0090] Step 401: Based on the fact that the fluctuation difference between the output data of the Nth time step and the estimated value of the output data of the N+1th time step meets the preset condition, determine the output data of the Nth time step stored in the N+1th time step multiplexing cache.

[0091] Meeting a preset condition can be defined as the numerical value representing the fluctuation difference between the estimated values ​​of the output data at time step N and time step N+1 being greater than or equal to a first preset threshold, or less than a second preset threshold. For example, if the fluctuation difference is represented by the similarity between the estimated values ​​of the output data at time step N and time step N+1, then a similarity greater than the first preset threshold indicates that the preset condition is met. For instance, if the first preset threshold is 0.92, a similarity greater than 0.92 indicates that the preset condition is met. If the fluctuation difference is represented by the difference between 1 and the similarity, then a similarity less than the second preset threshold indicates that the preset condition is met. For instance, if the second preset threshold is 0.05, and the similarity is 0.96, then 1 - 0.96 = 0.04, and 0.04 < 0.05, thus meeting the preset condition.

[0092] Following step 401, the following steps may also be included:

[0093] Step 402: In response to determining the output data of the Nth time step stored in the N+1th time step multiplexing buffer, the output data of the Nth time step is used as the output data of the N+1th time step.

[0094] In other words, if it is determined that the output data of the previous time step will be reused, the inference process to be executed in this round will be skipped. That is, the inference process of the Nth time step will not be executed, and the output data obtained by the inference of the Nth time step will be directly used as the output data of the N+1th time step.

[0095] Or, based on Figure 2 The illustrated embodiment, as Figure 5 As shown, step 205 may specifically include the following process:

[0096] Step 501: Based on the fact that the fluctuation difference between the output data of the Nth time step and the estimated value of the output data of the N+1th time step does not meet the preset condition, it is determined that the output data of the Nth time step stored in the cache will not be reused in the N+1th time step.

[0097] If the preset conditions are not met, it can be that the numerical value representing the fluctuation difference between the estimated values ​​of the output data at time step N and time step N+1 is less than a first preset threshold, or that the numerical value representing the fluctuation difference between the estimated values ​​of the output data at time step N and time step N+1 is greater than a second preset threshold. For example, if the fluctuation difference is represented by the similarity between the estimated values ​​of the output data at time step N and time step N+1, then if the similarity is less than the first preset threshold, the preset conditions are not met. For example, if the first preset threshold is 0.92, a similarity less than 0.92 is considered not to meet the preset conditions. If the fluctuation difference is represented by the difference between 1 and the similarity, then if the difference between 1 and the similarity is greater than the second preset threshold, the preset conditions are met. For example, if the second preset threshold is 0.05, the similarity is 0.85, 1 - 1.85 = 0.15, and 0.15 > 0.05, then the preset conditions are not met.

[0098] Following step 501, the following steps may also be included:

[0099] Step 502: In response to determining that the output data of the Nth time step stored in the buffer is not reused at the N+1th time step, the input data of the N+1th time step is input to the video generation model so that the video generation model performs the inference process of the N+1th time step and outputs the output data of the N+1th time step.

[0100] If the output data obtained from the previous time step is not reused, the video generation model needs to perform the N+1th inference process to obtain the output data of the N+1th time step.

[0101] After obtaining the output data of the (N+1)th time step, write it to the cache and predict the estimated value of the output data of the (N+2)th time step. Compare the fluctuation difference between the output data of the (N+1)th time step and the estimated value of the output data of the (N+2)th time step. If the fluctuation difference meets the preset condition, skip the inference process of the (N+2)th time step and directly use the output data of the (N+1)th time step as the output data of the (N+2)th time step.

[0102] Figure 4 or Figure 5In the illustrated embodiment, the system determines whether to reuse the output data generated in the previous time step based on whether the fluctuation difference between the output data of two adjacent time steps meets a preset condition. If the preset condition is met, the output data of the previous time step is reused and the inference process of the current time step is skipped. If the preset condition is not met, the output data of the previous time step is not reused and the inference process of the current time step is executed. In this way, dynamic caching can be achieved based on the real-time comparison results (fluctuation differences). Compared with static caching, which requires caching the output data of each time step, dynamic caching can reduce the occupation of cache space and save cache resources.

[0103] In one alternative embodiment, based on Figure 2 In the illustrated embodiment, the video generation model can be a video diffusion model that supports the addition of conditions. Conditions refer to guiding information or constraints input during the generation process. The methods for adding conditions can include: conditional input: providing text descriptions, tags, etc., as input data to the model, directly affecting the generation result; conditional embedding: integrating conditional information into the parameters of each layer of the generation model to ensure that multiple network layers follow specific constraints; and loss function adjustment: modifying the loss function during the training phase to include conditional constraints, thereby optimizing the controllability of the generation result. In this optional embodiment, control conditions are added using conditional embedding, such as... Figure 6 As shown, for example, the conditions to be embedded include trajectories, 3D bounding boxes, high-precision maps, and text. The trajectory is used to constrain the motion trajectory of foreground objects in the generated video; the 3D bounding box, i.e., the 3D bounding box corresponding to the foreground object, is used to constrain the specific position and boundaries of the foreground object in the image; the high-precision map is used to provide or constrain background scene information; and the text is text that can be input by the user to represent the natural semantics of the video content to be generated, such as... Figure 6 The "real-world driving scenario" shown is illustrated. Based on the above conditions, feature extraction is performed to obtain individual features (called conditional features). Multiple conditional features are input into the ControlNet model, where feature fusion is performed. The fused feature (the first feature) is embedded into the intermediate layer of the video diffusion model. The intermediate layer of the video diffusion model can be at least one network layer between the input layer and the output layer.

[0104] Specifically, in combination Figure 6 ,like Figure 7 As shown, prior to step 201, this method may also include the following steps:

[0105] Step 701: Obtain at least one condition feature corresponding to at least one condition.

[0106] For example, to obtain Figure 6 The conditions shown include trajectory, 3D bounding box, high-precision map, text, and other conditions, each corresponding to a specific feature.

[0107] Step 702: Input at least one conditional feature into the ControlNet model, and obtain the fused first feature through the ControlNet model.

[0108] Step 703: Embed the first feature into the intermediate layer of the video generation model.

[0109] An intermediate layer embedded in a video generation model can be achieved by fusing the first feature with the output of the previous layer and using it as the input to the next layer. The fusion method could be vector concatenation, for example.

[0110] Figure 6 and Figure 7 In the illustrated embodiment, embedding conditions can generate videos that meet the constraints, making the generated videos more in line with user expectations and improving the controllability of the video content, thereby facilitating the acquisition of more accurate and controllable 4D driving scenarios.

[0111] It should be noted that, as Figure 8a As shown, the inference process of each time step from the 1st to the (N+1th)th time step in the video diffusion model can include a first inference process with conditional embedding (conditional branch) and a second inference process with unconditional embedding (unconditional branch). In other words, the inference process corresponding to a time step can actually include conditional and unconditional branches. In the inference process of each time step, the same input data is input to the conditional branch and the unconditional branch respectively, resulting in the output data corresponding to the conditional branch and the output data corresponding to the unconditional branch respectively. The two output data can be fused to obtain the fused output data as the output data of the entire time step. For example, the fusion method can be weighted summation. The specific calculation formula is as follows:

[0112] V_pred=pred_uncond+guidance_scale*(pred_cond-pred_uncond)

[0113] Where V_pred represents the fused output data, pred_uncond represents the output data of the unconditional branch, guidance_scale represents the weight, which can take values ​​from 2 to 10; and pred_cond represents the output data of the conditional branch.

[0114] like Figure 8b As shown, in an optional embodiment, the above dynamic caching strategy can be applied only to conditional branches, that is, only the output data of conditional branches is dynamically cached, while unconditional branches do not use the above dynamic caching strategy. Specifically, based on Figure 2 The illustrated embodiment, as Figure 9As shown, this optional embodiment may specifically include the following steps:

[0115] Step 901: Obtain the first input data and first output data corresponding to the first inference process of the video generation model at the Nth time step, and the first input data at the N+1th time step.

[0116] It should be noted that the input data for the conditional branch and the unconditional branch are the same in each time step. That is, the first input data of the first inference process (i.e., the conditional branch) is the same as the second input data of the second inference process (the unconditional branch), which is the input data for each time step.

[0117] Step 902: Write the first output data obtained from the first inference process corresponding to the Nth time step into the cache.

[0118] That is, the first output data (pred_cond) obtained from the conditional branch in the Nth time step is written into the buffer.

[0119] Step 903: Based on the first input data and first output data at the Nth time step and the first input data at the N+1th time step, determine the first estimated value of the first output data corresponding to the first inference process in the N+1th time step through a mapping function.

[0120] The first input data refers to the input data for the entire time step. The first output data refers to the output data obtained from the conditional branch. Since this optional embodiment applies the dynamic caching strategy only to the conditional branch, it is only necessary to predict the estimated value of the first output data of the conditional branch. By comparing the fluctuation difference between the estimated values ​​of the first output data of the conditional branch at the Nth time step and the first output data at the (N+1)th time step, it is determined whether to reuse the first output data obtained from the conditional branch at the Nth time step. Due to the embedded condition, the first output data obtained from the conditional branch is different from the second output data obtained from the unconditional branch.

[0121] Step 904: Based on whether the fluctuation difference between the first output data obtained from the first inference process at the Nth time step and the first estimated value meets the preset conditions, determine whether to reuse the first output data obtained from the first inference process at the Nth time step stored in the cache.

[0122] It should be noted that dynamic caching strategies can be applied to both conditional and unconditional branches; that is, dynamic caching strategies can be applied to at least one of conditional and unconditional branches. Figure 9The illustrated embodiment applies a dynamic caching strategy to the conditional branches of the video diffusion model, while the unconditional branches sequentially execute the inference process from the first time step to the (N+1)th time step. Compared to applying the dynamic caching strategy to both conditional and unconditional branches, applying it only to the conditional branches halves the computational cost based on prediction, as only the estimated value of the first output data of the conditional branch needs to be predicted, without needing to predict the estimated value of the second output data of the unconditional branch. The conditional branches, due to the embedding of conditional features, further increase computational complexity. Executing the inference process at each time step would consume significant computational resources and be time-consuming. By employing a dynamic caching strategy for the conditional branches, caching the first output data and reusing it when preset conditions are met, the number of inference steps can be reduced, computational cost lowered, and model inference efficiency improved without affecting the quality of the generated video.

[0123] In one optional embodiment, the video generation model is a video diffusion model, specifically a video diffusion model including a spatial attention layer and a temporal attention layer, such as SVD. The temporal and spatial attention mechanisms in the video diffusion model optimize generation efficiency and quality by separating the spatiotemporal dimensions. The temporal attention mechanism focuses on the temporal coherence between video frames, reducing the generation of duplicate frames and improving the efficiency of long video processing by dynamically adjusting the time step and diffusion steps. The spatial attention mechanism optimizes the generation quality of a single frame, reducing computational complexity. The combination of these two mechanisms enables end-to-end inference acceleration and improves generation efficiency. Furthermore, temporal attention ensures consistency across different frames in the temporal dimension by encoding temporal information, avoiding jumps or repetitions in the generated content and guaranteeing content coherence or consistency. Spatial attention enhances the visual quality of a single frame, ensuring that each frame is coherent and logical, reducing visual discontinuities caused by inconsistencies between frames. In this optional embodiment, in order to further reduce the amount of computation and reduce the demand and consumption of computing resources, each element in the query vector Q, key vector K, value vector V and attention score vector P involved in the attention mechanism is quantized. Taking SVD as an example, Q, K, V and P that need to be processed by the spatial attention layer and the temporal attention layer can be quantized respectively.

[0124] Specifically, based on Figure 2 In the embodiment shown, the following steps may be included before step S201:

[0125] The inference process is performed at least one of the time steps from the first time step to the Nth time step using a video diffusion model.

[0126] Among them, such as Figure 10 As shown, during the inference process at each time step in at least one time step, the method further includes:

[0127] Step 1001: Convert each element in the query vector Q and key vector K processed by the attention layer into four-digit integer INT4 data.

[0128] For example, the attention layer can be a spatial attention layer and / or a temporal attention layer.

[0129] Step 1002: Convert each element in the value vector V and attention score vector P processed by the attention layer into 8-bit floating-point FP8 data.

[0130] Where Q represents the query, K represents the key, and V represents the value. The query vector Q, key vector K, and value vector V are obtained by mapping the input data at each time step through different weight matrices, or by linearly transforming the input data at each time step. The different weight matrices are trainable parameters in the video diffusion model. P represents the attention score obtained after calculating the inner product of the query vector Q and the key vector K and then normalizing it.

[0131] Figure 10 In the illustrated embodiment, to address the significant computational load of the attention component in the diffusion model, quantization is used to quantize Q, K, V, and P in the attention operation. The inventors discovered that reducing the precision of Q and K data has minimal impact on the performance of the video diffusion model in generating videos. Therefore, INT4 quantization was performed on Q and K, converting them into INT4 data for computation, reducing computational load without affecting video generation quality. However, reducing the precision of V and the attention score vector P has a more significant impact on the video generation quality. Therefore, FP8 quantization was performed on V and P. This method of applying different quantization techniques to different data can reduce computational precision to decrease computational resource requirements while maintaining video generation quality and improving inference efficiency. Experimental tests show that the quantization method provided in this optional embodiment can further reduce inference time by 31 seconds.

[0132] Thus, if combined Figure 2 The above dynamic caching strategies in other embodiments and Figure 10 The quantization processing method shown, compared with the original video diffusion model that did not adopt the above dynamic caching strategy and quantization processing method, can reduce the model inference time from 615 seconds to 278 seconds or less when generating a 233-frame long temporal video.

[0133] Combination Figure 1 As shown, based on Figure 2 In the illustrated embodiment, after obtaining the first image sequence, the first image sequence can be input into the 4D driving scene reconstruction architecture to generate a 4D driving scene. For example... Figure 11As shown, after obtaining the first image sequence, the method further includes the following steps:

[0134] Step 1101: Obtain image features based on the first image sequence.

[0135] Combination Figure 1 As shown, if the first image sequence is not in video format, it can be decoded first to obtain multiple frames of video format images. Then, the multiple frames of images are encoded to obtain image features.

[0136] Step 1102: Input the image features into the pre-trained depth estimation network and the pre-trained pose estimation network respectively. The depth estimation network outputs depth information, and the pose estimation network outputs pose information.

[0137] Pre-trained means that the model has been pre-trained and its performance meets the expected metrics. For example, meeting the expected metrics can be achieved by training the model in a supervised manner until the model converges.

[0138] For example, the depth estimation network could be DepthNet, and the pose estimation network could be PoseNet.

[0139] The depth information output by the depth estimation network can specifically be multiple frames of depth images. The pose information output by the pose estimation network can specifically be the camera pose information corresponding to each of the multiple frames.

[0140] Step 1103: Input the depth information and pose information into the pre-trained Gaussian prediction network, and obtain the target parameters for displaying the 3D Gaussian point cloud through the Gaussian prediction network.

[0141] It should be noted that the camera pose information output by the pose estimation network is in the camera coordinate system. It needs to be converted to pose information in the world coordinate system before being input into the Gaussian prediction network.

[0142] Gaussian prediction networks can be residual networks (ResNet) that include attention computation and convolution operations.

[0143] Target parameters are the various parameters required to generate a 3D Gaussian point cloud.

[0144] Step 1104: Reconstruct the driving scenario based on the target parameters and time information.

[0145] Figure 11The embodiment shown in this disclosure proposes a novel network architecture for reconstructing 4D driving scenes. Unlike the motion-static separation mechanism used in related technologies, this method does not require foreground object and background segmentation, avoiding a series of problems caused by the motion-static separation mechanism. For example, it avoids motion blur artifacts and poor consistency in dynamic elements (such as pedestrians and cyclists) in a static background caused by foreground object and background segmentation, thus improving the consistency of dynamic or static elements in the generated scene and maintaining extremely high spatial fidelity. Experimental tests show that the reconstruction scheme based on the above network architecture can generate a complete 4D scene model within seconds, meeting the stringent requirements of real-time applications such as simulation while ensuring that the quality of the reconstructed driving scene is not compromised.

[0146] based on Figure 11 The illustrated embodiment, as Figure 12 As shown, a Gaussian prediction network can include multiple sets of residual networks (ResNet); one set of residual networks includes multiple serially connected residual blocks and a head connected to the multiple residual blocks, and the set of residual networks corresponds to the output of a target parameter.

[0147] For example, the target parameters used to construct a 3D Gaussian point cloud may include four parameters: covariance, position, opacity, and spherical harmonic function. This corresponds to setting up four sets of residual networks, with each residual network outputting one set of parameters. It should be noted that, due to space limitations, only two sets of residual networks are shown. In other embodiments, the number of residual networks can be adaptively adjusted according to actual needs; these embodiments are not listed individually in this disclosure.

[0148] In each set of residual networks, at least two residual blocks among the serially connected residual blocks are skipped, for example, as shown below. Figure 12 As shown, the first residual block and the last residual block are skipped and connected in serial order.

[0149] like Figure 12 As shown, one of the multiple residual blocks includes a first convolutional layer, a first activation layer, an attention layer, and a batch normalization layer. The first convolutional layer performs convolution operations on the input data of the Gaussian prediction network, which includes pose information, depth information, and image features.

[0150] The first activation layer uses the Rectified Linear Unit (ReLU) activation function, which introduces non-linear characteristics into the neural network and enhances the model's expressive power. This first activation layer is used to activate the output of the first convolutional layer.

[0151] An attention layer is a network layer used to perform attention calculations on the input object through an attention mechanism. The input object can be the output data of the previous layer; for example, an attention layer is used to perform attention calculations on the output data of the first activation layer.

[0152] The Batch Normalization (BN) layer is used to perform batch normalization on the data output from the attention layer.

[0153] like Figure 12 As shown, a set of residual networks corresponds to a head network (or simply head). A head network includes a second convolutional layer and a second activation layer. For example, the second activation layer also uses the ReLU function. The second convolutional layer performs convolution operations on the output data of the residual networks, and the second activation layer activates the output data of the second convolutional layer using the ReLU function.

[0154] In related technologies, reconstructing a 4D driving scene requires the use of a 3D Gaussian splash model, while this disclosure... Figure 12 The embodiment shown demonstrates a novel Gaussian prediction network architecture based on a neural network, which can replace the 3D Gaussian splash model. This architecture can output target parameters for reconstructing 3D Gaussian point clouds frame by frame based on the input depth information, pose estimation information, and image features, thereby obtaining multi-frame 3D Gaussian point clouds. When these multi-frame 3D Gaussian point clouds are played in chronological order, a 4D driving scene is obtained. This Gaussian prediction network can maintain the consistency of the 4D scene while reconstructing the 3D Gaussian model frame by frame.

[0155] It's important to note that during the training phase, the depth estimation network, pose estimation network, and Gaussian prediction network can be trained using pre-captured real-world videos, rather than image sequences generated by the video generation model. Real-world videos do not require decoding; they are directly encoded to obtain image features. For example, panoramic view images or temporally adjacent images can be used as training data for these networks. Panoramic view images refer to images captured or acquired through a panoramic lens, and videos captured or acquired through a panoramic lens include multiple frames of panoramic view images. Temporally adjacent images are several images that are adjacent on the timeline within the video. The Gaussian prediction network predicts 3D Gaussian parameters based on multiple frames from the captured video, thereby reconstructing a 3D Gaussian point cloud. In the application phase, the video obtained by decoding the image sequence generated by the video generation model can be used as input to reconstruct the 3D Gaussian point cloud and further obtain a reconstructed 4D driving scene.

[0156] Furthermore, it should be noted that in related technologies, reconstruction methods based on static / dynamic separation, besides the problems caused by the static / dynamic separation mechanism itself, also require on-site training and learning based on user-input videos to improve adaptability to different types of videos. This means online learning based on user-provided videos, sometimes requiring tens or even hundreds of thousands of iterative inference processes, which is extremely time-consuming. The network architecture for reconstructing 4D driving scenes proposed in this disclosure can pre-train a Gaussian prediction network using videos of various categories or scenes as training data, enabling it to learn the features of various videos under different scene categories and possess prior knowledge of different scene categories. This eliminates the need for further online learning for different types of videos during the application phase.

[0157] In one alternative embodiment, based on Figure 11 The illustrated embodiment, as Figure 13 As shown, the image features corresponding to the first image sequence can be obtained through the following process:

[0158] Step 1301: Decode the first image sequence to obtain a second image sequence in 2D video format.

[0159] If the input data itself is in video format, then decoding is not required.

[0160] For example, decoding can be implemented using a decoder based on a variational autoencoder (VAE), and encoding can be implemented using a VAE encoder.

[0161] Step 1302: Based on the time attention mechanism, the images of the (M-1)th frame, the Mth frame, and the M+1th frame in the second image sequence are encoded to obtain the encoded Mth frame image, and the encoded multi-frame images are used as image features.

[0162] M is an integer, and M≥2. Specifically, for the first frame image in the second image sequence, the first frame image and the second frame image are encoded to obtain the encoded first frame image.

[0163] This embodiment only illustrates the process of fusing the features of the frame before and after the Mth frame with the features of the Mth frame image during encoding to obtain the encoded Mth frame image. In fact, in other embodiments, the features of the images of the first 2 or 3 frames and the features of the images of the last 2 or 3 or more frames can also be fused with the Mth frame image, and the fused image can be used as the encoded Mth frame image.

[0164] Figure 13In the illustrated embodiment, the previous frame image carries contextual information, and the next frame image carries contextual information. During encoding, the features of the previous and next frames are fused with those of the current frame image, which is equivalent to referencing contextual information. The reconstruction method achieved by jointly utilizing historical and future contextual information can effectively capture dynamic scene elements, maintain the consistency between dynamic and static elements in the scene, and achieve extremely high spatial fidelity.

[0165] It should be noted that, Figure 1 The driving video generation section and the 4D driving scene generation section shown can be used independently. For example, Figure 1 The right side of the diagram shows the driving scene generation section, which can use a user-specified video as input to generate the corresponding 4D driving scene. Figure 1 The driving video generation section on the left can also generate image sequences for other purposes, not limited to video material for reconstructing 4D driving scenes.

[0166] like Figure 14 As shown, Figure 14 This is a flowchart illustrating a driving scene reconstruction method provided in another exemplary embodiment of this disclosure. The method may include the following steps:

[0167] Step 1401: Obtain image features based on pre-acquired video or a first image sequence generated in real time.

[0168] Step 1402: Input the image features into the pre-trained depth estimation network and the pre-trained pose estimation network respectively. The depth estimation network outputs depth information, and the pose estimation network outputs pose information.

[0169] Step 1403: Input the depth information and pose information into the pre-trained Gaussian prediction network, and obtain the target parameters for displaying the 3D Gaussian point cloud through the Gaussian prediction network.

[0170] The Gaussian prediction network is a residual network that includes attention calculation and convolution operations.

[0171] Step 1404: Reconstruct the driving scenario based on the target parameters and time information.

[0172] Figure 14 In the illustrated embodiment, the image features can be the first image sequence output by the video generation model, or images or videos pre-captured in the real world using image acquisition devices such as cameras.

[0173] Any of the driving scene reconstruction methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the driving scene reconstruction methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the driving scene reconstruction methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0174] Exemplary device

[0175] Figure 15 This is a schematic diagram of one embodiment of the driving scene reconstruction apparatus of this disclosure. The apparatus of this embodiment can be used to implement the corresponding driving scene reconstruction method embodiments of this disclosure. Figure 15 The device shown includes: an acquisition module 1500, a cache module 1501, a mapping module 1502, a comparison module 1503, a discrimination module 1504, and an output module 1505.

[0176] The acquisition module 1500 is used to acquire the input and output data of the video generation model at the Nth time step, as well as the input data at the (N+1)th time step. Here, N is an integer, N≥1.

[0177] The cache module 1501 is used to write the output data of the Nth time step into the cache.

[0178] The mapping module 1502 is used to determine the output data at the (N+1)th time step based on the input and output data at the Nth time step and the input data at the (N+1)th time step, using a mapping function. The mapping function is pre-fitted based on samples of multiple input and output data from the video generation model.

[0179] The comparison module 1503 is used to compare the fluctuation difference between the output data at the Nth time step and the estimated value of the output data at the N+1th time step.

[0180] The discrimination module 1504 is used to determine whether the output data of the Nth time step stored in the cache is reused in the N+1th time step to obtain the output data of the N+1th time step, based on whether the fluctuation difference between the output data of the Nth time step and the estimated value of the output data of the N+1th time step meets the preset conditions.

[0181] Output module 1505 is used to obtain a first image sequence to reconstruct a driving scene based on the first image sequence; the first image sequence includes output data from the first time step to the N+1th time step.

[0182] like Figure 16 As shown, Figure 16This is a schematic diagram of another embodiment of the driving scene reconstruction device disclosed herein. The mapping module 1502 may specifically include a first acquisition unit 15021, a second acquisition unit 15022, a third acquisition unit 15023, and a mapping unit 15024.

[0183] The first acquisition unit 15021 is used to acquire multiple input data and corresponding multiple output data of the video generation model as multiple samples.

[0184] The second acquisition unit 15022 is used to acquire the ratio of the difference between the input data at the (N+1)th time step and the input data at the Nth time step in multiple samples to the input data at the Nth time step, as the first variable.

[0185] The third acquisition unit 15023 is used to acquire the ratio of the difference between the output data of the (N+1)th time step and the output data of the Nth time step in multiple samples to the output data of the Nth time step, as the second variable.

[0186] The mapping unit 15024 is used to fit multiple first variables and multiple second variables using the least squares method to obtain a multi-order polynomial as the mapping function.

[0187] Here, the mapping function represents the functional relationship from the first variable to the second variable.

[0188] In an optional embodiment, the discrimination module is used to determine the output data of the Nth time step stored in the N+1 time step multiplexing cache based on the fact that the fluctuation difference between the estimated values ​​of the output data of the Nth time step and the output data of the N+1 time step meets a preset condition.

[0189] Alternatively, if the fluctuation difference between the estimated values ​​of the output data at the Nth time step and the output data at the N+1th time step does not meet the preset conditions, it is determined that the output data of the Nth time step stored in the cache will not be reused at the N+1th time step.

[0190] The apparatus also includes a response module, configured to, in response to determining that the output data of the Nth time step is stored in the N+1th time step multiplexing buffer, use the output data of the Nth time step as the output data of the N+1th time step;

[0191] Alternatively, in response to determining that the output data of the Nth time step stored in the buffer is not reused at the N+1th time step, the input data of the N+1th time step is input to the video generation model, so that the video generation model performs the inference process of the N+1th time step and outputs the output data of the N+1th time step.

[0192] In one optional embodiment, the video generation model is a video diffusion model, which includes an attention layer; the apparatus further includes:

[0193] The execution module is used to perform the inference process of at least one of the time steps from the first time step to the Nth time step through the video diffusion model.

[0194] During the inference process at each time step in at least one time step, the execution module further includes a quantization unit. The quantization unit converts the query vector Q and key vector K processed by the attention layer into four-bit integer INT4 data; and converts the value vector V and attention score vector P processed by the attention layer into 8-bit floating-point FP8 data. The query vector Q, key vector K, and value vector V are obtained by mapping the input data through different weight matrices. The attention score vector P is obtained by calculating the inner product of the query vector Q and key vector K and then normalizing it.

[0195] In an optional embodiment, the apparatus further includes a condition embedding module, which is configured to: acquire at least one condition feature corresponding to at least one condition; input the at least one condition feature into a ControlNet model to obtain a fused first feature through the ControlNet model; and embed the first feature into an intermediate layer of the video generation model.

[0196] In an optional embodiment, the inference process for each time step from the 1st time step to the (N+1th time step) includes a conditionally embedded first inference process and an unconditionally embedded second inference process. An acquisition module is used to acquire the first input data and first output data corresponding to the first inference process of the video generation model at the Nth time step, and the first input data at the (N+1th time step). A caching module is used to write the first output data obtained from the first inference process corresponding to the Nth time step into a cache. A prediction module is used to determine, based on the first input data and first output data at the Nth time step, and the first input data at the (N+1th time step), a first estimated value of the first output data corresponding to the first inference process in the (N+1th time step) using a mapping function. A discrimination module is used to determine whether the first output data obtained from the first inference process at the Nth time step, stored in the cache, should be reused at the (N+1th time step) based on whether the fluctuation difference between the first output data obtained from the first inference process at the Nth time step and the first estimated value meets a preset condition.

[0197] In an optional embodiment, the above-described apparatus further includes:

[0198] The first feature fusion module is used to obtain image features based on the first image sequence.

[0199] The first parameter estimation module is used to input image features into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively. The depth estimation network outputs depth information, and the pose estimation network outputs pose information.

[0200] The first parameter prediction module is used to input depth information, pose information, and image features into a pre-trained Gaussian prediction network, which then obtains the target parameters for displaying the 3D Gaussian point cloud. The Gaussian prediction network is a residual network that includes attention calculation and convolution operations.

[0201] The first reconstruction module is used to reconstruct the driving scene based on target parameters and time information.

[0202] In one optional embodiment, the target parameter includes multiple parameters such as covariance, position, opacity, and spherical harmonic function. The Gaussian prediction network includes multiple sets of residual networks; one set of residual networks includes multiple serially connected residual blocks and a head connected to the residual blocks, and each set of residual networks outputs a target parameter; one of the residual blocks includes a first convolutional layer, a first ReLU layer, an attention layer, and a batch normalization layer; at least two residual blocks are skip-connected; the head includes a second convolutional layer and a second ReLU layer.

[0203] In an optional embodiment, the feature fusion module is specifically used for: decoding the first image sequence to obtain a second image sequence in 2D video format; encoding the images of the (M-1)th frame, the Mth frame, and the M+1th frame in the second image sequence based on a time attention mechanism to obtain the encoded Mth frame image; where M is an integer and M≥2; encoding the first frame image in the second image sequence with the second frame image to obtain the encoded first frame image; and using the encoded multi-frame images as image features.

[0204] Figure 17 This is a schematic diagram of one embodiment of the driving scene reconstruction apparatus of this disclosure. The apparatus of this embodiment can be used to implement the corresponding driving scene reconstruction method embodiments of this disclosure. Figure 17 The device shown includes: a second feature fusion module 1701, a second parameter estimation module 1702, a second parameter prediction module 1703, and a second reconstruction module 1704.

[0205] The second feature fusion module 1701 is used to acquire image features corresponding to pre-acquired video or real-time generated first image sequence.

[0206] The second parameter estimation module 1702 is used to input image features into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively, and output depth information through the depth estimation network and pose information through the pose estimation network.

[0207] The second parameter prediction module 1703 is used to input depth information and pose information into a pre-trained Gaussian prediction network, and obtain target parameters for displaying 3D Gaussian point clouds through the Gaussian prediction network; the Gaussian prediction network is a residual network that includes attention calculation and convolution operation.

[0208] The second reconstruction module 1704 is used to reconstruct the driving scene based on target parameters and time information.

[0209] Exemplary electronic devices

[0210] Below, for reference Figure 18 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0211] Figure 18 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0212] like Figure 18 As shown, the electronic device includes one or more processors and memory.

[0213] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0214] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the driving scene reconstruction methods of the various embodiments of this disclosure described above and / or other desired functions.

[0215] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0216] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0217] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0218] Of course, for the sake of simplicity, Figure 18 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0219] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the driving scene reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0220] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0221] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the driving scene reconstruction method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0222] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0223] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0224] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0225] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0226] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0227] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0228] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0229] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for reconstructing a driving scene, characterized in that, The method includes: Obtain at least one conditional feature corresponding to at least one condition; input the at least one conditional feature into the ControlNet model, and obtain a fused first feature through the ControlNet model; embed the first feature into the intermediate layer of the video generation model; Obtain the input and output data of the video generation model at time step N and the input data at time step N+1; N is an integer, N≥1; Write the output data of the Nth time step into the cache; Based on the input and output data at the Nth time step and the input data at the (N+1)th time step, an estimated value of the output data at the (N+1)th time step is determined using a mapping function; the mapping function is pre-fitted based on samples of multiple input and output data of the video generation model. Compare the fluctuation difference between the estimated values ​​of the output data at the Nth time step and the output data at the (N+1)th time step; Based on whether the fluctuation difference between the estimated values ​​of the output data of the Nth time step and the output data of the N+1th time step meets the preset conditions, it is determined whether the output data of the Nth time step stored in the cache is reused in the N+1th time step to obtain the output data of the N+1th time step. A first image sequence is obtained to reconstruct a driving scene based on the first image sequence; the first image sequence includes output data from the first time step to the N+1th time step.

2. The method according to claim 1, characterized in that, Before determining the estimated value of the output data at the (N+1)th time step using a mapping function, the method further includes: Multiple input data and corresponding multiple output data of the video generation model are obtained as multiple samples; The ratio of the difference between the input data at the (N+1)th time step and the input data at the Nth time step in the plurality of samples to the input data at the Nth time step is used as the first variable; The ratio of the difference between the output data at the (N+1)th time step and the output data at the Nth time step in the plurality of samples to the output data at the Nth time step is used as a second variable; The least squares method is used to fit multiple first variables and multiple second variables to obtain a multi-order polynomial as a mapping function; the mapping function represents the functional relationship from the first variables to the second variables.

3. The method according to claim 1, characterized in that, Based on whether the fluctuation difference between the estimated values ​​of the output data at the Nth time step and the output data at the (N+1)th time step meets a preset condition, determining whether to reuse the output data of the Nth time step stored in the cache at the (N+1)th time step includes: Based on the fact that the fluctuation difference between the estimated values ​​of the output data at the Nth time step and the output data at the (N+1)th time step meets a preset condition, it is determined that the output data of the Nth time step stored in the cache will be reused at the (N+1)th time step. or, Based on the fact that the fluctuation difference between the estimated values ​​of the output data of the Nth time step and the output data of the N+1th time step does not meet the preset condition, it is determined that the output data of the Nth time step stored in the cache will not be reused in the N+1th time step. After determining whether the output data of the Nth time step stored in the cache is reused at the N+1th time step, the method further includes: In response to determining that the output data of the Nth time step stored in the cache is reused at the N+1th time step, the output data of the Nth time step is used as the output data of the N+1th time step; or, In response to determining that the output data of the Nth time step stored in the cache is not reused at the N+1th time step, the input data of the N+1th time step is input to the video generation model so that the video generation model performs the inference process of the N+1th time step and outputs the output data of the N+1th time step.

4. The method according to claim 1, characterized in that, The video generation model is a video diffusion model, which includes an attention layer; before acquiring the input and output data of the video generation model at time step N and the input data at time step N+1, the method further includes: The inference process of at least one of the time steps from the first time step to the Nth time step is performed through the video diffusion model; During the inference process of executing any one of the at least one time step, the method further includes: The query vector Q and key vector K processed by the attention layer are converted into four-digit integer INT4 data. The value vector V and attention score vector P processed by the attention layer are converted into 8-bit floating-point FP8 data; wherein, the query vector Q, the key vector K and the value vector V are obtained by mapping the input data through different weight matrices; the attention score vector P is obtained by calculating the inner product of the query vector Q and the key vector K and then normalizing it.

5. The method according to claim 1, characterized in that, The reasoning process for each time step from the first time step to the (N+1)th time step includes a first reasoning process with conditional embedding and a second reasoning process with unconditional embedding. Obtain the input and output data of the video generation model at time step N, and the input data at time step N+1, including: Obtain the first input data and first output data corresponding to the first inference process of the video generation model at the Nth time step, and the first input data at the N+1th time step; Writing the output data of the Nth time step into the cache includes: Write the first output data obtained from the first inference process corresponding to the Nth time step into the cache; Based on the input and output data at the Nth time step and the input data at the (N+1)th time step, an estimated value of the output data at the (N+1)th time step is determined using a mapping function, including: Based on the first input data and the first output data at the Nth time step, and the first input data at the N+1th time step, a first estimated value of the first output data corresponding to the first inference process in the N+1th time step is determined by a mapping function. Based on whether the fluctuation difference between the estimated values ​​of the output data at the Nth time step and the output data at the (N+1)th time step meets a preset condition, determining whether to reuse the output data of the Nth time step stored in the cache at the (N+1)th time step includes: Based on whether the fluctuation difference between the first output data obtained from the first inference process at the Nth time step and the first estimated value meets a preset condition, determine whether to reuse the first output data obtained from the first inference process at the Nth time step stored in the cache.

6. The method according to any one of claims 1-5, characterized in that, After obtaining the first image sequence, the method further includes: Based on the first image sequence, image features are obtained; The image features are input into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively. The depth estimation network outputs depth information, and the pose estimation network outputs pose information. The depth information, pose information, and image features are input into a pre-trained Gaussian prediction network, which then generates target parameters for displaying the 3D Gaussian point cloud. The Gaussian prediction network is a residual network that includes attention calculation and convolution operations. Based on the target parameters and time information, the driving scenario is reconstructed.

7. The method according to claim 6, characterized in that, The target parameters include multiple parameters such as covariance, position, opacity, and spherical harmonic function. The Gaussian prediction network includes multiple sets of residual networks; one set of residual networks includes multiple residual blocks connected in series and a header connected to the multiple residual blocks, and the set of residual networks outputs a target parameter. One of the plurality of residual blocks includes a first convolutional layer, a first ReLU layer, an attention layer, and a batch normalization layer; At least two of the plurality of residual blocks are skip-connected; The head includes a second convolutional layer and a second ReLU layer.

8. The method according to claim 6, characterized in that, Obtaining the image features corresponding to the first image sequence includes: Decode the first image sequence to obtain a second image sequence in 2D video format; Based on the time attention mechanism, the images of the (M-1)th frame, the Mth frame, and the M+1th frame in the second image sequence are encoded to obtain the encoded Mth frame image; M is an integer and M≥2; for the first frame image in the second image sequence, the first frame image and the second frame image are encoded to obtain the encoded first frame image; the encoded multi-frame images are used as the image features.

9. A method for reconstructing a driving scene, characterized in that, The method includes: Image features are obtained based on pre-acquired video or a first image sequence generated in real time. The image features are input into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively. The depth estimation network outputs depth information, and the pose estimation network outputs pose information. The depth information, pose information, and image features are input into a pre-trained Gaussian prediction network, which then generates target parameters for displaying the 3D Gaussian point cloud. The Gaussian prediction network is a residual network that includes attention calculation and convolution operations. Based on the target parameters and time information, the driving scenario is reconstructed.

10. A driving scene reconstruction device, characterized in that, The device includes: The conditional embedding module is used to obtain at least one conditional feature corresponding to at least one condition; input the at least one conditional feature into the ControlNet model, and obtain the fused first feature through the ControlNet model; and embed the first feature into the intermediate layer of the video generation model. The acquisition module is used to acquire the input and output data of the video generation model at the Nth time step and the input data at the (N+1)th time step; N is an integer, N≥1; The caching module is used to write the output data of the Nth time step into the cache; The mapping module is used to determine the output data of the (N+1)th time step based on the input data and output data of the Nth time step and the input data of the (N+1)th time step through a mapping function; the mapping function is obtained in advance based on samples of multiple input data and multiple output data of the video generation model. The comparison module is used to compare the fluctuation difference between the estimated values ​​of the output data at the Nth time step and the output data at the (N+1)th time step; The discrimination module is used to determine whether the output data of the Nth time step stored in the cache is reused at the N+1 time step to obtain the output data of the N+1 time step, based on whether the fluctuation difference between the estimated values ​​of the output data of the Nth time step and the output data of the N+1 time step meets a preset condition. An output module is used to obtain a first image sequence to reconstruct a driving scene based on the first image sequence; the first image sequence includes output data from the first time step to the N+1th time step.

11. A driving scene reconstruction device, characterized in that, The device includes: The feature fusion module is used to acquire image features corresponding to pre-acquired video or real-time generated first image sequence; The parameter estimation module is used to input the image features into a pre-trained depth estimation network and a pre-trained pose estimation network, respectively, and output depth information through the depth estimation network and pose information through the pose estimation network. The parameter prediction module is used to input the depth information and pose information into a pre-trained Gaussian prediction network, and obtain the target parameters for displaying the 3D Gaussian point cloud through the Gaussian prediction network; the Gaussian prediction network is a residual network that includes attention calculation and convolution operation; The reconstruction module is used to reconstruct the driving scenario based on the target parameters and time information.

12. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-9.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.

14. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.

Images