Dynamic streetscape reconstruction method and device based on four-dimensional Gaussian scene graph

Abstract

The invention discloses a dynamic street scene reconstruction method and device based on a four-dimensional Gaussian scene graph, which are used for synthesizing a new visual angle image of a dynamic street scene by optimizing four-dimensional Gaussian and realizing high-fidelity reconstruction of the street scene. The method comprises the following steps: giving an image of a dynamic street scene, a camera pose and image annotation information, respectively optimizing a static street background and a dynamic object through four-dimensional Gaussian, and combining into a four-dimensional Gaussian scene graph to represent the dynamic street scene; each node in the four-dimensional Gaussian scene graph stores a group of four-dimensional Gaussian parameters; and high-fidelity reconstruction of a dynamic street scene is realized based on the four-dimensional Gaussian of the graph. And giving any view angle of the dynamic street scene, selecting a corresponding four-dimensional Gaussian node from the four-dimensional Gaussian scene graph, and combining four-dimensional Gaussian according to the image annotation information to render the image of the dynamic street. According to the method, high-fidelity modeling of the dynamic street scene is realized through the four-dimensional Gaussian scene graph, and the rendering quality of the dynamic street scene is improved.

Description

Technical Field

[0001] The present invention belongs to the field of scene reconstruction, and relates to a dynamic street scene reconstruction method and device based on a four-dimensional Gaussian scene graph. Background Art

[0002] In the existing street scene reconstruction technologies, most methods focus on static street scenes. Recently, some three-dimensional Gaussian-based reconstruction methods have achieved the reconstruction of dynamic street scenes through three-dimensional Gaussian combination methods, but these methods can only handle rigid objects such as vehicles and have no way to reconstruct non-rigid objects such as pedestrians. Summary of the Invention

[0003] The purpose of the present invention is to propose a dynamic street scene reconstruction method and device based on a four-dimensional Gaussian scene graph in view of the deficiencies of the prior art. Based on the input view information, the corresponding four-dimensional Gaussian nodes are selected from the four-dimensional Gaussian scene graph to represent the geometry and appearance of the dynamic street scene corresponding to the view, and the loss value is calculated with the original image, and each four-dimensional Gaussian node is optimized through backpropagation to achieve the reconstruction of the dynamic street scene.

[0004] The purpose of the present invention is achieved through the following technical solutions: A dynamic street scene reconstruction based on a four-dimensional Gaussian scene graph, the method comprising:

[0005] (1) Given the images, camera poses, lidar data, and image annotation information of the dynamic street scene, a four-dimensional Gaussian scene graph is constructed to represent the dynamic street scene;

[0006] (2) Based on the input view information, the corresponding four-dimensional Gaussian nodes are selected from the four-dimensional Gaussian scene graph for sputtering and rasterization, and the loss value is calculated with the original image, and the four-dimensional Gaussian scene graph is optimized through backpropagation to achieve scene reconstruction.

[0007] Further, in step (1), the specific process of constructing the four-dimensional Gaussian scene graph is: Given a view, according to the image annotation information corresponding to the view, multiple four-dimensional Gaussian nodes are created, and different four-dimensional Gaussian nodes respectively reconstruct elements such as static backgrounds, dynamic vehicles, and dynamic pedestrians in the image, thereby constructing a four-dimensional Gaussian scene graph.

[0008] Furthermore, in step (1), given a perspective, the specific process of creating multiple four-dimensional Gaussian nodes according to the image annotation information corresponding to the perspective is as follows: First, use the annotation information of multiple frames of images to obtain the bounding boxes of dynamic vehicles and dynamic pedestrians in the world coordinate system. Filter the radar point cloud with the bounding boxes of vehicles or pedestrians in each frame of the image to obtain the radar point cloud of a certain vehicle or a certain pedestrian in each frame. Then, superimpose the radar point clouds of the same vehicle or the same pedestrian to make the radar point cloud denser. The densified multiple groups of point clouds will be used to initialize multiple groups of four-dimensional Gaussians. Subsequently, multiple groups of four-dimensional Gaussians are fitted with the images to further optimize each four-dimensional Gaussian node in the four-dimensional Gaussian scene graph.

[0009] Furthermore, in step (2), based on the input perspective information, the specific process of selecting the corresponding four-dimensional Gaussian nodes from the four-dimensional Gaussian scene graph for sputtering and rasterization to achieve scene reconstruction is as follows: Use the image annotation information to organize multiple groups of four-dimensional Gaussians into a four-dimensional Gaussian scene graph, and then implement combined rendering. Calculate the loss value between the rendering result and the original image, and optimize each four-dimensional Gaussian node in the four-dimensional Gaussian scene graph through backpropagation, thereby achieving the reconstruction of the scene.

[0010] In a second aspect, the present invention also provides a dynamic street scene reconstruction device based on a four-dimensional Gaussian scene graph, including a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, the described dynamic street scene reconstruction device based on a four-dimensional Gaussian scene graph is implemented.

[0011] In a third aspect, the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the described dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph is implemented.

[0012] In a fourth aspect, the present invention also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the described dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph is implemented.

[0013] Advantages of the present invention: The present invention uses a four-dimensional Gaussian scene graph composed of a group of four-dimensional Gaussian nodes to represent a dynamic street scene, thereby achieving a high-fidelity reconstruction of the dynamic street scene. In order to represent the moving vehicles and pedestrians in the dynamic street scene, the present invention provides a four-dimensional Gaussian scene graph that can organize multiple groups of four-dimensional Gaussians. The present invention also proposes a Gaussian initialization strategy for four-dimensional Gaussians for vehicles and pedestrians. While reconstructing the static background and dynamic vehicles in the dynamic street scene, it ensures the high-fidelity modeling of complex non-rigid bodies such as vehicles and pedestrians, improves the modeling ability of moving vehicles and pedestrians in the dynamic street scene, and realizes the complete representation of the dynamic street scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of a dynamic street scene reconstruction method based on a four - dimensional Gaussian scene graph provided by the present invention.

[0015] Figure 2 It is a schematic diagram of constructing a four - dimensional Gaussian scene graph from a series of dynamic street scene images in the present invention.

[0016] Figure 3 It is a schematic diagram of reconstructing a dynamic street scene using a four - dimensional Gaussian scene graph in the present invention.

[0017] Figure 4 It is a structural diagram of a dynamic street scene reconstruction device based on a four - dimensional Gaussian scene graph in the present invention. Detailed implementation manners

[0018] The following further describes the technical details and principles of the present invention in conjunction with the accompanying drawings:

[0019] As Figure 1 shown, the present invention proposes a dynamic street scene reconstruction method based on a four - dimensional Gaussian scene graph. It includes:

[0020] Constructing a four - dimensional Gaussian scene graph from a series of dynamic street scene images: Specifically, given the images of the dynamic street scene, camera poses, lidar data, and image annotation information, a four - dimensional Gaussian scene graph is constructed to represent the dynamic street scene;

[0021] Reconstructing the dynamic street scene using the four - dimensional Gaussian scene graph: Specifically, based on the input view information, the corresponding four - dimensional Gaussian nodes are selected from the four - dimensional Gaussian scene graph for sputtering and rasterization, and the loss value is calculated with the original image. The four - dimensional Gaussian scene graph is optimized through backpropagation to achieve scene reconstruction.

[0022] As Figure 2 shown, in the dynamic street scene reconstruction method based on a four - dimensional Gaussian scene graph proposed by the present invention, the specific steps of constructing a four - dimensional Gaussian scene graph from a series of dynamic street scene images are as follows:

[0023] 1. Given a view, according to the image annotation information corresponding to the view, first use the annotation information of multiple frames of images to obtain the bounding boxes of dynamic vehicles and dynamic pedestrians in the world coordinate system. The lidar point cloud of each frame of image is screened by the bounding boxes of vehicles or pedestrians, and the lidar point cloud of a certain vehicle or a certain pedestrian in each frame is obtained. Then, the lidar point clouds of the same vehicle or the same pedestrian are superimposed to make the lidar point cloud denser. The densified multiple groups of point clouds will be used to initialize multiple groups of four - dimensional Gaussians.

[0024] 2. Initialize multiple sets of four - dimensional Gaussians using the densified multiple sets of point clouds. Each set of four - dimensional Gaussians corresponds to elements such as static background, dynamic vehicles, and dynamic pedestrians in the image, and then construct a four - dimensional Gaussian scene graph. The four - dimensional Gaussian used in the present invention is defined by a set of parameters, including position μ, range S, rotation R, opacity α, and spherical harmonic coefficients SH. The formula for the Gaussian function G is defined as:

[0025] G(x) = exp(-0.5(x - μ) T Σ -1 (x - μ))

[0026] Σ = RSS T R T

[0027] where Σ is the covariance matrix of the Gaussian, and x is the spatial position information of the Gaussian. During the rendering process, the four - dimensional Gaussians are combined through the opacity α, and the color of each point is determined by the spherical harmonic coefficients SH.

[0028] As Figure 3 shown, in the dynamic street scene reconstruction method based on the four - dimensional Gaussian scene graph proposed by the present invention, based on the input view information, the specific process of selecting the corresponding four - dimensional Gaussian nodes from the four - dimensional Gaussian scene graph for sputtering and rasterization to achieve scene reconstruction is as follows:

[0029] 1. Use the image annotation information of the given view to combine the corresponding four - dimensional Gaussian nodes and connect these nodes according to spatial and temporal relationships to form a dynamic four - dimensional Gaussian scene graph. This process not only considers the information in the spatial dimension but also introduces the time dimension, enabling the scene graph to better reflect the changes in the dynamic street scene. Subsequently, based on the constructed four - dimensional Gaussian scene graph, combined rendering is performed. By superimposing and fusing multiple four - dimensional Gaussian nodes, a rendered image corresponding to the input view is generated. This step makes full use of the spatial distribution and time evolution characteristics of the four - dimensional Gaussian nodes, enabling the rendering result to more accurately reflect the details and dynamic changes of the actual scene.

[0030] 2. After the rendering is completed, compare the generated rendered image with the original input image and calculate the loss value between the two. The calculation of the loss value is based on the pixel - level difference to ensure that the rendering result is highly consistent with the original image visually. Through the backpropagation algorithm, the loss value is propagated layer - by - layer back to each four - dimensional Gaussian node in the four - dimensional Gaussian scene graph, and then the parameters of the nodes are optimized and adjusted. This optimization process not only improves the accuracy of individual four - dimensional Gaussian nodes but also enhances the overall consistency of the entire scene graph. Finally, after multiple iterative optimizations, the nodes in the four - dimensional Gaussian scene graph can more accurately represent the spatio - temporal characteristics of the dynamic street scene, thus achieving high - quality scene reconstruction.

[0031] Corresponding to the embodiments of the foregoing dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph, the present invention also provides embodiments of a dynamic street scene reconstruction apparatus based on a four-dimensional Gaussian scene graph. Refer to Figure 4 An embodiment of a dynamic street scene reconstruction apparatus based on a four-dimensional Gaussian scene graph provided by an embodiment of the present invention includes a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it is used to implement a dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph in the foregoing embodiment.

[0032] Embodiments of a dynamic street scene reconstruction apparatus based on a four-dimensional Gaussian scene graph provided by the present invention can be applied to any device with data processing capabilities. Such a device with data processing capabilities can be a device or apparatus such as a computer. The apparatus embodiments can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful apparatus, it is formed by a processor of any device with data processing capabilities reading corresponding computer program instructions in a non-volatile memory into the memory for operation. From a hardware perspective, as Figure 4 shown, it is a hardware structure diagram of any device with data processing capabilities where a dynamic street scene reconstruction apparatus based on a four-dimensional Gaussian scene graph provided by the present invention is located. In addition to Figure 4 the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities where the apparatus in the embodiment is located usually also includes other hardware according to the actual functions of the device with data processing capabilities. Details thereof will not be elaborated herein.

[0033] The specific implementation processes of the functions and roles of each unit in the foregoing apparatus are specifically detailed in the implementation processes of the corresponding steps in the foregoing method, and will not be elaborated herein.

[0034] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial descriptions of the method embodiments. The apparatus embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0035] Embodiments of the present invention also provide a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements a dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph in the foregoing embodiment.

[0036] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0037] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the dynamic street view reconstruction method based on a four-dimensional Gaussian scene graph described above.

[0038] The above embodiments are used to explain the present invention, rather than limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph, characterized in that The method includes: (1) Given an image of a dynamic street scene, camera poses, radar data, and image annotation information, constructing a four-dimensional Gaussian scene graph for representing the dynamic street scene; (2) Based on the dynamic street scene information, selecting corresponding four-dimensional Gaussian nodes from the four-dimensional Gaussian scene graph for sputtering and rasterization, calculating the loss value with the original image, and optimizing the four-dimensional Gaussian scene graph through backpropagation to achieve scene reconstruction.

2. The dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph according to claim 1, wherein In step (1), the specific process of constructing the four-dimensional Gaussian scene graph is as follows: Given a perspective, according to the image annotation information corresponding to the perspective, creating multiple four-dimensional Gaussian nodes, and different four-dimensional Gaussian nodes respectively reconstruct the static background, dynamic vehicles, and dynamic pedestrian elements in the image, thereby constructing the four-dimensional Gaussian scene graph.

3. A dynamic street view reconstruction method based on a four-dimensional Gaussian scene graph according to claim 2, characterized in that In step (1), the specific process of creating multiple four-dimensional Gaussian nodes given a perspective according to the image annotation information corresponding to the perspective is as follows: First, use the annotation information of multiple frames of images to obtain the bounding boxes of dynamic vehicles and dynamic pedestrians in the world coordinate system, screen the radar point cloud with the bounding box of the vehicle or pedestrian in each frame of the image to obtain the radar point cloud of a certain vehicle or a certain pedestrian in each frame, and then superimpose the radar point clouds of the same vehicle or the same pedestrian to make the radar point cloud denser; the densified multiple groups of point clouds will be used to initialize multiple groups of four-dimensional Gaussians, and then the multiple groups of four-dimensional Gaussians are fitted with the image, thereby optimizing each four-dimensional Gaussian node in the four-dimensional Gaussian scene graph.

4. A dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph according to claim 1, characterized in that, In step (2), the specific process of selecting corresponding four-dimensional Gaussian nodes from the four-dimensional Gaussian scene graph for sputtering and rasterization based on the input perspective information to achieve scene reconstruction is as follows: Use the image annotation information to organize multiple groups of four-dimensional Gaussian nodes into a four-dimensional Gaussian scene graph, thereby realizing combined rendering. Calculate the loss value between the rendering result and the original image, and optimize each four-dimensional Gaussian node in the four-dimensional Gaussian scene graph through backpropagation, thereby realizing the reconstruction of the scene.

5. A dynamic street scene reconstruction device based on a four-dimensional Gaussian scene graph, comprising a memory and one or more processors, wherein executable code is stored in the memory, characterized in that, When the processor executes the executable code, it implements a dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph as described in any one of claims 1-4.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph as described in any one of claims 1-4.

7. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, it implements a dynamic street scene reconstruction method based on a four-dimensional Gaussian scene graph as described in any one of claims 1-4.

Images