A scene reconstruction method, device and equipment

Abstract

The present disclosure belongs to the field of autonomous driving technology, and specifically relates to a scene reconstruction method, device and equipment. The method comprises: acquiring images of a target area from multiple perspectives, predicting a 3D occupancy grid of the target area based on the images, separating a dynamic vehicle from a static background based on semantic information in the 3D occupancy grid, and generating a corresponding initial point cloud; generating a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, respectively forming a three-dimensional Gaussian model of the dynamic vehicle and the static background; predicting the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, and completing the combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory.

Description

Technical Field

[0001] The present disclosure belongs to the field of autonomous driving technology, and specifically relates to a scene reconstruction method, device, and equipment. Background Art

[0002] To test and verify the safety and reliability of autonomous driving algorithms, virtual simulation environments have become an essential component of autonomous driving development. The construction of autonomous driving simulation environments typically relies on 3D scene reconstruction technology, whose core task is to convert data collected by multiple sensors (such as lidar and cameras) into realistic 3D scene models.

[0003] Currently, autonomous driving scene reconstruction systems require expensive LiDAR sensors to generate high-precision point cloud data to reconstruct 3D scenes. However, the high cost and complex data processing of LiDAR limit its economic viability in large-scale applications. Furthermore, LiDAR only captures geometric information and lacks rich semantic information (such as object category and dynamic state), which poses a challenge to fully understanding complex scenes.

[0004] At the same time, dynamic scene reconstruction relies on accurate modeling and estimation of object motion trajectories, which typically requires large pre-labeled datasets for training and validating algorithm performance. However, labeling the trajectories and poses of dynamic objects is a time-consuming and laborious task, especially in complex driving environments with numerous objects and complex motion states, making manual labeling extremely difficult. Summary of the Invention

[0005] The disclosed embodiments propose a scene reconstruction solution to provide a low-cost, label-free, computationally-efficient, and highly robust and accurate autonomous driving scene reconstruction method.

[0006] A first aspect of an embodiment of the present disclosure provides a scene reconstruction method, comprising:

[0007] Acquire images of a target area from multiple perspectives, predict a 3D occupancy grid of the target area based on the images, separate dynamic vehicles from static background in the 3D occupancy grid based on semantic information, and generate a corresponding initial point cloud;

[0008] Generating a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, respectively forming a three-dimensional Gaussian model of the dynamic vehicle and the static background;

[0009] By comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, the trajectory of the dynamic vehicle is predicted, and based on the trajectory, the combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space is completed.

[0010] In some embodiments of the present disclosure, separating the dynamic vehicle from the static background in the 3D occupancy grid based on semantic information includes:

[0011] Extracting a pixel-level vehicle object based on the detection and segmentation model. If the displacement of the vehicle object at adjacent time stamps is greater than a preset threshold, the vehicle object is considered a dynamic vehicle.

[0012] Whether the grid belongs to the dynamic vehicle is distinguished based on the semantic information of each grid in the 3D occupancy grid; the 3D occupancy grid of the dynamic vehicle is composed of all grids belonging to the dynamic vehicle, and the 3D occupancy grid of the static background is composed of all grids not belonging to the dynamic vehicle.

[0013] In some embodiments of the present disclosure, generating the corresponding initial point cloud includes:

[0014] Upsampling a 3D occupancy grid of the dynamic vehicle for voxels exceeding a preset threshold to generate points of an initial point cloud of the dynamic vehicle, wherein the color of the points is the same as the color of a pixel projected onto a two-dimensional image plane for acquiring image data of the target area;

[0015] generating a first point cloud of the static background based on the 3D occupancy grid of the static background, wherein positions of points in the first point cloud are taken from center coordinates of the grid;

[0016] A second point cloud is generated based on COLMAP for the multi-view images of the target area, and the first point cloud of the static background is aggregated with the second point cloud to generate an initial point cloud of the static background.

[0017] In some embodiments of the present disclosure, generating a corresponding Gaussian ellipsoid for each point in the initial point cloud includes:

[0018] For each point in the initial point cloud, a corresponding Gaussian ellipsoid is generated based on the pose covariance matrix and the position matrix of the point, wherein the pose covariance matrix includes a rotation matrix and a scaling matrix;

[0019] The Gaussian ellipsoid further includes an opacity parameter and a set of spherical association coefficients characterizing the appearance of the target area, and the opacity parameter and the spherical association coefficients are initialized according to a preset rule;

[0020] Points corresponding to the Gaussian ellipsoid are projected onto a two-dimensional image plane of the image data of the target area to obtain corresponding pixel points, and the opacity parameter and spherical association coefficient of the Gaussian ellipsoid are optimized based on the attributes of the pixel points.

[0021] In some embodiments of the present disclosure, predicting the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area includes:

[0022] Predicting a position matrix and a rotation matrix of a Gaussian ellipsoid of the dynamic vehicle in each frame based on a dynamic vehicle prediction model, projecting the Gaussian ellipsoid onto a two-dimensional image plane for obtaining image data of the target area, and adjusting the dynamic vehicle prediction model based on a difference between a projected point and a corresponding pixel point;

[0023] The posture of the dynamic vehicle is represented by a position matrix and a rotation matrix of the Gaussian ellipsoid of the dynamic vehicle;

[0024] The postures of the dynamic vehicle in multiple frames are tracked to form a trajectory of the dynamic vehicle.

[0025] In some embodiments of the present disclosure, completing the combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory includes:

[0026] Combining the three-dimensional Gaussian model of the dynamic vehicle and the static background based on the trajectory to generate a scene model;

[0027] The scene model is projected into the target image space to complete the rendering of the target area in the target image space.

[0028] In some embodiments of the present disclosure, generating a scene model by combining the three-dimensional Gaussian model of the dynamic vehicle and the static background based on the trajectory includes:

[0029] The coordinate system of the three-dimensional Gaussian model of the dynamic vehicle and the static background is set as the world coordinate system;

[0030] Obtaining a rendering timestamp, and combining the three-dimensional Gaussian model of the dynamic vehicle and the static background at the rendering timestamp based on the spherical association coefficient of the Gaussian ellipsoid to form a scene snapshot;

[0031] A plurality of frames of scene snapshots are generated based on the timestamp sequence of the trajectory to form the scene model.

[0032] In some embodiments of the present disclosure, projecting the scene model into a target image space includes:

[0033] All the Gaussian ellipsoids of the scene model are projected into a target image space and parameters of a pixel point projected into the target image space are calculated based on the parameters of each Gaussian ellipsoid, wherein the color of the pixel point is calculated by the opacity of the Gaussian ellipsoid.

[0034] A second aspect of the embodiments of the present disclosure provides a scene reconstruction method, including:

[0035] a separation module, configured to acquire images of a target area from multiple perspectives, predict a 3D occupancy grid of the target area based on the images, separate dynamic vehicles from static background in the 3D occupancy grid based on semantic information, and generate a corresponding initial point cloud;

[0036] a generating module, configured to generate a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, thereby forming three-dimensional Gaussian models of the dynamic vehicle and the static background respectively;

[0037] A rendering module is used to predict the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, and complete the combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory.

[0038] A third aspect of the embodiments of the present disclosure provides a scene reconstruction device, including a memory and a processor,

[0039] The memory is used to store computer programs;

[0040] The processor is configured to implement the method described in the first aspect of the present disclosure when executing the computer program.

[0041] In summary, the scene reconstruction methods, devices, and apparatuses provided in the various embodiments of the present disclosure avoid using expensive LiDAR sensors to generate point clouds by generating the initial point cloud in 3DGS scene reconstruction based on the occupancy grid predicted by a trained occupancy prediction network. They also avoid manual data labeling by separating dynamic and static objects based on the semantic information of the grid and using the position and trajectory of dynamic objects as learnable parameters to predict the trajectory. Furthermore, system verification has shown that the method described in the present disclosure has low computational complexity and strong robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0043] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0044] Figure 2 is an instance of a surround view image acquired by a camera deployed on an autonomous vehicle;

[0045] Figure 3 This is a schematic diagram of the OG-Gaussian algorithm framework;

[0046] Figure 4 is a flowchart of a scene reconstruction method according to some embodiments of the present disclosure;

[0047] Figure 5 is based on Figure 2 The occupancy grid predicted by the surround image is shown;

[0048] Figure 6 It is a specific process of generating initial point clouds of dynamic vehicles and static background based on occupancy grids in some embodiments of the present disclosure;

[0049] Figure 7 is an example of a parameter-optimized three-dimensional Gaussian model in one embodiment of the present disclosure;

[0050] Figure 8 It’s a real street scene;

[0051] Figure 9 It is a street scene reconstructed based on OG-Gaussian;

[0052] Figure 10 Yes Figure 8 The result of reconstructing the dynamic vehicle in the target image space separately;

[0053] Figure 11 Yes Figure 8 The result of reconstructing the static background in the target image space separately;

[0054] Figure 12 Comparison of OG-Gaussian and other algorithms on the waymoopen dataset;

[0055] Figure 13 is a schematic diagram of a scene reconstruction device according to some embodiments of the present disclosure;

[0056] Figure 14 is a schematic diagram of a scene reconstruction device according to an embodiment of the present disclosure; DETAILED DESCRIPTION

[0057] In the detailed description that follows, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing between different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.

[0058] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly on, connected to, coupled to, or in communication with the other device, unit, or module, or there may be intervening devices, units, or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated listed items.

[0059] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0060] These and other features and characteristics of the present disclosure, as well as the methods of operation, the functions of the related elements of the structure, the combination of parts, and the economy of manufacture may be better understood with reference to the following description and accompanying drawings, which form a part of this specification. However, it is to be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0061] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the preceding or following structures are not intended to limit the present disclosure. The scope of protection of the present disclosure is subject to the claims.

[0062] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1 The computer system shown includes a rendering server connected to multiple image sensor data. The image sensors obtain image data of a target area from multiple angles. The target area generally includes a dynamic vehicle and a static background. The rendering server completes three-dimensional reconstruction of the target area in the target image space based on the image data.

[0063] The image sensor can be a variety of monocular or binocular cameras or monocular or binocular cameras. It is used to obtain image data of the target area from multiple angles. Typically, in the construction of an autonomous driving simulation environment, the image sensor is a plurality of cameras deployed in an autonomous driving vehicle, which can obtain surround images of the target area, such as Figure 2 shown.

[0064] The rendering server acquires image data of the target area from multiple angles, renders the image data in the target image space based on the image data, and generates a three-dimensional reconstruction of the target area. The rendering server can be any of a single machine, a cluster, or a distributed server.

[0065] To test and verify the safety and reliability of autonomous driving algorithms, virtual simulation environments have become an essential component of autonomous driving development. The construction of autonomous driving simulation environments typically relies on 3D scene reconstruction technology, whose core task is to convert data collected by multiple sensors (such as lidar and cameras) into realistic 3D scene models.

[0066] Currently, autonomous driving scene reconstruction systems require expensive LiDAR sensors to generate high-precision point cloud data to reconstruct 3D scenes. However, the high cost and complex data processing of LiDAR limit its economic viability in large-scale applications. Furthermore, LiDAR only captures geometric information and lacks rich semantic information (such as object category and dynamic state), which poses a challenge to fully understanding complex scenes.

[0067] At the same time, dynamic scene reconstruction relies on accurate modeling and estimation of object motion trajectories, which typically requires large pre-labeled datasets for training and validating algorithm performance. However, labeling the trajectories and poses of dynamic objects is a time-consuming and laborious task, especially in complex driving environments with numerous objects and complex motion states, making manual labeling extremely difficult.

[0068] In view of this, the present disclosure proposes a scene reconstruction method named OG-Gaussian to reconstruct the autonomous driving scene at low cost and high efficiency. Figure 3 This is a schematic diagram of the OG-Gaussian algorithm framework. Figure 3 As shown in the figure, OG-Gaussian first uses the trained occupancy prediction network to predict the occupied grid, then separates the dynamic vehicle grid based on the semantic information in the occupied grid and the speed difference between the previous and next frames, processes the dynamic vehicle grid into a denser point cloud, and processes the remaining grid into the background point cloud. These two groups of point clouds are then used as the initial point clouds for 3DGS reconstruction. 3DGS will initialize a set of parameterized ellipsoids at the positions of these point clouds. After parameter optimization, the two optimized ellipsoids of dynamic vehicles and static streets are combined into a hybrid model based on the predicted trajectory of the dynamic vehicle, and then projected into 2D space to render a realistic reconstruction scene of the target area containing the dynamic vehicle.

[0069] Figure 4 is a flow chart of a scene reconstruction method according to some embodiments of the present disclosure. In some embodiments, the scene reconstruction method is Figure 1The rendering server shown in the figure executes the scene reconstruction method, which includes the following steps:

[0070] S410 , acquiring images of a target area from multiple perspectives, predicting a 3D occupancy grid of the target area based on the images, separating a dynamic vehicle from a static background in the 3D occupancy grid based on semantic information, and generating a corresponding initial point cloud.

[0071] In one embodiment of the present disclosure, the image data is only the surround view image acquired by the camera fixed on the vehicle (eg Figure 2 As shown), after acquiring the surround view image, the surround view image is input into the trained 3D occupancy prediction network to predict the 3D occupancy grid. Figure 2 The occupancy grid predicted by the surround image is as follows Figure 5 shown.

[0072] The autonomous driving scene contains multiple moving vehicles. The present disclosure uses mature detection and segmentation models to extract vehicle objects at the pixel level. After obtaining the occupied grid, because each grid of the occupied grid naturally has semantic information, the present disclosure distinguishes whether the grid belongs to a vehicle based on the semantic information of each grid, thereby extracting the grid of the vehicle, and when the displacement of the vehicle at adjacent timestamps is greater than a preset threshold, the vehicle is determined to be a dynamic vehicle. The grid occupied by the dynamic vehicle is upsampled, and the sampling points are projected onto the two-dimensional image plane of the image data of the target area. The sampling points are colored based on the color of the projected pixel points. The colored sampling points constitute the initial point cloud of the dynamic vehicle with color information.

[0073] Finally, the present disclosure converts the remaining occupied grids into a dense point cloud of the static background, with the positions taken from their center coordinates. In addition, the dense point cloud is aggregated with the point cloud generated by COLMAP to handle distant buildings. The specific process of generating the initial point cloud of the dynamic vehicle and the static background is as follows: Figure 6 shown.

[0074] S420 , generating a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, to form three-dimensional Gaussian models of the dynamic vehicle and the static background respectively.

[0075] OG-Gaussian represents point cloud points using a Gaussian ellipsoid. A Gaussian ellipsoid is a dynamic, parameterized ellipsoid with properties such as position, color, opacity, and a spherical correlation coefficient that characterizes the appearance of the target area.

[0076] The position attribute value of the Gaussian ellipsoid can be determined by the pose covariance matrix and position matrix of the initial point cloud. The pose covariance matrix can be decomposed into a rotation matrix and a scaling matrix.

[0077] The present invention first initializes each point in the initial point cloud into a Gaussian ellipsoid based on a preset rule. Then, the Gaussian ellipsoid is projected onto a two-dimensional image plane to obtain the image data of the target area, and the properties of the Gaussian ellipsoid are optimized based on the properties of the projected pixel points to generate a three-dimensional Gaussian model. An example of a three-dimensional Gaussian model after parameter optimization is shown in FIG. Figure 7 .

[0078] The present disclosure generates three-dimensional Gaussian models of the dynamic vehicle and the static background based on the initial point cloud of the target area.

[0079] S430, predicting the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, and completing the combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory.

[0080] The present disclosure represents the actual state of a dynamic vehicle by tracking the posture of the dynamic vehicle. The posture of the dynamic vehicle can be represented by the position matrix and rotation matrix of the Gaussian ellipsoid of the three-dimensional Gaussian model of the dynamic vehicle. Some embodiments of the present disclosure represent the position matrix and rotation matrix of the Gaussian ellipsoid of the dynamic vehicle with optimizable parameters, and construct a dynamic vehicle prediction model to predict future values ​​based on the historical values ​​of the optimizable parameters to predict the Gaussian ellipsoid of the dynamic vehicle in the future. At the same time, the predicted Gaussian ellipsoid is projected onto a two-dimensional image plane to obtain image data of the target area containing the dynamic vehicle, and the attributes of the projection point are calculated based on the attributes of the Gaussian ellipsoid. Based on the attribute difference between the projection point and the pixel point corresponding to the two-dimensional image plane, the dynamic vehicle prediction model is adjusted to obtain a more accurate prediction result.

[0081] By predicting the position matrix and rotation matrix of the Gaussian ellipsoid of the dynamic vehicle in the future time, the posture of the dynamic vehicle in the future frame can be obtained. The trajectory of the dynamic vehicle can be formed by tracking the posture of the dynamic vehicle in multiple frames.

[0082] Because the present disclosure predicts future trajectories based on a comparison of a three-dimensional Gaussian model of the dynamic vehicle at historical moments with images of the target area, there is no need to manually label the trajectory data.

[0083] After obtaining the trajectory of the dynamic vehicle, the three-dimensional Gaussian model of the dynamic vehicle and the static background can be combined based on the trajectory to generate a scene model.

[0084] In some embodiments of the present disclosure, the coordinate system of the three-dimensional Gaussian model of the dynamic vehicle and the static background is first unified as the world coordinate system; then the rendering timestamp is obtained, and the three-dimensional Gaussian model of the dynamic vehicle and the static background at the rendering timestamp is combined based on the spherical association coefficient to form a frame of scene snapshot corresponding to the rendering timestamp; finally, a timestamp sequence is extracted from the timeline of the dynamic vehicle trajectory, and a frame of scene snapshot is generated for each timestamp in the timestamp sequence, and a scene model is generated from this series of scene snapshots.

[0085] After generating the scene model, the Gaussian ellipsoid is projected onto a 2D plane based on the camera's extrinsic and intrinsic parameters. The parameters of each point in 2D space are calculated to render the target area in the target image space. The color of each pixel in the target image space is determined by the corresponding Gaussian ellipsoid's opacity, color, and other properties. Figure 8 It is a real street scene with dynamic vehicles and static background. Figure 9 It is a street scene reconstructed based on OG-Gaussian.

[0086] In some other embodiments of the present disclosure, the three-dimensional Gaussian models of the dynamic vehicle and the static background are rendered in the target image space to reconstruct the dynamic vehicle and the static background separately, such as Figure 10 and Figure 11 As shown, the reconstructed dynamic vehicle is merged with the static background based on the trajectory to complete the overall reconstruction, as shown in Figure 9 shown.

[0087] In one embodiment of the present disclosure, three groups of typical scenes are selected from the Waymo open dataset and the OG-Gaussian method is used. Figure 12 Ours for short) and various commonly used algorithms for reconstruction, such as Figure 12 Experiments show that in every scene, the OG-Gaussian method outperforms other algorithms in PSNR, a measure of reconstruction quality, demonstrating strong robustness and accuracy. It also leads in rendering speed (FPS), demonstrating the improvement in rendering speed due to reduced computational effort.

[0088] Figure 13 FIG. 1 is a schematic diagram of a scene reconstruction device according to some embodiments of the present disclosure. Figure 13 As shown, the scene reconstruction device 1300 includes a separation module 1310, a generation module 1320, and a rendering module 1330. In some embodiments of the present disclosure, the scene reconstruction function is performed by Figure 1 The rendering server executes as shown.

[0089] A separation module 1310 is configured to acquire images of a target area from multiple perspectives, predict a 3D occupancy grid of the target area based on the images, separate dynamic vehicles from static background in the 3D occupancy grid based on semantic information, and generate a corresponding initial point cloud;

[0090] A generating module 1320 is configured to generate a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, thereby forming three-dimensional Gaussian models of the dynamic vehicle and the static background respectively;

[0091] The rendering module 1330 is used to predict the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, and complete the combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory.

[0092] One embodiment of the present disclosure provides a scene reconstruction device. Figure 14 As shown, the scene reconstruction device 1400 includes a memory 1420 and a processor 1410, wherein the memory 1420 is used to store a computer program; the processor 1410 is used to implement Figure 4 The method described in S410-S430.

[0093] In summary, the scene reconstruction methods, devices, and apparatuses provided in the various embodiments of the present disclosure avoid using expensive LiDAR sensors to generate point clouds by generating the initial point cloud in 3DGS scene reconstruction based on the occupancy grid predicted by a trained occupancy prediction network. They also avoid manual data labeling by separating dynamic and static objects based on the semantic information of the grid and using the position and trajectory of dynamic objects as learnable parameters to predict the trajectory. Furthermore, system verification has shown that the method described in the present disclosure has low computational complexity and strong robustness and accuracy.

[0094] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0095] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations may also be performed in conjunction with other types of program modules. Generally speaking, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will appreciate that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like, and may also be used in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0096] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0097] It should be understood that the above-described specific embodiments of the present disclosure are merely illustrative of or explanation of the principles of the present disclosure and do not constitute limitations on the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure shall be included within the scope of protection of the present disclosure. In addition, the claims appended to the present disclosure are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents of such scope and metes and bounds.

Claims

1. A scene reconstruction method, characterized in that: include: Acquire images of a target area from multiple perspectives, predict a 3D occupancy grid of the target area based on the images, separate dynamic vehicles from static background in the 3D occupancy grid based on semantic information, and generate a corresponding initial point cloud; Generating a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, respectively forming a three-dimensional Gaussian model of the dynamic vehicle and the static background; By comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, the trajectory of the dynamic vehicle is predicted, and based on the trajectory, the three-dimensional Gaussian model of the dynamic vehicle and the static background is combined and rendered in the target image space; The separating of the dynamic vehicle from the static background based on semantic information in the 3D occupancy grid includes: Extracting a pixel-level vehicle object based on the detection and segmentation model. If the displacement of the vehicle object at adjacent time stamps is greater than a preset threshold, the vehicle object is considered a dynamic vehicle. Distinguishing whether a grid belongs to the dynamic vehicle based on semantic information of each grid in the 3D occupancy grid; forming a 3D occupancy grid of the dynamic vehicle from all grids belonging to the dynamic vehicle, and forming a 3D occupancy grid of the static background from all grids not belonging to the dynamic vehicle; Generating the corresponding initial point cloud includes: Upsampling a 3D occupancy grid of the dynamic vehicle for voxels exceeding a preset threshold to generate points of an initial point cloud of the dynamic vehicle, wherein the color of the points is the same as the color of a pixel projected onto a two-dimensional image plane for acquiring image data of the target area; generating a first point cloud of the static background based on the 3D occupancy grid of the static background, wherein positions of points in the first point cloud are taken from center coordinates of the grid; generating a second point cloud for the multi-view images of the target area based on COLMAP, and aggregating the first point cloud of the static background with the second point cloud to generate an initial point cloud of the static background; Generating a corresponding Gaussian ellipsoid for each point in the initial point cloud includes: For each point in the initial point cloud, a corresponding Gaussian ellipsoid is generated based on the pose covariance matrix and the position matrix of the point, wherein the pose covariance matrix includes a rotation matrix and a scaling matrix; The Gaussian ellipsoid further includes an opacity parameter and a set of spherical association coefficients characterizing the appearance of the target area, and the opacity parameter and the spherical association coefficients are initialized according to a preset rule; Projecting the points corresponding to the Gaussian ellipsoid onto the two-dimensional image plane of the image data of the target area to obtain corresponding pixel points, and optimizing the opacity parameter and spherical association coefficient of the Gaussian ellipsoid based on the attributes of the pixel points; Predicting the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area includes: Predicting a position matrix and a rotation matrix of a Gaussian ellipsoid of the dynamic vehicle in each frame based on a dynamic vehicle prediction model, projecting the Gaussian ellipsoid onto a two-dimensional image plane for obtaining image data of the target area, and adjusting the dynamic vehicle prediction model based on a difference between a projected point and a corresponding pixel point; The posture of the dynamic vehicle is represented by a position matrix and a rotation matrix of the Gaussian ellipsoid of the dynamic vehicle; The postures of the dynamic vehicle in multiple frames are tracked to form a trajectory of the dynamic vehicle.

2. The method according to claim 1, characterized in that The combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory includes: Combining the three-dimensional Gaussian model of the dynamic vehicle and the static background based on the trajectory to generate a scene model; The scene model is projected into the target image space to complete the rendering of the target area in the target image space.

3. The method according to claim 2, characterized in that The generating a scene model by combining the three-dimensional Gaussian model of the dynamic vehicle and the static background based on the trajectory includes: The coordinate system of the three-dimensional Gaussian model of the dynamic vehicle and the static background is set as the world coordinate system; Obtaining a rendering timestamp, and combining the three-dimensional Gaussian model of the dynamic vehicle and the static background at the rendering timestamp based on the spherical association coefficient of the dynamic ellipsoid to form a frame of scene snapshot; A plurality of frames of scene snapshots are generated based on the timestamp sequence of the trajectory to form the scene model.

4. The method according to claim 3, characterized in that Projecting the scene model to the target image space comprises: All the Gaussian ellipsoids of the scene model are projected into a target image space and parameters of a pixel point projected into the target image space are calculated based on the parameters of each Gaussian ellipsoid, wherein the color of the pixel point is calculated by the opacity of the Gaussian ellipsoid.

5. A scene reconstruction device, characterized in that: include: a separation module, configured to acquire images of a target area from multiple perspectives, predict a 3D occupancy grid of the target area based on the images, separate dynamic vehicles from static background in the 3D occupancy grid based on semantic information, and generate a corresponding initial point cloud; a generating module, configured to generate a corresponding Gaussian ellipsoid for each point of the initial point cloud of the dynamic vehicle and the static background, thereby forming three-dimensional Gaussian models of the dynamic vehicle and the static background respectively; a rendering module, configured to predict a trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area, and perform combined rendering of the three-dimensional Gaussian model of the dynamic vehicle and the static background in the target image space based on the trajectory; The separating of the dynamic vehicle from the static background based on semantic information in the 3D occupancy grid includes: Extracting a pixel-level vehicle object based on the detection and segmentation model. If the displacement of the vehicle object at adjacent time stamps is greater than a preset threshold, the vehicle object is considered a dynamic vehicle. Distinguishing whether a grid belongs to the dynamic vehicle based on semantic information of each grid in the 3D occupancy grid; forming a 3D occupancy grid of the dynamic vehicle from all grids belonging to the dynamic vehicle, and forming a 3D occupancy grid of the static background from all grids not belonging to the dynamic vehicle; Generating the corresponding initial point cloud includes: Upsampling a 3D occupancy grid of the dynamic vehicle for voxels exceeding a preset threshold to generate points of an initial point cloud of the dynamic vehicle, wherein the color of the points is the same as the color of a pixel projected onto a two-dimensional image plane for acquiring image data of the target area; generating a first point cloud of the static background based on the 3D occupancy grid of the static background, wherein positions of points in the first point cloud are taken from center coordinates of the grid; generating a second point cloud for the multi-view images of the target area based on COLMAP, and aggregating the first point cloud of the static background with the second point cloud to generate an initial point cloud of the static background; Generating a corresponding Gaussian ellipsoid for each point in the initial point cloud includes: For each point in the initial point cloud, a corresponding Gaussian ellipsoid is generated based on the pose covariance matrix and the position matrix of the point, wherein the pose covariance matrix includes a rotation matrix and a scaling matrix; The Gaussian ellipsoid further includes an opacity parameter and a set of spherical association coefficients characterizing the appearance of the target area, and the opacity parameter and the spherical association coefficients are initialized according to a preset rule; Projecting the points corresponding to the Gaussian ellipsoid onto the two-dimensional image plane of the image data of the target area to obtain corresponding pixel points, and optimizing the opacity parameter and spherical association coefficient of the Gaussian ellipsoid based on the attributes of the pixel points; Predicting the trajectory of the dynamic vehicle by comparing the three-dimensional Gaussian model of the dynamic vehicle with the image of the target area includes: Predicting a position matrix and a rotation matrix of a Gaussian ellipsoid of the dynamic vehicle in each frame based on a dynamic vehicle prediction model, projecting the Gaussian ellipsoid onto a two-dimensional image plane for obtaining image data of the target area, and adjusting the dynamic vehicle prediction model based on a difference between a projected point and a corresponding pixel point; The posture of the dynamic vehicle is represented by a position matrix and a rotation matrix of the Gaussian ellipsoid of the dynamic vehicle; The postures of the dynamic vehicle in multiple frames are tracked to form a trajectory of the dynamic vehicle.

6. A scene reconstruction device, characterized in that: including memory and processor, The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 4 when executing the computer program.

Images