Automatic driving scene three-dimensional reconstruction method, system and device and storage medium

Abstract

The invention relates to the technical field of computer vision, and relates to an automatic driving scene three-dimensional reconstruction method, which comprises the following steps: acquiring a training image data set of an automatic driving scene; removing a dynamic shielding object area and a sky area in the initial training image from the initial training image to obtain a second training image; based on the second training image, obtaining a sparse point cloud of the automatic driving scene after the dynamic shielding object and the sky are removed; constructing a three-dimensional Gaussian model of the automatic driving scene according to the sparse point cloud, and optimizing parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the automatic driving scene; and reconstructing a sky area and a dynamic shielding object area in the optimized automatic driving scene three-dimensional model to obtain a final automatic driving scene three-dimensional model. The invention further provides an automatic driving scene three-dimensional reconstruction system and device and a storage medium. According to the invention, the three-dimensional reconstruction quality in a scene with a dynamic shielding object can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, device and storage medium for three-dimensional reconstruction of an autonomous driving scene. Background Art

[0002] Three-dimensional scene reconstruction and new view synthesis is one of the important and complex research tasks in the field of computer vision, which aims to restore the three-dimensional information of the scene in space from a set of two-dimensional images. This task plays an important role in many fields such as medical image processing, virtual reality (VR), and autonomous driving.

[0003] Implicit Neural Radiance Fields (NeRF) is a method that uses deep learning technology to extract the geometry and texture information of objects from images with multiple viewpoints to generate high-quality 3D reconstruction models. NeRF usually uses volume ray marching to optimize the multi-layer perceptron (MLP) to represent the color and density of each point in the scene, achieving highly realistic 3D models from any angle and distance.

[0004] NeRF's variant method further improves the reconstruction effect and rendering speed, making the 3D reconstruction process more natural and efficient, but it still has problems such as high resource consumption and slow rendering speed.

[0005] 3D Gaussian Splatting for Real-Time Radiance Field Rendering (3DGS) is another 3D reconstruction method. 3DGS achieves a double improvement in training speed and rendering effect by displaying the expression and taking advantage of its differentiable rendering. 3DGS technology defines a set of anisotropic Gaussians in the 3D world and performs adaptive density control to achieve high-quality rasterization rendering results using only sparse point cloud input obtained by the Structure-from-Motion (SfM) technology.

[0006] However, 3DGS only performs well in static scenes, and artifacts and other problems often occur in the reconstruction of dynamic scenes. Existing 3DGS variants are difficult to overcome the challenges of 3D reconstruction of unbounded scenes in autonomous driving. Therefore, the reconstruction quality after removing dynamic occluders from autonomous driving scenes is an urgent research issue. Summary of the invention

[0007] In order to improve the quality of three-dimensional reconstruction in scenes with dynamic occluders, a method, system, device and storage medium for three-dimensional reconstruction of an autonomous driving scene are provided.

[0008] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for 3D reconstruction of an autonomous driving scene, comprising:

[0009] Obtain a training image dataset of an autonomous driving scenario; the training image dataset includes a plurality of initial training images taken from different perspectives while the vehicle is driving;

[0010] Removing the dynamic occluder region and the sky region in the initial training image from the initial training image to obtain a second training image;

[0011] Summarizing the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset;

[0012] Obtaining a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image data set;

[0013] Constructing a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimizing the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene;

[0014] The sky area and dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene are reconstructed to obtain the final three-dimensional model of the autonomous driving scene.

[0015] Optionally, the method further comprises:

[0016] The initial training image is parsed to obtain an instance segmentation image of the initial training image, an initial depth image, and pose information of the initial training image.

[0017] Optionally, constructing a three-dimensional Gaussian model of the autonomous driving scene according to the sparse point cloud, and optimizing the three-dimensional Gaussian model to obtain a final three-dimensional model of the autonomous driving scene includes:

[0018] Construct a 3D Gaussian model based on the sparse point cloud and initialize the parameters of the 3D Gaussian model;

[0019] Determine a two-dimensional plane coordinate system according to the posture information, render and splash the three-dimensional Gaussian model into the two-dimensional plane coordinate system, and generate a rendered image;

[0020] Compare the rendered image with the second training image to determine the loss function of the three-dimensional Gaussian model;

[0021] The parameters of the three-dimensional Gaussian model are optimized according to the loss function to obtain an optimized three-dimensional model of the autonomous driving scene.

[0022] Optionally, the loss function includes a reconstruction loss and optical flow loss

[0023] Reconstruction losses The expression is

[0024]

[0025] in, represents the mean absolute error loss, λ recon To balance the hyperparameters, represents the structural similarity loss, represents the rendered image, and I represents the second training image;

[0026] Optical flow loss The expression is

[0027]

[0028] Where F = {f fwd ,f bwd} represents the ground truth stream, represents Gaussian flow; f fwd represents the forward flow between two adjacent training image frames in the training image dataset, where the training image dataset includes a plurality of training image frames continuously acquired while the vehicle is driving; f bwd represents the backward flow between two adjacent training image frames in the training image set; represents the forward flow in a Gaussian scene predicted by a three-dimensional Gaussian model, Represents the backward flow in a Gaussian scene predicted by a 3D Gaussian model.

[0029] Optionally, the loss function also includes a depth map supervision loss

[0030] Depth map supervision loss The expression is

[0031]

[0032] Among them, D ras represents the depth map of the rendered image,

[0033] D rmv represents the depth map of the second training image after removing dynamic occluders and the sky from the autonomous driving scene,

[0034] Corr(D ras ,D rmv ) means D ras and D rmv The Pearson correlation coefficient between .

[0035] Optionally, the details of the sky area of ​​the autonomous driving scene are restored by a mapping method to complete the reconstruction of the sky area in the optimized three-dimensional model of the autonomous driving scene.

[0036] Optionally, the surfaces of the optimized three-dimensional model of the autonomous driving scene are aligned to achieve reconstruction of the dynamically occluded areas that have been removed from the optimized three-dimensional model of the autonomous driving scene.

[0037] In order to solve the above problems, the present invention also provides a three-dimensional reconstruction system for an autonomous driving scene, the system comprising:

[0038] An acquisition module is used to acquire a training image dataset of an autonomous driving scenario; the training image dataset includes a plurality of initial training images taken from different perspectives while the vehicle is driving;

[0039] A separation module, used for separating the dynamic occlusion area and the sky area in the initial training image from the initial training image to obtain a second training image;

[0040] A summarizing module, used for summarizing the second training images generated based on the initial training images in the training image data set to obtain a second training image data set;

[0041] A point cloud generation module, configured to obtain a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image data set;

[0042] A first reconstruction module is used to construct a three-dimensional Gaussian model of the autonomous driving scene according to the sparse point cloud, and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene;

[0043] The second reconstruction module is used to reconstruct the sky area and the dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

[0044] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned three-dimensional reconstruction method of the autonomous driving scene.

[0048] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned three-dimensional reconstruction method of the autonomous driving scene.

[0049] In summary, this application includes the following beneficial technical effects:

[0050] By inputting a sparse point cloud after removing dynamic occluders and the sky, and rendering the sky separately, the background holes caused by removing dynamic occluders are optimized; the depth map of the second training image is used for supervision to improve the three-dimensional reconstruction effect after removing dynamic occluders in the autonomous driving scene; in the input training image, the sky area is separated by instance segmentation image, and the rendering of the sky is not considered during the scene rendering process, but the sky is added when the final rendered image is generated. This can avoid the problem of floating shadows caused by Gaussian points in the sky area due to incorrect depth when the 3D Gaussian method is used for rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a flow chart of a method for three-dimensional reconstruction of an autonomous driving scene provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a process for generating a three-dimensional Gaussian model based on a coefficient point cloud and optimizing the three-dimensional Gaussian model according to an embodiment of the present invention;

[0053] Figure 3 A system block diagram of a method for three-dimensional reconstruction of an autonomous driving scene provided in a preferred embodiment of the present invention;

[0054] Figure 4 It is a comparison diagram of the 3D reconstruction effect of the 3D reconstruction method of the autonomous driving scene of the present application and the 3D reconstruction effect of the existing 3D Gaussian splash in a static scene;

[0055] Figure 5 It is a three-dimensional reconstruction effect diagram of the three-dimensional reconstruction method of the autonomous driving scene of the present application under a dynamic scene;

[0056] Figure 6 A functional module diagram of a three-dimensional reconstruction system for an autonomous driving scene provided by an embodiment of the present invention;

[0057] Figure 7 A schematic diagram of the structure of an electronic device for implementing the method for three-dimensional reconstruction of an autonomous driving scene provided in one embodiment of the present invention.

[0058] Figure numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface; 31, acquisition module; 32, separation module; 33, aggregation module; 34, point cloud generation module; 35, first reconstruction module; 36, second reconstruction module.

[0059] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0061] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0062] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0063] Reference Figure 1 FIG. 1 is a flow chart of a method for 3D reconstruction of an autonomous driving scene provided by an embodiment of the present invention. In this embodiment, the method for 3D reconstruction of an autonomous driving scene includes:

[0064] S1. Obtain a training image dataset for autonomous driving scenarios.

[0065] The training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving. The training image dataset can be formed by capturing several image frames in chronological order from the video taken while the vehicle is driving; or the training image dataset can be obtained by continuously collecting multiple images as initial training images during the driving of the car. The initial training images corresponding to the autonomous driving scene include road conditions, traffic signals, pedestrians, other vehicles and other elements. These image data are captured in real time by the camera of the autonomous driving vehicle and processed in combination with other sensor information to help the system perform environmental perception, path planning and decision execution.

[0066] The initial training image is an RGB image, which is also called a three-channel color image. Each pixel is composed of the intensity values ​​of the three color channels of red, green, and blue. It is widely used in object recognition and scene understanding. In this embodiment, the method for 3D reconstruction of an autonomous driving scene also includes:

[0067] The initial training image is parsed to obtain an instance segmentation image of the initial training image, an initial depth image, and pose information of the initial training image.

[0068] Specifically, for example, Depth Anything V2 is used to process multiple groups of RGB images of different perspectives to obtain an initial depth map of the autonomous driving scene. The initial depth map can determine the distance from each pixel in the autonomous driving scene to the camera, which helps to accurately identify obstacles or occlusions.

[0069] The RGB image is processed through the Segment Anything Model (SAM) to obtain an instance segmentation image of the autonomous driving scene. The instance segmentation image includes objects (such as roads, pedestrians, other vehicles, traffic lights, street lights, green plants, or the sky) existing in the initial training image and the areas corresponding to the objects;

[0070] The training images in the training image dataset are processed through Structure-from-Motion to obtain the pose information of the autonomous driving scene. The pose information can be used to determine the position and angle of the camera that took the initial training image, so as to facilitate the subsequent restoration of the three-dimensional model of the autonomous driving scene.

[0071] S2. Remove the dynamic occluder region and the sky region in the initial training image from the initial training image to obtain a second training image.

[0072] Specifically, the area corresponding to the dynamic occluder and the area corresponding to the sky are screened out from the objects recognized in the instance segmentation image; the area corresponding to the dynamic occluder and the area corresponding to the sky are separated from the initial training image to obtain the second training image.

[0073] S3. Summarize the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset.

[0074] S4. Based on the second training image data set, a sparse point cloud is obtained after dynamic occluders and the sky are removed from the autonomous driving scene.

[0075] In this embodiment, the second training images in the second training image data set are processed using the Structure-from-Motion (SfM) technology to obtain a sparse point cloud after removing dynamic occluders and the sky.

[0076] S5. Construct a three-dimensional Gaussian model of the autonomous driving scene according to the sparse point cloud, and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene.

[0077] Reference Figure 2 , step S5 comprises:

[0078] S51. Construct a three-dimensional Gaussian model according to the sparse point cloud, and initialize parameters of the three-dimensional Gaussian model.

[0079] The sparse point cloud data is preprocessed. The preprocessing process includes data cleaning and filtering. Data cleaning can remove abnormal points and outliers. Filtering can choose methods such as Gaussian filtering or median filtering to reduce the impact of noise on post-processing. Subsequently, the preprocessed point cloud data is voxelized, that is, the point cloud data is divided into multiple voxels. The center of each voxel is regarded as an anchor point and is assigned local context features, scaling factors and learnable offsets. After that, a three-dimensional Gaussian model is constructed based on the voxelization. Starting from the sparse point cloud, a set of three-dimensional Gaussian distributions is created. Each point in the sparse point cloud is regarded as the mean of a three-dimensional Gaussian distribution, and each point is assigned initial covariance, transparency, color and other attributes. The final three-dimensional Gaussian model is obtained by optimizing these Gaussian distributions.

[0080] The three-dimensional Gaussian model is a dynamic three-dimensional Gaussian model. Specifically, the parameters of the three-dimensional Gaussian model include a center position, a rotation matrix, a cosquare matrix, a scaling matrix, transparency and spherical harmonic coefficients. The color of the three-dimensional Gaussian model is determined by the spherical harmonic coefficients.

[0081] We get a set of video frames during the vehicle driving process, each frame is a moment, and when the video group has t frames, there are t moments. If the three-dimensional Gaussian model is expressed by n Gaussian balls (n is a positive integer), the position and rotation information of each Gaussian ball is modeled with time t. According to the properties of each dynamic Gaussian at time t, the center position and rotation matrix of the Gaussian ball at time t can be obtained. The color, transparency and scale of the Gaussian ball do not change with time.

[0082] The properties of the dynamic Gaussian at time t are as follows: The expression of the center position is x(t), y(t), z(t) are the spatial coordinates corresponding to the Gaussian sphere, [.] T represents the transpose of [x(t), y(t), z(t)], Refers to the space composed of vectors in three directions.

[0083] The rotation matrix of the dynamic Gaussian is represented by quaternion. Specifically, the expression of the rotation matrix is Among them, q w (t) is the real part, q x (t), q y (t) and q z (t) is the imaginary part.

[0084] The expression of the scaling matrix is s x ,s y ,s z is the scaling factor.

[0085] In this embodiment, the expression G of the probability density function for any point x in space at time t is t (x) is:

[0086]

[0087] Where x is the index of a point in the sparse point cloud in space, μ(t) is the center position of point x at time t, ∑(t) represents the 3D covariance matrix at time t, and its 2D covariance matrix ∑(t)′=JW∑(t)W T J T , W is the observation transformation, J is the Gaussian center position

[0088] The affine approximation Jacobian matrix of the projection transformation at μ(t). In order to prevent overfitting, the Fourier approximation is used to approximate the 3D position of the Gaussian ball. At time t, the 3D position of the Gaussian ball is expressed as follows:

[0089]

[0090]

[0091] where w ·,0 ,…,w ·,2L represents the intercept and coefficient of the 3D position, L represents the number of harmonics, and i represents the index of the sparse point cloud in space.

[0092] To simplify the rotation matrix, a linear approximation is used to approximate the quaternion, expressed as follows:

[0093] q x (t) = w qx,0 +w qx,1 t,q y (t) = w qy,0 +w qy,1 t

[0094] q z (t) = w qz,0 +w qz,1 t,q w (t) = w qw,0 +w qw,1 t

[0095] where w ·,0 The intercept of the 3D rotation, w ·,1 Represents the coefficients of 3D rotation.

[0096] S52, determining a two-dimensional plane coordinate system according to the posture information, rendering and splashing the three-dimensional Gaussian model onto the two-dimensional plane coordinate system to generate a rendered image;

[0097] The original 3D Gaussian rendering is used to splatter the 3D Gaussian to the 2D camera plane coordinate system. Each 2D plane image is divided into 16×16 blocks, and the 2D Gaussians in all blocks are sorted by depth. The color c of each pixel is calculated based on the rasterization rendering evaluation of the mixture of N ordered points overlapping with the pixel:

[0098]

[0099] Where N is the number of points where pixels overlap in the rendered image, i is the index of the point where pixels overlap in the rendered image, j is the index of the point before point i, and c is the distance between the points where pixels overlap. i represents the color of the Gaussian sphere evaluated by the spherical harmonic coefficients SH; α′ i Represents the opacity (density) value of the current point i, α′ i =∑′α i Indicates the opacity of the Gaussian sphere; α′ j Represents the opacity (density) value of the j points before i, using 1-α′ i Multiply it cumulatively as the weight, which means that the more transparent the previous point is, the greater the weight is. The current α′i The greater the influence weight.

[0100] S53, comparing the rendered image with the second training image to determine the loss function of the three-dimensional Gaussian model;

[0101] The loss function includes reconstruction loss and optical flow loss

[0102] Reconstruction losses The expression is

[0103]

[0104] in, represents the mean absolute error loss, λ recon is the first balancing hyperparameter, represents the structural similarity loss, represents the rendered image and I represents the second training image.

[0105] In the dynamic stage, in order to provide spatiotemporal consistency of n frames of images, the optical flow loss is introduced

[0106] Optical flow loss The expression is

[0107]

[0108] Where F = {f fwd ,f bwd} represents the ground truth stream, represents Gaussian flow; f fwd represents the forward flow between two adjacent training image frames in the training image dataset, where the training image dataset includes a plurality of training image frames continuously acquired while the vehicle is driving; f bwd represents the backward flow between two adjacent training image frames in the training image set, f fwd and f bwd They are obtained by RAFT algorithm (forward propagation and back propagation algorithm in neural network); represents the forward flow in a Gaussian scene predicted by a three-dimensional Gaussian model, Represents the backward flow in the Gaussian scene predicted by the three-dimensional Gaussian model; it should be noted that the forward flow is the mapping from the first frame to the second frame; the backward flow is the mapping from the second frame to the first frame.

[0109]

[0110] Δt is the difference between the time steps of two training image frames.

[0111] The depth rasterizer is used to perform depth testing to determine the occlusion relationship of pixels and ensure the correct rendering. The differentiable depth rasterizer receives the rendered image depth map D ras And the depth map D after removing dynamic occluders and sky in the autonomous driving scene rmv The depth of each pixel is calculated by accumulating the z-buffer d of the ordered Gaussians that contribute to that pixel. i get:

[0112]

[0113] At the same time, in order to reduce the rendering image depth map D ras And the depth map D after removing dynamic occluders and sky in the autonomous driving scene rmv The scale blur between them introduces the Pearson correlation coefficient:

[0114]

[0115] The loss function also includes the depth map supervision loss

[0116] Depth map supervision loss The expression is

[0117]

[0118] Among them, D ras Represents the rendered image depth map,

[0119] D rmv The depth map Corr(D) of the second training image after removing dynamic occluders and the sky in the autonomous driving scene ras ,D rmv ) means D ras and D rmv The Pearson correlation coefficient between .

[0120] The expression of Pearson correlation coefficient is:

[0121]

[0122] Loss Function The expression is

[0123] λ flow is the second balancing hyperparameter, λ depth is the third balancing hyperparameter.

[0124] S54. Optimize the parameters of the three-dimensional Gaussian model according to the loss function to obtain an optimized three-dimensional model of the autonomous driving scene.

[0125] S6. Reconstruct the sky area and the dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

[0126] In this embodiment, the details of the sky area of ​​the autonomous driving scene are restored by the mapping method, and the reconstruction of the sky area in the optimized three-dimensional model of the autonomous driving scene is completed; because 3DGS is defined in Euclidean space, they are not suitable for reconstructing distant areas such as the sky. Therefore, the mapping method is used to restore the details of the sky area. In the input training image, the sky area is separated by the instance segmentation image. In the process of scene rendering, the rendering of the sky is not considered, and the sky is added when the final rendered image is generated. This can avoid the problem of floating shadows caused by incorrect depth of Gaussian points in the sky area when rendering with the 3D Gaussian method.

[0127] The surfaces of the optimized 3D model of the autonomous driving scene are aligned to reconstruct the removed dynamic occlusion areas.

[0128] Specifically, the method of aligning the surface to the Gaussian model includes:

[0129] For any spatial position in the scene The calculation formula of the density function ρ(x) is as follows:

[0130]

[0131] The density function ρ(x) is the sum of Gaussian values, N represents the total number of Gaussian balls in the space, i represents the index of the Gaussian ball in the space, and the density function ρ(x) is calculated by adding the mixing coefficient α of each Gaussian ball i i With each Gaussian G t (i) Multiply and sum.

[0132] Reference Figure 3 In the preferred implementation of this embodiment, in order to better reconstruct the area where the dynamic occluder is removed, surface alignment normalization is applied by controlling the Gaussian density function. In order to control the Gaussian density function and apply surface alignment normalization, there are three assumptions for any point x close to the scene surface: (1) The Gaussian g closest to point x * has a greater contribution to ρ(x); (2) The ideal three-dimensional Gaussian distribution of the surface is flat, so each Gaussian g has one of the scale factors in the three axis directions approaching 0; (3) In order to describe the real surface, the Gaussian transparency α is required i = 1. The density function is approximated as:

[0133]

[0134] is the approximate value of the density function of the missing part of the three-dimensional Gaussian model, represents the minimum scale factor of Gaussian, is the Gaussian g closest to point x * The center point coordinates, is the Gaussian g closest to point x * The direction of the corresponding axis.

[0135] Based on the density function ρ(x) and Get the approximate signed distance function (SDF), specifically, the signed distance function:

[0136]

[0137] In order to reduce the calculation, Use the depth map D of the rendered image corresponding to point x ras The depth value is replaced by the value of d(x). Then the difference between d(x) and the depth value at point x is The regularization term with d(x) is expressed as:

[0138]

[0139] Among them, X represents all points sampled onto the surface of the three-dimensional Gaussian model.

[0140] At the same time, a regularization term is added for the normal direction:

[0141]

[0142] It represents the gradient vector of the function d(x) at point x.

[0143] In addition, to encourage opaque Gaussians, cross entropy loss is used for opacity regularization:

[0144]

[0145] Δσ represents the opacity increment or change of the j-th pixel or point, and σ represents the opacity.

[0146] In the preferred implementation of this embodiment, the loss function of the 3D reconstruction model is The expression is:

[0147]

[0148] Among them, λ surface is the fourth balancing hyperparameter, λ SDF is the fifth balancing hyperparameter, λ norm is the sixth balancing hyperparameter, λopacity is the seventh balance hyperparameter; is the signed distance loss, is the regularization loss, is the cross entropy loss.

[0149] In order to verify the capability of the 3D reconstruction method of the autonomous driving scene provided in this application, we used the existing realistic dataset Virtual KITTI (VKITTI) synthesized by autonomous driving scenes to verify the 3D reconstruction effect of the autonomous driving scene;

[0150] The dataset Virtual KITTI (VKITTI) comes from 5 different virtual worlds in urban environments (Scene01, Scene02, Scene06, Scene18, Scene20), which are automatically, accurately and completely annotated with category, instance and depth labels at the pixel level. In addition, in order to further verify the versatility of our method in real scenes, the nuScenes dataset is also selected for verification.

[0151] The experimental environment is: Ubuntu 22.04 operating system, NVIDIA RTX 3060 GPU, Python 3.8, PyTorch1.13.1, CUDA 11.7. The model has undergone 30,000 iterations, and the whole process includes 3,000 iterations of static optimization and 27,000 iterations of dynamic optimization. During the training process, the values ​​of the hyperparameters are as follows: L = 2, λ recon =0.8,λ flow =1.0,λ depth =0.05,λ surface =0.2,λ SDF =1.0,λ norm =1.0,λ opacity =0.5.

[0152] Figure 4 It is a comparison diagram of the 3D reconstruction effect of the 3D reconstruction method of the autonomous driving scene of the present application and the 3D reconstruction effect of the existing 3D Gaussian splash in a static scene; Figure 4 The first column GT on the left represents the input initial training image, the second column is the autonomous driving scene reconstructed using the autonomous driving scene 3D reconstruction method of the present application; the third column is the autonomous driving scene obtained by 3D reconstruction using the existing 3D Gaussian splash method.

[0153] Figure 5It is a three-dimensional reconstruction effect diagram of the three-dimensional reconstruction method of the autonomous driving scene of the present application under a dynamic scene. The images in the first row and the third row are initial training images collected under a dynamic autonomous driving scene. The images in the second row are schematic diagrams of the autonomous driving scene reconstructed after removing dynamic obstacles from the initial training images in the first row. The images in the fourth row are schematic diagrams of the autonomous driving scene reconstructed after removing dynamic obstacles from the initial training images in the third row.

[0154] Reference Figure 6 Based on the same inventive concept, an embodiment of the present invention provides a three-dimensional reconstruction system for an autonomous driving scene.

[0155] The three-dimensional reconstruction system of an autonomous driving scene described in the present invention can be loaded in an electronic device. According to the functions to be implemented, the three-dimensional reconstruction system of an autonomous driving scene includes an acquisition module 31, a separation module 32, a summary module 33, a point cloud generation module 34, a first reconstruction module 35, and a second reconstruction module 36. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0156] The acquisition module 31 can acquire a training image data set of an autonomous driving scene; the training image data set includes a plurality of initial training images taken from different perspectives when the vehicle is driving; the separation module 32 can remove the dynamic occlusion area and the sky area in the initial training image from the initial training image to obtain a second training image; the aggregation module 33 can aggregate the second training images generated based on the initial training images in the training image data set to obtain a second training image data set;

[0157] The point cloud generation module 34 can obtain a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image data set; the first reconstruction module 35 can construct a three-dimensional Gaussian model of the autonomous driving scene according to the sparse point cloud, and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene; the second reconstruction module 36 can reconstruct the sky area and the dynamic occluder area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

[0158] The various variations and specific examples of the three-dimensional reconstruction method of the autonomous driving scene provided in the above-mentioned embodiment are also applicable to the three-dimensional reconstruction system of the autonomous driving scene in the present embodiment. Through the above-mentioned detailed description of the three-dimensional reconstruction method of the autonomous driving scene, those skilled in the art can clearly know the implementation method of the three-dimensional reconstruction system of the autonomous driving scene in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0159] The present application also discloses an electronic device, such as Figure 7 , which is a schematic diagram of the structure of an electronic device for a method for 3D reconstruction of an autonomous driving scene provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to the at least one processor for communication, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for the method for 3D reconstruction of an autonomous driving scene.

[0160] Among them, the processor 10 can be composed of an integrated circuit in some embodiments, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes the program or module stored in the memory 11 (for example, executing the three-dimensional reconstruction method of the autonomous driving scene, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0161] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the method program for three-dimensional reconstruction of the autonomous driving scene, but also can be used to temporarily store data that has been output or is to be output.

[0162] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0163] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0164] Figure 7 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0165] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0166] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0167] The present application provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the method for three-dimensional reconstruction of an autonomous driving scene of the above embodiment.

[0168] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0169] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for three-dimensional reconstruction of an autonomous driving scene, characterized in that: Obtain a training image dataset of an autonomous driving scenario; the training image dataset includes a plurality of initial training images taken from different perspectives while the vehicle is driving; Removing the dynamic occluder region and the sky region in the initial training image from the initial training image to obtain a second training image; Summarizing the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset; Obtaining a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image data set; Constructing a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimizing the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene; The sky area and dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene are reconstructed to obtain the final three-dimensional model of the autonomous driving scene.

2. The method for 3D reconstruction of an autonomous driving scene according to claim 1, wherein: The method further comprises: The initial training image is parsed to obtain an instance segmentation image of the initial training image, an initial depth image, and pose information of the initial training image.

3. The method for 3D reconstruction of an autonomous driving scene according to claim 2, wherein: The three-dimensional Gaussian model of the autonomous driving scene is constructed according to the sparse point cloud, and the three-dimensional Gaussian model is optimized to obtain an optimized three-dimensional model of the autonomous driving scene, including: Construct a 3D Gaussian model based on the sparse point cloud and initialize the parameters of the 3D Gaussian model; Determine a two-dimensional plane coordinate system according to the posture information, render and splash the three-dimensional Gaussian model onto the two-dimensional plane coordinate system, and generate a rendered image; Compare the rendered image with the second training image to determine the loss function of the three-dimensional Gaussian model; The parameters of the three-dimensional Gaussian model are optimized according to the loss function to obtain an optimized three-dimensional model of the autonomous driving scene.

4. The method for 3D reconstruction of an autonomous driving scene according to claim 3, wherein: The loss function includes reconstruction loss and optical flow loss Reconstruction losses The expression is in, represents the mean absolute error loss, λ recon To balance the hyperparameters, represents the structural similarity loss, represents the rendered image, and I represents the second training image; Optical flow loss The expression is Where F = {f fwd ,f bwd } represents the ground truth stream, represents Gaussian flow; f fwd represents the forward flow between two adjacent training image frames in the training image dataset, where the training image dataset includes a plurality of training image frames continuously acquired while the vehicle is driving; f bwd represents the backward flow between two adjacent training image frames in the training image set; represents the forward flow in a Gaussian scene predicted by a three-dimensional Gaussian model, Represents the backward flow in a Gaussian scene predicted by a 3D Gaussian model.

5. The method for 3D reconstruction of an autonomous driving scene according to claim 4, wherein: The loss function also includes the depth map supervision loss Among them, D ras represents the depth map of the rendered image, D rmv represents the depth map of the second training image after removing dynamic occluders and the sky from the autonomous driving scene, Corr(D ras ,D rmv ) means D ras and D rmv The Pearson correlation coefficient between .

6. The method for three-dimensional reconstruction of an autonomous driving scene according to any one of claims 1 to 5, characterized in that: The details of the sky area of ​​the autonomous driving scene are restored through the mapping method, and the reconstruction of the sky area in the optimized three-dimensional model of the autonomous driving scene is completed.

7. The method for three-dimensional reconstruction of an autonomous driving scene according to any one of claims 1 to 5, characterized in that: The surfaces of the optimized three-dimensional model of the autonomous driving scene are aligned to achieve reconstruction of the dynamically occluded areas removed from the optimized three-dimensional model of the autonomous driving scene.

8. A 3D reconstruction system for an autonomous driving scene, used to implement the 3D reconstruction method for an autonomous driving scene according to any one of claims 1 to 7, characterized in that: include: An acquisition module is used to obtain a training image dataset for autonomous driving scenarios; The training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving; A separation module, used for separating the dynamic occlusion area and the sky area in the initial training image from the initial training image to obtain a second training image; A summarizing module, used for summarizing the second training images generated based on the initial training images in the training image data set to obtain a second training image data set; A point cloud generation module, configured to obtain a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image data set; A first reconstruction module is used to construct a three-dimensional Gaussian model of the autonomous driving scene according to the sparse point cloud, and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene; The second reconstruction module is used to reconstruct the sky area and the dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the method for three-dimensional reconstruction of an autonomous driving scene as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed by the processor, the method for three-dimensional reconstruction of an autonomous driving scene as described in any one of claims 1 to 7 is implemented.

Images