A method, system, device and storage medium for 3D reconstruction of autonomous driving scenes

Abstract

The present invention relates to the field of computer vision technology and to a method for three-dimensional reconstruction of an autonomous driving scene, comprising: obtaining a training image dataset of the autonomous driving scene; removing dynamic occluder regions and sky regions from the initial training image to obtain a second training image; obtaining a sparse point cloud of the autonomous driving scene after removing the dynamic occluders and sky regions based on the second training image; constructing a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimizing the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene; and reconstructing the sky region and dynamic occluder regions in the optimized three-dimensional model of the autonomous driving scene to obtain a final three-dimensional model of the autonomous driving scene. The present invention also provides a system, device, and storage medium for three-dimensional reconstruction of the autonomous driving scene. The present invention can improve the quality of three-dimensional reconstruction in scenes with dynamic occluders.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, device, and storage medium for three-dimensional reconstruction of an autonomous driving scene. Background Art

[0002] Three-dimensional scene reconstruction and the synthesis of new views are important and complex research tasks in computer vision. They aim to recover the three-dimensional spatial information of a scene from a set of two-dimensional images. This task plays a vital role in many fields, including medical image processing, virtual reality (VR), and autonomous driving.

[0003] Neural Radiance Fields (NeRF) is a method that uses deep learning to extract the geometry and texture of objects from images with multiple viewpoints to generate high-quality 3D reconstructions. NeRF typically uses volumetric ray marching to optimize a multi-layer perceptron (MLP) to represent the color and density of each point in the scene, enabling the rendering of highly realistic 3D models from any angle and distance.

[0004] NeRF's variant method further improves the reconstruction effect and rendering speed, making the 3D reconstruction process more natural and efficient, but it still has problems such as high resource consumption and slow rendering speed.

[0005] 3D Gaussian Splatting for Real-Time Radiance Field Rendering (3DGS) is another 3D reconstruction method. 3DGS achieves both improved training speed and rendering quality by leveraging its explicit representation and the advantages of differentiable rendering. 3DGS defines a set of anisotropic Gaussians in the 3D world and performs adaptive density control to achieve high-quality rasterized rendering results using only sparse point cloud input derived from Structure-from-Motion (SfM) techniques.

[0006] However, 3DGS only performs well in static scenes; dynamic scenes often suffer from artifacts and other issues. Existing 3DGS variants struggle to overcome the challenges of 3D reconstruction in unbounded scenes for autonomous driving. Therefore, improving the quality of reconstruction after removing dynamic occluders from autonomous driving scenes is a pressing research issue. Summary of the Invention

[0007] In order to improve the quality of three-dimensional reconstruction in scenes with dynamic occluders, a method, system, device and storage medium for three-dimensional reconstruction of autonomous driving scenes are provided.

[0008] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for 3D reconstruction of an autonomous driving scene, comprising:

[0009] Obtain a training image dataset for an autonomous driving scenario; the training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving;

[0010] removing the dynamic occlusion region and the sky region in the initial training image from the initial training image to obtain a second training image;

[0011] Summarizing the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset;

[0012] Obtaining a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image dataset;

[0013] Construct a 3D Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimize the parameters of the 3D Gaussian model to obtain an optimized 3D model of the autonomous driving scene;

[0014] The sky area and dynamic occlusion area in the optimized 3D model of the autonomous driving scene are reconstructed to obtain the final 3D model of the autonomous driving scene.

[0015] Optionally, the method further includes:

[0016] The initial training image is parsed to obtain an instance segmentation image, an initial depth image, and pose information of the initial training image.

[0017] Optionally, constructing a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud and optimizing the three-dimensional Gaussian model to obtain a final three-dimensional model of the autonomous driving scene includes:

[0018] Construct a 3D Gaussian model based on the sparse point cloud and initialize the parameters of the 3D Gaussian model;

[0019] Determine a two-dimensional plane coordinate system based on the pose information, render and splash the three-dimensional Gaussian model onto the two-dimensional plane coordinate system, and generate a rendered image;

[0020] Compare the rendered image with the second training image to determine the loss function of the 3D Gaussian model;

[0021] The parameters of the three-dimensional Gaussian model are optimized according to the loss function to obtain an optimized three-dimensional model of the autonomous driving scene.

[0022] Optionally, the loss function includes a reconstruction loss and optical flow loss

[0023] Reconstruction losses The expression is

[0024]

[0025] in, represents the mean absolute error loss, λ recon To balance the hyperparameters, represents the structural similarity loss, represents the rendered image, and I represents the second training image;

[0026] Optical flow loss The expression is

[0027]

[0028] Where F={f fwd ,f bwd} represents the ground truth stream, represents Gaussian flow; f fwd represents the forward flow between two adjacent training image frames in the training image dataset, where the training image dataset includes multiple training image frames continuously acquired while the vehicle is driving; f bwd represents the backward flow between two adjacent training image frames in the training image set; represents the forward flow in the Gaussian scene predicted by the 3D Gaussian model, Represents the backward flow in a Gaussian scene predicted by a 3D Gaussian model.

[0029] Optionally, the loss function also includes a depth map supervision loss

[0030] Depth map supervision loss The expression is

[0031]

[0032] Among them, D ras represents the depth map of the rendered image,

[0033] D rmv represents the depth map of the second training image after removing dynamic occluders and the sky from the autonomous driving scene.

[0034] Corr(D ras ,D rmv ) means D ras and D rmv The Pearson correlation coefficient between .

[0035] Optionally, the details of the sky area of ​​the autonomous driving scene are restored by a mapping method to complete the reconstruction of the sky area in the optimized three-dimensional model of the autonomous driving scene.

[0036] Optionally, the surfaces of the optimized three-dimensional model of the autonomous driving scene are aligned to reconstruct the dynamically occluded areas that have been removed from the optimized three-dimensional model of the autonomous driving scene.

[0037] In order to solve the above problems, the present invention further provides a three-dimensional reconstruction system for an autonomous driving scene, the system comprising:

[0038] An acquisition module is used to acquire a training image dataset for an autonomous driving scenario; the training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving;

[0039] a separation module, configured to separate the dynamic occlusion region and the sky region in the initial training image from the initial training image to obtain a second training image;

[0040] a summarizing module, configured to summarize the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset;

[0041] a point cloud generation module, configured to obtain, based on the second training image dataset, a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky;

[0042] A first reconstruction module is used to construct a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene;

[0043] The second reconstruction module is used to reconstruct the sky area and dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

[0044] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned three-dimensional reconstruction method of the autonomous driving scene.

[0048] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one computer program. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned three-dimensional reconstruction method of the autonomous driving scene.

[0049] In summary, this application has the following beneficial technical effects:

[0050] By inputting a sparse point cloud after removing dynamic occluders and the sky, and rendering the sky separately, the background holes caused by removing dynamic occluders are optimized; the depth map of the second training image is used for supervision to improve the 3D reconstruction effect after removing dynamic occluders in autonomous driving scenarios; in the input training image, the sky area is separated by instance segmentation image. During the scene rendering process, the sky is not considered for rendering, but is included when generating the final rendered image. This can avoid the problem of Gaussian points in the sky area causing floating shadows due to incorrect depth when rendering using the 3D Gaussian method. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a flow chart of a method for 3D reconstruction of an autonomous driving scene provided by one embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a flow chart of steps for generating a three-dimensional Gaussian model based on a coefficient point cloud and optimizing the three-dimensional Gaussian model according to an embodiment of the present invention;

[0053] Figure 3 This is a system block diagram of a method for 3D reconstruction of an autonomous driving scene provided in a preferred embodiment of the present invention;

[0054] Figure 4 This is a comparison chart of the 3D reconstruction effect of the 3D reconstruction method of the autonomous driving scene in the present application and the 3D reconstruction effect of the existing 3D Gaussian splash in a static scene;

[0055] Figure 5 This is a 3D reconstruction rendering of the 3D reconstruction method for autonomous driving scenes in this application under dynamic scenes;

[0056] Figure 6 This is a functional module diagram of a 3D reconstruction system for an autonomous driving scene provided by one embodiment of the present invention;

[0057] Figure 7 A schematic diagram of the structure of an electronic device for implementing the method for three-dimensional reconstruction of an autonomous driving scene provided in one embodiment of the present invention.

[0058] Figure numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface; 31, acquisition module; 32, separation module; 33, aggregation module; 34, point cloud generation module; 35, first reconstruction module; 36, second reconstruction module.

[0059] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0060] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0061] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0062] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0063] Reference Figure 1 FIG. 1 is a flow chart of a method for 3D reconstruction of an autonomous driving scene according to an embodiment of the present invention. In this embodiment, the method for 3D reconstruction of an autonomous driving scene includes:

[0064] S1. Obtain a training image dataset for autonomous driving scenarios.

[0065] The training image dataset includes multiple initial training images taken from different perspectives while the vehicle is in motion. This dataset can be formed by capturing several image frames in chronological order from the video captured while the vehicle is in motion. Alternatively, the dataset can be obtained by continuously capturing multiple images as initial training images while the vehicle is in motion. The initial training images corresponding to autonomous driving scenarios include elements such as road conditions, traffic signals, pedestrians, and other vehicles. This image data is captured in real time by the autonomous vehicle's cameras and processed in combination with other sensor information to assist the system in environmental perception, path planning, and decision execution.

[0066] The initial training image is an RGB image, also known as a three-channel color image. Each pixel is composed of the intensity values ​​of the three color channels of red, green, and blue. It is widely used in object recognition and scene understanding. In this embodiment, the 3D reconstruction method of the autonomous driving scene also includes:

[0067] The initial training image is parsed to obtain an instance segmentation image, an initial depth image, and pose information of the initial training image.

[0068] Specifically, for example, Depth Anything V2 processes multiple sets of RGB images from different perspectives to obtain an initial depth map of the autonomous driving scene. The initial depth map can determine the distance from each pixel in the autonomous driving scene to the camera, which helps to accurately identify obstacles or occlusions.

[0069] The RGB image is processed using the Segment Anything Model (SAM) to generate instance segmentation images of the autonomous driving scene. The instance segmentation images include objects (such as roads, pedestrians, other vehicles, traffic lights, streetlights, green plants, or the sky) present in the initial training image and the regions corresponding to the objects.

[0070] The training images in the training image dataset are processed through Structure-from-Motion to obtain the pose information of the autonomous driving scene. The pose information can be used to determine the position and angle of the camera that took the initial training images, so as to facilitate the subsequent restoration of the three-dimensional model of the autonomous driving scene.

[0071] S2. Remove the dynamic occluder region and the sky region in the initial training image from the initial training image to obtain a second training image.

[0072] Specifically, the area corresponding to the dynamic occluder and the area corresponding to the sky are screened out from the objects identified in the instance segmentation image; the area corresponding to the dynamic occluder and the area corresponding to the sky are separated from the initial training image to obtain the second training image.

[0073] S3. Summarize the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset.

[0074] S4. Obtain a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image dataset.

[0075] In this embodiment, the second training image in the second training image dataset is processed using the Structure-from-Motion (SfM) technology to obtain a sparse point cloud after removing dynamic occluders and the sky.

[0076] S5. Construct a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene.

[0077] Reference Figure 2 , step S5 includes:

[0078] S51. Construct a three-dimensional Gaussian model according to the sparse point cloud and initialize the parameters of the three-dimensional Gaussian model.

[0079] The sparse point cloud data is preprocessed. The preprocessing process includes data cleaning and filtering. Data cleaning can remove abnormal points and outliers. Filtering can choose methods such as Gaussian filtering or median filtering to reduce the impact of noise on post-processing. Subsequently, the preprocessed point cloud data is voxelized, that is, the point cloud data is divided into multiple voxels. The center of each voxel is regarded as an anchor point and is assigned local context features, scaling factors and learnable offsets. After that, a three-dimensional Gaussian model is constructed based on the voxelization. Starting from the sparse point cloud, a set of three-dimensional Gaussian distributions is created. Each point in the sparse point cloud is regarded as the mean of a three-dimensional Gaussian distribution, and each point is assigned initial covariance, transparency, color and other attributes. The final three-dimensional Gaussian model is obtained by optimizing these Gaussian distributions.

[0080] The three-dimensional Gaussian model is a dynamic three-dimensional Gaussian model. Specifically, the parameters of the three-dimensional Gaussian model include center position, rotation matrix, cosquare matrix, scaling matrix, transparency and spherical harmonic coefficients. The color of the three-dimensional Gaussian model is determined by the spherical harmonic coefficients.

[0081] We capture a set of video frames during vehicle movement, with each frame representing a moment in time. For a video set with t frames, there are t moments in time. If a 3D Gaussian model is represented by n Gaussian spheres (n is a positive integer), the position and rotation information of each Gaussian sphere is modeled using time t. Based on the properties of each dynamic Gaussian at time t, the center position and rotation matrix of the Gaussian sphere at that time can be determined. The color, transparency, and scale of the Gaussian sphere remain constant over time.

[0082] The properties of the dynamic Gaussian at time t are as follows: The expression of the center position is x(t), y(t), z(t) are the spatial coordinates corresponding to the Gaussian sphere, [.] T represents the transpose of [x(t), y(t), z(t)], Refers to the space composed of vectors in three directions.

[0083] The rotation matrix of the dynamic Gaussian is represented by quaternion. Specifically, the expression of the rotation matrix is Among them, q w (t) is the real part, q x (t),q y (t) and q z (t) is the imaginary part.

[0084] The expression of the scaling matrix is s x ,s y ,s z is the scaling factor.

[0085] In this embodiment, the expression G of the probability density function for any point x in space at time t is t (x) is:

[0086]

[0087] Where x is the index of a point in the sparse point cloud in space, μ(t) is the center position of point x at time t, ∑(t) represents the 3D covariance matrix at time t, and its 2D covariance matrix ∑(t)′=JW∑(t)W T J T , W is the observation transformation, J is the Gaussian center position

[0088] The affine approximation Jacobian matrix of the projection transformation at μ(t). To prevent overfitting, the Fourier approximation is used to approximate the 3D position of the Gaussian sphere. At time t, the 3D position of the Gaussian sphere is expressed as follows:

[0089]

[0090]

[0091] where w ·,0 ,…,w ·,2L represents the intercept and coefficient of the 3D position, L represents the number of harmonics, and i represents the index of the sparse point cloud in space.

[0092] To simplify the rotation matrix, a linear approximation is used to approximate the quaternion, which is expressed as follows:

[0093] q x (t) = w qx,0 +w qx,1 t,q y (t) = w qy,0 +w qy,1 t

[0094] q z (t) = w qz,0 +w qz,1 t,q w (t) = w qw,0 +w qw,1 t

[0095] where w ·,0 Indicates the intercept of the 3D rotation, w ·,1 The coefficient representing the 3D rotation.

[0096] S52, determining a two-dimensional plane coordinate system according to the posture information, rendering and splashing the three-dimensional Gaussian model onto the two-dimensional plane coordinate system to generate a rendered image;

[0097] Using the original 3D Gaussian rendering technique, the 3D Gaussian is splashed onto the 2D camera plane coordinate system. Each 2D plane image is divided into 16×16 blocks, and the 2D Gaussians within all blocks are sorted by depth. The color c of each pixel is calculated based on rasterization by evaluating the mixture of N ordered points that overlap with the pixel:

[0098]

[0099] Where N is the number of points where pixels overlap in the rendered image, i is the index of the point where pixels overlap in the rendered image, j is the index of the point before point i, and c is the index of the point before point i. i represents the color of the Gaussian sphere evaluated by the spherical harmonic coefficients SH; α′ i Represents the opacity (density) value of the current point i, α′ i =∑′α i Indicates the opacity of the Gaussian sphere; α′ j Represents the opacity (density) value of the j points before i, using 1-α′ i Multiply it cumulatively as weight, which means the more transparent the previous point is, the greater the weight is. The current α′i The greater the influence weight.

[0100] S53, comparing the rendered image with the second training image to determine the loss function of the three-dimensional Gaussian model;

[0101] The loss function includes reconstruction loss and optical flow loss

[0102] Reconstruction losses The expression is

[0103]

[0104] in, represents the mean absolute error loss, λ recon is the first balancing hyperparameter, represents the structural similarity loss, represents the rendered image, and I represents the second training image.

[0105] In the dynamic stage, in order to provide spatiotemporal consistency of n-frame images, the optical flow loss is introduced

[0106] Optical flow loss The expression is

[0107]

[0108] Where F={f fwd ,f bwd} represents the ground truth stream, represents Gaussian flow; f fwd represents the forward flow between two adjacent training image frames in the training image dataset, where the training image dataset includes multiple training image frames continuously acquired while the vehicle is driving; f bwd represents the backward flow between two adjacent training image frames in the training image set, f fwd and f bwd They are obtained by RAFT algorithm (forward propagation and back propagation algorithm in neural network); represents the forward flow in the Gaussian scene predicted by the 3D Gaussian model, Represents the backward flow in the Gaussian scene predicted by the three-dimensional Gaussian model; it should be noted that the forward flow is the mapping from the first frame to the second frame; the backward flow is the mapping from the second frame to the first frame.

[0109]

[0110] Δt is the difference between the time steps of two training image frames.

[0111] The depth rasterizer is used to perform depth testing to determine the occlusion relationship of pixels and ensure the correct rendering. The differentiable depth rasterizer receives the rendered image depth map D ras And the depth map D after removing dynamic occluders and sky in the autonomous driving scene rmv The error signal between the depth of each pixel is calculated by accumulating the z-buffer d of the ordered Gaussians that contribute to that pixel. i get:

[0112]

[0113] At the same time, in order to reduce the rendering image depth map D ras And the depth map D after removing dynamic occluders and sky in the autonomous driving scene rmv The scale fuzziness between them is introduced by the Pearson correlation coefficient:

[0114]

[0115] The loss function also includes the depth map supervision loss

[0116] Depth map supervision loss The expression is

[0117]

[0118] Among them, D ras represents the rendered image depth map,

[0119] D rmv The depth map Corr(D) of the second training image after removing dynamic occluders and sky in the autonomous driving scene ras ,D rmv ) means D ras and D rmv The Pearson correlation coefficient between .

[0120] The expression of Pearson correlation coefficient is:

[0121]

[0122] Loss Function The expression is

[0123] λ flow is the second balancing hyperparameter, λ depth is the third balancing hyperparameter.

[0124] S54. Optimize the parameters of the three-dimensional Gaussian model according to the loss function to obtain an optimized three-dimensional model of the autonomous driving scene.

[0125] S6. Reconstruct the sky area and the dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

[0126] In this embodiment, a mapping method is used to restore details of the sky region in the autonomous driving scene, completing the reconstruction of the sky region in the optimized 3D model of the autonomous driving scene. Because 3DGS are defined in Euclidean space, they are not suitable for reconstructing distant areas such as the sky. Therefore, a mapping method is used to restore details of the sky region. In the input training image, the sky region is separated through instance segmentation. During the scene rendering process, the sky is not considered, but is included in the final rendered image. This avoids the problem of Gaussian points in the sky region causing ghosting during rendering due to incorrect depth when using the 3D Gaussian method.

[0127] The surfaces of the optimized 3D model of the autonomous driving scene are aligned to reconstruct the removed dynamic occlusion areas.

[0128] Specifically, the method of surface alignment Gaussian model includes:

[0129] For any spatial position in the scene The calculation formula of the density function ρ(x) is as follows:

[0130]

[0131] The density function ρ(x) is the sum of the Gaussian values, N represents the total number of Gaussian balls in the space, i represents the index of the Gaussian ball in the space, and the density function ρ(x) is calculated by adding the mixing coefficient α of each Gaussian ball i i With each Gaussian G t (i) Multiply and sum.

[0132] Reference Figure 3 In the preferred implementation of this embodiment, in order to better reconstruct the area where the dynamic occluder is removed, surface alignment normalization is applied by controlling the density function of the Gaussian. In order to control the density function of the Gaussian and apply surface alignment normalization, there are three assumptions for any point x close to the scene surface: (1) The Gaussian g closest to point x * has a greater contribution to ρ(x); (2) The ideal three-dimensional Gaussian distribution of the surface is flat, so each Gaussian g has one of the scale factors in the three axis directions approaching 0; (3) In order to describe the real surface, the Gaussian transparency α is required i = 1. The density function is approximated as:

[0133]

[0134] is the approximate value of the density function of the missing part of the three-dimensional Gaussian model, represents the minimum scale factor of Gaussian, is the Gaussian g closest to point x * The center point coordinates of is the Gaussian g closest to point x * The direction of the corresponding axis.

[0135] Based on the density function ρ(x) and Get the approximate signed distance function (SDF), specifically, the signed distance function:

[0136]

[0137] In order to reduce the calculation, Use the depth map D of the rendered image corresponding to point x ras The depth value is replaced by the value of d(x). Then the difference between d(x) and the depth value at point x is The regularization term with d(x) is expressed as:

[0138]

[0139] Where X represents all points sampled onto the surface of the three-dimensional Gaussian model.

[0140] At the same time, a regularization term is added for the normal direction:

[0141]

[0142] It represents the gradient vector of the function d(x) at point x.

[0143] In addition, to encourage opaque Gaussians, cross entropy loss is used for opacity regularization:

[0144]

[0145] Δσ represents the opacity increment or change of the j-th pixel or point, and σ represents opacity.

[0146] In the preferred implementation of this embodiment, the loss function of the 3D reconstruction model is The expression is:

[0147]

[0148] Among them, λ surface is the fourth balancing hyperparameter, λ SDF is the fifth balancing hyperparameter, λ norm is the sixth balancing hyperparameter, λopacity is the seventh balance hyperparameter; is the signed distance loss, is the regularization loss, is the cross entropy loss.

[0149] To verify the capabilities of the 3D reconstruction method for autonomous driving scenes provided in this application, we used the existing realistic dataset Virtual KITTI (VKITTI) for synthesizing autonomous driving scenes to verify the 3D reconstruction effect of autonomous driving scenes.

[0150] The Virtual KITTI (VKITTI) dataset consists of five different virtual worlds (Scene01, Scene02, Scene06, Scene18, and Scene20) in urban environments. It is automatically, accurately, and fully annotated with pixel-level category, instance, and depth labels. Furthermore, to further verify the versatility of our approach in real-world scenarios, we also selected the nuScenes dataset for validation.

[0151] The experimental environment is: Ubuntu 22.04 operating system, NVIDIA RTX 3060 GPU, Python 3.8, PyTorch 1.13.1, CUDA 11.7. The model went through 30,000 iterations, including 3,000 iterations of static optimization and 27,000 iterations of dynamic optimization. During the training process, the hyperparameter values ​​are as follows: L = 2, λ recon =0.8,λ flow =1.0,λ depth =0.05,λ surface =0.2,λ SDF =1.0,λ norm =1.0,λ opacity =0.5.

[0152] Figure 4 This is a comparison chart of the 3D reconstruction effect of the 3D reconstruction method of the autonomous driving scene in the present application and the 3D reconstruction effect of the existing 3D Gaussian splash in a static scene; Figure 4 The first column GT on the left represents the input initial training image, the second column is the autonomous driving scene reconstructed using the autonomous driving scene 3D reconstruction method of this application; the third column is the autonomous driving scene obtained by 3D reconstruction using the existing 3D Gaussian splash method.

[0153] Figure 5This is a three-dimensional reconstruction rendering of the three-dimensional reconstruction method of the autonomous driving scene of the present application under a dynamic scene. The images in the first row and the third row are initial training images collected under a dynamic autonomous driving scene. The images in the second row are schematic diagrams of the autonomous driving scene reconstructed after removing dynamic obstacles from the initial training images in the first row. The images in the fourth row are schematic diagrams of the autonomous driving scene reconstructed after removing dynamic obstacles from the initial training images in the third row.

[0154] Reference Figure 6 Based on the same inventive concept, an embodiment of the present invention provides a three-dimensional reconstruction system for an autonomous driving scene.

[0155] The 3D reconstruction system for autonomous driving scenes described in the present invention can be installed in an electronic device. Based on the functionality implemented, the system includes an acquisition module 31, a separation module 32, a summary module 33, a point cloud generation module 34, a first reconstruction module 35, and a second reconstruction module 36. A module, also referred to as a unit, is a series of computer program segments that can be executed by an electronic device processor and perform a fixed function. These are stored in the electronic device's memory.

[0156] The acquisition module 31 can acquire a training image dataset of an autonomous driving scene; the training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving; the separation module 32 can remove the dynamic occlusion area and the sky area from the initial training image to obtain a second training image; the aggregation module 33 can aggregate the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset;

[0157] The point cloud generation module 34 can obtain a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image data set; the first reconstruction module 35 can construct a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene; the second reconstruction module 36 can reconstruct the sky area and the dynamic occluder area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

[0158] The various variations and specific examples of the three-dimensional reconstruction method for autonomous driving scenes provided in the above embodiments are also applicable to the three-dimensional reconstruction system for autonomous driving scenes in this embodiment. Through the above detailed description of the three-dimensional reconstruction method for autonomous driving scenes, those skilled in the art can clearly understand the implementation method of the three-dimensional reconstruction system for autonomous driving scenes in this embodiment. For the sake of brevity of the specification, they will not be described in detail here.

[0159] This application also discloses an electronic device, such as Figure 7 Figure 1 is a schematic diagram of the structure of an electronic device for implementing a method for 3D reconstruction of an autonomous driving scene according to an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 communicatively coupled to the at least one processor, a communication bus 12, and a communication interface 13. The electronic device may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for implementing the method for 3D reconstruction of an autonomous driving scene.

[0160] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the memory 11 (such as executing a three-dimensional reconstruction method for an autonomous driving scene) and calling data stored in the memory 11 to perform various functions of the electronic device and process data.

[0161] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various types of data installed in the electronic device, such as the code of the method program for three-dimensional reconstruction of an autonomous driving scene, but also to temporarily store data that has been output or is to be output.

[0162] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0163] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0164] Figure 7 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 7 The structure shown does not constitute a limitation of the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) to power each component. Preferably, the power supply can be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0165] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0166] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0167] The present application provides a computer-readable storage medium, for example, including any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). The computer-readable storage medium stores a computer program capable of being loaded by a processor and executing the method for three-dimensional reconstruction of an autonomous driving scene according to the above-described embodiment.

[0168] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0169] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for 3D reconstruction of an autonomous driving scene, characterized in that: The method comprises: Obtain a training image dataset for an autonomous driving scenario; the training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving; removing the dynamic occlusion region and the sky region in the initial training image from the initial training image to obtain a second training image; Summarizing the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset; Obtaining a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky based on the second training image dataset; Construct a 3D Gaussian model of the autonomous driving scene based on the sparse point cloud, and optimize the parameters of the 3D Gaussian model to obtain an optimized 3D model of the autonomous driving scene; Reconstruct the sky area and dynamic occlusion area in the optimized 3D model of the autonomous driving scene to obtain the final 3D model of the autonomous driving scene; Parse the initial training image to obtain the instance segmentation image, initial depth image, and pose information of the initial training image; A 3D Gaussian model of the autonomous driving scene is constructed based on the sparse point cloud, and the 3D Gaussian model is optimized to obtain an optimized 3D model of the autonomous driving scene, including: Construct a 3D Gaussian model based on the sparse point cloud and initialize the parameters of the 3D Gaussian model; Determine a two-dimensional plane coordinate system based on the pose information, render and splash the three-dimensional Gaussian model onto the two-dimensional plane coordinate system, and generate a rendered image; Compare the rendered image with the second training image to determine the loss function of the 3D Gaussian model; Optimize the parameters of the three-dimensional Gaussian model according to the loss function to obtain an optimized three-dimensional model of the autonomous driving scene; The loss function includes reconstruction loss and optical flow loss Reconstruction losses The expression is in, represents the mean absolute error loss, λ recon To balance the hyperparameters, represents the structural similarity loss, represents the rendered image, and I represents the second training image; Optical flow loss The expression is Where F={f fwd ,f bwd } represents the ground truth stream, represents Gaussian flow; f fwd represents the forward flow between two adjacent training image frames in the training image dataset, where the training image dataset includes multiple training image frames continuously acquired while the vehicle is driving; f bwd represents the backward flow between two adjacent training image frames in the training image set; represents the forward flow in the Gaussian scene predicted by the 3D Gaussian model, represents the backward flow in a Gaussian scene predicted by a 3D Gaussian model; In order to reduce the rendering of the image depth map D ras And the depth map D after removing dynamic occluders and sky in the autonomous driving scene rmv The scale fuzziness between them is introduced by the Pearson correlation coefficient: The loss function also includes the depth map supervision loss Among them, D ras represents the depth map of the rendered image, D rmv represents the depth map of the second training image after removing dynamic occluders and the sky from the autonomous driving scene. Corr(D ras ,D rmv ) means D ras and D rmv The Pearson correlation coefficient between .

2. The method for 3D reconstruction of an autonomous driving scene according to claim 1, wherein: The details of the sky area of ​​the autonomous driving scene are restored through the mapping method, and the reconstruction of the sky area in the optimized three-dimensional model of the autonomous driving scene is completed.

3. The method for 3D reconstruction of an autonomous driving scene according to claim 1, wherein: The surfaces of the optimized three-dimensional model of the autonomous driving scene are aligned to reconstruct the dynamically occluded areas removed from the optimized three-dimensional model of the autonomous driving scene.

4. A 3D reconstruction system for an autonomous driving scene, for implementing the 3D reconstruction method for an autonomous driving scene according to any one of claims 1 to 3, characterized in that: include: Acquisition module, used to obtain training image datasets for autonomous driving scenarios; The training image dataset includes multiple initial training images taken from different perspectives while the vehicle is driving; a separation module, configured to separate the dynamic occlusion region and the sky region in the initial training image from the initial training image to obtain a second training image; a summarizing module, configured to summarize the second training images generated based on the initial training images in the training image dataset to obtain a second training image dataset; a point cloud generation module, configured to obtain, based on the second training image dataset, a sparse point cloud of the autonomous driving scene after removing dynamic occluders and the sky; A first reconstruction module is used to construct a three-dimensional Gaussian model of the autonomous driving scene based on the sparse point cloud and optimize the parameters of the three-dimensional Gaussian model to obtain an optimized three-dimensional model of the autonomous driving scene; The second reconstruction module is used to reconstruct the sky area and dynamic occlusion area in the optimized three-dimensional model of the autonomous driving scene to obtain the final three-dimensional model of the autonomous driving scene.

5. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively coupled to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the method for three-dimensional reconstruction of an autonomous driving scene as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer storage medium stores a computer program; when the computer program is executed by the processor, the method for three-dimensional reconstruction of an autonomous driving scene according to any one of claims 1 to 3 is implemented.

Images