Automatic driving scene rapid high-fidelity reconstruction method based on semantic guidance

Abstract

The invention discloses an automatic driving scene rapid high-fidelity reconstruction method based on semantic guidance, and belongs to the technical field of automatic driving and computer vision. The invention aims to solve the problem that the fidelity of a security key object is insufficient due to the fact that resources are allocated to all elements in a scene without distinguishing in the prior art. Comprising the following steps: establishing a three-dimensional Gaussian full-scene synthesis model; obtaining pixel-level masks of all scene elements in the two-dimensional image of the target scene; calculating the overlapping degree of the two-dimensional Gaussian projection image and the key category semantic mask to obtain a static semantic score, and combining the gradient contribution score to obtain a mixed importance score; carrying out iterative optimization on the three-dimensional Gaussian full-scene synthesis model; carrying out adaptive pruning and random selective discarding based on semantic guidance; and according to the three-dimensional Gaussian configuration high-fidelity or low-fidelity label, rendering is carried out respectively, and a high-fidelity reconstruction image is obtained. According to the invention, task-oriented efficient and high-fidelity reconstruction is realized.

Description

TECHNICAL FIELD

[0001] The application relates to a semantic-guided fast high-fidelity reconstruction method for an autonomous driving scene, and belongs to the technical field of autonomous driving and computer vision. BACKGROUND

[0002] Three-dimensional reconstruction of dynamic and large-scale urban scenes is an important part of the simulation and testing of autonomous driving. With the development of 3D Gaussian Splatting (3DGS) technology, the realism and real-time performance of new view synthesis have been significantly improved.

[0003] Currently, a mainstream technical paradigm is to decompose a complex scene into a static background and multiple independently modeled dynamic foregrounds. However, there is a key limitation in existing technologies: they are essentially semantically agnostic. This means that these methods allocate equal computational resources and representation capabilities to all elements in the scene, whether it is a distant static building or a nearby pedestrian crossing the road. For safety-critical autonomous driving applications, this indiscriminate allocation of resources is suboptimal and wasteful, as it fails to prioritize high-fidelity reconstruction of safety-critical objects such as vehicles and pedestrians. This deficiency constitutes a major bottleneck for current technology optimization, and a new task-oriented reconstruction paradigm is urgently needed, i.e., the reconstruction process should be explicitly guided by the semantic importance of scene elements. SUMMARY

[0004] In view of the problem that the existing technology allocates resources to all elements in the scene without distinction, resulting in insufficient fidelity to safety-critical objects, the application provides a semantic-guided fast high-fidelity reconstruction method for an autonomous driving scene.

[0005] The semantic-guided fast high-fidelity reconstruction method for an autonomous driving scene provided by the application comprises the following steps:

[0006] A three-dimensional Gaussian is used to establish a static background model and a dynamic foreground model for the target scene, and a three-dimensional Gaussian full-scene synthesis model is obtained;

[0007] A pixel-level mask is generated for all scene elements in the two-dimensional image of the target scene using SAM, and a key category semantic mask is labeled for the moving target using CLIP;

[0008] The three-dimensional Gaussian full-scene synthesis model is projected onto the two-dimensional image of the target scene to obtain a two-dimensional Gaussian projection image, the degree of overlap between the two-dimensional Gaussian projection image and the key category semantic mask is calculated to obtain a static semantic score, and the gradient contribution score of the two-dimensional Gaussian projection image to the 2D projection of the image reconstruction loss is calculated; and a hybrid importance score is calculated based on the static semantic score and the gradient contribution score;

[0009] Iterative optimization of the 3D Gaussian full-scene synthesis model: based on the mixture importance score, the 3D Gaussian of the 3D Gaussian full-scene synthesis model is subjected to semantic-guided adaptive pruning; at the same time, the dropout rate is calculated based on the static semantic score, and the 3D Gaussian of the 3D Gaussian full-scene synthesis model is randomly and selectively dropped, and the opacity of the remaining 3D Gaussian is compensated.

[0010] After training, based on the static semantic score, the optimized 3D Gaussian full-scene synthesis model is assigned a high-fidelity or low-fidelity label. The 3D Gaussian with the high-fidelity label is rendered using the full spherical harmonic coefficients up to the third order, while the 3D Gaussian with the low-fidelity label is rendered using the zero-order spherical harmonic coefficients to obtain a high-fidelity reconstructed image.

[0011] According to the semantically guided fast high-fidelity reconstruction method for autonomous driving scenes of the present invention, a static background model in the world coordinate system is established using LiDAR point cloud and SfM point cloud collected by the vehicle; the SfM point cloud is obtained by colmap from the two-dimensional image of the target scene; and a dynamic foreground model in the local coordinate system is established using LiDAR point cloud.

[0012] By combining the rotation matrix and translation vector of the moving target, the dynamic foreground model in the local coordinate system is transformed into the world coordinate system, and then stitched with the static background model to obtain a 3D Gaussian full-scene synthesis model.

[0013] According to the semantically guided fast high-fidelity reconstruction method for autonomous driving scenes of the present invention, the four-dimensional spherical harmonic coefficients of the dynamic foreground model are represented by Fourier coefficients:

[0014] ,

[0015] In the formula Let be the m-th order l-th spherical harmonic coefficients, and k be the number of Fourier coefficients; Let i be the i-th Fourier coefficient. Let i be the i-th Fourier basis function. For frame number, .

[0016] According to the semantically guided fast high-fidelity reconstruction method for autonomous driving scenes of the present invention, the gradient contribution score of the j-th 3D Gaussian is represented as... :

[0017] ,

[0018] In the formula Let j be the 2D projection value of the j-th 3D Gaussian. This indicates rendering a two-dimensional image. Represents rendering a 2D image pair partial derivative of the loss function with respect to the jth three-dimensional Gaussian.

[0019] According to the method for quickly and high-fidelity reconstruction of an autonomous driving scene based on semantic guidance, the static semantic score is represented as ; and the mixed importance score is represented as .

[0020] ,

[0021] In the formula, the static semantic score is represented as .

[0022] According to the method for quickly and high-fidelity reconstruction of an autonomous driving scene based on semantic guidance, the adaptive pruning based on semantic guidance comprises:

[0023] During the training densification, in the selected number of iterations, the mixed importance score is used to sort all the current three-dimensional Gaussians, and the pruning operation is performed on the 60% of the sorted three-dimensional Gaussians; in the fine-tuning stage, the pruning operation is performed on the 30% of the current sorted three-dimensional Gaussians at a set frequency.

[0024] According to the method for quickly and high-fidelity reconstruction of an autonomous driving scene based on semantic guidance, the dropout rate of the jth three-dimensional Gaussian is represented as .

[0025] ,

[0026] In the formula, the current iteration round is represented as , is an hyperparameter related to the dropout rate, is the static semantic score of the jth three-dimensional Gaussian, is a dropout rate adjustment hyperparameter, is the total iteration round.

[0027] According to the method for quickly and high-fidelity reconstruction of an autonomous driving scene based on semantic guidance, the compensation factor for compensating the opacity of the remaining jth three-dimensional Gaussian is represented as .

[0028] .

[0029] According to the method for quickly and high-fidelity reconstruction of an autonomous driving scene based on semantic guidance, after the training is completed, the static semantic score greater than 0.5 in the optimized three-dimensional Gaussian full-scene synthesis model is configured with a high-fidelity label LOD0, and the static semantic score less than or equal to 0.5 is configured with a low-fidelity label LOD1.

[0030] ​The loss function used in the training process of the three-dimensional Gaussian full-scene synthesis model in the automatic driving scene rapid high-fidelity reconstruction method based on semantic guidance according to the application is:

[0031] ,

[0032] In the formula is a color reconstruction loss of a rendered two-dimensional image and a two-dimensional image of a target scene, is a depth supervision loss, is an entropy loss of foreground opacity, is a depth supervision loss weight, is an entropy loss weight.

[0033] The method of the application is used for three-dimensional reconstruction of dynamic and large-scale urban scenes. By embedding semantic importance into each stage of the three-dimensional Gaussian life cycle, a three-dimensional Gaussian splatting (3D Gaussian Splatting) representation method guided by semantic information is used to achieve task-oriented efficient and high-fidelity reconstruction. The method of the application solves the contradiction between reconstruction fidelity and computational efficiency in the prior art, and is particularly suitable for applications such as autonomous driving simulation testing, synthetic data generation, etc. that require efficient and high-fidelity reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the automatic driving scene rapid high-fidelity reconstruction method based on semantic guidance according to the application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0036] Specific embodiment one, in combination with Figure 1 shown, the application provides an automatic driving scene rapid high-fidelity reconstruction method based on semantic guidance, comprising:

[0037] A three-dimensional Gaussian is used to establish a static background model and a dynamic foreground model for a target scene, and a three-dimensional Gaussian full-scene synthesis model is obtained;

[0038] SAM is used to generate a pixel-level mask for all scene elements in a two-dimensional image of the target scene, and CLIP is used to label a key category semantic mask for a moving target;

[0039] The three-dimensional Gaussian full-scene synthesis model is projected onto a two-dimensional Gaussian projection image of a target scene two-dimensional image, an overlap degree of the two-dimensional Gaussian projection image and a key category semantic mask is calculated, and a static semantic score is obtained; meanwhile, a gradient contribution score of the two-dimensional Gaussian projection image to reducing the 2D projection of the image reconstruction loss is calculated; a hybrid importance score is calculated based on the static semantic score and the gradient contribution score;

[0040] The three-dimensional Gaussian full-scene synthesis model is iteratively optimized: the three-dimensional Gaussian of the three-dimensional Gaussian full-scene synthesis model is adaptively pruned based on semantic guidance according to the hybrid importance score; meanwhile, a discard rate is calculated according to the static semantic score, the three-dimensional Gaussian of the three-dimensional Gaussian full-scene synthesis model is randomly and selectively discarded, and the opacity of the remaining three-dimensional Gaussian is compensated;

[0041] After training, a high-fidelity or low-fidelity label is configured for the three-dimensional Gaussian of the optimized three-dimensional Gaussian full-scene synthesis model based on the static semantic score; the three-dimensional Gaussian configured with the high-fidelity label is rendered using complete spherical harmonic coefficients up to the third order, and the three-dimensional Gaussian configured with the low-fidelity label is rendered using zero-order spherical harmonic coefficients, thereby obtaining a high-fidelity reconstruction image.

[0042] The static background model is represented by a group of 3D Gaussian primitives fixed in the world coordinate system, and is initialized using LiDAR point clouds and SfM point clouds collected by the ego vehicle to cover a large area. The three-dimensional Gaussian parameters of each background point are initialized, including: position coordinates , rotation matrix , scaling matrix , opacity , and spherical harmonic coefficients .

[0043] Scene decomposition and semantic acquisition in the embodiment: first, a basic model (such as Segment Anything Model) is used to perform semantic decomposition on the training image, and scene elements are divided into “critical” (such as vehicles, pedestrians, and cyclists) and “non-critical” (such as buildings, roads, and vegetation) two categories. This one-time preprocessing step provides clear and strong semantic prior information for subsequent optimization and rendering. The key category corresponds to a labeled key category semantic mask, and the non-key category corresponds to a labeled non-key category semantic mask.

[0044] Further, a static background model under the world coordinate system is established using LiDAR point clouds and SfM point clouds collected by the ego vehicle; the SfM point cloud is obtained from a target scene two-dimensional image through colmap; a dynamic foreground model under a local coordinate system is established using LiDAR point clouds;

[0045] The dynamic foreground model in local coordinate system is converted to the world coordinate system by combining the rotation matrix and translation vector of the moving target, and spliced with the static background model to obtain a three-dimensional Gaussian full-scene synthesis model.

[0046] Each moving object (e.g. vehicle) is modeled as a set of independent Gaussian primitives in its own local coordinate system. The local Gaussians are converted to the world coordinate system for rendering by a series of time-varying optimizable poses (rotation matrix and translation vector ). To capture the time-varying appearance of dynamic objects due to lighting changes, the present embodiment employs a four-dimensional spherical harmonic model, where each SH coefficient is expressed as a function of time by a set of Fourier coefficients.

[0047] The four-dimensional spherical harmonic coefficients of the dynamic foreground model are expressed by Fourier coefficients:

[0048] ,

[0049] where is the m-th order l-th spherical harmonic coefficient, k is the number of Fourier coefficients; is the i-th Fourier coefficient, is the i-th Fourier basis function, is the frame number, .

[0050] Offline semantic scene decomposition: In the preprocessing stage, the Segment Anything Model (SAM) is used to generate pixel-level semantic masks for all training images.

[0051] The gradient contribution score of the j-th three-dimensional Gaussian is expressed as :

[0052] ,

[0053] where is the 2D projection value of the j-th three-dimensional Gaussian, denotes the rendered two-dimensional image, denotes the partial derivative of the rendered two-dimensional image with respect to .

[0054] The present embodiment proposes an innovative pruning strategy, the core of which is a hybrid importance metric. This metric combines two dimensions of scores: one is the "static semantic score" obtained by projecting the three-dimensional Gaussian onto the two-dimensional image and calculating its overlap with the key mask; the other is the "gradient contribution score" quantifying the instantaneous contribution of the Gaussian to reducing the image reconstruction loss.

[0055] The static semantic score is expressed as ; the hybrid importance score is denoted as :

[0056] ,

[0057] wherein is a static semantic score hyper-parameter, which can be set to 0.4.

[0058] Further, the semantic-guided adaptive pruning comprises:

[0059] During the training densification, at selected iteration times, the hybrid importance score is calculated to prune the top 60% of the three-dimensional Gaussians. In the fine-tuning stage, the top 30% of the three-dimensional Gaussians are pruned at a set frequency. The calculation of the hybrid importance score enables the model to adaptively retain the fine details of key objects (high ) while more aggressively pruning and simplifying non-key background elements.

[0060] As an example, in the first 5k steps of training, the static semantic score of each three-dimensional Gaussian is obtained by calculating the overlap coverage of the rendered picture (two-dimensional Gaussian projection image) and the pixel-level mask, and the gradient contribution score is obtained by squaring the gradient of the rendered image with respect to the two-dimensional projection value of the three-dimensional Gaussian obtained in the back propagation of the training.

[0061] As an example, during the training densification, the top 60% of the three-dimensional Gaussians are pruned at the 10k, 15k and 20k iterations; in the fine-tuning stage, starting from the 25k iteration, the pruning is performed at a rate of 30% every 5k iterations. Experiments show that the hyper-parameter is best when set to 0.4.

[0062] Adaptive random dropout: To solve the problem of overfitting of the model under sparse views, the present embodiment introduces an adaptive random dropout technique, in which the dropout rate of each Gaussian is inversely proportional to its semantic importance. This imposes stronger regularization on the background elements, forcing the model to establish a more robust representation for key objects.

[0063] During the entire training process, an adaptive random dropout strategy is applied. The dropout rate of a Gaussian cell j at time step T is adjusted by its semantic score . This mechanism works in conjunction with the hybrid pruning, and the frequently dropped cells are more likely to be permanently removed later due to receiving fewer gradient updates.

[0064] The dropout rate of the jth three-dimensional Gaussian is denoted as :

[0065] ,

[0066] where is the current iteration round, is a hyper-parameter related to the dropout rate, which can be chosen as 0.5; is the static semantic score of the jth three-dimensional Gaussian, is a dropout rate adjustment hyper-parameter, which can be 0.25; is the total iteration round.

[0067] To ensure that the overall contribution of each pixel remains unchanged, the opacity of the remaining three-dimensional Gaussian that is not randomly discarded in the current round is compensated by a compensation factor. The compensation factor for compensating the opacity of the remaining jth three-dimensional Gaussian is represented as :

[0068] .

[0069] Priority-driven differential rendering pipeline: extend the semantic priority principle to the rendering stage to greatly improve the rendering speed.

[0070] Semantic level of detail (LOD) assignment: after training, according to the final static semantic score, assign a discrete LOD level to each Gaussian. The optimized three-dimensional Gaussian full-scene synthesis model is The three-dimensional Gaussian with a static semantic score greater than 0.5 is configured with a high-fidelity label LOD0, and the three-dimensional Gaussian with a static semantic score less than or equal to 0.5 is configured with a low-fidelity label LOD1.

[0071] Differential rendering: the rendering pipeline reads the data of a Gaussian primitive, which checks the LOD label attached to this Gaussian primitive. For LOD0 primitives, complete, up to third-order SH coefficients are calculated to ensure photo-realistic, view-dependent appearance. For LOD1 primitives, the calculation is truncated and only zero-order SH coefficients, i.e., the basic color independent of the view angle, are used. This method achieves more than 2.2 times speed improvement in rendering while maintaining the highest reconstruction score by concentrating the computational load on semantically important pixels.

[0072] Differential spherical harmonic function calculation: during rendering, different LODs of Gaussians are processed differently. Key objects of LOD0 will calculate complete, up to third-order spherical harmonics (SH) to render realistic view-dependent effects (such as car body reflections). Non-key background elements of LOD1 only calculate zero-order SH coefficients (i.e., the basic color independent of the view angle), which is a computationally extremely inexpensive operation. This method concentrates expensive calculation tasks only on the pixels of key objects, thereby significantly improving the rendering frame rate while maintaining high perceptual quality in key areas.

[0073] In this embodiment, the loss function used in the training process of the three-dimensional Gaussian full scene synthesis model is :

[0074] ,

[0075] wherein is the color reconstruction loss of the rendered two-dimensional image and the target scene two-dimensional image, including L1+D-SSIM loss, which is used to measure the difference between the rendered image and the real image in color and structure, and to ensure the visual quality of the reconstructed image. is the depth supervision loss, is the L1 loss of the rendered depth and the LiDAR projected depth, which improves the accuracy of scene geometry reconstruction by constraining the consistency of the rendered depth and the LiDAR depth. is the entropy loss of the foreground opacity, is the depth supervision loss weight, is the entropy loss weight. Enhances the separation effect of the foreground and the background, and reduces the floating phenomenon of the foreground object in the background.

[0076] Although the present application is described herein with reference to particular embodiments, it is to be understood that these examples are merely illustrative of principles and applications of the present application. It will thus be appreciated that numerous modifications can be made to the illustrative embodiments and that other arrangements can be devised without departing from the spirit and scope of the present application as defined by the appended claims. It will be understood that the features described in relation to one embodiment can be used in other embodiments. It will be further understood that features described in relation to separate embodiments can be used in combination with one another.

Claims

1. A method for semantic-guided fast high-fidelity reconstruction of autonomous driving scenes, characterized in that, The method comprises the following steps: A three-dimensional Gaussian is used to establish a static background model and a dynamic foreground model for a target scene, and a three-dimensional Gaussian full-scene synthesis model is obtained; A pixel-level mask is generated for all scene elements in a two-dimensional image of the target scene using SAM, and a key category semantic mask is labeled for the moving target using CLIP; A two-dimensional Gaussian projection image is obtained by projecting the three-dimensional Gaussian full-scene synthesis model onto the two-dimensional image of the target scene, the degree of overlap between the two-dimensional Gaussian projection image and the key category semantic mask is calculated to obtain a static semantic score, and a gradient contribution score of the two-dimensional Gaussian projection image to reducing the 2D projection of the image reconstruction loss is calculated; a hybrid importance score is calculated based on the static semantic score and the gradient contribution score; Iterative optimization is performed on the three-dimensional Gaussian full-scene synthesis model: the three-dimensional Gaussian of the three-dimensional Gaussian full-scene synthesis model is adaptively pruned based on semantic guidance according to the hybrid importance score; a discard rate is calculated according to the static semantic score, and the three-dimensional Gaussian of the three-dimensional Gaussian full-scene synthesis model is randomly and selectively discarded, and the opacity of the remaining three-dimensional Gaussian is compensated; After training, the three-dimensional Gaussian of the optimized three-dimensional Gaussian full-scene synthesis model is configured with a high-fidelity or low-fidelity label based on the static semantic score; the three-dimensional Gaussian configured with the high-fidelity label is rendered using complete spherical harmonic coefficients up to the third order, and the three-dimensional Gaussian configured with the low-fidelity label is rendered using zero-order spherical harmonic coefficients, to obtain a high-fidelity reconstructed image.

2. The method of claim 1, wherein, A static background model in a world coordinate system is established using LiDAR point clouds and SfM point clouds collected by a self-driving vehicle; the SfM point clouds are obtained from two-dimensional images of a target scene by colmap; a dynamic foreground model in a local coordinate system is established using LiDAR point clouds; the dynamic foreground model in the local coordinate system is converted to the world coordinate system by combining the rotation matrix and translation vector of the moving target, and is spliced with the static background model to obtain a three-dimensional Gaussian full-scene synthesis model.

3. The method of claim 1, wherein, Four-dimensional spherical harmonic coefficients of the dynamic foreground model are represented by Fourier coefficients: ， wherein is the mth order, lth spherical harmonic coefficient, and k is the number of Fourier coefficients; is the ith Fourier coefficient, is the ith Fourier basis function, is the frame number, .

4. The method of claim 1, wherein, The gradient contribution score of the jth three-dimensional Gaussian is denoted as : ， wherein is the 2D projection value of the jth three-dimensional Gaussian, denotes rendering a two-dimensional image, denotes rendering a two-dimensional image pair of partial derivatives.

5. The method of claim 4, wherein, representing the static semantic score as representing the hybrid importance score as : ， In the formula is a static semantic score hyperparameter.

6. The method of claim 5, wherein, The adaptive pruning based on semantic guidance comprises: During training densification, for a selected number of iterations, the mixed importance score is used to Sort all current 3D Gaussians, prune 60% of the sorted 3D Gaussians; prune 30% of the current sorted 3D Gaussians at a set frequency during fine-tuning.

7. The method of claim 6, wherein, The dropping rate of the jth three-dimensional Gaussian is denoted as : ， wherein is the current iteration round, is a hyperparameter related to the drop rate, is the static semantic score of the jth three-dimensional Gaussian, is a drop rate adjustment hyperparameter, is the total iteration round.

8. The method of claim 7, wherein, The compensation factor to compensate for the opacity of the remaining jth three-dimensional Gaussian is represented as : 。 9. The method of claim 8, wherein, After training, the static semantic score in the optimized three-dimensional Gaussian full-scene synthesis model is synthesized three-dimensional Gaussian configuration high-fidelity label LOD0 greater than 0.5, static semantic score three-dimensional Gaussian configuration low-fidelity label LOD1 less than or equal to 0.

5.

10. The method of claim 1, wherein, Loss function used in the training process of the three-dimensional Gaussian full-scene synthesis model is: ， wherein is a color reconstruction loss for rendering a two-dimensional image with a two-dimensional image of a target scene, is a depth supervision loss, is an entropy loss for foreground opacity, is a depth supervision loss weight, is an entropy loss weight.

Images