A 3D gaussian splash based driving scene feedforward reconstruction method

Abstract

The application relates to a driving scene feedforward reconstruction method based on 3D Gaussian spray, which comprises the following steps: learning scale-aware depth in a self-supervised manner through a deep network, a pose network and a Gaussian network by using sparse visual angles of a vehicle-mounted surround-view camera, positioning the position of a 3D Gaussian, independently predicting the attributes of the 3D Gaussian from each visual angle, aggregating the 3D Gaussians of all visual angles to obtain a reconstructed scene; camera external parameters are not required during training, and the surround-view picture and the pose thereof are given during testing, and real-time reconstruction is realized through feedforward inference. Compared with the prior art, the application solves the problems that the driving scene has sparse visual angles, the overlapping area between the visual angles is small, and the geometric relationship between the visual angles is difficult to directly utilize, and the like, realizes generalizable real-time reconstruction on a driving scene dataset, and improves the accuracy of new visual angle synthesis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional scene reconstruction, in particular to a driving scene feedforward reconstruction method based on 3D Gaussian splatting. BACKGROUND

[0002] Three-dimensional scene reconstruction is of great significance in autonomous driving technology. Modern autonomous driving assistance systems rely on vehicle-mounted cameras to capture real-time surrounding panoramic images, which are then used for online mapping, bird's eye view perception, and three-dimensional target detection, among other key downstream tasks. However, the reconstruction of driving scenes faces numerous challenges, such as sparse camera viewpoints, small overlap between adjacent viewpoints, and the real-time requirements of downstream tasks, which collectively constitute the main difficulties in achieving accurate and efficient three-dimensional reconstruction in driving scenes.

[0003] In recent years, although techniques such as neural radiance fields and 3D Gaussian splatting have made significant progress in the field of three-dimensional scene reconstruction, they rely on dense viewpoint input and high computational cost single scene optimization, are mainly used for offline scene optimization, and lack generalizability, making it difficult to meet the real-time needs of driving scenes. Therefore, the practicality of these methods is limited.

[0004] To address the above challenges, current research is focused on developing three-dimensional reconstruction methods that can achieve real-time and good generalization ability from sparse input views. These methods use feedforward reasoning techniques to quickly construct three-dimensional scenes from sparse input views. However, due to the limited number of vehicle-mounted cameras, sparse viewpoints, small overlap between adjacent views, and the difficulty of obtaining camera extrinsic parameters, existing feedforward reconstruction methods perform poorly in driving scenes, limiting their practicality.

[0005] In addition, existing methods typically use fixed input modes, such as input based on binocular cameras or single-frame panoramic images. These methods cannot adapt flexibly to changes in the number of panoramic cameras in driving scenes and the continuous capture of multiple frames of panoramic images during vehicle travel. Therefore, these methods are limited in their ability to handle different numbers and angles of cameras and continuous image sequences, making it difficult to fully utilize the diverse data obtained during real-time driving.

[0006] In summary, developing an online reconstruction technique suitable for sparse viewpoint input conditions in driving scenes and with generalization ability is a key problem in the field of three-dimensional reconstruction that needs to be addressed. SUMMARY

[0007] The present application is to overcome the defects of the prior art and provide a driving scene feedforward reconstruction method based on 3D Gaussian splatting suitable for sparse viewpoint input conditions in driving scenes and with generalization ability.

[0008] The object of the present application can be realized by the following technical solutions:

[0009] A driving scene feedforward reconstruction method based on 3D Gaussian splashing, comprising the following steps:

[0010] A data set construction and preprocessing step: collecting surround view images taken by vehicle-mounted cameras in a driving scene and performing preprocessing to construct a data set;

[0011] A deep network and pose network construction and training step: constructing a deep network and a pose network, obtaining input images from the data set, extracting a depth map and a pose transformation matrix through the deep network and the pose network respectively, thereby constructing a warped image, and performing loss calculation based on the warped image and the input image, thereby performing self-supervised learning on the deep network and the pose network;

[0012] A Gaussian network construction and training step: constructing a Gaussian network, extracting features of the depth map through an encoder of the Gaussian network, and predicting 3D Gaussian attributes using the extracted depth map features and the depth map through a classification head of the Gaussian network, and aggregating 3D Gaussian attributes of all perspectives to obtain new perspective image rendering results, and comparing with corresponding input images for supervised training;

[0013] A feedforward inference step: inputting surround view images of a driving scene to be tested into the trained deep network to extract a depth map; inputting the depth map into the trained Gaussian network to predict 3D Gaussian attributes, and aggregating 3D Gaussian attributes of all perspectives to obtain a reconstruction result of the driving scene.

[0014] Further, the method further comprises: combining the training processes of the deep network, the pose network and the Gaussian network for joint training, specifically comprising:

[0015] Obtaining input images from the data set;

[0016] Inputting the input images into the deep network and the pose network respectively to output a depth map and a pose transformation matrix, thereby constructing a warped image, and performing positioning loss calculation based on the warped image and the input image;

[0017] Inputting the depth map into the Gaussian network, extracting features of the depth map through an encoder of the Gaussian network, and predicting 3D Gaussian attributes using the extracted depth map features and the depth map through a classification head of the Gaussian network;

[0018] Aggregating 3D Gaussian attributes of all perspectives to obtain new perspective image rendering results, and comparing with corresponding input images for rendering loss calculation;

[0019] Constructing an overall loss function by integrating the positioning loss and the rendering loss, thereby adjusting parameters of the deep network, the pose network and the Gaussian network, and performing iterative training.

[0020] Further, the construction process of the warped image is specifically:

[0021] The image at the time step t corresponding to the adjacent camera is recorded, and the warped image is obtained by warping the image of the adjacent camera or the image of the upper and lower frames of the current camera using the corresponding generated depth map and pose transformation matrix.

[0022] Further, the generation expression of the warped image is:

[0023]

[0024] In the formula, is the warped image corresponding to the adjacent camera i at the time step t, I s is the input image, K s and K i are the image I s and the image corresponding camera intrinsic parameters, is the image corresponding to the adjacent camera i at the time step t, and T is the pose transformation matrix, is the depth map.

[0025] Further, the loss calculation process of the depth network and the pose network is specifically:

[0026] According to the source of the input image, it is divided into time, space and space-time three categories; according to the difference of the input image, the corresponding pose transformation matrix is also divided into time, space and space-time three categories;

[0027] The warped image corresponding to the input image of the time, space and space-time three categories is calculated respectively, the corresponding loss function is obtained, and the Gaussian positioning loss is obtained by combining the depth smoothing loss, which is used for self-supervised learning of the depth network and the pose network.

[0028] Further, the calculation expression of the photometric loss is:

[0029]

[0030] In the formula, L reproj is the photometric loss calculation result, SSIM is the structural similarity degree measure, is the input image, is the corresponding warped image;

[0031] The expression of the Gaussian positioning loss is:

[0032] L loc =λ sp L sp +λ tm L tm +λsp-tm L sp-tm +λ smooth L smooth

[0033] In the formula, L loc is the calculation result of the Gaussian localization loss, L sp is the loss function corresponding to the spatial type source input image, L tm is the loss function corresponding to the time type source input image, L sp-tm is the loss function corresponding to the space-time type source input image, L smooth is the depth smoothing loss, λ sp , λ tm , λ sp-tm , λ smooth is a hyperparameter.

[0034] Further, the 3D Gaussian attribute includes transparency, a covariance matrix composed of a scaling factor and a rotation quaternion, and color represented by a spherical harmonic function.

[0035] Further, in the training process of the Gaussian network, the generated new view image rendering result is compared with the corresponding input image, the LPIPS similarity and the L2 distance are calculated, so as to obtain a rendering loss;

[0036] The calculation expression of the rendering loss is:

[0037] L render = L L2 + 0.05L LPIPS

[0038] In the formula, L render is the calculation result of the rendering loss, L L2 is the calculation result of the L2 distance, and L LPIPS is the calculation result of the LPIPS similarity.

[0039] Further, the pre-processing process includes image enhancement on the collected surround view images, and the image enhancement method includes random transformation of exposure, saturation and hue of the image.

[0040] Further, the depth network and the pose network both use a ResNet-18 network to extract features of the input image, the depth network outputs a depth map, and the pose network outputs a pose transformation matrix from the current frame view to the upper and lower frame views.

[0041] Compared with the prior art, the present application has the following advantages:

[0042] (1) The application utilizes the sparse view angle of the vehicle-mounted surround view camera, learns the scale-aware depth in a self-supervised manner through a depth network, a pose network and a Gaussian network, locates the position of the 3D Gaussian, independently predicts the attributes of the 3D Gaussian from each view angle, aggregates the 3D Gaussian of all view angles to obtain the reconstructed scene, and does not need camera external parameters during training and gives the surround view picture and its pose during testing, thereby realizing real-time reconstruction through feedforward inference. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is a depth network and pose network training process schematic diagram provided in the embodiment of the application;

[0044] Figure 2 It is a Gaussian network training process schematic diagram provided in the embodiment of the application;

[0045] Figure 3 It is a joint training process schematic diagram provided in the embodiment of the application;

[0046] Figure 4 It is an inference process schematic diagram provided in the embodiment of the application;

[0047] Figure 5 It is a whole process schematic diagram of the driving scene feedforward reconstruction method based on 3D Gaussian splash provided in the embodiment of the application;

[0048] Figure 6 It is a benchmark method comparison experiment result schematic diagram provided in the embodiment of the application;

[0049] Figure 7 It is a scene-by-scene reconstruction method comparison experiment result schematic diagram provided in the embodiment of the application;

[0050] Figure 8 It is a reconstruction effect schematic diagram provided in the embodiment of the application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. The components of the embodiments of the application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0052] The following detailed description of embodiments of the application in the drawings provided is not intended to limit the scope of the application as claimed, but merely represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0053] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0054] Embodiment 1

[0055] The embodiment provides a driving scene feedforward reconstruction method based on 3D Gaussian splash, comprising the following steps:

[0056] Data set construction and preprocessing step: collect the surround view images taken by the vehicle-mounted camera in the driving scene, and perform preprocessing to construct the data set;

[0057] Deep network and pose network construction and training step: constructing a deep network and a pose network, obtaining input images from the data set, extracting a depth map and a pose transformation matrix through the deep network and the pose network respectively, thereby constructing a warped image, calculating a loss based on the warped image and the input image, thereby performing self-supervised learning on the deep network and the pose network;

[0058] Gaussian network construction and training step: constructing a Gaussian network, extracting features of the depth map through an encoder of the Gaussian network, predicting 3D Gaussian attributes through a classification head of the Gaussian network using the extracted depth map features and the depth map, and aggregating 3D Gaussian attributes of all perspectives to obtain a new perspective image rendering result, and comparing with the corresponding input image for supervised training;

[0059] Feedforward inference step: inputting the surround view images of the driving scene to be tested into the trained deep network to extract a depth map; inputting the depth map into the trained Gaussian network to predict 3D Gaussian attributes, and aggregating 3D Gaussian attributes of all perspectives to obtain a reconstruction result of the driving scene.

[0060] Preferably, the method further comprises: combining the training processes of the depth network, the pose network and the Gaussian network for joint training, specifically comprising:

[0061] Obtaining input images from the data set;

[0062] Inputting the input images into the deep network and the pose network respectively to output a depth map and a pose transformation matrix, thereby constructing a warped image, and calculating a localization loss based on the warped image and the input image;

[0063] The depth map is input into a Gaussian network, the encoder of the Gaussian network extracts the features of the depth map, and the classification head of the Gaussian network uses the extracted depth map features and the depth map to predict 3D Gaussian properties.

[0064] Aggregate the 3D Gaussian properties of all viewpoints to obtain the rendering result of the new viewpoint image, and compare it with the corresponding input image to calculate the rendering loss;

[0065] A comprehensive loss function is constructed by combining localization loss and rendering loss, thereby adjusting the parameters of the deep network, pose network, and Gaussian network, and then iteratively training is performed.

[0066] This approach can train deep networks, pose networks, and Gaussian networks separately or jointly; there is no single limitation.

[0067] See Figure 1 This embodiment first trains a deep network and a pose network using a large number of continuous surround view image frames of driving scenes. The aim is to learn scale information from images from adjacent camera views and the overlap of images in consecutive frames, to locate 3D Gaussian positions, and to provide input for subsequent networks. Specifically, it includes the following steps:

[0068] A0: Data augmentation is used to perform a series of transformations on the input image to enhance the generalization of the model's feature representation; specifically, color dithering is used to randomly transform the image's exposure, saturation, and hue HSV.

[0069] A1: Deep networks and pose networks use ResNet-18 to extract the input image. Based on the characteristics of the image, the deep network outputs a depth map of the current image. The pose network outputs the pose transformation matrix from the current frame's viewpoint to the viewpoints of the previous and next frames. This avoids dependence on camera extrinsic parameters during training.

[0070] A2: Let C be the adjacent camera. i C j and its corresponding time step t image Using the depth map and pose transformation matrix output from A1, the images from adjacent cameras or the images from the current camera's previous and next frames are warped, as shown in the following formula:

[0071]

[0072] Among them, K s K i Image I s , The corresponding camera intrinsic parameters, It is a distorted image, I sAccording to its source, it can be divided into time, space, space-time three categories:

[0073]

[0074] T is I s to the transformation matrix, according to I s different, also can be divided into time, space, space-time three categories:

[0075]

[0076] A3: using the image from time, space, space-time distortion and the current true value image, through the photometric loss for self-supervised training, make the deep network learn the depth of scale perception, so as to ensure the multi-view consistency, make the 3D Gaussian positioning accurate. The photometric loss formula is as follows:

[0077]

[0078] Where, SSIM is the structural similarity measure, ||.|| is the L1 distance, according to the distorted image Source image I s different, the loss function composed of space, time, space-time source image is respectively represented as L sp , L tm , L sp-tm , combined with the depth smoothness loss L smoot h, get the Gaussian positioning loss:

[0079] L loc =λ sp L sp +λ tm L tm +λ sp-tm L sp-tm +λ smooth L smooth (5)

[0080] For deep network and pose network self-supervised training, wherein, λ Sp , λ tm , λ sp-tm , λ smooth is a hyperparameter.

[0081] Reference Figure 2 , the embodiment uses the depth map output by the depth network and the features extracted by the encoder to input the Gaussian network to predict the corresponding 3D Gaussian attribute of each image, aggregate the 3D Gaussian of all view images, get the reconstructed scene, render new view for training; Specifically, the following steps are included:

[0082] B0: The encoder of the Gaussian network extracts the features of the depth map, aiming to make full use of the scale information of the depth map when predicting the 3D Gaussian attributes, and improve the accuracy of the prediction;

[0083] B1: The classification head of the Gaussian network uses the extracted depth map features and the image features output by the depth network encoder to predict the attributes of each 3D Gaussian, including the covariance matrix composed of the scaling factor and the rotation quaternion, the transparency, and the color represented by the spherical harmonic function;

[0084] B2: Assign a 3D Gaussian to each pixel of each image, project each 3D Gaussian to the three-dimensional space using the depth map, and aggregate the 3D Gaussians projected by all view images to obtain a reconstructed scene represented by 3D Gaussians;

[0085] B3: Render new view images from the reconstructed scene, use the combination of the learned perceptual image block similarity LPIPS and L2 distance as the rendering loss, and supervise the Gaussian network with the ground truth images of the new view to train the Gaussian network:

[0086] L render =L L2 +0.05L LPIPS (6)

[0087] Referring to Figure 3 , the depth map output by the depth network and the image features extracted by its encoder serve as a link, and the depth network, the pose network, and the Gaussian network can be jointly trained; specifically including the following steps:

[0088] C0: The depth map output by the depth network and the image features extracted by its encoder jointly train the depth network, the pose network, and the Gaussian network, so that the three networks can be end-to-end jointly trained, and the loss function of the entire network is as follows:

[0089] L=L loc +λ render L render (7)

[0090] where L loc is the Gaussian localization loss, L render is the rendering loss, f render is a hyperparameter.

[0091] Referring to Figure 4 , given the surround view images and their poses, only the depth network and the Gaussian network are needed during inference, and real-time reconstruction can be achieved through feedforward inference; specifically including the following steps:

[0092] D0: Input the surround view images into the depth network, extract the image features and estimate the scale-aware depth map, and use the depth to localize the 3D Gaussian;

[0093] D1: Input the depth map into the Gaussian network and stitch together the image features. The classification head outputs the attributes of the 3D Gaussian network.

[0094] D2: Aggregate the 3D Gaussian from all viewpoint images to obtain the reconstructed scene.

[0095] See Figure 5 This scheme first constructs a deep network, a pose network, and a Gaussian network, and then performs joint training. The deep network and pose network learn scale information in a self-supervised manner to achieve scale-aware 3D Gaussian localization and improve the consistency of 3D Gaussian position across multiple views. The Gaussian network predicts other attributes of the 3D Gaussian. During inference, the deep network and Gaussian network are used to output the position and other attributes of the 3D Gaussian in a feedforward manner, and the 3D Gaussian from all views is aggregated to reconstruct the scene.

[0096] Specifically, in this embodiment, the dataset is first divided using the academic dataset nuScenes, which contains 1000 driving scenarios. The training data includes 700 scenarios, the validation data includes 150 scenarios, and the test data includes 150 scenarios. Each driving scenario consists of 20 seconds of continuous image frames, and all images are captured by six onboard cameras, covering the surround-view perspective. This embodiment uses the standard division of 700 training scenarios and 150 validation scenarios, supported by a single RTX A6000 GPU.

[0097] Then, deep networks, pose networks, and Gaussian networks are constructed.

[0098] Finally, joint training was conducted, and the review... Figure 3 The model takes multiple frames of panoramic images as input. The deep network and pose network learn scale information in a self-supervised manner to estimate depth. The Gaussian network predicts other properties of the 3D Gaussian network. After reconstructing the scene, a new perspective is rendered for optimization. The specific implementation steps are as follows:

[0099] 1) Perform data augmentation on A0;

[0100] 2) The A1 depth network outputs a depth map, and the pose network outputs the pose transformation matrix between the previous and next frames.

[0101] 3), A2 distorts the image, synthesizing the distorted image using the depth map and pose transformation matrix obtained in 2), and A3 uses the photometric loss between the distorted image and the current real image for self-supervised training.

[0102] 4) The B0 Gaussian network takes the depth map as input and extracts depth features. The B1 Gaussian network then stitches the image features extracted by the depth network in 2), and the classification head outputs 3D Gaussian attributes.

[0103] 5), B2 aggregates 3D Gauss of all view images, reconstructs the scene, B3 renders new view images, and optimizes the similarity loss with the real images of the new view.

[0104] The embodiment is tested on nuScenes, refer to Figure 4 , and the model is fed forward to infer, reconstruct the scene, and render a new view during testing. Figure 6 The embodiment outperforms the benchmark method in all three similarity indicators, proving that the embodiment can accurately reconstruct the driving scene. Figure 7 The embodiment does not need to be optimized during testing, realizes real-time inference, and has better reconstruction quality than the benchmark method of scene-by-scene optimization.

[0105] Refer to Figure 8 The embodiment realizes higher quality in new view synthesis, and other benchmark methods have obvious artifacts, while the embodiment significantly reduces artifacts and synthesizes clearer new view images, proving that the embodiment can achieve higher reconstruction quality.

[0106] The above describes preferred embodiments of the embodiment. It should be understood that those skilled in the art can make many modifications and changes without creative labor according to the concept of the embodiment. Therefore, any technical solution obtained by logical analysis, reasoning, or limited experiments based on the existing technology according to the concept of the embodiment shall be within the protection scope defined by the claims.

Claims

1. A 3D Gaussian splash based driving scene feedforward reconstruction method, characterized in that, The method comprises the following steps: A data set construction and preprocessing step: surround view images captured by vehicle-mounted cameras in a driving scene are collected and preprocessed to construct a data set; A deep network and pose network construction and training step: a deep network and a pose network are constructed, input images are obtained from the data set, a depth map and a pose transformation matrix are extracted by the deep network and the pose network respectively, a warped image is constructed, a loss is calculated based on the warped image and the input image, and the deep network and the pose network are self-supervised learned; A Gaussian network construction and training step: a Gaussian network is constructed, features of the depth map are extracted by an encoder of the Gaussian network, 3D Gaussian attributes are predicted by a classification head of the Gaussian network using the extracted depth map features and the depth map, and new view image rendering results are obtained by aggregating 3D Gaussian attributes of all views, and are compared with corresponding input images for supervised training; A feedforward inference step: surround view images of a driving scene to be tested are input into the trained deep network to extract a depth map, the depth map is input into the trained Gaussian network to predict 3D Gaussian attributes, and 3D Gaussian attributes of all views are aggregated to obtain a reconstruction result of the driving scene.

2. The method of claim 1, wherein the method is a 3D Gaussian splash based driving scene feed-forward reconstruction method. The method further comprises: combining the training processes of the deep network, the pose network and the Gaussian network for joint training, specifically comprising: obtaining input images from the data set; inputting the input images into the deep network and the pose network respectively to output a depth map and a pose transformation matrix, thereby constructing a warped image, and calculating a localization loss based on the warped image and the input image; inputting the depth map into the Gaussian network to extract features of the depth map by an encoder of the Gaussian network, predicting 3D Gaussian attributes by a classification head of the Gaussian network using the extracted depth map features and the depth map; aggregating 3D Gaussian attributes of all views to obtain new view image rendering results, and comparing the new view image rendering results with corresponding input images to calculate a rendering loss; constructing an overall loss function by combining the localization loss and the rendering loss, thereby adjusting parameters of the deep network, the pose network and the Gaussian network, and performing iterative training.

3. The method of claim 1, wherein the method is a 3D Gaussian splash based driving scene feed-forward reconstruction method. The construction process of the warped image is specifically: record the images at time step t corresponding to adjacent camera machines, use the corresponding generated depth map and pose transformation matrix to warp the images of adjacent cameras or the images of the current camera upper and lower frames to obtain a warped image.

4. The driving scene feed-forward reconstruction method based on 3D Gaussian splash of claim 3, wherein, The generation expression of the warped image is: wherein, is the warped image corresponding to the adjacent camera i at time t, I s is the input image, K s and K i are the image I s and image is the intrinsic parameter of the corresponding camera, is the image corresponding to the adjacent camera i at time t, T is the pose transformation matrix, is the depth map.

5. The method of claim 1, wherein, The loss calculation process of the deep network and the pose network is specifically: According to the source of the input image, it is divided into three categories of time, space and space; according to the difference of the input image, the corresponding pose transformation matrix is also divided into three categories of time, space and space; The warped images corresponding to the input images of the three categories of time, space and space are calculated respectively to obtain the corresponding loss function, and the Gaussian localization loss is obtained by combining the depth smoothing loss, which is used for self-supervised learning of the deep network and the pose network.

6. The 3D Gaussian splash based driving scene feed-forward reconstruction method of claim 5, wherein, The calculation expression of the photometric loss is: wherein L reproj is the luminance loss calculation result, SSIM is the structural similarity metric, is the input image, is the corresponding warped image; The expression of the Gaussian localization loss is: L loc = λ sp L sp + λ tm L tm + λ sp-tm L sp-tm + λ smooth L smooth In the formula, L loc is the calculation result of the Gauss localization loss, L sp is the loss function corresponding to the spatial type source input image, L tm is the loss function corresponding to the time type source input image, L sp-tm is the loss function corresponding to the space-time type source input image, L smooth is the depth smoothing loss, λ sp , λ tm , λ sp-tm , λ smooth is a hyperparameter.

7. The method of claim 1, wherein the method is a 3D Gaussian splash based driving scene feed-forward reconstruction method. The 3D Gaussian attributes include transparency, a covariance matrix composed of a scaling factor and a rotation quaternion, and a color represented by a spherical harmonic function.

8. The method of claim 1, wherein the method is a 3D Gaussian splash based driving scene feed-forward reconstruction method. In the training process of the Gaussian network, the generated new view image rendering result is compared with the corresponding input image, the LPIPS similarity and L2 distance are calculated, and a rendering loss is obtained; The calculation expression of the rendering loss is: L render = L L2 + 0.05L LPIPS In the formula, L render is a rendering loss calculation result, L L2 is an L2 distance calculation result, L LPIPS is an LPIPS similarity calculation result.

9. The method of claim 1, wherein the method is a 3D Gaussian splash based driving scene feed-forward reconstruction method. The pre-processing process includes image enhancement on the collected surround view images, and the image enhancement method includes random transformation on the exposure, saturation and hue of the image.

10. The method of claim 1, wherein the method is a 3D Gaussian splash based driving scene feed-forward reconstruction method. Both the depth network and the pose network use a ResNet-18 network to extract the features of the input image, the depth network outputs a depth map, and the pose network outputs a pose transformation matrix from the current frame view to the upper and lower frame views.

Images