Driving scene feedforward reconstruction method based on 3D Gaussian splashing

Abstract

The invention relates to a driving scene feedforward reconstruction method based on 3D Gaussian splashing, and the method comprises the steps: learning the depth perceived by a scale in a self-supervision manner through a deep network, an attitude network and a Gaussian network by employing the sparse view angle of a vehicle-mounted all-round view camera, positioning the position of 3D Gaussian, and independently predicting the attribute of the 3D Gaussian from each view angle, aggregating the 3D gausses of all the visual angles to obtain a reconstructed scene; during training, external parameters of a camera are not needed, and a look-around picture and a pose thereof are given during testing and are reconstructed in real time through feedforward reasoning. Compared with the prior art, the problems that the visual angles of the driving scene are sparse, the overlapping area between the visual angles is small, and the geometrical relation between the visual angles is difficult to directly use are solved, generalization real-time reconstruction is achieved on the driving scene data set, and the new visual angle synthesis precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scene reconstruction, and in particular to a driving scene feedforward reconstruction method based on 3D Gaussian scattering. Background Art

[0002] Three-dimensional scene reconstruction is of great significance in autonomous driving technology. Modern autonomous driving assistance systems rely on on-board cameras to capture surround images in real time, which are then used for key downstream tasks such as online mapping, bird's-eye view perception, and three-dimensional object detection. However, the reconstruction of driving scenes faces many challenges, such as sparse camera perspectives, small overlap between adjacent perspectives, and the real-time requirements of downstream tasks. These factors together constitute the main difficulties in achieving accurate and efficient three-dimensional reconstruction in driving scenes.

[0003] In recent years, although technologies such as neural radiation field and 3D Gaussian sputtering have made significant progress in the field of 3D scene reconstruction, they rely on dense view input and single scene optimization with high computational cost, are mainly used for offline scene optimization, and are not generalizable and difficult to meet the real-time requirements of driving scenes. Therefore, the practicality of these methods is limited.

[0004] To address the above challenges, current research is working on developing real-time 3D reconstruction methods with good generalization capabilities from sparse input views. These methods use feedforward reasoning techniques to quickly construct 3D scenes under the condition of sparse input views. However, due to the limited number of on-board cameras, sparse viewing angles, small overlap between adjacent views, and difficulty in obtaining camera extrinsic parameters, existing feedforward reconstruction methods perform poorly in driving scenarios, limiting their practicality.

[0005] In addition, existing methods usually adopt fixed input modes, such as binocular camera-based input or single-frame surround view images as input. These methods cannot flexibly adapt to the changes in the number of surround view cameras in the driving scene and the multiple frames of surround view images captured continuously during vehicle driving. Therefore, these methods are limited in their ability to handle cameras with different numbers and perspectives and continuous image sequences, making it difficult to fully utilize the diverse data obtained during real-time driving.

[0006] In summary, developing an online reconstruction technology that is suitable for the input conditions of sparse perspectives in driving scenes and has generalization capabilities is a key issue that needs to be urgently addressed in the field of 3D reconstruction. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a driving scene feedforward reconstruction method based on 3D Gaussian sparse input conditions that is suitable for driving scene sparse perspectives and has generalization capability.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A driving scene feedforward reconstruction method based on 3D Gaussian scattering includes the following steps:

[0010] Dataset construction and preprocessing steps: collect surround view images taken by on-board cameras in driving scenarios, perform preprocessing, and construct a dataset;

[0011] Steps for building and training the deep network and the posture network: Building the deep network and the posture network, obtaining the input image from the dataset, extracting the depth map and the posture transformation matrix through the deep network and the posture network respectively, thereby building the distorted image, and calculating the loss based on the distorted image and the input image, thereby performing self-supervised learning on the deep network and the posture network;

[0012] Gaussian network construction and training steps: construct a Gaussian network, extract the features of the depth map through the encoder of the Gaussian network, use the extracted depth map features and the depth map through the classification head of the Gaussian network to predict the 3D Gaussian properties, and aggregate the 3D Gaussian properties of all perspectives to obtain the rendering results of the new perspective image, and compare it with the corresponding input image for supervised training;

[0013] Feedforward reasoning step: Input the surround view image of the driving scene to be tested into the trained deep network to extract the depth map; input the depth map into the trained Gaussian network to predict the 3D Gaussian attributes, and aggregate the 3D Gaussian attributes of all perspectives to obtain the reconstruction result of the driving scene.

[0014] Furthermore, the method further includes: combining the training processes of the deep network, the posture network and the Gaussian network for joint training, specifically including:

[0015] Get the input image from the dataset;

[0016] Input the input image into the depth network and the posture network respectively, output the depth map and the posture transformation matrix, so as to construct the distorted image, and calculate the positioning loss based on the distorted image and the input image;

[0017] The depth map is input into the Gaussian network, the features of the depth map are extracted through the encoder of the Gaussian network, and the 3D Gaussian attributes are predicted by the classification head of the Gaussian network using the extracted depth map features and the depth map;

[0018] Aggregate the 3D Gaussian properties of all perspectives to obtain the rendering results of the new perspective image, and compare it with the corresponding input image to calculate the rendering loss;

[0019] The overall loss function is constructed by combining the positioning loss and rendering loss to adjust the parameters of the deep network, posture network and Gaussian network, and perform iterative training.

[0020] Furthermore, the construction process of the distorted image is specifically as follows:

[0021] Record the images of the adjacent camera machines at the corresponding time step t, and use the corresponding generated depth map and pose transformation matrix to distort the adjacent camera images or the images of the upper and lower frames of the current camera to obtain a distorted image.

[0022] Furthermore, the generation expression of the distorted image is:

[0023]

[0024] In the formula, is the distorted image corresponding to the adjacent camera i at time t, I s is the input image, K s and K i I s and images The corresponding camera intrinsic parameters, is the image corresponding to the adjacent camera i at time t, T is the pose transformation matrix, is the depth map.

[0025] Furthermore, the loss calculation process of the deep network and the posture network is specifically as follows:

[0026] According to the source of the input image, it is divided into three categories: time, space and time-space; according to the different input images, the corresponding posture transformation matrix is ​​also divided into three categories: time, space and time-space;

[0027] The photometric loss is calculated for the distorted images corresponding to the three types of input images: time, space, and spatiotemporal, and the corresponding loss function is obtained. Combined with the deep smoothing loss, the Gaussian positioning loss is obtained for self-supervised learning of deep networks and posture networks.

[0028] Furthermore, the calculation expression of the luminosity loss is:

[0029]

[0030] Where, L reproj is the calculation result of photometric loss, SSIM is the structural similarity metric, is the input image, is the corresponding distorted image;

[0031] The expression of the Gaussian positioning loss is:

[0032] L loc =λ sp L sp +λ tm L tm +λsp-tm L sp-tm +λ smooth L smooth

[0033] Where, L loc is the calculation result of Gaussian positioning loss, L sp is the loss function corresponding to the spatial type source input image, L tm is the loss function corresponding to the time type source input image, L sp-tm is the loss function corresponding to the input image of the spatiotemporal source, L smooth is the depth smoothing loss, λ sp , tm , sp-tm , smooth is a hyperparameter.

[0034] Furthermore, the 3D Gaussian attributes include transparency, a covariance matrix consisting of a scaling factor and a rotation quaternion, and a color represented by a spherical harmonic function.

[0035] Furthermore, during the training process of the Gaussian network, the generated rendering result of the new perspective image is compared with the corresponding input image, and the LPIPS similarity and L2 distance are calculated to obtain the rendering loss;

[0036] The calculation expression of the rendering loss is:

[0037] L render =L L2 +0.05L LPIPS

[0038] Where, L render is the result of rendering loss calculation, L L2 is the L2 distance calculation result, L LPIPS It is the result of LPIPS similarity calculation.

[0039] Furthermore, the preprocessing process includes: performing image enhancement on the collected surround view images, and the image enhancement method includes randomly transforming the exposure, saturation and hue of the image.

[0040] Furthermore, the deep network and the posture network both use the ResNet-18 network to extract features of the input image, the deep network outputs a depth map, and the posture network outputs a posture transformation matrix from the current frame perspective to the upper and lower frame perspectives.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) The present invention utilizes the sparse perspectives of the on-board surround camera, and learns the scale-aware depth in a self-supervised manner through a deep network, a posture network, and a Gaussian network, locates the position of the 3D Gaussian, and independently predicts the properties of the 3D Gaussian from each perspective, aggregates the 3D Gaussians of all perspectives, and obtains a reconstructed scene; the present invention does not require camera external parameters during training, and during testing, a surround image and its posture are given, and real-time reconstruction is performed through feedforward reasoning. Compared with the existing related technologies, the present invention solves the problems of sparse driving scene perspectives, small overlapping areas between perspectives, and difficulty in directly using the geometric relationship between perspectives. It realizes generalizable real-time reconstruction on driving scene datasets and improves the accuracy of new perspective synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a deep network and posture network training process provided in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a Gaussian network training process provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a joint training process provided in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a reasoning process provided in an embodiment of the present invention;

[0047] Figure 5 It is a schematic diagram of the overall process of a driving scene feedforward reconstruction method based on 3D Gaussian scattering provided in an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of comparative experimental results of a benchmark method provided in an embodiment of the present invention;

[0049] Figure 7 A schematic diagram of comparative experimental results of a scene-by-scene reconstruction method provided in an embodiment of the present invention;

[0050] Figure 8 A schematic diagram of a reconstruction effect provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] Example 1

[0055] This embodiment provides a driving scene feedforward reconstruction method based on 3D Gaussian scattering, comprising the following steps:

[0056] Dataset construction and preprocessing steps: collect surround view images taken by on-board cameras in driving scenarios, perform preprocessing, and construct a dataset;

[0057] Steps for building and training the deep network and the posture network: Building the deep network and the posture network, obtaining the input image from the dataset, extracting the depth map and the posture transformation matrix through the deep network and the posture network respectively, thereby building the distorted image, and calculating the loss based on the distorted image and the input image, thereby performing self-supervised learning on the deep network and the posture network;

[0058] Gaussian network construction and training steps: construct a Gaussian network, extract the features of the depth map through the encoder of the Gaussian network, use the extracted depth map features and the depth map through the classification head of the Gaussian network to predict the 3D Gaussian properties, and aggregate the 3D Gaussian properties of all perspectives to obtain the rendering results of the new perspective image, and compare it with the corresponding input image for supervised training;

[0059] Feedforward reasoning step: Input the surround view image of the driving scene to be tested into the trained deep network to extract the depth map; input the depth map into the trained Gaussian network to predict the 3D Gaussian attributes, and aggregate the 3D Gaussian attributes of all perspectives to obtain the reconstruction result of the driving scene.

[0060] Preferably, the method further comprises: combining the training processes of the deep network, the posture network and the Gaussian network for joint training, specifically comprising:

[0061] Get the input image from the dataset;

[0062] Input the input image into the depth network and the posture network respectively, output the depth map and the posture transformation matrix, so as to construct the distorted image, and calculate the positioning loss based on the distorted image and the input image;

[0063] The depth map is input into the Gaussian network, the features of the depth map are extracted through the encoder of the Gaussian network, and the 3D Gaussian attributes are predicted by the classification head of the Gaussian network using the extracted depth map features and the depth map;

[0064] Aggregate the 3D Gaussian properties of all perspectives to obtain the rendering results of the new perspective image, and compare it with the corresponding input image to calculate the rendering loss;

[0065] The overall loss function is constructed by combining the positioning loss and rendering loss to adjust the parameters of the deep network, posture network and Gaussian network, and perform iterative training.

[0066] This solution can train the deep network, posture network, and Gaussian network separately or jointly, without any limitation.

[0067] See also Figure 1 ,This embodiment first uses a large number of continuous surround view image frames of driving scenes to train the deep network and the posture network, aiming to learn scale information from the images of adjacent camera perspectives and the overlap of upper and lower frame images, locate the 3D Gaussian position, and provide input for the subsequent network, specifically including the following steps:

[0068] A0: Use data augmentation to perform a series of transformations on the input image to enhance the generalization of the model's feature representation; specifically, use color jitter to randomly transform the image's exposure, saturation, and hue HSV;

[0069] A1: Deep network and posture network use ResNet-18 to extract input image The deep network outputs the depth map of the current image. The posture network outputs the posture transformation matrix from the current frame perspective to the upper and lower frame perspectives This avoids the reliance on camera external parameters during training.

[0070] A2: Record the adjacent camera C i , C j and its corresponding image at time step t Using the depth map and pose transformation matrix output in A1, the adjacent camera images or the images of the upper and lower frames of the current camera are distorted. The formula is as follows:

[0071]

[0072] Among them, K s , K i is image I s , The corresponding camera intrinsic parameters, is the distorted image, I sAccording to their sources, they can be divided into three categories: time, space, and space-time:

[0073]

[0074] T = I s arrive The transformation matrix, according to I s Different from each other, it can also be divided into three categories: time, space, and space-time:

[0075]

[0076] A3: Using images distorted from time, space, and time and space with the current true value image, self-supervised training is performed through photometric loss, so that the deep network learns the depth of scale perception, thereby ensuring multi-view consistency and making 3D Gaussian positioning accurate. The photometric loss formula is as follows:

[0077]

[0078] Among them, SSIM is the structural similarity metric, ∥·∥ is the L1 distance, and the Source Image I s Different from the original image in space, time, and spatiotemporal, the loss functions of the original image in space, time, and spatiotemporal are expressed as L sp , L tm , L sp-tm , combined with the deep smoothing loss L smoot h, and get the Gaussian positioning loss:

[0079] L loc =λ sp L sp +λ tm L tm +λ sp-tm L sp-tm +λ smooth L smooth (5)

[0080] For self-supervised training of deep networks and posture networks, where λ Sp ,λ tm ,λ sp-tm ,λ smooth is a hyperparameter.

[0081] See also Figure 2 In this embodiment, the depth map output by the deep network and the features extracted by its encoder are input into the Gaussian network to predict the 3D Gaussian attributes corresponding to each image, the 3D Gaussian of all view images are aggregated, the reconstructed scene is obtained, and the new view is rendered for training; specifically, the following steps are included:

[0082] B0: The encoder of the Gaussian network extracts the features of the depth map, aiming to make full use of the scale information of the depth map when predicting the 3D Gaussian attributes and improve the accuracy of the prediction;

[0083] B1: The classification head of the Gaussian network uses the extracted depth map features and the image features output by the deep network encoder to predict the properties of each 3D Gaussian, including the covariance matrix composed of scaling factors and rotation quaternions, transparency, and color represented by spherical harmonics;

[0084] B2: Assign a 3D Gaussian to each pixel of each image, use the depth map to project each 3D Gaussian into three-dimensional space, aggregate the 3D Gaussians projected from all viewpoint images, and obtain the reconstructed scene represented by the 3D Gaussian;

[0085] B3: Render the new view image from the reconstructed scene, use the combination of learning-perceptual image block similarity LPIPS and L2 distance as the rendering loss, supervise with the real value image of the new view, and train the Gaussian network:

[0086] L render =L L2 +0.05L LPIPS (6)

[0087] See also Figure 3 , the depth map output by the deep network and the image features extracted by its encoder are used as a link. The deep network, posture network and Gaussian network can be trained jointly; specifically, the following steps are included:

[0088] C0: The depth map output by the deep network and the image features extracted by its encoder are combined with the deep network, posture network and Gaussian network so that the three networks can be trained end-to-end. The loss function of the entire network is as follows:

[0089] L=L loc +λ render L render (7)

[0090] Where L loc is the Gaussian localization loss, L render is the rendering loss, f render is a hyperparameter.

[0091] See also Figure 4 , given a surround image and its pose, only a deep network and a Gaussian network are needed for reasoning, and real-time reconstruction can be achieved through feedforward reasoning; specifically, the following steps are included:

[0092] D0: Input the surround image into the deep network, extract image features and estimate the scale-aware depth map, and use the depth to locate the position of the 3D Gaussian;

[0093] D1: Input the depth map into the Gaussian network and concatenate the image features, and the classification head outputs the attributes of the 3D Gaussian.

[0094] D2: Aggregate the 3D Gaussians of all viewpoint images to obtain the reconstructed scene.

[0095] See also Figure 5 ,This scheme first builds a deep network, a pose network and a Gaussian network, and then performs joint training. The deep network and the pose network learn the scale information in a self-supervised way, ,to achieve scale-aware 3D Gaussian positioning and improve the consistency of the ,position of multi-view 3D Gaussian, and the Gaussian network predicts other attributes of the 3D Gaussian; during reasoning, the deep network and Gaussian network are ,used to output the position and other attributes of the 3D Gaussian in a feedforward manner, and the 3D Gaussian of all ,views

[0096] Specifically, in this embodiment, the data set is first divided using the academic data set nuScenes, which contains 1,000 driving scenes, of which the training data includes 700 scenes, the verification data includes 150 scenes, and the test data includes 150 scenes. Each driving scene consists of 20s of continuous image frames, and all images are captured by 6 on-board cameras, covering the surround view. This embodiment uses 700 scene training data and 150 scene verification data divided by standard, and uses an RTXA6000 as hardware support;

[0097] Then build the deep network, posture network and Gaussian network;

[0098] Finally, joint training was conducted. Figure 3 The model inputs multiple frames of surround images, the deep network and the posture network learn scale information in a self-supervised manner, estimate the depth, and the Gaussian network predicts other properties of the 3D Gaussian. After reconstructing the scene, the new perspective is rendered for optimization. The specific implementation steps are as follows:

[0099] 1), A0 performs data enhancement;

[0100] 2), the A1 depth network outputs a depth map, and the posture network outputs the pose transformation matrix of the upper and lower frames;

[0101] 3), A2 distorts the image and uses the depth map and pose transformation matrix obtained in 2) to synthesize the distorted image. A3 uses the photometric loss of the distorted image and the current real image for self-supervised training;

[0102] 4), B0 Gaussian network inputs the depth map to extract the depth features, B1 Gaussian network splices the image features extracted by the deep network in 2), and the classification head outputs 3D Gaussian attributes;

[0103] 5), B2 aggregates the 3D Gaussians of all viewpoint images to reconstruct the scene, and B3 renders the new viewpoint image and calculates the similarity loss with the real image of the new viewpoint for optimization.

[0104] This example is tested in nuScenes, see Figure 4 During the test, a single frame or multiple frames of surround images are input, the model feeds forward inference, reconstructs the scene and renders the new perspective. The effectiveness and real-time performance of the present invention are demonstrated by comparing the similarity between the rendered new perspective image and the real new perspective image with the benchmark method (PSNR, SSIM, LPIPS are similarity indicators) and comparing the inference time with the scene-by-scene optimization method. The comparison results with the benchmark method are shown in Figure 6 The present invention outperforms the benchmark method in all three similarity indices, proving that the present invention can accurately reconstruct driving scenes. The results are compared with the scene-by-scene optimization method. Figure 7 ,The present method does not require test-time optimization, achieves real-time reasoning, and the reconstruction quality is better than the baseline method of ,scene-by-scene optimization.

[0105] See also Figure 8 The present invention achieves higher quality in new perspective synthesis. Other benchmark methods have obvious artifacts, while the present invention significantly reduces artifacts and synthesizes clearer new perspective images, proving that the present invention can achieve higher reconstruction quality.

[0106] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A driving scene feedforward reconstruction method based on 3D Gaussian scattering, characterized in that: The following steps are involved: Dataset construction and preprocessing steps: collect surround view images taken by on-board cameras in driving scenarios, perform preprocessing, and construct a dataset; Steps for building and training the deep network and the posture network: Building the deep network and the posture network, obtaining the input image from the dataset, extracting the depth map and the posture transformation matrix through the deep network and the posture network respectively, thereby building the distorted image, and calculating the loss based on the distorted image and the input image, thereby performing self-supervised learning on the deep network and the posture network; Gaussian network construction and training steps: construct a Gaussian network, extract the features of the depth map through the encoder of the Gaussian network, use the extracted depth map features and the depth map through the classification head of the Gaussian network to predict the 3D Gaussian properties, and aggregate the 3D Gaussian properties of all perspectives to obtain the rendering results of the new perspective image, and compare it with the corresponding input image for supervised training; Feedforward reasoning step: Input the surround view image of the driving scene to be tested into the trained deep network to extract the depth map; input the depth map into the trained Gaussian network to predict the 3D Gaussian attributes, and aggregate the 3D Gaussian attributes of all perspectives to obtain the reconstruction result of the driving scene.

2. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: The method further includes: combining the training processes of the deep network, the posture network and the Gaussian network for joint training, specifically including: Get the input image from the dataset; Input the input image into the depth network and the posture network respectively, output the depth map and the posture transformation matrix, so as to construct the distorted image, and calculate the positioning loss based on the distorted image and the input image; The depth map is input into the Gaussian network, the features of the depth map are extracted through the encoder of the Gaussian network, and the 3D Gaussian attributes are predicted by the classification head of the Gaussian network using the extracted depth map features and the depth map; Aggregate the 3D Gaussian properties of all perspectives to obtain the rendering results of the new perspective image, and compare it with the corresponding input image to calculate the rendering loss; The overall loss function is constructed by combining the positioning loss and rendering loss to adjust the parameters of the deep network, posture network and Gaussian network, and perform iterative training.

3. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: The construction process of the distorted image is specifically as follows: Record the images of the adjacent camera machines at the corresponding time step t, and use the corresponding generated depth map and pose transformation matrix to distort the adjacent camera images or the images of the upper and lower frames of the current camera to obtain a distorted image.

4. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 3, characterized in that: The generation expression of the distorted image is: In the formula, is the distorted image corresponding to the adjacent camera i at time t, I s is the input image, K s and K i I s and images The corresponding camera intrinsic parameters, is the image corresponding to the adjacent camera i at time t, T is the pose transformation matrix, is the depth map.

5. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: The loss calculation process of the deep network and the posture network is specifically as follows: According to the source of the input image, it is divided into three categories: time, space and space-time; according to the different input images, the corresponding posture transformation matrix is ​​also divided into three categories: time, space and space-time; The photometric loss is calculated for the distorted images corresponding to the three types of input images: time, space, and spatiotemporal, and the corresponding loss function is obtained. Combined with the deep smoothing loss, the Gaussian positioning loss is obtained for self-supervised learning of deep networks and posture networks.

6. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 5, characterized in that: The calculation expression of the luminosity loss is: Where, L reproj is the calculation result of photometric loss, SSIM is the structural similarity metric, is the input image, is the corresponding distorted image; The expression of the Gaussian positioning loss is: L loc =λ sp L sp +λ tm L tm +λ sp-tm L sp-tm +λ smooth L smooth Where, L loc is the calculation result of Gaussian positioning loss, L sp is the loss function corresponding to the spatial type source input image, L tm is the loss function corresponding to the time type source input image, L sp-tm is the loss function corresponding to the input image of the spatiotemporal source, L smooth is the depth smoothing loss, λ sp , tm , sp-tm , smooth is a hyperparameter.

7. The driving scene feed-forward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: The 3D Gaussian properties include transparency, a covariance matrix consisting of a scaling factor and rotation quaternions, and color represented by spherical harmonics.

8. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: During the training process of the Gaussian network, the generated new perspective image rendering result is compared with the corresponding input image, and the LPIPS similarity and L2 distance are calculated to obtain the rendering loss; The calculation expression of the rendering loss is: L render =L L2 +0.05L LPIPS Where, L render is the result of rendering loss calculation, L L2 is the L2 distance calculation result, L LPIPS It is the result of LPIPS similarity calculation.

9. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: The preprocessing process includes: performing image enhancement on the collected surround view images, and the image enhancement method includes randomly transforming the exposure, saturation and hue of the image.

10. The driving scene feedforward reconstruction method based on 3D Gaussian spurious according to claim 1, characterized in that: The deep network and the posture network both use the ResNet-18 network to extract features of the input image, the deep network outputs a depth map, and the posture network outputs a posture transformation matrix from the current frame perspective to the upper and lower frame perspectives.

Images