3D Gaussian sputtering monocular vision SLAM method combined with neural network

Abstract

The invention provides a 3D Gaussian sputtering monocular vision SLAM method combined with a neural network, and the method comprises the steps: carrying out the monocular depth estimation of a screened key frame through employing the neural network, and enabling a depth result to be used for the optimization of a 3D Gaussian primitive; meanwhile, the global significance score is introduced to measure the importance degree of the primitives in the 3D Gaussian scene, the global significance score and the opacity serve as comprehensive judgment standards of the importance of the 3D Gaussian primitives, rendering details are reserved through comprehensive judgment, and a rendering result with higher quality is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a 3D Gaussian splatting monocular vision SLAM method combined with a neural network. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology is an important research direction in the fields of computer vision and robotics, enabling a robot to gradually build a map of the surrounding environment while determining its own position and orientation in an unknown environment. SLAM was initially applied in the field of robotics and has subsequently been gradually applied to multiple fields such as augmented reality and autonomous driving. Visual SLAM is a subclass of SLAM, which mainly calculates the camera motion trajectory corresponding to the current video by processing consecutive RGB-D or RGB video frames, and realizes map construction by combining the camera pose with the corresponding video frames. Visual SLAM uses a monocular camera, a binocular camera, or an RGB-D camera, etc. to collect information to locate the camera itself and reconstruct the surrounding environment, and has the advantages of low cost, lightweight hardware, and rich application scenarios.

[0003] 3D Gaussian Splatting is an explicit radiation field technology for establishing a 3D scene that can be quickly and efficiently rendered. A 3D Gaussian scene consists of 3D Gaussian basis elements, which include the center point position, the covariance matrix that controls the shape of the basis element, the opacity, and the spherical harmonic function coefficients used to calculate the color. By receiving RGB images from multiple viewpoints and their corresponding camera poses as inputs, the loss function between the rendered result of the 3D Gaussian scene and the ground truth RGB image is calculated, and the attributes of each 3D Gaussian basis element in the scene are continuously adjusted. After training is completed, the 3D Gaussian scene can generate high-quality new view rendering results.

[0004] Matsuki, H et al. proposed a visual SLAM system that combines 3D Gaussian sputtering. In monocular mode, when tracking the camera pose, the system selects key frames in the received RGB images by calculating the co-visibility and maintains an optimization window, ensuring the running speed of the system. During the tracking process, a certain Gaussian noise is added to the fixed depth to complete the depth initialization, which is also used as the center point of the initial primitive in the 3D Gaussian scene. When mapping, based on the established scene, the densification of 3D Gaussian primitives and pruning based on opacity are carried out to optimize the attributes of each 3D Gaussian primitive in the scene. At the same time, in order to suppress the out-of-control absolute scale of 3D Gaussian primitives, the system introduces a regularization method for suppression. By calculating the loss function between the rendering result of the established 3D Gaussian scene and the input RGB image, the system continuously optimizes the camera pose and the 3D Gaussian scene, and finally obtains the camera movement trajectory and the 3D Gaussian scene result that can generate the rendering result of the new view. Compared with traditional visual SLAM, which usually only generates geometric scene mapping, the introduction of the 3D Gaussian sputtering method makes the scene rendering result of this system anisotropic with the camera pose, thus obtaining a high-quality rendering result of the new view.

[0005] Fan, Z et al. proposed a new pruning strategy for 3D Gaussian primitives, establishing a global significance score to comprehensively judge the importance of 3D Gaussian primitives. By calculating and sorting the global significance scores of all 3D Gaussian primitives and calculating the boundary according to a fixed ratio, all 3D Gaussian primitives with global significance scores lower than the boundary are pruned. Compared with pruning only based on opacity, which will lose detailed information, the global significance score comprehensively considers the factors affecting the rendering of 3D Gaussian primitives and can retain 3D Gaussian primitives with low opacity but are more important during rendering, which is a more reasonable pruning strategy.

[0006] Visual SLAM has a wide range of applications in fields such as smart homes, augmented reality, and robotics. Matsuki, H et al. proposed the integration of 3D Gaussian sputtering method and visual SLAM earlier, enabling the visual SLAM system to generate high-quality rendering results of new views. However, this system still has certain defects. The lack of depth information in monocular visual SLAM affects the accuracy of camera pose tracking and also affects the position of the center point during the initialization of the 3D Gaussian scene, resulting in a decline in mapping performance. In addition, during the mapping optimization process, the system still only adopts the strategy in the original 3D Gaussian sputtering method, using opacity as the only pruning standard for 3D Gaussian primitives, which will lead to the loss of a large amount of scene details in the rendering result of the scene. Summary of the Invention

[0007] To solve the above problems, the present invention provides a 3D Gaussian sputtering monocular vision SLAM method combined with a neural network, which introduces a global saliency score to measure the importance of primitives in a 3D Gaussian scene, enabling the pruning method to retain more important primitives to obtain a higher-quality rendering result.

[0008] A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network, comprising:

[0009] S1: Obtain the co-visibility between the current RGB video frame and the latest key frame within the key frame window;

[0010] S2: Determine whether the co-visibility is lower than a threshold. If so, include the current RGB video frame in the key frame window as the latest key frame, perform monocular depth estimation on the latest key frame, and enter S3 after obtaining the predicted depth values corresponding to each pixel position on the latest key frame; if not, enter S4;

[0011] S3: Use the pixel positions on the latest key frame as the central coordinates of newly generated 3D Gaussian primitives, and then optimize the attributes of the newly generated and original 3D Gaussian primitives to obtain an optimized 3D Gaussian scene, and enter S5;

[0012] S4: Determine whether the current RGB video frame is the last frame. If so, obtain the camera trajectory composed of the camera poses corresponding to all RGB video frames; if not, use the next frame of the RGB video as the current RGB video frame and execute S1 - S2;

[0013] S5: Determine whether the current RGB video frame is the last frame. If so, the optimized 3D Gaussian scene is the final 3D Gaussian mapping result; if not, enter S6;

[0014] S6: Determine whether the number of optimization times is the set number. If not, use the next frame of the RGB video as the current RGB video frame and execute S1 - S2; if so, after performing densification and pruning on each 3D Gaussian primitive in the optimized 3D Gaussian scene, then use the next frame of the RGB video as the current RGB video frame and execute S1 - S2.

[0015] Further, the densification and pruning of each 3D Gaussian primitive in the optimized 3D Gaussian scene are specifically as follows:

[0016] S51: Divide the 3D Gaussian scene into multiple sub-regions, determine whether the density of 3D Gaussian primitives in each sub-region is greater than the set value. For the sub-regions with a judgment result of yes, the number of primitives remains unchanged; for the sub-regions with a judgment result of no, increase the number of 3D Gaussian primitives contained in itself by means of cloning and splitting to complete densification;

[0017] S52: Obtain the opacity of all 3D Gaussian basis elements in the 3D Gaussian scene, and at the same time render the 3D Gaussian scene to obtain the global saliency scores of each 3D Gaussian basis element;

[0018] S53: Remove the 3D Gaussian basis elements with opacity lower than the set threshold T1 and global saliency score lower than the set threshold T2, and the remaining 3D Gaussian basis elements are retained in the 3D Gaussian scene to complete the pruning process.

[0019] Further, the determination method of the set threshold T2 is:

[0020] Sort the global saliency scores of all 3D Gaussian basis elements from high to low, and calculate the boundary value according to the set ratio after sorting to obtain the set threshold.

[0021] Further, in step S2, when the judgment result is yes, if the current RGB video frame is included in the key frame window as the latest key frame and the number of all key frames exceeds the length upper limit of the key frame window, then remove the earliest key frame in the key frame window.

[0022] Further, the attributes of the 3D Gaussian basis element include the central coordinates, the covariance matrix controlling the basis element shape, the opacity, and the spherical harmonic function coefficients for calculating the basis element color.

[0023] Further, the initialization method of the key frame is:

[0024] Set the length of the key frame window to L;

[0025] Directly set the camera pose corresponding to the first RGB video frame in the RGB video as the center of the world coordinate system, and calculate the camera poses corresponding to the subsequent L - 1 RGB video frames through the parallax between the subsequent L - 1 RGB video frames and the first RGB video frame;

[0026] Include the first L RGB video frames in the RGB video in the key frame window as key frames, and use the coordinates of each pixel position on all key frames in the world coordinate as the central coordinates of the initially generated 3D Gaussian basis elements. At the same time, directly set the initial values for the attributes of the initially generated 3D Gaussian basis elements, or use an activation function to determine the attributes of the initially generated 3D Gaussian basis elements.

[0027] Further, in step S1, obtain the co - visibility between the current RGB video frame and the latest key frame in the key frame window according to the camera pose corresponding to the current RGB video frame.

[0028] Further, in step S2, use the DPT - Hybrid model to perform monocular depth estimation on the latest key frame in the key frame window.

[0029] Further, in step S3, the attributes of all newly generated and original 3D Gaussian basis elements are optimized by using photometric loss combined with regularization loss.

[0030] Further, in steps S4 and S5, it is determined whether the current RGB video frame is the last frame of the RGB video.

[0031] Beneficial effects:

[0032] The present invention provides a 3D Gaussian sputtering monocular vision SLAM method combined with a neural network, which uses the neural network to perform monocular depth estimation on the selected key frames and uses the depth results for the optimization of 3D Gaussian basis elements. At the same time, the present invention introduces a global saliency score to measure the importance of basis elements in the 3D Gaussian scene, and uses the global saliency score and opacity as a comprehensive judgment criterion for the importance of 3D Gaussian basis elements, and retains rendering details through comprehensive judgment to obtain a higher-quality rendering result. Description of the drawings

[0033] Figure 1 is a flowchart of a 3D Gaussian sputtering monocular vision SLAM method combined with a neural network provided by the present invention;

[0034] Figure 2 is a schematic diagram of the DPT model provided by the present invention. Detailed implementation manners

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0036] As Figure 1 shown, a 3D Gaussian sputtering monocular vision SLAM method combined with a neural network includes the following steps:

[0037] S0: Initialize key frames:

[0038] Set the length of the key frame window to L;

[0039] Directly set the camera pose corresponding to the first RGB video frame in the RGB video as the center of the world coordinate system, and calculate the camera poses corresponding to the subsequent L-1 RGB video frames through the parallax between the subsequent L-1 RGB video frames and the first RGB video frame;

[0040] All the first L RGB video frames in the RGB video are included in the key frame window as key frames, and the coordinates of each pixel position on all key frames in the world coordinates are used as the central coordinates of each initially generated 3D Gaussian primitive. At the same time, initial values are directly set for the attributes of each initially generated 3D Gaussian primitive, or an activation function is used to determine the attributes of each initially generated 3D Gaussian primitive.

[0041] It should be noted that a map built based on 3D Gaussian primitives can produce a good rendering effect. 3D Gaussian mapping, point clouds, polygon meshes, etc. are all mapping forms; the result of 3D Gaussian mapping is composed of a very large number of 3D Gaussian primitives, and these 3D Gaussian primitives can be concretely understood as an ellipsoid, mainly including the following attributes: [the central position (x, y, z) of the ellipsoid in the world coordinates, the shape of the ellipsoid (3x3 covariance), opacity (1 data), and the spherical harmonic coefficients representing the color (here, 3 coefficients of the 0th order are taken)].

[0042] S1: Obtain the camera pose corresponding to the current RGB video frame, and obtain the co-visibility between the current RGB video frame and the latest key frame in the key frame window according to the camera pose;

[0043] It should be noted that the camera pose needs to be calculated for each RGB video frame. First, a camera pose is calculated based on the parallax, and then the existing map is rendered for this camera pose to obtain a rendering result. At this time, the newly received RGB video frame is the ground truth. By calculating the loss function between the rendering result and the RGB frame ground truth, when it reaches the minimum value, it is considered that the camera pose optimization of the current frame is completed, that is, the camera pose of the current frame is determined.

[0044] S2: Determine whether the co-visibility is lower than the set threshold. If so, include the current RGB video frame in the key frame window as the latest key frame, and use the DPT-Hybrid model to perform monocular depth estimation on the latest key frame in the key frame window to obtain the predicted depth values corresponding to each pixel position on the latest key frame, and then enter step S3; if not, enter step S4;

[0045] It should be noted that in order to further improve the system operation speed, when the judgment result is yes, a new key frame is generated. If the current RGB video frame is included in the key frame window as the latest key frame and the number of all key frames exceeds the length limit of the key frame window, the earliest key frame in the key frame window is removed. For the newly incoming key frame, monocular depth estimation is performed. The pixel position in the camera coordinate system combined with the corresponding predicted depth information undergoes coordinate transformation to become a point (x, y, z) in the world coordinate system, and this point is used as the initialization center point of the new 3D Gaussian primitive. Each pixel of the key frame is processed in this way, so many new Gaussian primitives will be generated.

[0046] S3: Use the coordinates of each pixel position on the latest key frame in the world coordinate system as the central coordinates of the newly generated 3D Gaussian basis elements. Then, optimize the attributes of all the newly generated and original 3D Gaussian basis elements by combining photometric loss and regularization loss to obtain an optimized 3D Gaussian scene, and then proceed to step S5;

[0047] Specifically, the attributes of the 3D Gaussian basis elements of the present invention include central coordinates, covariance matrices that control the shape of the basis elements, opacity, and spherical harmonic function coefficients used to calculate the color of the basis elements. Since adding all video frames to the optimization process will result in very slow calculation and little improvement in the effect, key frames need to be found from all video frames, defined as key frames, and only use key frames to optimize the mapping.

[0048] That is to say, during the mapping initialization and the running process after initialization, monocular depth estimation will be performed on all selected key frames. The pixel positions in the camera coordinate system combined with the corresponding predicted depth information are converted through the coordinate system to become a point (x, y, z) in the world coordinate system. This point is used as the initialization center point of the new 3D Gaussian basis element.

[0049] S4: Determine whether the current RGB video frame is the last frame of the RGB video. If so, obtain the camera trajectory composed of the camera poses corresponding to all RGB video frames. If not, use the next frame of the RGB video as the current RGB video frame and execute steps S1 - S2;

[0050] That is to say, for each video frame in the input RGB video, camera pose tracking is performed, and finally a trajectory is obtained. Each camera pose and RGB video frame correspond one by one in time from front to back; that is, one video frame corresponds to one camera pose. The video frames are played from front to back to form a video, and the camera trajectory corresponds to form a trajectory.

[0051] S5: Determine whether the current RGB video frame is the last frame of the RGB video. If so, the optimized 3D Gaussian scene at this time is the final 3D Gaussian mapping result; if not, proceed to step S6;

[0052] S6: Determine whether the number of optimizations is the set number. If not, directly use the next frame of the RGB video as the current RGB video frame and execute steps S1 - S2; if so, perform densification and pruning processing on each 3D Gaussian basis element in the optimized 3D Gaussian scene, and then use the next frame of the RGB video as the current RGB video frame and execute steps S1 - S2.

[0053] It should be noted that densification and pruning are triggered once every fixed number of iterations. Note that this number of iterations starts accumulating after initialization and does not start from 0 every time a new key frame comes in.

[0054] Among them, the densification and pruning of each 3D Gaussian primitive in the optimized 3D Gaussian scene are specifically as follows:

[0055] S51: Divide the 3D Gaussian scene into multiple sub-regions, and judge whether the density of the 3D Gaussian primitives in each sub-region is greater than the set value. For the sub-regions where the judgment result is yes, the number of primitives remains unchanged. For the sub-regions where the judgment result is no, the number of 3D Gaussian primitives contained in itself is increased by means of cloning and splitting to complete densification;

[0056] S52: Obtain the opacity of all 3D Gaussian primitives in the 3D Gaussian scene, and at the same time render the 3D Gaussian scene to obtain the global saliency score of each 3D Gaussian primitive;

[0057] S53: Remove the 3D Gaussian primitives with opacity lower than the set threshold T1 and global saliency score lower than the set threshold T2, and the remaining 3D Gaussian primitives are retained in the 3D Gaussian scene to complete the pruning process.

[0058] Furthermore, the determination method of the set threshold T2 is as follows:

[0059] Sort the global saliency scores of all 3D Gaussian primitives from high to low, and calculate the boundary value according to the set ratio after sorting to obtain the set threshold.

[0060] The basic principle of monocular depth estimation is introduced below.

[0061] One of the major drawbacks of monocular visual SLAM is the lack of depth information. The introduction of a monocular depth estimation network can better make up for this. The present invention adopts the DPT-Hybrid model to perform monocular depth estimation on key frames. As Figure 2 shown, the DPT model first extracts non-overlapping features, flattens them and performs linear projection (DPT-Base and DPT-Large) or converts them into tokens by applying a ResNet-50 feature extractor (DPT-Hybrid). Then, position embeddings are used to enhance the image embeddings, and output heads independent of the features are added. After the tokens pass through multiple Transformer stages, the Reassemble module then reassembles the tokens from different stages into image-like representations with multiple resolutions. The Fusion module then performs step-by-step fusion and upsampling to the depth prediction result with higher resolution. Figure 2As shown in the middle figure, the Reassemble module reassembles the tokens into a feature map with a resolution of 1 / s of the input image's spatial resolution. Figure 2 As shown in the right figure, the Fusion module uses residual convolutional units for feature fusion and upsamples the feature map.

[0062] The DPT model proposes to use a Transformer instead of a convolutional neural network as the backbone architecture for dense prediction tasks. The Transformer backbone structure can process features at a relatively stable high resolution and has a global receptive field at each stage. Compared with monocular depth estimation networks with convolutional neural networks as the main structure, the DPT model can produce more refined and consistent depth prediction results. DPT-Large and DPT-Hybrid have better performance than DPT-Base, but based on the real-time requirements of visual SLAM, the present invention uses DPT-Hybrid with fewer parameters and computational requirements for monocular depth estimation.

[0063] It should be noted that for the monocular depth estimation network and the pre-trained parameter results of the dataset, other types of neural networks or pre-trained parameter results can be considered as alternatives; different monocular depth estimation networks and pre-trained results should be adopted for different application scenarios to ensure the flexibility and scalability of the technical solution.

[0064] The basic principles of global saliency score and pruning are introduced below.

[0065] The 3D Gaussian sputtering method's 3D Gaussian primitive pruning strategy is based on the opacity attribute of the primitives, that is, setting a threshold and pruning 3D Gaussian primitives with opacity lower than the threshold. LightGaussian proposes a new criterion for measuring the importance of 3D Gaussian primitives - the global saliency score. After calculating the global saliency scores of all 3D Gaussian primitives, they are sorted and the boundary value is calculated according to a certain ratio to obtain the threshold, and the 3D Gaussian primitives with global saliency scores lower than the threshold in the whole set are pruned, so as to obtain results that can retain more scene details.

[0066] For specific primitives, the global saliency score calculation method is as follows:

[0067]

[0068] where j is the Gaussian index, i represents the rendered pixel, M, H, and W represent the number of views, image height, and width participating in the global saliency score calculation respectively. 1 is an indicator function to determine whether the j primitive participates in the rendering of the i pixel. Since 3D Gaussian primitives with larger volumes participate in the rendering of more pixels, their importance is often exaggerated, resulting in the pruning of 3D Gaussian primitives that represent scene details. Therefore, in the formula, γ(∑j ) represents an adaptive method for restricting the global significance of large-volume 3D Gaussian basis elements, which is calculated as follows:

[0069]

[0070] Normalize using 90% of the largest volume among all 3D Gaussian basis elements. Those exceeding 1 are limited to 1 to reduce the influence of basis element volume on global significance calculation. Introduce β to provide additional flexibility for different application scenarios.

[0071] The global significance score comprehensively considers factors such as whether 3D Gaussian basis elements participate in pixel rendering, opacity, and basis element volume. Since these factors are related to 3D Gaussian scene rendering, the pruning strategy based on the global significance score is more reasonable than simply judging based on opacity.

[0072] Finally, the present invention uses a DPT-Hybrid monocular depth model pre-trained on the omnidata dataset for depth prediction. To ensure the running speed of 3DGS monocular visual SLAM, the model is loaded during the initialization of the tracking phase, and only the depth information of key frames is obtained instead of all input RGB video frames. While improving the tracking accuracy, it provides better center point initialization information for 3D Gaussian basis elements, reduces the pressure of mapping optimization, and improves mapping performance and speed.

[0073] The global significance score pruning strategy is applied in the mapping optimization process. When redundant 3D Gaussian basis elements appear in the scene, the system will initiate an additional rendering process to calculate the global significance score and perform pruning. However, since the present invention uses a key frame window for optimization and the camera poses corresponding to the window cannot cover most viewpoints well, there will be a large fluctuation in the global significance scores calculated for the same basis element at different times, resulting in a serious map forgetting problem. To improve stability, when the opacity of a 3D Gaussian basis element is higher than the threshold, the basis element will not be pruned.

[0074] In summary, the present invention provides a 3D Gaussian sputtering monocular vision SLAM method combined with a neural network. After receiving an RGB video frame, it estimates the camera pose and calculates the co-visibility between this frame and the previous key frame. When the co-visibility is lower than the threshold, this video frame is selected as a new key frame and a DPT model is used for monocular depth estimation. 3D Gaussian densification is performed according to the predicted depth result, and the center point position of the newly generated 3D Gaussian primitive is initialized according to the depth information. When the pruning condition is met, the global saliency scores of all 3D Gaussian primitives in the current scene are calculated and sorted, and the primitives with a relatively low ranking and non-compliant transparency are pruned. After completing the density adjustment of the primitives in the scene and the initialization of the center positions, a new rendering is performed; the mapping optimization process calculates the photometric loss and regularization between this rendering and the previous rendering based on the key frames within the current optimization window, and optimizes the corresponding camera pose and the attributes of the 3D Gaussian primitives in the scene.

[0075] It can be seen that the present invention makes up for the shortcoming of the lack of depth information in 3D Gaussian sputtering monocular vision SLAM by introducing a pre-trained monocular depth estimation model to predict the depth information of key frames. On the premise of minimizing the impact on the running speed as much as possible, more information is provided for camera pose tracking. At the same time, the depth information can also enable the newly emerged primitives to obtain better center point position initialization during 3D Gaussian scene densification, reducing the computational complexity and time consumption of mapping optimization.

[0076] In addition, the present invention adopts a more reasonable 3D Gaussian scene pruning strategy in the mapping optimization process and solves the unstable problem caused by coupling with the system. By calculating the global saliency scores of all 3D Gaussian primitives, comprehensively considering the number of times the primitives participate in the calculation, opacity, and primitive volume, and measuring the importance of the primitives with a more reasonable criterion, the 3D Gaussian primitives representing details in the scene can be better retained. At the same time, in order to adapt to the global saliency score fluctuations caused by mapping optimization in the key frame window, the 3D Gaussian primitives with an opacity reaching the threshold during the optimization process will not participate in pruning.

[0077] Of course, the present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can certainly make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations should all fall within the protection scope of the appended claims of the present invention.

Claims

1. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network, characterized in that, Including: S1: Obtain the co-visibility between the current RGB video frame and the latest key frame within the key frame window; S2: Determine whether the co-visibility is lower than the threshold. If so, incorporate the current RGB video frame into the key frame window as the latest key frame, perform monocular depth estimation on the latest key frame, obtain the predicted depth values corresponding to each pixel position on the latest key frame, and then enter S3; if not, enter S4; S3: Take the pixel positions on the latest key frame as the central coordinates of newly generated 3D Gaussian basis elements, and then optimize the attributes of the newly generated and original 3D Gaussian basis elements to obtain an optimized 3D Gaussian scene, and enter S5; S4: Determine whether the current RGB video frame is the last frame. If so, obtain the camera trajectory composed of the camera poses corresponding to all RGB video frames. If not, use the next frame of the RGB video as the current RGB video frame and execute S1 - S2; S5: Determine whether the current RGB video frame is the last frame. If so, the optimized 3D Gaussian scene is the final 3D Gaussian mapping result; if not, enter S6; S6: Determine whether the number of optimization times is the set number. If not, use the next frame of the RGB video as the current RGB video frame and execute S1 - S2; if so, after densifying and pruning each 3D Gaussian basis element in the optimized 3D Gaussian scene, then use the next frame of the RGB video as the current RGB video frame and execute S1 - S2.

2. The 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, wherein The densifying and pruning process of each 3D Gaussian basis element in the optimized 3D Gaussian scene is specifically as follows: S51: Divide the 3D Gaussian scene into multiple sub-regions, determine whether the density of 3D Gaussian basis elements in each sub-region is greater than the set value. For sub-regions with a judgment result of yes, the number of basis elements remains unchanged. For sub-regions with a judgment result of no, increase the number of 3D Gaussian basis elements it contains by means of cloning and splitting to complete densification; S52: Obtain the opacity of all 3D Gaussian basis elements in the 3D Gaussian scene, and at the same time render the 3D Gaussian scene to obtain the global saliency score of each 3D Gaussian basis element; S53: Remove 3D Gaussian basis elements with an opacity lower than the set threshold T1 and a global saliency score lower than the set threshold T2, and the remaining 3D Gaussian basis elements are retained in the 3D Gaussian scene to complete the pruning process.

3. The 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 2, characterized in that The determination method of the set threshold T2 is: Sort the global saliency scores of all 3D Gaussian basis elements from high to low, and calculate the boundary value according to the set ratio after sorting to obtain the set threshold.

4. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that, In step S2, when the judgment result is yes, if after incorporating the current RGB video frame into the key frame window as the latest key frame, the number of all key frames exceeds the upper limit of the key frame window length, then remove the earliest key frame within the key frame window.

5. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that, The attributes of the 3D Gaussian basis element include the central coordinates, the covariance matrix controlling the basis element shape, the opacity, and the spherical harmonic function coefficients used to calculate the basis element color.

6. The 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that, The initialization method of the key frame is: Set the length of the key frame window to L; The camera pose corresponding to the first RGB video frame in the RGB video is directly set as the center of the world coordinate system, and the camera poses corresponding to the subsequent L - 1 RGB video frames are calculated through the parallax between the subsequent L - 1 RGB video frames and the first RGB video frame. All the first L RGB video frames in the RGB video are included in the key - frame window as key frames, and the coordinates of each pixel position on all key frames in the world coordinate are used as the center coordinates of each initially generated 3D Gaussian basis element. At the same time, the initial values of the attributes of each initially generated 3D Gaussian basis element are directly set, or an activation function is used to determine the attributes of each initially generated 3D Gaussian basis element.

7. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that, In step S1, the co - visibility between the current RGB video frame and the latest key frame in the key - frame window is obtained according to the camera pose corresponding to the current RGB video frame.

8. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that, In step S2, the DPT - Hybrid model is used to perform monocular depth estimation on the latest key frame in the key - frame window.

9. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that, In step S3, the attributes of all newly generated and original 3D Gaussian basis elements are optimized by combining photometric loss and regularization loss.

10. A 3D Gaussian sputtering monocular vision SLAM method combined with a neural network according to claim 1, characterized in that In steps S4 and S5, it is judged whether the current RGB video frame is the last frame of the RGB video.

Images