A three-dimensional gaussian-based incremental map optimization method

Abstract

The application provides a kind of based on three-dimensional Gauss incremental map optimization method, comprising: S1: obtaining current frame and estimated station posture, S2: judge whether the current frame is key frame;If yes, execute step S3;S3: based on the estimated station posture, rendering contour image is obtained;S4: obtain historical three-dimensional Gauss graph;Based on the contour image and the historical three-dimensional Gauss graph, determine the pixel corresponding to the new area;S5: based on the pixel corresponding to the new area, the historical three-dimensional Gauss graph is optimized and handled, and the first precision three-dimensional Gauss graph corresponding to the new area is obtained;S6: obtain historical key frame, based on the historical key frame, the first precision three-dimensional Gauss graph is optimized, and the second precision three-dimensional Gauss graph corresponding to the new area is obtained.In the process of gradually optimizing map, to reduce redundancy, improve efficiency and accuracy, realize high fidelity, high efficiency rendering.

Description

Technical Field

[0001] The present application relates to the field of map construction, and in particular to the field of incremental map update optimization. Background Art

[0002] Incremental map updates refer to a method of updating and modifying only the parts of existing map data that have changed. This method identifies and captures information about newly added, deleted, or modified features in geographic space, such as new road construction and widening, building demolition and new construction, and then integrates this change information into the original map data. This allows for timely and accurate map updates with minimal data updates, maintaining consistency and currency with the real world, and improving map practicality and reliability.

[0003] However, with existing technologies, it's difficult to achieve high-fidelity modeling of complex, small objects during incremental map updates. Furthermore, computational redundancy leads to low accuracy and efficiency during incremental map updates. Summary of the Invention

[0004] In order to overcome the above technical defects, the present application provides a three-dimensional Gaussian-based incremental map optimization method, storage medium and device.

[0005] To achieve the above objectives, this application is implemented according to the following technical solutions:

[0006] In a first aspect, the present application provides an incremental map optimization method based on three-dimensional Gaussian, which is applied to an incremental mapping submodule. The incremental mapping submodule and the multi-sensor visual odometry submodule constitute a positioning and mapping module, including:

[0007] S1: Obtain the current frame and estimated standing posture, wherein the estimated standing posture is obtained by the output of the multi-sensor visual odometry submodule;

[0008] S2: Determine whether the current frame is a key frame;

[0009] If yes, go to step S3;

[0010] S3: Rendering to obtain a contour image based on the estimated standing posture;

[0011] S4: Obtain historical three-dimensional Gaussian graph;

[0012] Determining pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian graph;

[0013] S5: Optimizing the historical three-dimensional Gaussian graph based on the pixels corresponding to the newly added area to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area;

[0014] S6: Acquire historical key frames, wherein the historical key frames include an initial key frame and a previous key frame;

[0015] Based on the historical key frames, loss optimization is performed on the first-precision three-dimensional Gaussian map to obtain a second-precision three-dimensional Gaussian map corresponding to the newly added area, wherein the first precision is less than the second precision.

[0016] Optionally, determining whether the current frame is a key frame is specifically as follows:

[0017] Determining whether the pixel similarity between the current frame and the previous key frame is greater than a first threshold;

[0018] If so, the current frame is used as a key frame.

[0019] Optionally, rendering a contour image based on the estimated standing posture includes:

[0020] Determining, based on the key frame, a current camera viewing angle corresponding to the key frame;

[0021] The estimated standing posture is rendered based on the current camera viewing angle to obtain a silhouette image.

[0022] Optionally, determining pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian graph includes:

[0023] generating a mask based on the contour image;

[0024] Matching the mask with the historical three-dimensional Gaussian graph to determine pixels corresponding to the newly added area;

[0025] Optionally, the optimizing process of the historical three-dimensional Gaussian graph based on the pixels corresponding to the newly added area to obtain the first-precision three-dimensional Gaussian graph corresponding to the newly added area includes:

[0026] Obtaining newly registered radar points in the time interval between the current frame and the previous key frame;

[0027] Randomly sampling one half of the newly registered radar points;

[0028] Determining an optimizable radar point based on pixels corresponding to the newly added area and half of the randomly sampled newly registered radar points;

[0029] Determining an optimizable Gaussian point based on the optimizable radar point;

[0030] Based on the optimizable Gaussian points, the historical three-dimensional Gaussian graph is optimized to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area.

[0031] Optionally, the determining the optimizable radar points based on the pixels corresponding to the newly added area and half of the randomly sampled newly registered radar points includes:

[0032] Projecting half of the randomly sampled newly registered radar points into pixels corresponding to the newly added area;

[0033] Determine whether one-half of the randomly sampled newly registered radar points can be projected into pixels corresponding to the newly added area;

[0034] If so, the radar point that can be projected into the pixel corresponding to the newly added area is used as the optimizable radar point.

[0035] Optionally, performing loss optimization on the first precision three-dimensional Gaussian map based on the historical key frames to obtain a second precision three-dimensional Gaussian map corresponding to the newly added area includes:

[0036] Step 701: Randomly disrupt the historical key frames, and use the following formula to iteratively process the first precision three-dimensional Gaussian image by minimizing the photometric rendering loss to obtain an optimized three-dimensional Gaussian rendering depth image;

[0037]

[0038] Among them, I and They are the images corresponding to the observed (collected) historical key frames and the images obtained by Gaussian representation of the rendered historical key frames, L D-SSIM To constrain structural consistency for structural similarity, λ refers to the weight, which is set manually, and L refers to the loss;

[0039] Step S702: Determine whether the number of iterative calculations for minimizing photometric rendering loss is greater than a first preset number of calculations;

[0040] If yes, then end the iterative calculation process and use the optimized three-dimensional Gaussian rendering depth map as the second-precision three-dimensional Gaussian map corresponding to the newly added area;

[0041] If not, determining whether the number of iterative calculations for minimizing the photometric rendering loss is equal to a second preset number of calculations; wherein the second preset number of calculations is the number of calculations obtained by dividing the first preset number of calculations according to a preset gradient;

[0042] If yes, execute step 703;

[0043] If not, the optimized three-dimensional Gaussian rendering depth map is used as the first-precision three-dimensional Gaussian map for the next round of iterative processing, and step 701 is repeated;

[0044] Step 703: Obtain an optimized 3D Gaussian rendering depth map;

[0045] Calculating cumulative opacity for each pixel in the optimized three-dimensional Gaussian rendering depth map;

[0046] If the cumulative opacity of any pixel among the pixels is greater than a second threshold, or the difference between the rendering depth corresponding to the pixel and the laser point cloud depth is greater than a third threshold, the pixel is marked as an unstable pixel;

[0047] determining, based on the unstable pixel, a three-dimensional Gaussian graph corresponding to the unstable pixel;

[0048] Processing the three-dimensional Gaussian map corresponding to the unstable image using an adaptive densification and pruning strategy to obtain a processed first optimized loss Gaussian map;

[0049] The processed first optimized loss Gaussian map is used as the first precision three-dimensional Gaussian map for the next repeated iterative process of minimizing the photometric rendering loss, and the above steps S701-S703 are repeated.

[0050] Optionally, the estimated standing posture is obtained by outputting the multi-sensor visual odometry submodule, including:

[0051] Input the radar point cloud map, current frame and inertial navigation data into the multi-sensor visual odometry submodule;

[0052] Determining the estimated stance based on the radar point cloud map, the current frame, and the inertial navigation data;

[0053] The estimated standing posture is output through the multi-sensor visual odometry sub-module.

[0054] In a second aspect, the present application provides a computer device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 8.

[0055] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 8.

[0056] This application has the following beneficial effects:

[0057] This application reduces the amount of image computation required during map updates by evaluating the current frame and registering it as a keyframe. By introducing 3D Gaussian representation as the foundation for real-time positioning and mapping of large-scale scene models, the application improves the quality of scene map construction by enabling high-fidelity modeling of complex, small objects. In parallel with the multi-sensor odometry submodule, the pose tracking results are used, along with an incremental map optimization method, to gradually optimize the map, reduce redundancy, improve efficiency and accuracy, and achieve high-fidelity, high-efficiency rendering.

[0058] In addition to the above-described purposes, features and advantages, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0060] Figure 1 This is a flow chart of the real-time construction method of a three-dimensional scene in this application.

[0061] Figure 2 Schematic diagram of a multi-sensor odometer submodule according to an embodiment of the present application.

[0062] Figure 3 This is a flowchart of a three-dimensional Gaussian-based incremental map optimization method provided in this application.

[0063] Figure 4 Schematic diagram of the incremental mapping submodule of an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application are described in detail below with reference to the accompanying drawings. However, the present application can be implemented in many different ways as defined and covered by the claims.

[0065] Since this application mentions multiple different maps and some conceptual judgments, in order to avoid misunderstandings, a brief explanation of them is given here:

[0066] Global LiDAR Map: A global LiDAR map is a comprehensive description of the environment constructed using LiDAR data. Through point cloud registration and feature extraction, it features high precision and rich three-dimensional information. It has important applications in many fields, including autonomous driving and robot navigation.

[0067] 3D Gaussian plot: A 3D Gaussian plot is a graph based on three-dimensional space and a Gaussian distribution to represent data features. It uses a Gaussian function to describe the shape, position, and uncertainty of an object in three-dimensional space, and has a wide range of applications in various fields.

[0068] It should be noted that the method in this application is mainly used for real-time construction of three-dimensional scenes in large-scale water scenes. Figure 1 As shown in the figure, the real-time construction of the 3D scene is mainly based on the joint calibration module and the positioning real-time mapping module. The following is a brief introduction to the functions of these modules:

[0069] The joint calibration module is used to perform high-precision joint calibration of the external parameters of the LiDAR, IMU, and camera in the hardware platform. This module fully utilizes the high-resolution visual information provided by the camera, the high-precision distance measurement provided by the LiDAR, and the dynamic information such as acceleration and angular velocity provided by the IMU to eliminate the potential impact of errors and drift of each sensor on system performance. Through joint calibration, it can ensure that the data of these sensors are aligned in the same coordinate system, thereby improving the accuracy of data fusion. By reducing these errors during the data fusion process, the overall stability and reliability of the system are improved, which plays a vital supporting role in subsequent perception, positioning, navigation, and decision-making tasks.

[0070] The positioning and real-time mapping module is used for unmanned platforms to efficiently determine their own position in large-scale water surface environments using multi-sensor information and to build a visually appealing and realistic three-dimensional map of dense water surface scenes in real time, providing precise location information and high-quality dense map information for subsequent tasks such as path planning and autonomous control.

[0071] Since the method of this application is mainly based on the positioning and real-time mapping modules, it is described in detail here:

[0072] The positioning and real-time mapping module includes a multi-sensor visual odometry submodule and an incremental mapping submodule.

[0073] The multi-sensor odometry submodule is a robust pose tracking system that combines camera, inertial navigation, and radar. The specific process is as follows: Figure 2 As shown in the figure, the radar point cloud map, the current image frame, and the inertial navigation data are input. First, the SURF feature points of the point cloud are extracted and the map points in the previous frame are projected to the current frame using optical flow tracking. Then, the radar point cloud and the map are scan-matched. The residuals of the radar, camera, and inertial navigation are calculated and optimized on a continuous-time trajectory represented by non-uniform B-splines. The map points are then marginalized and updated. The above steps are repeated after the sensor data at the new time point is input, and the continuous-time joint optimization trajectory is finally output.

[0074] Specifically, a global map is constructed using radar point clouds, and 3D map points are associated with pixels in camera images, avoiding the need for deep optimization of visual features. This allows for efficient fusion of heterogeneous radar, inertial, and camera data within a short sliding window optimization. Raw IMU measurements and a random walk process are used to construct IMU factors and bias factors. Subsequently, a continuous-time optimization using non-uniform B-splines is performed based on the point-to-surface error from the radar point cloud to the map, the reprojection error from the camera image to the map, and the IMU factors and bias factors. The output is an estimated stance pose for the radar point cloud map. During the optimization process, control points are dynamically and adaptively added to improve accuracy and real-time performance, taking into account dynamic changes in motion.

[0075] That is to say, the multi-sensor odometry submodule processes the input radar point cloud map, the current image frame and the inertial navigation data, and outputs the estimated standing posture.

[0076] The incremental mapping submodule is based on the 3D Gaussian scene representation method. It takes the current frame pose estimated by the multi-sensor odometry module as input and progressively builds a complete, high-fidelity, dense 3D scene map during step-by-step optimization.

[0077] In order to solve the problems raised by the above background technology, Figure 3 and Figure 4 The present application proposes a three-dimensional Gaussian-based incremental map optimization method, which is applied to the incremental mapping submodule. The incremental mapping submodule and the multi-sensor visual odometry submodule constitute a positioning and mapping module, including:

[0078] Step S1: obtaining a current frame and an estimated standing posture, wherein the estimated standing posture is output by the multi-sensor visual odometry submodule;

[0079] The current frame represents the sensor data captured by the system at the current moment. For example, it represents a point cloud data frame scanned by a LiDAR radar, or an image data frame captured by a camera at the same moment. This current frame data provides the system with real-time environmental information for decision-making and control.

[0080] Estimating pose refers to the process of estimating the position and attitude of an object in space. By acquiring data from various sensors and applying specific algorithms, the orientation, angle, and other information of the object relative to the reference frame are determined. This has important applications in robotics, autonomous driving, and surface scene construction. The estimated pose in this application is obtained from the output of the multi-sensor visual odometry submodule and input into the incremental mapping submodule for subsequent calculations and processing. The multi-sensor visual odometry submodule and the process of determining the estimated pose have been described above and will not be elaborated on here.

[0081] In this application, a set of control strategies are carefully designed to make three-dimensional Gaussian sputtering more suitable for incremental mapping. For the first time step, information input based on multiple sensors, such as the first frame image, estimated posture, and the lidar point cloud corresponding to the first frame, are used as the starting point for constructing a three-dimensional Gaussian map, thereby forming an original three-dimensional Gaussian map. The first frame of input image is set as the key frame by default. Therefore, all radar points included in the lidar point cloud corresponding to the first frame image in the first time step are used to initialize the three-dimensional Gaussian. Each Gaussian center point is placed at the corresponding radar point position, and its zero-degree spherical harmonics are initialized with the radar point color attribute, and a smaller scale is given to the nearby Gaussian, and a larger scale is given to the Gaussian in the principle imaging plane to reduce potential aliasing artifacts. The radius of each Gaussian point projected onto the image plane is 2 pixels:

[0082]

[0083] Where e is a 3*1 unit vector, d represents the depth of the radar point in the image coordinate system, f is the focal length of the camera, and S is the image plane.

[0084] It should be noted that the above steps are only for generating the original three-dimensional Gaussian image and do not involve other Gaussianization processes.

[0085] After generating the original three-dimensional Gaussian map, this application will subsequently perform continuous incremental updates and optimizations on the original three-dimensional Gaussian map based on the input information of multiple sensors. It should be noted that each subsequent incremental update and optimization of the three-dimensional Gaussian map is performed on the basis of the previous incremental update and optimization of the three-dimensional Gaussian map, and not every incremental update and optimization is performed on the basis of the original three-dimensional Gaussian map.

[0086] Step S2: Determine whether the current frame is a key frame, if so, execute step S3;

[0087] For online incremental mapping systems, in order to achieve efficient and real-time map construction, it is computationally infeasible to use all received images for optimization. In addition, in order to obtain a complete environmental map, the Gaussian scene representation should be able to simulate the geometry and appearance of the newly observed area. Therefore, the optimization of the three-dimensional scene representation is performed only on the keyframes. Instead of simply setting the number of time steps as the keyframe division standard, the keyframes are adaptively set based on the pixel similarity between the current frame and the previous keyframe, taking into full account the uncertainty of the unmanned platform's movement and steering.

[0088] Determine whether the pixel similarity between the current frame and the previous key frame is greater than the first threshold. If so, register the current frame as a key frame. If not, treat the current frame as a normal input frame and perform simple subsequent processing. The first threshold represents that compared with the previous key frame, the current frame is likely to have a new area. Therefore, by setting the first threshold, the current frame can be registered as a key frame in a timely manner, and then the incremental map update is completed based on the key frame. The specific value of the first threshold can be set by yourself based on the actual situation on site, the situation of the sensor, or the method of data processing. This article does not impose specific restrictions on this.

[0089] Step S3: Rendering to obtain a contour image based on the estimated standing posture;

[0090] Each frame received after the first one typically captures the geometry and appearance of the newly observed area. However, lidar points from different frames may contain duplicate or very similar information. To avoid redundancy, after determining that the current frame is a keyframe, we first determine the current camera viewpoint corresponding to the keyframe. We then render the estimated pose from the current camera viewpoint to obtain a silhouette image V.

[0091] Step S4: Obtain historical three-dimensional Gaussian graph;

[0092] Determining pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian graph;

[0093] The historical 3D Gaussian graph is a 3D Gaussian graph obtained by performing 3D Gaussian optimization on the historical key frames before the key frame using the method of the present application. For example, if the current frame is the fifth key frame, the historical 3D Gaussian graph is a 3D Gaussian graph obtained by performing 3D Gaussian optimization on the first four key frames using the method of the present application.

[0094] After obtaining the above-mentioned contour image V, a mask M is generated according to the contour image V, and the mask M is used to perform matching calculations with the three-dimensional Gaussian map constructed by the current Gaussian, so that pixels corresponding to unreliable and prone to observing new areas are selected in the mask, and the pixels corresponding to unreliable and prone to observing new areas are used as pixels corresponding to the newly added areas, so as to perform subsequent calculations.

[0095] The above-mentioned mask is used to perform matching calculation with the three-dimensional Gaussian map constructed by the current Gaussian, so that in the mask, pixels with a value of 0 in the mask are regarded as pixels corresponding to unreliable and tend to observe new areas, and are thus selected.

[0096] Step S5: Optimizing the historical three-dimensional Gaussian graph based on the pixels corresponding to the newly added area to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area;

[0097] Since this step uses the global lidar point cloud and the newly registered radar points in the global lidar point cloud, it is described in detail:

[0098] In particular, in order to use a radar point cloud with a more accurate geometric prior as a guide point set for three-dimensional Gaussian sputtering, instead of using the usual SFM method, the system corresponding to this application maintains a global lidar map stored in voxels with a resolution of ten meters to facilitate operation and access to the radar point cloud. After receiving the input frame and the corresponding radar point cloud map, the system randomly samples one-quarter of the radar points, converts them into world coordinates, and registers them to the voxels to improve efficiency, ensuring that the distance between radar points in each voxel is greater than five centimeters to reduce redundancy. For successfully registered radar points, they are mapped to the nearest image and the pixel value is queried to obtain its color attribute. It should be noted that the input frame mentioned here is not the key frame mentioned above. When applying the method mentioned in this application for three-dimensional Gaussian optimization, it can be understood as the current frame.

[0099] After determining the pixels corresponding to the newly added area, the historical 3D Gaussian map needs to be optimized. In other words, the Gaussian corresponding to the newly added area needs to be added to the historical 3D Gaussian map. Therefore, after determining the pixels corresponding to the newly added area, they are projected into the maintained global lidar point cloud, and the newly registered radar points in the time interval between the current frame (new keyframe) and the previous keyframe are accessed. Considering the sparsity of surface targets, in order to reduce unnecessary memory consumption, half of the radar points are randomly sampled. Then, based on the newly registered radar points and the pixels corresponding to the newly added area, optimizable radar points are determined. After determining the optimizable radar points, optimizable Gaussian points are determined based on the optimizable radar points. Then, based on the optimizable Gaussian points, the historical 3D Gaussian map is optimized to obtain the first-precision 3D Gaussian map corresponding to the newly added area.

[0100] The first-precision 3D Gaussian map corresponding to the newly added area can be understood as a low-precision 3D Gaussian map corresponding to the newly added area. This is because only a preliminary optimization of the historical 3D Gaussian map corresponding to the newly added area has been performed. Therefore, the precision of the 3D Gaussian map corresponding to the newly added area obtained at this time is relatively low. The specific setting of the numerical range of the first precision can be customized according to the actual situation on site, and this article does not impose specific restrictions on this. For example, the low-precision numerical range can be determined based on the camera's shooting pixels.

[0101] The radar points that can be optimized are determined based on the pixels corresponding to the newly registered radar points and the newly added areas of the randomly sampled one-half, specifically:

[0102] The new registered radar points of the random sampling half are projected into the pixels corresponding to the added region, and it is determined whether the new registered radar points of the random sampling half can be projected into the pixels corresponding to the added region, as shown in formula (2). Only the radar points that can be projected onto the selected pixels can be used as the optimizable radar points for subsequent initialization of Gauss.

[0103]

[0104] M is a contour image, V is a mask, and a is the contribution of each Gaussian point to a certain pixel point, which is calculated by the product of the opacity and the probability density function of the three-dimensional Gaussian projection onto the imaging plane.

[0105] After determining the optimizable radar points, they are projected into the world coordinate system to determine the optimizable Gaussian points. Then, the historical three-dimensional Gaussian map is optimized according to the optimizable Gaussian points in the world coordinate system, thereby completing the addition of the added region and obtaining the first-precision three-dimensional Gaussian map corresponding to the added region. It should be noted that the specific calculation process of the projection and the specific calculation process of the optimization are common calculation methods in the field, so the calculation process will not be described in detail.

[0106] S6: Obtain a historical key frame, wherein the historical key frame includes an initial key frame and a previous key frame;

[0107] Based on the historical key frame, the first-precision three-dimensional Gaussian map is loss-optimized to obtain a second-precision three-dimensional Gaussian map corresponding to the added region, wherein the first precision is less than the second precision. The historical key frame is stored in a multi-sensor vision odometer submodule.

[0108] The second-precision three-dimensional Gaussian map corresponding to the added region can be understood as a high-precision three-dimensional Gaussian map corresponding to the added region. Because the first-precision three-dimensional Gaussian map is loss-optimized for the three-dimensional Gaussian corresponding to the added region, the precision of the three-dimensional Gaussian map corresponding to the added region obtained at this time is high. The specific setting of the numerical range of the second precision can be self-set according to the actual situation, and this article does not make specific limitations. Exemplarily, the numerical range of high precision can be determined according to the presentation requirements of the Gaussian map.

[0109] In order to further limit the computational complexity to improve efficiency and improve the precision of the three-dimensional Gaussian corresponding to the added region, we select K historical key frames to optimize the Gaussian map in each key frame mapping process, including the initial key frame and the previous key frame, to avoid the catastrophic forgetting problem and maintain the geometric consistency of the global map. The specific steps are as follows:

[0110] Step 701: Randomly disrupt the historical key frames, and use the following formula (3) to iteratively process the first precision Gaussian map by minimizing the photometric rendering loss to obtain an optimized three-dimensional Gaussian rendering depth map:

[0111]

[0112] Among them, I and They are the images corresponding to the observed (collected) historical key frames and the images obtained by Gaussian representation of the rendered historical key frames, L D-SSIM To constrain structural consistency for structural similarity, λ refers to the weight, which is set manually, and L refers to the loss;

[0113] The first-precision Gaussian image is iteratively processed to minimize the photometric rendering loss. The purpose is to improve the image rendering quality, optimize the model parameters, enhance the robustness, achieve accurate photometric estimation, narrow the difference with the real image, make the generated image more realistic and natural, and enable the model to better fit the data and adapt to various scenarios.

[0114] Step S702: Determine whether the number of iterative calculations for minimizing photometric rendering loss is greater than a first preset number of calculations;

[0115] If yes, then end the iterative calculation process and use the optimized three-dimensional Gaussian rendering depth map as the second-precision three-dimensional Gaussian map corresponding to the newly added area;

[0116] If not, determining whether the number of iterative calculations for minimizing the photometric rendering loss is equal to a second preset number of calculations; wherein the second preset number of calculations is the number of calculations obtained by dividing the first preset number of calculations according to a preset gradient;

[0117] If yes, execute step 703;

[0118] If not, the optimized three-dimensional Gaussian rendering depth map is used as the first-precision three-dimensional Gaussian map for the next round of iterative processing, and step 701 is repeated;

[0119] Since erroneous displacement or deformation may occur during the iterative optimization of the first-precision 3D Gaussian map through photometric rendering loss, we render the cumulative opacity of pixels to classify them into stable and unstable categories. Adaptive densification and pruning strategies are used on the 3D Gaussians corresponding to unstable pixels to promote more refined mapping and compensate for the defects of LiDAR that cannot cover the entire scene, especially in unbounded large-scale water environments, and the possible initialization of erroneous 3D Gaussians. Ultimately, continuous-time 3D scene map construction is achieved in large water scenes.

[0120] When the first-precision three-dimensional Gaussian image is iteratively processed to minimize the photometric rendering loss, the number of iterative calculations will reach a certain number. Then, it is impossible to immediately perform corresponding processing on the optimized three-dimensional Gaussian rendering depth map after each iterative process. This will increase the calculation time and reduce efficiency. Therefore, we reduce the number of processing calculations by setting a first preset number of calculations and a second preset number of calculations. The second preset number of calculations is the number of calculations obtained after dividing the first preset number of calculations according to the preset gradient. For example, if the first preset number of calculations is 3000, the first preset number of calculations can be set according to a gradient of 100, and the order of the first preset number of calculations can be 100, 200, 300...2900, 3000.

[0121] Then, according to the first preset number of calculations and the second preset number of calculations, specific steps for optimizing the three-dimensional Gaussian rendering depth map are determined, and the following description is given in order:

[0122] If the number of iterative processing calculations for minimizing the photometric rendering loss (hereinafter simplified as the number of iterative processing calculations for ease of understanding) is greater than the first preset number of calculations, it means that the iterative processing calculations have been completed at this time. At this time, the optimized three-dimensional Gaussian rendering depth map is used as the second-precision three-dimensional Gaussian map to complete the optimization of the three-dimensional Gaussian area corresponding to the newly added area.

[0123] If the number of iterative processing calculations is less than or equal to the first preset number of calculations, it means that the optimization processing of the first precision three-dimensional Gaussian image has not been completed at this time. At this time, it is necessary to further determine whether the number of iterative processing calculations meets the second preset number of calculations to determine the subsequent processing of the first precision three-dimensional Gaussian image.

[0124] If the number of iterative processing calculations is equal to the second preset number of calculations, it means that the number of iterative processing calculations is sufficient. The possibility of erroneous displacement or deformation in the optimized three-dimensional Gaussian depth rendering image is greatly increased. At this time, step 703 can be executed.

[0125] If the number of iterative calculations is not equal to the second preset number of calculations, it means that the iterative processing of minimizing the photometric rendering loss is still being performed on the first precision three-dimensional Gaussian image using formula (3).

[0126] Step S703: Obtaining an optimized 3D Gaussian rendering depth map;

[0127] Calculating cumulative opacity for each pixel in the optimized three-dimensional Gaussian rendering depth map;

[0128] If the accumulated opacity of any pixel point in each pixel point is less than a second threshold value, or the rendering depth corresponding to the any pixel point and the laser point cloud depth differ, the pixel point is marked as an unstable pixel;

[0129] Based on the unstable pixel, a three-dimensional Gaussian graph corresponding to the unstable pixel is determined;

[0130] The three-dimensional Gaussian graph corresponding to the stable image is processed using an adaptive densification and pruning strategy to obtain a first processed optimized loss Gaussian graph.

[0131] The first processed optimized loss Gaussian graph is used as the first precision three-dimensional Gaussian graph for the next iteration of the repeated minimization of photometric rendering loss processing, and steps S701-S703 are repeated.

[0132] Because the area formed by the general unstable pixel is most likely to be used to capture new scene content, i.e., there may be missing new areas, and there is unstable adaptive control when using formula (3) optimization, new Gaussians may be generated, resulting in poor mapping quality. Therefore, in order to improve the precision of the three-dimensional Gaussian graph corresponding to the new area, the three-dimensional Gaussian area corresponding to the unstable pixel needs to be optimized. Specifically:

[0133] After obtaining the optimized three-dimensional Gaussian rendering depth map, first, the accumulated opacity of each pixel point in the optimized three-dimensional Gaussian rendering depth map is calculated. The accumulated opacity can be equivalent to a contour graph in form. After calculating the accumulated opacity, the accumulated opacity of each pixel point is judged. If the accumulated opacity of any pixel point in each pixel point is greater than a second threshold value, or the rendering depth corresponding to any pixel point and the laser point cloud differ by a value greater than a third threshold value, the pixel point is marked as an unstable pixel. Therefore, according to the unstable pixel, a three-dimensional Gaussian graph corresponding to the unstable pixel is generated by back projection into the world coordinate system. Then, the three-dimensional Gaussian graph corresponding to the unstable image is processed using an adaptive densification and pruning strategy to obtain a first processed optimized loss Gaussian graph. The specific range of the second threshold value and the third threshold value is set by the on-site technical personnel according to the specific situation of the site, and is not limited in this paper.

[0134] The adaptive densification and pruning strategy includes a densification process and a pruning process. The densification process is a densification operation on the three-dimensional Gaussian graph corresponding to the unstable pixel. The pruning process is to check all observable Gaussian points in the current three-dimensional Gaussian graph from the current camera frustum after adding new Gaussians, and explicitly reduce the opacity of Gaussian points far from the scene surface, so that such Gaussian points are periodically automatically excluded in the subsequent optimization process, thereby obtaining a first processed optimized loss Gaussian graph.

[0135] After obtaining the processed first optimization loss Gaussian map, since the iteration process is not completed, the processed first optimization loss Gaussian map is taken as the first precision three-dimensional Gaussian map for the next iteration of the minimization photometric rendering loss, and the steps S701-S703 are repeated to continue the iteration process and periodically optimize the first precision three-dimensional Gaussian map.

[0136] In summary, in the present application, the current frame is registered as a key frame after being judged, reducing the calculation amount of pictures in the map updating process, and a three-dimensional Gaussian representation is introduced as a basic method for real-time positioning and mapping of large-scale scene modeling, aiming at high-fidelity modeling of complex structure and small object, improving the quality of scene map construction. In parallel with the multi-sensor odometer sub-module, the pose tracking result is used and an incremental map optimization method is used to gradually optimize the map, reduce redundancy, improve efficiency and accuracy, and realize high-fidelity and efficient rendering.

[0137] In some embodiments, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, causes the processor to implement the three-dimensional Gaussian-based incremental map optimization method of the first aspect.

[0138] Those of ordinary skill in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the functional blocks / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer readable storage medium, which can include a computer readable storage medium (or non-transitory medium) and a communication medium (or transitory medium).

[0139] As is well known to those skilled in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically contains computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0140] Exemplarily, the computer-readable storage medium may be an internal storage unit of the network management device described in the aforementioned embodiment, such as a hard disk or memory of the network management device. The computer-readable storage medium may also be an external storage device of the network management device, such as a plug-in hard disk, a SmartMediaCard (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the network management device.

[0141] In some embodiments, a device is provided, comprising a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute the computer program and implement the three-dimensional Gaussian-based incremental map optimization method provided in the first aspect of the present application when executing the computer program.

[0142] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0143] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A 3D Gaussian-based incremental map optimization method, applied to the incremental mapping submodule, wherein the incremental mapping submodule and the multi-sensor visual odometry submodule constitute a positioning and mapping module, characterized in that: include: S1: Obtain the current frame and estimated standing posture, wherein the estimated standing posture is obtained by the output of the multi-sensor visual odometry submodule; S2: Determine whether the pixel similarity between the current frame and the previous key frame is greater than a first threshold; If yes, take the current frame as the key frame and execute step S3; S3: Determine the current camera angle corresponding to the key frame according to the key frame; Rendering the estimated standing posture based on the current camera viewing angle to obtain a silhouette image; S4: generating a mask based on the contour image; Get historical 3D Gaussian graph; Matching the mask with the historical three-dimensional Gaussian graph to determine pixels corresponding to the newly added area; S5: Obtain newly registered radar points in the time interval between the current frame and the previous key frame; Randomly sampling one half of the newly registered radar points; Projecting half of the randomly sampled newly registered radar points to pixels corresponding to the newly added area; Determine whether one-half of the randomly sampled newly registered radar points can be projected onto pixels corresponding to the newly added area; If yes, the radar point that can be projected into the pixel corresponding to the newly added area is used as the optimizable radar point; Determining an optimizable Gaussian point based on the optimizable radar point; Based on the optimizable Gaussian points, the historical three-dimensional Gaussian graph is optimized to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area; S6: Acquire historical key frames, wherein the historical key frames include an initial key frame and a previous key frame; Step 701: Randomly disrupt the historical key frames, and use the following formula to iteratively process the first precision three-dimensional Gaussian image by minimizing the photometric rendering loss to obtain an optimized three-dimensional Gaussian rendering depth image; Among them, I and They are the images corresponding to the observed (collected) historical key frames and the images obtained by Gaussian representation of the rendered historical key frames, L D-SSIM To constrain structural consistency for structural similarity, λ refers to the weight, which is set manually, and L refers to the loss; Step S702: Determine whether the number of iterative calculations for minimizing photometric rendering loss is greater than a first preset number of calculations; If yes, then end the iterative calculation process and use the optimized three-dimensional Gaussian rendering depth map as the second-precision three-dimensional Gaussian map corresponding to the newly added area; If not, determining whether the number of iterative calculations for minimizing the photometric rendering loss is equal to a second preset number of calculations; wherein the second preset number of calculations is the number of calculations obtained by dividing the first preset number of calculations according to a preset gradient; If yes, execute step 703; If not, the optimized three-dimensional Gaussian rendering depth map is used as the first-precision three-dimensional Gaussian map for the next round of iterative processing, and step 701 is repeated; Step 703: Calculate the cumulative opacity for each pixel in the optimized 3D Gaussian rendering depth map; If the cumulative opacity of any pixel among the pixels is greater than a second threshold, or the difference between the rendering depth corresponding to the pixel and the laser point cloud depth is greater than a third threshold, the pixel is marked as an unstable pixel; Based on the unstable pixel, determining a three-dimensional Gaussian map corresponding to the unstable pixel; Processing the three-dimensional Gaussian graph using an adaptive densification and pruning strategy to obtain a processed first optimized loss Gaussian graph; The first optimized loss Gaussian map is used as the first precision three-dimensional Gaussian map for the next repeated iterative process of minimizing the photometric rendering loss, and the above steps S701-S703 are repeated.

2. The method according to claim 1, characterized in that The estimated standing posture is obtained from the output of the multi-sensor visual odometry submodule, including: Input the radar point cloud map, current frame and inertial navigation data into the multi-sensor visual odometry submodule; Determining the estimated stance based on the radar point cloud map, the current frame, and the inertial navigation data; The estimated standing posture is output through the multi-sensor visual odometry sub-module.

3. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 2.

Images