Incremental map optimization method based on three-dimensional Gaussian

Abstract

The invention provides an incremental map optimization method based on three-dimensional Gaussian, which comprises the following steps: S1, acquiring a current frame and an estimated standing posture, S2, judging whether the current frame is a key frame; if yes, executing the step S3; s3, rendering to obtain a contour image based on the estimated standing posture; s4, acquiring a historical three-dimensional Gaussian map; based on the contour image and the historical three-dimensional Gaussian map, determining pixels corresponding to a newly added area; s5, optimizing the historical three-dimensional Gaussian image based on the pixels corresponding to the newly-added area to obtain a first-precision three-dimensional Gaussian image corresponding to the newly-added area; and S6, obtaining a historical key frame, and carrying out loss optimization on the first precision three-dimensional Gaussian map based on the historical key frame to obtain a second precision three-dimensional Gaussian map corresponding to the newly added region. The method and the device are used for reducing redundancy, improving efficiency and precision and realizing high-fidelity and high-efficiency rendering in a process of gradually optimizing a map.

Description

Technical Field

[0001] The present application relates to the field of map construction, and in particular to the field of incremental map update optimization. Background Art

[0002] Incremental map update refers to a map update method that updates and modifies only the changed parts based on the existing map data. It identifies and obtains the information of new, deleted or modified elements in the geographic space, such as the construction and widening of roads, the demolition and construction of buildings, etc., and then integrates these change information into the original map data, so as to achieve timely and accurate update of the map with a smaller amount of data update, maintain the consistency and timeliness of the map with the real world, and improve the practicality and reliability of the map.

[0003] However, under existing technical conditions, it is difficult to achieve high-fidelity modeling for small and complex objects during incremental map updates. At the same time, due to computational redundancy, the accuracy and efficiency of incremental map updates are both at a low level. Summary of the invention

[0004] In order to overcome the above technical defects, the present application provides an incremental map optimization method, storage medium and device based on three-dimensional Gaussian.

[0005] To achieve the above objectives, this application is implemented according to the following technical solutions:

[0006] In a first aspect, the present application provides an incremental map optimization method based on three-dimensional Gaussian, which is applied to an incremental mapping submodule. The incremental mapping submodule and the multi-sensor visual odometer submodule constitute a positioning and mapping module, including:

[0007] S1: Obtaining a current frame and an estimated standing posture, wherein the estimated standing posture is obtained by outputting the multi-sensor visual mileage submodule;

[0008] S2: Determine whether the current frame is a key frame;

[0009] If yes, execute step S3;

[0010] S3: Rendering to obtain a contour image based on the estimated standing posture;

[0011] S4: Obtain historical three-dimensional Gaussian graph;

[0012] Determine pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian image;

[0013] S5: Based on the pixels corresponding to the newly added area, the historical three-dimensional Gaussian map is optimized to obtain a first-precision three-dimensional Gaussian map corresponding to the newly added area;

[0014] S6: Acquire historical key frames, wherein the historical key frames include an initial key frame and a previous key frame;

[0015] Based on the historical key frames, the first precision three-dimensional Gaussian map is loss optimized to obtain a second precision three-dimensional Gaussian map corresponding to the newly added area, wherein the first precision is less than the second precision.

[0016] Optionally, determining whether the current frame is a key frame is specifically as follows:

[0017] Determine whether the pixel similarity between the current frame and the previous key frame is greater than a first threshold;

[0018] If so, the current frame is used as a key frame.

[0019] Optionally, rendering to obtain a contour image based on the estimated standing posture includes:

[0020] According to the key frame, determining a current camera viewing angle corresponding to the key frame;

[0021] Based on the current camera viewing angle, the estimated standing posture is rendered to obtain a contour image.

[0022] Optionally, determining pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian image includes:

[0023] Based on the contour image, generating a mask;

[0024] Based on the mask and the historical three-dimensional Gaussian map, matching is performed to determine the pixels corresponding to the newly added area;

[0025] Optionally, the optimizing process of the historical three-dimensional Gaussian graph based on the pixels corresponding to the newly added area to obtain the first-precision three-dimensional Gaussian graph corresponding to the newly added area includes:

[0026] Acquire the newly registered radar points in the time interval between the current frame and the previous key frame;

[0027] Randomly sampling one half of the newly registered radar points;

[0028] Determine the radar points that can be optimized based on the pixels corresponding to the newly added area and one-half of the newly registered radar points randomly sampled;

[0029] Based on the optimizable radar points, determining optimizable Gaussian points;

[0030] Based on the optimizable Gaussian points, the historical three-dimensional Gaussian graph is optimized to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area.

[0031] Optionally, the determining the optimizable radar points based on the pixels corresponding to the newly added area and one-half of the randomly sampled newly registered radar points includes:

[0032] Projecting one-half of the randomly sampled newly registered radar points into pixels corresponding to the newly added area;

[0033] Determine whether one half of the newly registered radar points randomly sampled can be projected into pixels corresponding to the newly added area;

[0034] If so, the radar point that can be projected into the pixel corresponding to the newly added area is used as the optimizable radar point.

[0035] Optionally, the performing loss optimization on the first precision three-dimensional Gaussian graph based on the historical key frame to obtain a second precision three-dimensional Gaussian graph corresponding to the newly added area includes:

[0036] Step 701: randomly disrupt the historical key frames, and use the following formula to iteratively process the first precision three-dimensional Gaussian image by minimizing the photometric rendering loss to obtain an optimized three-dimensional Gaussian rendering depth image;

[0037]

[0038] Among them, I and They are the images corresponding to the observed (collected) historical key frames and the images obtained by Gaussian representation of the rendered historical key frames, L D-SSIM To constrain structural consistency for structural similarity, λ refers to the weight, which is set manually, and L refers to the loss;

[0039] Step S702: determining whether the number of iterative calculations for minimizing photometric rendering loss is greater than a first preset number of calculations;

[0040] If yes, the iterative calculation process is terminated, and the optimized three-dimensional Gaussian rendering depth map is used as the second precision three-dimensional Gaussian map corresponding to the newly added area;

[0041] If not, determining whether the number of iterative calculations for minimizing the photometric rendering loss is equal to a second preset number of calculations; wherein the second preset number of calculations is the number of calculations obtained by dividing the first preset number of calculations according to a preset gradient;

[0042] If yes, execute step 703;

[0043] If not, use the optimized three-dimensional Gaussian rendering depth map as the first-precision three-dimensional Gaussian map for the next round of iterative processing, and repeat step 701;

[0044] Step 703: Obtain an optimized three-dimensional Gaussian rendering depth map;

[0045] For each pixel in the optimized three-dimensional Gaussian rendering depth map, calculating the cumulative opacity;

[0046] If the cumulative opacity of any pixel among the pixels is greater than the second threshold, or the difference between the rendering depth corresponding to the pixel and the laser point cloud depth is greater than the third threshold, the pixel is marked as an unstable pixel;

[0047] Based on the unstable pixel, determining a three-dimensional Gaussian graph corresponding to the unstable pixel;

[0048] The three-dimensional Gaussian graph corresponding to the unstable image is processed using an adaptive densification and pruning strategy to obtain a processed first optimized loss Gaussian graph;

[0049] The processed first optimized loss Gaussian graph is used as the first precision three-dimensional Gaussian graph for the next repeated iterative processing of minimizing the photometric rendering loss, and the above steps S701-S703 are repeated.

[0050] Optionally, the estimated standing posture is obtained by outputting the multi-sensor visual mileage submodule, including:

[0051] Input the radar point cloud map, current frame and inertial navigation data into the multi-sensor visual mileage submodule;

[0052] Determining the estimated standing posture based on the radar point cloud map, the current frame and the inertial navigation data;

[0053] The estimated standing posture is output through the multi-sensor visual mileage sub-module.

[0054] In a second aspect, the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 8.

[0055] In a third aspect, the present application provides a computer-readable storage medium, wherein the readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 8.

[0056] This application has the following beneficial effects:

[0057] In this application, the current frame is judged and registered as a key frame, which reduces the amount of image calculations during the map update process. By introducing three-dimensional Gaussian representation as the basic method for real-time positioning and mapping of large-scale scene modeling, high-fidelity modeling of complex and small objects is used to improve the quality of scene map construction. In parallel with the multi-sensor odometer submodule, the pose tracking results are used and the incremental map optimization method is used to gradually optimize the map, reduce redundancy, improve efficiency and accuracy, and achieve high-fidelity and high-efficiency rendering.

[0058] In addition to the above-described purposes, features and advantages, the present application also has other purposes, features and advantages. The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1 This is a flow chart of the real-time construction method of a three-dimensional scene of the present application.

[0061] Figure 2 It is a schematic diagram of a multi-sensor odometer submodule of an embodiment of the present application.

[0062] Figure 3 It is a flowchart of a three-dimensional Gaussian-based incremental map optimization method provided in this application.

[0063] Figure 4 It is a schematic diagram of the incremental mapping submodule of an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application are described in detail below with reference to the accompanying drawings; however, the present application can be implemented in many different ways as defined and covered by the claims.

[0065] Since this application will mention multiple different maps and some conceptual judgments, in order to avoid misunderstandings, a brief explanation is given here:

[0066] Global LiDAR map: The global LiDAR map is a comprehensive description of the environment constructed through LiDAR data collection. After point cloud registration and feature extraction, it has the characteristics of high precision and rich three-dimensional information. It has important applications in many fields such as autonomous driving and robot navigation.

[0067] 3D Gaussian graph: 3D Gaussian graph is a graph based on 3D space and Gaussian distribution to represent data features. It describes the shape, position and uncertainty of objects in 3D space through Gaussian function and has wide applications in many fields.

[0068] It should be noted that the method in this application is mainly used for real-time construction of three-dimensional scenes of large-scale water surface scenes. Figure 1 As shown in the figure, the real-time construction of the 3D scene is mainly based on the joint calibration module and the positioning real-time mapping module. The functions of the above modules are briefly introduced below:

[0069] The joint calibration module is used to perform high-precision joint calibration of the external parameters of the LiDAR, IMU and camera in the hardware platform, making full use of the high-resolution visual information provided by the camera, the high-precision distance measurement provided by the LiDAR, and the dynamic information such as acceleration and angular velocity provided by the IMU, eliminating the potential impact of the errors and drifts of each sensor on the performance of the system. Through joint calibration, it can ensure that the data of these sensors can be aligned in the same coordinate system, thereby improving the accuracy of data fusion, and by reducing these errors in the data fusion process, the overall stability and reliability of the system can be improved, which plays an important supporting role in subsequent perception, positioning, navigation and decision-making tasks.

[0070] The positioning and real-time mapping module is used for unmanned platforms to use multi-sensor information in large-scale water environment space to efficiently determine their own positions and build real-time dense three-dimensional maps of water scenes with visual appearance and realistic content, providing accurate position information and high-quality dense map information for subsequent tasks such as path planning and autonomous control.

[0071] Since the method of this application is mainly implemented based on the positioning and real-time mapping modules, it is described in detail here:

[0072] The positioning and real-time mapping module includes a multi-sensor visual odometer submodule and an incremental mapping submodule.

[0073] The multi-sensor odometer submodule is a robust pose tracking system that combines camera, inertial navigation and radar. The specific process is as follows Figure 2 As shown. Input the radar point cloud map, current image frame and inertial navigation data, first extract the point cloud SURF feature points and use optical flow tracking to project the map points in the previous frame image to the current frame, then scan and match the radar point cloud with the map, calculate the residuals of the radar, camera and inertial navigation, optimize on the continuous time trajectory represented by non-uniform B-spline, marginalize and update the map points, input the sensor data at the new time point and repeat the above steps, and finally output the continuous time joint optimization trajectory.

[0074] Specifically, the global map is constructed using radar point clouds, and the 3D map points are associated with the pixels in the camera image, avoiding the need for deep optimization of visual features, thereby achieving efficient fusion of heterogeneous radar-inertial-camera data in short-time sliding window optimization. The IMU raw measurements and random walk process are used to construct IMU factors and bias factors. Subsequently, based on the point-surface error from the radar point cloud to the map, the reprojection error from the camera image to the map, and the IMU factors and bias factors, non-uniform B-splines are used for continuous time optimization to output the estimated standing posture of the radar point cloud map. In the optimization process, dynamic changes in motion are considered, and control points are added dynamically and adaptively to improve accuracy and real-time performance.

[0075] That is to say, the multi-sensor odometer submodule processes the input radar point cloud map, the current image frame and the inertial navigation data, and outputs the estimated standing posture.

[0076] The incremental mapping submodule is based on the three-dimensional Gaussian scene representation method. It takes the output of the multi-sensor odometer module to estimate the current frame pose as input, and progressively builds a complete, high-fidelity, and dense three-dimensional scene map during step-by-step optimization.

[0077] In order to solve the problems raised by the above background technology, Figure 3 and Figure 4 The present application proposes an incremental map optimization method based on three-dimensional Gaussian, which is applied to the incremental mapping submodule. The incremental mapping submodule and the multi-sensor visual odometer submodule constitute a positioning and mapping module, including:

[0078] Step S1: obtaining a current frame and an estimated standing posture, wherein the estimated standing posture is obtained by outputting the multi-sensor visual mileage submodule;

[0079] The current frame refers to the data frame corresponding to the sensor data obtained by the system at the current moment. For example, the point cloud data frame scanned by the lidar at a certain moment, or the image data frame taken by the camera at the same moment. These current frame data provide the system with real-time environmental information for decision-making and control.

[0080] Estimated pose refers to the process of estimating the position and posture of an object in space. By acquiring data through various sensors and applying specific algorithms, the orientation, angle and other information of the object relative to the reference system are determined. It has important applications in the fields of robotics, autonomous driving, and surface scene construction. The estimated pose in this application is obtained by the output of the multi-sensor visual mileage submodule, and it is input into the incremental mapping submodule for subsequent calculation and processing. The previous article has described the multi-sensor visual mileage submodule and the process of determining the estimated pose, so it will not be elaborated on in detail here.

[0081] In this application, a set of control strategies are carefully designed to make three-dimensional Gaussian sputtering more suitable for incremental mapping. For the first time step, based on the information input of multiple sensors, for example, the first frame of image, the estimated posture, and the lidar point cloud corresponding to the first frame, it is used as the beginning of constructing the three-dimensional Gaussian map, thereby forming the original three-dimensional Gaussian map. The first frame of image input is used as the key frame by default. Therefore, the three-dimensional Gaussian is initialized using all the radar points included in the lidar point cloud corresponding to the first frame of image in the first time step. Each Gaussian center point is placed at the corresponding radar point position, and its zero-degree spherical harmonics are initialized with the color attributes of the radar point, and a smaller scale is given to the nearby Gaussian, and a larger scale is given to the Gaussian in the principle imaging plane to reduce potential aliasing artifacts. The radius of each Gaussian point projected to the image plane is 2 pixels:

[0082]

[0083] Among them, e is a 3*1 unit vector, d represents the depth of the radar point in the image coordinate system, f is the focal length of the camera, and S is the image plane.

[0084] It should be noted that the above steps are only for generating the original three-dimensional Gaussian image and do not involve other Gaussianization processes.

[0085] After generating the original three-dimensional Gaussian map, the present application will subsequently perform continuous incremental updates and optimizations on the original three-dimensional Gaussian map based on the input information of multiple sensors. It should be noted that each subsequent incremental update and optimization of the three-dimensional Gaussian map is performed on the basis of the last incremental update and optimization of the three-dimensional Gaussian map, not every incremental update and optimization is performed on the basis of the original three-dimensional Gaussian map.

[0086] Step S2: Determine whether the current frame is a key frame, if so, execute step S3;

[0087] For online incremental mapping systems, in order to achieve efficient and real-time map construction, it is not feasible to use all received images for optimization in terms of complex calculations. And in order to obtain a complete environmental map, the Gaussian scene representation should be able to simulate the geometry and appearance of the new observation area. Therefore, the optimization of the three-dimensional scene representation is only done in the key frame. It is not a simple setting of the number of time steps as the key frame division standard, but it fully considers the uncertainty of the unmanned platform's movement and steering changes, and adaptively sets the key frame according to the pixel similarity between the current frame and the previous key frame. Specifically:

[0088] Determine whether the pixel similarity between the current frame and the previous key frame is greater than the first threshold. If so, register the current frame as a key frame. If not, treat the current frame as a normal input frame for simple subsequent processing. The first threshold means that compared with the previous key frame, the current frame is likely to have a new area. Therefore, by setting the first threshold, the current frame can be registered as a key frame in time, and then the incremental map update is completed based on the key frame. The specific value of the first threshold can be set by yourself based on the actual situation on site, the situation of the sensor, or the method of data processing. This article does not impose specific restrictions on this.

[0089] Step S3: Rendering to obtain a contour image based on the estimated standing posture;

[0090] Each frame received after the first frame usually captures the geometry and appearance of the new observation area, but lidar points from different frames may contain repeated or very similar information. To avoid redundancy, after determining that the current frame is a keyframe, we first determine the current camera view corresponding to the keyframe based on the keyframe, and then render the estimated pose from the current camera view to obtain a contour image V.

[0091] Step S4: Obtain historical three-dimensional Gaussian graph;

[0092] Determining pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian image;

[0093] The historical three-dimensional Gaussian graph is a three-dimensional Gaussian graph obtained by performing three-dimensional Gaussian optimization on the historical key frames before the key frame using the method of the present application. For example, if the current frame is the fifth key frame, the historical three-dimensional Gaussian graph is a three-dimensional Gaussian graph obtained by performing three-dimensional Gaussian optimization on the first four key frames using the method of the present application.

[0094] After obtaining the above-mentioned contour image V, a mask M is generated according to the contour image V, and then the mask M is used to perform matching calculations with the three-dimensional Gaussian image constructed by the current Gaussian, so that in the mask, pixels corresponding to unreliable areas that tend to be observed are selected, and pixels corresponding to unreliable areas that tend to be observed are used as pixels corresponding to the newly added areas, so as to perform subsequent calculations.

[0095] The above-mentioned mask is used to match the three-dimensional Gaussian map constructed by the current Gaussian, so that in the mask, the pixels with a value of 0 in the mask are regarded as pixels corresponding to unreliable and tend to observe new areas, and thus are selected.

[0096] Step S5: Based on the pixels corresponding to the newly added area, the historical three-dimensional Gaussian graph is optimized to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area;

[0097] Since this step uses the global lidar point cloud and the newly registered radar points in the global lidar point cloud, it is described in detail:

[0098] In particular, in order to use a radar point cloud with a more accurate geometric prior as a guide point set for three-dimensional Gaussian sputtering, instead of using the usual SFM method, the system corresponding to the present application maintains a global lidar map stored in voxels with a resolution of ten meters to facilitate operation and access to the radar point cloud. After receiving the input frame and the corresponding radar point cloud map, the system randomly samples one-fourth of the radar points, converts them into world coordinates and registers them to the voxels to improve efficiency, ensuring that the distance between radar points in each voxel is greater than five centimeters to reduce redundancy. For successfully registered radar points, they are mapped to the nearest image and the pixel value is queried to obtain its color attribute. It should be noted that the input frame mentioned here is not the key frame mentioned above. When applying the method mentioned in this application for three-dimensional Gaussian optimization, it can be understood as the current frame.

[0099] After determining the pixels corresponding to the newly added area, it is necessary to optimize the historical three-dimensional Gaussian map, that is, it is necessary to add the Gaussian corresponding to the newly added area to the historical three-dimensional Gaussian map. Therefore, after determining the pixels corresponding to the newly added area, project them into the maintained global lidar point cloud, access the newly registered radar points in the time interval between the current frame (new key frame) and the previous key frame, and consider the sparsity of the surface targets. In order to reduce unnecessary memory consumption, half of the radar points are randomly sampled, and then the radar points that can be optimized are determined based on the pixels corresponding to the newly registered radar points and the newly added area of ​​the randomly sampled half. After determining the optimizable radar points, the optimizable Gaussian points are determined based on the optimizable radar points, and then the historical three-dimensional Gaussian map is optimized based on the optimizable Gaussian points to obtain the first-precision three-dimensional Gaussian map corresponding to the newly added area.

[0100] The first precision three-dimensional Gaussian map corresponding to the newly added area can be understood as a low-precision three-dimensional Gaussian map corresponding to the newly added area, because only the three-dimensional Gaussian map corresponding to the newly added area is preliminarily optimized for the historical three-dimensional Gaussian map, so the three-dimensional Gaussian map corresponding to the newly added area obtained at this time has a lower precision. The specific setting of the numerical range of the first precision can be set according to the actual situation on site, and this article does not make specific restrictions on this. For example, the low-precision numerical range can be determined based on the shooting pixels of the camera.

[0101] The radar points that can be optimized are determined based on the pixels corresponding to the newly registered radar points and the newly added areas of the randomly sampled one-half, specifically:

[0102] Project one-half of the newly registered radar points randomly sampled are projected into the pixels corresponding to the newly added area, and determine whether one-half of the newly registered radar points randomly sampled can be projected into the pixels corresponding to the newly added area. As shown in formula (2), only radar points that can be projected onto the selected pixels can be used as optimizable radar points for subsequent Gaussian initialization.

[0103]

[0104] Among them, M is the contour image, V is the mask, and α is the contribution of each Gaussian point to a certain pixel, which is calculated by multiplying the opacity and the probability density function after its three-dimensional Gaussian projection to the imaging plane.

[0105] After determining the radar points that can be optimized, project them into the world coordinate system to determine the Gaussian points that can be optimized, and then in the world coordinate system, according to the Gaussian points that can be optimized, optimize the historical three-dimensional Gaussian graph, so as to complete the addition of the new area and obtain the first-precision three-dimensional Gaussian graph corresponding to the new area. It should be noted that the specific calculation process of the projection mentioned in this paragraph and the specific calculation process of the optimization process are common calculation methods in this field, so the calculation process will not be described in detail.

[0106] S6: Acquire historical key frames, wherein the historical key frames include an initial key frame and a previous key frame;

[0107] Based on the historical key frame, the first precision three-dimensional Gaussian graph is loss optimized to obtain a second precision three-dimensional Gaussian graph corresponding to the newly added area, wherein the first precision is less than the second precision. The historical key frame is stored in the multi-sensor visual odometry submodule.

[0108] The second precision three-dimensional Gaussian map corresponding to the newly added area can be understood as a high-precision three-dimensional map corresponding to the newly added area, because the first precision three-dimensional Gaussian map corresponding to the newly added area has been loss-optimized, so the three-dimensional Gaussian map corresponding to the newly added area obtained at this time has a higher precision. The specific setting of the numerical range of the second precision can be set according to the actual situation on site, and this article does not make specific restrictions on this. For example, the high-precision numerical range can be determined based on the presentation requirements of the Gaussian map.

[0109] In order to further limit the computational complexity to improve efficiency and improve the accuracy of the 3D Gaussian corresponding to the newly added area, in each keyframe mapping process, we select K historical keyframes to optimize the Gaussian mapping, including the initial keyframe and the previous keyframe, to avoid catastrophic forgetting problems and maintain the geometric consistency of the global map. The specific steps are:

[0110] Step 701: randomly disrupt the historical key frames, and use the following formula (3) to iteratively process the first precision Gaussian image by minimizing the photometric rendering loss to obtain an optimized three-dimensional Gaussian rendering depth image:

[0111]

[0112] Among them, I and They are the images corresponding to the observed (collected) historical key frames and the images obtained by Gaussian representation of the rendered historical key frames, L D-SSIM To constrain structural consistency for structural similarity, λ refers to the weight, which is set manually, and L refers to the loss;

[0113] The first precision Gaussian image is iteratively processed to minimize the photometric rendering loss, with the aim of improving image rendering quality, optimizing model parameters, enhancing robustness, achieving accurate photometric estimation, narrowing the difference with the real image, making the generated image more realistic and natural, and allowing the model to better fit the data and adapt to a variety of scenarios.

[0114] Step S702: determining whether the number of iterative calculations for minimizing photometric rendering loss is greater than a first preset number of calculations;

[0115] If yes, the iterative calculation process is terminated, and the optimized three-dimensional Gaussian rendering depth map is used as the second precision three-dimensional Gaussian map corresponding to the newly added area;

[0116] If not, determining whether the number of iterative calculations for minimizing the photometric rendering loss is equal to a second preset number of calculations; wherein the second preset number of calculations is the number of calculations obtained by dividing the first preset number of calculations according to a preset gradient;

[0117] If yes, execute step 703;

[0118] If not, use the optimized three-dimensional Gaussian rendering depth map as the first-precision three-dimensional Gaussian map for the next round of iterative processing, and repeat step 701;

[0119] Since erroneous displacement or deformation may occur in the iterative optimization process of the first-precision 3D Gaussian map through photometric rendering loss, we render the cumulative opacity of pixels to divide the pixels into stable and unstable categories, and use adaptive densification and pruning strategies on the 3D Gaussians corresponding to unstable pixels to promote more refined mapping and lidar to make up for the inability to cover the entire scene, especially in unbounded large-scale water environments, and the possible defects of initialization erroneous 3D Gaussians, and finally realize the construction of continuous-time 3D scene maps in large water scenes.

[0120] When the first-precision three-dimensional Gaussian image is iteratively processed to minimize the photometric rendering loss, the number of iterative calculations will reach a certain number. Then, it is impossible to immediately perform corresponding processing on the optimized three-dimensional Gaussian rendering depth image after each iterative processing. This will increase the calculation time and reduce efficiency. Therefore, we reduce the number of processing calculations by setting a first preset number of calculations and a second preset number of calculations. The second preset number of calculations is the number of calculations obtained after dividing the first preset number of calculations according to the preset gradient. For example, if the first preset number of calculations is 3000, the first preset number of calculations can be set according to the gradient of 100, and the order of the first preset number of calculations can be 100, 200, 300...2900, 3000.

[0121] Then, according to the first preset calculation times and the second preset calculation times, specific steps for optimizing the three-dimensional Gaussian rendering depth map are determined, and the following description is given in order:

[0122] If the number of iterative processing calculations for minimizing the photometric rendering loss (hereinafter simplified as the number of iterative processing calculations for ease of understanding) is greater than the first preset number of calculations, it means that the iterative processing calculations have been completed at this time. At this time, the optimized three-dimensional Gaussian rendering depth map is used as the second precision three-dimensional Gaussian map to complete the optimization of the three-dimensional Gaussian area corresponding to the newly added area.

[0123] If the number of iterative processing calculations is less than or equal to the first preset number of calculations, it means that the optimization processing of the first precision three-dimensional Gaussian image has not been completed at this time. At this time, it is necessary to further determine whether the number of iterative processing calculations meets the second preset number of calculations to determine the subsequent processing of the first precision three-dimensional Gaussian image.

[0124] If the number of iterative processing calculations is equal to the second preset number of calculations, it means that the number of iterative processing calculations is sufficient. The possibility of erroneous displacement or deformation in the optimized three-dimensional Gaussian depth rendering image is greatly increased. At this time, step 703 can be executed.

[0125] If the number of iterative processing calculations is not equal to the second preset number of calculations, it means that the iterative processing of minimizing the photometric rendering loss is still being performed on the first precision three-dimensional Gaussian image using formula (3).

[0126] Step S703: Obtain an optimized three-dimensional Gaussian rendering depth map;

[0127] For each pixel in the optimized three-dimensional Gaussian rendering depth map, calculating the cumulative opacity;

[0128] If the cumulative opacity of any pixel among the pixels is less than the second threshold, or the rendering depth corresponding to the pixel differs from the laser point cloud depth, the pixel is marked as an unstable pixel;

[0129] Based on the unstable pixel, determining a three-dimensional Gaussian graph corresponding to the unstable pixel;

[0130] The three-dimensional Gaussian graph corresponding to the stabilized image is processed using an adaptive densification and pruning strategy to obtain a processed first optimized loss Gaussian graph;

[0131] The processed first optimized loss Gaussian map is used as the first precision three-dimensional Gaussian map for the next repeated iterative processing of minimizing the photometric rendering loss, and the above steps S701-S703 are repeated.

[0132] Since the area formed by the unstable pixels is likely to be used to capture new scene content, there may be omissions in the newly added area, and the unstable adaptive control when using formula (3) for optimization may generate new Gaussians, resulting in poor mapping quality. Therefore, in order to improve the accuracy of the three-dimensional Gaussian map corresponding to the newly added area, it is necessary to optimize the three-dimensional Gaussian area corresponding to the unstable pixels. Specifically:

[0133] After obtaining the optimized three-dimensional Gaussian rendering depth map, first calculate the cumulative opacity for each pixel in the optimized three-dimensional Gaussian rendering depth map. The cumulative opacity can be equivalent to the contour map in form. After the cumulative opacity is calculated, the cumulative opacity of each pixel is judged. If the cumulative opacity of any pixel in each pixel is greater than the second threshold, or the difference between the rendering depth corresponding to any pixel and the laser point cloud is greater than the third threshold, the pixel is marked as an unstable pixel, and then the unstable pixel is back-projected into the world coordinate system to generate a three-dimensional Gaussian map corresponding to the unstable pixel, and then the three-dimensional Gaussian map corresponding to the unstable image is processed using an adaptive densification and pruning strategy to obtain the optimized loss Gaussian map after the first processing. The specific range of the second threshold and the third threshold depends on the specific situation on site, or is set by the on-site technicians. This article does not make specific restrictions here.

[0134] The adaptive densification and pruning strategy includes a densification process and a pruning process. The densification process is to perform a densification operation on the three-dimensional Gaussian map corresponding to the unstable pixel. The pruning process is to check all observable Gaussian points in the current three-dimensional Gaussian map from the current camera cone after adding a new Gaussian, and explicitly reduce the opacity of the Gaussian points far away from the scene surface, so that such Gaussian points are periodically and automatically eliminated in the subsequent optimization process, thereby obtaining the processed first optimization loss Gaussian map.

[0135] After obtaining the processed first optimized loss Gaussian map, the iterative processing is not completed yet. Therefore, the processed first optimized loss Gaussian map is used as the first precision three-dimensional Gaussian map for the next repeated iterative processing of minimizing the photometric rendering loss. The above steps S701-S703 are repeated to continue the iterative processing to achieve periodic optimization of the first precision three-dimensional Gaussian map.

[0136] In summary, this application reduces the amount of image calculations during map updates by registering the current frame as a key frame after judging it. By introducing three-dimensional Gaussian representation as the basic method for real-time positioning and mapping of large-scale scene modeling, high-fidelity modeling of complex and small objects is used to improve the quality of scene map construction. In parallel with the multi-sensor odometer submodule, the pose tracking results are used and the incremental map optimization method is used to gradually optimize the map, reduce redundancy, improve efficiency and accuracy, and achieve high-fidelity and high-efficiency rendering.

[0137] In some embodiments, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor implements the three-dimensional Gaussian-based incremental map optimization method provided in the first aspect.

[0138] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional blocks / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0139] As is known to those of ordinary skill in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0140] Exemplarily, the computer-readable storage medium may be an internal storage unit of the network management device described in the foregoing embodiment, such as a hard disk or memory of the network management device. The computer-readable storage medium may also be an external storage device of the network management device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the network management device.

[0141] In some embodiments, a device is provided, including a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute the computer program and implement the three-dimensional Gaussian-based incremental map optimization method provided in the first aspect of the present application when executing the computer program.

[0142] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0143] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional Gaussian-based incremental map optimization method, applied to an incremental mapping submodule, wherein the incremental mapping submodule and a multi-sensor visual odometer submodule constitute a positioning mapping module, characterized in that: include: S1: Obtaining a current frame and an estimated standing posture, wherein the estimated standing posture is obtained by outputting the multi-sensor visual mileage submodule; S2: Determine whether the current frame is a key frame; If yes, execute step S3; S3: Rendering to obtain a contour image based on the estimated standing posture; S4: Obtain historical three-dimensional Gaussian graph; Determine pixels corresponding to the newly added area based on the contour image and the historical three-dimensional Gaussian image; S5: Based on the pixels corresponding to the newly added area, the historical three-dimensional Gaussian map is optimized to obtain a first-precision three-dimensional Gaussian map corresponding to the newly added area; S6: Acquire historical key frames, wherein the historical key frames include an initial key frame and a previous key frame; Based on the historical key frames, the first precision three-dimensional Gaussian map is loss optimized to obtain a second precision three-dimensional Gaussian map corresponding to the newly added area, wherein the first precision is less than the second precision.

2. The method according to claim 1, characterized in that Determine whether the current frame is a key frame, specifically: Determine whether the pixel similarity between the current frame and the previous key frame is greater than a first threshold; If so, the current frame is used as a key frame.

3. The method according to claim 2, characterized in that The rendering to obtain a contour image based on the estimated standing posture includes: According to the key frame, determining a current camera viewing angle corresponding to the key frame; Based on the current camera viewing angle, the estimated standing posture is rendered to obtain a contour image.

4. The method according to claim 3, characterized in that The determining, based on the contour image and the historical three-dimensional Gaussian graph, pixels corresponding to the newly added area includes: Based on the contour image, generating a mask; Based on matching the mask and the historical three-dimensional Gaussian graph, pixels corresponding to the newly added area are determined.

5. The method according to claim 4, characterized in that The optimizing process of the historical three-dimensional Gaussian graph based on the pixels corresponding to the newly added area to obtain the first-precision three-dimensional Gaussian graph corresponding to the newly added area includes: Acquire the newly registered radar points in the time interval between the current frame and the previous key frame; Randomly sampling one half of the newly registered radar points; Determine the radar points that can be optimized based on the pixels corresponding to the newly added area and one-half of the newly registered radar points randomly sampled; Based on the optimizable radar points, determining optimizable Gaussian points; Based on the optimizable Gaussian points, the historical three-dimensional Gaussian graph is optimized to obtain a first-precision three-dimensional Gaussian graph corresponding to the newly added area.

6. The method according to claim 5, characterized in that The determining of the optimizable radar points based on the pixels corresponding to the newly added area and one-half of the randomly sampled newly registered radar points includes: Projecting one-half of the randomly sampled newly registered radar points into pixels corresponding to the newly added area; Determine whether one half of the newly registered radar points randomly sampled can be projected into pixels corresponding to the newly added area; If so, the radar point that can be projected into the pixel corresponding to the newly added area is used as the optimizable radar point.

7. The method according to claim 6, characterized in that The step of performing loss optimization on the first precision three-dimensional Gaussian graph based on the historical key frame to obtain a second precision three-dimensional Gaussian graph corresponding to the newly added area includes: Step 701: randomly disrupt the historical key frames, and use the following formula to iteratively process the first precision three-dimensional Gaussian image by minimizing the photometric rendering loss to obtain an optimized three-dimensional Gaussian rendering depth image; Among them, I and They are the images corresponding to the observed (collected) historical key frames and the images obtained by Gaussian representation of the rendered historical key frames, L D-SSIM To constrain structural consistency for structural similarity, λ refers to the weight, which is set manually, and L refers to the loss; Step S702: determining whether the number of iterative calculations for minimizing photometric rendering loss is greater than a first preset number of calculations; If yes, the iterative calculation process is terminated, and the optimized three-dimensional Gaussian rendering depth map is used as the second precision three-dimensional Gaussian map corresponding to the newly added area; If not, determining whether the number of iterative calculations for minimizing the photometric rendering loss is equal to a second preset number of calculations; The second preset number of calculations is the number of calculations obtained by dividing the first preset number of calculations according to a preset gradient; If yes, execute step 703; If not, use the optimized three-dimensional Gaussian rendering depth map as the first-precision three-dimensional Gaussian map for the next round of iterative processing, and repeat step 701; Step 703: Obtain an optimized three-dimensional Gaussian rendering depth map; For each pixel in the optimized three-dimensional Gaussian rendering depth map, calculating the cumulative opacity; If the cumulative opacity of any pixel among the pixels is greater than the second threshold, or the difference between the rendering depth corresponding to the pixel and the laser point cloud depth is greater than the third threshold, the pixel is marked as an unstable pixel; Based on the unstable pixel, determining a three-dimensional Gaussian graph corresponding to the unstable pixel; The three-dimensional Gaussian graph corresponding to the unstable image is processed using an adaptive densification and pruning strategy to obtain a processed first optimized loss Gaussian graph; The processed first optimized loss Gaussian graph is used as the first precision three-dimensional Gaussian graph for the next repeated iterative processing of minimizing the photometric rendering loss, and the above steps S701-S703 are repeated.

8. The method according to claim 1, characterized in that The estimated standing posture is obtained by outputting the multi-sensor visual mileage submodule, including: Input the radar point cloud map, current frame and inertial navigation data into the multi-sensor visual mileage submodule; Determining the estimated standing posture based on the radar point cloud map, the current frame and the inertial navigation data; The estimated standing posture is output through the multi-sensor visual mileage sub-module.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the three-dimensional Gaussian-based incremental map optimization method as described in any one of claims 1 to 8.

Images