Real-time positioning and mapping method and device based on two-dimensional Gaussian splashing

Abstract

The invention relates to the technical field of computer graphics and computer vision, in particular to a real-time positioning and mapping method and device based on two-dimensional Gaussian splashing, and the method comprises the steps: carrying out the two-dimensional Gaussian splashing scene representation of a current visual angle environment image collected by a target image collection device, so as to obtain a current Gaussian map; calculating the depth distortion measure of the current Gaussian map so as to carry out adaptive surface reconstruction on the current Gaussian map; calculating the current radial gradient of the reconstructed Gaussian map so as to adjust the current pose of the target image acquisition equipment; and acquiring a new-view-angle environment image by using the target image acquisition equipment with the adjusted pose, and iteratively executing the process until the loss function is converged, thereby obtaining a final geometrical relationship between the target image acquisition equipment and the acquired environment. Therefore, the problems that an existing SLAM method cannot fully capture geometric details and normal vector information, and consequently reconstruction results are poor in the aspects of detail representation and geometric alignment are solved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer graphics and computer vision, and particularly relates to a real-time positioning and mapping method and device based on two-dimensional Gaussian splashing. Background Art

[0002] As an important research direction in the fields of computer vision and robot navigation, Simultaneous Localization and Mapping (SLAM) shows great potential in a wide range of applications such as autonomous driving, unmanned aerial vehicles, virtual reality, and augmented reality.

[0003] Traditional SLAM methods usually rely on sparse feature point matching or dense point clouds, meshes, voxels and other representation methods to achieve environmental mapping, but these methods often have problems such as large computational complexity, sparse scene representation, and insufficient reconstruction accuracy in large-scale complex scenes.

[0004] During the pose tracking process of the SLAM system, due to factors such as complex scenes, occlusions, and excessive movement amplitudes between adjacent frames, feature matching fails, thus affecting the accuracy of pose estimation. In addition, due to the optimization algorithm falling into a local optimal solution, the pose estimation deviates from the true value, resulting in a slow convergence speed or inaccurate pose update, and even an extreme situation of pose failure. As time goes by, errors gradually accumulate, causing the tracking trajectory of the SLAM system to drift or even completely fail.

[0005] Although the SLAM method based on deep learning has improved performance to a certain extent, it has a strong dependence on data, requires a large amount of labeled data for training, and may face the problem of insufficient generalization ability in practical applications. In addition, its computational complexity is relatively high, and the demand for hardware resources is large, which limits its wide application in scenarios with high real-time requirements or resource constraints.

[0006] With the proposal of Neural Radiance Fields (NeRF), high-quality novel view rendering has been achieved through continuous scene representation methods. However, NeRF-based SLAM systems face problems of large computational complexity and low efficiency during the optimization process, making it difficult to meet the requirements of real-time updates. At the same time, 3D Gaussian Splatting (3DGS) has gradually become a research hotspot due to its fast rendering ability and intuitive geometric expression advantages. Researchers have proposed various visual SLAM methods based on 3DGS, such as SplaTAM, MonoGS, RTG-SLAM, etc., achieving real-time and highly accurate visual SLAM systems. However, existing 3DGS-based SLAM methods mainly focus on the fitting of image information, lacking in the expression of depth information, edge details, and surface normal vectors of objects in the scene. There are problems of blurring and distortion during scene depth rendering, unable to fully capture the geometric details and normal vector information of the real object surface, resulting in deficiencies in the detail performance and geometric alignment of the reconstruction results. At the same time, for edge regions or local regions with obvious geometric changes, it is difficult for existing methods to keep the details clear when fusing multi-view information, resulting in discontinuous or blurred details in the reconstruction results. Summary of the Invention

[0007] The present invention provides a real-time positioning and mapping method and device based on 2D Gaussian splatting to solve the problems that existing SLAM methods have blurring and distortion during scene depth rendering, unable to fully capture the geometric details and normal vector information of the real object surface, which affect subsequent pose estimation and map construction, etc.

[0008] In a first aspect embodiment of the present invention, a real-time positioning and mapping method based on 2D Gaussian splatting is provided, including the following steps: performing 2D Gaussian splatting scene representation on the current view environment image collected by the target image acquisition device to obtain the current Gaussian map; calculating the depth distortion metric of the current Gaussian map, and adaptively reconstructing the surface of the current Gaussian map according to the depth distortion metric to obtain the reconstructed Gaussian map; calculating the current radial gradient of the reconstructed Gaussian map, and adjusting the current pose of the target image acquisition device according to the current radial gradient; using the target image acquisition device with the adjusted pose to collect a new view environment image, and iteratively executing the foregoing 2D Gaussian splatting scene representation, adaptive surface reconstruction, and pose adjustment processes until the pre-constructed mapping and pose optimization loss function converges, to obtain the final geometric relationship between the target image acquisition device and the collected environment.

[0009] Optionally, the performing 2D Gaussian splatting scene representation on the current view environment image collected by the target image acquisition device to obtain the current Gaussian map includes:

[0010] Based on two-dimensional Gaussian splashing, each two-dimensional Gaussian patch of the current perspective environment image is defined, and each defined two-dimensional Gaussian patch is rendered to obtain each rendered two-dimensional Gaussian patch; the rendered two-dimensional Gaussian patches are projected and aligned to obtain an initial Gaussian map; the opacity of each two-dimensional Gaussian patch and the error between the depth-rendered image and the current perspective environment image are calculated respectively; the initial Gaussian map is adjusted by adding or removing two-dimensional Gaussian patches according to the opacity of each two-dimensional Gaussian patch and the error to obtain the current Gaussian map.

[0011] Optionally, calculating a depth distortion metric of the current Gaussian map to adaptively reconstruct the surface of the current Gaussian map according to the depth distortion metric to obtain a reconstructed Gaussian map includes:

[0012] Calculating a depth distortion metric of the current Gaussian map to determine high-distortion regions on the current Gaussian map; selecting the patch with the largest transparency blending weight as the dominant patch in the high-distortion regions, and extracting depth information and normal information from the dominant patch; replacing the rendering result of the high-distortion regions with the depth information and normal information from the dominant patch to adaptively reconstruct the surface of the current Gaussian map to obtain the reconstructed Gaussian map.

[0013] Optionally, calculating a current radial gradient of the reconstructed Gaussian map to adjust the current pose of the target image acquisition device according to the current radial gradient includes:

[0014] Calculating the current radial gradient of the reconstructed Gaussian map through a Lie algebra framework; adjusting the current pose of the target image acquisition device according to the current radial gradient so that the adjusted target image acquisition device is aligned with the surface normal direction of the reconstructed Gaussian map.

[0015] In the second aspect of the embodiments of the present invention, a real-time positioning and mapping device based on two-dimensional Gaussian splashing is provided, including: a rendering module for performing a two-dimensional Gaussian splashing scene representation on the current perspective environment image collected by the target image acquisition device to obtain the current Gaussian map; a reconstructed map module for calculating the depth distortion metric of the current Gaussian map and adaptively reconstructing the surface of the current Gaussian map according to the depth distortion metric to obtain the reconstructed Gaussian map; an adjustment module for calculating the current radial gradient of the reconstructed Gaussian map and adjusting the current pose of the target image acquisition device according to the current radial gradient; an iterative positioning module for using the target image acquisition device with the adjusted pose to collect a new perspective environment image and iteratively executing the foregoing two-dimensional Gaussian splashing scene representation, adaptive surface reconstruction, and pose adjustment processes until the pre-constructed mapping and pose optimization loss function converges to obtain the final geometric relationship between the target image acquisition device and the acquired environment.

[0016] Optionally, the rendering module includes:

[0017] A rendering unit for defining each two-dimensional Gaussian bin of the current perspective environment image based on two-dimensional Gaussian splashing and rendering each two-dimensional Gaussian bin to obtain each rendered two-dimensional Gaussian bin;

[0018] An alignment unit for projecting and aligning each rendered two-dimensional Gaussian bin to obtain an initial Gaussian map;

[0019] A first calculation unit for respectively calculating the opacity of each two-dimensional Gaussian bin and the error between the depth-rendered image and the current perspective environment image;

[0020] Increasing or decreasing the two-dimensional Gaussian bins of the initial Gaussian map according to the opacity of each two-dimensional Gaussian bin and the error to obtain the current Gaussian map.

[0021] Optionally, the reconstructed map module includes:

[0022] A second calculation unit for calculating the depth distortion metric of the current Gaussian map to determine the high-distortion regions on the current Gaussian map;

[0023] An extraction unit for selecting the bin with the largest transparency blending weight as the dominant bin in the high-distortion regions and extracting the depth information and normal information in the dominant bin;

[0024] A reconstruction unit for replacing the rendering result of the high-distortion regions with the depth information and normal information in the dominant bin to perform adaptive surface reconstruction on the current Gaussian map to obtain the reconstructed Gaussian map.

[0025] Optionally, the adjustment module includes:

[0026] A third calculation unit, configured to calculate a current radial gradient of the reconstructed Gaussian map through a Lie algebra framework;

[0027] An adjustment unit, configured to adjust a current pose of the target image acquisition device according to the current radial gradient, so that the adjusted target image acquisition device is aligned with a surface normal direction of the reconstructed Gaussian map.

[0028] An embodiment of the third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the real-time positioning and mapping method based on two-dimensional Gaussian splash as described in the above embodiments.

[0029] An embodiment of the fourth aspect of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the program is executed by a processor, the real-time positioning and mapping method based on two-dimensional Gaussian splash as described above is implemented.

[0030] The real-time positioning and mapping method and device based on two-dimensional Gaussian splash proposed in the embodiments of the present invention utilize two-dimensional Gaussian construction to align with a real surface height, and through fusing color, depth, and normal vector information, realize forward rendering of a high-precision scene and map construction; by introducing a Gaussian distortion metric to evaluate the uncertainty of a pixel region, directly adopt the two-dimensional Gaussian information with the largest weight for reconstruction in a high-uncertainty region, thereby significantly improving the accuracy of edge and detail geometric representations; based on Lie groups and Lie algebras, derive a key gradient expression for two-dimensional Gaussian pose optimization, and combine a pose estimation method of rendering error and local geometric information, effectively improving the high robustness and convergence speed of camera tracking.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0033] Figure 1 is a flowchart of a real-time positioning and mapping method based on two-dimensional Gaussian splash provided by an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of an overall framework of a real-time positioning and mapping method based on two-dimensional Gaussian splash provided by an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of an adaptive surface reconstruction method provided by an embodiment of the present invention;

[0036] Figure 4 Schematic diagram for comparing the depth rendering results of the images of the embodiment of the present invention and the prior art;

[0037] Figure 5 Schematic diagram for comparing the surface details of the reconstructed scenes of the embodiment of the present invention and the prior art;

[0038] Figure 6 Schematic diagram of the convergence performance of the pose estimation algorithm provided by the embodiment of the present invention in extreme cases;

[0039] Figure 7 Schematic block diagram of a real-time positioning and mapping device based on two-dimensional Gaussian splashing provided by an embodiment of the present invention;

[0040] Figure 8 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0041] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0042] The real-time positioning and mapping method and device based on two-dimensional Gaussian splashing according to the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0043] Figure 1 Schematic flowchart of a real-time positioning and mapping method based on two-dimensional Gaussian splashing provided by an embodiment of the present invention.

[0044] As Figure 1 shown, the real-time positioning and mapping method based on two-dimensional Gaussian splashing includes the following steps:

[0045] In step S101, a two-dimensional Gaussian splashing scene representation is performed on the current perspective environment image collected by the target image acquisition device to obtain the current Gaussian map.

[0046] In some embodiments, performing a two-dimensional Gaussian splashing scene representation on the current perspective environment image collected by the target image acquisition device to obtain the current Gaussian map includes:

[0047] Based on two-dimensional Gaussian splatting, each two-dimensional Gaussian surface element of the current view environment image is defined, and each two-dimensional Gaussian surface element is rendered to obtain each rendered two-dimensional Gaussian surface element;

[0048] The rendered two-dimensional Gaussian surface elements are projected and aligned to obtain an initial Gaussian map;

[0049] The opacity, and the error between the depth-rendered image and the current view environment image of each two-dimensional Gaussian surface element are calculated respectively;

[0050] Based on the opacity and error of each two-dimensional Gaussian surface element, the two-dimensional Gaussian surface elements of the initial Gaussian map are increased or decreased to obtain the current Gaussian map.

[0051] As Figure 2 shown, in the actual execution process, based on two-dimensional Gaussian splatting, each two-dimensional Gaussian surface element (surfel) of the current view environment image is defined by the center point p k , the tangent vector t u , t v , the scaling factor s u , s v and the normal vector t w =t u ×t v is defined to achieve surface geometry alignment, and the color c k and the transparency α k of each surface element are defined for rendering; the color, depth, and normal are rendered through alpha blending to ensure multi-view geometric consistency, thereby obtaining an initial Gaussian map. The specific definition formula is as follows:

[0052]

[0053] Among them, ω i is the transparency blending weight of the surface element, α i is the opacity of a single surface element, and the opacity weight of a single surface element is calculated by weighting according to the front-back arrangement order under the current view. G i (u(x)) represents the Gaussian distribution probability value corresponding to the projection coordinates, u(x) is the local coordinate where the image coordinates are projected onto the surface element, C(x), D(x), and N(x) respectively represent the rendered image, depth map, and normal map, ω k is the opacity blending weight of the surface element, c k is the color of a single surface element, z k is the depth value of the pixel projected onto the surface element, and t w is the normal vector of a single surface element.

[0054] Further, calculate the opacity of each two-dimensional Gaussian surface element, the error between the depth-rendered image and the current perspective environment image, and add or delete Gaussian surface elements according to the opacity and error of the current frame. For pixels in the current frame with transparency greater than 0.5, depth or color error greater than the threshold (δ C = δ D = 0.1), new surface elements are generated, and surface elements with cumulative error exceeding twice the threshold are removed to obtain the current Gaussian map.

[0055] In addition, in the process of the embodiment of the present invention, a mapping and mapping and pose optimization loss function is also defined. By calculating the loss function of the real frame and the rendering result, the Jacobian matrix of the Gaussian parameters and the pose is obtained, and then the scene model (i.e., the reconstructed Gaussian map) and the current frame pose (i.e., the final geometric relationship between the target image acquisition device and the acquired environment) can be optimized. Among them, the specific expressions of the mapping loss function and the mapping and pose optimization loss function are:

[0056]

[0057] Among them, L map is the mapping loss function, C t is the Gaussian-rendered image, is the real image, γ D is the depth loss weight, D t is the Gaussian-rendered depth, is the real depth.

[0058] In step S102, calculate the depth distortion metric of the current Gaussian map to adaptively reconstruct the surface of the current Gaussian map based on the depth distortion metric to obtain the reconstructed Gaussian map.

[0059] In some embodiments, calculating the depth distortion metric of the current Gaussian map to adaptively reconstruct the surface of the current Gaussian map based on the depth distortion metric to obtain the reconstructed Gaussian map includes:

[0060] Calculate the depth distortion metric of the current Gaussian map to determine the high-distortion regions on the current Gaussian map;

[0061] Select the surface element with the largest transparency blending weight as the dominant surface element in the high-distortion region, and extract the depth information and normal information in the dominant surface element;

[0062] Replace the rendering result of the high-distortion region with the depth information and normal information in the dominant surface element to adaptively reconstruct the surface of the current Gaussian map to obtain the reconstructed Gaussian map.

[0063] As Figure 2As shown, in the actual execution process, in a traditional SLAM system, due to limited perspective or insufficient number of surface elements, depth and normal rendering with hybrid weight averaging easily lead to geometric blurring (such as loss of edge details and surface degradation). To solve this problem, the embodiments of the present invention propose a depth distortion metric (Depth Distortion) and an adaptive geometric rendering strategy based on the distortion degree. The specific steps are as follows:

[0064] Define a pixel-level distortion metric:

[0065]

[0066] where, is the depth distortion value, z i is the depth of the intersection of the ray and the i-th surface element, z j is the depth of the intersection of the ray and the j-th surface element, ω i is the transparency mixing weight of the surface element. This formula quantifies the depth inconsistency when the ray of the same pixel passes through multiple surface elements. A high distortion value indicates regions of edges, occlusions, or under-optimized areas.

[0067] As Figure 3 shown, for high-distortion regions, discard the hybrid weight averaging and select the surface element with the largest hybrid weight as the dominant surface element (Dominant Surfel), that is, select the dominant surface element instead of the average depth as the geometric representation of the scene, thereby avoiding geometric blurring caused by averaging and significantly improving edge sharpness.

[0068] Furthermore, to ensure the integrity of the planar region, only when the hybrid depth D(x)>D c (x), replace the rendering result of the high-distortion region with the depth and normal of the dominant surface element to perform adaptive surface reconstruction on the current Gaussian map and obtain the reconstructed Gaussian map. This mechanism prevents holes or normal jumps from occurring on large planes while retaining details. Among them, the formulas for depth and normal are as follows:

[0069]

[0070] where, D c (x) is the Gaussian rendering depth, ω k is the opacity mixing weight of the surface element, z k is the depth value of the pixel projected onto the surface element, N c (x) is the Gaussian rendering normal vector, t w is the normal vector of a single surface element.

[0071] In step S103, calculate the current radial gradient of the reconstructed Gaussian map to adjust the current pose of the target image acquisition device according to the current radial gradient.

[0072] In some embodiments, calculating the current radial gradient of the reconstructed Gaussian map to adjust the current pose of the target image acquisition device according to the current radial gradient includes:

[0073] Calculating the current radial gradient of the reconstructed Gaussian map through the Lie algebra framework;

[0074] Adjusting the current pose of the target image acquisition device according to the current radial gradient so that the adjusted target image acquisition device is aligned with the surface normal direction of the reconstructed Gaussian map.

[0075] As Figure 2 shown, in the actual execution process, traditional 3D Gaussian splash pose estimation relies on color gradients and insufficiently utilizes geometric constraints, resulting in tracking failure under large-angle movements. Therefore, the embodiments of the present invention jointly optimize color and depth consistency according to the mapping loss function and the mapping and pose optimization loss function. To ensure convergence, the initial pose is set to the estimated pose of the previous frame to calculate the pose parameters of the target image acquisition device, and the pose parameters include the rotation parameter θ and the translation parameter t.

[0076] Calculating the radial gradient of the reconstructed Gaussian map through the Lie algebra framework:

[0077]

[0078] where represents the radial vector from the center of the surface element to the ray intersection point, R is the rotation matrix in the current view, u is the coordinate corresponding to the tangent vector s u direction, t u is the scaling coefficient corresponding to the tangent vector s u direction, v is the coordinate corresponding to the tangent vector s v direction, t v is the scaling coefficient corresponding to the tangent vector s v direction. This radial gradient directly reflects the geometric relationship between the local plane of the surface element in the reconstructed Gaussian map and the target image acquisition device. By adjusting the rotation parameter θ, the attitude of the image acquisition device is pushed towards alignment with the surface normal direction. In the depth map loss function, the radial gradient is unique to the two-dimensional Gaussian, and at the same time, the simplified form of the Jacobian matrix that propagates the loss function (through the vector) to the rotation parameter is given:

[0079]

[0080] To ensure the running speed, the derivative of the pose parameters is implemented using CUDA based on two-dimensional Gaussian rendering.

[0081] In step S104, the target image acquisition device with adjusted pose is used to acquire new perspective environment images, and the foregoing two-dimensional Gaussian splash scene representation, adaptive surface reconstruction, and pose adjustment processes are iteratively executed until the pre-constructed mapping and pose optimization loss function converges, and the final geometric relationship between the target image acquisition device and the acquired environment is obtained.

[0082] As Figure 2 shown, during the actual execution process, according to the relative movement between the target image acquisition device with adjusted pose and the acquired environment, when the rotation angle is greater than the preset angle threshold or the translation distance is greater than the preset distance threshold (for example, rotation angle > 30° or translation distance > 0.3 m), the new perspective environment image acquired by the target image acquisition device with adjusted pose is determined as a key frame, and steps S101 - S104 are iteratively executed, and the key frame is fused with all the previous key frame data, so as to optimize the reconstructed Gaussian map and the current pose of the target image acquisition device until the mapping and pose optimization loss function converges, and the final Gaussian map of the acquired environment and the final geometric relationship between the target image acquisition device and the acquired environment are obtained.

[0083] It should be noted that the embodiment of the present invention mainly adopts a pose tracking strategy that combines the main mode and the auxiliary mode. Among them, the main mode is based on surface element gradient optimization, uses radial gradient to enhance rotation convergence, and adapts to violent movements (such as scenes with too large rotation angles between adjacent frames in some datasets). The auxiliary mode refers to the Plane-to-plane Iterative Closest Point algorithm, which combines rendered depth and normal map to improve the computational efficiency of low-motion scenes.

[0084] The real-time positioning and mapping method based on two-dimensional Gaussian splash proposed by the embodiment of the present invention will be further described below through a specific embodiment.

[0085] When evaluating the quality of the scene reconstruction map generated by the embodiment of the present invention and the accuracy of camera pose estimation, the present invention is compared with existing visual SLAM technologies such as those based on neural radiance fields and three-dimensional Gaussian splashes. The existing methods for comparison include NICE-SLAM, Point-SLAM, SplaTAM, MonoGS, RTG-SLAM, and ORB-SLAM3. The datasets used include the Replica dataset for testing scene reconstruction accuracy and the ScanNet++ dataset for testing pose estimation accuracy.

[0086] For the scene reconstruction accuracy, several metrics proposed in iMAP are selected as evaluation metrics in the embodiments of the present invention, including accuracy (Acc, cm), precision (P, %), completeness (Comp, cm), recall (R, %), F1 score, and L1 average depth error (cm). Among them, accuracy is defined as the average distance error of mapping the reconstructed point cloud to the real point cloud, completeness represents the average distance from the real point cloud to the reconstructed point cloud, while precision and recall are respectively the ratios of point clouds with two distances less than 0.05 cm. F1 is the harmonic mean of precision and recall. The comparison results of the scene reconstruction between the embodiments of the present invention and existing SLAM methods are shown in the following table (bold represents the best):

[0087]

[0088] It can be seen that the present invention is superior to the existing 3DGS-based SLAM methods in scene reconstruction, and the effect is comparable to the existing best NeRF-based SLAM method. However, the NeRF-based SLAM method consumes more time in the optimization and rendering stages and cannot achieve real-time performance. In the experimental results, the completeness of the present invention is 2.19 cm and the recall is 88.6%, which are higher than all existing methods.

[0089] Some examples of the specific generation results of the present invention are as Figure 4 shown, including visual images and depth maps. It can be seen that the images generated by the present invention have higher quality and more accurate scene depth estimation. Some examples of scene reconstruction are as Figure 5 shown. The scene reconstructed in the embodiments of the present invention has a smoother surface and retains more geometric details of the objects.

[0090] For the evaluation of the pose estimation effect, the present invention selects three representative camera sequences (S0, S1, S2) on the ScanNet++ dataset, and tests the pose estimation results of the truncated sequence (S1*) and the non-truncated sequence (S1) for S1. The absolute trajectory error (ATE) is selected as the evaluation metric for pose estimation. The comparison results of the pose estimation between the present invention and existing SLAM methods are shown in the following table (X represents tracking failure and excessive error):

[0091] S0 S1* S1 S2 ORB-SLAM3 X X X X Point-SLAM X X X X MonoGS X X X X SplaTAM 0.63 1.90 X X The present invention 1.37 1.45 1.69 3.97

[0092] Among them, both S1 and S2 include adjacent frames with drastic relative pose changes. It can be seen that both the traditional method and the rendering-based method fail to track the camera pose in some of the sequences, while the present invention based on two-dimensional Gaussian splash optimization pose can converge on those adjacent frames with drastic changes, thereby successfully locating the camera pose in these sequences while maintaining good accuracy. The performance of the present invention on the large span of adjacent frames in S1 is shown in Figure 2. Figure 6 shown.

[0093] In summary, the real-time positioning and mapping method based on two-dimensional Gaussian splashing proposed in the embodiment of the present invention includes the following:

[0094] Beneficial effects:

[0095] (1) The two-dimensional Gaussian splash model of the scene can be reconstructed while the camera is moving, and the camera positioning can be realized based on the two-dimensional Gaussian.

[0096] (2) The uncertainty of the pixel area is evaluated by introducing Gaussian distortion measurement. In the high uncertainty area, the two-dimensional Gaussian information with the largest weight is directly used for reconstruction to ensure that the local geometric information in the overall map is strictly corresponding and the details are faithful, thereby improving the overall geometric reconstruction accuracy of the scene;

[0097] (3) Based on Lie groups and Lie algebras, the gradient information back-propagated during the two-dimensional Gaussian rendering process is used to derive the gradient expression of the rotation component. Combined with the pose estimation method based on rendering error and local geometric information, rapid convergence of pose updates is achieved, ensuring stable tracking even in scenes with restricted viewing angles.

[0098] Next, a real-time positioning and mapping device based on two-dimensional Gaussian splashing according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0099] Figure 7 4 is a block diagram of a real-time positioning and mapping device based on two-dimensional Gaussian splashing according to an embodiment of the present invention.

[0100] like Figure 7 As shown, the real-time positioning and mapping device 70 based on two-dimensional Gaussian splashing includes: a rendering module 701, a map reconstruction module 702, an adjustment module 703 and an iterative positioning module 704.

[0101] Among them, the rendering module 701 is used to perform a two-dimensional Gaussian splash scene representation on the current perspective environment image collected by the target image acquisition device to obtain the current Gaussian map. The map reconstruction module 702 is used to calculate the depth distortion metric of the current Gaussian map, and adaptively reconstruct the surface of the current Gaussian map according to the depth distortion metric to obtain the reconstructed Gaussian map. The adjustment module 703 is used to calculate the current radial gradient of the reconstructed Gaussian map, and adjust the current pose of the target image acquisition device according to the current radial gradient. The iterative positioning module 704 is used to collect a new perspective environment image by using the target image acquisition device with the adjusted pose, and iteratively execute the foregoing two-dimensional Gaussian splash scene representation, adaptive surface reconstruction, and pose adjustment processes until the pre-constructed mapping and pose optimization loss function converges, so as to obtain the final geometric relationship between the target image acquisition device and the environment to be collected.

[0102] In some embodiments, the rendering module 701 includes:

[0103] A rendering unit, configured to define each two-dimensional Gaussian bin of the current perspective environment image based on two-dimensional Gaussian splash, and render each two-dimensional Gaussian bin to obtain each rendered two-dimensional Gaussian bin;

[0104] An alignment unit, configured to project-align each rendered two-dimensional Gaussian bin to obtain an initial Gaussian map;

[0105] A first calculation unit, configured to calculate the opacity of each two-dimensional Gaussian bin and the error between the depth-rendered image and the current perspective environment image respectively;

[0106] Increase or decrease the two-dimensional Gaussian bins of the initial Gaussian map according to the opacity and error of each two-dimensional Gaussian bin to obtain the current Gaussian map.

[0107] In some embodiments, the map reconstruction module 702 includes:

[0108] A second calculation unit, configured to calculate the depth distortion metric of the current Gaussian map to determine the high-distortion area on the current Gaussian map;

[0109] An extraction unit, configured to select the bin with the largest transparency mixing weight in the high-distortion area as the dominant bin, and extract the depth information and normal information in the dominant bin;

[0110] A reconstruction unit, configured to replace the rendering result of the high-distortion area with the depth information and normal information in the dominant bin to perform adaptive surface reconstruction on the current Gaussian map to obtain the reconstructed Gaussian map.

[0111] In some embodiments, the adjustment module 703 includes:

[0112] a third computing unit, for computing a current radial gradient of the reconstructed Gaussian map through a Lie algebra framework;

[0113] The adjustment unit is used to adjust the current posture of the target image acquisition device according to the current radial gradient, so that the target image acquisition device after the adjusted posture is aligned with the surface normal direction of the reconstructed Gaussian map.

[0114] It should be noted that the aforementioned explanation of the embodiment of the real-time positioning and mapping method based on two-dimensional Gaussian splashing is also applicable to the real-time positioning and mapping device based on two-dimensional Gaussian splashing of this embodiment, which will not be repeated here.

[0115] The real-time positioning and mapping device based on two-dimensional Gaussian splashing proposed in an embodiment of the present invention includes the following beneficial effects:

[0116] (1) The two-dimensional Gaussian splash model of the scene can be reconstructed while the camera is moving, and the camera positioning can be realized based on the two-dimensional Gaussian.

[0117] (2) The uncertainty of the pixel area is evaluated by introducing Gaussian distortion measurement. In the high uncertainty area, the two-dimensional Gaussian information with the largest weight is directly used for reconstruction to ensure that the local geometric information in the overall map is strictly corresponding and the details are faithful, thereby improving the overall geometric reconstruction accuracy of the scene;

[0118] (3) Based on Lie groups and Lie algebras, the gradient information back-propagated during the two-dimensional Gaussian rendering process is used to derive the gradient expression of the rotation component. Combined with the pose estimation method based on rendering error and local geometric information, rapid convergence of pose updates is achieved, ensuring stable tracking even in scenes with restricted viewing angles.

[0119] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0120] A memory 801 , a processor 802 , and a computer program stored in the memory 801 and executable on the processor 802 .

[0121] When the processor 802 executes the program, the real-time positioning and mapping method based on two-dimensional Gaussian splashing provided in the above embodiment is implemented.

[0122] Furthermore, the electronic device further comprises:

[0123] The communication interface 803 is used for communication between the memory 801 and the processor 802 .

[0124] The memory 801 is used to store computer programs that can be executed on the processor 802 .

[0125] The memory 801 may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0126] If the memory 801, the processor 802, and the communication interface 803 are implemented independently, the communication interface 803, the memory 801, and the processor 802 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0127] Optionally, in a specific implementation, if the memory 801, the processor 802, and the communication interface 803 are integrated on a single chip, the memory 801, the processor 802, and the communication interface 803 can communicate with each other through an internal interface.

[0128] The processor 802 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0129] The embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above real-time positioning and mapping method based on two-dimensional Gaussian splash is implemented.

[0130] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0131] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0132] Any process or method description in the flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or N executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0133] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or N wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0134] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0135] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0136] In addition, in each embodiment of the present invention, each functional unit can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0137] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A real-time positioning and mapping method based on two-dimensional Gaussian splashing, characterized in that: The following steps are involved: Performing a two-dimensional Gaussian splash scene representation on the current viewing angle environment image acquired by the target image acquisition device to obtain a current Gaussian map; Calculating a depth distortion metric of the current Gaussian map to adaptively reconstruct the surface of the current Gaussian map according to the depth distortion metric to obtain a reconstructed Gaussian map; Calculating a current radial gradient of the reconstructed Gaussian map to adjust a current position and posture of the target image acquisition device according to the current radial gradient; The target image acquisition device with adjusted posture is used to capture the new perspective environment image, and the aforementioned two-dimensional Gaussian splash scene representation, adaptive surface reconstruction and posture adjustment process are iteratively performed until the pre-constructed mapping and posture optimization loss function converges, thereby obtaining the final geometric relationship between the target image acquisition device and the captured environment.

2. The real-time positioning and mapping method based on two-dimensional Gaussian splashing according to claim 1 is characterized in that: The performing a two-dimensional Gaussian splash scene representation on the current viewing angle environment image acquired by the target image acquisition device to obtain a current Gaussian map includes: Based on the two-dimensional Gaussian splash, define each two-dimensional Gaussian surface element of the current viewing angle environment image, and render each two-dimensional Gaussian surface element to obtain each rendered two-dimensional Gaussian surface element; Project and align each rendered two-dimensional Gaussian surface element to obtain an initial Gaussian map; Calculating the opacity of each two-dimensional Gaussian facet and the error between the depth rendering image and the current viewing angle environment image respectively; The initial Gaussian map is subjected to adding or subtracting two-dimensional Gaussian facets according to the opacity of each two-dimensional Gaussian facet and the error, so as to obtain the current Gaussian map.

3. The real-time positioning and mapping method based on two-dimensional Gaussian splashing according to claim 1 is characterized in that: The calculating the depth distortion metric of the current Gaussian map to adaptively reconstruct the surface of the current Gaussian map according to the depth distortion metric to obtain a reconstructed Gaussian map includes: Calculating a depth distortion metric of the current Gaussian map to determine high distortion areas on the current Gaussian map; Selecting a plane element with the largest transparency blending weight in the high distortion area as a dominant plane element, and extracting depth information and normal information from the dominant plane element; The rendering result of the high-distortion area is replaced with the depth information and normal information in the dominant surface element to perform adaptive surface reconstruction on the current Gaussian map to obtain the reconstructed Gaussian map.

4. The real-time positioning and mapping method based on two-dimensional Gaussian splashing according to claim 1 is characterized in that: The step of calculating the current radial gradient of the reconstructed Gaussian map to adjust the current posture of the target image acquisition device according to the current radial gradient includes: Calculate the current radial gradient of the reconstructed Gaussian map through a Lie algebra framework; The current posture of the target image acquisition device is adjusted according to the current radial gradient, so that the target image acquisition device after the adjusted posture is aligned with the surface normal direction of the reconstructed Gaussian map.

5. A real-time positioning and mapping device based on two-dimensional Gaussian splashing, characterized in that: A rendering module, used for performing a two-dimensional Gaussian splash scene representation on the current viewing angle environment image acquired by the target image acquisition device to obtain a current Gaussian map; A map reconstruction module, used for calculating a depth distortion metric of the current Gaussian map, so as to perform adaptive surface reconstruction on the current Gaussian map according to the depth distortion metric to obtain a reconstructed Gaussian map; An adjustment module, used for calculating a current radial gradient of the reconstructed Gaussian map, so as to adjust a current posture of the target image acquisition device according to the current radial gradient; The iterative positioning module is used to use the target image acquisition device after adjusting the posture to acquire the new perspective environment image, and iteratively execute the above-mentioned two-dimensional Gaussian splash scene representation, adaptive surface reconstruction and posture adjustment process until the pre-constructed mapping and posture optimization loss function converges, thereby obtaining the final geometric relationship between the target image acquisition device and the captured environment.

6. The real-time positioning and mapping device based on two-dimensional Gaussian splashing according to claim 5, characterized in that: The rendering module includes: A rendering unit, used for defining each two-dimensional Gaussian surface element of the current viewing angle environment image based on two-dimensional Gaussian splashing, and rendering each two-dimensional Gaussian surface element to obtain each rendered two-dimensional Gaussian surface element; An alignment unit, used for projecting and aligning each rendered two-dimensional Gaussian facet to obtain an initial Gaussian map; A first calculation unit, used for respectively calculating the opacity of each two-dimensional Gaussian surface element and the error between the depth rendering image and the current viewing angle environment image; The initial Gaussian map is subjected to adding or subtracting two-dimensional Gaussian facets according to the opacity of each two-dimensional Gaussian facet and the error, so as to obtain the current Gaussian map.

7. The real-time positioning and mapping device based on two-dimensional Gaussian splashing according to claim 5, characterized in that: The map reconstruction module includes: A second calculation unit, configured to calculate a depth distortion metric of the current Gaussian map to determine a high distortion area on the current Gaussian map; An extraction unit, configured to select a plane element with the largest transparency blending weight in the high distortion area as a dominant plane element, and extract depth information and normal information from the dominant plane element; A reconstruction unit is used to replace the rendering result of the high-distortion area with the depth information and normal information in the dominant surface element to perform adaptive surface reconstruction on the current Gaussian map to obtain the reconstructed Gaussian map.

8. The real-time positioning and mapping device based on two-dimensional Gaussian splashing according to claim 5, characterized in that: The adjustment module comprises: A third calculation unit, configured to calculate a current radial gradient of the reconstructed Gaussian map by using a Lie algebra framework; An adjustment unit is used to adjust the current posture of the target image acquisition device according to the current radial gradient, so that the target image acquisition device after the adjusted posture is aligned with the surface normal direction of the reconstructed Gaussian map.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the real-time positioning and mapping method based on two-dimensional Gaussian splashing as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the real-time positioning and mapping method based on two-dimensional Gaussian splashing as described in any one of claims 1 to 4.

Images