Panoramic three-dimensional mapping and positioning method and system based on multi-face projection and accompanying optimization

Abstract

The application discloses a panoramic three-dimensional mapping and positioning method and system based on multi-face projection and accompanying optimization, and belongs to the technical field of computer vision and robot navigation. The method comprises the following steps: acquiring a panoramic image sequence of an environment to be mapped, and obtaining multiple local perspective views of each panoramic image; obtaining a pose gradient of each local perspective view based on a current three-dimensional Gaussian map and a view pose of each local perspective view, and obtaining a main pose result of a current frame panoramic image by using a Lie algebra accompanying matrix; when the current frame is a key frame, generating a new three-dimensional Gaussian point based on each local perspective view, updating the three-dimensional Gaussian map, updating a key frame sliding window, and then jointly iteratively optimizing and adaptively densifying a geometric attribute and an appearance attribute of the three-dimensional Gaussian point, and finally obtaining a final three-dimensional Gaussian map and a main pose result of each panoramic image. The application can improve the mapping and positioning stability in a panoramic scene.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, 3D reconstruction and mobile robot navigation technology, and specifically relates to a panoramic 3D mapping and positioning method and system based on multi-plane projection and adjoint optimization. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) plays a crucial role in perception systems in high-tech fields such as autonomous driving, autonomous mobile robots, virtual reality (VR), and augmented reality (AR). In recent years, SLAM methods based on 3D Gaussian sputtering (3DGS) have attracted significant attention from academia and industry due to their ability to provide extremely high-fidelity novel view synthesis and achieve real-time rendering speeds.

[0003] However, most existing 3DGS SLAM systems are designed specifically for standard pinhole cameras, which have limited field of view (FOV) and are prone to losing tracking in areas lacking texture. Panoramic cameras, on the other hand, offer a 360° ultra-wide field of view, theoretically providing more stable pose tracking and more comprehensive environmental mapping. However, directly applying panoramic images to existing 3DGS SLAM frameworks faces numerous technical challenges: First, panoramic images (such as ERP equidistant cylindrical projection) have extremely strong nonlinear spatial distortion. Directly performing Gaussian projection and photometric error calculation on the distorted image will lead to severe numerical instability or even collapse in the optimization process.

[0004] When processing large-scale outdoor scenes, boundless regions such as the sky lack geometric parallax and have depths approaching infinity. Traditional unconstrained Gaussian initialization methods struggle to handle these regions, causing the system to generate numerous floating artifacts around the camera trajectory in an attempt to fit the sky color. This not only consumes a significant amount of video memory but also severely compromises structural accuracy and rendering fidelity.

[0005] Therefore, developing a high-fidelity panoramic SLAM method that can effectively eliminate panoramic distortion, ensure the physical rigidity of multi-faceted pose updates, and intelligently handle outdoor boundless scenes has significant theoretical and engineering application value. Summary of the Invention

[0006] To address the problems in the prior art, this invention provides a panoramic 3D mapping and positioning method and system based on multi-plane projection and accompanying optimization.

[0007] The technical solution of the present invention is as follows: In a first aspect, the present invention discloses a panoramic 3D mapping and positioning method based on multi-plane projection and accompanying optimization, comprising the following steps: 1) Obtain a panoramic image sequence of the environment to be mapped, and perform multi-plane projection on each panoramic image in the panoramic image sequence to obtain multiple local perspective views corresponding to each panoramic image; generate an initial three-dimensional Gaussian map using the multiple local perspective views corresponding to the first panoramic image. 2) Based on the view poses of the current 3D Gaussian map and the local perspective views of the current frame panoramic image, generate the prediction images corresponding to the local perspective views of the current frame panoramic image. Obtain the pose gradient of each local perspective view according to the photometric error loss between the prediction image and the corresponding local perspective view. Then, use the Lie algebra adjoint matrix to obtain the main pose estimate of the current frame panoramic image. 3) Update the view pose of the current frame panoramic image based on the master pose obtained in step 2), and repeat step 2) until the preset number of rounds to obtain the master pose result of the current frame panoramic image; determine whether the current frame panoramic image is a keyframe panoramic image based on the master pose result. If it is, generate new 3D Gaussian points based on each local perspective view of the current frame panoramic image and update the current 3D Gaussian map, and then execute step 4); otherwise, return to step 2) to process the next frame panoramic image. 4) Update the keyframe sliding window based on the keyframe panoramic image. Based on the panoramic image in the keyframe sliding window, perform joint iterative optimization and adaptive densification on the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map to obtain the updated 3D Gaussian map. 5) Return to step 2) to process the next frame of the panoramic image until all panoramic images have been processed to obtain the final 3D Gaussian map and the principal pose results of each panoramic image.

[0008] Further, in step 1), the panoramic image is projected onto multiple surfaces, including: according to the spatial orientation of each pixel in the panoramic image, each frame of the panoramic image is resampled to multiple sides of a virtual polyhedron to obtain multiple local perspective views that share the same camera optical center and have a fixed relative pose relationship.

[0009] Furthermore, the method for generating the initial 3D Gaussian map includes: performing monocular depth estimation on each local perspective view of the first frame panoramic image to obtain the predicted depth value corresponding to each pixel in each local perspective view; determining the target depth value of each pixel in each local perspective view based on the predicted depth value, a preset maximum effective depth threshold, and a preset fixed distance constant; back-projecting the pixels to the world coordinate system based on the target depth value, the principal pose result of the current frame panoramic image, and the fixed relative pose relationship between each local perspective view to generate 3D Gaussian points; and constructing the 3D Gaussian points into an initial 3D Gaussian map.

[0010] Further, in step 2), generating the predicted image corresponding to each local perspective view of the current frame panoramic image includes: projecting the three-dimensional Gaussian points in the current three-dimensional Gaussian map onto the image plane of the local perspective view according to the view pose of each local perspective view of the current frame panoramic image, and generating the predicted image corresponding to each local perspective view according to the projection position, opacity and color attributes of each three-dimensional Gaussian point.

[0011] Further, in step 2), obtaining the pose gradient of each local perspective view includes: applying a preset pose perturbation to the view pose of each local perspective view based on the photometric error loss between the predicted image and the corresponding local perspective view, and obtaining the pose gradient corresponding to each local perspective view based on the change in photometric error loss before and after the perturbation.

[0012] Further, in step 2), obtaining the principal pose of the current frame panoramic image using the Lie algebra adjoint matrix includes: constructing a Lie algebra adjoint matrix based on the fixed relative pose relationship between each local perspective view, and using the Lie algebra adjoint matrix to transform the pose gradients corresponding to each local perspective view to the coordinate system corresponding to the principal pose of the current frame panoramic image; fusing the transformed pose gradients into a joint pose gradient, then converting the joint pose gradient into a principal pose update amount according to a preset step size, and updating the principal pose of the current frame panoramic image based on the principal pose update amount to obtain the updated principal pose.

[0013] Further, in step 4), updating the keyframe sliding window based on the keyframe panoramic image includes: determining whether the number of keyframe panoramic images stored in the keyframe sliding window has reached a preset maximum capacity; if the preset maximum capacity has not been reached, inserting the current keyframe panoramic image and its principal pose result into the keyframe sliding window; if the preset maximum capacity has been reached, determining the multi-view intersection-union ratio between the current keyframe panoramic image and each historical keyframe panoramic image in the keyframe sliding window, removing the historical keyframe panoramic image with the highest multi-view intersection-union ratio from the keyframe sliding window, and inserting the current keyframe panoramic image and its principal pose result into the keyframe sliding window; wherein, the multi-view intersection-union ratio is the ratio of the sum of the intersection pixels of the effective visible areas of two keyframe panoramic images in each local perspective view to the sum of the union pixels.

[0014] Further, in step 4), the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map are jointly iteratively optimized and adaptively densified based on the panoramic image in the keyframe sliding window. This includes: randomly selecting a keyframe panoramic image from the keyframe sliding window, and selecting an optimized view from multiple local perspective views corresponding to the selected keyframe panoramic image; generating a prediction image corresponding to the optimized view based on the current 3D Gaussian map, and jointly iteratively optimizing the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map according to the photometric error loss between the prediction image and the optimized view; during the joint iterative optimization process, determining the 3D Gaussian points to be densified according to the degree of influence of each 3D Gaussian point on the photometric error loss, and performing cloning or splitting operations on the 3D Gaussian points to be densified; and removing 3D Gaussian points that meet the removal conditions according to the opacity of the 3D Gaussian points to obtain the updated 3D Gaussian map.

[0015] Secondly, the present invention also discloses a panoramic 3D mapping and positioning system based on multi-plane projection and adjoint optimization for implementing the method, comprising: The multi-projection module is used to acquire a panoramic image sequence of the environment to be mapped, and to perform multi-projection on each panoramic image in the panoramic image sequence to obtain multiple local perspective views corresponding to each panoramic image. The pose estimation module is used to generate the view poses of each local perspective view based on the current 3D Gaussian map and the current frame panoramic image, and the main pose result of the current frame panoramic image; wherein, an initial 3D Gaussian map is generated from multiple local perspective views corresponding to the first frame panoramic image. The keyframe mapping module is used to determine whether the current frame panoramic image is a keyframe panoramic image based on the main pose result. If it is, it generates new three-dimensional Gaussian points based on each local perspective view of the current frame panoramic image and updates the current three-dimensional Gaussian map. The mapping optimization and result output module is used to determine whether the current frame panoramic image is a keyframe panoramic image based on the principal pose result. When the current frame panoramic image is a keyframe panoramic image, it generates new 3D Gaussian points based on its local perspective views and updates the current 3D Gaussian map and the keyframe sliding window. Based on the panoramic image in the keyframe sliding window, it performs joint iterative optimization and adaptive densification processing on the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map, and outputs the final 3D Gaussian map and the principal pose results of each panoramic image.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) This invention reduces the impact of nonlinear distortion caused by panoramic imaging methods such as equidistant cylindrical projection on 3D Gaussian projection and photometric error calculation by decomposing panoramic image projection into multiple local perspective views. Since each local perspective view shares the same camera optical center and has a fixed relative pose relationship, it can preserve the large field of view information of the panoramic image while making the subsequent pose tracking and map building process closer to the perspective imaging model, thereby improving the stability of panoramic image sequence mapping and positioning.

[0017] 2) In the pose tracking process, this invention generates predicted images and obtains pose gradients based on each local perspective view. Then, it uses the Lie algebra adjoint matrix to transform and fuse the pose gradients of each local perspective view to update the principal pose of the current frame panoramic image. This processing method avoids the disruption of their fixed relative pose relationships caused by multiple local perspective views updating their poses independently. It allows the photometric error information of multiple views to participate in pose updating within the same principal pose optimization framework, thereby improving the consistency of pose estimation under multi-view projection conditions.

[0018] 3) This invention employs monocular depth estimation and depth truncation during keyframe mapping. For areas with significant depth or insufficient parallax, such as the sky and distant background, the target depth is limited by fixing a distance constant. This reduces the occurrence of abnormal floating points in the 3D Gaussian map in boundless areas, thereby improving the structural purity and rendering quality of the 3D Gaussian map in large-scale outdoor scenes.

[0019] 4) This invention is applicable to panoramic image inputs such as ERP format or panoramic loop format. By uniformly converting different panoramic image inputs into multiple local perspective views for subsequent processing, it reduces the dependence of subsequent 3D Gaussian mapping and positioning processes on the original panoramic projection format, improving the method's adaptability to different panoramic image formats. It eliminates the need for complex recalibration for specific cameras; coupled with an innovative multi-view keyframe management mechanism, it enables high-fidelity reconstruction while achieving high-frame-rate real-time online operation, making it directly applicable to various real-world robots and autonomous driving platforms.

[0020] 5) This invention manages the keyframe panoramic images involved in map optimization through a keyframe sliding window, and performs joint iterative optimization and adaptive densification processing on the geometric and appearance attributes of 3D Gaussian points based on the panoramic images in the keyframe sliding window. This method can reduce redundant calculations caused by repeated mapping in adjacent frames and enhance the expressive power of 3D Gaussian points in areas where map representation is insufficient, thus balancing the update quality and computational cost of the 3D Gaussian map. Attached Figure Description

[0021] Figure 1 This is a flowchart of the panoramic 3D mapping and positioning method based on multi-plane projection and accompanying optimization of the present invention; Figure 2 This is a trajectory comparison diagram between the present invention and the most advanced three-dimensional Gaussian mapping and positioning method. Detailed Implementation

[0022] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0023] This invention proposes a panoramic 3D mapping and localization method and system based on multi-faceted projection and adjoint optimization. Its core lies in eliminating distortion through cubic multi-faceted projection, unifying pose gradient by combining Lie algebra adjoint matrix, and using depth truncation to process unbounded scenes, thereby achieving high-quality panoramic SLAM.

[0024] like Figure 1 As shown, the method of the present invention mainly includes the following steps: (1) Multi-projection and distortion elimination of panoramic images This step aims to transform panoramic input with strong spatial distortion into a perspective view suitable for Gaussian sputtering optimization. It receives a sequence of panoramic images continuously acquired by a mobile device equipped with a panoramic camera as it moves through the environment to be mapped. This sequence is in ERP (quirectangular projection) format or Panoramic Angle (PAL) format. The environment to be mapped refers to the surrounding spatial environment in which the mobile device needs to perform 3D map construction and camera pose localization during its movement, including indoor environments, outdoor environments, or continuous indoor-outdoor environments.

[0025] For each frame of a panoramic image in an ERP format image sequence, let the width of the panoramic image be W and the height be H, and the two-dimensional pixel coordinates of any pixel in the panoramic image be... ,in Horizontal pixel coordinates These are the vertical pixel coordinates. First, according to the latitude and longitude mapping rules of equidistant cylindrical projection, the two-dimensional pixel coordinates are... Convert to longitude θ and latitude φ, where θ = 2πu / W - π, φ = π / 2-πv / H; then, it is projected inversely onto a unit sphere with the camera's optical center as the origin, establishing a correspondence between panoramic image pixels and spatial points on the unit sphere. Specifically, the unit direction vector d=(x, y, z) is calculated based on longitude θ and latitude φ, pointing from the camera's optical center to the corresponding spatial direction of the pixel, where x=cosφ·sinθ, y=sinφ, and z=cosφ·cosθ. This allows the pixel to be projected inversely onto the unit sphere with the camera's optical center as the origin, establishing a correspondence between panoramic image pixels and spatial directions on the unit sphere. Next, a virtual cube enclosing the unit sphere is constructed with the camera's optical center as the origin, and fixed orientations are preset for the front, back, left, and right sides. Since each side of the virtual cube can be considered a local imaging plane with a fixed relative orientation to the camera's optical center, the front, back, left, and right sides are used as imaging planes for four local perspective views, generating corresponding front, back, left, and right views respectively.

[0026] For each local perspective view to be generated, the target pixels in the local perspective view are traversed. Based on the two-dimensional position of the target pixel on the side of the corresponding virtual cube, a sampling ray is determined, originating from the camera's optical center and passing through the side of the virtual cube. The unit direction vector corresponding to this sampling ray is calculated. Then, this unit direction vector is converted into the corresponding longitude and latitude, and its sampling coordinates in the ERP format panoramic image are calculated based on the longitude and latitude. Finally, interpolation sampling is performed on the ERP format panoramic image based on the sampling coordinates to obtain the pixel value of the target pixel. Through the above resampling process, four local perspective views (front, back, left, and right) are generated from the same frame of panoramic image, forming a multi-faceted cubemap.

[0027] The four partial perspective views—front, rear, left, and right—are all projected from the same panoramic image frame, thus sharing the same camera optical center. Furthermore, these four partial perspective views have a fixed relative rotation relationship: the right view rotates 90° relative to the front view around the vertical axis, the rear view rotates 180° relative to the front view around the vertical axis, and the left view rotates -90° relative to the front view around the vertical axis. This converts the original ERP format panoramic image into multiple partial perspective views with minimal distortion and defined relative pose relationships, for subsequent use in 3D Gaussian map construction and camera pose tracking.

[0028] In this panoramic image, each frame has a corresponding front view, back view, left view, and right view, each with its own local view camera coordinate system. These local view coordinate systems share the same camera optical center and have a fixed relative rotation relationship. They are four virtual local perspective views projected from the same panoramic camera. These four local perspective views share the same camera optical center, meaning they have the same origin; however, their viewing directions are different, so their coordinate axis directions are different. Therefore, each local perspective view can be considered to have its own corresponding local view coordinate system.

[0029] (2) Initial keyframe mapping and initial 3D Gaussian map generation For the first panoramic image in the panoramic image sequence, the system uses the first panoramic image as the initial keyframe and the corresponding forward view as the initial main view. The system uses the local view camera coordinate system corresponding to the initial main view as the world coordinate system and sets the principal pose of the initial main view relative to the world coordinate system as the initial unit pose; wherein, the initial unit pose means that the rotation matrix of the initial main view relative to the world coordinate system is the identity matrix and the translation vector is the zero vector. For a given initial keyframe, a pre-trained monocular depth estimation model (source of the pre-trained monocular depth estimation model: YANG L, KANG B, HUANG Z, et al. Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data [PP / OL]. arXiv (2024-04-07) [2026-06-09]. http: / / arxiv.org / abs / 2401.10891. DOI:10.48550 / arXiv.2401.10891.) is used to process each local perspective view corresponding to the initial keyframe, resulting in monocular depth maps for each local perspective view. Each pixel in the monocular depth map has a corresponding predicted depth, and the predicted depth value is used to characterize the distance of the scene point corresponding to that pixel relative to the camera coordinate system of the corresponding local perspective view.

[0030] When the predicted depth in the monocular depth map output by the pre-trained monocular depth estimation model is a relative depth, the relative depth value is scaled according to the preset scale coefficient, camera height, known scene scale, or pose change results of adjacent frames to obtain the predicted depth value used for back projection calculation; when the depth output by the pre-trained monocular depth estimation model is the actual scale depth, it is directly used as the predicted depth value used for back projection calculation.

[0031] Set a maximum effective depth threshold and fixed distance constant ,in, Maximum effective depth threshold A distant view threshold representing the actual physical distance is used to distinguish between the effective depth region and the distant view or parallax-invalid region; a fixed distant distance constant is used to provide a limited initial depth for the distant view or parallax-invalid region.

[0032] For any local perspective view in the initial keyframe, depth truncation and backprojection calculations are performed as follows to generate and insert new 3D Gaussian points: Let the two-dimensional pixel coordinates of the pixel to be processed in the partial perspective view be... And its corresponding camera intrinsic parameter matrix (the virtual camera intrinsic parameter matrix corresponding to the local perspective view, which is predetermined based on the image size and field of view of the local perspective view) is The predicted depth value of the pixel to be processed in the corresponding monocular depth map is Then based on the predicted depth value Determine the target depth value of the pixel to be processed. : When the predicted depth value Less than or equal to the maximum effective depth threshold When the pixel to be processed is determined to belong to the effective depth region, the target depth value of the pixel to be processed is directly set. Equal to the predicted depth value When the predicted depth value Greater than the maximum effective depth threshold When parallax is invalid (i.e., it is identified as an outdoor boundless background area such as the sky), the system does not discard the pixels corresponding to that area, but instead performs depth truncation, forcibly removing the target depth value of the pixel to be processed. Set to a fixed long-distance constant In a specific embodiment of the present invention, a fixed long-distance constant is used. It is set to 10 meters.

[0033] Determine the target depth value for each pixel. Then, the three-dimensional spatial coordinates of the pixel in the local view camera coordinate system are calculated using the back projection formula. : Next, the 3D spatial coordinates of each local view camera coordinate system corresponding to the initial keyframe are uniformly transformed to the world coordinate system. Specifically, since the local view camera coordinate system corresponding to the front view of the initial keyframe is the world coordinate system, the 3D spatial coordinates of the pixels obtained by backprojection in the local view camera coordinate system are the 3D spatial coordinates of the pixels in the front view in the world coordinate system. For the pixels in the rear view, left view, or right view, the system transforms the 3D spatial coordinates obtained by backprojection to the world coordinate system according to the fixed relative rotation relationship of each local perspective view relative to the initial main view, thus obtaining the 3D spatial coordinates of the pixels in each local perspective view in the world coordinate system.

[0034] After obtaining the three-dimensional spatial coordinates in the world coordinate system, the system uses these three-dimensional spatial coordinates as the initial center position of the three-dimensional Gaussian point. Based on the color information of the pixel to be processed in the local perspective view corresponding to the three-dimensional spatial coordinates, the system sets the initial color attribute of the three-dimensional Gaussian point (i.e., using the RGB color value of the pixel to be processed as the initial color attribute of the corresponding three-dimensional Gaussian point). Simultaneously, it sets the initial scaling scale, initial rotation parameters, and initial opacity for the three-dimensional Gaussian point, thus generating the initial three-dimensional Gaussian point. Therefore, the two-dimensional image information corresponding to the pixel to be processed is converted into a three-dimensional Gaussian point with an initial spatial position and initial color attributes.

[0035] The system performs the aforementioned processing on all pixels to be processed in the first frame of the panoramic image, obtaining the initial 3D Gaussian points corresponding to each pixel, thus obtaining an initial 3D Gaussian point set. Next, the system establishes a map data structure to store the 3D Gaussian points and inserts each initial 3D Gaussian point from the initial 3D Gaussian point set into the map data structure. For each initial 3D Gaussian point, the system records its 3D center coordinates, scaling scale, rotation parameters, opacity, and color attributes in the world coordinate system. After insertion, the map data structure contains the initial 3D Gaussian point set with the world coordinate system as a unified reference, thereby obtaining the initial 3D Gaussian map. The rotation parameters can be represented using rotation quaternions.

[0036] (3) Subsequent frame location and key frame determination For input frames following the first panoramic image, the system uses the pose corresponding to the forward view of the current input frame as the principal pose of the current input frame. Specifically, the system uses the principal pose already tracked in the previous input frame as the initial value of the principal pose of the current input frame. If the current input frame is the second panoramic image in the panoramic image sequence, then the principal pose already tracked in the previous input frame is the initial unit pose of the first panoramic image.

[0037] After obtaining the initial principal pose of the current input frame, the system fixes the geometric and appearance attributes of each 3D Gaussian point in the current 3D Gaussian map, and uses the principal pose of the current input frame as the variable to be optimized. Based on the initial principal pose and the fixed relative pose relationships between each local perspective view of the current input frame and the forward view of the current input frame, the system determines the view poses corresponding to the forward, backward, left, and right views of the current input frame. The view pose corresponding to the forward view is the principal pose of the current input frame, and the view poses corresponding to the backward, left, and right views are obtained by combining the principal pose of the current input frame with the corresponding fixed relative pose relationships.

[0038] For any local perspective view in the current input frame, the system obtains the camera intrinsic parameters, the current 3D Gaussian map, and the view pose corresponding to the local perspective view. The view pose is used to characterize the position and orientation of the local view camera coordinate system relative to the world coordinate system. The current 3D Gaussian map includes multiple 3D Gaussian points, each of which has 3D center coordinates, scaling scale, rotation parameters, opacity, and color attributes.

[0039] Based on the view pose corresponding to the local perspective view, the system transforms the 3D center coordinates of each 3D Gaussian point in the current 3D Gaussian map from the world coordinate system to the local view camera coordinate system corresponding to the local perspective view. Based on the camera intrinsic parameters, the system projects the 3D Gaussian points within the visible range of the local perspective view onto the image plane, thereby obtaining the 2D projection position and depth value of each 3D Gaussian point on the image plane.

[0040] The system determines the two-dimensional influence region of each 3D Gaussian point after it is projected onto the image plane based on the scaling and rotation parameters of each point. For any pixel in the local perspective view, the system determines the 2D influence region that covers the 3D Gaussian points of that pixel, and determines the depth order based on the depth values ​​of the 3D Gaussian points covering the pixel. Subsequently, the system performs transparency synthesis according to the opacity and color attributes of the 3D Gaussian points covering the pixel, based on the depth order, to obtain the predicted color of the pixel. The system traverses all pixels in the local perspective view to obtain the predicted image corresponding to the local perspective view.

[0041] The predicted image is then compared with the true local perspective view corresponding to the current input frame to obtain the photometric loss for each local perspective view. This photometric loss is calculated using a weighted combination of L1 loss and structural similarity (SSIM) loss. in, This is due to photometric error loss; This is the predicted image corresponding to the partial perspective view. This is a true partial perspective view; These are preset weight hyperparameters; For structural similarity loss; It is an L1 norm.

[0042] After obtaining the photometric error loss corresponding to each local perspective view of the current input frame, the system calculates the pose gradient of the corresponding local perspective view based on each photometric error loss.

[0043] Specifically, for any local perspective view, the system uses the current view pose of the local perspective view as a reference and applies preset minute pose changes along three translational directions and three rotational directions of the local view camera coordinate system corresponding to the local perspective view. In each direction, based on the view pose after the minute pose change, the system regenerates the predicted image corresponding to the local perspective view based on the current 3D Gaussian map and recalculates the photometric error loss between the predicted image and the real local perspective view. The system determines the gradient component corresponding to that direction based on the change in photometric error loss before and after the minute pose change. The system obtains the gradient components in the three translational directions and the three rotational directions respectively, and combines the gradient components in the three translational directions and the three rotational directions to obtain the six-dimensional pose gradient corresponding to the local perspective view, i.e., the pose gradient corresponding to the local perspective view. The six-dimensional pose gradient is used to characterize the trend of photometric error loss when the view pose of the local perspective view undergoes minute changes in the three-dimensional translational and three-dimensional rotational directions.

[0044] The four partial perspective views—front, back, left, and right—have a fixed relative rotation relationship, that is, a fixed relative pose. Therefore, the system only updates the main pose of the current input frame independently. The poses of the left, right, and back views are not updated independently, but are calculated from the main pose and the corresponding fixed relative pose.

[0045] For the forward view, the system uses its pose gradient as the pose gradient of the main view. For the left, right, and rear views, the system first obtains the pose gradient of each auxiliary view in its own coordinate system. Since the viewing directions of each auxiliary view are different from those of the forward view, the system transforms the pose gradient of each auxiliary view to the coordinate system of the main view corresponding to the forward view based on the fixed relative pose relationship between each auxiliary view and the forward view, thus obtaining the equivalent main pose gradient of each auxiliary view.

[0046] Specifically, the system uses a fixed relative pose for gradient transformation. Construct the corresponding Lie algebra adjoint matrix, where, , A pose transformation form used to represent the position and orientation of a camera or partial perspective view in three-dimensional space. Includes rotation matrix Translation vector The specific form of the adjoint matrix of a Lie algebra is a The matrix, specifically: in, Translation vector An antisymmetric matrix.

[0047] The system utilizes the Lie algebra adjoint matrix to convert the pose gradient of the auxiliary view in its own coordinate system into the equivalent principal pose gradient in the coordinate system of the corresponding principal view. If all local perspective views share the same optical center, then the translation vector... The zero vector is the Lie algebraic adjoint matrix, which is determined by the fixed relative rotation matrix of each auxiliary view relative to the front view.

[0048] Calculate the Lie algebraic adjoint matrices of the left, right, and rear auxiliary views relative to the front view, and then calculate the pose gradient of each auxiliary view in its own coordinate system. Multiply by the transpose of its corresponding Lie algebraic adjoint matrix Strictly unify the pose gradient mapping of all auxiliary views to the equivalent principal pose gradient in the forward view coordinate system.

[0049] The system accumulates the pose gradient of the main view corresponding to the forward view with the equivalent main pose gradients of each auxiliary view to obtain the joint pose gradient of the current input frame. Then, the system determines the main pose update amount of the current input frame based on the joint pose gradient. Specifically, the system scales the joint pose gradient in the reverse direction according to a preset step size to obtain the six-dimensional main pose update amount for the current iteration. This six-dimensional main pose update amount includes updates in three translational directions and three rotational directions, representing the correction magnitude and direction of the main pose of the current input frame in this iteration. Subsequently, the system converts the six-dimensional main pose update amount into a pose increment matrix using a Lie algebra exponential mapping method, and combines the pose increment matrix with the main pose of the current input frame to obtain the updated main pose. After the main pose update, the system recalculates the view poses corresponding to each auxiliary view based on the updated main pose and the fixed relative pose relationships between each auxiliary view and the forward view, thus completing absolutely rigid multi-faceted joint pose tracking.

[0050] The system repeatedly performs the following steps: predictive image generation, photometric error loss calculation, pose gradient calculation, auxiliary view pose gradient transformation, joint pose gradient calculation, and principal pose update, until the preset number of tracking iterations is reached or the photometric error loss meets the preset convergence condition. The final updated principal pose is then used as the principal pose that has been tracked in the current input frame (i.e., the principal pose result of the current input frame).

[0051] The next step is to determine whether the current input frame is a keyframe: after the principal pose of the current input frame converges, the system determines whether the current input frame should be used as a new keyframe based on the principal pose of the current input frame.

[0052] Specifically, the system uses the most recently determined keyframe as the reference keyframe and obtains its frame number, acquisition time, and principal pose. To confirm the current input frame as a keyframe, both of the following conditions must be met simultaneously: One is the temporal interval condition, which means that the relative interval between the current input frame and the reference key frame in the panoramic image sequence exceeds the preset frame number threshold (such as setting the frame number threshold to 5). Secondly, there is the spatial motion condition. The system calculates the relative pose change of the current input frame relative to the reference keyframe based on the main pose already tracked in the current input frame and the main pose already tracked in the reference keyframe. Based on this relative pose change, the system obtains the relative translation distance and relative rotation angle. When the relative translation distance is greater than a preset translation threshold, or the relative rotation angle is greater than a preset rotation threshold, the current input frame is deemed to meet the spatial motion condition.

[0053] The system will determine the current input frame as the new keyframe if and only if the above-mentioned temporal interval condition and spatial motion condition are met simultaneously.

[0054] If the current input frame is not a keyframe, the system outputs the principal pose result of the current input frame and continues processing the next frame of panoramic image. If the current input frame is a keyframe, the system designates the current input frame as the new keyframe and inputs the new keyframe, the corresponding principal pose result, and each local perspective view into the 3D Gaussian map update step. In the 3D Gaussian map update step, the system uses the same depth estimation, depth truncation, and backprojection methods as the initial keyframe mapping. Based on the local perspective views corresponding to the new keyframe, camera intrinsics, target depth values, principal pose results, and the fixed relative pose relationships of each local perspective view relative to the forward view, the system generates new 3D Gaussian points and inserts these new 3D Gaussian points into the current 3D Gaussian map to expand the current 3D Gaussian map.

[0055] (4) Keyframe sliding window update and 3D Gaussian map expansion optimization This step is used to maintain a set of keyframes for map optimization with limited computational resources, and to perform joint optimization and adaptive densification of the geometric and appearance attributes of the 3D Gaussian map based on the set of keyframes.

[0056] (4.1) Dynamic management of keyframes based on intersection-union ratio The system maintains a keyframe sliding window, which stores keyframes participating in global joint optimization and their corresponding tracked principal poses. The keyframe sliding window has a preset maximum capacity N, where N is an integer greater than 1. In a specific embodiment of the invention, N is 8.

[0057] When a new keyframe enters the mapping thread, the system determines whether the keyframe sliding window has reached the preset maximum capacity N. If the keyframe sliding window has not reached the preset maximum capacity N, the system directly inserts the new keyframe and its corresponding already tracked principal pose into the keyframe sliding window.

[0058] If the keyframe sliding window has reached the preset maximum capacity N, the system calculates the multi-face common view intersection-union ratio between the new keyframe and each historical keyframe in the keyframe sliding window. Then, the historical keyframe with the highest multi-face common view intersection-union ratio with the new keyframe is identified as a redundant keyframe and removed from the keyframe sliding window. Subsequently, the new keyframe and its corresponding master pose that has been tracked are inserted into the keyframe sliding window.

[0059] Specifically, for two keyframes whose multi-view intersection-union ratio (XOR) needs to be calculated, the system determines the effective visible areas of the two keyframes in the front view, back view, left view, and right view, based on the principal poses and multiple local perspective views corresponding to each keyframe. The effective visible area is the effective pixel area formed by projecting or rendering the 3D Gaussian points in the current 3D Gaussian map onto the corresponding local perspective view. The system counts the intersection and union of the effective visible areas of the two keyframes in each corresponding local perspective view, sums the intersection pixels corresponding to each local perspective view to obtain the total number of intersecting pixels, and sums the union pixels corresponding to each local perspective view to obtain the total number of pixels. The ratio of the total number of intersecting pixels to the total number of pixels is then used as the multi-view intersection-union ratio between the two keyframes. Specifically, when a pixel position of a 3D Gaussian point is projected or rendered onto the corresponding local perspective view, and the rendering opacity of that pixel position is greater than a preset opacity threshold, that pixel position is determined as a valid pixel; the effective visible area is composed of these valid pixels.

[0060] (4.2) Global joint optimization and adaptive intensification The system selects at least one keyframe from a keyframe sliding window and chooses at least one local perspective view from the forward, backward, left, and right views corresponding to the selected keyframe as the optimized view. For each optimized view, the system obtains the corresponding real image, camera intrinsic matrix, and camera pose matrix (if the optimized view is not a forward view, the camera pose matrix is ​​obtained from the already tracked principal pose of the keyframe corresponding to the optimized view and the fixed relative pose of the corresponding auxiliary view relative to the forward view). Each 3D Gaussian point in the current 3D Gaussian map has corresponding geometric and appearance attributes. The geometric attributes include 3D center coordinates, scaling scale, and rotation parameters, and the appearance attributes include opacity and color attributes. When the 3D Gaussian point is first generated, the geometric and appearance attributes are initial values, and they are gradually updated during subsequent joint iterative optimization. The geometric attributes include the 3D center coordinates, scaling scale, and rotation parameters of the 3D Gaussian point, and the appearance attributes include the opacity and spherical harmonic coefficients of the 3D Gaussian point.

[0061] The system transforms the 3D center coordinates of each 3D Gaussian point in the current 3D Gaussian map from the world coordinate system to the camera coordinate system corresponding to the optimized view based on the camera pose matrix, and projects each 3D Gaussian point onto the image plane of the optimized view based on the camera intrinsic parameter matrix, thereby obtaining the 2D projection position of each 3D Gaussian point on the image plane.

[0062] The system determines the spatial distribution range of each 3D Gaussian point in 3D space based on its current scaling and rotation parameters. Combining this with the camera pose matrix and intrinsic parameter matrix corresponding to the optimized view, the system projects this spatial distribution range onto the image plane, obtaining a 2D elliptical region on the image plane for each 3D Gaussian point. This elliptical region is then used as the 2D influence region of the corresponding 3D Gaussian point. For each pixel in the optimized view, the system identifies the 3D Gaussian point covering that pixel and obtains its depth value in the camera coordinate system of the optimized view. This depth value is the coordinate value of the 3D center coordinates of the 3D Gaussian point after transformation to the camera coordinate system of the optimized view, along the camera optical axis (i.e., the z-axis direction). The system sorts the 3D Gaussian points covering the same pixel according to their depth values, obtaining a depth order. This depth order characterizes the occlusion relationship of the 3D Gaussian points covering the same pixel relative to the optimized view. Subsequently, the system performs transparency synthesis based on the depth order, the current opacity, and color attributes of each 3D Gaussian point to obtain the predicted color of that pixel. The system iterates through each pixel in the optimized view to obtain the predicted image corresponding to the optimized view. After obtaining the predicted image corresponding to the optimized view, the photometric error loss between the predicted image and the real image corresponding to the optimized view is calculated.

[0063] The system then performs backpropagation based on the photometric error loss to jointly iteratively optimize the geometric and appearance attributes of the 3D Gaussian points in the 3D Gaussian map. Specifically, the system uses the geometric and appearance attributes of the 3D Gaussian points as parameters to be optimized. Based on the aforementioned predicted image generation process, the system performs joint iterative optimization of the geometric and appearance attributes (including 3D center coordinates, scaling, rotation parameters, opacity, and spherical harmonic coefficients) of the 3D Gaussian points through global backpropagation.

[0064] During the joint iterative optimization of the 3D Gaussian map, the system performs adaptive densification processing on each 3D Gaussian point. Specifically, in each optimization iteration, the system projects each 3D Gaussian point onto the image plane of the optimized view to obtain the two-dimensional projection position of each 3D Gaussian point, and calculates the gradient of the photometric error loss relative to the two-dimensional projection position. The magnitude of this gradient is used as the view space position gradient of the corresponding 3D Gaussian point in the optimized view. The system averages the view space position gradients of the same 3D Gaussian point in multiple optimized views or multiple optimization iterations to obtain the average view space position gradient of the 3D Gaussian point. When the average view space position gradient is greater than a preset densification threshold, it is determined that the map representation of the area where the 3D Gaussian point is located is insufficient and densification processing is required.

[0065] After determining that densification is required, the system selects a densification method based on the scaling scale of the 3D Gaussian point. If the scaling scale of the 3D Gaussian point is smaller than a preset scale boundary, it indicates that the coverage area of ​​the 3D Gaussian point is small. The system performs a cloning operation on the 3D Gaussian point, that is, copies the geometric and appearance attributes of the 3D Gaussian point to generate at least one new 3D Gaussian point, and adds a preset perturbation amount to the 3D center coordinates of the new 3D Gaussian point so that the new 3D Gaussian point is distributed near the original 3D Gaussian point. If the scaling scale of the 3D Gaussian point is greater than or equal to the preset scale boundary, it indicates that the coverage area of ​​the 3D Gaussian point is large. The system performs a splitting operation on the 3D Gaussian point, that is, generates at least two new 3D Gaussian points based on the 3D center coordinates of the original 3D Gaussian point, and makes the scaling scale of the at least two new 3D Gaussian points smaller than the scaling scale of the original 3D Gaussian point, so as to improve the map representation accuracy of the area.

[0066] The system also resets or updates the opacity of the 3D Gaussian points according to a preset iteration interval, and determines whether the opacity of each 3D Gaussian point is lower than a preset opacity threshold during each removal check. If a 3D Gaussian point is lower than the preset opacity threshold in a preset number of consecutive removal checks, it is determined that the 3D Gaussian point has a small contribution to the color of the predicted image, and the 3D Gaussian point is removed from the 3D Gaussian map.

[0067] After step (4), the system obtains an updated keyframe sliding window and an updated 3D Gaussian map. The updated 3D Gaussian map is used for pose tracking of subsequent input frames and map expansion of subsequent keyframes.

[0068] (5) Panoramic 3D mapping and positioning For subsequent input frames, repeat steps (3)-(4) until all input frames in the panoramic image sequence are processed to obtain the final three-dimensional Gaussian map and the principal pose result corresponding to each panoramic image, thereby realizing panoramic three-dimensional mapping and localization.

[0069] In a preferred embodiment of the present invention, the present invention also provides a panoramic 3D mapping and positioning system based on multi-projection and accompanying optimization, including a multi-projection module, a pose estimation module, a keyframe mapping module, and a mapping optimization and result output module. The pose estimation module includes a pose tracking module and a pose iteration module, and the mapping optimization and result output module includes a map optimization module and a loop processing module.

[0070] The multi-projection module is used to acquire a panoramic image sequence of the environment to be mapped, and to perform multi-projection on each panoramic image in the panoramic image sequence to obtain multiple local perspective views corresponding to each panoramic image.

[0071] The pose tracking module is used to generate predicted images corresponding to each local perspective view of the current frame panoramic image based on the view poses of the current 3D Gaussian map and each local perspective view of the current frame panoramic image. The pose gradient of each local perspective view is obtained based on the photometric error loss between the predicted image and the corresponding local perspective view. Then, the main pose estimate of the current frame panoramic image is obtained by using the Lie algebra adjoint matrix.

[0072] The pose iteration module is used to update the view pose of the current frame panoramic image based on the master pose obtained by the pose tracking module, and trigger the pose tracking module to repeat the process until the master pose result of the current frame panoramic image is obtained.

[0073] The keyframe mapping module is used to determine whether the current frame panoramic image is a keyframe panoramic image based on the principal pose result. If so, it generates new three-dimensional Gaussian points based on each local perspective view of the current frame panoramic image and updates the current three-dimensional Gaussian map.

[0074] The map optimization module updates the keyframe sliding window based on the keyframe panoramic image, and performs joint iterative optimization and adaptive densification processing on the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map based on the panoramic image in the keyframe sliding window, to obtain the updated 3D Gaussian map. The loop processing module triggers the pose tracking module to process the next frame panoramic image when the current frame panoramic image is not a keyframe panoramic image, or after the map optimization module obtains the updated 3D Gaussian map, until all panoramic images have been processed, obtaining the final 3D Gaussian map and the principal pose results of each panoramic image.

[0075] Implementation Examples To verify the effectiveness of the method described in this invention, an experiment was conducted on a local workstation equipped with an NVIDIA GeForce RTX 4070 Ti GPU and an Intel Core i5 processor.

[0076] 1. Experimental environment and dataset: Large-scale multimodal synthetic panoramic datasets (SynPano, OmniBlender) and real-world indoor panoramic datasets (PALVIO) were used. The input data covered narrow indoor corridors, open conference rooms, and outdoor architectural scenes with large areas of sky and complex lighting (such as lone-monk sequences and desert sequences).

[0077] 2. Example and comparative analysis of the running process: After the system starts up, it directly receives panoramic images in ERP format from the video stream and converts them into cubemaps in real time for processing.

[0078] (1) Regarding pose tracking accuracy: In the real PALVIO dataset, when the camera rotates drastically or is in a region with repetitive textures, traditional monocular Gaussian SLAM (such as MonoGS) frequently loses tracking due to limited field of view, with an absolute trajectory error (ATE) exceeding 2.8 meters. However, the system of this invention relies on an ultra-large field of view and adjoint consistency optimization to tightly lock onto features, with an average ATE stabilizing at around 0.07 meters, achieving sub-decimeter-level positioning accuracy.

[0079] (2) Regarding the mapping effect in outdoor scenes (the effect of depth truncation): Taking an outdoor scene (including a large area of ​​sky) as an example. When optimizing the sky area at infinity, existing technologies (Photo-SLAM, MonoGS) cause the optimizer to go out of control, generating a large number of chaotic Gaussian spheres (dirty artifacts) above the camera's motion trajectory (in mid-air), which severely obscures normal buildings, and the PSNR of the new perspective synthesis is only about 22dB. However, after adopting the depth truncation strategy of this invention, the system smoothly initializes the sky pixels to the set far-distance virtual sphere, and the generated scene has a clean and clear sky, sharp building edges, no floating artifacts, and the PSNR of the rendered image is significantly improved to over 29dB.

[0080] Figure 2 The present invention is compared with three other methods—Photo-SLAM (a real-time synchronous localization and photorealistic mapping method for monocular, binocular, and RGB-D cameras), MonoGS (a monocular synchronous localization and mapping method based on 3D Gaussian sputtering), and S3PO-GS (an outdoor monocular synchronous localization and mapping method based on globally scale-consistent 3D Gaussian point maps)—in terms of camera trajectory estimation results on different datasets and scene sequences. Figure 2 Multiple trajectory comparison sub-images are arranged in rows and columns. From top to bottom, each row corresponds to the ID02 sequence from the PALVIO dataset, the dojo sequence from the SynPano dataset, the livingroom sequence from the SynPano dataset, and the barbershop sequence from the OmniBlender dataset. From left to right, each column corresponds to the Photo-SLAM method, the MonoGS method, the S3PO-GS method, and the method of this invention. In each sub-image, the gray dashed line GT represents the real camera trajectory in the corresponding scene, and the colored solid line represents the estimated camera trajectory obtained by the corresponding method. The color of the colored solid line indicates the magnitude of the trajectory error at different positions on the estimated trajectory according to the color scale on the right side of each row. Blue corresponds to a smaller trajectory error, and the trajectory error gradually increases as the color changes to cyan, green, yellow, and red. Therefore, the higher the overlap between the colored solid line and the gray dashed line, and the closer the colored solid line is to blue, the closer the pose estimation result obtained by the corresponding method is to the real pose, and the higher the trajectory tracking accuracy.

[0081] exist Figure 2In the PALVIO-ID02 sequence shown in the first row, the actual trajectory is an irregular closed trajectory with two ring regions. The estimated trajectory obtained by the Photo-SLAM method is only close to the actual trajectory in some sections, with significant deviations in the left side, the central intersection area, and the right curved area, with a trajectory error of 1.726. The estimated trajectory obtained by the MonoGS method shows obvious trajectory clustering, intersection, and drift in the central area, failing to recover the two ring structures of the actual trajectory, and appears red or orange in several sections, with a trajectory error of 3.047. The S3PO-GS method can recover the general outline of the actual trajectory, but there are still significant offsets in the upper, right, and central connecting areas, with a trajectory error of 1.932. In contrast, the estimated trajectory obtained by the method of this invention basically extends along the actual trajectory shown by the gray dashed line, and can completely recover the two ring regions on the left and right and their central connecting relationship. The colored solid line is mainly blue and cyan, with slight color changes only at local turning and connecting positions, and the trajectory error is reduced to 0.046, indicating that the method of this invention can effectively suppress pose drift in real panoramic sequences with complex motion paths and multiple directional changes.

[0082] exist Figure 2 In the SynPano-dojo sequence shown in the second row, the actual trajectory is an approximately elliptical closed trajectory. While the estimated trajectory obtained by the Photo-SLAM method can recover part of the elliptical outline, it deviates from the actual trajectory on the right side and at the closed position, with a trajectory error of 0.229. The estimated trajectory obtained by the MonoGS method differs significantly from the actual trajectory in scale, shape, and position; the estimated trajectory is stretched into a narrow ellipse with a noticeable red high-error section at the bottom, with a trajectory error of 1.091. The estimated trajectory obtained by the S3PO-GS method is generally close to the actual trajectory, but there are still local deviations at the top, left, and right sides of the elliptical trajectory, with a trajectory error of 0.137. The estimated trajectory obtained by the method of this invention basically coincides with the actual elliptical trajectory, forming a continuous and stable closed trajectory. The colored solid lines are almost entirely blue, with a trajectory error of only 0.007, indicating that the method of this invention can maintain high global pose consistency in closed-loop motion scenes and reduce trajectory scale deviation and closed-loop cumulative error.

[0083] exist Figure 2In the SynPano-livingroom sequence shown in the third row, the true trajectory is a non-closed curve with multiple consecutive bends. The Photo-SLAM method only approximates the true trajectory in the upper part of the trajectory, failing to fully recover the true motion path in the remaining sections, with a trajectory error of 0.252. Although the MonoGS method can obtain a bending trend roughly similar to the true trajectory, it exhibits significant lateral offset in the upper bending region and has large error sections in red and orange, with a trajectory error of 0.145. The S3PO-GS method can recover the true trajectory relatively completely, but the gap between the estimated trajectory and the true trajectory can still be observed at the upper and middle bending positions, with a trajectory error of 0.036. The estimated trajectory obtained by the method of this invention basically coincides with the gray dashed line at each consecutive bending position, maintaining the same bending direction and spatial shape as the true trajectory. The colored solid line is generally blue, with a trajectory error of only 0.004, indicating that the method of this invention can still maintain stable pose tracking results under continuous turning and curvilinear motion conditions.

[0084] exist Figure 2 In the OmniBlender-barbershop sequence shown in the fourth row, the actual trajectory is an approximately circular closed trajectory. The estimated trajectory obtained by the Photo-SLAM method is generally close to the actual trajectory, but there are still local offsets in the left, top, and lower right regions of the trajectory, with a trajectory error of 0.0065. The trajectory obtained by the MonoGS method shows obvious shape distortion, positional offset, and scale changes relative to the actual trajectory. Some estimated trajectories are located outside the actual trajectory and have many red and orange high-error segments, with a trajectory error of 0.1577. The S3PO-GS method can recover the closed trajectory, but the estimated trajectory is obviously polygonal and deviates from the actual trajectory in the top, left, and bottom regions, with a trajectory error of 0.1056. The estimated trajectory obtained by the method of this invention forms a complete closed loop along the actual trajectory. Each trajectory segment basically coincides with the gray dashed line, and the colored solid line is dark blue overall, with a trajectory error of only 0.0006, indicating that the method of this invention can effectively maintain the shape, scale, and spatial position of the closed loop trajectory.

[0085] comprehensive Figure 2The four sets of experimental results show that the trajectory errors of the method of this invention in the four test sequences PALVIO-ID02, SynPano-dojo, SynPano-livingroom, and OmniBlender-barbershop are 0.046, 0.007, 0.004, and 0.0006, respectively, all lower than the trajectory errors of Photo-SLAM, MonoGS, and S3PO-GS methods under the same test sequences. Meanwhile, the color-estimated trajectory corresponding to the method of this invention maintains a high degree of overlap with the true trajectory shown by the gray dashed line in each sub-figure, and the trajectory color is mainly concentrated in the blue low-error range; other comparative methods exhibit varying degrees of trajectory offset, shape distortion, scale variation, local drift, trajectory clustering, or inaccurate loop closure. This demonstrates that the present invention obtains pose gradients based on multiple local perspective views and uses the Lie algebra adjoint matrix to transform each pose gradient to the principal pose coordinate system for joint optimization. This can comprehensively utilize image information from different viewing directions, reduce pose estimation errors caused by insufficient texture, limited field of view, or changes in motion direction in a single local view, thereby improving the camera pose tracking accuracy and trajectory consistency in panoramic image sequences.

[0086] In summary, the method of this invention can robustly handle highly distorted panoramic images and extremely challenging outdoor unbounded scenes, and the generated optical trajectories and 3D maps are far superior to existing technologies, making it highly valuable for industrial deployment.

[0087] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A panoramic 3D mapping and positioning method based on multi-plane projection and adjoint optimization, characterized in that, Includes the following steps: 1) Obtain the panoramic image sequence of the environment to be mapped, and perform multi-plane projection on each panoramic image in the panoramic image sequence to obtain multiple local perspective views corresponding to each panoramic image. An initial 3D Gaussian map is generated from multiple local perspective views corresponding to the first frame of the panoramic image. 2) Based on the view poses of the current 3D Gaussian map and the local perspective views of the current frame panoramic image, generate the prediction images corresponding to the local perspective views of the current frame panoramic image. Obtain the pose gradient of each local perspective view according to the photometric error loss between the prediction image and the corresponding local perspective view. Then, use the Lie algebra adjoint matrix to obtain the main pose estimate of the current frame panoramic image. 3) Update the view pose of the current frame panoramic image based on the master pose obtained in step 2), and repeat step 2) until the preset number of rounds to obtain the master pose result of the current frame panoramic image; determine whether the current frame panoramic image is a keyframe panoramic image based on the master pose result; if so, generate new 3D Gaussian points based on each local perspective view of the current frame panoramic image and update the current 3D Gaussian map, and then execute step 4). Conversely, return to step 2) to process the next frame of the panoramic image; 4) Update the keyframe sliding window based on the keyframe panoramic image. Based on the panoramic image in the keyframe sliding window, perform joint iterative optimization and adaptive densification on the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map to obtain the updated 3D Gaussian map. 5) Return to step 2) to process the next frame of the panoramic image until all panoramic images have been processed to obtain the final 3D Gaussian map and the principal pose results of each panoramic image.

2. The method according to claim 1, characterized in that, In step 2), multiple local perspective views of each panoramic image share the same camera optical center and have a fixed relative pose relationship; Partial perspective views include the front view, back view, left view, and right view; The view poses of each local perspective view of the current frame panoramic image include the main pose of the front view and the view poses of the back view, left view, and right view. The main pose result of the previous frame panoramic image is used as the initial value of the main pose of the current frame panoramic image. The view pose of the front view is determined by the main pose of the current frame panoramic image. The view poses of the back view, left view, and right view are obtained by combining the main pose of the front view with the fixed relative pose relationship.

3. The method according to claim 1, characterized in that, In step 2), generating the predicted image corresponding to each local perspective view of the current frame panoramic image includes: projecting the three-dimensional Gaussian points in the current three-dimensional Gaussian map onto the image plane of the local perspective view according to the view pose of each local perspective view of the current frame panoramic image, and generating the predicted image corresponding to each local perspective view according to the projection position, opacity and color attributes of each three-dimensional Gaussian point.

4. The method according to claim 1, characterized in that, In step 2), obtaining the pose gradient of each local perspective view includes: applying a preset pose perturbation to the view pose of each local perspective view based on the photometric error loss between the predicted image and the corresponding local perspective view, and obtaining the pose gradient corresponding to each local perspective view based on the change in photometric error loss before and after the perturbation.

5. The method according to claim 1, characterized in that, In step 2), obtaining the principal pose of the current frame panoramic image using the Lie algebra adjoint matrix includes: constructing a Lie algebra adjoint matrix based on the fixed relative pose relationship between each local perspective view, and using the Lie algebra adjoint matrix to transform the pose gradients corresponding to each local perspective view to the coordinate system corresponding to the principal pose of the current frame panoramic image; fusing the transformed pose gradients into a joint pose gradient, then converting the joint pose gradient into a principal pose update amount according to a preset step size, and updating the principal pose of the current frame panoramic image based on the principal pose update amount to obtain the updated principal pose.

6. The method according to claim 1, characterized in that, In step 3), determining whether the current frame panoramic image is a keyframe panoramic image based on the main pose result includes: obtaining the frame interval between the current frame panoramic image and the previous keyframe panoramic image, and determining the relative pose change of the current frame panoramic image relative to the previous keyframe panoramic image based on the main pose result of the current frame panoramic image and the main pose result of the previous keyframe panoramic image; when the frame interval exceeds a preset frame threshold and the relative pose change meets a preset spatial motion condition, the current frame panoramic image is determined to be a keyframe panoramic image. The relative pose change includes relative translation distance and relative rotation angle. The relative pose change satisfies the preset spatial motion condition that the relative translation distance is greater than a preset translation threshold or the relative rotation angle is greater than a preset rotation threshold.

7. The method according to claim 1, characterized in that, In step 3), generating new 3D Gaussian points and updating the current 3D Gaussian map based on each local perspective view of the current frame panoramic image includes: performing monocular depth estimation on each local perspective view of the current frame panoramic image to obtain the predicted depth value corresponding to each pixel in each local perspective view; and determining the target depth value of each pixel in each local perspective view based on the predicted depth value, the preset maximum effective depth threshold, and the preset fixed distance constant. Based on the target depth value, the main pose result of the current frame panoramic image, and the fixed relative pose relationship between each local perspective view, the pixel points are back-projected onto the world coordinate system to generate new three-dimensional Gaussian points; the new three-dimensional Gaussian points are inserted into the current three-dimensional Gaussian map to update the current three-dimensional Gaussian map.

8. The method according to claim 1, characterized in that, In step 4), updating the keyframe sliding window based on the keyframe panoramic image includes: determining whether the number of keyframe panoramic images stored in the keyframe sliding window has reached a preset maximum capacity; if the preset maximum capacity has not been reached, inserting the current keyframe panoramic image and its principal pose result into the keyframe sliding window; if the preset maximum capacity has been reached, determining the multi-view intersection-union ratio between the current keyframe panoramic image and each historical keyframe panoramic image in the keyframe sliding window, removing the historical keyframe panoramic image with the highest multi-view intersection-union ratio from the keyframe sliding window, and inserting the current keyframe panoramic image and its principal pose result into the keyframe sliding window; wherein, the multi-view intersection-union ratio is the ratio of the sum of the intersection pixels of the effective visible areas of two keyframe panoramic images in each local perspective view to the sum of the union pixels.

9. The method according to claim 1, characterized in that, In step 4), the geometric and appearance attributes of the three-dimensional Gaussian points in the current three-dimensional Gaussian map are jointly iteratively optimized and adaptively densified based on the panoramic image in the keyframe sliding window, including: randomly selecting a keyframe panoramic image from the keyframe sliding window, and selecting an optimized view from multiple local perspective views corresponding to the selected keyframe panoramic image. Based on the current 3D Gaussian map, a predicted image corresponding to the optimized view is generated. Then, based on the photometric error loss between the predicted image and the optimized view, the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map are jointly iteratively optimized. During the joint iterative optimization process, the 3D Gaussian points to be densified are determined according to the degree of influence of each 3D Gaussian point on the photometric error loss, and cloning or splitting operations are performed on these points. Finally, 3D Gaussian points that meet the rejection criteria are removed based on their opacity, resulting in an updated 3D Gaussian map.

10. A panoramic 3D mapping and positioning system based on multi-plane projection and adjoint optimization, implementing the method of claim 1, characterized in that, include: The multi-projection module is used to acquire a panoramic image sequence of the environment to be mapped, and to perform multi-projection on each panoramic image in the panoramic image sequence to obtain multiple local perspective views corresponding to each panoramic image. The pose estimation module is used to generate the view poses of each local perspective view based on the current 3D Gaussian map and the current frame panoramic image, and the main pose result of the current frame panoramic image; wherein, an initial 3D Gaussian map is generated from multiple local perspective views corresponding to the first frame panoramic image. The keyframe mapping module is used to determine whether the current frame panoramic image is a keyframe panoramic image based on the main pose result. If it is, it generates new three-dimensional Gaussian points based on each local perspective view of the current frame panoramic image and updates the current three-dimensional Gaussian map. The mapping optimization and result output module is used to determine whether the current frame panoramic image is a keyframe panoramic image based on the principal pose result. When the current frame panoramic image is a keyframe panoramic image, it generates new 3D Gaussian points based on its local perspective views and updates the current 3D Gaussian map and the keyframe sliding window. Based on the panoramic image in the keyframe sliding window, it performs joint iterative optimization and adaptive densification processing on the geometric and appearance attributes of the 3D Gaussian points in the current 3D Gaussian map, and outputs the final 3D Gaussian map and the principal pose results of each panoramic image.

Images