A 3DGS visual relocalization method, system, terminal, and storage medium based on intrinsic image decomposition.

Abstract

This invention relates to the field of image localization technology, and discloses a 3DGS visual relocalization method, system, terminal, and storage medium based on intrinsic image decomposition. The method includes: acquiring multi-view scene image data; inputting the multi-view scene image data into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model; acquiring a query image to be localized and a set of landmarks; inputting the query image into the intrinsic 3DGS map model for decomposition to obtain a target albedo map; performing feature matching based on the target albedo and the set of landmarks to obtain an initial pose; rendering the initial pose based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map; and adjusting the initial pose based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the localization camera pose. This invention achieves accurate visual localization by visually relocalizing a query image using an intrinsic 3DGS map model.

Description

Technical Field

[0001] This invention relates to the field of image localization technology, and in particular to a 3DGS visual relocalization method, system, terminal, and computer-readable storage medium based on intrinsic image decomposition. Background Technology

[0002] Visual relocalization, which accurately calculates the camera's position and orientation in 3D space (six degrees of freedom pose) based on a query image, is a core technology for cutting-edge applications such as Augmented Reality (AR), autonomous driving, and robot navigation. Visual relocalization relies on a pre-built 3D scene map and completes localization by matching the query image against this map. In recent years, 3D Gaussian Splatting (3DGS) has become a mainstream technology for building high-precision scene maps due to its ability to render photorealistic 3D scenes in real time.

[0003] Currently, traditional 3DGS-based visual relocalization methods mainly rely on photometric consistency, estimating pose by matching visual features extracted under stable lighting or by minimizing the color error between the rendered image and the query image. For example, some methods establish 2D-3D correspondences by rendering synthetic views, while others directly optimize the localization camera pose in the RGB image domain. Furthermore, to improve the representational power of 3DGS, feature Gaussian techniques have emerged, distilling high-dimensional features from 2D pre-trained models (e.g., CLIP or DINO) onto each Gaussian unit to support downstream tasks such as scene segmentation or feature matching.

[0004] Currently, traditional 3DGS-based visual relocalization methods mainly rely on photometric consistency. However, appearance inconsistencies caused by changes in illumination lead to a significant drop in localization accuracy when illumination changes drastically, making it impossible to meet the application requirements of all-weather, all-scenario applications. This has become an urgent problem to be solved.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a 3DGS visual relocalization method, system, terminal, and computer-readable storage medium based on intrinsic image decomposition. This invention aims to solve the problem of appearance inconsistency caused by changes in illumination in the prior art, which leads to a severe decrease in visual positioning accuracy when illumination changes drastically, resulting in the inability to meet the application requirements of all weather and all scenarios.

[0007] To achieve the above objectives, the present invention provides a 3DGS visual relocalization method based on intrinsic image decomposition, the 3DGS visual relocalization method based on intrinsic image decomposition comprising the following steps:

[0008] Acquire multi-view scene image data, perform image decomposition on the multi-view scene image data through a preset image decomposition model to obtain intrinsic optical properties, input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training, and obtain an intrinsic 3DGS map model.

[0009] The query image and landmark set to be located are obtained. The query image is input into the intrinsic 3DGS map model for decomposition to obtain the target albedo map. Feature matching is performed based on the target albedo and the landmark set to obtain the initial pose.

[0010] The initial pose is rendered based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map. The initial pose is then adjusted based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose.

[0011] Optionally, in the 3DGS visual relocalization method based on intrinsic image decomposition, the intrinsic optical properties include an albedo map, a shading map, and a residual map.

[0012] The process of acquiring multi-view scene image data, performing image decomposition on the multi-view scene image data using a preset image decomposition model to obtain intrinsic optical properties, and inputting the intrinsic optical properties into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model, specifically includes:

[0013] Acquire multi-view scene image data, and perform image decomposition on the multi-view scene image data using a preset image decomposition model to obtain an albedo map, a shading map, and a residual map;

[0014] The albedo map, the shading map, and the residual map are rendered to obtain an RGB image;

[0015] The RGB image is input into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model.

[0016] Optionally, the 3DGS visual relocalization method based on intrinsic image decomposition, wherein the step of inputting the RGB image into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model, further includes:

[0017] The loss is calculated on the albedo map, the shading map and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image;

[0018] The intrinsic attribute loss and the reconstructed RGB image are input into the intrinsic 3DGS map model for backpropagation to obtain the optimal intrinsic 3DGS map model.

[0019] Optionally, in the 3DGS visual relocalization method based on intrinsic image decomposition, the intrinsic attribute loss includes L distance;

[0020] The step of calculating the loss of the albedo map, the shading map, and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image specifically includes:

[0021] The difference between the albedo map, the shading map, and the residual map is calculated based on the preset image decomposition model to obtain the L distance;

[0022] Multiply the albedo map and the shading map and add the residual map to obtain the reconstructed RGB image;

[0023] The L distance is used to optimize the loss function of the intrinsic 3DGS map model.

[0024] Optionally, in the 3DGS visual relocalization method based on intrinsic image decomposition, the initial pose includes a six-degree-of-freedom pose.

[0025] The process of acquiring the query image and landmark set to be located, inputting the query image into the intrinsic 3DGS map model for decomposition to obtain a target albedo map, and performing feature matching based on the target albedo and the landmark set to obtain the initial pose specifically includes:

[0026] Obtain the query image to be located and the set of landmarks, and extract features from the set of landmarks according to the matching algorithm to obtain the landmarks;

[0027] The query image is input into the intrinsic 3DGS map model for decomposition to obtain the target albedo map;

[0028] The landmarks are matched with the target albedo map to obtain sparse matching results.

[0029] The pose of the sparse matching results is estimated by using the PnP-RANSAC algorithm to obtain the six-degree-of-freedom pose.

[0030] The six-degree-of-freedom pose is used to view the position and perspective of the scene image.

[0031] Optionally, the 3DGS visual relocalization method based on intrinsic image decomposition, wherein the step of performing feature matching between the landmark points and the target albedo map to obtain sparse matching results specifically includes:

[0032] Key points were obtained by detecting the target albedo map using Efficient LoFTR feature matching technology;

[0033] In the albedo domain, feature matching is performed between the landmark points and the key points to obtain sparse matching results.

[0034] Optionally, the 3DGS visual relocalization method based on intrinsic image decomposition, wherein rendering the initial pose based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map, and adjusting the initial pose according to the dense candidate albedo map, the depth map, and the target albedo map to obtain the localization camera pose, specifically includes:

[0035] The initial pose is rendered based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map. Dense feature matching is performed between the dense candidate albedo map and the target albedo map to obtain the point correspondence.

[0036] The point correspondence is upgraded based on the depth map to obtain the target point correspondence. The initial pose is then adjusted based on the target point correspondence to obtain the positioning camera pose.

[0037] Furthermore, to achieve the above objectives, the present invention also provides a 3DGS visual relocalization system based on intrinsic image decomposition, wherein the 3DGS visual relocalization system based on intrinsic image decomposition:

[0038] The model training module is used to acquire multi-view scene image data, perform image decomposition on the multi-view scene image data through a preset image decomposition model to obtain intrinsic optical properties, and input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training to obtain an intrinsic 3DGS map model.

[0039] The feature matching module is used to acquire the query image to be located and the set of landmarks, input the query image into the internal 3DGS map model for decomposition to obtain the target albedo map, and perform feature matching based on the target albedo and the set of landmarks to obtain the initial pose;

[0040] The positioning adjustment module is used to render the initial pose based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map, and adjust the initial pose according to the dense candidate albedo map, the depth map and the target albedo map to obtain the positioning camera pose.

[0041] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a 3DGS visual relocalization program based on intrinsic image decomposition, and the 3DGS visual relocalization program based on intrinsic image decomposition, when executed by a processor, implements the steps of the 3DGS visual relocalization method based on intrinsic image decomposition as described above.

[0042] In this invention, multi-view scene image data is acquired, and the multi-view scene image data is decomposed using a preset image decomposition model to obtain intrinsic optical properties. These intrinsic optical properties are then input into a 3D Gaussian sputtering model for training, resulting in an intrinsic 3DGS map model. A query image and a set of landmarks to be located are acquired, and the query image is input into the intrinsic 3DGS map model for decomposition, resulting in a target albedo map. Feature matching is performed based on the target albedo and the set of landmarks to obtain an initial pose. The initial pose is then rendered based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map. The initial pose is adjusted based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose. This invention achieves accurate visual positioning by visually relocalizing a query image using an intrinsic 3DGS map model. Attached Figure Description

[0043] Figure 1 This is a flowchart of a preferred embodiment of the 3DGS visual relocalization method based on intrinsic image decomposition of the present invention;

[0044] Figure 2 This is a schematic diagram of the overall framework of a preferred embodiment of the 3DGS visual relocalization method based on intrinsic image decomposition of the present invention;

[0045] Figure 3 This is a schematic diagram of the map construction framework of a preferred embodiment of the 3DGS visual relocalization method based on intrinsic image decomposition of the present invention;

[0046] Figure 4 This is a schematic diagram of the localization framework of a preferred embodiment of the 3DGS visual relocalization method based on intrinsic image decomposition of the present invention;

[0047] Figure 5 This is a structural diagram of a preferred embodiment of the 3DGS visual relocalization system based on intrinsic image decomposition of the present invention;

[0048] Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the device of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] Currently, traditional 3DGS-based visual relocalization methods mainly rely on photometric consistency. However, appearance inconsistencies caused by lighting changes lead to a significant drop in positioning accuracy when lighting conditions change drastically, making it impossible to meet the application requirements of all-weather, all-scene applications. Therefore, a 3DGS visual relocalization method based on intrinsic image decomposition is needed. Intrinsic albedo decomposition and 3D Gaussian sputtering modeling are used to estimate and optimize the camera pose of the query image, achieving a method for robust and accurate visual localization in the face of lighting changes.

[0051] The 3DGS visual relocalization method based on intrinsic image decomposition described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the 3DGS visual relocalization method based on intrinsic image decomposition includes the following steps:

[0052] Step S10: Acquire multi-view scene image data, perform image decomposition on the multi-view scene image data using a preset image decomposition model to obtain intrinsic optical properties, input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training, and obtain an intrinsic 3DGS map model.

[0053] Step S10 includes:

[0054] Step S11: Acquire multi-view scene image data, and perform image decomposition on the multi-view scene image data using a preset image decomposition model to obtain an albedo map, a shading map, and a residual map.

[0055] Step S12: Render the albedo map, the shading map, and the residual map to obtain an RGB image;

[0056] Step S13: Input the RGB image into the three-dimensional Gaussian sputtering model for training to obtain the intrinsic 3DGS map model.

[0057] Specifically, multi-view scene image data is acquired (for each multi-view image in the multi-view scene image data, the multi-view scene image data includes: normal indoor lighting (lighted) and dim or unlighted lighting with only a small amount of ambient light (unlighted)). The multi-view scene image data is decomposed by a preset image decomposition model to obtain an albedo map, a shading map and a residual map (the intrinsic image decomposition technology (preset image decomposition model) decomposes it to generate the corresponding albedo map, shading map and residual map). The albedo map, the shading map and the residual map are rendered to obtain an RGB image. The RGB image is input into a three-dimensional Gaussian sputtering model for training to obtain an intrinsic 3DGS map model.

[0058] In this embodiment, the Diffuse Albedo is a three-dimensional, view-independent albedo vector that represents the stable, inherent surface color of an object. The Shading map is a set of low-order spherical harmonic coefficients used to represent the diffuse reflection brightness and darkness effects formed by ambient light, indirect lighting, etc., that vary with the viewpoint. The Residual map is a set of high-order spherical harmonic coefficients used to capture non-Lambertian effects that vary drastically and are strongly view-dependent, such as highlights.

[0059] For example, such as Figure 2 As shown, in the offline mapping stage, the system utilizes 3D Gaussian sputtering (3DGS) technology to learn and construct a semantically rich feature 3DGS scene map from the collected data. This map can not only render high-quality color images by calculating loss, but also decouple material attribute images such as albedo (albedo, color difference, and residual) and process lighting images, thus forming a scene prior knowledge base (normal lighting images and rendered lighting images) with strong representation capabilities for ambient lighting and material attribute invariants. Subsequently, in the online localization stage, when a query image with weak lighting is input... First, a material attribute image is extracted from an effect model. Then, the localization process employs a two-stage strategy from coarse to fine. In the sparse stage, an initial pose is calculated based on a small number of 2D-2D matching points using the PnP-RANSAC (PnP+RANSAC, RANSAC&PnP) algorithm. In the dense stage, based on this initial pose and combined with dense information such as the depth map, the PnP-RANSAC algorithm is used for fine optimization, ultimately outputting a high-precision dense pose. This achieves stable and accurate pose estimation under complex lighting conditions. For example: Figure 2 In the diagram, (a) represents the offline 3DGS mapping stage: a 3DGS scene map containing decoupling features is trained using a teacher-student model on normally lit images. Figure 2 In the diagram, (b) represents the online localization stage: the optimized dense pose is obtained by using a sparse-dense pipeline and the PnP-RANSAC algorithm.

[0060] In this embodiment, as Figure 3 As shown, in the offline 3DGS mapping stage: input multi-view dataset images are used to generate albedo, shading, and residual pseudo-real values ​​using a 2D teacher model. At the same time, the 3DGS student model containing intrinsic attributes outputs the corresponding feature map through a differentiable renderer. By jointly optimizing the pseudo-real value supervision loss and the rendered image reconstruction loss, the intrinsic 3DGS map model is trained.

[0061] Following step S13, the following is also included:

[0062] Step S14: Perform loss calculation on the albedo map, the shading map and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image;

[0063] Step S15: Input the intrinsic attribute loss and the reconstructed RGB image into the intrinsic 3DGS map model for backpropagation to obtain the optimal intrinsic 3DGS map model.

[0064] Specifically, loss calculations are performed on the albedo map, the shading map, and the residual map according to the preset image decomposition model to obtain intrinsic attribute loss and reconstructed RGB image. The intrinsic attribute loss and the reconstructed RGB image are then input into the intrinsic 3DGS map model for backpropagation (by backpropagating the total loss (intrinsic attribute loss and reconstructed RGB image) and combining it with the standard 3DGS adaptive density control strategy, the geometry and intrinsic attributes of all Gaussian elements are optimized end-to-end to obtain the optimal intrinsic 3DGS map model.

[0065] Step S14 includes:

[0066] Step S141: Calculate the difference between the albedo map, the shading map, and the residual map according to the preset image decomposition model to obtain the L distance;

[0067] Step S142: Multiply the albedo map and the shading map and add the residual map to obtain the reconstructed RGB image.

[0068] Specifically, the difference between the albedo map, the shading map, and the residual map is calculated according to the preset image decomposition model to obtain the L distance. The albedo map and the shading map are multiplied and the residual map is added to obtain the reconstructed RGB image. The intrinsic attribute loss includes the L distance, which is used to optimize the loss function of the intrinsic 3DGS map model.

[0069] In this embodiment, the intrinsic attribute loss is used to minimize the L-distance between the albedo, shading, and residual map rendered by the student model and the corresponding "ground truth (albedo map, shading map, and residual map)" generated by the teacher model (3D Gaussian sputtering model). The reconstructed image loss is obtained by multiplying the rendered albedo and shading map and adding the residual map to obtain the reconstructed RGB image, and minimizing the L1 (L1 Loss, mean absolute error loss) and D-SSIM (D-SSIM Loss, structural similarity index) losses between it and the original training image to ensure rendering quality.

[0070] Step S20: Obtain the query image and landmark set to be located, input the query image into the intrinsic 3DGS map model for decomposition to obtain the target albedo map, and perform feature matching based on the target albedo and the landmark set to obtain the initial pose.

[0071] Step S20 includes:

[0072] Step S21: Obtain the query image to be located and the set of landmarks, and extract features from the set of landmarks according to the matching algorithm to obtain the landmarks;

[0073] Step S22: Input the query image into the intrinsic 3DGS map model for decomposition to obtain the target albedo map;

[0074] Step S23: Perform feature matching between the landmark and the target albedo map to obtain sparse matching results;

[0075] Step S24: Use the PnP-RANSAC algorithm to estimate the pose of the sparse matching results to obtain the six-degree-of-freedom pose.

[0076] Specifically, the query image to be located and a set of landmarks are obtained. Features are extracted from the set of landmarks using a matching algorithm to obtain the landmarks. The query image is input into the intrinsic 3DGS map model for decomposition to obtain a target albedo map (firstly, the target albedo map is extracted using a teacher model). The landmarks are then matched with the target albedo map to obtain sparse matching results. The sparse matching results are then used to estimate the pose using the PnP-RANSAC (Perspective-n-Point Random Sample Consensus) algorithm to obtain a six-degree-of-freedom pose (a robust initial camera pose is obtained by solving the sparse matching results using the PnP-RANSAC algorithm). The six-degree-of-freedom pose is used to view the position and perspective of the scene image, and the initial pose includes the six-degree-of-freedom pose.

[0077] In this embodiment, to achieve efficient matching, a sparse but high-quality set of landmarks is first selected from the complete intrinsic 3DGS map. Specifically, an "albedo consistency score" is calculated for each Gaussian primitive. The albedo consistency score measures the consistency of its albedo attribute with the 2D pseudo-true value in all visible views. Then, through an anchor-guided selection strategy (matching algorithm), primitives with high scores are preferentially selected as landmarks while ensuring spatial uniformity.

[0078] Step S23 includes:

[0079] Step S231: Detect the target albedo map using Efficient LoFTR feature matching technology to obtain key points;

[0080] Step S232: In the albedo domain, perform feature matching between the landmark points and the key points to obtain sparse matching results.

[0081] Specifically, the target albedo map is detected using Efficient LoFTR feature matching technology (scene-specific detector) to obtain key points (2D key points corresponding to the above 3D landmarks are detected in the albedo map). In the albedo domain, the landmarks and key points are feature matched to obtain sparse matching results (in the albedo domain where the illumination is constant, these key points (2D key points) are feature matched with the 3D landmarks).

[0082] Step S30: Render the initial pose based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map. Adjust the initial pose according to the dense candidate albedo map, the depth map and the target albedo map to obtain the positioning camera pose.

[0083] Step S30 includes:

[0084] Step S31: Render the initial pose based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map. Perform dense feature matching on the dense candidate albedo map and the target albedo map to obtain the point correspondence.

[0085] Step S32: Perform dimensionality upscaling on the point correspondence based on the depth map to obtain the target point correspondence, and adjust the initial pose based on the target point correspondence to obtain the positioning camera pose.

[0086] Specifically, the initial pose is rendered based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map. Dense feature matching is performed on the dense candidate albedo map and the target albedo map to obtain point correspondence (the correspondence between points in the dense candidate albedo map and the target albedo map). The point correspondence is then upgraded based on the depth map (from 2D-2D to 2D-3D) to obtain the target point correspondence. The initial pose is then adjusted based on the target point correspondence to obtain the positioning camera pose (the final, higher-precision positioning camera pose is obtained again using the PnP-RANSAC algorithm).

[0087] In this embodiment, as Figure 4 As shown, using the initial pose obtained in the sparse stage, dense candidate albedo maps and depth maps corresponding to the pose are rendered from the internal 3DGS map. Coarse-to-fine dense feature matching is performed between the target albedo map of the query image and the albedo map of the candidate view. The resulting dense 2D-2D matching is improved to a 2D-3D correspondence using the depth map, and the final, higher-precision localization camera pose is solved again using the PnP-RANSAC (RANSAC & PnP) algorithm. (Online localization stage: In the sparse localization stage, the albedo map is extracted from the input query image, 2D keypoints are detected by the scene detector and matched with 3D landmarks, and the initial sparse pose is solved using PnP-RANSAC. In the dense localization stage, the albedo map and depth map are rendered using the initial pose, dense matching is performed, and the result is improved to a 2D-3D correspondence. The optimized dense pose is solved again using PnP-RANSAC.)

[0088] Furthermore, inputting multi-view dataset images, the model obtains albedo, shading, and residual maps through intrinsic image decomposition, and incorporates these as new features into 3DGS primitives to construct an intrinsic 3DGS scene representation. During training, the model utilizes a differentiable renderer to output various pseudo-intrinsic attribute losses and reconstructed RGB images in parallel. Joint optimization is achieved through the total loss function. (Combined with RGB loss) and intrinsic attribute loss This is performed to learn a lighting-invariant 3DGS scene representation;

[0089] The calculation formula is as follows:

[0090] ;

[0091] ;

[0092] in, These represent the weighting coefficients of the reconstructed RGB image. The weighting coefficients represent the intrinsic attribute loss. This represents the weighting coefficients of the albedo map. This represents the weighting coefficients of the coloring map. This represents the weighting coefficients of the residual plot. Represents the loss function. Represents an albedo map. Represents a colorized image. Representing the residual plot, This represents the true value of the albedo map. Represents the true value of the shaded map. This represents the true value of the residual plot.

[0093] Furthermore, such as Figure 5 As shown, based on the above-mentioned 3DGS visual relocalization method based on intrinsic image decomposition, the present invention also provides a 3DGS visual relocalization system based on intrinsic image decomposition, wherein the 3DGS visual relocalization system based on intrinsic image decomposition includes:

[0094] Model training module 51 is used to acquire multi-view scene image data, perform image decomposition on the multi-view scene image data through a preset image decomposition model to obtain intrinsic optical properties, and input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training to obtain an intrinsic 3DGS map model.

[0095] The feature matching module 52 is used to acquire the query image to be located and the set of landmarks, input the query image into the internal 3DGS map model for decomposition to obtain the target albedo map, and perform feature matching based on the target albedo and the set of landmarks to obtain the initial pose;

[0096] The positioning adjustment module 53 is used to render the initial pose based on the internal 3DGS map model to obtain a dense candidate albedo map and a depth map, and adjust the initial pose according to the dense candidate albedo map, the depth map and the target albedo map to obtain the positioning camera pose.

[0097] Furthermore, such as Figure 6 As shown, based on the above-mentioned 3DGS visual relocalization method and system based on intrinsic image decomposition, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0098] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a 3DGS visual relocalization program 40 based on intrinsic image decomposition, which can be executed by the processor 10 to implement the 3DGS visual relocalization method based on intrinsic image decomposition in this application.

[0099] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the 3DGS visual relocalization method based on intrinsic image decomposition.

[0100] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminals communicate with each other via a system bus.

[0101] In one embodiment, when the processor 10 executes the 3DGS visual relocalization program 40 based on intrinsic image decomposition in the memory 20, the following steps are performed:

[0102] Acquire multi-view scene image data, perform image decomposition on the multi-view scene image data through a preset image decomposition model to obtain intrinsic optical properties, input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training, and obtain an intrinsic 3DGS map model.

[0103] The query image and landmark set to be located are obtained. The query image is input into the intrinsic 3DGS map model for decomposition to obtain the target albedo map. Feature matching is performed based on the target albedo and the landmark set to obtain the initial pose.

[0104] The initial pose is rendered based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map. The initial pose is then adjusted based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose.

[0105] The intrinsic optical properties include an albedo map, a shading map, and a residual map;

[0106] The process of acquiring multi-view scene image data, performing image decomposition on the multi-view scene image data using a preset image decomposition model to obtain intrinsic optical properties, and inputting the intrinsic optical properties into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model, specifically includes:

[0107] Acquire multi-view scene image data, and perform image decomposition on the multi-view scene image data using a preset image decomposition model to obtain an albedo map, a shading map, and a residual map;

[0108] The albedo map, the shading map, and the residual map are rendered to obtain an RGB image;

[0109] The RGB image is input into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model.

[0110] The step of inputting the RGB image into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model further includes:

[0111] The loss is calculated on the albedo map, the shading map and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image;

[0112] The intrinsic attribute loss and the reconstructed RGB image are input into the intrinsic 3DGS map model for backpropagation to obtain the optimal intrinsic 3DGS map model.

[0113] The intrinsic attribute loss includes the L distance;

[0114] The step of calculating the loss of the albedo map, the shading map, and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image specifically includes:

[0115] The difference between the albedo map, the shading map, and the residual map is calculated based on the preset image decomposition model to obtain the L distance;

[0116] Multiply the albedo map and the shading map and add the residual map to obtain the reconstructed RGB image;

[0117] The L distance is used to optimize the loss function of the intrinsic 3DGS map model.

[0118] The initial pose includes a six-degree-of-freedom pose;

[0119] The process of acquiring the query image and landmark set to be located, inputting the query image into the intrinsic 3DGS map model for decomposition to obtain a target albedo map, and performing feature matching based on the target albedo and the landmark set to obtain the initial pose specifically includes:

[0120] Obtain the query image to be located and the set of landmarks, and extract features from the set of landmarks according to the matching algorithm to obtain the landmarks;

[0121] The query image is input into the intrinsic 3DGS map model for decomposition to obtain the target albedo map;

[0122] The landmarks are matched with the target albedo map to obtain sparse matching results.

[0123] The pose of the sparse matching results is estimated by using the PnP-RANSAC algorithm to obtain the six-degree-of-freedom pose.

[0124] The six-degree-of-freedom pose is used to view the position and perspective of the scene image.

[0125] Specifically, the step of performing feature matching between the landmark points and the target albedo map to obtain sparse matching results includes:

[0126] Key points were obtained by detecting the target albedo map using Efficient LoFTR feature matching technology;

[0127] In the albedo domain, feature matching is performed between the landmark points and the key points to obtain sparse matching results.

[0128] Specifically, the process of rendering the initial pose based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map, and adjusting the initial pose based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose, includes:

[0129] The initial pose is rendered based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map. Dense feature matching is performed between the dense candidate albedo map and the target albedo map to obtain the point correspondence.

[0130] The point correspondence is upgraded based on the depth map to obtain the target point correspondence. The initial pose is then adjusted based on the target point correspondence to obtain the positioning camera pose.

[0131] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a 3DGS visual relocalization program based on intrinsic image decomposition, and the 3DGS visual relocalization program based on intrinsic image decomposition, when executed by a processor, implements the steps of the 3DGS visual relocalization method based on intrinsic image decomposition as described above.

[0132] In summary, this invention provides a 3DGS visual relocalization method, system, terminal, and storage medium based on intrinsic image decomposition. The method includes: acquiring multi-view scene image data; performing image decomposition on the multi-view scene image data using a preset image decomposition model to obtain intrinsic optical properties; inputting the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training to obtain an intrinsic 3DGS map model; acquiring a query image to be located and a set of landmarks; inputting the query image into the intrinsic 3DGS map model for decomposition to obtain a target albedo map; performing feature matching based on the target albedo and the set of landmarks to obtain an initial pose; rendering the initial pose based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map; and adjusting the initial pose based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose. This invention achieves accurate visual positioning by visually relocalizing a query image using an intrinsic 3DGS map model.

[0133] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal system that includes that element.

[0134] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0135] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A 3DGS visual relocalization method based on intrinsic image decomposition, characterized in that, The 3DGS visual relocalization method based on intrinsic image decomposition includes: Acquire multi-view scene image data, perform image decomposition on the multi-view scene image data through a preset image decomposition model to obtain intrinsic optical properties, input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training, and obtain an intrinsic 3DGS map model. The query image and landmark set to be located are obtained. The query image is input into the intrinsic 3DGS map model for decomposition to obtain the target albedo map. Feature matching is performed based on the target albedo map and the landmark set to obtain the initial pose. The initial pose is rendered based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map. The initial pose is then adjusted based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose. The intrinsic optical properties include albedo maps, shading maps, and residual maps; The process of acquiring multi-view scene image data, performing image decomposition on the multi-view scene image data using a preset image decomposition model to obtain intrinsic optical properties, and inputting the intrinsic optical properties into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model, specifically includes: Acquire multi-view scene image data, and perform image decomposition on the multi-view scene image data using a preset image decomposition model to obtain an albedo map, a shading map, and a residual map; The albedo map, the shading map, and the residual map are rendered to obtain an RGB image; The RGB image is input into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model.

2. The 3DGS visual relocalization method based on intrinsic image decomposition according to claim 1, characterized in that, The step of inputting the RGB image into a 3D Gaussian sputtering model for training to obtain an intrinsic 3DGS map model further includes: The loss is calculated on the albedo map, the shading map and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image; The intrinsic attribute loss and the reconstructed RGB image are input into the intrinsic 3DGS map model for backpropagation to obtain the optimal intrinsic 3DGS map model.

3. The 3DGS visual relocalization method based on intrinsic image decomposition according to claim 2, characterized in that, The intrinsic attribute loss includes the L distance; The step of calculating the loss of the albedo map, the shading map, and the residual map according to the preset image decomposition model to obtain the intrinsic attribute loss and the reconstructed RGB image specifically includes: The difference between the albedo map, the shading map, and the residual map is calculated based on the preset image decomposition model to obtain the L distance; Multiply the albedo map and the shading map and add the residual map to obtain the reconstructed RGB image; The L distance is used to optimize the loss function of the intrinsic 3DGS map model.

4. The 3DGS visual relocalization method based on intrinsic image decomposition according to claim 1, characterized in that, The initial pose includes a six-degree-of-freedom pose; The process of acquiring the query image and landmark set to be located, inputting the query image into the intrinsic 3DGS map model for decomposition to obtain a target albedo map, and performing feature matching based on the target albedo map and the landmark set to obtain the initial pose specifically includes: Obtain the query image to be located and the set of landmarks, and extract features from the set of landmarks according to the matching algorithm to obtain the landmarks; The query image is input into the intrinsic 3DGS map model for decomposition to obtain the target albedo map; The landmarks are matched with the target albedo map to obtain sparse matching results. The pose of the sparse matching results is estimated by using the PnP-RANSAC algorithm to obtain the six-degree-of-freedom pose. The six-degree-of-freedom pose is used to view the position and perspective of the scene image.

5. The 3DGS visual relocalization method based on intrinsic image decomposition according to claim 4, characterized in that, The step of performing feature matching between the landmarks and the target albedo map to obtain sparse matching results specifically includes: Key points were obtained by detecting the target albedo map using Efficient LoFTR feature matching technology; In the albedo domain, feature matching is performed between the landmark points and the key points to obtain sparse matching results.

6. The 3DGS visual relocalization method based on intrinsic image decomposition according to claim 1, characterized in that, The process of rendering the initial pose based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map, and adjusting the initial pose based on the dense candidate albedo map, the depth map, and the target albedo map to obtain the positioning camera pose, specifically includes: The initial pose is rendered based on the intrinsic 3DGS map model to obtain a dense candidate albedo map and a depth map. Dense feature matching is performed between the dense candidate albedo map and the target albedo map to obtain the point correspondence. The point correspondence is upgraded based on the depth map to obtain the target point correspondence. The initial pose is then adjusted based on the target point correspondence to obtain the positioning camera pose.

7. A 3DGS visual relocalization system based on intrinsic image decomposition, characterized in that, The 3DGS visual relocalization system based on intrinsic image decomposition is applied to the 3DGS visual relocalization method based on intrinsic image decomposition as described in any one of claims 1-6, wherein the 3DGS visual relocalization system based on intrinsic image decomposition comprises: The model training module is used to acquire multi-view scene image data, perform image decomposition on the multi-view scene image data through a preset image decomposition model to obtain intrinsic optical properties, and input the intrinsic optical properties into a three-dimensional Gaussian sputtering model for training to obtain an intrinsic 3DGS map model. The feature matching module is used to acquire the query image to be located and the set of landmarks, input the query image into the internal 3DGS map model for decomposition to obtain the target albedo map, and perform feature matching based on the target albedo and the set of landmarks to obtain the initial pose; The positioning adjustment module is used to render the initial pose based on the inherent 3DGS map model to obtain a dense candidate albedo map and a depth map, and adjust the initial pose according to the dense candidate albedo map, the depth map and the target albedo map to obtain the positioning camera pose.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a 3DGS visual relocalization program based on intrinsic image decomposition stored in the memory and executable on the processor. When the 3DGS visual relocalization program based on intrinsic image decomposition is executed by the processor, it implements the steps of the 3DGS visual relocalization method based on intrinsic image decomposition as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a 3DGS visual relocalization program based on intrinsic image decomposition, which, when executed by a processor, implements the steps of the 3DGS visual relocalization method based on intrinsic image decomposition as described in any one of claims 1-6.

Images