Visual repositioning method and system based on 3D Gaussian scene and storage medium

Through 3D Gaussian sputtering technology and deep neural network, combined with 2D-3D matching points to calculate the camera pose, the problems of sparseness and high computing cost in the existing technology are solved, and efficient and accurate visual repositioning is achieved.

CN118941629AActive Publication Date: 2024-11-12COWA TECHNOLOGY CO LTD +1

Patent Information

Application Number
CN202410947827.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-12
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

It is difficult for the existing technology to perform efficient and accurate visual relocation based on 3D Gaussian scenes. The 3D model built by traditional SLAM methods is sparse and has high computational cost. Neural rendering technology and 3D Gaussian splattering technology have not yet fully utilized their scene reconstruction and rendering capabilities.

Method used

采用3D高斯喷溅技术重建场景,通过深度神经网络提取3D空间点特征,结合2D图像特征,使用图像检索网络和视觉特征编码器构建2D-3D匹配点,利用PnP算法计算相机位姿,实现迭代优化。

Benefits of technology

It realizes efficient and accurate visual repositioning, makes full use of the feature representation ability of 3D Gaussian scenes, improves the accuracy of 2D-3D matching, and reduces computing costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941629A_ABST
    Figure CN118941629A_ABST
Patent Text Reader

Abstract

The invention discloses a visual repositioning method and system based on a 3D Gaussian scene and a storage medium, and the method comprises the steps: S1, initial input: carrying out the training of a training image with a known camera pose based on a 3D Gaussian splashing technology, and obtaining the 3D Gaussian scene; s2, image retrieval: retrieving the image most similar to the query image in the training image, and taking the pose of the image as the initial pose of the camera of the query image; s3, feature coding: extracting 2D point features and 3D point features based on the initial pose; s4, similarity matching: constructing a similarity matching matrix of the 2D point features and the 3D point features, and selecting 2D-3D matching points according to similarity scores; and S5, pose resolving: according to the 2D-3D matching points obtained in the step S4, calculating a current matching pose result as an initial pose of next iteration, and repeating the steps S3 to S5 to obtain a final camera pose. According to the method, the feature representation capability of 3D Gaussian for a scene is fully utilized, and the deep neural network is combined to realize 3D space point feature extraction, so that 2D-3D matching is realized more accurately.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of visual relocation technology, and in particular to a visual relocation method, system and storage medium based on a 3D Gaussian scene. Background Art

[0002] In the technical fields of autonomous driving, robot navigation, AR / VR, etc., scene reconstruction and relocalization technology provide intelligent agents with the ability to perceive and interact with the environment, which is of great significance. Relocalization technology based on visual information has become a research hotspot in the field of computer vision due to its advantages such as rich sensor information and low cost. The key to the relocalization algorithm based on image information is how to reconstruct a high-quality 3D model based on 2D images in order to make full use of 3D scene information and obtain accurate 2D-3D matching points. Due to the characteristics of image sensing information, the traditional SLAM method is relatively sparse in constructing 3D models and has a high computational cost. In recent years, the rise of neural rendering technology and 3D Gaussian splashing technology has provided a more efficient and accurate scene reconstruction method.

[0003] Neural rendering technology and 3D Gaussian splash technology are emerging 3D scene reconstruction technologies in the past two years. They are relatively mature and have rich practical value. 3D Gaussian splash is more efficient than neural rendering and is a hot research topic in the current computer vision field. The CVPR2024 paper The NeRFect Match: Exploring NeRF Features for Visual Localization provides ideas for visual relocalization based on neural rendering technology, and how to perform visual relocalization based on 3D Gaussian scene reconstruction has become a key technical problem to be solved. Summary of the invention

[0004] The present invention expects to obtain 3D scene point features and establish 2D-3D matching points based on the scene reconstruction and image rendering capabilities of the 3D Gaussian splashing technology, thereby achieving efficient and accurate visual repositioning.

[0005] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a visual relocalization method based on a 3D Gaussian scene, which comprises the following steps: S1 initial input: based on the 3D Gaussian splashing technology, a 3D Gaussian scene is obtained by training training images with known camera poses; S2 image retrieval: the image most similar to the query image is retrieved in the training images, and its pose is used as the initial camera pose of the query image; S3 feature encoding: based on the initial pose, the query image and the 3D Gaussian scene, 2D point features and 3D point features are extracted; S4 similarity matching: a similarity matching matrix of 2D point features and 3D point features is constructed, and 2D-3D matching points are selected according to the similarity score; S5 pose solution: according to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next round of iteration, and steps S3-S5 are repeated to obtain the final camera pose.

[0006] Further, in step S3, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual Transformer ViT, and traditional visual feature operator SIFT.

[0007] Furthermore, in step S3, according to the initial camera pose and the camera intrinsic parameter model, the depth corresponding to each pixel point in the 3D Gaussian scene is obtained, thereby obtaining the 3D sampling point and its incident direction Among them, x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point Then the 3D point features are obtained through neural network encoding

[0008] Furthermore, in step S4, the 2D point feature F is constructed by performing forward and reverse retrieval through the dual-softma× matching function. q and 3D point features F s The similarity matching matrix

[0009] Furthermore, in step S4, the similarity matching matrix is ​​supervised by constructing the true value by projecting the 3D points to the 2D points.

[0010] Furthermore, in step S5, the pose result is calculated using the classic PnP algorithm.

[0011] Further, in step S1, the training image {I i}i training to obtain a 3D Gaussian scene, which consists of a three-dimensional Gaussian volume set {(μj ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian, respectively.

[0012] Furthermore, in step S2, the existing image retrieval network NetVLAD or DenseVLAD is used to perform image retrieval.

[0013] The technical solution of the present invention also provides a visual repositioning system based on a 3D Gaussian scene, which is used to execute the method as described above, and the system includes the following modules: an initial input module, which is used to obtain a 3D Gaussian scene by training training images with known camera poses based on a 3D Gaussian splashing technique; an image retrieval module, which is used to retrieve the image most similar to the query image in the training image, and use its pose as the initial camera pose of the query image; a feature encoding module, which is used to extract 2D point features and 3D point features based on the initial pose, the query image and the 3D Gaussian scene; a similarity matching module, which is used to construct a similarity matching matrix of 2D point features and 3D point features, and select 2D-3D matching points according to the similarity score; and a pose solving module, which is used to calculate the pose result of the current match based on the 2D-3D matching points obtained by the similarity matching module.

[0014] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program. When the computer program is executed by one or more processors, the visual relocalization method based on the 3D Gaussian scene as described above is implemented.

[0015] The present invention provides a solution for visual relocalization using 3D Gaussian splatting technology, which fully utilizes the feature representation capability of 3D Gaussian for scenes, combines deep neural networks to realize 3D spatial point feature extraction, and realizes 2D-3D matching more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a flow chart of the visual relocalization method based on 3D Gaussian scene of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The present invention utilizes scene prior information provided by 3D Gaussian splashing, combined with deep learning, to extract features for 3D spatial sampling points and achieve more accurate 2D-3D matching.

[0020] The solution of the present invention first realizes 3D scene reconstruction based on 3D Gaussian splashing technology, inputs 2D image features and 3D Gaussian point features through deep neural network learning, and constructs 2D-3D matching points, and finally uses the PnP algorithm to solve the camera pose. Figure 1 As shown, the specific technical route of the solution is as follows:

[0021] (1) Initial input: Given a known camera pose {P i} i The video sequence {I i} i The trained 3D Gaussian radiation field is composed of a three-dimensional Gaussian volume set {(μ j ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing scale and rotation, opacity, and the spherical harmonic function representing color. The task goal is to input a query image I with an unknown camera pose in a 3D Gaussian scene. q , solve its camera pose P q .

[0022] (2) Image retrieval: Using existing image retrieval networks, such as NetVLAD and DenseVLAD, we can i} i=1,...,N Retrieval and query images in I q The most similar image I k , and use its pose Pk as the initial pose of the query image.

[0023] (3) Feature encoding: For 2D points, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual transformer ViT, traditional visual feature operator SIFT. For 3D points, according to the initial camera pose P k And the camera intrinsic parameter model, the depth corresponding to each pixel in the 3D Gaussian scene can be obtained, thereby obtaining the 3D sampling point and its incident direction where x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point And obtained through neural network encoding

[0024] (4) Similarity matching: Perform forward and reverse retrieval through the dual-softmax matching function to construct the 2D point feature F q and 3D point features F s The similarity matching matrix The 2D-3D matching points are selected based on the similarity score. During the training phase, the matching matrix can be supervised by constructing the true value by projecting 3D points to 2D points.

[0025] (5) Posture solution: Based on the 2D-3D matching points, for example, the classic PnP algorithm can be used to calculate the pose result of the current match, which is used as the initial pose for the next round of iteration. Steps (3) to (5) are repeated to obtain the final result and complete the relocalization task. That is, steps (3) to (5) are repeated twice (initial once + iterative once).

[0026] In an embodiment of the present invention, a visual relocalization method based on a 3D Gaussian scene is provided, which comprises the following steps: S1 initial input: based on the 3D Gaussian splashing technology, a 3D Gaussian scene is obtained by training training images with known camera poses; S2 image retrieval: the image most similar to the query image is retrieved in the training images, and its pose is used as the initial camera pose of the query image; S3 feature encoding: based on the initial pose, the query image and the 3D Gaussian scene, 2D point features and 3D point features are extracted; S4 similarity matching: a similarity matching matrix of 2D point features and 3D point features is constructed, and 2D-3D matching points are selected according to the similarity score; S5 pose solution: according to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next round of iteration, and steps S3-S5 are repeated to obtain the final camera pose.

[0027] Further, in step S3, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual TransformerViT, and traditional visual feature operator SIFT.

[0028] Furthermore, in step S3, according to the initial camera pose and the camera intrinsic parameter model, the depth corresponding to each pixel point in the 3D Gaussian scene is obtained, thereby obtaining the 3D sampling point and its incident direction Among them, x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point Then the 3D point features are obtained through neural network encoding

[0029] Furthermore, in step S4, the 2D point feature F is constructed by performing forward and reverse retrieval through the dual-softma× matching function. q and 3D point features F s Similarity matching matrix

[0030] Furthermore, in step S4, the similarity matching matrix is ​​supervised by constructing the true value by projecting the 3D points to the 2D points.

[0031] Furthermore, in step S5, the pose result is calculated using the classic PnP algorithm.

[0032] Further, in step S1, the training image {I i} i The 3D Gaussian scene is obtained by training. The 3D Gaussian scene consists of a three-dimensional Gaussian volume set {(μ j ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian, respectively.

[0033] Furthermore, in step S2, the existing image retrieval network NetVLAD or DenseVLAD is used to perform image retrieval.

[0034] In another embodiment of the present invention, a visual relocalization system based on a 3D Gaussian scene is also provided, which is used to execute the method as described above, and the system includes the following modules: an initial input module, which is used to obtain a 3D Gaussian scene by training training images with known camera poses based on a 3D Gaussian splashing technique; an image retrieval module, which is used to retrieve an image most similar to a query image in the training images, and use its pose as the initial camera pose of the query image; a feature encoding module, which is used to extract 2D point features and 3D point features based on the initial pose, the query image and the 3D Gaussian scene; a similarity matching module, which is used to construct a similarity matching matrix of 2D point features and 3D point features, and select 2D-3D matching points according to the similarity scores; and a pose solving module, which is used to calculate the pose result of the current match based on the 2D-3D matching points obtained by the similarity matching module.

[0035] In other embodiments of the present invention, a computer-readable storage medium containing a computer program is further provided. When the computer program is executed by one or more processors, the visual relocalization method based on the 3D Gaussian scene as described above is implemented.

[0036] The present invention provides a solution for visual relocalization using 3D Gaussian splatting technology, which fully utilizes the feature representation capability of 3D Gaussian for scenes, combines deep neural networks to realize 3D spatial point feature extraction, and realizes 2D-3D matching more accurately.

Claims

1. A visual relocalization method based on 3D Gaussian scene, characterized in that: The steps include: S1 initial input: Based on the 3D Gaussian splashing technology, a 3D Gaussian scene is obtained by training with training images of known camera poses; S2 image retrieval: retrieve the image most similar to the query image in the training images, and use its pose as the initial camera pose of the query image; S3 feature encoding: extract 2D point features and 3D point features based on the initial pose, query image and 3D Gaussian scene; S4 Similarity Matching: Construct a similarity matching matrix between 2D point features and 3D point features, and select 2D-3D matching points based on the similarity score; S5 pose calculation: According to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next iteration, and steps S3-S5 are repeated to obtain the final camera pose.

2. The method according to claim 1, characterized in that In step S3, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual Transformer ViT, and traditional visual feature operator SIFT.

3. The method according to claim 2, characterized in that In step S3, according to the initial camera pose and the camera intrinsic parameter model, the depth corresponding to each pixel in the 3D Gaussian scene is obtained, thereby obtaining the 3D sampling point and its incident direction. Among them, x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point Then the 3D point features are obtained through neural network encoding 4. The method according to claim 3, characterized in that In step S4, the dual-softmax matching function is used to perform forward and reverse retrieval to construct the 2D point feature F q and 3D point features F s Similarity matching matrix 5. The method according to claim 4, characterized in that In step S4, the similarity matching matrix is ​​supervised by constructing the true value by projecting 3D points to 2D points.

6. The method according to claim 5, characterized in that In step S5, the pose result is calculated using the classic PnP algorithm.

7. The method according to any one of claims 1 to 6, characterized in that In step S1, a training image {I i }i training to obtain a 3D Gaussian scene, which consists of a three-dimensional Gaussian volume set {(μ j ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian, respectively.

8. The method according to any one of claims 1 to 6, characterized in that In step S2, image retrieval is performed using the existing image retrieval network NetVLAD or DenseVLAD.

9. A visual relocalization system based on 3D Gaussian scenes, characterized in that: For executing the method according to any one of claims 1 to 8, the system comprises the following modules: The initial input module is used to obtain a 3D Gaussian scene from training images with known camera poses based on the 3D Gaussian splashing technology; An image retrieval module is used to retrieve the image most similar to the query image in the training images and use its pose as the initial camera pose of the query image; Feature encoding module, used to extract 2D point features and 3D point features based on the initial pose, query image and 3D Gaussian scene; Similarity matching module, used to construct the similarity matching matrix of 2D point features and 3D point features, and select 2D-3D matching points according to the similarity score; The pose solving module is used to calculate the pose result of the current matching based on the 2D-3D matching points obtained by the similarity matching module.

10. A computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the visual relocalization method based on a 3D Gaussian scene described in any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Indoor service robot repositioning method based on dense feature matching

    CN115578426A

  • Relocation method based on image retrieval and neural radiation field

    CN116049477A

  • Pose estimation method and apparatus, related model training method and apparatus, electronic device, computer readable medium and computer program product

    WO2024012333A1

Cited By

  • 3D Gaussian splash visual repositioning method based on visual basic model driving

    CN119963641A

  • A Visual Relocalization Method for 3D Gaussian Splash Based on a Visual Base Model

    CN119963641B

  • Infrared image absolute pose regression repositioning method based on 3D Gaussian spatter

    CN120070564A

  • Unmanned aerial vehicle indoor positioning method and system based on three-dimensional Gaussian splash rendering scene

    CN120318481A

  • UAV indoor positioning method and system based on 3D Gaussian splash rendering scene

    CN120318481B