Visual relocalization method, system and storage medium based on 3D Gaussian scene

Abstract

The present invention discloses a visual repositioning method, system and storage medium based on 3D Gaussian scene, the method comprising: S1 initial input: based on 3D Gaussian splash technology, 3D Gaussian scene is obtained by training training images with known camera poses; S2 image retrieval: the image most similar to the query image is retrieved in the training image, and its pose is used as the camera initial pose of the query image; S3 feature encoding: based on the initial pose, 2D point features and 3D point features are extracted; S4 similarity matching: constructing a similarity matching matrix of 2D point features and 3D point features, and selecting 2D-3D matching points according to the similarity score; S5 pose solution: according to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next round of iteration, and steps S3-S5 are repeated to obtain the final camera pose. The present invention makes full use of the feature representation ability of 3D Gaussian for scenes, combines deep neural networks to realize 3D space point feature extraction, and realizes 2D-3D matching more accurately.

Description

Technical Field

[0001] The present invention relates to the field of visual relocation technology, and in particular to a visual relocation method, system and storage medium based on a 3D Gaussian scene. Background Art

[0002] In the technical fields of autonomous driving, robot navigation, AR / VR, etc., scene reconstruction and relocalization technology provide intelligent agents with the ability to perceive and interact with the environment, which is of great significance. Relocalization technology based on visual information has become a research hotspot in the field of computer vision due to its advantages such as rich sensor information and low cost. The key to the relocalization algorithm based on image information is how to reconstruct a high-quality 3D model based on 2D images in order to make full use of 3D scene information and obtain accurate 2D-3D matching points. Due to the characteristics of image sensing information, the traditional SLAM method is relatively sparse in constructing 3D models and has a high computational cost. In recent years, the rise of neural rendering technology and 3D Gaussian splashing technology has provided a more efficient and accurate scene reconstruction method.

[0003] Neural rendering technology and 3D Gaussian splash technology are emerging 3D scene reconstruction technologies in the past two years. They are relatively mature and have rich practical value. 3D Gaussian splash is more efficient than neural rendering and is a hot research topic in the current computer vision field. The CVPR2024 paper The NeRFect Match: Exploring NeRF Features for Visual Localization provides ideas for visual relocalization based on neural rendering technology, and how to perform visual relocalization based on 3D Gaussian scene reconstruction has become a key technical problem to be solved. Summary of the invention

[0004] The present invention expects to obtain 3D scene point features and establish 2D-3D matching points based on the scene reconstruction and image rendering capabilities of the 3D Gaussian splashing technology, thereby achieving efficient and accurate visual repositioning.

[0005] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a visual relocalization method based on a 3D Gaussian scene, which comprises the following steps: S1 initial input: based on the 3D Gaussian splashing technology, a 3D Gaussian scene is obtained by training training images with known camera poses; S2 image retrieval: the image most similar to the query image is retrieved in the training images, and its pose is used as the initial camera pose of the query image; S3 feature encoding: based on the initial pose, the query image and the 3D Gaussian scene, 2D point features and 3D point features are extracted; S4 similarity matching: a similarity matching matrix of 2D point features and 3D point features is constructed, and 2D-3D matching points are selected according to the similarity score; S5 pose solution: according to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next round of iteration, and steps S3-S5 are repeated to obtain the final camera pose.

[0006] Further, in step S3, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual Transformer ViT, and traditional visual feature operator SIFT.

[0007] Furthermore, in step S3, according to the initial camera pose and the camera intrinsic parameter model, the depth corresponding to each pixel point in the 3D Gaussian scene is obtained, thereby obtaining the 3D sampling point and its incident direction Among them, x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point Then the 3D point features are obtained through neural network encoding

[0008] Furthermore, in step S4, the 2D point feature F is constructed by performing forward and reverse retrieval through the dual-softma× matching function. q and 3D point features F s Similarity matching matrix

[0009] Furthermore, in step S4, the similarity matching matrix is ​​supervised by constructing the true value by projecting the 3D points to the 2D points.

[0010] Furthermore, in step S5, the pose result is calculated using the classic PnP algorithm.

[0011] Further, in step S1, the training image {I i}i training to obtain a 3D Gaussian scene, which consists of a three-dimensional Gaussian volume set {(μj ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian, respectively.

[0012] Furthermore, in step S2, the existing image retrieval network NetVLAD or DenseVLAD is used to perform image retrieval.

[0013] The technical solution of the present invention also provides a visual repositioning system based on a 3D Gaussian scene, which is used to execute the method as described above, and the system includes the following modules: an initial input module, which is used to obtain a 3D Gaussian scene by training training images with known camera poses based on a 3D Gaussian splashing technique; an image retrieval module, which is used to retrieve the image most similar to the query image in the training image, and use its pose as the initial camera pose of the query image; a feature encoding module, which is used to extract 2D point features and 3D point features based on the initial pose, the query image and the 3D Gaussian scene; a similarity matching module, which is used to construct a similarity matching matrix of 2D point features and 3D point features, and select 2D-3D matching points according to the similarity score; and a pose solving module, which is used to calculate the pose result of the current match based on the 2D-3D matching points obtained by the similarity matching module.

[0014] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program. When the computer program is executed by one or more processors, the visual relocalization method based on the 3D Gaussian scene as described above is implemented.

[0015] The present invention provides a solution for visual relocalization using 3D Gaussian splatting technology, which fully utilizes the feature representation capability of 3D Gaussian for scenes, combines deep neural networks to realize 3D spatial point feature extraction, and realizes 2D-3D matching more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a flow chart of the visual relocalization method based on 3D Gaussian scene of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The present invention utilizes scene prior information provided by 3D Gaussian splashing, combined with deep learning, to extract features for 3D spatial sampling points and achieve more accurate 2D-3D matching.

[0020] The solution of the present invention first realizes 3D scene reconstruction based on 3D Gaussian splashing technology, inputs 2D image features and 3D Gaussian point features through deep neural network learning, and constructs 2D-3D matching points, and finally uses the PnP algorithm to solve the camera pose. Figure 1 As shown, the specific technical route of the solution is as follows:

[0021] (1) Initial input: Given a known camera pose {P i} i The video sequence {I i} i The trained 3D Gaussian radiation field is composed of a three-dimensional Gaussian volume set {(μ j ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing scale and rotation, opacity, and the spherical harmonic function representing color. The task goal is to input a query image I with an unknown camera pose in a 3D Gaussian scene. q , solve its camera pose P q .

[0022] (2) Image retrieval: Using existing image retrieval networks, such as NetVLAD and DenseVLAD, we can i} i＝1，...，N Retrieval and query images in I q The most similar image I k , and use its pose Pk as the initial pose of the query image.

[0023] (3) Feature encoding: For 2D points, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual transformer ViT, traditional visual feature operator SIFT. For 3D points, according to the initial camera pose P k And the camera intrinsic parameter model, the depth corresponding to each pixel in the 3D Gaussian scene can be obtained, thereby obtaining the 3D sampling point and its incident direction where x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point And obtained through neural network encoding

[0024] (4) Similarity matching: Perform forward and reverse retrieval through the dual-softmax matching function to construct the 2D point feature F q and 3D point features F s Similarity matching matrix The 2D-3D matching points are selected based on the similarity score. During the training phase, the matching matrix can be supervised by constructing the true value by projecting 3D points to 2D points.

[0025] (5) Posture solution: Based on the 2D-3D matching points, for example, the classic PnP algorithm can be used to calculate the pose result of the current match, which is used as the initial pose for the next round of iteration. Steps (3) to (5) are repeated to obtain the final result and complete the relocalization task. That is, steps (3) to (5) are repeated twice (initial once + iterative once).

[0026] In an embodiment of the present invention, a visual relocalization method based on a 3D Gaussian scene is provided, which comprises the following steps: S1 initial input: based on the 3D Gaussian splashing technology, a 3D Gaussian scene is obtained by training training images with known camera poses; S2 image retrieval: the image most similar to the query image is retrieved in the training images, and its pose is used as the initial camera pose of the query image; S3 feature encoding: based on the initial pose, the query image and the 3D Gaussian scene, 2D point features and 3D point features are extracted; S4 similarity matching: a similarity matching matrix of 2D point features and 3D point features is constructed, and 2D-3D matching points are selected according to the similarity score; S5 pose solution: according to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next round of iteration, and steps S3-S5 are repeated to obtain the final camera pose.

[0027] Further, in step S3, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features Convolutional neural network CNN, visual TransformerViT, and traditional visual feature operator SIFT.

[0028] Furthermore, in step S3, according to the initial camera pose and the camera intrinsic parameter model, the depth corresponding to each pixel point in the 3D Gaussian scene is obtained, thereby obtaining the 3D sampling point and its incident direction Among them, x represents the three-dimensional coordinate, θ and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point Then the 3D point features are obtained through neural network encoding

[0029] Furthermore, in step S4, the 2D point feature F is constructed by performing forward and reverse retrieval through the dual-softma× matching function. q and 3D point features F s Similarity matching matrix

[0030] Furthermore, in step S4, the similarity matching matrix is ​​supervised by constructing the true value by projecting the 3D points to the 2D points.

[0031] Furthermore, in step S5, the pose result is calculated using the classic PnP algorithm.

[0032] Further, in step S1, the training image {I i} i The 3D Gaussian scene is obtained by training. The 3D Gaussian scene consists of a three-dimensional Gaussian volume set {(μ j ,∑ j , α j , c j )} j Represented by, where μ, ∑, α, and c are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian, respectively.

[0033] Furthermore, in step S2, the existing image retrieval network NetVLAD or DenseVLAD is used to perform image retrieval.

[0034] In another embodiment of the present invention, a visual relocalization system based on a 3D Gaussian scene is also provided, which is used to execute the method as described above, and the system includes the following modules: an initial input module, which is used to obtain a 3D Gaussian scene by training training images with known camera poses based on a 3D Gaussian splashing technique; an image retrieval module, which is used to retrieve an image most similar to a query image in the training images, and use its pose as the initial camera pose of the query image; a feature encoding module, which is used to extract 2D point features and 3D point features based on the initial pose, the query image and the 3D Gaussian scene; a similarity matching module, which is used to construct a similarity matching matrix of 2D point features and 3D point features, and select 2D-3D matching points according to the similarity scores; and a pose solving module, which is used to calculate the pose result of the current match based on the 2D-3D matching points obtained by the similarity matching module.

[0035] In other embodiments of the present invention, a computer-readable storage medium containing a computer program is further provided. When the computer program is executed by one or more processors, the visual relocalization method based on the 3D Gaussian scene as described above is implemented.

[0036] The present invention provides a solution for visual relocalization using 3D Gaussian splatting technology, which fully utilizes the feature representation capability of 3D Gaussian for scenes, combines deep neural networks to realize 3D spatial point feature extraction, and realizes 2D-3D matching more accurately.

Claims

1. A visual relocalization method based on 3D Gaussian scene, characterized in that: The steps include: S1 initial input: Based on the 3D Gaussian splashing technology, a 3D Gaussian scene is obtained by training with training images of known camera poses; S2 image retrieval: retrieve the image most similar to the query image in the training images, and use its pose as the initial camera pose of the query image; S3 feature encoding: extract 2D point features and 3D point features based on the initial pose, query image and 3D Gaussian scene; S4 Similarity Matching: Construct a similarity matching matrix between 2D point features and 3D point features, and select 2D-3D matching points based on the similarity score; S5 pose calculation: According to the 2D-3D matching points obtained in step S4, the pose result of the current match is calculated as the initial pose of the next iteration, and steps S3-S5 are repeated to obtain the final camera pose; wherein, In step S3, the query image As input, use any of the following visual feature encoders to perform feature extraction to obtain 2D point features : Convolutional neural network CNN, visual TransformerViT, traditional visual feature operator SIFT; and In step S3, according to the initial camera pose and the camera intrinsic parameter model, the depth corresponding to each pixel in the 3D Gaussian scene is obtained, thereby obtaining the 3D sampling point and its incident direction. ,in, represents three-dimensional coordinates, and Represents the incident spherical angle, and then obtains the radiation field characteristics corresponding to each sampling point , and then obtain 3D point features through neural network encoding ,in They are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian.

2. The method according to claim 1, characterized in that In step S4, forward and reverse retrieval is performed through the dual-softmax matching function to construct 2D point features and 3D point features The similarity matching matrix .

3. The method according to claim 2, characterized in that In step S4, the similarity matching matrix is ​​supervised by constructing the true value by projecting 3D points to 2D points.

4. The method according to claim 3, characterized in that In step S5, the pose result is calculated using the classic PnP algorithm.

5. The method according to any one of claims 1 to 4, characterized in that In step S1, a training image with a known camera pose is The 3D Gaussian scene is obtained by training. The 3D Gaussian scene is composed of a set of three-dimensional Gaussian bodies. Indicates that They are the center position of the Gaussian, the three-dimensional covariance matrix representing the scaling and rotation of the Gaussian, the opacity of the Gaussian, and the spherical harmonic function representing the color of the Gaussian.

6. The method according to any one of claims 1 to 4, characterized in that In step S2, image retrieval is performed using the existing image retrieval network NetVLAD or DenseVLAD.

7. A visual relocalization system based on 3D Gaussian scenes, characterized in that: For executing the method according to any one of claims 1 to 6, the system comprises the following modules: The initial input module is used to obtain a 3D Gaussian scene from training images with known camera poses based on the 3D Gaussian splashing technology; An image retrieval module is used to retrieve the image most similar to the query image in the training images and use its pose as the initial camera pose of the query image; Feature encoding module, used to extract 2D point features and 3D point features based on the initial pose, query image and 3D Gaussian scene; Similarity matching module, used to construct the similarity matching matrix of 2D point features and 3D point features, and select 2D-3D matching points according to the similarity score; The pose solving module is used to calculate the pose result of the current matching based on the 2D-3D matching points obtained by the similarity matching module.

8. A computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the visual relocalization method based on a 3D Gaussian scene described in any one of claims 1 to 6 is implemented.

Images