Three-dimensional relocalization method and device for known map video sequence based on SFM

Abstract

This invention discloses a method and device for 3D relocalization of known map video sequences based on SFM (Sparse Visual Relocalization). The method includes: sparsely reconstructing a scene using SFM technology to obtain a sparse 3D point cloud model of the scene and camera intrinsic and extrinsic parameters for each captured image; for the input video or image sequence requiring relocalization, obtaining the corresponding camera intrinsic parameters; selecting the first frame of the video or image sequence and matching it with the entire known map to obtain the camera pose of the first frame in the scene; based on the obtained camera pose of the first frame in the scene, establishing a matching relationship between adjacent frames through feature matching, and using PnP-RANSAC to locate the camera pose of the remaining frames in the scene. This invention can perform relocalization operations quickly, significantly improving the speed of traditional SFM visual relocalization.

Description

Technical Field

[0001] This invention relates to the field of video sequence relocation technology, specifically to a method and device for three-dimensional relocation of known map video sequences based on SFM. Background Technology

[0002] After acquiring data about a scene using a camera, SFM (Simultaneous Front-End Modeling) technology can recover the scene's 3D structure and camera intrinsic and extrinsic parameters from the captured image set. Once a 3D model of the scene is obtained, new captures of the scene can be repositioned within the existing 3D model using visual relocalization. Visual relocalization technology has wide applications in virtual reality, augmented reality, autonomous driving, robotics, spatial computing, and 3D modeling.

[0003] Current mainstream SFM methods, in order to establish a matching relationship between the new capture and the original scene, need to match it one by one with the original captured image set, thereby assigning the scene 3D point cloud corresponding to the original acquisition to the new capture to construct the PnP problem for solving camera pose. However, using the above steps to perform relocalization has some problems:

[0004] 1. Performing such relocation steps on a set of time-related captures is redundant and does not take advantage of the view continuity between sequential captures;

[0005] 2. The above relocation process involves full-scene capture and 2D matching with newly registered data each time, which consumes a lot of time and makes it impossible to perform real-time positioning tasks. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and device for three-dimensional relocation of known map video sequences based on SFM.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a three-dimensional relocalization method for known map video sequences based on SFM, comprising:

[0008] Step 1: Perform sparse reconstruction of the scene based on SFM technology to obtain the sparse 3D point cloud model of the scene and the camera intrinsic and extrinsic parameters of each captured image.

[0009] Step 2: For the input video or image sequence that needs to be relocated, obtain the camera intrinsic parameters corresponding to the video or image sequence;

[0010] Step 3: Select the first frame of the video or image sequence and match it with the full known map to obtain the camera pose of the first frame in the scene;

[0011] Step 4: Based on the camera pose of the first frame image in the scene, establish the matching relationship between the first frame image and the second frame image through feature matching, and use PnP-RANSAC to locate the camera pose of the second frame image in the scene. Then, according to the location method of the camera pose of the second frame image in the scene, locate the camera pose of the remaining frames in the scene in sequence.

[0012] Furthermore, in step 4, during the process of sequentially locating the camera pose of the remaining frames in the scene, if the PnP solution for a certain frame fails or the number of points in RANSAC is lower than a set threshold, the camera pose in the scene obtained from the previous frame is used, and the M closest images are selected from the existing map image set based on the camera pose obtained from the previous frame. The M images are then matched with the current frame image, and the camera pose of the current frame image is relocated in the scene using PnP-RANSAC. If the solution still fails or the number of points in RANSAC is lower than the set threshold, the N closest images (N > M) are selected from the existing map image set based on the camera pose obtained from the previous frame image to provide more common view relationships, and the camera pose of the current frame image is relocated in the scene. If the solution still fails or the number of points in RANSAC is lower than the set threshold, the current frame image is marked as a location failure, and the process continues until the input video or image sequence is traversed.

[0013] Furthermore, step 1 specifically includes:

[0014] Image data is acquired from the scene to obtain scene capture;

[0015] The colmap software was used to sequentially perform feature extraction, feature matching, incremental reconstruction, and global BA operations on the captured images to obtain the camera intrinsic and extrinsic parameters of the captured images, as well as the sparse point cloud model constructed from them.

[0016] Furthermore, step 2 specifically includes:

[0017] Select a camera model to describe the projection relationship of the image that needs to be repositioned;

[0018] Read the first frame of a video or image sequence, perform EXIF ​​parsing, obtain the camera focal length information, and convert it to pixel space representation;

[0019] Use the acquired focal length information to initialize a predefined camera model.

[0020] Furthermore, the camera model is a pinhole camera model, which includes the camera principal point coordinates (cx, cy) and the camera focal length focal. The camera principal point coordinates (cx, cy) are set to half the width and length of the image.

[0021] Furthermore, step 3 specifically includes:

[0022] The superpoint algorithm is used to extract features from the first frame of the image;

[0023] Traverse the existing map image set and use the Superglue algorithm to calculate the matching relationship between each image in the image set and the first frame image;

[0024] Based on the mapping relationship between the coordinates of 2D feature points on the existing map image set and the sparse point cloud in the sparse 3D point cloud model, and the matching relationship between the first frame image and the image set, the correspondence between the first frame image and the sparse point cloud is constructed as follows:

[0025]

[0026] Where, m i Let x be the 2D coordinates of the point in the first frame image. i For existing map image sets and m i The corresponding 2D feature point, X i In a sparse 3D point cloud model, x i The corresponding sparse point cloud;

[0027] Based on the camera's intrinsic parameter K, the following mathematical model is established:

[0028]

[0029] Among them, a i b i c i For sparse point clouds X i The three-dimensional coordinates, f x f y These represent the pixel values ​​of the camera's focal length in the horizontal and vertical directions, respectively. i v i The coordinates of 2D feature points in the existing map image set are given, and n is the number of 2D feature points in the first frame image that match the sparse 3D point cloud model.

[0030] The camera pose is solved using the PnP-RANSAC algorithm as follows:

[0031] (R, t, N) inliers success) = PnP - RANSAC(x i X i K)

[0032] Where R is the rotation matrix of the first frame image in the existing map space, t is the position matrix of the first frame image in the existing map space, and N inliersThe number of interior points calculated by the RANSAC algorithm is used to analyze whether the calculated camera pose is robust, and success is the sign of successful execution.

[0033] Furthermore, the value of M is 10, and the value of N is 30.

[0034] Furthermore, the method for selecting images from the existing map image set based on the camera pose obtained from the previous frame image localization is as follows:

[0035] Calculate the distance and angle between the camera pose of all registered cameras in the existing map image set and the previous frame image. Sort all registered cameras in the existing map image set in descending order of distance from nearest to farthest. Remove registered cameras whose angle with the previous frame image is greater than a set angle threshold. Then, select the corresponding number of images sequentially from the top of the sort.

[0036] Furthermore, the set angle threshold is 30°.

[0037] In a second aspect, the present invention provides a three-dimensional relocation device for known map video sequences based on SFM, comprising a storage medium and a processor, wherein the storage medium stores a computer program, which, when executed by the processor, is used to implement the above-described method.

[0038] Beneficial effects: This invention models the scene using SFM technology to obtain a sparse scene model; then, relocalization can be performed in this scene. First, the first frame of the input sequence is matched with the whole image to determine its position; subsequently, by taking advantage of the continuity of the input sequence, the relocalization operation is quickly performed through inter-frame matching and local matching with known map capture, which greatly improves the speed of the original SFM visual relocalization. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the scene after sparse reconstruction based on SFM technology according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the initialization process according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the camera trajectory obtained by positioning according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0043] like Figures 1 to 3As shown, this embodiment of the invention provides a three-dimensional relocalization method for known map video sequences based on SFM, including:

[0044] Step 1: Perform sparse reconstruction of the scene using SFM technology to obtain a sparse 3D point cloud model of the scene and the camera's intrinsic and extrinsic parameters for each captured image. Specifically, this includes:

[0045] Image data is acquired from the scene to obtain scene capture.

[0046] The colmap software was used to sequentially perform feature extraction, feature matching, incremental reconstruction, and global basis comparison (BA) operations on the captured images to obtain the camera intrinsic and extrinsic parameters of the captured images, and the sparse point cloud model constructed accordingly. See details in [link to documentation]. Figure 1 , Figure 1 The constructed sparse scene map is shown, which includes the camera's position in the scene (small triangle boxes) and the sparse scene point cloud. Each camera parameter consists of camera intrinsic parameters and camera extrinsic parameters, which describe the camera's projection characteristics and spatial position, respectively.

[0047] Step 2: For the input video or image sequence requiring relocalization, obtain the corresponding camera intrinsic parameters. The video or image sequence requiring relocalization can be captured by freely moving around the scene. Specifically, this includes:

[0048] A camera model is selected to describe the projection relationship of the image to be repositioned. The first frame of the video or image sequence is read, EXIF ​​data is parsed to obtain the camera focal length information, and then converted to pixel space representation. The obtained focal length information is used to initialize a predefined camera model. The camera model described above is a pinhole camera model, which includes three parameters: the camera principal point coordinates (cx, cy) and the camera focal length (focal). When initializing the predefined camera model, the camera principal point coordinates (cx, cy) are set to half the width and height of the image, and the camera focal length (focal) is obtained through the EXIF ​​data parsing described above.

[0049] Step 3: Initialization: Select the first frame of the video or image sequence and match it with the entire known map to obtain the camera pose of the first frame in the scene. It should be noted that if this step fails to solve the PnP problem or the number of points in RANSAC falls below a set threshold, it indicates initialization failure. In this case, another image frame needs to be selected and matched with the entire known map. See [link to relevant documentation]. Figure 2 This step specifically includes:

[0050] The superpoint algorithm is used to extract features from the first frame of the image.

[0051] Traverse the existing map image set and use the Superglue algorithm to calculate the matching relationship between each image in the image set and the first frame image.

[0052] Based on the mapping relationship between the coordinates of 2D feature points on the existing map image set and the sparse point cloud in the sparse 3D point cloud model, and the matching relationship between the first frame image and the image set, the correspondence between the first frame image and the sparse point cloud is constructed as follows:

[0053]

[0054] Where, m i Let x be the 2D coordinates of the point in the first frame image. i For existing map image sets and m i The corresponding 2D feature point, X i In a sparse 3D point cloud model, x i The corresponding sparse point cloud;

[0055] Based on the camera's intrinsic parameter K, the following mathematical model is established:

[0056]

[0057] Among them, a i b i c i For sparse point clouds X i The three-dimensional coordinates, f x f y These represent the pixel values ​​of the camera's focal length in the horizontal and vertical directions, respectively. i v i represents the coordinates of 2D feature points in the existing map image set, and n represents the number of 2D feature points in the first frame image that match the sparse 3D point cloud model.

[0058] The camera pose is solved using the PnP-RANSAC algorithm as follows:

[0059] (R, t, N) inliers success) = PnP - RANSAC(x i X i ,K)

[0060] Where R is the rotation matrix of the first frame image in the existing map space, t is the position matrix of the first frame image in the existing map space, and N inliers This represents the number of interior points calculated by the RANSAC algorithm, used to analyze whether the calculated camera pose is robust. "Success" indicates successful execution; if the output is "False", it means the algorithm failed.

[0061] Step 4: Based on the camera pose of the first frame image in the scene, establish a matching relationship between the first and second frames image through feature matching. Use PnP-RANSAC to locate the camera pose of the second frame image in the scene. Then, following the same method as locating the camera pose of the second frame image, locate the camera pose of the remaining frames image in the scene. After completion, the camera pose of the entire video or image sequence can be obtained. See details in [link to documentation]. Figure 3 , Figure 3 The image shows the camera trajectory obtained by performing relocalization on a set of image sequences.

[0062] See Figure 3 Furthermore, during the process of sequentially locating the camera pose of the remaining frames in the scene, if the PnP solution for a certain frame fails or the number of points in RANSAC is lower than a set threshold, the camera pose in the scene obtained from the previous frame is used. Based on the camera pose obtained from the previous frame, the M closest images are selected from the existing map image set, and a matching relationship is established between the M images and the current frame. The PnP-RANSAC is then used to relocate the camera pose of the current frame in the scene. If the solution still fails or the number of points in RANSAC is lower than the set threshold, the N closest images (N > M) are selected from the existing map image set based on the camera pose obtained from the previous frame to provide more common views and a more lenient local map matching strategy. The camera pose of the current frame is then attempted to be located again. If the solution still fails or the number of points in RANSAC is lower than the set threshold, the current frame is marked as a location failure, and the process continues until the input video or image sequence has been traversed. The preferred value of M is 10, and the preferred value of N is 30. The number of images selected was increased from 10 to 30 to ensure that there are enough feature points to calculate the camera pose.

[0063] The specific method for selecting images from the existing map image set based on the camera pose obtained from the previous frame image is as follows:

[0064] Calculate the distance and angle between the registered cameras in the existing map image set and the camera pose of the previous frame image. Sort all registered cameras in the existing map image set in descending order of distance, and remove registered cameras whose camera pose angle with the previous frame image is greater than a set angle threshold. Then, select the corresponding number of images sequentially from the top of the sorted list. The set angle threshold is preferably 30°.

[0065] The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, other parts not specifically described are existing technology or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional relocalization method for known map video sequences based on SFM, characterized in that, include: Step 1: Perform sparse reconstruction of the scene based on SFM technology to obtain the sparse 3D point cloud model of the scene and the camera intrinsic and extrinsic parameters of each captured image. Step 2: For the input video or image sequence that needs to be relocated, obtain the camera intrinsic parameters corresponding to the video or image sequence; Step 3: Select the first frame of the video or image sequence and match it with the full known map to obtain the camera pose of the first frame in the scene; Step 4: Based on the camera pose of the first frame image in the scene, establish the matching relationship between the first frame image and the second frame image through feature matching, and use PnP-RANSAC to locate the camera pose of the second frame image in the scene. Then, according to the location method of the camera pose of the second frame image in the scene, locate the camera pose of the remaining frames in the scene in sequence.

2. The method for 3D relocalization of known map video sequences based on SFM according to claim 1, characterized in that, In step 4, during the process of sequentially locating the camera pose of the remaining frames in the scene, if the PnP solution for a certain frame fails or the number of points in RANSAC is lower than a set threshold, the camera pose in the scene obtained from the previous frame is used. Based on the camera pose obtained from the previous frame, the M closest images are selected from the existing map image set, and a matching relationship is established between the M images and the current frame. The camera pose of the current frame is then relocated in the scene using PnP-RANSAC. If the solution still fails or the number of points in RANSAC is lower than the set threshold, the N closest images (N > M) are selected from the existing map image set based on the camera pose obtained from the previous frame to provide more common view relationships. The camera pose of the current frame is then attempted to be located in the scene again. If the solution still fails or the number of points in RANSAC is lower than the set threshold, the current frame is marked as a location failure, and the process continues until the input video or image sequence has been traversed.

3. The method for 3D relocalization of known map video sequences based on SFM according to claim 1, characterized in that, Step 1 specifically includes: Image data is acquired from the scene to obtain scene capture; The colmap software was used to sequentially perform feature extraction, feature matching, incremental reconstruction, and global BA operations on the captured images to obtain the camera intrinsic and extrinsic parameters of the captured images, as well as the sparse point cloud model constructed from them.

4. The method for 3D relocalization of known map video sequences based on SFM according to claim 1, characterized in that, Step 2 specifically includes: Select a camera model to describe the projection relationship of the image that needs to be repositioned; Read the first frame of a video or image sequence, perform EXIF ​​parsing, obtain the camera focal length information, and convert it to pixel space representation; Use the acquired focal length information to initialize a predefined camera model.

5. The SFM-based three-dimensional relocalization method for known map video sequences according to claim 4, characterized in that, The camera model is a pinhole camera model, which includes the camera principal point coordinates (cx, cy) and the camera focal length focal. The camera principal point coordinates (cx, cy) are set to half the width and length of the image.

6. The method for 3D relocalization of known map video sequences based on SFM according to claim 1, characterized in that, Step 3 specifically includes: The superpoint algorithm is used to extract features from the first frame of the image; Traverse the existing map image set and use the Superglue algorithm to calculate the matching relationship between each image in the image set and the first frame image; Based on the mapping relationship between the coordinates of 2D feature points on the existing map image set and the sparse point cloud in the sparse 3D point cloud model, and the matching relationship between the first frame image and the image set, the correspondence between the first frame image and the sparse point cloud is constructed as follows: Where, m i Let x be the 2D coordinates of the point in the first frame image. i For existing map image sets and m i The corresponding 2D feature point, X i In a sparse 3D point cloud model, x i The corresponding sparse point cloud; Based on the camera's intrinsic parameter K, the following mathematical model is established: Among them, a i 、b i c i For sparse point clouds X i The three-dimensional coordinates, f x f y These represent the pixel values ​​for the camera's focal length in the horizontal and vertical directions, respectively. i v i The coordinates of 2D feature points in the existing map image set are given, and n is the number of 2D feature points in the first frame image that match the sparse 3D point cloud model. The camera pose is solved using the PnP-RANSAC algorithm as follows: (R,t,N inliers ,success)=PnP-RANSAC(x i ,X i ,K) Where R is the rotation matrix of the first frame image in the existing map space, t is the position matrix of the first frame image in the existing map space, and N inliers The number of interior points calculated by the RANSAC algorithm is used to analyze whether the calculated camera pose is robust, and success is the sign of successful execution.

7. The SFM-based three-dimensional relocalization method for known map video sequences according to claim 2, characterized in that, The value of M is 10, and the value of N is 30.

8. The method for 3D relocalization of known map video sequences based on SFM according to claim 2, characterized in that, The specific method for selecting images from an existing map image set based on the camera pose obtained from the previous frame image is as follows: Calculate the distance and angle between the camera pose of all registered cameras in the existing map image set and the previous frame image. Sort all registered cameras in the existing map image set in descending order of distance from nearest to farthest. Remove registered cameras whose angle with the previous frame image is greater than a set angle threshold. Then, select the corresponding number of images sequentially from the top of the sort.

9. The method for three-dimensional relocalization of known map video sequences based on SFM according to claim 8, characterized in that, The set angle threshold is 30°.

10. A three-dimensional relocalization device for known map video sequences based on SFM, comprising a storage medium and a processor, wherein the storage medium stores a computer program, characterized in that, When the computer program is executed by a processor, it is used to implement the method described in any one of claims 1-9.

Images