SFM-based known map video sequence three-dimensional relocation method and device

Abstract

The invention discloses a known map video sequence three-dimensional relocation method and device based on SFM. The method comprises the steps of performing sparse reconstruction on a scene based on an SFM technology, and obtaining a sparse three-dimensional point cloud model of the scene and camera internal and external parameters of each captured image; for an input video or image sequence needing to be repositioned, camera internal reference corresponding to the video or image sequence is obtained; selecting a first frame image of the video or the image sequence to be matched with a known full map so as to obtain a camera posture of the first frame image in the scene; and according to the obtained camera attitude of the first frame of image in the scene, establishing a matching relationship between two adjacent frames of images through feature matching, and positioning camera attitudes of other frames of images in the scene by using PnP-RANSAC. According to the method, the relocation operation can be quickly executed, and the speed of original SFM visual relocation is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video sequence repositioning, and in particular to a method and device for three-dimensional repositioning of a known map video sequence based on SFM. Background Art

[0002] After using a camera to collect data about a scene, SFM technology can recover the 3D structure of the scene and the internal and external parameters of the camera from the captured image set. After obtaining the 3D model of the scene, the position of the new capture of the scene in the existing 3D model can be restored by visual relocalization. Visual relocalization technology has a wide range of applications in virtual reality, augmented reality, unmanned driving, robotics, spatial computing, and 3D modeling.

[0003] In order to establish the matching relationship between the new capture and the original scene, the current mainstream SFM method needs to match it with the original captured image set one by one, so as to assign the scene 3D point cloud corresponding to the original acquisition to the new capture, so as to construct the PnP problem to solve the camera pose. However, there are some problems in performing relocalization using the above steps:

[0004] 1. Performing such a relocalization step on a set of temporally related captures is redundant and does not take advantage of the view continuity between sequential captures;

[0005] 2. The above relocalization process performs 2D matching of full scene capture and newly registered acquisition each time, which is time-consuming and cannot perform high-real-time positioning tasks. Summary of the invention

[0006] The purpose of the present invention is to provide a method and device for three-dimensional repositioning of a known map video sequence based on SFM in view of the deficiencies in the prior art.

[0007] To achieve the above object, in a first aspect, the present invention provides a method for three-dimensional repositioning of a video sequence with a known map based on SFM, comprising:

[0008] Step 1: Sparsely reconstruct the scene based on the SFM technology to obtain a sparse 3D point cloud model of the scene and the camera internal and external parameters of each captured image;

[0009] Step 2: for the input video or image sequence that needs to be relocated, obtain the camera intrinsic parameters corresponding to the video or image sequence;

[0010] Step 3: Select the first frame of the video or image sequence and match it with the known map to obtain the camera pose of the first frame in the scene;

[0011] Step 4: Based on the obtained camera posture of the first frame image in the scene, a matching relationship between the first frame image and the second frame image is established through feature matching, and the camera posture of the second frame image in the scene is located using PnP-RANSAC. In addition, the camera postures of the remaining frames of images in the scene are located in turn according to the positioning method of the camera posture of the second frame image in the scene.

[0012] Furthermore, in step 4, in the process of sequentially locating the camera poses of the remaining frames in the scene, if the PnP solution of a certain frame fails or the number of RANSAC internal points is lower than the set threshold, the camera pose in the scene obtained by locating the previous frame of the frame is used, and the closest M images are selected from the image set of the existing map based on the camera pose obtained by locating the previous frame, and a matching relationship is established between the M images and the frame, and the camera pose of the frame in the scene is relocated using PnP-RANSAC; if the solution still fails or the number of RANSAC internal points is lower than the set threshold, the closest N images are selected from the image set of the existing map based on the camera pose obtained by locating the previous frame, N>M, to provide more common view relationships, and try to locate the camera pose of the frame in the scene again, if the solution still fails or the number of RANSAC internal points is lower than the set threshold, the frame is marked as a positioning failure, and continues to execute backward until the input video or image sequence is traversed.

[0013] Furthermore, the step 1 specifically includes:

[0014] Performing image data acquisition on the scene to obtain scene capture;

[0015] The colmap software is used to perform feature extraction, feature matching, incremental reconstruction and global BA operations on the captured images in sequence to obtain the camera internal and external parameters of the captured images and the sparse point cloud model constructed thereby.

[0016] Furthermore, the step 2 specifically includes:

[0017] Select a camera model to describe the projection relationship of the image that needs to be relocated;

[0018] Read the first frame of a video or image sequence, parse it in EXIF ​​format, obtain the camera focal length information, and convert it to pixel space representation;

[0019] Initialize the predefined camera model using the acquired focal length information.

[0020] Furthermore, the camera model is a pinhole camera model, which includes camera principal point coordinates (cx, cy) and camera focal length focal, and the camera principal point coordinates (cx, cy) are set to half of the image width and length.

[0021] Furthermore, the step 3 specifically includes:

[0022] Use the superpoint algorithm to extract features from the first frame image;

[0023] Traverse the image set of the existing map, and use the superglue algorithm to calculate the matching relationship between each image in the image set and the first frame image;

[0024] According to the mapping relationship between the 2D feature point coordinates on the existing map image set and the sparse point cloud in the sparse 3D point cloud model, and the matching relationship between the first frame image and the image set, the corresponding relationship between the first frame image and the sparse point cloud is constructed as follows:

[0025]

[0026] Among them, m i is the 2D coordinate point in the first frame image, x i For the existing map image set and m i The corresponding 2D feature point, X i is the sparse 3D point cloud model with x i The corresponding sparse point cloud;

[0027] The following mathematical model is established based on the camera intrinsic parameter K:

[0028]

[0029] Among them, a i , b i 、c i For the sparse point cloud X i The three-dimensional coordinates of x 、f y Represents the pixel value of the camera's focal length in the horizontal and vertical directions, u i 、v i is the coordinate value of the 2D feature point in the existing map image set, and n is the number of matches between the 2D feature points of the first frame image and the sparse 3D point cloud model;

[0030] The camera pose is solved using the PnP-RANSAC algorithm:

[0031] (R,t,N inliers , success)=PnP-RANSAC(x i , X i , K)

[0032] Among them, R is the rotation matrix of the first frame image in the existing map space, t is the position matrix of the first frame image in the existing map space, N inliersIt is the number of inliers calculated by the RANSAC algorithm, which is used to analyze whether the calculated camera pose is robust. Success is a sign of successful operation.

[0033] Furthermore, the value of M is 10, and the value of N is 30.

[0034] Furthermore, the method of selecting an image from an image set of an existing map according to the camera posture obtained by positioning the previous frame of image is as follows:

[0035] Calculate the distance and angle between all registered cameras in the image set of the existing map and the camera posture of the previous frame image, and sort all registered cameras in the image set of the existing map in descending order according to the distance from near to far, eliminate the registered cameras whose angle with the camera posture of the previous frame image is greater than the set angle threshold, and then select the corresponding number of images from the top of the sort.

[0036] Furthermore, the set angle threshold is 30°.

[0037] In a second aspect, the present invention provides a known map video sequence three-dimensional repositioning device based on SFM, comprising a storage medium and a processor, wherein the storage medium stores a computer program, and the computer program is used to implement the above method when executed by the processor.

[0038] Beneficial effects: The present invention models the scene through the SFM technology to obtain a sparse scene model; then relocalization can be performed in this scene, firstly, the first frame of the input sequence is fully matched to determine its position; subsequently, with the help of the continuity of the input sequence, the relocalization operation is quickly performed through inter-frame matching and local matching with known map capture, which greatly improves the speed of the original SFM visual relocalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of a scene after sparse reconstruction based on the SFM technology according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of the initialization process of an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the camera trajectory obtained by positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0043] like Figures 1 to 3As shown, the embodiment of the present invention provides a method for three-dimensional repositioning of a known map video sequence based on SFM, including:

[0044] Step 1: Sparsely reconstruct the scene based on the SFM technology to obtain the sparse 3D point cloud model of the scene and the internal and external parameters of each captured image. Specifically including:

[0045] Image data is collected on the scene to obtain a scene capture.

[0046] Use colmap software to perform feature extraction, feature matching, incremental reconstruction and global BA operations on the captured images in sequence to obtain the camera internal and external parameters of the captured images and the sparse point cloud model constructed therefrom. For details, see Figure 1 , Figure 1 The constructed sparse scene map is shown, which contains the position of the camera in the scene (a small triangular box) and the sparse scene point cloud. Each camera parameter consists of camera intrinsic parameters and camera extrinsic parameters, which describe the projection characteristics and spatial position of the camera respectively.

[0047] Step 2: For the input video or image sequence that needs to be relocated, obtain the camera internal parameters corresponding to the video or image sequence. The video or image sequence that needs to be relocated can be captured by randomly walking around in the scene. Specifically including:

[0048] Select a camera model to describe the projection relationship of the image that needs to be relocated. Read the first frame of the video or image sequence, perform EXIF ​​parsing, obtain the camera focal length information, and convert it to pixel space representation. Use the acquired focal length information to initialize the predefined camera model. The above camera model is a pinhole camera model. The pinhole camera model includes three parameters, namely the camera principal point coordinates (cx, cy) and the camera focal length focal. When initializing the predefined camera model, the camera principal point coordinates (cx, cy) are set to half of the image width and length. The camera focal length focal is obtained through the above EXIF ​​parsing.

[0049] Step 3: Initialize: Select the first frame of the video or image sequence and match it with the known map to obtain the camera pose of the first frame in the scene. It should be noted that if the PnP solution fails or the number of RANSAC internal points is lower than the set threshold in this step, it means that the initialization has failed and you need to select another frame of the image to match with the known map. Figure 2 , the step specifically includes:

[0050] Use the superpoint algorithm to extract features from the first frame image.

[0051] The image set of the existing map is traversed, and the matching relationship between each image in the image set and the first frame image is calculated using the superglue algorithm.

[0052] According to the mapping relationship between the 2D feature point coordinates on the existing map image set and the sparse point cloud in the sparse 3D point cloud model, and the matching relationship between the first frame image and the image set, the corresponding relationship between the first frame image and the sparse point cloud is constructed as follows:

[0053]

[0054] Among them, m i is the 2D coordinate point in the first frame image, x i For the existing map image set and m i The corresponding 2D feature point, X i is the sparse 3D point cloud model with x i The corresponding sparse point cloud;

[0055] The following mathematical model is established based on the camera intrinsic parameter K:

[0056]

[0057] Among them, a i , b i 、c i For the sparse point cloud X i The three-dimensional coordinates of x 、f y Represents the pixel value of the camera's focal length in the horizontal and vertical directions, u i 、v i is the coordinate value of the 2D feature point in the existing map image set, and n is the number of matches between the 2D feature points of the first frame image and the sparse 3D point cloud model.

[0058] The camera pose is solved using the PnP-RANSAC algorithm:

[0059] (R, t, N inliers , success) = PnP-RANSAC(x i , X i ,K)

[0060] Among them, R is the rotation matrix of the first frame image in the existing map space, t is the position matrix of the first frame image in the existing map space, N inliers It is the number of internal points calculated by the RANSAC algorithm, which is used to analyze whether the calculated camera posture is robust. Success is a sign of successful operation. If the output is False, it means that the algorithm has failed.

[0061] Step 4: Based on the obtained camera pose of the first frame image in the scene, the matching relationship between the first frame image and the second frame image is established through feature matching, and the camera pose of the second frame image in the scene is located using PnP-RANSAC. The camera poses of the remaining frames in the scene are located in turn according to the positioning method of the camera pose of the second frame image in the scene. After the execution is completed, the camera pose of the entire video or image sequence can be obtained. For details, please refer to Figure 3 , Figure 3 The camera trajectory obtained by performing relocalization on a set of image sequences is shown.

[0062] See also Figure 3 In addition, in the process of sequentially locating the camera poses of the remaining frames in the scene, if the PnP solution of a frame image fails or the number of RANSAC inliers is lower than the set threshold, the camera pose in the scene obtained by locating the previous frame image of the frame is used, and the closest M images are selected from the image set of the existing map according to the camera pose obtained by locating the previous frame image, and the M images are matched with the frame image, and the camera pose of the frame image in the scene is relocated using PnP-RANSAC; if the solution still fails or the number of RANSAC inliers is lower than the set threshold, the closest N images are selected from the image set of the existing map according to the camera pose obtained by locating the previous frame image, N>M, to provide more common view relationships, use a more relaxed map local matching strategy, and try to locate the camera pose of the frame image in the scene again, if the solution still fails or the number of RANSAC inliers is lower than the set threshold, the frame image is marked as positioning failure, and the execution continues until the input video or image sequence traversal is completed. The value of the above M is preferably 10, and the value of the above N is preferably 30. Increase the number of selected images from 10 to 30 to ensure that there are enough feature points to calculate the camera pose.

[0063] The above method of selecting images from the image set of the existing map based on the camera posture obtained by positioning the previous frame of image is as follows:

[0064] Calculate the distance and angle between all registered cameras in the image set of the existing map and the camera posture of the previous frame image, and sort all registered cameras in the image set of the existing map in descending order according to the distance from near to far, remove the registered cameras whose angle with the camera posture of the previous frame image is greater than the set angle threshold, and then select the corresponding number of images from the top of the sorting. The above-set angle threshold is preferably 30°.

[0065] The above is only a preferred embodiment of the present invention. It should be noted that for ordinary technicians in this technical field, other parts not specifically described belong to the prior art or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A 3D relocalization method for video sequences with known maps based on SFM, characterized in that: include: Step 1: Sparsely reconstruct the scene based on the SFM technology to obtain a sparse 3D point cloud model of the scene and the camera internal and external parameters of each captured image; Step 2: for the input video or image sequence that needs to be relocated, obtain the camera intrinsic parameters corresponding to the video or image sequence; Step 3: Select the first frame of the video or image sequence and match it with the known map to obtain the camera pose of the first frame in the scene; Step 4: Based on the obtained camera posture of the first frame image in the scene, a matching relationship between the first frame image and the second frame image is established through feature matching, and the camera posture of the second frame image in the scene is located using PnP-RANSAC. In addition, the camera postures of the remaining frames of images in the scene are located in turn according to the positioning method of the camera posture of the second frame image in the scene.

2. The SFM-based 3D relocation method for a known map video sequence according to claim 1, characterized in that: In step 4, in the process of sequentially locating the camera poses of the remaining frames in the scene, if the PnP solution of a frame fails or the number of RANSAC internal points is lower than the set threshold, the camera pose in the scene obtained by locating the previous frame of the frame is used, and the closest M images are selected from the image set of the existing map according to the camera pose obtained by locating the previous frame, and a matching relationship is established between the M images and the frame, and the camera pose of the frame in the scene is relocated using PnP-RANSAC; if the solution still fails or the number of RANSAC internal points is lower than the set threshold, the closest N images are selected from the image set of the existing map according to the camera pose obtained by locating the previous frame, N>M, to provide more common view relationships, and try to locate the camera pose of the frame in the scene again, if the solution still fails or the number of RANSAC internal points is lower than the set threshold, the frame is marked as a positioning failure, and continues to execute backward until the input video or image sequence is traversed.

3. The SFM-based 3D relocation method for video sequences with known maps according to claim 1, characterized in that: The step 1 specifically includes: Performing image data acquisition on the scene to obtain scene capture; The colmap software is used to perform feature extraction, feature matching, incremental reconstruction and global BA operations on the captured images in sequence to obtain the camera internal and external parameters of the captured images and the sparse point cloud model constructed thereby.

4. The SFM-based 3D relocation method for video sequences with known maps according to claim 1, characterized in that: The step 2 specifically includes: Select a camera model to describe the projection relationship of the image that needs to be relocated; Read the first frame of a video or image sequence, parse it in EXIF ​​format, obtain the camera focal length information, and convert it to pixel space representation; Initialize the predefined camera model using the acquired focal length information.

5. The SFM-based 3D relocation method for video sequences with known maps according to claim 4, characterized in that: The camera model is a pinhole camera model, which includes camera principal point coordinates (cx, cy) and camera focal length focal, and the camera principal point coordinates (cx, cy) are set to half of the width and length of the image.

6. The SFM-based 3D relocation method for video sequences with known maps according to claim 1, characterized in that: The step 3 specifically includes: Use the superpoint algorithm to extract features from the first frame image; Traverse the image set of the existing map, and use the superglue algorithm to calculate the matching relationship between each image in the image set and the first frame image; According to the mapping relationship between the 2D feature point coordinates on the existing map image set and the sparse point cloud in the sparse 3D point cloud model, and the matching relationship between the first frame image and the image set, the corresponding relationship between the first frame image and the sparse point cloud is constructed as follows: Among them, m i is the 2D coordinate point in the first frame image, x i For the existing map image set and m i The corresponding 2D feature point, X i is the sparse 3D point cloud model with x i The corresponding sparse point cloud; The following mathematical model is established based on the camera intrinsic parameter K: Among them, a i 、b i 、c i For the sparse point cloud X i The three-dimensional coordinates of x 、f y Represents the pixel value of the camera's focal length in the horizontal and vertical directions, u i 、v i is the coordinate value of the 2D feature point in the existing map image set, and n is the number of matches between the 2D feature points of the first frame image and the sparse 3D point cloud model; The camera pose is solved using the PnP-RANSAC algorithm: (R,t,N inliers ,success)=PnP-RANSAC(x i ,X i ,K) Among them, R is the rotation matrix of the first frame image in the existing map space, t is the position matrix of the first frame image in the existing map space, N inliers It is the number of inliers calculated by the RANSAC algorithm, which is used to analyze whether the calculated camera pose is robust. Success is a sign of successful operation.

7. The SFM-based 3D relocation method for known map video sequences according to claim 2, characterized in that: The value of M is 10, and the value of N is 30.

8. The SFM-based 3D relocation method for known map video sequences according to claim 2, characterized in that: The specific method of selecting images from the image set of the existing map based on the camera posture obtained by positioning the previous frame of image is as follows: Calculate the distance and angle between all registered cameras in the image set of the existing map and the camera posture of the previous frame image, and sort all registered cameras in the image set of the existing map in descending order according to the distance from near to far, eliminate the registered cameras whose angle with the camera posture of the previous frame image is greater than the set angle threshold, and then select the corresponding number of images from the top of the sort.

9. The SFM-based 3D relocation method for video sequences with known maps according to claim 8, characterized in that: The set angle threshold is 30°.

10. A device for three-dimensional repositioning of a known map video sequence based on SFM, comprising a storage medium and a processor, wherein the storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 9.

Images