A time series based visual map positioning method and system

Abstract

The application provides a time series-based visual map positioning method and system, which comprises the following steps: based on a first pose set T and a second pose set P2, a third pose set P3 of each sequence image relative to other sequence images is calculated; given a projection error threshold Th, according to the pose element set of each sequence image relative to other sequence images in the third pose set P3, an error value set ΔX between the three-dimensional point projection of the image feature of each sequence image corresponding to the pose element and the corresponding two-dimensional point in the device coordinate system is calculated; based on the error value set ΔX, the inlier number set inlier of the sequence image pose element projection error value less than the threshold Th is counted; and the pose element set corresponding to the sequence image corresponding inlier number set maximum value is taken as the current pose set P4 of the sequence image and is output. The application can improve the positioning accuracy of the visual map.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a visual map localization method and system based on time series. Background Technology

[0002] Existing computer vision positioning technology reconstructs a region using multiple sensors such as cameras, generates a visual map, and then uses a mobile device containing sensors such as visible light to collect certain data. The position of the mobile device in the visual map is then calculated.

[0003] Visual mapping refers to the process of capturing images of a scene using a camera to create a map. This map is then generated using techniques such as Structure From Motion, which may or may not incorporate data from other sensors (depth, laser, etc.). The resulting map file contains visual information and primarily includes the generated spatial map and 2D features from the images. Visual mapping is a technique that uses the captured image, with or without additional sensor information, to locate the current position of the image within the visual map. For pure point cloud maps, features and descriptors of the image to be located are typically extracted and compared with the descriptors of the 3D point cloud. The most similar points are then paired, and the pose is calculated using these point pairs. For keyframe-based maps, a descriptor is typically generated from the images, and the overall image is compared with the descriptors generated from the keyframes. Keyframe candidates similar to the current image are searched for. Finally, the pose is calculated through local descriptor matching between keyframes or by directly estimating the SIM3 transform between images.

[0004] Existing positioning technologies are all based on a single image as the data source. However, the 2D-3D matching in positioning algorithms always contains errors, and all matches may be wrong. As a result, the calculated pose error is relatively large, or even a very absurd pose, which limits the use of related products. Summary of the Invention

[0005] The technical problem solved by the present invention is: how to improve the positioning accuracy of visual maps and how to calculate a more robust pose for the current frame.

[0006] To address the aforementioned technical problems, the present invention provides a time-series-based visual map localization method, comprising:

[0007] Acquire visual map data, which includes: a first image feature set and a corresponding first three-dimensional point cloud coordinate set;

[0008] Receive the sequence of images F to be located, F = {F1, F2, F3, ..., F...} M}; M is the image sequence number, where M is a natural number greater than 1;

[0009] Calculate the initial pose set P1 of the sequence image F in the device coordinate system, P1 = {P1...} 1 P2 1 P3 1 ,…,P M 1};

[0010] Based on the initial pose set P1, the relative poses between each pair of sequential images F are obtained to obtain the first pose set T, T = {T1, T2, T3, ..., T...}. M};

[0011] Extract image features from each image sequence to obtain the corresponding second image feature set J = {J1, J2, J3, ..., J...} M} and the two-dimensional image feature coordinate set R={R1,R2,R3,…,R M};

[0012] Based on the image features of each sequence of images in the second image feature set J, find the matching image features in the first image feature set to form a third image feature set J', where J' = {J1', J2', J3', ..., J...} M '};

[0013] Based on the third image feature set J', a three-dimensional point cloud coordinate set corresponding to the image features in the first three-dimensional point cloud coordinate set is obtained from the third image feature set J', so as to form a second three-dimensional point cloud coordinate set S = {S1, S2, S3, ..., S} corresponding to the second image feature set J. M};

[0014] Based on the two-dimensional image feature coordinate set and the corresponding second three-dimensional point cloud coordinate set, the second pose set P2 of the sequence image is obtained, where P2 = {P1} 2 P2 2 P3 2 ,…,P M 2};

[0015] Based on the first pose set T and the second pose set P2, calculate the third pose set P3 of each image sequence relative to the other image sequences, where P3 = {P1} 3 P2 3 P3 3 ,…,P M 3};P1 3 For the sequence image F1 in its pose P1 2Based on the relative pose set T1, the sequence images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P2 3 For the sequence image F2 in its pose P2 2 Based on the relative pose set T2, the sequence of images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P3 3 For the sequence image F3 in its pose P3 2 Based on the relative pose set T3, the sequence of images F1, F2, F3, F4, ..., F are calculated. M The keyframe pose set; and so on, P M 3 For the sequence image F M In its pose P M 2 Based on the relative pose set T M The calculated sequence images F1, F2, F3, ..., F (M-1) ,F M Keyframe pose set;

[0016] Given a projection error threshold Th, based on the pose element set P1 of each sequence image in the third pose set P3 relative to other sequence images... 3 P2 3 P3 3 ,…,P M 3 Calculate the set of error values ​​ΔX between the projection of the 3D points of the image features corresponding to each pose element onto the corresponding 2D points in the device coordinate system, where ΔX = ΔX1, ΔX2, ΔX3, ..., ΔX M ;

[0017] Based on the error value set △X={△X1,△X2,△X3,…,△X M}, statistically analyze the sequence images F1, F2, F3, ..., F M The set of inliers whose pose element projection error values ​​are less than the threshold Th is inlier; inlier = {inlier1, inlier2, inlier3, ..., inlier} M};

[0018] Based on the sequence images F1, F2, F3, ..., F M The corresponding number of inliers, counting the number of inliers in the set inlier1, inlier2, inlier3, ..., inlier M Let the set corresponding to the maximum number of interior points be inlier. max, where max is the image number;

[0019] The corresponding pose element set P max 3 As the sequence images F1, F3, ..., F M The current pose P4, P4 = P max 3 .

[0020] Optionally, based on the first pose set T, since T = {T1, T2, T3, ..., T M}:

[0021] T1={T 11 ,T 12 ,T 13 ,T 14 ,…,T 1M}, T1 is the sequence image F1 with sequence number 1, relative to the sequence images F1, F2, F3, ..., Fm with sequence numbers 1 to M. M The relative pose set;

[0022] T2={T 21 ,T 22 ,T 23 ,T 24 ,…,T 2M}, T2 is the sequence image F2 with image sequence number 2 relative to the sequence images F1, F2, F3, ..., F2 with image sequence numbers 1 to M. M The relative pose set;

[0023] T3={T 31 ,T 32 ,T 33 ,…,T 3M}, T3 is the sequence image F3 with sequence number 3, relative to the sequence images F1, F2, F3, ..., F with sequence numbers 1 to M. M The relative pose set; and so on,

[0024] T M ={T M1 ,T M2 ,T M3 ,…,T MM}, T M Image sequence F with sequence number M M Relative to the sequence of images F1, F2, F3, ..., Fm numbered from 1 to M M The relative pose set.

[0025] Optionally, let N1, N2, N3, ..., (NM) be natural numbers greater than 1, and let F1, F2, F3, ..., FM be the sequence images respectively. M The number of image features extracted;

[0026] Based on the second image feature set J = {J1, J2, J3, ..., J...} M}, Two-dimensional image feature coordinate set R = {R1, R2, R3, ..., R M}, the third image feature set J', J'={J1',J2',J3',…,J M '} and the second three-dimensional point cloud coordinate set S={S1,S2,S3,…,S M};

[0027] J1={j 11 ,j 12 ,j 13 ,…,j 1N1}、R1={r 11 ,r 12 ,r 13 ,…,r 1N1}、J1'={j 11 ',j 12 ',j 13 ',…,j 1N1 '}、S1={s 11 ,s 12 ,s 13 ,…,s 1N1};j 11 ,j 12 ,j 13 ,…,j 1N1 r represents the image feature elements extracted from the sequence image F1. 11 ,r 12 ,r 13 ,…,r 1N1 Image feature element j extracted from sequence image F1 11 ,j 12 ,j 13 ,…,j 1N1 In the device coordinate system, the two-dimensional coordinate element j 11 ',j 12 ',j 13 ',…,j 1N1 'For the first image feature set, respectively with j 11 ,j 12 ,j 13 ,…,j 1N1 Matched image features, s 11 ,s 12 ,s 13 ,…,s 1N1For the first three-dimensional point cloud coordinate set and image features j 11 ',j 12 ',j 13 ',…,j 1N1 'Corresponding 3D point cloud coordinate elements;

[0028] J2={j 21 ,j 22 ,j 23 ,…,j 2N2}、R2={r 21 ,r 22 ,r 23 ,…,r 2N2}、J2'={j 21 ',j 22 ',j 23 ',…,j 2N2 '}、S2={s 21 ,s 22 ,s 23 ,…,s 2N2};j 21 ,j 22 ,j 23 ,…,j 2N2 r represents the image feature elements extracted from the sequence image F2. 21 ,r 22 ,r 23 ,…,r 2N2 Image feature element j extracted from sequence image F2 21 ,j 22 ,j 23 ,…,j 2N2 In the device coordinate system, the two-dimensional coordinate element j 21 ',j 22 ',j 23 ',…,j 2N2 'For the first image feature set, respectively with j 21 ,j 22 ,j 23 ,…,j 2N2 Matched image features, s 21 ,s 22 ,s 23 ,…,s 2N2 For the first three-dimensional point cloud coordinate set and image features j 21 ',j 22 ',j 23 ',…,j 2N2 'Corresponding 3D point cloud coordinate elements;

[0029] J3={j 31 ,j 32 ,j33 ,…,j 3N3}、R3={r 31 ,r 32 ,r 33 ,…,r 3N3}、J3'={j 31 ',j 32 ',j 33 ',…,j 3N3 '}、S3={s 31 ,s 32 ,s 33 ,…,s 3N3};j 31 ,j 32 ,j 33 ,…,j 3N3 r represents the image feature elements extracted from the sequence image F3. 31 ,r 32 ,r 33 ,…,r 3N3 Image feature element j extracted from sequence image F3 31 ,j 32 ,j 33 ,…,j 3N3 In the device coordinate system, the two-dimensional coordinate element j 31 ',j 32 ',j 33 ',…,j 3N3 'For the first image feature set, respectively with j 31 ,j 32 ,j 33 ,…,j 3N3 Matched image features, s 31 ,s 32 ,s 33 ,…,s 3N3 For the first three-dimensional point cloud coordinate set and image features j 31 ',j 32 ',j 33 ',…,j 3N3 'Corresponding 3D point cloud coordinate elements;

[0030] And so on,

[0031] J M ={j M1 ,j M2 ,j M3 ,…,j M(NM)}、R M ={r M1 ,r M2 ,r M3 ,…,r M(NM)}、J M'={j M1 ',j M2 ',j M3 ',…,j M(NM) '}、S M ={s M1 ,s M2 ,s M3 ,…,s M(NM)};j M1 ,j M2 ,j M3 ,…,j M(NM) For the sequence image F M The extracted image feature elements, r M1 ,r M2 ,r M3 ,…,r M(NM) For the sequence image F M Extracted image feature element j M1 ,j M2 ,j M3 ,…,j M(NM) In the device coordinate system, the two-dimensional coordinate element j M1 ',j M2 ',j M3 ',…,j M(NM) 'For the first image feature set, respectively with j M1 ,j M2 ,j M3 ,…,j M(NM) Matched image features, s M1 ,s M2 ,s M3 ,…,s M(NM) For the first three-dimensional point cloud coordinate set and image features j M1 ',j M2 ',j M3 ',…,j M(NM) 'Corresponding 3D point cloud coordinate elements.

[0032] Optionally, based on the third pose set P3 = {P1} 3 P2 3 P3 3 ,…,P M 3}:

[0033] P1 3 For image sequence number 1, image F1 is positioned at pose P1. 2 Based on the relative pose set T1={T 11 ,T 12 ,T 13 ,T 14 ,…,T 1MThe sequence image F1 is calculated relative to the sequence images F1, F2, F3, ..., Fm numbered from 1 to M. M Keyframe pose set, P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 For image sequence F1, relative to image sequence F1, F2, F3, ..., F M Pose elements;

[0034] P2 3 ={P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3}, P2 3 Image sequence number 2, image F2, is positioned at pose P2. 2 Based on the relative pose set T2={T 21 ,T 22 ,T 23 ,T 24 ,…,T 2M The sequence image F2 is calculated to be relative to the sequence images F1, F2, F3, ..., F2 numbered from 1 to M. M Keyframe pose set, P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 For image sequence F2 relative to image sequence F1, F2, F3, ..., F M Pose elements;

[0035] P3 3 ={P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3}, P3 3Image sequence number 3, image F3, is positioned at pose P3. 2 Based on the relative pose set T3={T 31 ,T 32 ,T 33 ,…,T 3M The sequence image F3 is calculated to be relative to the sequence images F1, F2, F3, ..., Fm numbered from 1 to M. M Keyframe pose set, P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 For image sequence F3, relative to image sequence F1, F2, F3, ..., F M The pose element; and so on.

[0036] P M 3 ={P M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3}, P M 3 Image sequence F with sequence number M M In its pose P M 2 Based on the relative pose set T M ={T M1 ,T M2 ,T M3 ,…,T MM The sequence image F was calculated. M Relative to the sequence of images F1, F2, F4, ..., Fm numbered from 1 to M M Keyframe pose set, P M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 For the sequence of images F1, F2, F4, ..., F M Pose elements.

[0037] Optional, △X1 is the pose element P 11 3 ,P 12 3 ,P13 3 ,P 14 3 ,…,P 1M 3 The corresponding set of error values, △X1={△X 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M};△X 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between;

[0038] △X2 is the pose element P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The corresponding set of error values, △X2={△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M};△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image 21 3,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between;

[0039] △X3 is the pose element P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding set of error values, △X3={△X 31 ,△X 32 ,△X 33 ,…,△X 3M};△X 31 ,△X 32 ,△X 33 ,…,△X 3M These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between them; and so on,

[0040] △X M P is the pose element M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding set of error values, ΔX M ={△X M1 ,△X M2 ,△X M3 ,…,△X MM};△X M1 ,△XM2 ,△X M3 ,…,△X MM These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between them.

[0041] Optionally, the sequence images F1, F2, F3, ..., F can be statistically analyzed in the following manner. M The set of inliers whose pose element projection error values ​​are less than the threshold Th:

[0042] inlier1 is the pose element P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The sum of the corresponding interior point counts, inlier1 = inlier 11 +inlier 12 +inlier 13 +…+inlier 1M ; Calculate the pose elements P respectively 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The corresponding set of error values ​​△X 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M The number of error values ​​less than the threshold Th is counted as the pose element P. 11 3 ,P 12 3,P 13 3 ,P 14 3 ,…,P 1M 3 The corresponding number of interior points 11 inlier 12 ,inlier 13 ,inlier 14 ,…,inlier 1M ;

[0043] inlier2 is the pose element P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The sum of the corresponding interior point counts, inlier2 = inlier 21 +inlier 22 +inlier 23 +…+inlier 2M ; Calculate the pose elements P respectively 21 3 ,P 22 3 ,P 23 3 ,…,P 2M 3 The corresponding set of error values ​​△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M The number of error values ​​less than the threshold Th is counted as the pose element P. 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The corresponding number of interior points 21 ,inlier 22 ,inlier 23 ,inlier 14 ,…,inlier 2M ;

[0044] inlier3 is the pose element P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The sum of the corresponding interior point counts, inlier3 = inlier 31 +inlier 32 +inlier 33 +…+inlier 3M ; Calculate the pose elements P respectively 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding set of error values ​​△X 31 ,△X 32 ,△X 33 ,…,△X 3M The number of error values ​​less than the threshold Th is counted, and the sum of these counts is used as the pose element P. 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding number of interior points 31 ,inlier 32 ,inlier 33 ,…,inlier 3M And so on,

[0045] inlier M P is the pose element M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding set of interior point count statistics, inlier M ={inlier M1 ,inlier M2 ,inlier M3 ,…,inlier MM}; Calculate the pose elements P separately M1 3 ,P M2 3,P M3 3 ,…,P MM 3 The corresponding set of error values ​​△X M1 ,△X M2 ,△X M3 ,…,△X MM The number of error values ​​less than the threshold Th is counted, and the sum of these counts is used as the pose element P. M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding number of interior points M1 ,inlier M2 ,inlier M3 ,…,inlier MM Take the set of interior point counts, inlier = {inlier1, inlier2, inlier3, ..., inlier}. M The maximum number of interior points in a given number is denoted as inlier. max , where max is the corresponding image sequence number;

[0046] Let the number of interior points be inlier max The corresponding keyframe pose P max 3 ;P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3};

[0047] The sequence of images F1, F3, ..., F M The current pose set P4 is:

[0048] P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3}

[0049] Optionally, the time-series-based visual map localization method further includes:

[0050] Let P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3}={P1 4 P2 4 P3 4 ,…,P M 4}, which are the sequential images F1, F2, F3, ..., F M The current frame pose;

[0051] Calculate F1, F2, F3, ..., F for each image sequence. M The set of error values ​​△X' between the projection of the 3D points of the image features corresponding to the current frame pose onto the corresponding 2D points in the device coordinate system, where △X' = {△X1', △X2', △X3', ..., △X M '};

[0052] Based on the error value set △X'={△X1',△X2',△X3',…,△X M '}, for the current frame pose P1 in the current pose set P4 4 P2 4 P3 4 ,…,P M 4 Optimize to ensure that the error value elements △X1', △X2', △X3', ..., △X in the error value set △X' are optimized. M Minimize the sum of the error values ​​of '.

[0053] Let the optimized current pose set be P5, P5 = {P1} 5 P2 5 P3 5 ,…,P M 5}, P1 5 P2 5 P3 5 ,…,P M 5 These are the optimized pose elements P1 4 P2 4 P3 4 ,…,P M 4 ;

[0054] Output the optimized current pose set P5.

[0055] Optionally, the current frame pose P1 in the current pose set P4 can be modified as follows: 4 P2 4 P3 4 ,…,P M 4 Optimize:

[0056] △X1',△X2',△X3',…,△X M 'Represents the sequence images F1, F2, F3, ..., F M The corresponding image feature sets J1, J2, J3, ..., J M The corresponding second three-dimensional point cloud coordinate set S1, S2, S3, ..., S M Based on the current frame pose P1 4 P2 4 P3 4 ,…,P M 4 The projected coordinate set obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1, R2, R3, ..., R M The error value between;

[0057] Let E min =△X1'+△X2'+△X3'+,…,+△X M Adjust the pose of the current frame P1 4 P2 4 P3 4 ,…,P M 4 The value of E min Minimize the error value;

[0058] Let the adjusted current frame pose be P1. 5 P2 5 P3 5 ,…,P M 5 The optimized current pose set P5 = {P1} 5 P2 5 P3 5 ,…,P M 5}

[0059] To address the aforementioned technical problems, the present invention also provides a time-series-based visual map positioning system, comprising:

[0060] The first acquisition unit is adapted to acquire visual map data, the visual map data including: a first image feature set and a corresponding first three-dimensional point cloud coordinate set;

[0061] The receiving unit is adapted to receive a sequence of images F to be located, where F = {F1, F2, F3, ..., F...} M}; M is the image sequence number, where M is a natural number greater than 1;

[0062] The first computing unit is adapted to calculate the initial pose set P1 of the sequence image F in the device coordinate system, P1 = {P1...} 1 P2 1 P3 1 ,…,P M 1};

[0063] The second computational unit is adapted to obtain the relative poses between each pair of sequential images F based on the initial pose set P1 to obtain the first pose set T, T = {T1, T2, T3, ..., T...} M};

[0064] Extraction unit, adapted to extract image features from each sequence of images to obtain the corresponding second image feature set J = {J1, J2, J3, ..., J...} M} and the two-dimensional image feature coordinate set R={R1,R2,R3,…,R M};

[0065] The matching unit is adapted to find image features in the first image feature set that match the image features of each sequence of images in the second image feature set J, thereby forming a third image feature set J', where J' = {J1', J2', J3', ..., J...} M '};

[0066] The second acquisition unit is adapted to obtain a three-dimensional point cloud coordinate set corresponding to the image features in the third image feature set J' from the first three-dimensional point cloud coordinate set, so as to form a second three-dimensional point cloud coordinate set S = {S1, S2, S3, ..., S} corresponding to the second image feature set J. M};

[0067] The third calculation unit is adapted to obtain the second pose set P2 of the sequence image based on the two-dimensional image feature coordinate set and the corresponding second three-dimensional point cloud coordinate set, where P2 = {P1} 2 P2 2 P3 2 ,…,P M 2};

[0068] The fourth calculation unit is adapted to calculate, based on the first pose set T and the second pose set P2, a third pose set P3 for each sequence of images relative to other sequence of images, where P3 = {P1} 3 P2 3 P33 ,…,P M 3};P1 3 For the sequence image F1 in its pose P1 2 Based on the relative pose set T1, the sequence images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P2 3 For the sequence image F2 in its pose P2 2 Based on the relative pose set T2, the sequence of images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P3 3 For the sequence image F3 in its pose P3 2 Based on the relative pose set T3, the sequence of images F1, F2, F3, F4, ..., F are calculated. M The keyframe pose set; and so on, P M 3 For the sequence image F M In its pose P M 2 Based on the relative pose set T M The calculated sequence images F1, F2, F3, ..., F (M-1) ,F M Keyframe pose set;

[0069] The fifth calculation unit, adapted to a given projection error threshold Th, calculates the pose element set P1 of each sequence image relative to other sequence images in the third pose set P3. 3 P2 3 P3 3 ,…,P M 3 Calculate the set of error values ​​ΔX between the projection of the 3D points of the image features corresponding to each pose element onto the corresponding 2D points in the device coordinate system, where ΔX = ΔX1, ΔX2, ΔX3, ..., ΔX M ;

[0070] The statistical unit is adapted to be based on the error value set △X={△X1,△X2,△X3,…,△X M}, statistically analyze the sequence images F1, F2, F3, ..., F M The set of inliers whose pose element projection error values ​​are less than the threshold Th is inlier; inlier = {inlier1, inlier2, inlier3, ..., inlier} M}; based on the sequence images F1, F2, F3, ..., F MThe corresponding number of inliers, counting the number of inliers in the set inlier1, inlier2, inlier3, ..., inlier M The maximum value; let the set corresponding to the maximum number of interior points be inlier. max , where max is the image number;

[0071] The first output unit is suitable for processing the corresponding pose element set P. max 3 As the sequence images F1, F3, ..., F M The current pose P4, P4 = P max 3 .

[0072] Optionally, the time-series-based visual map localization system further includes:

[0073] Let P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3}={P1 4 P2 4 P3 4 ,…,P M 4}, which are the sequential images F1, F2, F3, ..., F M The current frame pose;

[0074] The sixth computational unit is suitable for calculating F1, F2, F3, ..., F for each sequence of images. M The set of error values ​​△X' between the projection of the 3D points of the image features corresponding to the current frame pose onto the corresponding 2D points in the device coordinate system, where △X' = {△X1', △X2', △X3', ..., △X M '};

[0075] The optimization unit is adapted to be based on the error value set △X'={△X1',△X2',△X3',…,△X M '}, for the current frame pose P1 in the current pose set P4 4 P2 4 P3 4 ,…,P M 4 Optimize to minimize the sum of error values ​​of error value elements △X1', △X2', △X3', ..., △X0' in the error value set △X'; let the optimized current pose set be P5, P5 = {P15 P2 5 P3 5 ,…,P M 5}, P1 5 P2 5 P3 5 ,…,P M 5 These are the optimized pose elements P1 4 P2 4 P3 4 ,…,P M 4 ;

[0076] The second output unit is suitable for outputting the optimized current pose set P5.

[0077] The beneficial effects of the technical solution of the present invention include at least the following:

[0078] The technical solution of this invention can solve the problem of large positioning errors in single images in visual map positioning applications in the prior art. By considering the inherent relationship between consecutive frames of a sequence of images in existing visual maps through sequential image poses and relative poses, the technical solution of this invention can calculate a more robust pose for the current frame.

[0079] The technical solution of this invention can be applied to the discretization of sequential images or videos within a computer window into sequential images. Through 2D-3D matching of sequential image frames, the relative pose (Delta Pose) between any two frames within the window can be calculated using VIO. The sequential frames within the window are matched and the initial pose of the current frame with the smallest residual is calculated using Ransac. Then, the initial pose is optimized using rigBA to obtain the final high-precision pose.

[0080] The technical solution of this invention is applicable to all sequence image data within a computer processing window. The data is more redundant, reducing the impact of incorrect matching. Redundant data allows for more constraints, which effectively improves positioning accuracy. Even if all matches in the current positioning frame are incorrect, the relative pose (Delta Pose) provided by VIO can be used to inversely calculate the poses of other sequence frames to the current frame, similarly directly eliminating erroneous poses in single-frame positioning. Attached Figure Description

[0081] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0082] Figure 1A flowchart illustrating a time-series-based visual map localization method provided by the present invention.

[0083] Figure 2 This is a partial flowchart illustrating another time-series-based visual map localization method provided by the present invention. Detailed Implementation

[0084] To better illustrate the technical solution of the present invention, the present invention will be further described below with reference to the accompanying drawings.

[0085] In this invention, to address the problem of large errors in existing visual map localization algorithms and obtain high-precision device pose, a time-series-based visual map localization scheme is proposed. The basic idea is as follows:

[0086] Within an existing visual map (denoted as basemap), a video segment is captured within the corresponding real-world space using a mobile device or other visible light device. Simultaneously, information from gyroscopes, accelerometers, magnetometers, and GPS is collected. Based on the acquired data, Visual-Inertial Odometry (VIO) is run to obtain the relative relationships between all images in the video sequence. A suitable window size is selected, and the sequence images within that window (images discretized from the video) are simultaneously subjected to 2D-3D matching. VIO calculates the relative pose (Deltapose) between any two images within the window. The matched frames within the window are then processed using Ransac to calculate the initial pose (Pose) of the current frame with the smallest residual. RigBA is then used to optimize the initial pose (Pose) to obtain the final high-precision pose.

[0087] Based on the above-mentioned inventive concept, this invention provides a time-series-based visual map localization method, such as... Figure 1 As shown, it includes the following steps:

[0088] Step S100: Obtain visual map data, which includes: a first image feature set and a corresponding first three-dimensional point cloud coordinate set.

[0089] In specific applications, video sequences within a certain spatial range can be acquired using devices such as panoramic / fisheye / digital cameras, and a high-precision visual map, denoted as a Basemap, can be created. Basemap data includes, but is not limited to: keyframe poses, 2D feature point positions and corresponding descriptors, 2D-3D point correspondences, camera parameters, and the positions and errors of 3D points.

[0090] Based on the collected visual map (Basemap) data, a series of image feature sets and corresponding three-dimensional point cloud coordinate sets that constitute the image sequence of the visual map can be obtained, namely the first image feature set and the corresponding first three-dimensional point cloud coordinate set.

[0091] Step S101: Receive the sequence of images F to be located, F = {F1, F2, F3, ..., F...} M}; M is the image sequence number, where M is a natural number greater than 1.

[0092] Specifically, video sequences are captured within the Basemap spatial range using mobile devices such as smartphones and cameras. The images in this video sequence are denoted as queries. Since a query is a series of positioned images, it is also denoted as F in this embodiment, where F = {F1, F2, F3, ..., F...}. M}

[0093] Step S102: Calculate the initial pose set P1 of the sequence image F in the device coordinate system, P1 = {P1...} 1 P2 1 P3 1 ,…,P M 1}

[0094] The sequence of images F, including all images F1, F2, F3, ..., F, can be acquired via VIO (Visual Inertial Odometry). M Initial pose set P1, let P1 = {P1 1 P2 1 P3 1 ,…,P M 1 The pose here is in the mobile device coordinate system and has no relation to the Basemap spatial range.

[0095] Step S103: Based on the initial pose set P1, obtain the relative poses between each pair of sequential images F to obtain the first pose set T.

[0096] Let T = {T1, T2, T3, ..., T} M}:

[0097] T1={T 11 ,T 12 ,T 13 ,T 14 ,…,T 1M}, T1 is the sequence image F1 with sequence number 1, relative to the sequence images F1, F2, F3, ..., Fm with sequence numbers 1 to M. M The relative pose set;

[0098] T2={T21 ,T 22 ,T 23 ,T 24 ,…,T 2M}, T2 is the sequence image F2 with image sequence number 2 relative to the sequence images F1, F2, F3, ..., F2 with image sequence numbers 1 to M. M The relative pose set;

[0099] T3={T 31 ,T 32 ,T 33 ,T 34 ,…,T 3M}, T3 is the sequence image F3 with sequence number 3, relative to the sequence images F1, F2, F3, ..., F with sequence numbers 1 to M. M The relative pose set; and so on,

[0100] T M ={T M1 ,T M2 ,T M3 ,…,T MM}, T M Image sequence F with sequence number M M Relative to the sequence of images F1, F2, F3, ..., Fm numbered from 1 to M M The relative pose set.

[0101] When performing image localization in a computer, an image sequence processing window is set up in the computer, and the camera parameters of all sequence images in the window are read. The relative pose T1 between any two images in the window is calculated based on the pose P1, so as to calculate the 2D-3D matching data between all sequence images and visual map image data through subsequent steps S104 to S106.

[0102] In this step, the camera parameters and the first pose set T also need to be obtained through the following process:

[0103] Given a window by the computer, read the camera parameters of all image sequences within the window; calculate the relative pose T between any two images within the window based on pose P1. Then, perform 2D-3D matching between all queries used for localization and the map within the window. (Continue to refer to...) Figure 1 :

[0104] Step S104: Extract image features from each image sequence to obtain the corresponding second image feature set J = {J1, J2, J3, ..., J...} M} and the two-dimensional image feature coordinate set R={R1,R2,R3,…,R M}

[0105] Step S105: Based on the image features of each sequence of images in the second image feature set J, find the matching image features in the first image feature set to form a third image feature set J', where J' = {J1', J2', J3', ..., J...} M '}.

[0106] Step S106: Based on the third image feature set J', obtain the three-dimensional point cloud coordinate set corresponding to the image features in the first three-dimensional point cloud coordinate set from the third image feature set J', so as to form the second three-dimensional point cloud coordinate set S = {S1, S2, S3, ..., S} corresponding to the second image feature set J. M}

[0107] Let N1, N2, N3, ..., (NM) be natural numbers greater than 1, and let F1, F2, F3, ..., FM be the sequence images respectively. M The number of image features extracted;

[0108] Based on the second image feature set J = {J1, J2, J3, ..., J...} M}, Two-dimensional image feature coordinate set R = {R1, R2, R3, ..., R M}, the third image feature set J', J'={J1',J2',J3',…,J M '} and the second three-dimensional point cloud coordinate set S={S1,S2,S3,…,S M};

[0109] J1={j 11 ,j 12 ,j 13 ,…,j 1N1}、R1={r 11 ,r 12 ,r 13 ,…,r 1N1}、J1'={j 11 ',j 12 ',j 13 ',…,j 1N1 '}、S1={s 11 ,s 12 ,s 13 ,…,s 1N1};j 11 ,j 12 ,j 13 ,…,j 1N1 r represents the image feature elements extracted from the sequence image F1. 11 ,r 12 ,r 13 ,…,r 1N1 Image feature element j extracted from sequence image F111 ,j 12 ,j 13 ,…,j 1N1 In the device coordinate system, the two-dimensional coordinate element j 11 ',j 12 ',j 13 ',…,j 1N1 'For the first image feature set, respectively with j 11 ,j 12 ,j 13 ,…,j 1N1 Matched image features, s 11 ,s 12 ,s 13 ,…,s 1N1 For the first three-dimensional point cloud coordinate set and image features j 11 ',j 12 ',j 13 ',…,j 1N1 'Corresponding 3D point cloud coordinate elements;

[0110] J2={j 21 ,j 22 ,j 23 ,…,j 2N2}、R2={r 21 ,r 22 ,r 23 ,…,r 2N2}、J2'={j 21 ',j 22 ',j 23 ',…,j 2N2 '}、S2={s 21 ,s 22 ,s 23 ,…,s 2N2};j 21 ,j 22 ,j 23 ,…,j 2N2 r represents the image feature elements extracted from the sequence image F2. 21 ,r 22 ,r 23 ,…,r 2N2 Image feature element j extracted from sequence image F2 21 ,j 22 ,j 23 ,…,j 2N2 In the device coordinate system, the two-dimensional coordinate element j 21 ',j 22 ',j 23 ',…,j 2N2 'For the first image feature set, respectively with j 21,j 22 ,j 23 ,…,j 2N2 Matched image features, s 21 ,s 22 ,s 23 ,…,s 2N2 For the first three-dimensional point cloud coordinate set and image features j 21 ',j 22 ',j 23 ',…,j 2N2 'Corresponding 3D point cloud coordinate elements;

[0111] J3={j 31 ,j 32 ,j 33 ,…,j 3N3}、R3={r 31 ,r 32 ,r 33 ,…,r 3N3}、J3'={j 31 ',j 32 ',j 33 ',…,j 3N3 '}、S3={s 31 ,s 32 ,s 33 ,…,s 3N3};j 31 ,j 32 ,j 33 ,…,j 3N3 r represents the image feature elements extracted from the sequence image F3. 31 ,r 32 ,r 33 ,…,r 3N3 Image feature element j extracted from sequence image F3 31 ,j 32 ,j 33 ,…,j 3N3 In the device coordinate system, the two-dimensional coordinate element j 31 ',j 32 ',j 33 ',…,j 3N3 'For the first image feature set, respectively with j 31 ,j 32 ,j 33 ,…,j 3N3 Matched image features, s 31 ,s 32 ,s 33 ,…,s 3N3 For the first three-dimensional point cloud coordinate set and image features j 31 ',j 32',j 33 ',…,j 3N3 'Corresponding 3D point cloud coordinate elements;

[0112] And so on,

[0113] J M ={j M1 ,j M2 ,j M3 ,…,j M(NM)}、R M ={r M1 ,r M2 ,r M3 ,…,r M(NM)}、J M '={j M1 ',j M2 ',j M3 ',…,j M(NM) '}、S M ={s M1 ,s M2 ,s M3 ,…,s M(NM)};j M1 ,j M2 ,j M3 ,…,j M(NM) For the sequence image F M The extracted image feature elements, r M1 ,r M2 ,r M3 ,…,r M(NM) For the sequence image F M Extracted image feature element j M1 ,j M2 ,j M3 ,…,j M(NM) In the device coordinate system, the two-dimensional coordinate element j M1 ',j M2 ',j M3 ',…,j M(NM) 'For the first image feature set, respectively with j M1 ,j M2 ,j M3 ,…,j M(NM) Matched image features, s M1 ,s M2 ,s M3 ,…,s M(NM) For the first three-dimensional point cloud coordinate set and image features j M1 ',j M2 ',j M3 ',…,j M(NM) 'Corresponding 3D point cloud coordinate elements.

[0114] Step S107: Based on the two-dimensional image feature coordinate set and the corresponding second three-dimensional point cloud coordinate set, obtain the second pose set P2 of the sequence image, P2 = {P1} 2 P2 2 P3 2 ,…,P M 2}

[0115] In the above steps, a robust pose can be calculated using Ransac. Specifically, first, any image within the computer window is randomly obtained, and then the pose P2 of the sequence of images is calculated using Ransac and the PnP algorithm.

[0116] Step S108: Based on the first pose set T and the second pose set P2, calculate the third pose set P3 of each image sequence relative to other image sequences, where P3 = {P1} 3 P2 3 P3 3 ,…,P M 3}

[0117] in:

[0118] P1 3 ={P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3}, P1 3 For image sequence number 1, image F1 is positioned at pose P1. 2 Based on the relative pose set T1={T 11 ,T 12 ,T 13 ,T 14 ,…,T 1M The sequence image F1 is calculated relative to the sequence images F1, F2, F3, ..., Fm numbered from 1 to M. M Keyframe pose set, P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 For pose elements;

[0119] P23 ={P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3}, P2 3 Image sequence number 2, image F2, is positioned at pose P2. 2 Based on the relative pose set T2={T 21 ,T 22 ,T 23 ,T 24 ,…,T 2M The sequence image F2 is calculated to be relative to the sequence images F1, F2, F3, ..., F2 numbered from 1 to M. M Keyframe pose set, P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 For pose elements;

[0120] P3 3 ={P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3}, P3 3 Image sequence number 3, image F3, is positioned at pose P3. 2 Based on the relative pose set T3={T 31 ,T 32 ,T 33 ,…,T 3M The sequence image F3 is calculated to be relative to the sequence images F1, F2, F3, ..., Fm numbered from 1 to M. M Keyframe pose set, P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 For pose elements; and so on,

[0121] P M 3 ={P M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3}, P M 3 Image sequence F with sequence number M M In its pose P M 2 Based on the relative pose set T M ={T M1 ,T M2 ,T M3 ,…,T MM The sequence image F was calculated. M Relative to the sequence of images F1, F2, F3, ..., Fm numbered from 1 to M M Keyframe pose set, P M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 For pose elements;

[0122] Step S109: Given a projection error threshold Th, based on the pose element set P1 of each sequence image in the third pose set P3 relative to other sequence images... 3 P2 3 P3 3 ,…,P M 3 Calculate the set of error values ​​△X between the projection of the three-dimensional points of the image features corresponding to each pose element onto the corresponding two-dimensional points in the device coordinate system.

[0123] Let △X = {△X1, △X2, △X3, ..., △X} M}:

[0124] △X1 is the pose element P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The corresponding set of error values, △X1={△X 11 ,△X 12 ,△X13 ,△X 14 ,…,△X 1M Specifically: △X 11 It is the 3D point cloud coordinate set S1 corresponding to the sequence image F1, based on the pose element P of the corresponding sequence image. 11 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1; △X 12 It is the 3D point cloud coordinate set S2 corresponding to the sequence image F2, based on the pose element P of the corresponding sequence image. 12 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R2; △X 13 It is the 3D point cloud coordinate set S3 corresponding to the sequence image F3, based on the pose element P of the corresponding sequence image. 13 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R3; and so on, △X 1M It is a sequence of images F M The corresponding 3D point cloud coordinate set S M Based on the pose element P of the corresponding sequence image 1M 3 The projected coordinates obtained by projecting onto the device coordinate system, and the two-dimensional image feature coordinate set R M The set of error values ​​between them. More specifically, let the sequence of images be F1, F2, F3, ..., F... M The three-dimensional point cloud coordinate set S1, S2, S3, ..., S M They are respectively:

[0125] S1={s 11 ,s 12 ,s 13 ,…,s 1N1}={(X 11 Y 11 Z 11 ), (X 12 Y 12 Z 12 ),(X 13 Y 13 Z 13 ), ..., (X 1N1 Y 1N1 Z 1N1 )};

[0126] S2={s 21 ,s 22 ,s 23 ,…,s 2N2}={(X 21, Y 21 , Z 21 ), (X 22 , Y 22 , Z 22 ), (X 23 , Y 23 , Z 23 ), ..., (X 2N2 , Y 2N2 , Z 2N2 )};

[0127] S3={s 31 ,s 32 ,s 33 ,...,s 3N3}={(X 31 , Y 31 , Z 31 ), (X 32 , Y 32 , Z 32 ),(X 33 , Y 33 , Z 33 ), ..., (X 3N3 , Y 3N3 , Z 3N3 )};

[0128] ......

[0129] S M ={s M1 ,s M2 ,s M3 ,...,s M(NM)}={(X M1 , Y M1 , Z 11 ), (X M2 , Y M2 , Z M2 ),(X M3 , Y M3 , Z M3 ), ..., (X M(NM) , Y M(NM) , Z M(NM) )};

[0130] Two-dimensional image feature coordinate sets R1, R2, R3, ..., R M are respectively:

[0131] R1={r 11 ,r 12 ,r 13 ,...,r 1N1}={(x 11 , y 11 ), (x 12 , y12 ),(x 13 y 13 ), ..., (x 1N1 y 1N1 );

[0132] R2={r 21 ,r 22 ,r 23 ,…,r 2N2}={(x 21 y 21 ), (x 22 y 22 ),(x 23 y 23 ), ..., (x 2N2 y 2N2 )};

[0133] R3={r 31 ,r 32 ,r 33 ,…,r 3N3}={(x 31 y 31 ), (x 32 y 32 ),(x 33 y 33 ), ...,

[0134] (x 3N3 y 3N3 )};

[0135] ...

[0136] R M ={r M1 ,r M2 ,r M3 ,…,r M(NM)}={(x M1 y M1 ), (x M2 y M2 ),(x M3 y M3 ), ...,

[0137] (x M(NM) y M(NM) If the camera intrinsic parameter matrix of the image is K, then the error value set ΔX is statistically analyzed as follows: 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M Error values ​​in:

[0138] △X 11={|x 11 -X 11 ×P 11 3 ×K|,|x 12 -X 12 ×P 11 3 ×K|,|x 13 -X 13 ×P 11 3 ×K|,…,|x 1N1 -X 1N1 ×P 11 3 ×K|};

[0139] △X 12 ={|x 21 -X 21 ×P 12 3 ×K|,|x 22 -X 22 ×P 12 3 ×K|,|x 23 -X 23 ×P 12 3 ×K|,…,|x 2N2 -X 2N2 ×P 12 3 ×K|};

[0140] △X 13 ={|x 31 -X 31 ×P 13 3 ×K|,|x 32 -X 32 ×P 13 3 ×K|,|x 33 -X 33 ×P 13 3 ×K|,…,|x 3N3 -X 3N3 ×P 13 3 ×K|};

[0141] ……

[0142] △X 1M ={|x M1 -X M1 ×P 1M 3 ×K|,|x M2 -XM2 ×P 1M 3 ×K|,|x M3 -X M3 ×P 1M 3 ×K|,…,|x M(NM) -X M(NM) ×P 1M 3 ×K|};

[0143] △X2 is the pose element P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The corresponding set of error values, △X2={△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M Specifically: △X 21 It is the 3D point cloud coordinate set S1 corresponding to the sequence image F1, based on the pose element P of the corresponding sequence image. 21 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1; △X 22 It is the 3D point cloud coordinate set S2 corresponding to the sequence image F2, based on the pose element P of the corresponding sequence image. 22 3 The error between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R2; △X 23 It is the 3D point cloud coordinate set S3 corresponding to the sequence image F3, based on the pose element P of the corresponding sequence image. 23 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R3; and so on, △X 2M It is a sequence of images F M The corresponding 3D point cloud coordinate set S M Based on the pose element P of the corresponding sequence image 2M 3 The projected coordinates obtained by projecting onto the device coordinate system, and the two-dimensional image feature coordinate set R M The set of error values ​​between them. More specifically, let the sequence of images be F1, F2, F3, ..., F... M The three-dimensional point cloud coordinate set S1, S2, S3, ..., S MRespectively:

[0144] S1={s 11 ,s 12 ,s 13 ,…,s 1N1}={(X 11 , Y 11 , Z 11 ), (X 12 , Y 12 , Z 12 ),(X 13 , Y 13 , Z 13 ), …, (X 1N1 , Y 1N1 , Z 1N1 )};

[0145] S2={s 21 ,s 22 ,s 23 ,…,s 2N2}={(X 21 , Y 21 , Z 21 ), (X 22 , Y 22 , Z 22 ),(X 23 , Y 23 , Z 23 ), …, (X 2N2 , Y 2N2 , Z 2N2 )};

[0146] S3={s 31 ,s 32 ,s 33 ,…,s 3N3}={(X 31 , Y 31 , Z 31 ), (X 32 , Y 32 , Z 32 ),(X 33 , Y 33 , Z 33 ), …, (X 3N3 , Y 3N3 , Z 3N3 )};

[0147] ……

[0148] S M ={s M1 ,s M2 ,s M3 ,…,s M(NM)}={(XM1 Y M1 Z 11 ), (X M2 Y M2 Z M2 ),(X M3 Y M3 Z M3 ), ..., (X M(NM) Y M(NM) Z M(NM) )};

[0149] Two-dimensional image feature coordinate set R1,R2,R3,…,R M They are respectively:

[0150] R1={r 11 ,r 12 ,r 13 ,…,r 1N1}={(x 11 y 11 ), (x 12 y 12 ),(x 13 y 13 ), ..., (x 1N1 y 1N1 );

[0151] R2={r 21 ,r 22 ,r 23 ,…,r 2N2}={(x 21 y 21 ), (x 22 y 22 ),(x 23 y 23 ), ..., (x 2N2 y 2N2 )};

[0152] R3={r 31 ,r 32 ,r 33 ,…,r 3N3}={(x 31 y 31 ), (x 32 y 32 ),(x 33 y 33 ), ..., (x 3N3 y 3N3 )};

[0153] ...

[0154] R M ={rM1 ,r M2 ,r M3 ,…,r M(NM)}={(x M1 y M1 ), (x M2 y M2 ),(x M3 y M3 ), ..., (x M(NM) y M(NM) If the camera intrinsic parameter matrix of the image is K, then the error value set ΔX is statistically analyzed as follows: 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M Error values ​​in:

[0155] △X 21 ={|x 11 -X 11 ×P 21 3 ×K|,|x 12 -X 12 ×P 21 3 ×K|,|x 13 -X 13 ×P 21 3 ×K|,…,|x 1N1 -X 1N1 ×P 21 3 ×K|};

[0156] △X 22 ={|x 21 -X 21 ×P 22 3 ×K|,|x 22 -X 22 ×P 22 3 ×K|,|x 23 -X 23 ×P 22 3 ×K|,…,|x 2N2 -X 2N2 ×P 22 3 ×K|};

[0157] △X 23 ={|x 31 -X 31 ×P 23 3 ×K|,|x32 -X 32 ×P 23 3 ×K|,|x 33 -X 33 ×P 23 3 ×K|,…,|x 3N3 -X 3N3 ×P 23 3 ×K|};

[0158] ...

[0159] △X 2M ={|x M1 -X M1 ×P 2M 3 ×K|,|x M2 -X M2 ×P 2M 3 ×K|,|x M3 -X M3 ×P 2M 3 ×K|,…,|x M(NM) -X M(NM) ×P 2M 3 ×K|};

[0160] △X3 is the pose element P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding set of error values, △X3={△X 31 ,△X 32 ,△X 33 ,…,△X 3M Specifically: △X 31 It is the 3D point cloud coordinate set S1 corresponding to the sequence image F1, based on the pose element P of the corresponding sequence image. 31 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1; △X 32 It is the 3D point cloud coordinate set S2 corresponding to the sequence image F2, based on the pose element P of the corresponding sequence image. 32 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R2; △X 33It is the 3D point cloud coordinate set S3 corresponding to the sequence image F3, based on the pose element P of the corresponding sequence image. 33 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R3; and so on, △X 3M It is a sequence of images F M The corresponding 3D point cloud coordinate set S M Based on the pose element P of the corresponding sequence image 3M 3 The projected coordinates obtained by projecting onto the device coordinate system, and the two-dimensional image feature coordinate set R M The set of error values ​​between them. More specifically, let the sequence of images be F1, F2, F3, ..., F... M The three-dimensional point cloud coordinate set S1, S2, S3, ..., S M They are respectively:

[0161] S1={s 11 ,s 12 ,s 13 ,…,s 1N1}={(X 11 Y 11 Z 11 ), (X 12 Y 12 Z 12 ),(X 13 Y 13 Z 13 ), ..., (X 1N1 Y 1N1 Z 1N1 )};

[0162] S2={s 21 ,s 22 ,s 23 ,…,s 2N2}={(X 21 Y 21 Z 21 ), (X 22 Y 22 Z 22 ),(X 23 Y 23 Z 23 ), ..., (X 2N2 Y 2N2 Z 2N2 )};

[0163] S3={s 31 ,s 32 ,s 33 ,…,s 3N3}={(X 31Y 31 Z 31 ), (X 32 Y 32 Z 32 ),(X 33 Y 33 Z 33 ), ..., (X 3N3 Y 3N3 Z 3N3 )};

[0164] ...

[0165] S M ={s M1 ,s M2 ,s M3 ,…,s M(NM)}={(X M1 Y M1 Z 11 ), (X M2 Y M2 Z M2 ),(X M3 Y M3 Z M3 ), ..., (X M(NM) Y M(NM) Z M(NM) )};

[0166] Two-dimensional image feature coordinate set R1,R2,R3,…,R M They are respectively:

[0167] R1={r 11 ,r 12 ,r 13 ,…,r 1N1}={(x 11 y 11 ), (x 12 y 12 ),(x 13 y 13 ), ..., (x 1N1 y 1N1 );

[0168] R2={r 21 ,r 22 ,r 23 ,…,r 2N2}={(x 21 y 21 ), (x 22 y 22 ),(x 23 y 23 ), ..., (x 2N2 y2N2 )};

[0169] R3={r 31 ,r 32 ,r 33 ,…,r 3N3}={(x 31 y 31 ), (x 32 y 32 ),(x 33 y 33 ), ...,

[0170] (x 3N3 y 3N3 )};

[0171] ...

[0172] R M ={r M1 ,r M2 ,r M3 ,…,r M(NM)}={(x M1 y M1 ), (x M2 y M2 ),(x M3 y M3 ), ...,

[0173] (x M(NM) y M(NM) If the camera intrinsic parameter matrix of the image is K, then the error value set ΔX is statistically analyzed as follows: 31 ,△X 32 ,△X 33 ,…,△X 3M Error values ​​in:

[0174] △X 31 ={|x 11 -X 11 ×P 31 3 ×K|,|x 12 -X 12 ×P 31 3 ×K|,|x 13 -X 13 ×P 31 3 ×K|,…,|x 1N1 -X 1N1 ×P 31 3 ×K|};

[0175] △X 32 ={|x21 -X 21 ×P 32 3 ×K|,|x 22 -X 22 ×P 32 3 ×K|,|x 23 -X 23 ×P 32 3 ×K|,…,|x 2N2 -X 2N2 ×P 32 3 ×K|};

[0176] △X 33 ={|x 31 -X 31 ×P 33 3 ×K|,|x 32 -X 32 ×P 33 3 ×K|,|x 33 -X 33 ×P 33 3 ×K|,…,|x 3N3 -X 3N3 ×P 33 3 ×K|};

[0177] ...

[0178] △X 3M ={|x M1 -X M1 ×P 3M 3 ×K|,|x M2 -X M2 ×P 3M 3 ×K|,|x M3 -X M3 ×P 3M 3 ×K|,…,|x M(NM) -X M(NM) ×P 3M 3 ×K|};

[0179] And so on,

[0180] △X M P is the pose element M1 3 ,P M2 3 ,PMM3 ,…,P MM 3 The corresponding set of error values, ΔX M ={△X M1 ,△X M2 ,△X M3 ,…,△X MM Specifically: △X M1 It is the 3D point cloud coordinate set S1 corresponding to the sequence image F1, based on the pose element P of the corresponding sequence image. M1 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1; △X M2 It is the 3D point cloud coordinate set S2 corresponding to the sequence image F2, based on the pose element P of the corresponding sequence image. M2 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R2; △X M3 It is the 3D point cloud coordinate set S3 corresponding to the sequence image F3, based on the pose element P of the corresponding sequence image. M3 3 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R3; and so on, △X MM It is a sequence of images F M The corresponding 3D point cloud coordinate set S M Based on the pose element P of the corresponding sequence image MM 3 The projected coordinates obtained by projecting onto the device coordinate system, and the two-dimensional image feature coordinate set R M The set of error values ​​between them. More specifically, let the sequence of images be F1, F2, F3, ..., F... M The three-dimensional point cloud coordinate set S1, S2, S3, ..., S M They are respectively:

[0181] S1={s 11 ,s 12 ,s 13 ,…,s 1N1}={(X 11 Y 11 Z 11 ), (X 12 Y 12 Z 12 ),(X 13 Y 13 Z 13 ), ..., (X 1N1 Y 1N1 Z 1N1 )};

[0182] S2={s 21 ,s 22 ,s 23 ,…,s 2N2}={(X 21 Y 21 Z 21 ), (X 22 Y 22 Z 22 ),(X 23 Y 23 Z 23 ), ..., (X 2N2 Y 2N2 Z 2N2 )};

[0183] S3={s 31 ,s 32 ,s 33 ,…,s 3N3}={(X 31 Y 31 Z 31 ), (X 32 Y 32 Z 32 ),(X 33 Y 33 Z 33 ), ..., (X 3N3 Y 3N3 Z 3N3 )};

[0184] ...

[0185] S M ={s M1 ,s M2 ,s M3 ,…,s M(NM)}={(X M1 Y M1 Z 11 ), (X M2 Y M2 Z M2 ),(X M3 Y M3 Z M3 ), ..., (X M(NM) Y M(NM) Z M(NM) )};

[0186] Two-dimensional image feature coordinate set R1,R2,R3,…,R M They are respectively:

[0187] R1={r 11 ,r 12,r 13 ,…,r 1N1}={(x 11 y 11 ), (x 12 y 12 ),(x 13 y 13 ), ..., (x 1N1 y 1N1 );

[0188] R2={r 21 ,r 22 ,r 23 ,…,r 2N2}={(x 21 y 21 ), (x 22 y 22 ),(x 23 y 23 ), ..., (x 2N2 y 2N2 )};

[0189] R3={r 31 ,r 32 ,r 33 ,…,r 3N3}={(x 31 y 31 ), (x 32 y 32 ),(x 33 y 33 ), ..., (x 3N3 y 3N3 )};

[0190] ...

[0191] R M ={r M1 ,r M2 ,r M3 ,…,r M(NM)}={(x M1 y M1 ), (x M2 y M2 ),(x M3 y M3 ), ..., (x M(NM) y M(NM) If the camera intrinsic parameter matrix of the image is K, then the error value set ΔX is statistically analyzed as follows: M1 ,△X M2 ,△X M3 ,…,△X MM Error values ​​in:

[0192] △X M1 ={|x 11 -X 11 ×P M1 3 ×K|,|x 12 -X 12 ×P M1 3 ×K|,|x 13 -X 13 ×P M1 3 ×K|,…,|x 1N1 -X 1N1 ×P M1 3 ×K|};

[0193] △X M2 ={|x 21 -X 21 ×P M2 3 ×K|,|x 22 -X 22 ×P M2 3 ×K|,|x 23 -X 23 ×P M2 3 ×K|,…,|x 2N2 -X 2N2 ×P M2 3 ×K|};

[0194] △X M3 ={|x 31 -X 31 ×P M3 3 ×K|,|x 32 -X 32 ×P M3 3 ×K|,|x 33 -X 33 ×P M3 3 ×K|,…,|x 3N3 -X 3N3 ×P M3 3 ×K|};

[0195] ……

[0196] △X MM ={|x M1 -X M1 ×P MM 3 ×K|,|xM2 -X M2 ×P MM 3 ×K|,|x M3 -X M3 ×P MM 3 ×K|,…,|x M(NM) -X M(NM) ×P MM 3 ×K|};

[0197] Step S110, based on the error value set △X={△X1,△X2,△X3,…,△X M}, count the number of interior points of the pose element inlier.

[0198] In step S110, let inlier = {inlier1, inlier2, inlier3, ..., inlier} M}:

[0199] inlier1 is the pose element P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The sum of the corresponding interior points, let inlier1 = inlier 11 +inlier 11 +inlier 13 +…+inlier 1M Statistical pose element P 11 3 The corresponding set of error values ​​△X 11 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 11 This is used as the pose element P 11 3 The corresponding number of interior points; count the pose element P. 12 3 The corresponding set of error values ​​△X 12 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 12 This is used as the pose element P 12 3 The corresponding number of interior points; count the pose element P. 13 3The corresponding set of error values ​​△X 13 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 13 This is used as the pose element P 13 3 The corresponding number of interior points; and so on, to count the pose element P. 1M 3 The corresponding set of error values ​​△X 1M The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 1M This is used as the pose element P 1M 3 The corresponding number of interior points.

[0200] inlier2 is the pose element P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The sum of the corresponding interior point counts, let inlier2 = inlier 21 +inlier 22 +inlier 23 +…+inlier 2M Statistical pose element P 21 3 The corresponding set of error values ​​△X 21 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 21 This is used as the pose element P 21 3 The corresponding number of interior points; count the pose element P. 22 3 The corresponding set of error values ​​△X 22 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 22 This is used as the pose element P 22 3 The corresponding number of interior points; count the pose element P. 23 3 The corresponding set of error values ​​△X 23 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 23 This is used as the pose element P23 3 The corresponding number of interior points; and so on, to count the pose element P. 2M 3 The corresponding set of error values ​​△X 2M The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 2M This is used as the pose element P 2M 3 The corresponding number of interior points.

[0201] inlier3 is the pose element P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The sum of the corresponding interior point counts, let inlier3 = inlier 31 +inlier 32 +inlier 33 +…+inlier 3M Statistical pose element P 31 3 The corresponding set of error values ​​△X 31 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 31 This is used as the pose element P 31 3 The corresponding number of interior points; count the pose element P. 32 3 The corresponding set of error values ​​△X 32 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 32 This is used as the pose element P 32 3 The corresponding number of interior points; count the pose element P. 33 3 The corresponding set of error values ​​△X 33 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 33 This is used as the pose element P 33 3 The corresponding number of interior points; and so on, to count the pose element P. 3M 3 The corresponding set of error values ​​△X 3MThe number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. 3M This is used as the pose element P 3M 3 The corresponding number of interior points. And so on.

[0202] inlier M P is the pose element M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 Let inlier be the set of statistics on the number of interior points. M ={inlier M1 ,inlier M2 ,inlier M3 ,…,inlier MM}: Statistical pose element P M1 3 The corresponding set of error values ​​△X M1 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. M1 This is used as the pose element P M1 3 The corresponding number of interior points; count the pose element P. M2 3 The corresponding set of error values ​​△X M2 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. M2 This is used as the pose element P M2 3 The corresponding number of interior points; count the pose element P. M3 3 The corresponding set of error values ​​△X M3 The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. M3 This is used as the pose element P M3 3 The corresponding number of interior points; and so on, to count the pose element P. MM 3 The corresponding set of error values ​​△X MM The number of error values ​​less than the threshold Th is recorded as the number of inliers, denoted as inlier. MM This is used as the pose element P MM 3The corresponding number of interior points.

[0203] Step S111: Count the number of interior points of the pose element in the set inlier, i.e., inlier1, inlier2, inlier3, ..., inlier M The maximum value and corresponding sequence number.

[0204] Take inlier1, inlier2, inlier3, ..., inlier from the inlier set. M The maximum value is denoted as inlier. max , where max is the corresponding image sequence number.

[0205] Let the number of interior points be inlier max The corresponding keyframe pose P max 3 ;P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3};

[0206] The sequence of images F1, F3, ..., F M The current pose set P4 is:

[0207] P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3}

[0208] Step S112, output the current pose set P4 of the sequence image F, P4 = {P1} 4 P2 4 P3 4 ,…,P M 4}

[0209] Where: P4 = {P1} 4 P2 4 P3 4 ,…,P M 4}=P max 3 ={P max13 ,P max2 3 ,P max3 3 ,…,P max(M) 3}

[0210] In the above embodiments, the technical solution of the present invention calculates the pose P3 of the current frame and the poses P4 of the remaining frames based on the relative pose T1, and counts the number of inliers that meet the projection error. Through multiple iterations, the pose P3 with the most inliers is found as the initial value for localization. When selecting inliers, the backprojection error is calculated using the pose obtained for each image. Matching pairs with a value less than a certain threshold are classified as inliers, and the model with the most inliers is selected as the current pose corresponding to the sequence of images.

[0211] Corresponding to the above method steps, the present invention also provides a time-series-based visual map positioning system, including: a first acquisition unit, a receiving unit, a first calculation unit, a second calculation unit, an extraction unit, a matching unit, a second acquisition unit, a third calculation unit, a fourth calculation unit, a fifth calculation unit, a first statistics unit, a second statistics unit, and a first output unit, respectively adapted to perform the above steps S100 to S112.

[0212] In one optional embodiment of the present invention, based on the above embodiments, a time-series-based visual map localization method is also provided, such as... Figure 2 As shown, it includes the following steps:

[0213] Steps S200 to S212 are based on the same principles as steps S100 to S112 described above, and will not be repeated in this embodiment.

[0214] Let P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3}={P1 4 P2 4 P3 4 ,…,P M 4}, which are the sequential images F1, F2, F3, ..., F M The current frame pose.

[0215] The time-series-based visual map localization method in this embodiment further includes the following steps:

[0216] Step S213: Calculate F1, F2, F3, ..., F for each sequence of images. M The set of error values ​​△X' between the projection of the 3D points of the image features corresponding to the current frame pose onto the corresponding 2D points in the device coordinate system, where △X' = {△X1', △X2', △X3', ..., △X M '}.

[0217] Step S214, based on the error value set △X'={△X1',△X2',△X3',…,△X M '}, for the current frame pose P1 in the current pose set P4 4 P2 4 P3 4 ,…,P M 4 Optimize to ensure that the error value elements △X1', △X2', △X3', ..., △X in the error value set △X' are optimized. M Minimize the sum of the error values.

[0218] Let the optimized current pose set be P5, P5 = {P1} 5 P2 5 P3 5 ,…,P M 5}, P1 5 P2 5 P3 5 ,…,P M 5 These are the optimized pose elements P1 4 P2 4 P3 4 ,…,P M 4 ;

[0219] The time-series-based visual map localization method in this embodiment further includes the following steps:

[0220] Step S215: Output the optimized current pose set P5.

[0221] During steps S213 to S215:

[0222] The current frame pose P1 in the current pose set P4 is determined as follows: 4 P2 4 P3 4 ,…,P M 4 Optimize:

[0223] △X1',△X2',△X3',…,△X M 'Represents the sequence images F1, F2, F3, ..., FM The corresponding image feature sets J1, J2, J3, ..., J M The corresponding second three-dimensional point cloud coordinate set S1, S2, S3, ..., S M Based on the current frame pose P1 4 P2 4 P3 4 ,…,P M 4 The projected coordinate set obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1, R2, R3, ..., R M The error value between;

[0224] Let E min =△X1'+△X2'+△X3'+,…,+△X M Adjust the pose of the current frame P1 4 P2 4 P3 4 ,…,P M 4 The value of E makes min Minimize the error value;

[0225] Let the adjusted current frame pose be P1. 4 ',P2 4 ',P3 4 ',…,P M 4 ', The optimized current pose set P5 = {P1} 5 P2 5 P3 5 ,…,P M 5}={P1 4 ',P2 4 ',P3 4 ',…,P M 4 '}.

[0226] More specifically,

[0227] Let △X' = {△X1', △X2', △X3', ..., △X M '}:

[0228] △X1' is the 3D point cloud coordinate set S1 corresponding to the sequence image F1, based on the current frame pose P1 of the corresponding sequence image. 4 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1; △X2' is the three-dimensional point cloud coordinate set S2 corresponding to the sequence image F2, based on the current frame pose P2 of the corresponding sequence image. 4The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R2; △X3' is the three-dimensional point cloud coordinate set S3 corresponding to the sequence image F3, based on the current frame pose P3 of the corresponding sequence image. 4 The set of error values ​​between the projected coordinates obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R3; and so on, △X M 'Is a sequence of images F M The corresponding 3D point cloud coordinate set S M Based on the current frame pose P of the corresponding image sequence M 4 The projected coordinates obtained by projecting onto the device coordinate system, and the two-dimensional image feature coordinate set R M The set of error values ​​between them.

[0229] E min =△X1'+△X2'+△X3'+,…,+△X M ', that is, adjusting the pose of the current frame P1 4 P2 4 P3 4 ,…,P M 4 The values ​​are such that the set of error values ​​△X1'+△X2'+△X3'+,…,+△X M The sum of the error values ​​within each frame is minimized, thus obtaining the optimal pose for the current frame.

[0230] More specifically, let the sequence of images be F1, F2, F3, ..., F M The three-dimensional point cloud coordinate set S1, S2, S3, ..., S M They are respectively:

[0231] S1={s 11 ,s 12 ,s 13 ,…,s 1N1}={(X 11 Y 11 Z 11 ), (X 12 Y 12 Z 12 ),(X 13 Y 13 Z 13 ), ..., (X 1N1 Y 1N1 Z 1N1 )};

[0232] S2={s 21 ,s 22 ,s 23 ,…,s 2N2}={(X 21 Y 21 Z 21 ), (X 22 Y 22 Z 22 ),(X 23 Y 23 Z 23 ), ..., (X 2N2 Y 2N2 Z 2N2 )};

[0233] S3={s 31 ,s 32 ,s 33 ,…,s 3N3}={(X 31 Y 31 Z 31 ), (X 32 Y 32 Z 32 ),(X 33 Y 33 Z 33 ), ..., (X 3N3 Y 3N3 Z 3N3 )};

[0234] ...

[0235] S M ={s M1 ,s M2 ,s M3 ,…,s M(NM)}={(X M1 Y M1 Z 11 ), (X M2 Y M2 Z M2 ),(X M3 Y M3 Z M3 ), ..., (X M(NM) Y M(NM) Z M(NM) )};

[0236] Two-dimensional image feature coordinate set R1,R2,R3,…,R M They are respectively:

[0237] R1={r 11 ,r 12 ,r 13 ,…,r 1N1}={(x 11 y 11 ), (x12 y 12 ),(x 13 y 13 ), ..., (x 1N1 y 1N1 );

[0238] R2={r 21 ,r 22 ,r 23 ,…,r 2N2}={(x 21 y 21 ), (x 22 y 22 ),(x 23 y 23 ), ..., (x 2N2 y 2N2 )};

[0239] R3={r 31 ,r 32 ,r 33 ,…,r 3N3}={(x 31 y 31 ), (x 32 y 32 ),(x 33 y 33 ), ..., (x 3N3 y 3N3 )};

[0240] ...

[0241] R M ={r M1 ,r M2 ,r M3 ,…,r M(NM)}={(x M1 y M1 ), (x M2 y M2 ),(x M3 y M3 ), ..., (x M(NM) y M(NM) If the camera intrinsic parameter matrix of the image is k, then the set of error values ​​ΔX1'+ΔX2'+ΔX3'+,…,+ΔX is calculated as follows: M Error values ​​in ':

[0242] △X1'={|x 11 -X 11 ×P1 4 ×k|,|x 12 -X 12 ×P14 ×k|,|x 13 -X 13 ×P1 4 ×k|,…,|x 1N1 -X 1N1 ×P1 4 ×k|};

[0243] △X2’={|x 21 -X 21 ×P2 4 ×k|,|x 22 -X 22 ×P2 4 ×k|,|x 23 -X 23 ×P2 4 ×k|,…,|x 2N2 -X 2N2 ×P2 4 ×k|};

[0244] △X3’={|x 31 -X 31 ×P3 4 ×k|,|x 32 -X 32 ×P3 4 ×k|,|x 33 -X 33 ×P3 4 ×k|,…,|x 3N3 -X 3N3 ×P3 4 ×k|};

[0245] ……

[0246] △X M ’={|x M1 -X M1 ×P M 4 ×k|,|x M2 -X M2 ×P M 4 ×k|,|x M3 -X M3 ×P M 4 ×k|,…,|x M(NM) -X M(NM) ×P M 4 ×k|}。

[0247] E min =△X1’+△X2’+△X3’+,…,+△X M ’=|x 11-X 11 ×P1 4 ×k|+|x 12 -X 12 ×P1 4 ×k|+|x 13 -X 13 ×P1 4 ×k|+…+|x 1N1 -X 1N1 ×P1 4 ×k|+|x 21 -X 21 ×P2 4 ×k|+|x 22 -X 22 ×P2 4 ×k|+|x 23 -X 23 ×P2 4 ×k|+…+|x 2N2 -X 2N2 ×P2 4 ×k|+|x 31 -X 31 ×P3 4 ×k|+|x 32 -X 32 ×P3 4 ×k|+|x 33 -X 33 ×P3 4 ×k|+…+|x 3N3 -X 3N3 ×P3 4 ×k|++…++|x M1 -X M1 ×P M 4 ×k|+|x M2 -X M2 ×P M 4 ×k|+|x M3 -X M3 ×P M 4 ×k|+…+|x M(NM) -X M(NM) ×P M 4 ×k|。

[0248] In the above optional solutions, the technical solution of the present invention uses the initial value P3 of the current frame and the relative pose T1 to calculate the initial pose of other frames within the window, and marks all 2D-3D matching pairs of images within the window that satisfy the projection error as inliers. In addition to the constraint of the conventional backprojection error, the constraint of the image pair pose T1 is added to form rigBA, resulting in a more accurate pose P5, which is the pose of the current frame to be determined. The backprojection error is composed of the errors of all frames, and is the difference between the projection of the corresponding 3D point of each frame onto P3 and the 2D projection of each frame.

[0249] Based on the above method steps, the present invention also provides a time-series-based visual map positioning system, comprising: a first acquisition unit, a receiving unit, a first calculation unit, a second calculation unit, an extraction unit, a matching unit, a second acquisition unit, a third calculation unit, a fourth calculation unit, a fifth calculation unit, a first statistics unit, a second statistics unit, and a first output unit, respectively adapted to perform the above steps S200 to S212; and further comprising: a sixth calculation unit, an optimization unit, and a second output unit, respectively adapted to perform the above steps S213 to S215.

[0250] In an application example of the technical solution of the present invention, a visual map localization method based on time series may include the following steps:

[0251] Step a1: Acquire video sequences and build a basemap. Step a1 specifically includes: Step a11, acquiring video sequences using devices such as panoramic / fisheye / digital cameras; Step a12, building a high-precision visual map, denoted as the basemap, which includes, but is not limited to: the pose of keyframes, the positions of 2D feature points and their corresponding descriptors, the correspondence between 2D and 3D points, camera parameters, the positions of 3D points, and errors.

[0252] Step a2: Obtain the pose {P1} of the mobile device. Step a2 specifically includes: Step a21: The mobile device acquires a video sequence within the basemap area, and the images in the video sequence are denoted as {query}; Step a22: Use visual inertial odometry (VIO) to obtain the pose {P1} between all images in the video sequence.

[0253] Step a3: Obtain camera parameters and relative pose {T}. Step a3 specifically includes: Step a31, given a window, read the camera parameters of all sequence image queries within the window; Step a32, calculate the relative pose {T1} between any two images within the window based on pose P1; Step a33, perform 2D-3D matching between all queries used for localization and the map within the window.

[0254] Step a4: Ransac calculates a more robust pose. Step a4 specifically includes: step a41, let the window length be N, and the number of positioning images in the queue be M, then the effective window length k=min(N,M); for each positioning image query i (i={1,2,3...K}), based on the 2D-3D matching results, if Ransac+PnP is successfully used, the pose of the query image can be calculated all together constitute P2; step a42, for the in step a41 through the relative pose {T1} in step a32, the pose of each frame can be obtained (i indicates which pose the current pose is derived from, and j indicates which keyframe pose is推算ed according to {T1}); step a43, for the pose of each image j and a given projection error threshold Th, the number of matching pairs within the range conforming to this threshold can be calculated, and the number of such matching pairs is the number of inliers. Step a43 further includes:

[0255] Step a431, there is a set of three-dimensional points X=[X,Y,Z], corresponding image coordinates x=[x,y], the camera internal parameter matrix K of the image and the calculated transformation pose such that the error between the projection x^ of the three-dimensional point on the image and the actual observed x can be expressed as:

[0256]

[0257] Step a432, if Δx<Th, it is an inlier. Corresponding to m matched 2D-3D matching pairs for an image, the number of inliers is recorded as inlier i j ;

[0258] Step a433, through corresponding inlier i j number, it can be statistically obtained corresponding the image serial number n corresponding to the maximum number of inliers is obtained through comparison, and the pose of the current frame can be obtained as

[0259] Step a5: Calibrate the initial pose and rigBA. Step a5 specifically includes: Step a51, using P4 and the relative pose {T1}, the poses P5 of other frames within the window can be calculated. For each image within the window, the pose P can be optimized using the projection method in step a431 to minimize Δx. This is the conventional projection error constraint; Step a52, based on the projection error in step a51, the constraint of the relative pose T1 is added for overall optimization. At this point, the only variable being optimized is the pose P4. Through optimization, a more accurate pose P6 is obtained, and P6 is the pose of the current frame to be determined.

[0260] In the above steps, based on the Δx of each image, let the image index i = {1, 2, 3... K}, the objective optimization equation is: e = Δx1 + Δx2 + ... + Δx k The overall optimization objective of step a5 is to minimize equation e.

[0261] In the above steps, let the current image be i, and Δx i =||x i -K i ·P4·T i k ·X i ||,T i k This represents the relative pose from the current frame (index k) to the i-th image, that is, the relative pose between the current frame image with index k and the image with index i, obtained based on the set {T1} in step a32.

[0262] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A visual map localization method based on time series, characterized in that, include: Acquire visual map data, which includes: a first image feature set and a corresponding first three-dimensional point cloud coordinate set; Receive the sequence of images F to be located, F = {F1, F2, F3, ..., F...} M }; M is the image sequence number, where M is a natural number greater than 1; Calculate the initial pose set P1 of the sequence image F in the device coordinate system, P1 = {P1...} 1 P2 1 P3 1 ,…,P M 1 }; Based on the initial pose set P1, the relative poses between each pair of sequential images F are obtained to obtain the first pose set T, T = {T1, T2, T3, ..., T...}. M }; Extract image features from each image sequence to obtain the corresponding second image feature set J = {J1, J2, J3, ..., J...} M } and the two-dimensional image feature coordinate set R={R1,R2,R3,…,R M }; Based on the image features of each sequence of images in the second image feature set J, find the matching image features in the first image feature set to form a third image feature set J', where J' = {J1', J2', J3', ..., J...} M '}; Based on the third image feature set J', a three-dimensional point cloud coordinate set corresponding to the image features in the first three-dimensional point cloud coordinate set is obtained from the third image feature set J', so as to form a second three-dimensional point cloud coordinate set S = {S1, S2, S3, ..., S} corresponding to the second image feature set J. M }; Based on the two-dimensional image feature coordinate set and the corresponding second three-dimensional point cloud coordinate set, the second pose set P2 of the sequence image is obtained, where P2 = {P1} 2 P2 2 P3 2 ,…,P M 2 }; Based on the first pose set T and the second pose set P2, calculate the third pose set P3 of each image sequence relative to the other image sequences, where P3 = {P1} 3 P2 3 P3 3 ,…,P M 3 };P1 3 For the sequence image F1 in its pose P1 2 Based on the relative pose set T1, the sequence images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P2 3 For the sequence image F2 in its pose P2 2 Based on the relative pose set T2, the sequence of images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P3 3 For the sequence image F3 in its pose P3 2 Based on the relative pose set T3, the sequence of images F1, F2, F3, F4, ..., F are calculated. M The keyframe pose set; and so on, P M 3 For the sequence image F M In its pose P M 2 Based on the relative pose set T M The calculated sequence images F1, F2, F3, ..., F (M-1) ,F M Keyframe pose set; Given a projection error threshold Th, based on the pose element set P1 of each sequence image in the third pose set P3 relative to other sequence images... 3 P2 3 P3 3 ,…,P M 3 Calculate the set of error values ​​ΔX between the projection of the 3D points of the image features corresponding to each pose element onto the corresponding 2D points in the device coordinate system, where ΔX = ΔX1, ΔX2, ΔX3, ..., ΔX M ; Based on the error value set △X={△X1,△X2,△X3,…,△X M }, statistically analyze the sequence images F1, F2, F3, ..., F M The set of inliers whose pose element projection error values ​​are less than the threshold Th is inlier; inlier = {inlier1, inlier2, inlier3, ..., inlier} M }; Based on the sequence images F1, F2, F3, ..., F M The corresponding number of inliers, counting the number of inliers in the set inlier1, inlier2, inlier3, ..., inlier M Let the set corresponding to the maximum number of interior points be inlier. max , where max is the image number; The corresponding pose element set P max 3 As the sequence images F1, F3, ..., F M The current pose P4, P4 = P max 3 .

2. The time-series-based visual map localization method as described in claim 1, characterized in that, Based on the first pose set T, since T = {T1, T2, T3, ..., T M }: T1={T 11 ,T 12 ,T 13 ,T 14 ,…,T 1M }, T1 is the sequence image F1 with sequence number 1, relative to the sequence images F1, F2, F3, ..., Fm with sequence numbers 1 to M. M The relative pose set; T2={T 21 ,T 22 ,T 23 ,T 24 ,…,T 2M }, T2 is the sequence image F2 with image sequence number 2 relative to the sequence images F1, F2, F3, ..., F2 with image sequence numbers 1 to M. M The relative pose set; T3={T 31 ,T 32 ,T 33 ,…,T 3M }, T3 is the sequence image F3 with sequence number 3, relative to the sequence images F1, F2, F3, ..., F with sequence numbers 1 to M. M The relative pose set; and so on, T M ={T M1 ,T M2 ,T M3 ,…,T MM }, T M Image sequence F with sequence number M M Relative to the sequence of images F1, F2, F3, ..., Fm numbered from 1 to M M The relative pose set.

3. The time-series-based visual map localization method as described in claim 1, characterized in that, Let N1, N2, N3, ..., (NM) be natural numbers greater than 1, and let F1, F2, F3, ..., FM be the sequence images respectively. M The number of image features extracted; Based on the second image feature set J = {J1, J2, J3, ..., J...} M }, Two-dimensional image feature coordinate set R = {R1, R2, R3, ..., R M }, the third image feature set J', J'={J1',J2',J3',…,J M '} and the second three-dimensional point cloud coordinate set S={S1,S2,S3,…,S M }; J1={j 11 ,j 12 ,j 13 ,…,j 1N1 }、R1={r 11 ,r 12 ,r 13 ,…,r 1N1 }、J1’={j 11 ’,j 12 ’, j 13 ',…,j 1N1 '}、S1={s 11 ,s 12 ,s 13 ,…,s 1N1 };j 11 ,j 12 ,j 13 ,…,j 1N1 r represents the image feature elements extracted from the sequence image F1. 11 ,r 12 ,r 13 ,…,r 1N1 Image feature element j extracted from sequence image F1 11 ,j 12 ,j 13 ,…,j 1N1 In the device coordinate system, the two-dimensional coordinate element j 11 ',j 12 ',j 13 ',…,j 1N1 'For the first image feature set, respectively with j 11 ,j 12 ,j 13 ,…,j 1N1 Matched image features, s 11 ,s 12 ,s 13 ,…,s 1N1 For the first three-dimensional point cloud coordinate set and image features j 11 ',j 12 ',j 13 ',…,j 1N1 'Corresponding 3D point cloud coordinate elements; J2={j 21 ,j 22 ,j 23 ,…,j 2N2 }、R2={r 21 ,r 22 ,r 23 ,…,r 2N2 }、J2’={j 21 ’,j 22 ’, j 23 ',…,j 2N2 '}、S2={s 21 ,s 22 ,s 23 ,…,s 2N2 };j 21 ,j 22 ,j 23 ,…,j 2N2 r represents the image feature elements extracted from the sequence image F2. 21 ,r 22 ,r 23 ,…,r 2N2 Image feature element j extracted from sequence image F2 21 ,j 22 ,j 23 ,…,j 2N2 In the device coordinate system, the two-dimensional coordinate element j 21 ',j 22 ',j 23 ',…,j 2N2 'For the first image feature set, respectively with j 21 ,j 22 ,j 23 ,…,j 2N2 Matched image features, s 21 ,s 22 ,s 23 ,…,s 2N2 For the first three-dimensional point cloud coordinate set and image features j 21 ',j 22 ',j 23 ',…,j 2N2 'Corresponding 3D point cloud coordinate elements; J3={j 31 ,j 32 ,j 33 ,…,j 3N3 }、R3={r 31 ,r 32 ,r 33 ,…,r 3N3 }、J3’={j 31 ’,j 32 ’, j 33 ',…,j 3N3 '}、S3={s 31 ,s 32 ,s 33 ,…,s 3N3 };j 31 ,j 32 ,j 33 ,…,j 3N3 r represents the image feature elements extracted from the sequence image F3. 31 ,r 32 ,r 33 ,…,r 3N3 Image feature element j extracted from sequence image F3 31 ,j 32 ,j 33 ,…,j 3N3 In the device coordinate system, the two-dimensional coordinate element j 31 ',j 32 ',j 33 ',…,j 3N3 'For the first image feature set, respectively with j 31 ,j 32 ,j 33 ,…,j 3N3 Matched image features, s 31 ,s 32 ,s 33 ,…,s 3N3 For the first three-dimensional point cloud coordinate set and image features j 31 ',j 32 ',j 33 ',…,j 3N3 'Corresponding 3D point cloud coordinate elements; And so on, J M ={j M1 ,j M2 ,j M3 ,…,j M(NM) }、R M ={r M1 ,r M2 ,r M3 ,…,r M(NM) }、J M '={j M1 ',j M2 ', j M3 ',…,j M(NM) '}、S M ={s M1 ,s M2 ,s M3 ,…,s M(NM) };j M1 ,j M2 ,j M3 ,…,j M(NM) for Sequence Image F M The extracted image feature elements, r M1 ,r M2 ,r M3 ,…,r M(NM) For the sequence image F M Extracted image feature element j M1 ,j M2 ,j M3 ,…,j M(NM) In the device coordinate system, the two-dimensional coordinate element j M1 ',j M2 ',j M3 ',…,j M(NM) 'For the first image feature set, respectively with j M1 ,j M2 ,j M3 ,…,j M(NM) Matched image features, s M1 ,s M2 ,s M3 ,…,s M(NM) For the first three-dimensional point cloud coordinate set and image features j M1 ',j M2 ',j M3 ',…,j M(NM) 'Corresponding 3D point cloud coordinate elements.

4. The time-series-based visual map localization method as described in claim 1, characterized in that, Based on the third pose set P3 = {P1} 3 P2 3 P3 3 ,…,P M 3 }: P1 3 ={P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 }, P1 3 For image sequence number 1, image F1 is positioned at pose P1. 2 Based on the relative pose set T1={T 11 ,T 12 ,T 13 ,T 14 ,…,T 1M The sequence image F1 is calculated relative to the sequence images F1, F2, F3, ..., Fm numbered from 1 to M. M Keyframe pose set, P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 For image sequence F1, relative to image sequence F1, F2, F3, ..., F M Pose elements; P2 3 ={P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 }, P2 3 Image sequence number 2, image F2, is positioned at pose P2. 2 Based on the relative pose set T2={T 21 ,T 22 ,T 23 ,T 24 ,…,T 2M The sequence image F2 is calculated to be relative to the sequence images F1, F2, F3, ..., F2 numbered from 1 to M. M Keyframe pose set, P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 For image sequence F2 relative to image sequence F1, F2, F3, ..., F M Pose elements; P3 3 ={P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 }, P3 3 Image sequence number 3, image F3, is positioned at pose P3. 2 Based on the relative pose set T3={T 31 ,T 32 ,T 33 ,…,T 3M The sequence image F3 is calculated to be relative to the sequence images F1, F2, F3, ..., Fm numbered from 1 to M. M Keyframe pose set, P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 For image sequence F3, relative to image sequence F1, F2, F3, ..., F M The pose element; and so on. P M 3 ={P M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 }, P M 3 Image sequence F with sequence number M M In its pose P M 2 Based on the relative pose set T M ={T M1 ,T M2 ,T M3 ,…,T MM The sequence image F was calculated. M Relative to the sequence of images F1, F2, F4, ..., Fm numbered from 1 to M M Keyframe pose set, P M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 For the sequence of images F1, F2, F4, ..., F M Pose elements.

5. The time-series-based visual map localization method as described in claim 4, characterized in that, △X1 is the pose element P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The corresponding set of error values, △X1={△X 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M };△X 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between; △X2 is the pose element P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The corresponding set of error values, △X2={△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M };△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between; △X3 is the pose element P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding set of error values, △X3={△X 31 ,△X 32 ,△X 33 ,…,△X 3M };△X 31 ,△X 32 ,△X 33 ,…,△X 3M These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between them; and so on, △X M P is the pose element M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding set of error values, ΔX M ={△X M1 ,△X M2 ,△X M3 ,…,△X MM };△X M1 ,△X M2 ,△X M3 ,…,△X MM These are the sequence images F1, F2, F4, ..., F M The corresponding 3D point cloud coordinate set S1, S2, S3, ..., S M Based on the pose element P of the corresponding sequence image M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The set of projected coordinates obtained by projecting onto the device coordinate system and the set of two-dimensional image feature coordinates R1, R2, R3, ..., R M The error value between them.

6. The time-series-based visual map localization method as described in claim 5, characterized in that, The sequence images F1, F2, F3, ..., F are statistically analyzed in the following manner. M The set of inliers whose pose element projection error values ​​are less than the threshold Th: Let inlier = {inlier1, inlier2, inlier3, ..., inlier} M }; inlier1 is the pose element P 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The sum of the corresponding interior point counts, inlier1 = inlier 11 +inlier 12 +inlier 13 +…+inlier 1M ; Calculate the pose elements P respectively 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The corresponding error value △X 11 ,△X 12 ,△X 13 ,△X 14 ,…,△X 1M The number of error values ​​less than the threshold Th is counted as the pose element P. 11 3 ,P 12 3 ,P 13 3 ,P 14 3 ,…,P 1M 3 The sum of the corresponding interior points; inlier2 is the pose element P 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The sum of the corresponding interior point counts, inlier2 = inlier 21 +inlier 22 +inlier 23 +…+inlier 2M ; Calculate the pose elements P respectively 21 3 ,P 22 3 ,P 23 3 ,…,P 2M 3 The corresponding set of error values ​​△X 21 ,△X 22 ,△X 23 ,△X 24 ,…,△X 2M The number of error values ​​less than the threshold Th is counted as the pose element P. 21 3 ,P 22 3 ,P 23 3 ,P 24 3 ,…,P 2M 3 The corresponding number of interior points inlier 21 ,inlier 22 ,inlier 23 ,inlier 14 ,…,inlier 2M ; inlier3 is the pose element P 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The sum of the corresponding interior point counts, inlier3 = inlier 31 +inlier 32 +inlier 33 +…+inlier 3M ; Calculate the pose elements P respectively 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding set of error values ​​△X 31 ,△X 32 ,△X 33 ,…,△X 3M The number of error values ​​less than the threshold Th is counted, and the sum of these counts is used as the pose element P. 31 3 ,P 32 3 ,P 33 3 ,…,P 3M 3 The corresponding number of interior points inlier 31 ,inlier 32 ,inlier 33 ,…,inlier 3M And so on, inlier M P is the pose element M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding set of interior point count statistics, inlier M ={inlier M1 ,inlier M2 ,inlier M3 ,…,inlier MM }; Calculate the pose elements P separately. M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding set of error values ​​△X M1 ,△X M2 ,△X M3 ,…,△X MM The number of error values ​​less than the threshold Th is counted, and the sum of these counts is used as the pose element P. M1 3 ,P M2 3 ,P M3 3 ,…,P MM 3 The corresponding number of interior points inlier M1 ,inlier M2 ,inlier M3 ,…,inlier MM ; Take the set of interior point counts, inlier = {inlier1, inlier2, inlier3, ..., inlier} M The maximum number of interior points in a given number is denoted as inlier. max , where max is the corresponding image sequence number; Let the number of interior points be inlier max The corresponding keyframe pose P max 3 ; P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3 }; The sequence of images F1, F3, ..., F M The current pose set P4 is: P4=P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3 }。 7. The time-series-based visual map localization method as described in any one of claims 1 to 6, characterized in that, Also includes: Let P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3 }={P1 4 P2 4 P3 4 ,…,P M 4 }, which are the sequential images F1, F2, F3, ..., F M The current frame pose; Calculate F1, F2, F3, ..., F for each image sequence. M The set of error values ​​△X' between the projection of the 3D points of the image features corresponding to the current frame pose onto the corresponding 2D points in the device coordinate system, where △X' = {△X1', △X2', △X3', ..., △X M '}; Based on the error value set △X'={△X1',△X2',△X3',…,△X M '}, for the current frame pose P1 in the current pose set P4 4 P2 4 P3 4 ,…,P M 4 Optimize to ensure that the error value elements △X1', △X2', △X3', ..., △X in the error value set △X' are optimized. M Minimize the sum of the error values ​​of '. Let the optimized current pose set be P5, P5 = {P1} 5 P2 5 P3 5 ,…,P M 5 }, P1 5 P2 5 P3 5 ,…,P M 5 These are the optimized pose elements P1 4 P2 4 P3 4 ,…,P M 4 ; Output the optimized current pose set P5.

8. The time-series-based visual map localization method as described in claim 7, characterized in that, The current frame pose P1 in the current pose set P4 is determined as follows: 4 P2 4 P3 4 ,…,P M 4 Optimize: △X1',△X2',△X3',…,△X M 'Represents the sequence images F1, F2, F3, ..., F M The corresponding image feature sets J1, J2, J3, ..., J M The corresponding second three-dimensional point cloud coordinate set S1, S2, S3, ..., S M Based on the current frame pose P1 4 P2 4 P3 4 ,…,P M 4 The projected coordinate set obtained by projecting onto the device coordinate system and the two-dimensional image feature coordinate set R1, R2, R3, ..., R M The error value between; Let E min =△X1'+△X2'+△X3'+,…,+△X M Adjust the pose of the current frame P1 4 P2 4 P3 4 ,…,P M 4 The value of E min Minimize the error value; Let the adjusted current frame pose be P1. 5 P2 5 P3 5 ,…,P M 5 The optimized current pose set P5 = {P1} 5 P2 5 P3 5 ,…,P M 5 } 9. A time-series-based visual map positioning system, characterized in that, include: The first acquisition unit is adapted to acquire visual map data, which includes: a first image feature set and a corresponding first three-dimensional point cloud coordinate set; The receiving unit is adapted to receive a sequence of images F to be located, where F = {F1, F2, F3, ..., F...} M }; M is the image sequence number, where M is a natural number greater than 1; The first computing unit is adapted to calculate the initial pose set P1 of the sequence image F in the device coordinate system, P1 = {P1...} 1 P2 1 P3 1 ,…,P M 1 }; The second computational unit is adapted to obtain the relative poses between each pair of sequential images F based on the initial pose set P1 to obtain the first pose set T, T = {T1, T2, T3, ..., T...} M }; Extraction unit, adapted to extract image features from each sequence of images to obtain the corresponding second image feature set J = {J1, J2, J3, ..., J...} M } and the two-dimensional image feature coordinate set R={R1,R2,R3,…,R M }; The matching unit is adapted to find image features in the first image feature set that match the image features of each sequence of images in the second image feature set J, thereby forming a third image feature set J', where J' = {J1', J2', J3', ..., J...} M '}; The second acquisition unit is adapted to obtain a three-dimensional point cloud coordinate set corresponding to the image features in the third image feature set J' from the first three-dimensional point cloud coordinate set, so as to form a second three-dimensional point cloud coordinate set S = {S1, S2, S3, ..., S} corresponding to the second image feature set J. M }; The third calculation unit is adapted to obtain the second pose set P2 of the sequence image based on the two-dimensional image feature coordinate set and the corresponding second three-dimensional point cloud coordinate set, where P2 = {P1} 2 P2 2 P3 2 ,…,P M 2 }; The fourth calculation unit is adapted to calculate, based on the first pose set T and the second pose set P2, a third pose set P3 for each sequence of images relative to other sequence of images, where P3 = {P1} 3 P2 3 P3 3 ,…,P M 3 };P1 3 For the sequence image F1 in its pose P1 2 Based on the relative pose set T1, the sequence images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P2 3 For the sequence image F2 in its pose P2 2 Based on the relative pose set T2, the sequence of images F1, F2, F3, ..., F are calculated. M Keyframe pose set; P3 3 For the sequence image F3 in its pose P3 2 Based on the relative pose set T3, the sequence of images F1, F2, F3, F4, ..., F are calculated. M The keyframe pose set; and so on, P M 3 For the sequence image F M In its pose P M 2 Based on the relative pose set T M The calculated sequence images F1, F2, F3, ..., F (M-1) ,F M Keyframe pose set; The fifth calculation unit, adapted to a given projection error threshold Th, calculates the pose element set P1 of each sequence image in the third pose set P3 relative to other sequence images. 3 P2 3 P3 3 ,…,P M 3 Calculate the set of error values ​​ΔX between the projection of the 3D points of the image features corresponding to each pose element onto the corresponding 2D points in the device coordinate system, where ΔX = ΔX1, ΔX2, ΔX3, ..., ΔX M ; The statistical unit is adapted to be based on the error value set △X={△X1,△X2,△X3,…,△X M }, statistically analyze the sequence images F1, F2, F3, ..., F M The set of inliers whose pose element projection error values ​​are less than the threshold Th is inlier; inlier = {inlier1, inlier2, inlier3, ..., inlier} M }; based on the sequence images F1, F2, F3, ..., F M The corresponding number of inliers, counting the number of inliers in the set inlier1, inlier2, inlier3, ..., inlier M The maximum value; let the set corresponding to the maximum number of interior points be inlier. max , where max is the image number; The first output unit is suitable for processing the corresponding pose element set P. max 3 As the sequence images F1, F3, ..., F M The current pose P4, P4 = P max 3 .

10. A time-series-based visual map positioning system as described in claim 9, characterized in that, Also includes: Let P4 = P max 3 ={P max1 3 ,P max2 3 ,P max3 3 ,…,P max(M) 3 }={P1 4 P2 4 P3 4 ,…,P M 4 }, which are the sequential images F1, F2, F3, ..., F M The current frame pose; The sixth computational unit is suitable for calculating F1, F2, F3, ..., F for each sequence of images. M The set of error values ​​△X' between the projection of the 3D points of the image features corresponding to the current frame pose onto the corresponding 2D points in the device coordinate system, where △X' = {△X1', △X2', △X3', ..., △X M '}; The optimization unit is adapted to be based on the error value set △X'={△X1',△X2',△X3',…,△X M '}, for the current frame pose P1 in the current pose set P4 4 P2 4 P3 4 ,…,P M 4 Optimize to ensure that the error value elements △X1', △X2', △X3', ..., △X in the error value set △X' are optimized. M Minimize the sum of error values; let the optimized current pose set be P5, P5 = {P1} 5 P2 5 P3 5 ,…,P M 5 }, P1 5 P2 5 P3 5 ,…,P M 5 These are the optimized pose elements P1 4 P2 4 P3 4 ,…,P M 4 ; The second output unit is suitable for outputting the optimized current pose set P5.

Images