A method for online real-time tracking and matching of circular non-coding landmarks in binocular sequence images

Abstract

The present invention discloses a method for online real-time tracking and matching of circular non-encoded markers in binocular sequence images, comprising: initializing left and right view pipelines; constructing a corresponding relationship between left and right view pipelines; associating pipelines with marker IDs; synchronously capturing a frame of image with left and right cameras; real-time detection of markers; real-time tracking of markers between frames; real-time matching of markers between left and right views; pipeline updating; determining whether to stop real-time acquisition of sequence images; if so, terminating real-time tracking and matching; otherwise, repeating the process to capture images, track, and match in real time. The present invention combines pipeline filtering algorithms, shape context algorithms, homography transformations, and epipolar constraints to construct left and right view pipeline queues and corresponding relationships between pipelines, achieving automatic matching of markers in the left and right views while completing inter-frame marker tracking, without the need for special matching. Therefore, the efficiency of tracking and matching is greatly improved, with the advantages of being fast, accurate, and robust.

Description

Technical Field

[0001] The present invention belongs to the technical field of binocular stereo vision, relates to a method for tracking and matching marker points in binocular sequence images, and further relates to a method for online real-time tracking and matching of circular non-coding marker points in binocular sequence images. Background Art

[0002] Binocular stereo vision, a key branch of computer vision, aims to mimic human binocular vision using two cameras, one on each side, to calculate three-dimensional information about the target scene. Due to its convenience, low cost, strong real-time performance, and high precision, it has been widely used in robotic vision, aerial mapping, reverse engineering, defense and military industry, medical imaging, and industrial inspection. Binocular stereo vision typically uses artificial landmarks or natural feature points (corner points) as measurement targets. Among these, measurement methods based on artificial landmarks are widely used due to their high precision and strong real-time performance. Obtaining the matching relationship between artificial landmarks (primarily non-coded landmarks) in the image sequences of the left and right cameras is a key step and the foundation for ensuring the real-time and precision of binocular stereo vision. The accuracy of tracking and matching determines the accuracy of detection, while the speed of tracking and matching determines the efficiency of detection.

[0003] Chinese invention patent ZL200710307748.6 proposes a Kalman filter-based method for inter-frame landmark tracking. However, this method requires manual initialization of a synchronized correspondence between two sets of video images under binocular vision, preventing automatic matching. During tracking, both a two-dimensional Kalman filter and a three-dimensional Kalman filter are used for prediction, and the three-dimensional Kalman filter is used to correct the two-dimensional Kalman filter predictions, making the entire process time-consuming. Chinese invention patent ZL 202010054037.8 matches the same landmark in the pre- and post-deformation images based on feature vectors arranged in a random neighborhood topology. When matching each landmark, 2 / 3 to 3 / 4 of the total number of landmarks must be obtained as adjacent landmarks to determine their relative position. Consequently, the matching process is time-consuming and inefficient.

[0004] Chinese invention patent ZL201910403384.4 constructs a third-perspective image based on calibration parameters, and uses the dual epipolar constraints of the left camera image and the constructed third-perspective image in the right image to complete the matching of feature landmarks. The dual limit constraint reduces the probability of mismatching to a certain extent, but it takes a lot of time to construct a third-perspective image in each frame image matching process. Chinese invention patent ZL201910224546.8 proposes a method of using epipolar correction to obtain coplanar and aligned binocular stereo vision images, and then obtains the matching relationship of the landmark points of the left and right camera sequence images through the Euclidean distance between the landmark points. However, when the landmark points are distributed too densely, the calculated epipolar lines will overlap in the image, making the above method ineffective. At the same time, in the actual matching process, due to the influence of inaccurate orientation parameters, the distance error between the image point and the corresponding epipolar line will also be large, reducing the stability of the method. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, this paper combines a pipeline filtering algorithm, a shape context algorithm, a homography, and epipolar constraints to propose an online real-time tracking and matching method for circular non-coding landmarks in binocular image sequences. First, during the pipeline initialization phase, a pipeline is constructed for each circular non-coding landmark in the left and right views (the images captured by the left and right cameras are referred to as the left and right views). Based on this, real-time tracking of circular non-coding landmarks in the image sequences captured by the left and right cameras is achieved across frames. Second, the shape context algorithm is combined with epipolar constraints to construct a correspondence between the pipelines in the left and right view pipeline queues, based on which fast matching of circular non-coding landmarks in the left and right views is achieved. Finally, for the few circular non-coding landmarks that fail to match using the pipeline filtering algorithm due to occlusion, a homography algorithm is used for fast and accurate matching, thereby ensuring the accuracy and robustness of stereo matching. The most time-consuming step in this method is the construction of the left and right view pipeline correspondences using the shape context algorithm and epipolar constraints. However, this process is performed during the pipeline initialization phase and is not involved in the subsequent real-time tracking and matching phases, thus ensuring real-time tracking and matching. In summary, compared with the existing technology, this method takes into account the real-time and accuracy of circular non-coding landmark matching, and has the advantages of being fast, precise, and robust.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for online real-time tracking and matching of circular non-coding landmarks in binocular sequence images comprises the following steps:

[0008] The first step is to initialize the left and right view pipelines. Control the left and right cameras to synchronously capture N frames of images (N is the pipeline length, typically 8 to 10) to complete pipeline initialization.

[0009] The second step is to build the correspondence between the left and right view pipelines. The shape context algorithm and the epipolar constraint are combined to build the correspondence between the left and right view pipelines.

[0010] The third step is to associate the pipeline with the landmark ID. The corresponding pipeline in the left and right views constructed in the second step is associated with the ID of the circular non-coding landmark.

[0011] Step 4: The left and right cameras synchronously capture a frame of image. Control the left and right cameras to synchronously capture a frame of circular non-coded marker image in real time.

[0012] Step 5: Real-time detection of landmark points. Extract the center coordinates of the landmark points in the image collected in step 4.

[0013] Step 6: Real-time tracking of landmarks between frames. Based on the constructed pipeline and its corresponding relationship, real-time tracking of the circular non-coding landmarks extracted in step 5 is achieved between frames.

[0014] Step 7: Real-time matching of left and right view landmarks: Based on the inter-frame landmark tracking results of step 6, the circular non-encoding landmarks in the left and right views are matched in real time.

[0015] Step 8: Pipeline update: Update the pipeline based on the landmark tracking and matching results.

[0016] In the ninth step, it is determined whether to stop the real-time acquisition of the sequence images; if so, the real-time tracking and matching of the circular non-coding markers is terminated; otherwise, steps 4 to 8 are repeated to acquire images, track and match in real time.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] (1) The method of the present invention uses a pipeline filtering algorithm to achieve rapid matching of circular non-coding landmarks in a single-camera sequence image. By establishing a pipeline queue for left and right views and its update method, rapid inter-frame matching of non-coding landmarks in the left and right views is achieved. Therefore, the method of the present invention is more efficient than a strategy that requires simultaneous prediction using a two-dimensional Kalman filter and a three-dimensional Kalman filter, and then using a three-dimensional Kalman filter to correct the two-dimensional Kalman filter prediction result.

[0019] (2) By constructing a pipeline and its corresponding relationship, the circular non-coding markers in the left and right views are automatically matched during tracking. After tracking is completed, the circular non-coding markers with the same ID are the corresponding points in the left and right views. Separate matching is no longer required, and only epipolar constraints are required for verification. Therefore, the real-time matching is guaranteed and is less time-consuming than existing methods that require constructing a third-person perspective image or performing image correction.

[0020] (3) For a small number of circular non-encoding landmarks that fail to match using the pipeline filtering algorithm due to occlusion or other reasons, the homography transformation algorithm is used to quickly and accurately match them, thereby ensuring the accuracy and robustness of stereo matching. This is more reliable than existing matching methods that rely solely on epipolar constraints.

[0021] (4) The most time-consuming step in the method of the present invention is the construction of the left and right view pipeline correspondence by combining the shape context algorithm with the polar line constraint. However, this process is placed in the pipeline initialization stage and is not involved in the subsequent real-time tracking and matching stage. Therefore, the real-time performance of tracking and matching is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of the specific operation steps of the present invention.

[0023] Figure 2 This is the flowchart for real-time detection of circular non-coding landmarks.

[0024] Figure 3 Schematic diagram of the center and cross-sectional area of ​​a pipeline constructed for a circular non-coded landmark point.

[0025] Figure 4 It is a frame of circular non-coded marker point image captured by the left and right cameras in real time, where a is the image captured by the left camera and b is the image captured by the right camera.

[0026] Figure 5 for Figure 4 Real-time tracking and matching results of the circular non-coding landmarks, where a is Figure 4 The real-time tracking and matching results of a and b are Figure 4 Real-time tracking and matching results of b.

[0027] Figure 6 Update the flow chart for the pipeline.

[0028] Figure 7 The real-time tracking and matching results of the circular non-encoded landmarks in the 40th frame image captured by the left and right cameras are shown, where a is the result of the left camera and b is the result of the right camera.

[0029] Figure 8 The real-time tracking and matching results of the circular non-encoded landmarks in the 60th frame image captured by the left and right cameras are shown, where a is the result of the left camera and b is the result of the right camera. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings.

[0031] The present invention proposes a method for online real-time tracking and matching of circular non-coding markers in binocular sequence images. The process is as follows: Figure 1Shown, including:

[0032] The first step is to initialize the left and right view pipelines. Control the left and right cameras to synchronously capture N frames of images (N is the pipeline length, generally 8 to 10) to complete pipeline initialization. The specific steps are as follows:

[0033] 1) Control the left and right cameras to synchronously capture the first image frame. The image captured by the left camera is called the left view, and the image captured by the right camera is called the right view. Furthermore, before capturing images, binocular calibration is required to obtain the intrinsic and extrinsic parameters of the left and right cameras, which will be used to solve the epipolar lines in the subsequent steps.

[0034] 2) Using a circular non-coding mark point real-time detection algorithm and other methods to extract the center coordinates of the circular non-coding mark points in the left and right views collected in step 1). The process of the circular non-coding mark point real-time detection algorithm is as follows: Figure 2 As shown, it is mainly implemented on the GPU side. First, the CPU side and the GPU side are initialized, that is, sufficient page-locked memory is allocated on the CPU side to store the image collected in step 1), and sufficient memory area is allocated in the global memory of the GPU side for data storage during the GPU side processing. The initialization of the CPU side and the GPU side is time-consuming, but it only needs to be initialized once, that is, after the first initialization is completed, this step will no longer be executed. Secondly, the left and right views collected in step 1) are uploaded from the CPU side to the GPU side, and asynchronous transmission is used during transmission, that is, after the left view is transmitted, the binarization processing of the left view is started, and the asynchronous upload of the right view is started at the same time. The binarization on the GPU side adopts the Sauvola local adaptive binarization algorithm. Left view Figure 2 After binarization, the GPU uses a two-step method (Two-Pass algorithm) to mark and analyze connected domains, and on this basis, extract the integer pixel edges of circular non-coded markers; next, the GPU uses the spatial moment method to extract sub-pixel edges, and uses the least squares ellipse fitting algorithm to obtain the center coordinates of the circular non-coded markers; finally, the left view detection results are asynchronously transmitted from the GPU to the CPU, and the binarization processing of the right view is started at the same time. After binarization, connected domain marking and analysis, integer pixel and sub-pixel edge extraction, and least squares ellipse fitting operations are performed to obtain the center coordinates of the circular non-coded markers in the right view, and the detection results are asynchronously transmitted to the CPU to complete the online real-time detection of circular non-coded markers in the left and right views.

[0035] 3) Build a pipeline for each circular non-coding landmark point in the left and right views, such as Figure 3As shown, the center of the circular non-coding marker is the center of the pipeline, and the cross-sectional area of ​​the pipeline is (2L+1)×(2L+1). The value of L is related to the camera frame rate and is generally set to L = (40-200)d / F, where d is the pixel size and F is the camera frame rate. Next, set the pipeline counter to 1 and the missing frame counter to 0. Finally, in the order in which the pipelines were constructed, add the pipelines corresponding to all circular non-coding markers in the left view to a queue and mark it as the left view pipeline queue. Add the pipelines corresponding to all circular non-coding markers in the right view to another queue and mark it as the right view pipeline queue.

[0036] 4) Control the left and right cameras to synchronously capture the a-th frame image, where a = 2, 3, 4, ..., N, and extract the center coordinates of all circular non-coding landmarks in the a-th frame image. The extraction method can use the circular non-coding landmark real-time detection algorithm described in step 2).

[0037] 5) Assign the circular non-coding landmarks in the left view extracted in step 4) to a pipeline. The assignment method is as follows: For each circular non-coding landmark in the left view extracted in step 4), determine whether it falls into a pipeline in the left view pipeline queue constructed in step 3). If so, increment the pipeline counter by 1, and update the pipeline center to the center of the circular non-coding landmark, indicating successful assignment. Otherwise, assignment fails, and a new pipeline is constructed for the circular non-coding landmark according to the method described in step 3) and added to the left view pipeline queue.

[0038] The method for judging whether a circular non-coding mark point falls into a certain pipeline is as follows: if the absolute value of the difference between the x-coordinate component of the center coordinate of the circular non-coding mark point and the center coordinate of a certain pipeline is less than L, and the absolute value of the difference between the y-coordinate component is also less than L, then it is determined that the circular non-coding mark point falls into the pipeline.

[0039] 6) Assign the circular non-coding landmarks in the right view extracted in step 4) to a pipeline. The assignment method is as follows: For each circular non-coding landmark in the right view extracted in step 4), determine whether it falls into a pipeline in the right view pipeline queue constructed in step 3). If so, increment the pipeline counter by 1, and update the pipeline center to the center of the circular non-coding landmark, indicating successful assignment. Otherwise, assignment fails, and a new pipeline is constructed for the circular non-coding landmark according to the method described in step 3) and added to the right view pipeline queue.

[0040] 7) Pipeline numbering. If the current frame is the Nth frame, i.e., i=N, the pipelines in the left and right view pipeline queues are numbered. First, the counter of each pipeline in the left view pipeline queue is compared with the threshold K. c Do a comparison, if the counter is greater than or equal to K c, then set the number of the pipeline to M l +1, M l The maximum value of the pipeline number in the left view pipeline queue, M l The initial value is 0; if the counter is less than K c , then set the pipeline number to -1. Secondly, compare the counter of each pipeline in the right view pipeline queue with the threshold K c Do a comparison, if the counter is greater than or equal to K c , then set the number of the pipeline to M r +1, M r The maximum value of the pipeline number in the pipeline queue of the right view, M r The initial value is also 0; if the counter is less than K c , then the number of the pipeline is set to -1. Threshold K c Generally, it is 5 to 7.

[0041] The second step is to establish the correspondence between the left and right view pipelines. This is done by combining the shape context algorithm with the epipolar constraint, and includes the following steps:

[0042] 1) Let the set of pipeline centers whose numbers are not -1 in the left view pipeline queue be P = {p1, p2, ..., p n}, n represents the number of pipelines. First, select the center of one pipeline p i As the reference point, 1≤i≤n, establish a i Next, create a circle with the pole as the center and R as the radius, R = log(max(p b -p i ), 1≤b≤n, max() represents the maximum value function, and log() represents the logarithmic function. Next, the circle is divided into k1 equal parts along the polar radius and k2 equal parts along the polar angle. This results in the area being divided into k1×k2 sub-areas. k1 is typically 5–7, and k2 is typically 10–12.

[0043] 2) Count the number of remaining pipeline centers in P in the sub-areas divided in step 1) to obtain the histogram h i (k), 1≤k≤k1×k2, called p i The shape context.

[0044] 3) Using the centers of the remaining channels in P as reference points, repeat steps 1) and 2) to calculate the shape context of the centers of the remaining channels in P.

[0045] 4) Let the set of pipeline centers whose numbers are not -1 in the right view pipeline queue be Q = {q1,q2...,q m}, m represents the number of channels. According to the process described in steps 1) to 3), the shape context of the channel center in Q is calculated.

[0046] 5) Similarity measure matrix calculation. The value C in the i-th row and j-th column of the similarity measure matrix C is ij The calculation method is as follows:

[0047]

[0048] Among them, h i (k) and h j (k) represents the pipeline center p in the calculated left view pipeline queue i And the center of the pipeline q in the right view pipeline queue j The shape context of , 1≤i≤n, 1≤j≤m.

[0049] 6) Based on the similarity measure matrix C calculated in step 5), the bipartite graph matching algorithm is used to solve the optimal matching π. The optimization objective function for solving the optimal matching π is:

[0050]

[0051] Among them, π represents the corresponding relationship between the pipeline center in the left view pipeline queue and the pipeline center in the right view pipeline queue, and π(i) represents the i-th pipeline center p in the left view pipeline queue. i Corresponding to the π(i)th pipeline center q in the right view pipeline queue π(i) , 1≤π(i)≤m. The π corresponding to the minimum of the objective function H(π) is the optimal match sought.

[0052] 7) For the pair of pipe centers p that were successfully matched in step 6 i With q π(i) If their pipeline numbers are not -1, the basic matrix is ​​calculated based on the internal and external parameters of the left and right cameras, and p is solved based on the basic matrix. i Epipolar line f in the right view ri , and q π(i) to f ri The distance d ri ; Secondly, calculate q based on the internal and external parameters of the left and right cameras π(i) The epipolar line f in the left view li , and p i to f li The distance d li ; Let d i =d li +d ri , given a distance threshold K d , if d i ≤K d, then mark the i-th pipeline in the left view pipeline queue and the π(i)-th pipeline in the right view pipeline queue as corresponding to each other. Distance threshold K d Generally, 3 to 6 pixels are used.

[0053] The third step is to associate pipelines with landmark IDs. The corresponding pipelines in the left and right views constructed in step 2 are associated with the IDs of the circular non-encoded landmarks. Assuming that the i-th pipeline in the left view pipeline queue and the j-th pipeline in the right view pipeline queue correspond to each other, they are associated with a landmark ID of 100 + i. Otherwise, if a pipeline does not correspond to any other pipeline, it is associated with a landmark ID of -1.

[0054] Step 4: The left and right cameras synchronously capture a frame of image. Control the left and right cameras to synchronously capture a frame of circular non-coded marker image in real time. Figure 4 Figures a and b are frames of circular non-coded marker point images of a wind turbine blade during a fatigue excitation test, captured synchronously in real time by the left and right cameras.

[0055] Step 5: Real-time marker detection. Extract the center coordinates of the circular non-coding markers in the left and right views collected in step 4, and set the initial ID value of the circular non-coding marker to -1. This extraction method can use the aforementioned real-time circular non-coding marker detection algorithm.

[0056] Step 6: Real-time tracking of markers between frames. Based on the constructed pipelines and their corresponding relationships, the real-time tracking of the circular non-coding markers in the left and right views detected in the fifth step is realized. For any circular non-coding marker in the left view detected in the fifth step, determine whether it falls into a pipeline in the constructed left view pipeline queue; if it does, it is assigned to the pipeline, and its ID is set to the marker ID associated with the pipeline, and its status is marked as tracking success; otherwise, its status is marked as tracking failure. Similarly, for any circular non-coding marker in the right view detected in the fifth step, determine whether it falls into a pipeline in the constructed right view pipeline queue; if it does, it is assigned to the pipeline, and its ID is set to the marker ID associated with the pipeline, and its status is marked as tracking success; otherwise, its status is marked as tracking failure.

[0057] Step 7: Real-time matching of left and right view landmarks. Real-time matching of the circular non-encoding landmarks in the left and right views that were successfully tracked between frames in step 6 is performed. The specific steps are as follows:

[0058] 1) For the circular non-coding marker pairs with the same ID in the left and right views after the sixth step of inter-frame tracking and both of them are not -1, perform epipolar verification; if the verification is successful, the pair of circular non-coding markers is successfully matched, otherwise the IDs of the pair of circular non-coding markers are both set to -1. The epipolar verification rule is: for a pair of circular non-coding markers with the same ID in the left and right views, i with b j , first calculate a based on the basic matrix calculated in the second step i Epipolar line f in the right view ri , and b j to f ri The distance d ri ; Secondly, calculate b j The epipolar line f in the left view lj , and a i to f lj The distance d lj ; Let d i =d lj +d ri , if d i <K d , verification is successful. Among them, a i Indicates the i-th circular non-coding landmark point in the left view, 1≤i≤n1, n1 represents the number of circular non-coding landmark points in the left view, b j Indicates the jth circular non-coding landmark point in the right view, 1≤j≤m1, m1 represents the number of circular non-coding landmark points in the right view, and the threshold K d The value of is as described in the first step.

[0059] 2) For any circular non-coding mark point a that is not successfully matched in the left view after step 1) verification i , 1≤i≤n1, if the number of the pipeline to which it belongs is not -1, then search for its four nearest neighboring points that are successfully matched in the left view and the corresponding points of these nearest neighboring points in the right view, and calculate the homography matrix H based on the searched four pairs of corresponding points, and then convert a i Transform to the right view. Assume a i The transformed coordinates are a i ′, search for its nearest neighbor b among the unmatched landmarks in the right view j If b j If the number of the pipeline to which it belongs is not -1, then a is calculated based on the basic matrix calculated in the second step. i Epipolar line f in the right view ri , and b j to f ri The distance d ri If d i <K d / 2, then the circular non-coding mark point a in the left view is considered i Compared with the circular non-coding landmark b in the right view j For corresponding points, the match is successful. i If the ID is not -1, b j The ID is changed to a i ID; if a i The ID of b is -1, and b j If the ID is not -1, then a i The ID is changed to b j ID; if a i with b j If the IDs of all are -1, then their IDs are set to ID m +1; among them, ID m Indicates the maximum ID associated with the pipelines in the left and right pipeline queues, threshold K d The value of is as described in the first step.

[0060] Figure 5 A in the middle shows Figure 4 Real-time tracking and matching results of the circular non-coding landmark in a. Figure 5 b in the figure is Figure 4 Figure b shows the real-time tracking and matching results for circular non-coding markers. The figure shows that both stationary circular non-coding markers in the background, such as those with IDs 118 to 120, and those moving along the wind turbine blades, such as those with IDs 102, 104, and 109, can be accurately tracked and matched.

[0061] Step 8: Pipeline update. According to the tracking and matching results of the landmark points, such as Figure 6 As shown, the pipeline is updated in three cases. In the first case, if a circular non-coding marker is tracked successfully, the counter of the pipeline to which it belongs is increased by 1, the missing frame counter is set to 0, and the center coordinates of the pipeline are set to the center coordinates of the circular non-coding marker; at the same time, if the ID of the circular non-coding marker is not -1, the ID associated with the pipeline to which it belongs is set to the circular non-coding marker ID. In addition, if the pipeline number to which it belongs is -1, and the pipeline counter is greater than the threshold K c , then set the number of the pipeline to M l +1 (when the pipeline is in the left view pipeline queue) or M r +1 (when the pipeline is in the right view pipeline queue). Among them, the threshold K c 、M l and M r The meaning and value of are as described in the first step.

[0062] In the second case, if the tracking of a circular non-coding marker fails, a new pipeline is constructed for the circular non-coding marker according to the pipeline construction method described in the first step; secondly, the new pipeline is added to the pipeline queue (when the circular non-coding marker belongs to the left view, the new pipeline is added to the left view pipeline queue, otherwise it is added to the right view pipeline queue), and a marker ID with a value of -1 is associated.

[0063] In the third case, for the pipe where no circular non-coding mark falls, its counter is set to 0, the missing frame counter is increased by 1, and it is determined whether the missing frame counter is greater than the threshold K e If it is greater than, delete the pipeline. Threshold K e, Generally, 3 to 5 are taken.

[0064] In the ninth step, it is determined whether the left and right cameras have stopped real-time acquisition of sequential images; if so, the real-time tracking and matching of the circular non-coding markers is terminated; otherwise, steps 4 to 8 are repeated to acquire images, track, and match in real time. Figure 7 Figures a and b are the real-time tracking and matching results of the circular non-coding markers in the 40th frame of the image captured synchronously by the left and right cameras. Figure 8 Figures a and b show the real-time tracking and matching results of circular non-coding landmarks in the 60th frame of the image, captured synchronously by the left and right cameras. As can be seen from the figure, despite the large range of motion of the circular non-coding landmarks with the blade, all of them are correctly tracked and matched, demonstrating the accuracy and robustness of the proposed method. The above processing was performed on a laptop equipped with an Intel Core i9-11900H CPU and an RTX 3060 graphics card. The image resolution was 5 megapixels (2448 x 2048), containing 20 circular non-coding landmarks, and the online processing rate was 70 Hz. Excluding the time consumed by the real-time landmark detection in the fifth step, the total time consumed by the inter-frame landmark tracking in the sixth step, the real-time matching of the left and right view landmarks in the seventh step, and the pipeline update in the eighth step is less than 2 milliseconds. When the number of landmarks in the image increases to more than 100, the online real-time tracking and matching time of steps 6 to 8 is less than 3 milliseconds, demonstrating the strong real-time performance of the proposed method and its ability to effectively meet the practical application requirements of online real-time tracking and matching.

[0065] In summary, the present invention combines the pipeline filtering algorithm, the shape context algorithm, the homography transformation and the epipolar constraint to construct the left and right view pipeline queues and the mutual correspondence between the pipelines, and realizes the automatic matching of the marker points in the left and right views while completing the inter-frame marker tracking. There is no need for special matching, so the tracking and matching efficiency is greatly improved. In addition, for a small number of circular non-coding marker points that fail to match using the pipeline filtering algorithm due to interference factors such as occlusion, secondary matching is performed using the homography transformation, and the matching accuracy and robustness are stronger. In short, compared with the prior art, the method of the present invention has the advantages of being fast, accurate, and robust, and can realize online real-time tracking and matching of circular non-coding marker points in binocular sequence images.

Claims

1. A method for online real-time tracking and matching of circular non-coding landmarks in binocular sequence images, characterized in that: The steps include: The first step is to initialize the left and right view pipelines Control the left and right cameras to synchronously capture N frames of images to complete pipeline initialization, where N is the pipeline length; The second step is to build the corresponding relationship between the left and right view pipelines Combine the shape context algorithm with the epipolar constraint to establish the left and right view pipeline correspondence. The method is as follows: 1) Let the set of pipeline centers whose numbers are not -1 in the left view pipeline queue be P = {p1, p2, ..., p n }, n represents the number of pipelines; First, select one of the pipeline centers p i As the reference point, 1≤i≤n, establish the i As the polar coordinate system of the pole; secondly, create a circle with the pole as the center and R as the radius; finally, divide the circle into k1 parts in the polar direction according to the logarithmic distance, and divide it into k2 parts along the polar angle direction, so that the area is divided into k1×k2 sub-areas; the number is the counter of each pipeline in the left and right view pipeline queues and the threshold K c Do a comparison, if the counter is less than K c , then the number of the pipeline is set to -1; 2) Count the number of remaining pipeline centers in P in the sub-areas divided in step 1) to obtain the histogram h i (k), 1≤k≤k1×k2, called p i The shape context; 3) Using the centers of the remaining channels in P as reference points, repeat steps 1) and 2) to calculate the shape context of the centers of the remaining channels in P; 4) Let the set of pipeline centers whose numbers are not -1 in the right view pipeline queue be Q = {q1,q2...,q m }, m represents the number of channels. According to the process described in steps 1) to 3), the shape context of the channel center in Q is calculated; 5) Calculate the similarity measure matrix; 6) Based on the similarity measure matrix C calculated in step 5), a bipartite graph matching algorithm is used to solve the optimal matching π; 7) For the pair of pipe centers p that were successfully matched in step 6 i With q π(i) If their pipeline numbers are not -1, the basic matrix is ​​calculated according to the internal and external parameters of the left and right cameras, and p is solved according to the basic matrix. i Epipolar line f in the right view ri , and q π(i) to f ri The distance d ri ; Secondly, calculate q based on the internal and external parameters of the left and right cameras π(i) The epipolar line f in the left view li , and p i to f li The distance d li ; Let d i =d li +d ri , given a distance threshold K d , if d i ≤K d , then mark the i-th pipeline in the left view pipeline queue and the π(i)-th pipeline in the right view pipeline queue as corresponding to each other; The third step is to associate the pipeline with the landmark ID Associate the corresponding pipelines in the left and right views constructed in the second step with the IDs of the circular non-coding landmarks; Step 4: The left and right cameras synchronously capture a frame of image Control the left and right cameras to synchronously capture a frame of circular non-coded marker image in real time; Step 5: Real-time detection of landmark points Extract the center coordinates of the landmark points in the image collected in the fourth step; Step 6: Real-time tracking of landmarks between frames Based on the constructed pipeline and its corresponding relationship, the real-time tracking of the circular non-coding landmarks extracted in the fifth step is realized; Step 7: Real-time matching of left and right view landmarks According to the inter-frame landmark tracking results of the sixth step, the circular non-encoding landmarks in the left and right views are matched in real time; Step 8: Pipeline Update Update the pipeline based on the landmark tracking and matching results; Step 9: Determine whether to stop the real-time acquisition of the sequence image; if so, end the real-time tracking and matching of the circular non-encoding marker; Otherwise, repeat steps 4 to 8 to collect images, track and match in real time.

2. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 1, characterized in that: The first step of left and right view pipeline initialization includes the following steps: 1) Control the left and right cameras to synchronously capture the first frame of image. Before capturing the image, perform binocular calibration on the left and right cameras to obtain the intrinsic and extrinsic parameters of the left and right cameras for subsequent epipolar line calculation. 2) extracting the center coordinates of the circular non-encoding landmarks in the left and right views collected in step 1); 3) Construct a pipeline for each circular non-coding marker in the left and right views. The center of the pipeline is the center of the circular non-coding marker, and the cross-sectional area of ​​the pipeline is (2L+1)×(2L+1). L is a calculation parameter related to the camera frame rate, and L=(40-200)d / F, where d is the pixel size and F is the camera frame rate. Set the pipeline counter to 1 and the missing frame counter to 0. In the order of pipeline construction, add the pipelines corresponding to all circular non-coding markers in the left view to a queue and mark it as the left view pipeline queue. Add the pipelines corresponding to all circular non-coding markers in the right view to another queue and mark it as the right view pipeline queue. 4) Control the left and right cameras to synchronously capture the a-th frame image, where a=2, 3, 4, ..., N, and extract the center coordinates of all circular non-encoding markers in the captured a-th frame image; 5) Pipeline assignment is performed on the circular non-coding marker points in the left view detected in step 4), and the assignment method is as follows: for each circular non-coding marker point in the left view extracted in step 4), determine whether it falls into a pipeline in the left view pipeline queue constructed in step 3); if it does, the counter of the pipeline is incremented by 1, and the center of the pipeline is updated to the center of the circular non-coding marker point, and the assignment is successful; otherwise, the assignment fails, and a new pipeline is constructed for the circular non-coding marker point according to the method described in step 3) and added to the left view pipeline queue; The method for determining whether a circular non-coding mark point falls into a certain pipeline is as follows: if the absolute value of the difference between the center coordinates of the circular non-coding mark point and the center coordinates of a certain pipeline is less than L, and the absolute value of the difference between the y-coordinate components is also less than L, then it is determined that the circular non-coding mark point falls into the pipeline; 6) Pipeline assignment is performed on the circular non-coding marker points in the right view detected in step 4), and the assignment method is as follows: for each circular non-coding marker point in the right view extracted in step 4), determine whether it falls into a pipeline in the right view pipeline queue constructed in step 3); if it does, the counter of the pipeline is incremented by 1, and the center of the pipeline is updated to the center of the circular non-coding marker point, and the assignment is successful; otherwise, the assignment fails, and a new pipeline is constructed for the circular non-coding marker point according to the method described in step 3) and added to the right view pipeline queue; 7) Pipeline number If the current frame is the Nth frame, that is, a=N, the pipelines in the left and right view pipeline queues are numbered. The process is as follows: First, the counter of each pipeline in the left view pipeline queue is compared with the threshold K. c Do a comparison, if the counter is greater than or equal to K c , then set the number of the pipeline to M l +1, M l The maximum value of the pipeline number in the left view pipeline queue, M l The initial value is 0; secondly, the counter of each pipeline in the right view pipeline queue is compared with the threshold K c Do a comparison, if the counter is greater than or equal to K c , then set the number of the pipeline to M r +1, M r The maximum value of the pipeline number in the pipeline queue of the right view, M r The initial value is also 0.

3. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 2, characterized in that: The 2), 4) and the fifth step all utilize a circular non-coding marker point real-time detection algorithm to extract the center coordinates. The circular non-coding marker point real-time detection algorithm process includes: first, initialization is performed on the CPU side and the GPU side, that is, sufficient page-locked memory is allocated on the CPU side for storing the collected images, and sufficient memory area is allocated in the global memory of the GPU side for storing data during the GPU side processing; secondly, the collected left and right views are uploaded from the CPU side to the GPU side, and asynchronous transmission is used during transmission, that is, after the left view transmission is completed, the binarization processing of the left view is started, and the asynchronous upload of the right view is started at the same time. After the left view is binarized, two images are uploaded to the GPU side. The connected domain is marked and analyzed based on the step method, and on this basis, the integer pixel edges of the circular non-coding markers are extracted; next, the sub-pixel edges are extracted based on the spatial moment method on the GPU side, and the center coordinates of the circular non-coding markers are obtained by the least squares ellipse fitting algorithm; finally, the left view detection result is asynchronously transmitted from the GPU side to the CPU side, and the binarization processing of the right view is started at the same time. After binarization, the connected domain is marked and analyzed, the integer pixel and sub-pixel edges are extracted, and the least squares ellipse fitting operations are performed to obtain the center coordinates of the circular non-coding markers in the right view, and the detection results are asynchronously transmitted to the CPU side to complete the online real-time detection of the circular non-coding markers in the left and right views.

4. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 2, characterized in that: The second step: The radius of the circle R = log(max(p b -p i )), 1≤b≤n, max() represents the maximum value function, log() represents the logarithmic function; the number of copies k1 is 5 to 7, and k2 is 10 to 12; The value C in the i-th row and j-th column of the similarity measurement matrix C ij The calculation method is as follows: Among them, h i (k) and h j (k) represents the pipeline center p in the calculated left view pipeline queue i And the center of the pipeline q in the right view pipeline queue j The shape context of , 1≤i≤n, 1≤j≤m; The optimization objective function when solving the optimal matching π is: Among them, π represents the corresponding relationship between the pipeline center in the left view pipeline queue and the pipeline center in the right view pipeline queue, and π(i) represents the i-th pipeline center p in the left view pipeline queue. i Corresponding to the π(i)th pipeline center q in the right view pipeline queue π(i) , 1≤π(i)≤m; the π corresponding to the minimum of the objective function H(π) is the optimal match sought; The distance threshold K d Take 3 to 6 pixels.

5. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 4, characterized in that: The method for associating pipelines with marker point IDs in the third step is as follows: if the i-th pipeline in the left view pipeline queue corresponds to the j-th pipeline in the right view pipeline queue, then a marker point ID with a value of 100+i is associated with them; otherwise, if a pipeline does not correspond to other pipelines, then a marker point ID with a value of -1 is associated with it.

6. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 4, characterized in that: The sixth step of the real-time tracking method of inter-frame markers is as follows: for any circular non-coding marker in the left view detected in the fifth step, determine whether it falls into a certain pipeline in the constructed left view pipeline queue; if it does, it is assigned to the pipeline, and its ID is set to the marker ID associated with the pipeline, and its status is marked as tracking success; otherwise, its status is marked as tracking failure; for any circular non-coding marker in the right view detected in the fifth step, determine whether it falls into a certain pipeline in the constructed right view pipeline queue; if it does, it is assigned to the pipeline, and its ID is set to the marker ID associated with the pipeline, and its status is marked as tracking success; otherwise, its status is marked as tracking failure.

7. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 4, characterized in that: The seventh step of the real-time matching process of the left and right view landmarks includes the following steps: 1) For the circular non-coding marker pairs with the same ID in the left and right views after the sixth step of inter-frame tracking and both of them are not -1, perform epipolar line verification; if the verification is successful, the circular non-coding marker pair is successfully matched, otherwise the ID of the circular non-coding marker pair is set to -1; the epipolar line verification rule is: for a pair of circular non-coding marker points with the same ID in the left and right views, i with b j , first calculate a based on the basic matrix calculated in the second step i Epipolar line f in the right view ri , and b j to f ri The distance d ri ; Secondly, calculate b j The epipolar line f in the left view lj , and a i to f lj The distance d lj ; Let d i =d lj +d ri , if d i <K d , verification is successful; among them, a i Indicates the i-th circular non-coding landmark point in the left view, 1≤i≤n1, n1 represents the number of circular non-coding landmark points in the left view, b j Indicates the jth circular non-coding landmark point in the right view, 1≤j≤m1, m1 represents the number of circular non-coding landmark points in the right view, and the threshold K d The value of is as described in the first step; 2) For any circular non-coding mark point a that is not successfully matched in the left view after step 1) verification i , 1≤i≤n1, if the number of the pipeline to which it belongs is not -1, then search for its four nearest neighboring points that are successfully matched in the left view and the corresponding points of these nearest neighboring points in the right view, and calculate the homography matrix H based on the searched four pairs of corresponding points, and then convert a i Transform to the right view; let a i The transformed coordinates are a i ′, search for its nearest neighbor b among the unmatched landmarks in the right view j , if b j If the number of the pipeline to which it belongs is not -1, then a is calculated based on the basic matrix calculated in the second step. i Epipolar line f in the right view ri , and b j to f ri The distance d ri If d i <K d / 2, then the circular non-coding mark point a in the left view is considered i Compared with the circular non-coding landmark b in the right view j For corresponding points, the match is successful; if a i If the ID is not -1, b j The ID is changed to a i ID; if a i The ID of b is -1, and b j If the ID is not -1, then a i The ID is changed to b j ID; if a i with b j If the IDs of all are -1, then their IDs are set to ID m +1; among them, ID m Indicates the maximum ID associated with the pipes in the left and right pipe queues.

8. The method for online real-time tracking and matching of circular non-coding markers in binocular sequence images according to claim 4, characterized in that: The pipeline update in step 8 is divided into three cases: In the first case, if a circular non-coding marker is successfully tracked, the counter of the pipeline to which it belongs is incremented by 1, the missing frame counter is set to 0, and the center coordinates of the pipeline are set to the center coordinates of the circular non-coding marker. At the same time, if the ID of the circular non-coding marker is not -1, the ID associated with the pipeline to which it belongs is set to the ID of the circular non-coding marker. Moreover, if the pipeline number to which it belongs is -1 and the pipeline counter is greater than the threshold K c , then when the pipeline is in the left view pipeline queue, set the pipeline number to M l +1, when the pipeline is in the right view pipeline queue, set the pipeline number to M r +1; In the second case, if a circular non-coding landmark fails to be tracked, a new pipeline is constructed for the circular non-coding landmark according to the pipeline construction method described in the first step. Secondly, the new pipeline is added to the pipeline queue and associated with a landmark ID with a value of -1. If the circular non-coding landmark belongs to the left view, the new pipeline is added to the left view pipeline queue; otherwise, it is added to the right view pipeline queue. In the third case, for the pipe where no circular non-coding mark falls, its counter is set to 0, the missing frame counter is increased by 1, and it is determined whether the missing frame counter is greater than the threshold K e If it is greater than, delete the pipeline, threshold K e , take 3 to 5.

Images