Augmented reality target tracking registration method based on twin network combined with DSST scale estimation

Abstract

The present invention discloses a twin network augmented reality target tracking and registration method combined with DSST scale estimation, wherein the DSST filter is introduced into the twin network tracking process, and the HOG feature is used to compensate for the deep feature in the twin network, so as to suppress the drift to the similar target; the candidate target transferred to the DSST is screened more accurately by SiamFC, so as to alleviate the boundary effect of DSST; after the target position and scale are transmitted back to the SiamFC network, the relevant filter coefficients are updated by linear interpolation to relocate the target, so as to obtain a more accurate target area to be registered; the features of the target to be registered are detected and matched by the ORB algorithm, and the mismatched pairs are eliminated by RANSAC after matching by the Hamming distance, and the registration matrix is ​​obtained according to the feature relationship between the matching adjacent frames, and the registration of the virtual information is completed after rendering with the cube virtual model generated by OpenGL, so as to ensure the real-time performance while improving the accuracy, robustness and robustness of the results of the traditional augmented reality tracking and registration algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and relates to a twin network augmented reality target tracking registration method combined with DSST scale estimation. Background Art

[0002] Augmented Reality (AR) is based on virtual reality technology. It uses computer graphics and computer vision technology to superimpose virtual information on real scenes, realize virtual-real fusion and human-computer interaction, and enhance users' perception of the real world. It has been applied to many fields such as medicine, education, and navigation. With the application of mobile augmented reality technology by more and more users, people pay more attention to the user experience of AR technology. In order to perfectly superimpose virtual information in reality to realize augmented reality, it is necessary to use target tracking and registration technology in three-dimensional space to align virtual information with real scenes. The speed and accuracy of the tracking algorithm and the robustness of the registration determine the performance of the augmented reality method. The main difficulty of augmented reality is how to improve the accuracy of tracking and registration while ensuring real-time performance. Therefore, it is of great significance to develop a method that can achieve accurate tracking and registration of augmented reality targets while ensuring real-time performance in complex motion scenes.

[0003] At present, among the mainstream three-dimensional tracking and registration technologies, the markerless tracking and registration method based on natural feature points has a wider range of application scenarios. The mainstream tracking algorithms are mainly divided into correlation filter algorithms and deep learning algorithms. Among the correlation filter tracking algorithms, the kernel correlation filter tracking (KCF) algorithm uses multi-channel features in the CF-based tracker, and has good tracking speed and accuracy performance; DSST (Discriminative Scale Space Tracking) introduces a scale filter to respond to the change of target scale, which has good portability, but due to the limited scale pool, it cannot achieve accurate tracking when the target moves quickly. In the deep learning algorithm, HCF (Hierarchical Convolutional Features for Visual Tracking) integrates deep features into the filter to improve the tracking performance, but the algorithm also has the problem of scale change, so it is not robust when large-scale changes occur in target tracking; the target tracking algorithm based on the twin network (SiamFC) uses a fully convolutional network structure for similarity prediction. It is a tracker based on CNNs for end-to-end tracking and achieves real-time tracking speed. At the same time, tracking algorithms CFnet and DCFnet have emerged that combine correlation filters with twin networks. Both algorithms interpret correlation filters as differentiable layers in deep neural networks and train the network to find the features that best suit the correlation filters, achieving better tracking performance. However, due to boundary effects, the performance improvement is very limited.

[0004] The above-mentioned correlation filtering and twin network-based tracking methods have good target tracking performance, and each has its own advantages in terms of accuracy and efficiency. However, in complex situations, the tracking accuracy is not ideal when the target is occluded or out of view, resulting in poor stability and robustness. Specifically, the twin network-based tracker has high tracking accuracy, but due to the semantic feature representation and lack of model update, it tends to drift to similar target areas; the tracking algorithm based on the correlation filtering framework is fast, but due to the use of a single feature (HOG, CN, etc.) and no target occlusion processing, it is not robust to scenes with strong background edges and target deformation; therefore, the overall tracking effect still needs to be improved. Summary of the invention

[0005] The purpose of the present invention is to provide a twin network augmented reality target tracking and registration method combined with DSST scale estimation, in order to address the technical problem that the existing augmented reality tracking and registration methods are prone to target tracking failure in complex motion scenes due to similar interference caused by motion and the external environment, target occlusion, local deformation of the target, etc., introduce the DSST filter into the twin network tracking process, and use HOG features to make up for the lack of deep features in the twin network, so as to improve the tracking accuracy and tracking effect while maintaining real-time performance.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The twin network augmented reality target tracking registration method combined with DSST scale estimation provided by the present invention comprises the following steps:

[0008] Step 1, SiamFC initial target position estimation: Use SiamFC to regard tracking as similarity learning, train a deep twin network offline, and use the deep twin network to compare the similarity between the search area and the target template during online tracking. Find the sample that is most similar to the target template marked in the first frame from many candidate frames, and convolve the sample and the target template to generate a response map. The position with the highest response value in the response map is the position of the initial target.

[0009] Step 2: DSST target scale estimation: pass the initial target position to the DSST tracker, collect multi-scale image composition samples at the initial target position, independently train the scale filter, and estimate the target scale based on the response value in the scale filter response map of the sample;

[0010] Step 3: Target relocation: The estimated target scale is transmitted back to the SiamFC tracker, and the initial target position and target scale are used to indicate the tracking of the next frame; the learning rate is adaptively adjusted using the difference between the two frames, and the relevant filter coefficients are updated using linear interpolation to locate the target, and the target area to be registered is obtained;

[0011] Step 4: ORB feature matching and 3D registration: Use the ORB algorithm for feature detection and matching, use RANSAC to eliminate mismatches through Hamming distance, and obtain the registration matrix based on the feature relationship between adjacent frames. Superimpose the virtual color cube model drawn by OpenGL on the real scene through the registration matrix to complete the tracking registration.

[0012] Furthermore, in step 1, SiamFC uses a large search area to improve tracking accuracy, but it is also more likely to introduce similar interference. We set a threshold based on the distribution of local maxima on the response graph to screen out potential target positions, thereby ensuring that the target position passed to the next module is more accurate.

[0013] Further, the threshold is an inverse Gaussian distribution with a peak value at the center of the response graph.

[0014] Furthermore, the target relocation is to fuse the response map score based on twin network tracking in step one with the response map score based on DSST tracking in step two by linear summation, update the position filter, complete accurate target tracking, and obtain the target area to be registered.

[0015] Furthermore, in step 4, the tracking registration is to directly match the 2D feature points tracked by the camera with the 3D space coordinate points updated in real time by the virtual color cube model, and calculate the changed camera posture so that the virtual color cube is accurately superimposed on the target object.

[0016] Compared with the existing target tracking registration method, the present invention has the following beneficial effects:

[0017] The present invention introduces the DSST filter into the twin network tracking process, uses the HOG feature to compensate for the deep features in the twin network, and suppresses the drift to similar targets; the candidate targets transferred to the DSST are screened more accurately by SiamFC, which can alleviate the boundary effect of DSST; after the target position and scale are passed back to the SiamFC network, the relevant filter coefficients are updated by linear interpolation to relocate the target, and a more accurate target area to be registered is obtained; the features of the target to be registered are detected and matched by the ORB algorithm, and after matching by the Hamming distance, the mismatched pairs are eliminated by RANSAC (random sampling consensus algorithm), and the registration matrix is ​​obtained according to the feature relationship between matching adjacent frames, and the registration of virtual information is completed after rendering with the cube virtual model generated by OpenGL, which ensures real-time performance while improving the accuracy, robustness and robustness of the results of the traditional augmented reality tracking registration algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a twin network augmented reality target tracking and registration method combined with DSST scale estimation according to an embodiment of the present invention;

[0019] Figure 2 This is a tracking effect diagram of an embodiment of the present invention under complex conditions, in which from top to bottom are a.Coupon, b.Deer, c.Liquor, d.Bird1, and e.Box;

[0020] Figure 3 Graph showing the accuracy and success rate of the embodiment of the present invention under the OTB2015 benchmark;

[0021] Figure 4 A comparison chart of the success rates of the embodiments of the present invention under four different attributes;

[0022] Figure 5 1 is a diagram of the tracking and registration results of an embodiment of the present invention under four different interferences: from top to bottom in the figure are (1) Box sequence diagram and (2) Liquor sequence diagram; a is a frontal registration diagram, b is a 180° rotated registration diagram, c is a certain viewing angle registration diagram, and d is a partially occluded registration diagram. DETAILED DESCRIPTION

[0023] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In the current mainstream three-dimensional tracking and registration technologies, the markerless tracking and registration method based on natural feature points is relatively widely used. With the continuous changes in the registration scene, the tracking accuracy is not ideal when the target is occluded or out of the field of view in complex situations, resulting in poor stability and robustness of the tracking method. The embodiment of the present invention overcomes the influence of similar interferences on the accurate tracking of the SiamFC twin network through the integration of DSST filters, and provides a twin network augmented reality target tracking and registration method combined with DSST scale estimation. The flow chart of the tracking and registration method is shown in the figure. Figure 1 As shown; specifically including the following steps:

[0025] Step 101: Input a target video sequence, determine the target size and position of the initial frame as the target template image, and crop a search image from the next tracking frame.

[0026] Step 102: Use the two CNN branches that share the same parameters contained in the SiamFC network structure to extract the convolution features of the target template image and the search image, and then use the existing similarity function of the SiamFC network: Calculate the similarity scores of the two branch features. The scores at each position here constitute a similarity score graph, where: z is the target template image; x is the search image; is the feature extraction corresponding to the embedded deep network; * represents the convolution operation, through which the part of x that is closest to z is extracted; b represents the value of each position in the similarity score map; f(z,x) represents the similarity score of x and z, and the similarity score of each position is classified into high and low scores, and the highest score is the target position. When tracking, the search image centered on the target template position of the previous frame is used to obtain the response map.

[0027] Step 103: After obtaining the response map of the target area position, find the local maximum of the similarity score on all response maps as the initial potential target position according to the distribution of the local maximum on the response map, and design a threshold for each position on the response map. The threshold is the inverse Gaussian distribution that reaches the peak at the center of the response map, and the positions with response values ​​greater than the given threshold are screened out. The specific process is as follows:

[0028] The response score map of the tth frame is represented by an N*M matrix, where [u,v] is the center position of the response score map, and the threshold at the [p,q] position is: The number of candidates is limited to N. When the number of selected positions is less than the threshold, the selected position is returned as the final candidate, otherwise the threshold position with the maximum response value is screened out as a candidate, thereby screening out the initial target position.

[0029] Step 104: Taking advantage of the fact that the DSST filter tracker can update the model online, the relatively accurate initial target position obtained by SiamFC in step 103 is passed to the DSST tracker for scale estimation. In the scale estimation stage, multi-scale images are collected at the target position to form a sample-independent training scale filter, and the target scale is estimated according to the scale filter response value of the sample. The HOG feature description in DSST is referenced to extract the multi-dimensional features of the image block where the target is located to construct the optimal filter. Finally, the image feature extracted in the new frame is z, and the two-dimensional discrete Fourier transform corresponding to each dimensional feature is taken to obtain Z l , using the formula Calculate the response score f hog (I), where the new position of the target corresponds to the maximum value.

[0030] Step 105: Update the SiamFC tracker related filter coefficients using linear interpolation, and convert the calculated twin neural network tracking response map score f t (z,x) and the response map score f based on DSST tracking hog (I) Using the formula f(I) = γf t (z,x)+(1-γ)f hog (I) performing fusion by linear summation, where γ is a parameter of the fusion perturbation perception model, relocating the target obtained in step 104, finding the position with the maximum score in the final response score map to obtain the accurate positioning of the final target, and obtaining the target area to be registered, that is, the accurate position of the tracked target;

[0031] Step 106: For the relatively accurate target area to be registered obtained by the above tracking method, an ORB algorithm which is more suitable for an augmented reality system is used to perform feature detection and extract corner points;

[0032] Step 107: The matching degree of the feature points detected in step 106 is measured according to the Hamming distance between the feature vectors. The smaller the value, the higher the similarity. Under the premise of having enough feature points, the RANSAC (random sampling consensus) algorithm is used to eliminate mismatched pairs to obtain high-quality matching feature points.

[0033] Step 108: Calculate the parameters of the three-dimensional registration matrix based on the accurately matched feature relationship between adjacent frames. For the calculation of the three-dimensional registration matrix, first determine the three-dimensional coordinates and mutual conversion relationship between the camera, virtual information, and real scene, then determine the position information of the camera and the relative position of the camera and the real scene, and finally determine the mapping position of the virtual information in the real scene to obtain the transformation matrix between the camera coordinate system and the real scene coordinate system. After completing the coordinate system conversion, calculate the camera pose parameters for three-dimensional registration.

[0034] Step 109: Draw a virtual color cube model using the OpenGL three-dimensional graphics library;

[0035] Step 110: The virtual color cube model generated by OpenGL is rendered into the real scene in real time according to the three-dimensional registration matrix in step 108, the 2D feature points tracked by the camera are directly matched one by one with the 3D space coordinate points updated in real time by the virtual color cube model, and the changed camera posture is calculated so that the virtual color cube is accurately superimposed on the target object, and finally the tracking registration is completed to achieve an augmented reality effect.

[0036] The present invention makes full use of the advantages of SiamFC-based end-to-end tracking and online model update based on DSST filter tracker to realize interference-aware tracking, and compensates for the deficiency of deep features in SiamFC through HOG features, uses fast ORB algorithm for matching and registration, and develops a twin network augmented reality target tracking and registration method combined with scale estimation and tracking DSST. The method uses SiamFC to regard tracking as similarity learning, and performs initial positioning of the target according to the position with the largest response value in the correlation response score map; further, the relatively accurate candidate target position screened by SiamFC is passed to the DSST tracker for target scale estimation, thereby obtaining a more accurate target area to be registered; in order to enable the augmented reality system to superimpose virtual information at the position to be registered, the present invention adopts the ORB algorithm with good real-time performance to detect and match the features of the target to be registered, ensuring real-time while improving the accuracy and robustness of the results of the traditional augmented reality tracking and registration algorithm.

[0037] The effect of the present invention is further described below in conjunction with experiments:

[0038] 1. Experimental conditions

[0039] The experimental environment is Windows 10 (64-bit) operating system, Intel(R) Core(TM) i7-8750H@3.40GH, 32G memory, the experimental platform is Matlab 2015b and Visual Studio 2015, based on computer vision library (Open CV), graphics library (OpenGL) and deep learning toolbox MatConvNet, and uses Logitech C270 camera.

[0040] The experimental data is selected from the OTB2015 dataset, which contains 100 fully annotated videos with substantial changes. The sequences in the dataset are annotated with 11 different attributes: scale change (SV), occlusion (OCC), deformation (DEF), fast motion (FM), in-plane rotation (IPR), out of view (OV), background clutter (BC), etc. Five video sequences of Bird1, Coupon, Box, Liquor, and Deer are selected.

[0041] The evaluation index includes two basic parameters: (1) center position error, which is represented by the precision plot to show the average pixel distance (Euclidean distance) between the center position of the tracked target and the manually annotated accurate position; (2) area overlap ratio, which is measured by the area overlap ratio between the bounding box obtained by the tracking algorithm and the manually annotated accurate bounding box. First, the overlap score (OS) is defined. The bounding box obtained by the tracking algorithm is denoted as a, and the box given by the ground-truth is denoted as b. The overlap rate is defined as: OS = |a∩b| / |a∪b|, |·| represents the number of pixels in the region; when the OS of a frame is greater than the set threshold, the frame is considered successful (Success), and the percentage of successful frames in all frames is the success rate (Success rate). The value range of OS is 0-1. The area under the curve (AUC) of each success rate graph is used to rank the tracking algorithms.

[0042] 2. Experiment and results analysis

[0043] (1) Moving target tracking and result analysis

[0044] The performance of different tracking algorithms is compared between the method of the present invention and deep learning algorithms: CFnet, DCFnet, SiamFC and related filter algorithms: Staple, DSST, KCF on OTB2015. The algorithm of the present invention is compared and analyzed with SiamFC, DSST and other algorithms on a video dataset sequence containing similar background, occlusion, fast motion and motion blur. The tracking situation is as follows Figure 2 shown. Figure 2 It is shown that SiamFC performs well in sequences with occlusion and fast motion (Box, Deer) and motion blur (Deer), but fails to track when similar distractors appear (Coupon, Liquor, Bird1), which is due to its semantic feature representation and lack of online model update; DSST tracker learns correlation filters on HOG features, which performs well in sequences with partial deformation and similar distractors (Coupon, Bird1), but drifts when the target is severely occluded (Box, Deer) and fast motion (Liquor). Due to the complementary characteristics of SiamFC and DSST filters, the method of the present invention overcomes the limitations of the two trackers by introducing DSST into the tracking process of SiamFC. In complex background situations where similar target interference, partial occlusion of the target, etc. may cause target tracking failure, the experimental results show that the tracking methods proposed in the present invention can track the target well.

[0045] In order to more clearly show the tracking performance of each tracking algorithm, the present invention evaluates the algorithm based on two indicators: accuracy and success rate: accuracy represents the proportion of frames whose distance does not exceed the accurate value of the threshold to the total number of frames, and the present invention uses a 50-pixel threshold; success rate represents the overlap degree of the BenchMark where the predicted target is located. The one-pass evaluation (OPE) result is obtained based on the exact position of the ground-truth initialized in the first frame, and then the algorithm of the present invention is compared with other tracking algorithms with better performance. Figure 3 Figure 2 shows the accuracy and success rate graphs on the OTB2015 benchmark. The numbers in the legend indicate the representative accuracy of the 20-pixel accuracy graph and the area under the curve (AUC) of the success rate graph. Figure 4 A comparison of the success rates under four different attributes is shown. Figure 4As can be seen in the figure, due to the limited search area caused by the boundary effect, correlation filters such as KCF and DSST usually perform poorly in the case of fast motion, motion blur and out-of-view; Staple uses the feature combination of HOG and CN (color histogram), so it shows better performance; except for the case where similar interferers are included in the background clutter, SiamFC's overall performance is better than the correlation filter tracking algorithm, which can be explained as the depth feature is better than the hand-crafted feature; the tracking algorithm of the present invention combines the advantages of the deep features of SiamFC and the HOG features of DSST, so as to more effectively handle all challenging scenes. In particular, an absolute gain of nearly 10% is obtained in the case of background clutter, which further proves the effectiveness of the method of the present invention in mitigating the impact of similar interference. Table 1 shows the quantitative comparison of the overlap rate and 20-pixel accuracy under the overlap threshold of 0.5. The algorithm of the present invention is excellent in both accuracy and overlap rate score. Specifically, the algorithm of the present invention improves the accuracy (DP) by 3.9% and the overlap rate score (OS) by 1.4% compared with SiamFC, and achieves an absolute gain of 13.2% accuracy and 12.1% overlap rate on the basis of DSST; in addition, the comprehensive performance of the algorithm of the present invention is better than that of the combination of DCFnet and CFnet based on the correlation filter tracker and the twin network. The experimental results show that the algorithm of the present invention has excellent performance and maintains real-time speed.

[0046] Table 1 Quantitative comparison results of accuracy and success rate of each algorithm %

[0047]

[0048] (2) Analysis of registration results of motion targets

[0049] Registration result analysis: We select each frame of the Box and Liquor video sequences with IV, SV, OCC, MB, IPR, OPR, OV, BC, LR, FM and other attributes in OTB2015, and combine the hybrid tracking registration with ORB feature detection and matching. The rendered virtual cube is superimposed in front of the box and bottle. The results are shown in Figure 2. Figure 5As shown. In the figure, (1) is the Box, (2) is the Liquor video sequence, and the number of video sequence frames increases gradually from (a) to (d). (a) is the front registration, (b) is the registration result after rotating 180°, (c) is the registration result after changing a certain viewing angle, and (d) is the registration result when the target is partially blocked by the pencil. It can be observed that during the target movement, the tracking and registration method of the present invention can still accurately track the target under four interference conditions, and the virtual color cube can be more accurately superimposed on the target area to be registered. This is because the registration method based on natural feature points has good resistance to partial occlusion and the like. As long as there are a sufficient number of visible feature points, the tracking and registration effect will not be affected. In general, the augmented reality target tracking and registration method of the present invention shows good accuracy and robustness.

Claims

1. A twin network augmented reality target tracking registration method combined with DSST scale estimation, characterized in that: The following steps are involved: Step 1, SiamFC initial target position estimation: Use SiamFC to regard tracking as similarity learning, train a deep twin network offline, and use the deep twin network to compare the similarity between the search area and the target template during online tracking. Find the sample that is most similar to the target template marked in the first frame from many candidate frames, and convolve the sample and the target template to generate a response map. The position with the highest response value in the response map is the position of the initial target. Step 2: DSST target scale estimation: The initial target position is passed to the DSST tracker, multi-scale image composition samples are collected at the initial target position, the scale filter is trained independently, and the target scale is estimated based on the response value in the scale filter response map of the sample; Step 3: Target relocation: The estimated target scale is transmitted back to the SiamFC tracker, and the initial target position and target scale are used to indicate the tracking of the next frame; the learning rate is adaptively adjusted using the difference between the two frames, and the SiamFC tracker related filter coefficients are updated using linear interpolation. The calculated tracking response map score based on the twin neural network is Response plot scores based on DSST tracking Using the formula The fusion is performed by linear summation, where is the target template image, To search for images, In order to fuse the disturbance perception model parameters, the target obtained in step 1 is relocated, and the position with the largest score in the final response score map is found to obtain the accurate positioning of the final target, and the target area to be registered is obtained, that is, the accurate position of the tracked target; Step 4: ORB feature matching and 3D registration: Use the ORB algorithm for feature detection and matching, use RANSAC to eliminate mismatches through Hamming distance, and obtain the registration matrix based on the feature relationship between adjacent frames. Superimpose the virtual color cube model drawn by OpenGL on the real scene through the registration matrix to complete the tracking registration.

2. A twin network augmented reality target tracking registration method combined with DSST scale estimation as claimed in claim 1, characterized in that: In the step 1, a threshold is set according to the distribution of local maximum values ​​on the response graph to screen out potential target locations.

3. A twin network augmented reality target tracking registration method combined with DSST scale estimation as described in claim 2, characterized in that: The threshold is an inverse Gaussian distribution peaked at the center of the response map.

4. The twin network augmented reality target tracking registration method combined with DSST scale estimation as claimed in claim 1, characterized in that: The step three is to fuse the response map score based on twin network tracking in step one with the response map score based on DSST tracking in step two by linear summation, update the position filter, complete accurate target tracking, and obtain the target area to be registered.

5. The twin network augmented reality target tracking registration method combined with DSST scale estimation as described in claim 1, characterized in that: In step 4, the tracking registration is to directly match the 2D feature points tracked by the camera with the 3D space coordinate points updated in real time by the virtual color cube model, and calculate the changed camera posture so that the virtual color cube is accurately superimposed on the target object.

Images