Long-term tracking method based on infrared and visible light fusion

Abstract

The invention relates to the field of target tracking technology, and in particular to a long-term tracking method based on infrared and visible light fusion, comprising manually selecting a tracking target and training a motion correlation filter on the target; finding the best scale of the target and recording a second frame response value τ of the target; establishing a new frame target search area, which is at the same position as a target frame of a previous frame and has an area 1.5 times that of the target frame; extracting directional gradient histogram features and color features from a visible light image, extracting directional gradient histogram features and grayscale features from an infrared image, cascading the four groups of feature vectors to form a fused new feature vector and weighting it by a cosine window; judging whether a tracker fails in tracking, and, after judging that the tracker fails in tracking, searching for the lost target in two source images of infrared and visible light; the invention solves the problem that the target is easily lost in complex environments such as occlusion, low resolution, rainy days, dusk, etc., and realizes robust tracking of the target in the complex environment.

Description

Technical Field

[0001] The invention relates to the technical field of target tracking, and in particular to a long-term tracking method based on the fusion of infrared and visible light. Background Art

[0002] As a hot research direction in the field of computer vision, the fusion tracking of infrared and visible light has received widespread attention due to its technical advantages and execution efficiency. Especially in this "new crown pneumonia" epidemic, binocular temperature detection and personnel monitoring technology based on infrared and visible light has played an extremely important role in the work of checkpoint inspection. Compared with target tracking based on a single sensor, fusion tracking under multiple sensors has developed relatively slowly due to the lack of a unified representation method for the target and available data sets. At present, target tracking is mainly focused on the research of single-source data, among which single target tracking under visible light is the most prominent. Since visible light images and infrared images are isomorphic data and these two images have good complementarity, fusion tracking based on visible light and infrared has gradually attracted the attention of technicians.

[0003] The biggest advantage of correlation filter tracking is that it can easily meet the requirements of real-time tracking, which is beyond the reach of other tracking methods. As far as we know, most of the fusion tracking algorithms based on the correlation filter framework adopt decision-level fusion. Wang et al. proposed a soft consistency correlation filter, which uses soft consistency to simultaneously consider the collaboration and heterogeneity of visible light and infrared modalities, and then fuses the response maps of the two modalities together under a weighted fusion mechanism to obtain the final response. Zhai et al. proposed a correlation filter tracking algorithm with cross-modality. The authors observed in the experiment that similar filters can be learned under different modal features, and this feature can be used to consistently locate the target object. Therefore, they introduced low-rank constraints to jointly learn cross filters to better obtain the mutual dependence between modalities. Yun et al. proposed a discriminative fusion correlation learning model, which mainly estimates the target first and then performs weighted fusion on the response map. In addition, some studies have combined correlation filtering with other methods. For example, Luo et al. combined the correlation filter-based tracker with the histogram-based tracker, and Ren et al. combined the correlation filter-based visible light tracking with the Markov chain Monte Carlo-based infrared tracking.

[0004] Although image fusion technology is relatively mature, it still faces three prominent problems. First, methods based on deep learning are rarely used for the fusion of infrared and visible light images. Second, fusion methods based on saliency are prone to mistaking areas with severe noise as target areas. Third, traditional fusion methods only focus on generating images with richer information but ignore the actual application needs of image fusion in specific scenarios. For target tracking with infrared and visible light image fusion, there are three main problems. First, there is a lack of a unified representation method for the target model, especially in environments such as time asynchrony and image mismatch. Second, there is a lack of long-term tracking solutions under multi-source image fusion. There is currently no good solution to tracking failures caused by interference factors such as severe occlusion, sudden changes in illumination, and morphological distortion. Third, there is a lack of a decision-making mechanism for fusion tracking. Most of the current algorithms use decision-level fusion, but ignore the available information under pixel-level and feature-level fusion. In general, image fusion and fusion tracking technology have made significant progress in recent years, and the future development trend of the field will also focus on demand-driven and provide solutions corresponding to the needs. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a long-term tracking method based on the fusion of infrared and visible light, which is particularly suitable for target tracking in conditions of obstruction, low light, and night.

[0006] A long-term tracking method based on infrared and visible light fusion, characterized in that the method comprises:

[0007] Manually select the target to be tracked and use the following formula (4) to train the motion correlation filter w on the target;

[0008] w=argmin∑|φ(x m,n )·wy(m,n)| 2 +λ|w| 2 (4)

[0009] Among them, φ represents mapping to the kernel space; according to the shift amount, a Gaussian label is given to the training image. If the shift amount is smaller, the label value is closer to 1, otherwise it is closer to 0; λ is the regularization parameter;

[0010] Use formula (8) to find the optimal scale β of the target i , enter the next frame, where λ=10 -獈 , Gaussian kernel width σ = 0.1;

[0011]

[0012] Record the second frame response value τ of the target;

[0013] Establish a new frame target search area, which is in the same position as the target frame of the previous frame and has an area 1.5 times that of the target frame;

[0014] The oriented gradient histogram features and color features are extracted from the visible light image, and the oriented gradient histogram features and grayscale features are extracted from the infrared image. The four sets of feature vectors are cascaded to form a fused new feature vector and weighted by a cosine window. The maximum response value under motion and scale is obtained by combining equations (6) and (8). The maximum response value under motion and scale is selected. The position of is taken as the translation estimation result of the target, and at the same time, the β corresponding to the maximum response value is selected i The best measure as a goal;

[0015] It is determined whether the tracker fails to track based on the continuous response value, and after determining that the tracker fails to track, the lost target is found in the infrared and visible light source images.

[0016] In a further technical solution, the method further includes, after manually selecting the target in the first frame, the correlation tracker regresses a cyclically shifted version of the input feature into a Gaussian kernel function and locates the target by searching for a maximum value on the response map.

[0017] In a further technical solution, the method for determining whether the tracker fails to track specifically includes:

[0018] Evaluate the tracking results based on the changing trend of the response value:

[0019] Y=[y(1),y(2),y(3),y(4),y(5)]<d·τ

[0020] sum(Y<θ·d·τ)≥2,θ<1

[0021] Where y(i) is the response value, y(i) is the element of Y, θ is the coefficient, and the operator sum() is used to calculate the number of y(i) < θ·d·τ in the set Y. When the response value of five consecutive frames meets the above conditions, the target tracking is judged to have failed, then the tracking is stopped and the re-detection mechanism is started.

[0022] In a further technical solution, the method for finding the lost target in both infrared and visible light source images specifically includes:

[0023] First, a template pool construction method is used to maintain the uncontaminated target template;

[0024] The GFCE algorithm is used to fuse the infrared image with the visible light image, and then the fused image is input into the heavy detector to transform the target detection of multi-source images into the target detection of single-source images.

[0025] Use EdgeBox to detect the single-source image and obtain a proposal;

[0026] The proposals are filtered by restricting appearance, and then the corresponding positions of the infrared and visible light images are marked according to the remaining detection results. Finally, the features of these marked areas are extracted and fused into new feature vectors, which are then input into the correlation filter for correlation operations.

[0027] In a further technical solution, the steps of fusing the new feature vector and inputting it into the correlation filter for correlation operation include:

[0028] After traversing each annotated area, a candidate sample set is formed;

[0029] For each candidate sample, use equations (6) and (8) to perform correlation filtering on it in turn and take out the maximum response value. If the maximum response value is greater than ντ, the target frame is output and used as a new initial condition to start the tracker; otherwise, enter the next frame for detection until the target is detected; after obtaining the target frame of the current frame, enter the next frame.

[0030] In a further technical solution, the method further comprises:

[0031] During tracking, for each frame, condition (12) is used to determine whether the template needs to be updated.

[0032] The updated plan is:

[0033]

[0034] Where η is the learning rate, and the above steps are repeated until the image sequence ends.

[0035] In order to achieve the above object, the present invention adopts the following technical solutions:

[0036] Compared with the prior art, the present invention has the following technical effects:

[0037] The long-term tracking method based on infrared and visible light fusion provided by the present invention solves the problem that the target is easily lost in complex environments such as occlusion, low resolution, rainy days, and dusk, and realizes robust tracking of targets in complex environments.

[0038] In addition, compared with the existing algorithm, the tracking method provided by the present invention is only for short-term tracking of the target. When the target is lost, the algorithm in the prior art cannot continue to perform the tracking task. The tracking method provided by the present invention effectively solves the problem of long-term tracking in the current fusion tracking field and has great practical value.

[0039] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The theoretical framework of the real-time long-term tracking method based on correlation filtering constructed for the present invention;

[0041] Figure 2 It is a typical case of the relevant filter response value curve and result reliability involved in the present invention;

[0042] Figure 3 The results of the present invention on the RGBT210 dataset are compared with those of the other eight most recent tracking methods in terms of overlap success rate;

[0043] Figure 4 The results of the present invention on the RGBT234 dataset are compared with those of eight other robust tracking methods in terms of overlap success rate.

[0044] Figure 5 The overlapping success rate of the present invention is compared with other 8 robust tracking methods on the RGBT234 dataset under background clutter attributes;

[0045] Figure 6 The overlap success rate of the present invention is compared with that of other 8 robust tracking methods under camera motion attributes on the RGBT234 dataset;

[0046] Figure 7 The overlap success rate of the present invention is compared with that of other 8 robust tracking methods on the RGBT234 dataset under morphological distortion attributes;

[0047] Figure 8 The overlap success rate of the present invention is compared with other 8 robust tracking methods under fast motion attributes on the RGBT234 dataset;

[0048] Fig. 9 The overlap success rate of the present invention is compared with that of other 8 robust tracking methods under severe occlusion attributes on the RGBT234 dataset;

[0049] Fig.10 The overlap success rate comparison between the present invention and other 8 robust tracking methods under low light attributes on the RGBT234 dataset;

[0050] Fig.11 The overlap success rate of the present invention is compared with other 8 robust tracking methods on the RGBT234 dataset under low-resolution attributes;

[0051] Fig.12The overlap success rate of the present invention is compared with other 8 robust tracking methods under motion blur attributes on the RGBT234 dataset;

[0052] Fig.13 The overlap success rate of the present invention is compared with other 8 robust tracking methods on the RGBT234 dataset under no occlusion attributes;

[0053] Fig.14 The overlap success rate of the present invention is compared with that of other 8 robust tracking methods under partial occlusion attributes on the RGBT234 dataset;

[0054] Fig.15 The overlap success rate comparison between the present invention and other 8 robust tracking methods under the scale estimation attribute on the RGBT234 dataset is shown;

[0055] Fig.16 The overlap success rate of the present invention is compared with that of other 8 robust tracking methods on the RGBT234 dataset under the thermal cross attribute;

[0056] Fig.17 This is a visual comparison of the results of some sequences of the present invention on the RGBT234 dataset with those of eight other robust tracking methods. DETAILED DESCRIPTION

[0057] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific drawings.

[0058] The present invention provides a long-term tracking method based on infrared and visible light fusion, the method comprising: manually selecting a tracking target, using the following formula (4), and training a motion correlation filter w on the target;

[0059] w=argmin∑|φ(x m,n )·wy(m,n)| 2 +λ|w| 2 (4)

[0060] Among them, φ represents mapping to the kernel space; according to the shift amount, a Gaussian label is given to the training image. If the shift amount is smaller, the label value is closer to 1, otherwise it is closer to 0; λ is the regularization parameter;

[0061] Use formula (8) to find the optimal scale β of the target i , enter the next frame, where λ=10 -獈 , Gaussian kernel width σ = 0.1;

[0062]

[0063] Record the second frame response value τ of the target;

[0064] Establish a new frame target search area, which is in the same position as the target frame of the previous frame and has an area 1.5 times that of the target frame;

[0065] The oriented gradient histogram features and color features are extracted from the visible light image, and the oriented gradient histogram features and grayscale features are extracted from the infrared image. The four sets of feature vectors are cascaded to form a fused new feature vector and weighted by a cosine window. The maximum response value under motion and scale is obtained by combining equations (6) and (8). The maximum response value under motion and scale is selected. The position of is taken as the translation estimation result of the target, and at the same time, the β corresponding to the maximum response value is selected i The best measure as a goal;

[0066] It is determined whether the tracker fails to track based on the continuous response value, and after determining that the tracker fails to track, the lost target is found in the infrared and visible light source images.

[0067] The method also includes, after manually selecting the target in the first frame, the correlation tracker regresses a cyclically shifted version of the input features into a Gaussian kernel function and locates the target by searching for a maximum value on the response map.

[0068] It should be pointed out that the basic assumption of the above methods is that the cyclically shifted version of the input features is close to the dense samples of the target at different positions.

[0069] The characteristic of correlation filtering is its high speed. For the initial target position of the input, for the subsequent frames, the filter will convolve the image block of the frame near the position of the previous frame, and the output after convolution corresponds to a grayscale response map. The grayscale reflects the degree of correlation. The higher the grayscale value, the greater the degree of correlation. The position with the maximum grayscale in the response map is the new position of the target. The input image and the correlation filter are transferred to the Fourier domain by Fast Fourier Transform (FFT). In this domain, the correlation operation becomes a dot product, which greatly improves the computational efficiency, as shown in formula (1).

[0070] G=F⊙H * (1) Where F = F(f im ) and H = F(h) is represented as image f im and filter h are transformed to the Fourier domain, where ⊙ and * are the point product operation and conjugate transpose, respectively.

[0071] Then use the inverse FFT transform F -1 Transform G back to the spatial domain to obtain the response map. The computational complexity of the whole process is only O(P log P), where P is the number of pixels in the tracking window.

[0072] The present invention uses a motion correlation filter in a KCF tracker framework, and uses a scale pooling method in a SAMF (Scale Adaptive Correlation Filter) algorithm to estimate the scale of a target.

[0073] Assuming that the selected target box size is M×N, first use circular shift to increase the number of samples to obtain sample x m， n, where (m, n)∈{0, 1, ..., M-1}×{0, 1, ..., N-1}.

[0074] The specific principle of cyclic shift is: Assume x = [x1, x2, ..., x n ] T is an n-dimensional column vector, P is the permutation matrix for cyclic shift of x, then the shifted sample is P l x, so we get the sample set {P l x|l=0,1,...,n-1}. The permutation matrix is:

[0075]

[0076] Combine all shifts of an n×1 vector x into a circulant matrix X:

[0077]

[0078] The first row is the original column vector x, the second row is the elements in x shifted one position to the right, and so on for the other rows. The purpose of circular shift is to circularly shift the convolution of the encoded vector. Due to the cyclic characteristics, the reference sample can be obtained periodically every n shifts. Similarly, it can be equivalently regarded as the first half of the circulant matrix X shifting the elements of vector x in the positive direction, and the second half shifting in the reverse direction.

[0079] In getting x m，n Finally, the mean square error between the training image and the regression target is minimized by ridge regression to obtain the motion correlation filter w∈R M×N , as shown in formula (4):

[0080]

[0081] Where φ represents the mapping to the kernel space. According to the shift amount, the training image is given a Gaussian label. If the shift amount is smaller, the label value is closer to 1, otherwise it is closer to 0. λ is the regularization parameter.

[0082] After mapping and discrete Fourier transform, the solution of w can be expressed as a linear combination of samples: w = ∑ m,n a(m,n)φ(x m,n), where the coefficient α can be obtained by formula (5), using the Gaussian kernel The kernel mapping φ is defined as κ = φ(x)·φ(x').

[0083]

[0084] When processing the next frame, the filter w will perform correlation operations with the M×N image block near the target position of the previous frame. After inverse discrete Fourier transform to the spatial domain, a response map will be obtained, which is calculated by formula (6):

[0085]

[0086] Where H = F(h)w, h i =κ(x,z i ) is an element of h, z i is the training sample obtained in the new frame, and x is the target model obtained in the previous frame. The position with the maximum value in is the target position of the obtained target in the new frame.

[0087] Since the kernel matrix only needs to calculate the dot product and vector norm, multiple channels can be applied to image features. Considering that visible light images have rich color information, while infrared images have more prominent grayscale information, we extract directional gradient histogram features and color features in visible light images, and directional gradient histogram features and grayscale features in infrared images. Then, these four sets of feature vectors are cascaded to form a fused new input sample x = [x1, x2, ...x L ], where L is the number of channels. For the kernel function in equation (5) under the multi-feature fusion of infrared and visible light images, its solution can be further obtained by:

[0088]

[0089] Where * represents the conjugate composite operation, ||x|| is the modulus of vector x, x′ is the transpose of vector x, and δ is the Gaussian kernel bandwidth. The KCF algorithm determines the target center position of each frame by iterating (5)-(7).

[0090] Considering that KCF cannot adapt to the scale change of the target, in order to further improve the performance, we use the scale pooling method, β = [β1…β i ]. By setting several scale candidate regions and obtaining the target’s response value, the scale corresponding to the maximum response value compared with the target in the previous frame is used as the target scale in the current frame.

[0091]

[0092] In the present invention, the method for determining whether the tracker fails to track specifically includes:

[0093] Evaluate the tracking results based on the changing trend of the response value:

[0094] Y=[y(1),y(2),y(3),y(4),y(5)]<d·τ

[0095] sum(Y<θ·d·τ)≥2,θ<1

[0096] Where y(i) is the response value, y(i) is the element of Y, θ is the coefficient, and the operator sum() is used to calculate the number of y(i) < θ·d·τ in the set Y. When the response value of five consecutive frames meets the above conditions, the target tracking is judged to have failed, then the tracking is stopped and the re-detection mechanism is started.

[0097] The target tracking result of the correlation filter depends on the position of the maximum response value. If the target is intact and not affected by the environment, the response graph is clear and the white dots are prominent. Otherwise, it is dim and blurred, such as when the target is occluded. When occlusion begins and the target is not completely occluded, the filter may still locate the target based on the previous training results. However, as time goes by, the occlusion area gradually increases, and the degree of pollution to the filter gradually deepens, eventually causing the polluted filter to be unable to re-track the target after exiting the occlusion, so the tracking fails.

[0098] In the process of target tracking, the tracker often fails to track due to the occlusion of the target, morphological distortion, and out of view. When the target is lost, the intervention of the re-detector helps to improve the tracking efficiency of the tracker, and whether to enable the re-detection mechanism depends on the reliability of the tracking result. In fact, the tracker is usually based on the output result of the previous frame, and then estimates the state of the target in the next frame according to the maximum response value, so the reliability of the tracking result is related to the time axis. To a certain extent, the larger the response value, the more reliable the tracking result, and the smaller the response value, the more reliable the tracker may have lost the target, so the change trend of the response value can intuitively reflect whether the tracking result is reliable. For target tracking under visible light, the inventor of the present application has extensively studied the connection between the response value and the tracking result, and from the perspective of the time axis, the reliability of the tracking result is evaluated by the fluctuation trend of the response value of 5 consecutive frames. Since the method of the present invention has only one feature template and also predicts the state of the target based on the maximum response value, it is still applicable to evaluate the tracking result based on the change trend of the response value for feature-level fusion tracking.

[0099] Y=[y(1),y(2),y(3),y(4),y(5)]<d·τ (9)

[0100] sum(Y<θ·d·τ)≥2,θ<1 (10)

[0101] Where y(i) is the response value, y(i) is an element of Y, and θ is the coefficient. The operator sum(·) is used to count the number of y(i) < θ·d·τ in the set Y. When the response values ​​of five consecutive frames meet the above conditions, we consider the target to be occluded. Then the tracking is stopped and the re-detection mechanism is started. In addition to the sharp drop in response values ​​caused by occlusion, other factors may also lead to the same result, such as illumination changes, scale changes, in-plane rotations, etc. Therefore, the proposed criterion is still able to identify tracking failures caused by other attributes.

[0102] The method of finding the lost target in both infrared and visible light source images specifically includes:

[0103] First, a template pool construction method is used to maintain the uncontaminated target template;

[0104] The GFCE algorithm is used to fuse the infrared image with the visible light image, and then the fused image is input into the heavy detector to transform the target detection of multi-source images into the target detection of single-source images.

[0105] Use EdgeBox to detect the single-source image and obtain a proposal;

[0106] The proposals are filtered by restricting appearance, and then the corresponding positions of the infrared and visible light images are marked according to the remaining detection results. Finally, the features of these marked areas are extracted and fused into new feature vectors, which are then input into the correlation filter for correlation operations.

[0107] The steps of fusing to form a new feature vector and then inputting it into the correlation filter for correlation operation include:

[0108] After traversing each annotated area, a candidate sample set is formed;

[0109] For each candidate sample, use equations (6) and (8) to perform correlation filtering on it in turn and take out the maximum response value. If the maximum response value is greater than ντ, the target frame is output and used as a new initial condition to start the tracker; otherwise, enter the next frame for detection until the target is detected; after obtaining the target frame of the current frame, enter the next frame.

[0110] When the tracking result is judged to be unreliable, the template pool construction method is first used to maintain the uncontaminated target template, and then the recovery mechanism is started to detect the target on the subsequent input infrared image and visible light image. The present invention provides two schemes for reacquiring the target, each of which covers two core steps: one is to extract features from the area of ​​interest and construct candidate samples; the other is to select the most likely sample from the candidate samples as the final output of the detector.

[0111] Different from target tracking under single-source data, tracking based on the fusion of infrared and visible light has two sets of input images, and the imaging methods of each set of images are significantly different. If EdgeBox is used for detection on each set of images, multiple different suspected targets will be obtained in each set of images. Since the size and position of each suspected target are different, it is very difficult to find the correct target from them, which also hinders the application of EdgeBox in the field of fusion tracking. To solve this problem, we first fuse the infrared image with the visible light image, and then input the fused image into the heavy detector, thereby converting the target detection problem of multi-source images into target detection of single-source images.

[0112] The core idea of ​​the EdgeBox algorithm is to calculate a score for each sliding window, which represents the possibility that the area corresponding to the sliding window contains an object. If the score is large, the area may contain an object. On the contrary, if the score is small, the area is less likely to contain an object. The main steps are: (1) edge segment and similarity and algorithm; (2) edge segment weight calculation; (3) sliding window score calculation.

[0113] In the process of image fusion, the GFCE algorithm is used, which can enhance the details of visible light and infrared images and retain the significant features of the images. Therefore, the obtained fused image can take into account the information of visible light and infrared images at the same time. After the image is detected by EdgeBox, many proposals will be obtained, but most of them do not meet the requirements, so it is necessary to conduct preliminary screening of these proposals to improve the real-time and accuracy of the tracker. In the method of the present invention, we use the method of limiting appearance to filter the proposals. Then, according to the remaining detection results, the corresponding positions of the infrared and visible light images are marked respectively. Finally, the features of these marked areas are extracted respectively and fused into new feature vectors, which are then input into the correlation filter for correlation operations.

[0114] The test results are finalized:

[0115] In the re-detector, a correlation filtering operation is implemented on each sliding window. If the response value of the target corresponding to the bounding box reaches the threshold ντ, the tracker is reinitialized using the detection result. However, environmental interference and changes in the target itself have a huge impact on the re-detector. Due to factors such as lighting changes, background clutter, and rapid motion, the response value corresponding to the sliding window will be relatively small. In order to effectively detect potential targets, the constraints on the detection results we set are as follows.

[0116] Max(R)>ντ (11)

[0117] Among them, R is the response value vector in all candidate sample sets, the operator Max(·) is the maximum value operation, ν is the constraint coefficient, which is 0.3 here, and then the maximum response value is compared with the set threshold to determine whether the detected proposal contains the target we want. The present invention will perform relevant operations on all filtered detection frames in turn, and compare the corresponding response value with the threshold. If the response value is greater than the set threshold, it will be adopted as the final detection result to restart the tracker to continue working, otherwise it will enter the next frame for detection until the correct target is detected.

[0118] Template update strategy:

[0119] During tracking, the appearance of the target will change due to rotation, deformation, etc. Therefore, the target template should be updated during tracking to achieve strong performance. If the target template is updated too frequently, the template is easily corrupted by noise. On the contrary, if the target template is updated too slowly, the template cannot capture the normal appearance changes of the target. A suitable update scheme is crucial for the tracker. For each frame, we use formula (12) to determine whether the template needs to be updated.

[0120] The model update strategy is described as follows

[0121]

[0122] Where η is the learning rate, and the above steps are repeated until the image sequence ends.

[0123] The core implementation step of the present invention is the target re-detection link after target tracking fails, while the existing algorithm is only for short-term tracking of the target. When the target is lost, the existing tracking algorithm cannot continue to perform the tracking task. This method proposes a long-term fusion tracking algorithm of infrared and visible light, which effectively solves the problem of long-term tracking in the current fusion tracking field. The core steps of its re-detection are as follows:

[0124]

[0125]

[0126] In order to better understand the technical solution of the present invention, further description is given below in conjunction with the accompanying drawings.

[0127] (1) Construct Figure 1 The theoretical framework of the long-term tracking method based on the fusion of infrared and visible light is shown;

[0128] (2) Manually select the target to be tracked, use formula (4), train the motion correlation filter w on the target, and use formula (8) to find the optimal scale β of the target i , enter the next frame, in the present invention, λ=10 -4 , Gaussian kernel width σ = 0.1;

[0129] (3) Record the second frame response value τ of the target;

[0130] (4) Establish a new frame target search area, which is located at the same position as the target frame of the previous frame and has an area 1.5 times that of the target frame.

[0131] (5) Extract the feature vectors of the infrared and visible light image targets respectively, concatenate them into a new feature x and weight them by a cosine window, and use equations (6) and (8) together to obtain the maximum response value under motion and scale. The position of the target is taken as the translation estimation result of the target. At the same time, the β corresponding to the maximum response value is selected i The best scale to aim for.

[0132] (6) Determine whether the most recent five consecutive frames meet the reliability evaluation criteria of the present invention, namely:

[0133] 1)Y=[y(1),y(2),y(3),y(4),y(5)]<d·τ

[0134] 2)sum(Y<θ·d·τ)≥2,θ<1

[0135] In the present invention, condition 1) is the basic condition, and if the response value continues to drop seriously in condition 2), it is determined as tracking failure. Wherein, d=0.3, θ=0.7.

[0136] (7) If the target is judged to be lost, tracking is stopped and the re-detector is enabled.

[0137] (8) Fusion of the subsequently input infrared and visible light images.

[0138] (9) Perform object detection on the fused image and filter the proposals by limiting the area to eliminate proposals that do not meet the requirements.

[0139] (10) Annotate the remaining proposals and extract image features in infrared and visible light images respectively.

[0140] (11) After traversing each labeled area, a candidate sample set is formed.

[0141] (12) For each candidate sample, use equations (6) and (8) to perform correlation filtering on it in turn and take out the maximum response value. If the maximum response value is greater than ντ, then output the target frame and use it as the new initial condition to start the tracker; otherwise, enter the next frame for detection until the target is detected. The final detection result is as follows: Figure 5 After obtaining the target frame of the current frame, enter the next frame.

[0142] (13) During tracking, the appearance of the target will change due to rotation, deformation, etc. For each frame, we use condition (12) to determine whether the template needs to be updated. The update scheme is: Where η is the learning rate. Repeat the above steps until the image sequence ends.

[0143] (14) In order to verify the effectiveness of the present invention, the inventors of this application chose to test on the RGBT210 and RGBT234 datasets. The RGBT210 dataset was proposed in 2017. Furthermore, RGBT210 was expanded to RGB-T234 in 2019. In the RGBT234 dataset, there are 234 pairs of infrared and visible light videos, with a total video frame number of 234K, and a maximum frame number of a single pair of videos of 8K. It also includes tracking scenes such as rainy days, nights, cold and hot days. Compared with other datasets, RGBT234 has the following advantages: a large number of videos and a large total number of video frames; rich tracking scenes and diverse tracking objects; very accurate registration of infrared images and visible light images; each group of videos is annotated with different attributes and the corresponding levels of the attributes are divided. In general, RGBT234 is the latest and most comprehensive fusion tracking dataset.

[0144] (15) Although the infrared and visible light video registration of the RGBT210 and RGBT234 datasets is very accurate, there are still small registration errors. Therefore, the maximum accuracy and maximum success rate are used instead of the accuracy and success rate. Specifically, the Euclidean distance on the infrared and visible light modalities is calculated for each frame, and the smaller distance is used to calculate the accuracy to obtain the maximum accuracy, while the maximum success rate is obtained by changing the threshold. Since the maximum accuracy can only reflect the center error of the target frame, and the maximum success rate takes into account both the center error and the size of the target frame, the maximum success rate is more valuable than the maximum accuracy in the evaluation index. The present invention is compared with some recent advanced fusion tracking algorithms SGT, SOWP+RGBT, CFnet+RGBT, JSR, CSR, MEEM+RGBT, KCF+RGBT and L1-PF.

[0145] (16) In order to more comprehensively evaluate the overall performance of the tracker, the inventors of this application conducted an attribute-based evaluation of the algorithm on the public RGBT234 benchmark dataset. All of these sequences are annotated with 12 attributes, covering a variety of challenging factors, including scale changes, severe occlusion, partial occlusion, no occlusion, motion blur, fast motion, distortion, low light, background clutter, camera movement, heat intersection, and low resolution. We use the overlap success rate in the benchmark dataset to characterize the performance.

[0146] (17) Figure 3 and Figure 4 It can be seen that the performance of the proposed method on both RGBT210 and RGBT234 datasets ranks first, which fully proves the effectiveness of the proposed algorithm.

[0147] (18) For 12 types of challenge attributes, Figures 5 to 16 It can be seen that in terms of the overlap success rate indicator, our algorithm ranks first among the six challenging attributes of camera movement, severe occlusion, low light, motion blur, partial occlusion, and scale estimation, ranks second in the thermal intersection challenging attribute, and ranks third in the four challenging attributes of background clutter, distortion, fast motion, and low resolution. The proposed algorithm not only solves the problems of target occlusion, low light, and perspective movement, but also effectively solves the problem of tracking drift caused by other factors.

[0148] (19) From Fig.17 Comparison of the mid- and long-term tracking results shows that the proposed tracker has obvious advantages over other trackers. In the sequence baby, the video shook severely before the 364th frame, and only the proposed method recovered the lost target, while other trackers failed to track. In the sequence car10, it can be seen that the target showed huge differences in the infrared and visible light images, and also underwent drastic scale changes and deformations, so most of the trackers drifted, and only the proposed algorithm tracked more accurately. As for the sequence elecbike, the main challenging factors are the severe occlusion of the target, scale changes and camera movement. From the tracking results, it can be seen that in the sequence elecbike, only the proposed method did not lose the target, while the other methods drifted into the background. In the sequence run, due to the interference of similar targets and severe occlusion of the target, some trackers drifted, but the proposed method still tracked the target robustly. From the visualization results, it can be seen that the present invention has very obvious advantages over other existing advanced algorithms in long-term stable tracking, and the feature fusion component and the target re-detection component do play an important role in improving the performance of the tracker.

[0149] The above shows and describes the basic principles, main features and characteristics of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the attached claims and their equivalents.

Claims

1. A long-term tracking method based on infrared and visible light fusion, characterized in that: The method includes: Manually select the target to be tracked and use the following formula (4) to train the motion correlation filter w on the target; w=argmin∑|φ(x m，n )·w-y(m,n)| 2 +λ|w| 2 (4) Among them, φ represents mapping to the kernel space; according to the shift amount, a Gaussian label is given to the training image. If the shift amount is smaller, the label value is closer to 1, otherwise it is closer to 0; I is the regularization parameter; For the initial target position of the input, for the subsequent frames, the filter will convolve the image block of the frame near the position of the previous frame. The output after convolution corresponds to a grayscale response map. The grayscale reflects the degree of correlation. The higher the grayscale value, the better the image quality. The greater the correlation, the position with the maximum grayscale in the response map is the new position of the target. The input image and the correlation filter are transformed into the Fourier domain by fast Fourier transform, and the correlation operation in this domain becomes the dot product, as shown in formula (1) G=F⊙H * (1) Where F = F(f im ) and H = F(h) is represented as image f im and filter h to the Fourier domain, ⊙ and * are the point multiplication operation and conjugate transpose, respectively. Then use the inverse FFT transform F -1 Transform G back to the spatial domain and obtain the response graph, The motion correlation filter in the KCF tracker framework is used, and the scale pooling method in the SAMF algorithm is used to estimate the scale of the target. Assuming that the selected target box size is M×N, first use circular shift to increase the number of samples to obtain sample x m，n , where (m, n)∈{0, 1, ..., M-1}×{0, 1, ..., N-1}, The specific principle of cyclic shift is: Assume x = [x1, x2, ..., x n ] T is an n-dimensional column vector, P is the permutation matrix for cyclic shift of x, then the shifted sample is P l x, so we get the sample set {P l x|l=0,1,...,n-1}, where the permutation matrix is: Combine all shifts of an n×1 vector x into a circulant matrix X: The first row is the original column vector x, and the second row is the elements in x shifted one position to the right. The purpose of the circular shift is to circularly shift the convolution of the encoded vector. Due to the circular characteristics, the reference sample can be periodically obtained after each shift n times. Similarly, it can be equivalently regarded as the first half of the circular matrix X shifting the elements of vector x in the positive direction, and the second half shifting in the reverse direction. In getting x m，n Finally, the mean square error between the training image and the regression target is minimized by ridge regression to obtain the motion correlation filter w∈R M×N , as shown in formula (4): Among them, φ represents mapping to the kernel space. According to the shift amount, the training image is given a Gaussian label. If the shift amount is smaller, the label value is closer to 1, otherwise it is closer to 0. λ is the regularization parameter. After mapping and discrete Fourier transform, the solution of w can be expressed as a linear combination of samples: w = ∑ m，n a(m,n)φ(x m，n ), where the coefficient α can be obtained by formula (5), using the Gaussian kernel Define the kernel mapping φ as κ = φ(x)·φ(x′), When processing the next frame, the filter w will perform correlation operations with the M×N image block near the target position of the previous frame. After inverse discrete Fourier transform to the spatial domain, a response map will be obtained, which is calculated by formula (6): Where H = F(h)w, h i =κ(x,z i ) is an element of h, z i is the training sample obtained in the new frame, x is the target model obtained from the previous frame, The position with the maximum value in is the target position of the obtained target in the new frame. Use formula (8) to find the optimal scale β of the target i , enter the next frame, where A = 10 -4 , Gaussian kernel width σ = 0.1; Record the second frame response value τ of the target; Establish a new frame target search area, which is in the same position as the target frame of the previous frame and has an area 1.5 times that of the target frame; The oriented gradient histogram features and color features are extracted from the visible light image, and the oriented gradient histogram features and grayscale features are extracted from the infrared image. The four sets of feature vectors are cascaded to form a fused new feature vector and weighted by a cosine window. The maximum response value under motion and scale is obtained by combining equations (6) and (8). The maximum response value under motion and scale is selected. The position of is taken as the translation estimation result of the target, and at the same time, the β corresponding to the maximum response value is selected i The best measure as a goal; Determining whether the tracker fails to track based on the continuous response value, and, after determining that the tracker fails to track, searching for the lost target in both the infrared and visible light source images; The method of finding the lost target in both infrared and visible light source images specifically includes: First, a template pool construction method is used to maintain the uncontaminated target template; The GFCE algorithm is used to fuse the infrared image with the visible light image, and then the fused image is input into the heavy detector to transform the target detection of multi-source images into the target detection of single-source images; Use EdgeBox to detect the single-source image and obtain a proposal; The proposals are filtered by restricting the appearance, and then the corresponding positions of the infrared and visible light images are marked according to the remaining detection results. Finally, the features of these marked areas are extracted and fused into new feature vectors, which are then input into the correlation filter for correlation operation. The steps of fusing to form a new feature vector and then inputting it into the correlation filter for correlation operation include: After traversing each annotated area, a candidate sample set is formed; For each candidate sample, use equations (6) and (8) to perform correlation filtering on it in turn and take out the maximum response value. If the maximum response value is greater than vτ, the target frame is output and used as a new initial condition to start the tracker; otherwise, enter the next frame for detection until the target is detected; after obtaining the target frame of the current frame, enter the next frame.

2. The long-term tracking method based on infrared and visible light fusion according to claim 1 is characterized in that: The method also includes, after manually selecting the target in the first frame, the correlation tracker regresses a cyclically shifted version of the input features into a Gaussian kernel function and locates the target by searching for a maximum value on the response map.

Images