Correlation filter tracking method based on saliency perception and spatio-temporal regularization

Abstract

The application provides a correlation filter tracking method based on saliency perception and space-time regularization, introduces a space regularization and a time constraint term into a target function, establishes a space regularization and time constraint model, converts the model into a frequency domain, decomposes the converted model into multiple sub-problems, solves the sub-problems respectively, obtains an optimized model, acquires a saliency map of a target region, fuses the saliency map of the target into initial space regularization weight coefficients to obtain new weight coefficients based on saliency perception, learns a correlation filter according to the optimized model, uses the correlation filter to locate the target and simultaneously completes scale estimation of the target, and finally updates model parameters to complete filter tracking, which is helpful to enhance time continuity and consistency of the model, effectively reduce boundary effects, effectively improve tracking performance and tracking efficiency, and enable the tracker to adapt to appearance changes and suppress background interference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a correlation filter tracking method based on saliency perception and spatio-temporal regularization. BACKGROUND

[0002] Since the method of least squares filter proposed by Bolme et al, because of its fast speed, method performance characteristics, a series of corresponding improved method is put forward. Henriques et al. use the properties of circulant matrix, put forward a target tracking method based on kernel circulant structure (Circulant Structure Kernels, CSK), and then put forward a kernel correlation filter tracking method (Kernel Correlation Filter, KCF), using the circulant shift matrix to increase the training samples, using ridge regression to learn the target tracker, and expressing the features in the form of multi-channel. After that, most of the correlation filter based tracking methods are based on the basic framework of KCF. The traditional discriminative correlation filter method uses the properties of circulant matrix to obtain the training samples, and solves the correlation filter model in the frequency domain through fast Fourier transform. The kernel correlation filter method greatly improves the speed of target tracking, but because the training samples of the kernel correlation filter method are obtained by circular shift of the center target, it is easy to cause the undesirable boundary effect, and some non-real samples are generated, thereby reducing the discriminative performance of the correlation filter. When the target moves quickly or is disturbed by the background, the tracker is easy to fail. Most methods add a cosine window when processing the feature map to weaken the influence of the boundary effect, but the effect of the cosine window is limited, and only part of the foreground information is learned while the background information is ignored. The SRDCF (Spatially Regularized Correlation Filters) method proposed in the prior art adds a spatial correlation regularization component to the standard DCF objective function, and solves the optimization problem in the frequency domain by using the sparse characteristics of the regularization. Although SRDCF can effectively suppress the boundary effect, the main problem of this method is that the calculation amount is large. Because the spatial regularization term cannot perfectly use the traditional CF circulant matrix method, the calculation amount is large, and the large linear equation and Gauss Seidel solution are very time-consuming, resulting in a large overall calculation amount. The BACF (Background-Aware Correlation Filters) method uses a binary mask matrix to act on the search samples, and uses dense sampling to obtain real positive and negative samples, reduces the influence of the boundary effect caused by the circular shift, and solves the problem of large calculation amount of the SRDCF method by using the Alternating Direction Method of Multipliers (ADMM, Alternating Direction Method of Multipliers) to alternately optimize the objective function, but it does not consider the influence of time, and the performance is general. SUMMARY

[0003] The present application aims to provide a correlation filter tracking method based on saliency perception and space-time regularization, so as to overcome the influence of the boundary effect of the correlation filter tracker in the prior art, and to effectively solve the problem of the large change of the filter template between the current frame and the previous frame caused by the too large change of the target region, enhance the time continuity and consistency of the model, effectively reduce the boundary effect, improve the tracking performance, and improve the tracking efficiency, and the tracker can adapt to the appearance change and suppress the background interference.

[0004] To achieve the above object, the present application provides the following technical scheme.

[0005] The correlation filter tracking method based on saliency perception and space-time regularization comprises the following steps.

[0006] S1: constructing a space regularization and time constraint model;

[0007] S2: converting the space regularization and time constraint model into a frequency domain, optimizing the converted model, decomposing the converted model into a plurality of sub-problems, solving the plurality of sub-problems respectively, and obtaining an optimized space regularization and time constraint model;

[0008] S3: obtaining a saliency map of the target, according to the saliency map of the target, the original weight coefficient of the target, and fusing the saliency map of the target into the optimized space regularization and time constraint model to obtain a new weight coefficient based on saliency perception;

[0009] S4: predicting the target position according to the optimized space regularization and time constraint model, and simultaneously completing the scale estimation of the target;

[0010] S5: updating the model parameters according to the optimized space regularization and time constraint model, the new weight coefficient based on saliency perception, the target positioning and the target scale estimation, so as to track the target in the next frame of image.

[0011] Preferably, the space regularization and time constraint model constructed in S1 is:

[0012]

[0013] wherein, is a time constraint term, is a space weight coefficient, represents the target feature of the i-th channel extracted in the region, the value of is , , represents the total number of channels, represents a Gaussian type expected response, denotes the first element of the denotes the correlation filter under the denotes the correlation filter learned in the previous frame. is a binary mask matrix, which represents the binary matrix cropped from the target feature with size are the second term spatial regularization parameters in the above formula, respectively, is the temporal constraint term parameter, denotes the circular shift operation, denotes the discrete circular shift operation of the step on the feature of the channel, and the symbol represents the calculation of the conjugate transpose.

[0014] Preferably, the spatial regularization and the temporal constraint model in S2 are equivalently converted into the frequency domain by the Parseval theory, and an auxiliary variable is introduced to solve.

[0015] Preferably, the converted formula in S2 is as follows:

[0016]

[0017] wherein the symbol denotes the discrete Fourier transform of a signal, is a spatial weight parameter, is a vector, denotes the expected response of the Gaussian type, is a matrix with a size of is a binary mask matrix denotes a constant matrix, is an auxiliary variable.

[0018] Preferably, the multiple sub-problems in S2 are the solving problem of the vector , the solving problem of the auxiliary vector , and the updating problem of the Lagrange multiplier.

[0019] Preferably, the solving formula of the vector is as follows:

[0020]

[0021] wherein, is a spatial weight parameter, is a Lagrange multiplier,​​​​​​ a constant matrix of is a binary mask matrix, is an auxiliary variable, is a regularization constant.

[0022] Preferably, the auxiliary vector is solved by the formula:

[0023]

[0024] wherein, is a matrix of size , represents the expected response of the Gaussian type, is a time constraint term parameter.

[0025] Preferably, the updating formula of the Lagrange multiplier is:

[0026] .

[0027] Preferably, the target weight coefficient in S3 is obtained by first using a saliency detection algorithm to obtain a saliency map of the target, and then fusing the saliency map of the target into the optimized spatial regularization and time constraint model, and using the saliency map to regularize the original weight coefficient to obtain a new target weight coefficient.

[0028] Preferably, the positioning mode of the target in S4 is:

[0029]

[0030] wherein, represents the number of channels of the feature map, represents a relevant filter learned in the previous frame in the frequency domain, represents the target feature of the i-th channel extracted in the region, represents an inverse Fourier transform.

[0031] ​​​Compared with the prior art, the present application has the following beneficial effects: the present application provides a correlation filtering tracking method based on saliency perception and space-time regularization, by introducing a space-time regularization term, a space regularization and time constraint model is established, then the space regularization and time constraint model is converted into a frequency domain, the converted model is decomposed into a plurality of sub-problems, the plurality of sub-problems are solved respectively, an optimized space regularization and time constraint model is obtained, a saliency map of a target is acquired, and the saliency map of the target is fused into the optimized space regularization and time constraint model to obtain a target weight coefficient; according to the optimized space regularization and time constraint model, target positioning is acquired, and scale estimation of the target is completed at the same time, so that target tracking is performed on a next frame of image, the problem of large change of a filtering template between a current frame and a previous frame caused by too large change of a target region is effectively solved, time continuity and consistency of the model are enhanced, boundary effect is effectively reduced, tracking performance is improved, tracking efficiency is improved, and the tracker can adapt to appearance change and suppress background interference. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flow chart of the correlation filtering tracking method based on saliency perception and space-time regularization of the present application;

[0033] Figure 2 is a saliency map obtained by saliency detection;

[0034] Figure 3 is a specific implementation flow framework of the present application;

[0035] Figure 4 is an OPE success rate curve graph of the present application and other nine methods;

[0036] Figure 5 is an EAO ranking of the present application and other methods. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below in combination with specific embodiments, which are an explanation of the present application rather than a limitation.

[0038] Construction of space regularization and time constraint model

[0039] Objective function of the correlation filtering model based on space-time regularization

[0040] In order to reduce the influence of the undesirable boundary effect caused by the cyclic shift, the method of the present application adds an adaptive space constraint term on the basis of the BACF method to improve the robustness of the filter. First, similar to the method of BACF, a binary mask matrix of space constraint is introduced Unlike BACF, which operates on target features, this invention uses a spatially constrained binarized matrix that directly acts on the filter before performing correlation operations on the target features. Then, spatial regularization is added to the model to accommodate appearance changes of different targets at different times. Simultaneously, when constructing the target appearance model, to enhance the model's temporal continuity and consistency, a temporal constraint term is added to the objective function to penalize large changes in the correlation filter between consecutive frames. The objective function is introduced as a time constraint term, where This represents the correlation filter learned in the previous frame. The temporal constraint effectively prevents excessive changes in the correlation filter between frames, keeping the filter's temporal changes smoother, thus making the learned appearance model more robust. By combining the adaptive spatial and temporal constraints, a new objective function is obtained, namely:

[0041] … (1)

[0042] in, For time constraints, Spatial weighting coefficient, Indicates the number extracted in this region Target characteristics of each channel The value is , This represents the total number of channels. This represents the expected response of the Gaussian type. express The One element, Indicates the first Correlation filters for each channel This represents the relevant filter learned in the previous frame. It is a binary mask matrix, representing the target features. medium cutting size is The binary matrix, These are the space regularization parameters in the second term of the above equation. These are parameters for the time constraint term. This indicates a cyclic shift operation. Indicates the first The discrete cyclic shift operation in step 1 Channel characteristics , symbol To calculate the conjugate transpose.

[0043] This method introduces prior information into the spatial weights, which effectively avoids model degradation.

[0044] Objective function optimization solution

[0045] According to the relevant filtering theory and typical relevant filtering method, it is known that the relevant filter is usually solved in the frequency domain more efficiently, so as to improve the calculation efficiency, the formula (1) is converted into the frequency domain by the equivalent conversion of the Pascval theory, and an auxiliary variable is introduced to solve, the converted formula is as follows:

[0046] … (2)

[0047] Wherein, , the size is , which is composed of K channel relevant filtering vectorization cascade , , the size is . Symbol represents the discrete Fourier transform of a certain signal, represents Constant matrix of, used to map any Dimension vector signal to the Fourier domain, for example . Auxiliary variable , wherein, Similar to the vector , which is composed of K channel relevant filtering vectorization cascade, the size is . Symbol is the conjugate transpose of the calculation complex vector or complex matrix.

[0048] It is found that the formula (2) is convex optimization, which can be solved by ADMM iteration. First, the augmented Lagrangian method (ALM) is used to write formula (2) into the following form:

[0049] … (3)

[0050] Wherein, Indicates the Lagrange multiplier vector, the size is , is the penalty factor. Then, formula (3) can be solved by ADMM method iteration. Next, by using ADMM to solve the subproblem , , and .

[0051] Subproblem :

[0052] The and are considered as known quantities, the optimization problem of the equation becomes a convex optimization problem which is differentiable and has a unique closed-form solution, thus, The minimization of

[0053] … (4)

[0054] Let , then , we have

[0055] …(5)

[0056] Next, take the partial derivative of with respect to each element of , and set it equal to 0, i.e.

[0057]

[0058] where , and by using the fast Fourier transform principle and , substitute this equation into the above equation, and let 0, we have

[0059]

[0060] Simplify it to

[0061] (λ1W T W+μTPP T )h-TPξ-μTPg=0… (8)

[0062] Then we can get

[0063] … (9)

[0064] where represents a row vector composed of the mask matrix, and . By applying the inverse transform estimation to each element in , , we can get the corresponding spatial values of and .

[0065] The subproblem :

[0066] Similarly, for solving the subproblem , we have , and are considered as known quantities, the optimization problem of the equation becomes a convex optimization problem which is differentiable and has a unique closed-form solution, thus, The minimization of

[0067] … (10)

[0068] It can be observed from equation (10) that each element of , denoted as , only depends on K values in and . Therefore, the solution of in equation (10) can be decomposed into solving linear subsystems of size , i.e., each independent objective function can be expressed as:

[0069] … (11)

[0070] where , .

[0071] Similarly, taking the partial derivative of equation (11) with respect to and setting it to 0, we have:

[0072] … (12)

[0073] Then taking the partial derivative of equation (12) with respect to , we have:

[0074] … (13)

[0075] Let , we can solve:

[0076] … (14)

[0077] According to the Sherman-Morrison theorem, , we can substitute , into equation (14) and express the inverse matrix of equation (14) as:

[0078] …(15)

[0079] Substituting the above equation into equation (14), we can obtain:

[0080] … (16)

[0081] where , and the scalar .

[0082] Sub-problems

[0083] Solving the above sub-problems Then, the Lagrange multiplier can be updated The update equation is expressed as:

[0084] … (17)

[0085] where, and denote the solutions of the sub-problems and in the Fourier domain at the th iteration, respectively, and the update of the regularization constant is performed according to where takes the value 1000, is a scale factor that takes the value 10.

[0086] The spatio-temporal constrained model constructed in this section is a convex optimization model, and it can be seen from the above sub-problem solutions that each sub-problem has a closed-form solution in the ADMM algorithm, so the model satisfies the Eckstein-Bertsekas condition. It is guaranteed to converge to the global optimum, and the ADMM algorithm can achieve convergence through three iterations on most sequences.

[0087] Significance-aware weight acquisition

[0088] In order to achieve the purpose of suppressing background information interference, this method obtains the saliency map of the target through a saliency detection algorithm, and then merges it into the spatial weight coefficient The method in this paper is to multiply the saliency map with the original spatial weight coefficient, and the new weight coefficient can better reflect the shape of the target.

[0089] First, this method crops the search area to be 2 times larger than the tracking target, uses the existing saliency detection method [7] to obtain the relevant saliency map , and the relevant saliency detection result is shown in Figure 1 . Then, in order to suppress the influence of the background, the size of the spatial weight coefficient is kept unchanged, and its size is adjusted to be the same size as the relevant filter, and is used to regularize the original weight coefficient . Therefore, the new weight coefficient mapping can be obtained as follows:

[0090] …(18)​​

[0091] For online object tracking, a step is added in the previous correlation tracking algorithm to calculate the weight map of the first frame using equation (1.18), where the object is labeled with a given bounding box. Based on the above fixed spatial regularization method, a regularization generation method that can adapt to the shape changes of the object is proposed. At the same time, since the discriminative ability of the learned spatial weight coefficient is limited by the context information, leading to poor tracking effect. Compared with the original spatial weight coefficient , is a continuous weight function. By using the new weight coefficient based on saliency perception in the tracking process, it is helpful to learn more discriminative correlation filters by using the context information around the target.

[0092] Target positioning and scale estimation

[0093] In the target tracking stage, the position of the tracked target can be obtained by using the results of the previous frame ADMM iteration and the feature map of the search region to respond, and the response is as follows:

[0094] …(19)

[0095] where denotes the number of channels of the feature map, denotes the correlation filter learned in the previous frame in the frequency domain, which is obtained by ADMM iteration. After obtaining the target response by equation (19), the maximum value of the response is the position of the target in the current frame.

[0096] For scale estimation, this method refers to the ASRCF method and obtains another scale correlation filter model by learning, and the positioning correlation filter model is complementary, where the scale correlation filter model selects 5 different scales of the target, and only uses the shallow HOG feature of the hand-crafted type when obtaining the feature map, which can greatly reduce the model calculation amount, while the positioning correlation filter model uses a method combining deep CNN features and shallow HOG features. The deep CNN features used in the positioning stage directly use the network framework VGG-16 that has been trained on ImageNet, and select the Conv4-3 layer as the feature extraction layer.

[0097] In each frame, first, the target position is estimated using the positioning correlation filter model, and then the target scale is refined using the scale correlation filter model to obtain the optimal target scale. The updating method of the target appearance model is as follows:

[0098] … (20)

[0099] where, and denote the target appearance model of the frame and the frame, respectively, denotes the online learning rate of the correlation filter model update.

[0100] Method description

[0101] Based on the optimization solution process of the above objective function, the introduction of the saliency-aware weight, the target localization, the scale estimation and the model update, the specific description of the proposed method is as follows:

[0102] Input: initial location information of the target in the first frame and scale information ;

[0103] Output: predicted target location and scale in the frame ;

[0104] 1. Extract the appearance features of the target according to the target location and scale information, including HOG features and CNN features, and obtain two initialized appearance templates, one of which contains only the scale template of 31-dimensional HOG features, and the other contains the localization template of 111-dimensional HOG and CNN combined features;

[0105] 2. Initialize , two appearance templates are calculated by formula (9) (16) to solve the sub-problems , , and update the corresponding sub-problems

[0106] 3. Calculate the saliency map, and obtain the new spatial weight based on saliency awareness by formula (18).

[0107] Repeat the following process:

[0108] 4. Scale estimation selects five different sizes to extract the corresponding HOG features, and localization estimation extracts deep and shallow combined features;

[0109] 5. The localization template calculates the correlation response by formula (19) , and the maximum value of the appearance response is taken as the target location, while the appearance features of the five scales and the maximum template response are taken as the target scale of the frame;

[0110] 6. Update the localization template and the appearance template by formula (20);

[0111] 7. Compute sub-problems by formula (9) (16) , , and update the computation of the corresponding sub-problems for 3 times .

[0112] 8. Compute saliency map , obtain new saliency-aware spatial weights using formula (18).

[0113] Advantages of the present application:

[0114] (1) The introduced spatio-temporal regularization term is integrated into the related filtering objective function, realizing the coupling of DCF learning and model updating, preventing large inter-frame filter changes, improving the accuracy and robustness of tracking, and adapting to the appearance changes of different targets at different times.

[0115] (2) The saliency detection method is used to obtain saliency-aware weights, which are fused into the original spatial weight coefficients, making the tracker adapt to appearance changes and suppress background interference.

[0116] (3) Unlike SRDCF which has high complexity due to the formula on multiple training images, the saliency and spatio-temporal regularization model (SSTCF) of the present application can be effectively optimized by ADMM algorithm, and all sub-problems have analytical solutions.

[0117] (4) The method of the present application is evaluated on two classic tracking benchmarks and one recent long-term tracking dataset, namely OTB2015 dataset, VOT2018 dataset and LaSOT dataset. The experimental results show that compared with existing trackers, the model has higher accuracy and real-time performance.

[0118] Embodiment:

[0119] Take all 100 videos in the OTB2015 dataset as the test dataset to test the overall performance of the present method and other 9 methods, and draw the One-Pass Evaluation (OPE) accuracy curve and OPE success rate curve as the evaluation index for experimental comparison. The comparison methods are: CNN_SVM, GradNet, Staple, DSST, SRDCF, DeepSRDCF, TADT, STRCF, and SiamFC.

[0120] The OPE accuracy curve and OPE success rate curve of the present method and other 9 methods are shown in Figure 4 .

[0121] (2) VOT dataset

[0122] The comparison of the method of the present application is the accuracy-robustness (AR) chart and table, and the average overlap expectation EAO. The compared methods are: DSTRCF, ECO, UpdateNetSRDCF, DAT, LSART, SiamFC and Staple. The EAO ranking is shown in Table 1. The accuracy table and the robustness table are shown in Table 1 and Table 2. Figure 5

[0123] Table 1. Accuracy size comparison

[0124]

[0125] Table 2. Robustness table

[0126]

[0127] Although the embodiments of the present application have been described in connection with the accompanying drawings, the present application is not limited to the above-described specific embodiments and application fields, and the above-described specific embodiments are merely illustrative, instructive, and not restrictive. Those of ordinary skill in the art can make many modifications under the teachings of the specification, without departing from the scope of the claims of the present application, and these are also included in the present application.​

Claims

1. A method of correlation filter tracking based on saliency perception and spatio-temporal regularization, characterized in that, The method comprises the following steps: S1: constructing a spatial regularization and time constraint model; S2: converting the spatial regularization and time constraint model into a frequency domain, optimizing the converted model, decomposing the converted model into multiple sub-problems, solving the multiple sub-problems respectively, and obtaining an optimized spatial regularization and time constraint model; S3: obtaining a saliency map of a target, obtaining an original weight coefficient of the target according to the saliency map of the target, and fusing the saliency map of the target into the optimized spatial regularization and time constraint model to obtain a new weight coefficient based on saliency perception; S4: predicting a target position according to the optimized spatial regularization and time constraint model and simultaneously completing scale estimation of the target; S5: updating model parameters according to the optimized spatial regularization and time constraint model, the new weight coefficient based on saliency perception, the target positioning and the target scale estimation, so as to perform target tracking on a next frame of image. The spatial regularization and time constraint model constructed in S1 is: wherein, is a temporal constraint term, is a spatial weight coefficient, represents the target feature of the i-th channel extracted in the region, represents the total number of channels, represents a Gaussian-type expected response, represents the i-th element of represents the correlation filter under the i-th channel, represents the correlation filter learned in the previous frame, is a binary mask matrix, representing a binary matrix with a size of cropped from the target feature are respectively the second term spatial regularization parameters in the above formula, is a temporal constraint term parameter, represents a circular shift operation, represents a discrete circular shift operation of the i-th step on the feature of the i-th channel, , and the symbol is a calculation of the conjugate transpose;​​​​​​​​ The converted formula in S2 is: wherein the symbol denotes a discrete Fourier transform of a certain signal, a spatial weight parameter, is a vector, denotes a desired response of Gaussian type, is a matrix of size is a binary mask matrix denotes a constant matrix, is a constant matrix, is an auxiliary variable.

2. The method of claim 1, wherein, In S2, the spatial regularization and time constraint model are converted into frequency domain by the Pascual-Leone theory, and auxiliary variables are introduced Solving is performed.

3. The method of claim 1, wherein, The sub-problems in S2 are the solution of vector , the solution of auxiliary vector , and the update of Lagrange multiplier.

4. The method of claim 3, wherein, Vector The solution formula is: wherein a spatial weight parameter, is a Lagrange multiplier, denotes a constant matrix, is a binary mask matrix, is an auxiliary variable, is a regularization constant.

5. The method of claim 4, wherein, auxiliary vector The solution formula is: wherein, is a matrix of size is a matrix of size represents an expected response of Gaussian type, is a parameter of the temporal constraint term.

6. The method of claim 5, wherein the spatial-temporal regularization is performed by using a spatial-temporal regularization term. The updating formula of the Lagrange multiplier is: 。 7. The method of claim 1, wherein, In S3, the target weight coefficient is obtained by first obtaining a saliency map of a target by using a saliency detection algorithm, then fusing the saliency map of the target into the optimized spatial regularization and time constraint model, and finally performing regularization processing on the original weight coefficient by using the saliency map to obtain a new target weight coefficient.

8. The method of claim 1, wherein, The target positioning mode in S4 is: wherein, represents the number of channels of the feature map, represents a correlation filter learned in the previous frame in the frequency domain, represents the target feature of the first channel extracted in the region, represents an inverse Fourier transform.

Images