A long-time target tracking method based on distortion suppression and low rank constraint

Abstract

The application discloses a long-time target tracking method based on distortion suppression and low-rank constraint, solves the problem that in the prior art, when a target is accompanied by an adverse factor, too much background information is introduced, the appearance of the target changes sharply in a short time, and model drift is easily caused, and even tracking fails, and the long-time target tracking method based on distortion suppression and low-rank constraint provided by the application fuses the background information around the target itself in the process of constructing a filter (model), effectively alleviates the boundary effect, simultaneously increases the search area, and improves the identification ability in a fast motion, occlusion or background clutter scene.

Description

Technical Field

[0001] The present invention relates to a target tracking method, and in particular to a long-term target tracking method based on distortion suppression and low-rank constraint. Background Art

[0002] Object tracking methods primarily utilize intelligent detection algorithms and precise manual annotation to determine the target's center position and range in an initial image sequence. This allows for continuous, stable, and real-time tracking of the target in subsequent image sequences, while also predicting its trajectory and state. Current object tracking frameworks focus on two main categories: Discriminant Correlation Filters (DCFs) and Siamese Networks (SNs).

[0003] Although the tracking paradigm based on the twin network has shown strong performance, its complex and time-consuming convolution operations and high dependence on the GPU platform limit its practical application. In contrast, correlation filtering achieves a high degree of coordination in the time domain and frequency domain by cleverly combining the fast Fourier transform and the cyclic structure of the tracking paradigm, thereby significantly improving the efficiency of target tracking. After continuous exploration, many innovative target tracking methods have been spawned, which not only enriches the theoretical system of target tracking, but also provides strong technical support for practical applications. For example:

[0004] The Chinese invention patent with publication number CN115760906A discloses a method for "anti-occlusion single target tracking with self-attention mechanism in UAV target tracking scenarios". It adopts HOG and position features, and cleverly uses the attention mechanism to achieve feature enhancement, and then obtains the optimal match by maximizing the total matching score, thereby realizing target position estimation.

[0005] The Chinese invention patent with publication number CN114926497A discloses "A single-target anti-occlusion tracking method and device based on ECO", which optimizes the model calculation amount through parameters such as detection, search area and threshold, and perceives the current tracking reliability through peak sidelobe ratio. When the feedback is unreliable, it uses global search mode and threshold adjustment to reposition the tracking.

[0006] The Chinese invention patent with publication number CN116468752A discloses "A single target tracking method for drones". It is aimed at drone scenarios and combines correlation filtering and Kalman filtering technology. In the target prediction stage, a response dynamic perception correlation filter is trained to deal with the smoothness between responses. Then, in the subsequent sequence, Kalman filtering is used to search for the position of the area of ​​interest, effectively dealing with the problem of target disappearance caused by rapid movement.

[0007] The above-mentioned literature all alleviates the impact of boundary effects by introducing an expanded search area. However, a more difficult problem is that when the target appears with similar objects, or when the target is partially / completely occluded, or when other unfavorable factors arise, the introduction of excessive background information causes the target appearance to change dramatically in a short period of time, which makes it very easy for model drift or even tracking failure to occur. Summary of the Invention

[0008] In order to solve the technical problem in the prior art that when the target is accompanied by unfavorable factors, the introduction of excessive background information causes the target appearance to change dramatically in a short period of time, which is very likely to cause model drift or even tracking failure, the present invention provides a long-term target tracking method based on distortion suppression and low-rank constraints.

[0009] The inventive concept of the present invention is to significantly improve tracking performance through two core strategies:

[0010] First, the present invention cleverly integrates distortion suppression and context-aware low-rank theory, aiming to explicitly integrate the target and its background information to enhance the discriminative power of the filter (model); imposes distortion suppression constraints during the filter (model) learning phase to adapt to the impermanent changes in target appearance; and emphasizes temporal consistency by guiding the filter (model) distribution on a low-dimensional discriminative manifold to improve tracking stability and accuracy.

[0011] Second, the present invention integrates the historical average peak energy (APCE) to accurately evaluate the reliability of the tracking results and dynamically coordinates the filter (model) and detector tracking model. When the tracking state is stable, the filter (model) constructed in the first stage is used for tracking; once tracking instability or anomalies are detected, the target recapture method is activated. First, a background subtraction strategy is used to generate coarse candidate regions. Then, a fine screening is performed based on the target's scale and position prior information. Finally, by optimizing the state and response confidence of the fine-grained candidate regions, the target is accurately captured and recovered, effectively avoiding model degradation problems.

[0012] In order to achieve the above objectives and complete the above invention concept, the present invention adopts the following technical solutions:

[0013] A long-term target tracking method based on distortion suppression and low-rank constraint is special in that it includes the following steps:

[0014] Step 1: Select multiple sets of continuous multi-frame images in the image sequence whose tracking status is continuously unreliable, and the previous frame image I of each set of continuous multi-frame images is t-1 The state and properties of are known. For each set of continuous multi-frame images, according to its previous frame image I t-1 The target state S t-1 =[x t-1 ,y t-1 ,vt-1 ,h t-1 ]Extract the current frame image I t The target feature x0 and its context block feature x i ; The current frame image I t is the image I close to the previous frame in the corresponding continuous multi-frame image t-1 images;

[0015] Among them, i∈[1,k], k is a positive integer, t is the current frame time, (x t-1 ,y t-1 ) represents the previous frame image I t-1 The center position of the target, (v t-1 ,h t-1 ) represents the previous frame image I t-1 The width and height of the target;

[0016] Step 2: Based on the target feature x0 and the context block feature x i , construct the current frame image I t The filter w t ;

[0017] Step 3: Use the sliding average method to update the filter w according to the preset frame interval t , and obtain the weighted average filter

[0018] Among them, α∈[0,1] represents the learning rate;

[0019] Step 4: Based on the previous frame image I t-1 The target state S t-1 =[x t-1 ,y t-1 ,v t-1 ,h t-1 ], in the current frame image I t Crop the region of interest (ROI) around the center point of the target and extract the interesting features x of the region of interest (ROI) t,ROI ;

[0020] Step 5: For the feature of interest x t,ROI With weighted average filter Perform convolution operation to obtain the current frame image I t Response graph R t :

[0021]

[0022] in, represents the inverse discrete Fourier transform, and Respectively and xt,ROI The discrete Fourier transform of

[0023] Step 6: Get the recapture flag Re ;

[0024] Step 7: According to the recapture flag Re Get the current frame image I t target state;

[0025] If the recapture flag is set Re =1, obtain the current frame image I through the recapture method t The target state S t =[x t ,y t ,v t ,h t ];

[0026] If the recapture flag is set Re =0, according to the response graph R t The peak value of the current frame image I is obtained t The target state S t =[x t ,y t ,v t ,h t ];

[0027] Among them, (x t ,y t ) represents the current frame image I t The center position of the target, (v t ,h t ) represents the current frame image I t The width and height of the target;

[0028] Step 8: For each frame image in each group of continuous multi-frame images in the image sequence whose tracking status is continuously unreliable, perform steps 1 to 7 in sequence in order to obtain the target state of each frame image in all continuous multi-frame images, and combine it with the target state of each frame image with a reliable tracking status to complete long-term target tracking.

[0029] Furthermore, step 2 specifically includes:

[0030] 2.1. Target feature x0 and context block feature x extracted in step 1 i , explicitly fuse the target and its surrounding background information to construct the first objective function:

[0031]

[0032] Among them, X 0,t and Xi,t Represent the target feature x0 and the context block feature x respectively i The circulant matrix, λ1, λ2 are constants, y is an ideal two-dimensional Gaussian response;

[0033] 2.2. Introducing distortion suppression constraints into the first objective function The second objective function is obtained:

[0034]

[0035] in, Represents the circular shift operation, p and q represent the previous frame image I t-1 and the current frame image I t The difference in peak displacement on the x-axis and the peak displacement on the y-axis, R t-1 Represents the previous frame image I t-1 Response plot of λ AS is the distortion suppression penalty parameter;

[0036] 2.3. Apply low-rank constraint rank(W to the second objective function t )-rank(W t-1 )≤N, to emphasize temporal consistency, the third objective function is obtained:

[0037]

[0038] strank(W t )-rank(W t-1 )≤N

[0039] in, The history filter w is stored in h , h∈[2,t-1], N is a negative integer; rank(.) represents the matrix rank operation; W t-1 is the historical filter matrix at time t-1;

[0040] 2.4. Use the sufficient condition of low-rank constraint to replace the low-rank constraint rank of the third objective function (W t )-rank(W t-1 )≤N, to reconstruct the third objective function and obtain the fourth objective function:

[0041]

[0042] Where μ is a constant and μ>0;

[0043] 2.5. Introducing equality constraints w into the fourth objective function t =g t , we get the augmented Lagrangian function:

[0044]

[0045] Among them, γ is the penalty parameter, s t is the Lagrange multiplier, g t is an auxiliary variable;

[0046] 2.6. Use the alternating direction multiplier method to decompose the augmented Lagrangian function and obtain the fifth objective function:

[0047]

[0048] Among them, the penalty parameter γ follows the rule γ=min(γ max ,ργ) update,γ max represents the maximum value of the penalty parameter γ, ρ represents the iteration step size, and m+1 represents the m+1th iteration;

[0049] 2.7. Iteratively optimize the fifth objective function and update the filter w in sequence t , auxiliary variable g t , Lagrange multiplier s t , penalty parameter γ, until the maximum preset number of iterations is reached, and the current frame image I is obtained t The optimal filter w t,opt , with the optimal filter w t,opt Replace the current frame image I t The filter w t .

[0050] Furthermore, step 6 specifically includes:

[0051] 6.1. Get the previous frame image I t-1 Weighted average APCE of historical average peak energy APCE avg,t-1 , and obtain the number of consecutive multi-frame images whose tracking status is continuously unreliable Count;

[0052] 6.2. Calculate the current frame image I t Response graph R t Average peak energy APCE t :

[0053]

[0054] Where W and H represent the response map R t The width and height, R max and R min Response graph R t The maximum and minimum response values ​​in R i,j Represents the response graph R t The response value of row i and column j in ;

[0055] 6.3. Average Peak Energy APCE t and weighted mean APCE avg,t-1 Make comparisons;

[0056] If APCE t ≤σAPCE avg,t-1 , then the frame number Count increases by 1;

[0057] If APCE t >σAPCE avg,t-1 , then the frame number Count is set to 0;

[0058] Among them, σ∈[0,1] represents a constant;

[0059] 6.4. Determine whether the number of frames Count satisfies the condition Count ≥ T, where T represents the minimum value for activating the background difference mechanism;

[0060] If Count≥T, then recapture flag flag Re is 1;

[0061] If Count < T, then recapture flag flag Re is 0.

[0062] Furthermore, in step 7, the current frame image I is obtained by recapture method. t The target state S t =[x t ,y t ,v t ,h t ], specifically including the following steps:

[0063] 7.1. Construct a mixed Gaussian background model and pass B t =αB t-1 +(1-α)B t-1 Update the mixed Gaussian background model to obtain the background model B t ;

[0064] 7.2 Extracting background model B t and the current frame image I t ORB features;

[0065] 7.3. Constructing background model B t With the current frame image I t The motion model P;

[0066] 7.4. Solve the motion model P using ORB features;

[0067] 7.5. Based on the motion model P, the background model Bt With the current frame image I t Align to get the aligned background model

[0068] 7.6. Align the background model and the current frame image I t Perform the difference to obtain the coarse-grained binary difference map D corse ;

[0069] 7.7. Coarse-grained binary difference map D corse Perform morphological filtering to obtain a fine-grained binary difference map D fine ;

[0070] 7.8. From the fine-grained binary difference map D fine Extracting coarse-grained candidate regions from Among them, (x n ,y n ) represents the center position of the nth candidate region in the coarse-grained candidate region set, (v n ,h n ) represents the width and height of the nth candidate region in the coarse-grained candidate region set, and M represents the total number of candidate regions contained in the coarse-grained candidate region set;

[0071] 7.9. Combined with the previous frame image I t-1 The scale and position prior information of the target in the image is obtained from the coarse-grained candidate region set. Filter out fine-grained candidate region sets Among them, b n =(x n ,y n ,w n ,h n ) represents the state of the nth fine-grained candidate region in the fine-grained candidate region set, (x n ,y n ) represents the center position of the nth fine-grained candidate region in the fine-grained candidate region set, (v n ,h n ) represents the width and speed of the nth fine-grained candidate region in the fine-grained candidate region set, and A represents the total number of candidate regions contained in the fine-grained candidate region set;

[0072] 7.10. Calculate the fine-grained candidate region set B fine The state confidence of each candidate region in

[0073]

[0074] Where δ is a constant;

[0075] Calculate the fine-grained candidate region set Bfine The response confidence of each fine-grained proposal region in

[0076]

[0077] in, and Respectively and b n The discrete Fourier transform of

[0078] 7.11 Fusion State Confidence and response confidence Obtain the comprehensive confidence set Q of the fine-grained candidate region:

[0079]

[0080] in, θ is a constant;

[0081] 7.12. Obtain the maximum confidence value Q in the comprehensive confidence set Q max , through the maximum confidence value Q max Index to the optimal candidate region b opt =[x opt ,y opt ,v opt ,h opt ];

[0082] Among them, (x opt ,y opt ) represents the center position of the optimal candidate region, (v opt ,h opt ) represents the width and height of the optimal candidate region;

[0083] 7.13. According to the optimal candidate region b opt =[x opt ,y opt ,v opt ,h opt ], get the current frame image I t The center position of the target (x t ,y t )=(x opt ,y opt ), the target scale remains unchanged, then (v t ,h t )=(v t-1 ,h t-1 ), the target state is S t =[x t ,y t ,v t ,h t].

[0084] Furthermore, in step 1:

[0085] The target feature x0 and the context block feature x i Both include HOG, CN, Gray and deep convolution features.

[0086] Furthermore, step 7.1 is specifically as follows:

[0087] The mixed Gaussian background model is constructed by the neighborhood pixel relationship and regional modeling method. t =αB t-1 +(1-α)B t-1 Update the mixed Gaussian background model to obtain the background model B t .

[0088] Furthermore, step 7.4 is specifically as follows:

[0089] The motion model P is solved by using the RSNSNC method through ORB features.

[0090] Furthermore, in step 3:

[0091] The preset frame interval is 2 frames.

[0092] Furthermore, in step 2.7:

[0093] The maximum preset number of iterations is 5 times.

[0094] Beneficial effects of the present invention:

[0095] 1. The present invention provides a long-term target tracking method based on distortion suppression and low-rank constraint. In the process of constructing the filter (model), the target itself and its surrounding background information are integrated, which effectively alleviates the boundary effect, while increasing the search area and improving the identification ability in fast-moving, occluded or background-cluttered scenes.

[0096] 2. The present invention provides a long-term target tracking method based on distortion suppression and low-rank constraints. By imposing distortion suppression constraints on the change rate of the response graph, it can effectively suppress the unpredictable changes in the target appearance caused by adverse factors such as partial / complete occlusion and illumination changes when accompanied by background noise; at the same time, the introduction of low-rank constraints ensures the stability of the filter (model) and effectively avoids the occurrence of filter (model) drift.

[0097] 3. The present invention provides a long-term target tracking method based on distortion suppression and low-rank constraints, and designs a high-confidence target recapture method based on dynamic target detection. The current frame image is differentiated from the background model, and the differential result is morphologically filtered. Then, multiple constraints are used to finely screen the candidate areas, and those candidate areas with large differences are effectively eliminated; finally, by combining regional state analysis and response map calculation, high-confidence recapture of the target is successfully achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 This is a flow chart of an embodiment of a long-term target tracking method based on distortion suppression and low-rank constraint of the present invention;

[0099] Figure 2 Schematic diagram of the recapture method in an embodiment of the present invention. DETAILED DESCRIPTION

[0100] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0101] The embodiment of the present invention provides a long-term target tracking method based on distortion suppression and low-rank constraint, such as Figure 1 As shown, the following steps are included:

[0102] Step 1: Select multiple sets of continuous multi-frame images in the image sequence whose tracking status is continuously unreliable through intelligent detection or manual precise screening. The previous frame image I of each set of continuous multi-frame images is t-1 The state and properties of are known. For each set of continuous multi-frame images, according to its previous frame image I t-1 The target state S t-1 =[x t-1 ,y t-1 ,v t-1 ,h t-1 ]Extract the current frame image I t The target feature x0 and its context block feature x i ; Current frame image I t is the image I close to the previous frame in the corresponding continuous multi-frame image t-1 images;

[0103] Among them, i∈[1,k], k is a positive integer, t is the current frame time, (x t-1 ,y t-1 ) represents the previous frame image I t-1The center position of the target, (v t-1 ,h t-1 ) represents the previous frame image I t-1 The width and height of the target in ; where the target feature x0 and the context block feature x i Both include HOG, CN, Gray and deep convolution features.

[0104] Step 2: Based on the target feature x0 and the context block feature x i , construct the current frame image I t The filter w t ;

[0105] 2.1. Target feature x0 and context block feature x extracted in step 1 i , explicitly fuse the target and its surrounding background information to construct the first objective function:

[0106]

[0107] Among them, X 0,t and X i,t Represent the target feature x0 and the context block feature x respectively i The circulant matrix, λ1, λ2 are constants, y is an ideal two-dimensional Gaussian response;

[0108] 2.2. Considering the impermanent interference in actual tracking scenarios, multiple unfavorable factors overlap, such as the target being partially / completely blocked by other objects and non-rigid deformation, which causes the response map in the detection phase to vigorous jitter, and the fusion of excessive error information causes the response map to be distorted, which eventually leads to the filter w t Drift or even tracking failure. For this reason, this embodiment constructs the filter w t It not only integrates the contextual information of the target and its surroundings, but also suppresses potential distortion to obtain a more stable filter w t Therefore, this embodiment introduces the distortion suppression constraint into the first objective function The second objective function is obtained:

[0109]

[0110] in, Represents the circular shift operation, p and q represent the previous frame image I t-1 and the current frame image I t The difference in peak displacement on the x-axis and the peak displacement on the y-axis, R t-1 Represents the previous frame image I t-1 Response plot of λ AS is the distortion suppression penalty parameter;

[0111] 2.3. To further enhance the filter w t The discrimination power of this embodiment is to construct the filter w t When applying low-rank constraints on cross-frame sequences, we can emphasize temporal smoothness, so that the learned filter w t On the low-dimensional identification manifold, we improve the adaptability under complex conditions such as deformation and partial occlusion. Specifically:

[0112] Apply a low-rank constraint rank(W to the second objective function t )-rank(W t-1 )≤N, to emphasize temporal consistency, the third objective function is obtained:

[0113]

[0114] strank(W t )-rank(W t-1 )≤N

[0115] in, The history filter w is stored in h , h∈[2,t-1], N is a negative integer; rank(.) represents the matrix rank operation; W t and W t-1 is the variable in the low-rank constraint;

[0116] 2.4. Considering that the matrix rank operation rank(.) is an NP-hard problem, fortunately, this type of problem can be replaced by a sufficient condition of low-rank constraint to optimize the third objective function. Therefore, this embodiment uses the sufficient condition of low-rank constraint to replace the low-rank constraint rank(W t )-rank(W t-1 )≤N, to reconstruct the third objective function and obtain the fourth objective function:

[0117]

[0118] Where μ is a constant and μ>0;

[0119] 2.5. In order to improve the filter w t Adaptability to complex environments, in the filter w t During optimization, this embodiment introduces the equality constraint w into the fourth objective function t =g t , we get the augmented Lagrangian function:

[0120]

[0121] Among them, γ is the penalty parameter, s t is the Lagrange multiplier, g tis an auxiliary variable;

[0122] 2.6. Use the alternating direction multiplier method to decompose the augmented Lagrangian function and obtain the fifth objective function:

[0123]

[0124] Among them, the penalty parameter γ follows the rule γ=min(γ max ,ργ) update,γ max represents the maximum value of the penalty parameter γ, ρ represents the iteration step size, and m+1 represents the m+1th iteration;

[0125] 2.7. Iteratively optimize the fifth objective function:

[0126] Subproblems in the fifth objective function optimization:

[0127] Subproblems The objective function can be expressed as:

[0128]

[0129] Due to the sub-problem is a strictly quadratic convex optimization problem, so the subproblem is directly Differentiate and get the filter w t :

[0130]

[0131] Considering the sub-problem The optimal solution involves the inversion of the matrix, and the heavy computational effort makes it difficult to meet the requirements of real-time tracking. However, by using the diagonal properties of the circulant matrix in the frequency domain, the filter w is further simplified. t The matrices X0 and X in i (i∈[1,k]):

[0132]

[0133] The above matrices X0 and X i The simplified result of (i∈[1,k]) is brought into the filter w of the previous step t In the equation, we get the following:

[0134]

[0135] in represents the discrete Fourier transform of signal a, that is The symbol ⊙ represents the dot product operation of the corresponding elements of the vector. It is worth noting that, except for the FFT and IFFT operations, the subproblem All operations in the optimal solution are bit-wise operations, and do not involve matrix multiplication and inversion operations, so it is consistent with actual real-time tracking.

[0136] Subproblems Optimization:

[0137] About the sub-problems The objective function can be expressed as:

[0138]

[0139] Directly on the objective function Taking the derivative we get:

[0140]

[0141] The Lagrange multiplier s and penalty parameter γ can be updated according to the following rules:

[0142]

[0143] γ=min(γ max ,ργ)

[0144] Finally, update the filter w in sequence t , auxiliary variable g t , Lagrange multiplier s t , penalty parameter γ, until the maximum preset number of iterations is reached, and the current frame image I is obtained t The optimal filter w t,opt In this embodiment, the maximum preset number of iterations is 5 times.

[0145] Then, the optimal filter w t,opt Replace the current frame image I t The filter w t ;

[0146] Step 3: Use the sliding average method to update the filter w according to the preset frame interval t , and obtain the weighted average filter

[0147] Among them, α∈[0,1] represents the learning rate;

[0148] In this embodiment, the preset frame interval is 2 frames.

[0149] Step 4: Based on the previous frame image I t-1 The target state S t-1 =[x t-1 ,y t-1 ,v t-1 ,h t-1 ], in the current frame image I tCrop the region of interest (ROI) around the center point of the target and extract the interesting features x of the region of interest (ROI) t,ROI ;

[0150] Step 5: For the feature of interest x t,ROI With weighted average filter Perform convolution operation to obtain the current frame image I t Response graph R t :

[0151]

[0152] in, represents the inverse discrete Fourier transform, and Respectively and x t,ROI The discrete Fourier transform of

[0153] Step 6: Get the recapture flag Re ;

[0154] 6.1. Get the previous frame image I t-1 The weighted average APCE of the historical average peak energy APCE avg,t-1 , and obtain the number of consecutive multi-frame images whose tracking status is continuously unreliable Count;

[0155] 6.2. Calculate the current frame image I t Response graph R t Average peak energy APCE t :

[0156]

[0157] Where W and H represent the response map R t The width and height, R max and R min Response graph R t The maximum and minimum response values ​​in R i,j Represents the response graph R t The response value of row i and column j in ;

[0158] 6.3. Average Peak Energy APCE t and weighted mean APCE avg,t-1 Make comparisons;

[0159] If APCE t ≤σAPCE avg,t-1 , then the frame number Count increases by 1;

[0160] If APCEt >σAPCE avg,t-1 , then the frame number Count is set to 0;

[0161] Among them, σ∈[0,1] represents a constant;

[0162] 6.4. Determine whether the number of frames Count satisfies the condition Count ≥ T, where T represents the minimum value for activating the background difference mechanism;

[0163] If Count≥T, then recapture flag flag Re is 1;

[0164] If Count < T, then recapture flag flag Re is 0.

[0165] Step 7: According to the recapture flag Re Get the current frame image I t target state;

[0166] If the recapture flag is set Re =1, obtain the current frame image I through the recapture method t The target state S t =[x t ,y t ,v t ,h t ];

[0167] Actual tracking applications may generally track minute-level image sequences. The target may be subject to more severe interference factors, and the filter (model) may be contaminated, resulting in cumulative positioning errors and tracking failures. Therefore, it is particularly important for the tracker to configure a re-detection module to handle situations such as long-term occlusion and frequent out-of-field and re-entry into the field of view. Currently, Kalman filtering or target detection strategies are usually used to address the re-detection problem. Although such methods are effective, they are based on global search methods and are prone to catastrophic consequences. Therefore, if Figure 2 As shown, in this embodiment, the candidate region is first determined based on the moving target detection method, and then the pre-built background model is used to inject shape and position constraints. Finally, the target is recaptured based on the response map and spatiotemporal constraints. The details are as follows:

[0168] 7.1、If the recapture flag is set Re = 1, a mixed Gaussian background model is constructed by neighborhood pixel relationship and regional modeling method, and B t =αB t-1 +(1-α)B t-1 Update it to get the background model B t ;

[0169] 7.2 Extracting background model B t and the current frame image I t ORB features;

[0170] 7.3. Constructing background model B t With the current frame image I t The motion model P;

[0171] Considering that actual tracking tasks often face the situation where the target and background move simultaneously, as well as the mobility of the target and acquisition platform, this embodiment assumes that the motion relationship between the two image sequences is an affine transformation. Based on the above analysis, because after the affine transformation, the properties and spatial relative relationships of the straight lines in the image will not change, and the affine transformation matrix can be regarded as a combination of translation, scaling, rotation, reflection and shearing. It can be formulated in two-dimensional space as:

[0172]

[0173] in is the product of rotation, shear and scaling, the matrix Represents a translation transformation.

[0174] 7.4. Use the RSNSNC method to solve the motion model P through ORB features;

[0175] Considering the grayscale difference between the previous and next frames and the complexity of the target's motion pattern in actual tracking scenarios, a feature-based method is used to estimate the platform motion relationship; the details are as follows:

[0176] 7.4.1. Get the current frame image I t and background model B t ;

[0177] 7.4.2. Extract the current frame image I t and background model B t ORB feature point set And construct a motion vector field, where K represents the maximum number of feature point pairs extracted.

[0178] 7.4.3. Filter the motion vector field and calculate the motion model P based on the random sampling consistency algorithm;

[0179] 7.5. Based on the motion model P, the background model B t With the current frame image I t Align to get the aligned background model

[0180] 7.6. Align the background model and the current frame image I t Perform the difference to obtain the coarse-grained binary difference map D corse ;

[0181] 7.7. Coarse-grained binary difference map D corse Perform morphological filtering to obtain a fine-grained binary difference map D fine ;

[0182] 7.8. From the fine-grained binary difference map D fine Extracting coarse-grained candidate regions from Among them, (x n ,y n ) represents the center position of the nth candidate region in the coarse-grained candidate region set, (v n ,h n ) represents the width and height of the nth candidate region in the coarse-grained candidate region set, and M represents the total number of candidate regions contained in the coarse-grained candidate region set;

[0183] 7.9. Combined with the previous frame image I t-1 The scale and position prior information of the target in the image is obtained from the coarse-grained candidate region set. Filter out fine-grained candidate region sets Among them, b n =(x n ,y n ,v n ,h n ) represents the state of the nth fine-grained candidate region in the fine-grained candidate region set, (x n ,y n ) represents the center position of the nth fine-grained candidate region in the fine-grained candidate region set, (v n ,h n ) represents the width and speed of the nth fine-grained candidate region in the fine-grained candidate region set, and A represents the total number of candidate regions contained in the fine-grained candidate region set;

[0184] 7.9.1. According to the previous frame image I t-1 The scale and position prior information of the target in the image is used to obtain a coarse-grained candidate region set. Center position flag set and range flags set in:

[0185]

[0186] α and β are both known parameters;

[0187] 7.9.2. Based on the center position flag and range flag generated in 7.9.1, the comprehensive flag set is obtained according to the following formula:

[0188]

[0189] 7.9.3. Comprehensive flag bit set generated according to 7.9.2 Eliminate the coarse-grained candidate regions and concentrate on the previous frame image I t-1 The candidate regions with large differences in target range, center position and state are selected to select fine-grained candidate region sets.

[0190] 7.10. Calculate the fine-grained candidate region set B fine The state confidence of each candidate region in

[0191]

[0192] Where δ is a constant;

[0193] Calculate the fine-grained candidate region set B fine The response confidence of each fine-grained proposal region in

[0194]

[0195] in, and Respectively and b n The discrete Fourier transform of

[0196] 7.11 Fusion State Confidence and response confidence Obtain the comprehensive confidence set Q of the fine-grained candidate region:

[0197]

[0198] in, θ is a constant;

[0199] 7.12. Obtain the maximum confidence value Q in the comprehensive confidence set Q max , through the maximum confidence value Q max Index to the optimal candidate region b opt =[x opt ,y opt ,v opt ,h opt ];

[0200] Among them, (x opt ,y opt ) represents the center position of the optimal candidate region, (v opt ,hopt ) represents the width and height of the optimal candidate region;

[0201] 7.13. According to the optimal candidate region b opt =[x opt ,y opt ,v opt ,h opt ], get the current frame image I t The center position of the target (x t ,y t )=(x opt ,y opt ), the target scale remains unchanged, then (v t ,h t )=(v t-1 ,h t-1 ), the target state is S t =[x t ,y t ,v t ,h t ].

[0202] If the recapture flag is set Re =0, according to the response graph R t The peak value of the current frame image I is obtained t The target state S t =[x t ,y t ,v t ,h t ];

[0203] Among them, (x t ,y t ) represents the current frame image I t The center position of the target, (v t ,h t ) represents the current frame image I t The width and height of the target;

[0204] Step 8: For each frame image in each group of continuous multi-frame images in the image sequence whose tracking status is continuously unreliable, perform steps 1 to 7 in sequence in order to obtain the target state of each frame image in all continuous multi-frame images, and combine it with the target state of each frame image with a reliable tracking status to complete long-term target tracking.

[0205] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention shall be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A long-term target tracking method based on distortion suppression and low-rank constraint, characterized in that: The following steps are involved: Step 1: Select multiple sets of continuous multi-frame images in the image sequence whose tracking status is continuously unreliable, and the previous frame image I of each set of continuous multi-frame images is t-1 The state and properties of are known. For each set of continuous multi-frame images, according to its previous frame image I t-1 The target state S t-1 =[x t-1 ,y t-1 ,v t-1 ,h t-1 ]Extract the current frame image I t The target feature x0 and its context block feature x i ; The current frame image I t is the image I close to the previous frame in the corresponding continuous multi-frame image t-1 images; Among them, i∈[1,k], k is a positive integer, t is the current frame time, (x t-1 ,y t-1 ) represents the previous frame image I t-1 The center position of the target, (v t-1 ,h t-1 ) represents the previous frame image I t-1 The width and height of the target; Step 2: Based on the target feature x0 and the context block feature x i , construct the current frame image I t The filter w t ; Step 3: Use the sliding average method to update the filter w according to the preset frame interval t , and obtain the weighted average filter Among them, α∈[0,1] represents the learning rate; Step 4: Based on the previous frame image I t-1 The target state S t-1 =[x t-1 ,y t-1 ,v t-1 ,h t-1 ], in the current frame image I t Crop the region of interest (ROI) around the center point of the target and extract the interesting features x of the region of interest (ROI) t,ROI ; Step 5: For the feature of interest x t,ROI With weighted average filter Perform convolution operation to obtain the current frame image I t Response graph R t ; Step 6: Get the recapture flag Re ; Step 7: According to the recapture flag Re Get the current frame image I t target state; If the recapture flag is set Re =1, obtain the current frame image I through the recapture method t The target state S t =[x t ,y t ,v t ,h t ]; If the recapture flag is set Re =0, according to the response graph R t The peak value of the current frame image I is obtained t The target state S t =[x t ,y t ,v t ,h t ]; Among them, (x t ,y t ) represents the current frame image I t The center position of the target, (v t ,h t ) represents the current frame image I t The width and height of the target; The current frame image I is obtained by the recapture method t The target state S t =[x t ,y t ,v t ,h t ], specifically including the following steps: 7.

1. Construct a mixed Gaussian background model and pass B t =αB t-1 +(1-α)B t-1 Update the mixed Gaussian background model to obtain the background model B t ; 7.2 Extracting background model B t and the current frame image I t ORB features; 7.

3. Constructing background model B t With the current frame image I t The motion model P; 7.

4. Calculate the motion model P using ORB features; specifically: 7.4.

1. Get the current frame image I t and background model B t ; 7.4.

2. Extract the current frame image I t and background model B t ORB feature point set And construct a motion vector field, where K represents the maximum number of feature point pairs extracted; 7.4.

3. Filter the motion vector field and calculate the motion model P based on the random sampling consistency algorithm; 7.

5. Based on the motion model P, the background model B t With the current frame image I t Align to get the aligned background model 7.

6. Align the background model and the current frame image I t Perform the difference to obtain the coarse-grained binary difference map D corse ; 7.

7. Coarse-grained binary difference map D corse Perform morphological filtering to obtain a fine-grained binary difference map D fine ; 7.

8. From the fine-grained binary difference map D fine Extracting coarse-grained candidate regions from Among them, (x n ,y n ) represents the center position of the nth candidate region in the coarse-grained candidate region set, (v n ,h n ) represents the width and height of the nth candidate region in the coarse-grained candidate region set, and M represents the total number of candidate regions contained in the coarse-grained candidate region set; 7.

9. Combined with the previous frame image I t-1 The scale and position prior information of the target in the image is obtained from the coarse-grained candidate region set. Filter out fine-grained candidate region sets Among them, b n =(x n ,y n ,v n ,h n ) represents the state of the nth fine-grained candidate region in the fine-grained candidate region set, (x n ,y n ) represents the center position of the nth fine-grained candidate region in the fine-grained candidate region set, (v n ,h n ) represents the width and height of the nth fine-grained candidate region in the fine-grained candidate region set, and A represents the total number of candidate regions contained in the fine-grained candidate region set; 7.

10. Calculate the fine-grained candidate region set B fine The state confidence of each candidate region in : Where δ is a constant; Calculate the fine-grained candidate region set B fine The response confidence of each fine-grained proposal region in : in, and Respectively and b n The discrete Fourier transform of 7.11 Fusion State Confidence and response confidence Obtain the comprehensive confidence set Q of the fine-grained candidate region: in, θ is a constant; 7.

12. Obtain the maximum confidence value Q in the comprehensive confidence set Q max , through the maximum confidence value Q max Index to the optimal candidate region b opt =[x opt ,y opt ,v opt ,h opt ]; Among them, (x opt ,y opt ) represents the center position of the optimal candidate region, (v opt ,h opt ) represents the width and height of the optimal candidate region; 7.

13. According to the optimal candidate region b opt =[x opt ,y opt ,v opt ,h opt ], get the current frame image I t The center position of the target (x t ,y t )=(x opt ,y opt ), the target scale remains unchanged, then (v t ,h t )=(v t-1 ,h t-1 ), the target state is S t =[x t ,y t ,v t ,h t ]; Step 8: For each frame image in each group of continuous multi-frame images in the image sequence whose tracking status is continuously unreliable, perform steps 1 to 7 in sequence in order to obtain the target state of each frame image in all continuous multi-frame images, and combine it with the target state of each frame image with a reliable tracking status to complete long-term target tracking.

2. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 1, characterized in that: Step 2 specifically includes: 2.

1. Target feature x0 and context block feature x extracted in step 1 i , explicitly fuse the target and its surrounding background information to construct the first objective function: Among them, X 0,t and X i,t Represent the target feature x0 and the context block feature x respectively i The circulant matrix, λ1, λ2 are constants, y is an ideal two-dimensional Gaussian response; 2.

2. Introducing distortion suppression constraints into the first objective function The second objective function is obtained: in, Represents the circular shift operation, p and q represent the previous frame image I t-1 and the current frame image I t The difference in peak displacement on the x-axis and the peak displacement on the y-axis, R t-1 Represents the previous frame image I t-1 Response plot of λ AS is the distortion suppression penalty parameter; 2.

3. Apply low-rank constraint rank(W to the second objective function t )-rank(W t-1 )≤N, to emphasize temporal consistency, the third objective function is obtained: s.t.rank(W t )-rank(W t-1 )≤N in, The history filter w is stored in h , h∈[2,t-1], N is a negative integer; rank(.) represents the matrix rank operation; W t-1 is the historical filter at time t-1; 2.

4. Use the sufficient condition of low-rank constraint to replace the low-rank constraint rank of the third objective function (W t )-rank(W t-1 )≤N, to reconstruct the third objective function and obtain the fourth objective function: Where μ is a constant and μ>0; 2.

5. Introducing equality constraints w into the fourth objective function t =g t , we get the augmented Lagrangian function: Among them, γ is the penalty parameter, s t is the Lagrange multiplier, g t is an auxiliary variable; 2.

6. Use the alternating direction multiplier method to decompose the augmented Lagrangian function and obtain the fifth objective function: Among them, the penalty parameter γ follows the rule γ=min(γ max ,ργ) update,γ max represents the maximum value of the penalty parameter γ, ρ represents the iteration step size, and m+1 represents the m+1th iteration; 2.

7. Iteratively optimize the fifth objective function and update the filter w in sequence t , auxiliary variable g t , Lagrange multiplier s t , penalty parameter γ, until the maximum preset number of iterations is reached, and the current frame image I is obtained t The optimal filter w t,opt , with the optimal filter w t,opt Replace the current frame image I t The filter w t .

3. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 2, characterized in that: Step 6 specifically includes: 6.

1. Get the previous frame image I t-1 Weighted average APCE of historical average peak energy APCE avg,t-1 , and obtain the number of consecutive multi-frame images whose tracking status is continuously unreliable Count; 6.

2. Calculate the current frame image I t Response graph R t Average peak energy APCE t : Where W and H represent the response map R t The width and height, R max and R min Response graph R t The maximum and minimum response values ​​in R i,j Represents the response graph R t The response value of row i and column j in ; 6.

3. Average Peak Energy APCE t and weighted mean APCE avg,t-1 Make comparisons; If APCE t ≤σAPCE avg,t-1 , then the frame number Count increases by 1; If APCE t >σAPCE avg,t-1 , then the frame number Count is set to 0; Among them, σ∈[0,1] represents a constant; 6.

4. Determine whether the number of frames Count satisfies the condition Count ≥ T, where T represents the minimum value for activating the background difference mechanism; If Count≥T, then recapture flag flag Re is 1; If Count < T, then recapture flag flag Re is 0.

4. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 3, characterized in that: Step 5 is as follows: For the feature of interest x t,ROI With weighted average filter Perform convolution operation to obtain the current frame image I t Response graph R t : in, represents the inverse discrete Fourier transform, and Respectively and x t,ROI The symbol ⊙ represents the dot product operation of the corresponding elements of the vector.

5. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 4, characterized in that: In step 1: The target feature x0 and the context block feature x i Both include HOG, CN, Gray and deep convolution features.

6. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 5, characterized in that: Step 7.1 is as follows: The mixed Gaussian background model is constructed by the neighborhood pixel relationship and regional modeling method. t =αB t-1 +(1-α)B t-1 Update the mixed Gaussian background model to obtain the background model B t .

7. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 6, characterized in that: In step 3: The preset frame interval is 2 frames.

8. The long-term target tracking method based on distortion suppression and low-rank constraint according to claim 7, characterized in that: In step 2.7: The maximum preset number of iterations is 5 times.

Images