Anti-shielding tracking method based on multi-scale spatial constraint

Abstract

The invention discloses an anti-shielding tracking method based on multi-scale spatial constraint, which comprises the following steps of: extracting HOG (Histogram of Oriented Gradient) features, CN (Content Network) features and gray features of a candidate target area, and reducing dimensions of the HOG features and the CN features by using a PCA (Principal Component Analysis) method to accelerate the operation efficiency; a channel weight fusion method is adopted, a filter is independently trained for each layer of features, the response effect of an effective feature layer is more prominent through self-adaptive fusion weight, and the problem that multi-feature response fusion is insufficient is solved; a step space constraint method is provided, and a color space constraint model, constraint model error covering target information and an airspace constraint model limitation effect are optimized; through an adaptive learning rate and a diffusion search method, irrelevant information learned by a filter is reduced, and the tracking accuracy when a target is shielded is improved; a ternary tree scale acceleration method is provided, a scale filter is introduced, and on the basis of the scale filter, a parallel structure of the scale filter is transformed into a ternary tree classification structure.

Description

Technical Field

[0001] The invention belongs to the technical field of target tracking, and in particular relates to a tracking method based on multi-scale spatial constraint anti-occlusion. Background Art

[0002] In recent decades, the field of computer vision has developed rapidly, and computers have acquired various image processing capabilities, including target detection, target recognition, target tracking, image segmentation, image super-resolution, and image fusion. At the same time, target tracking has been widely used in many fields such as video surveillance and security, autonomous driving technology, smart home and Internet of Things, and medical image processing, playing an important role in various fields. In-depth research on the principles and methods of target tracking technology is of great significance to promoting technological innovation and application in related fields.

[0003] Traditional correlation filter-based target tracking methods usually use feature fusion methods to solve the problem of a single feature extraction method. However, this method only fuses different types of features at the response level and does not consider a single feature layer. In terms of boundary effects, targets are not always square. When the target information is spatially restricted, irrelevant information will still be trained as positive samples, affecting the filter model. When the target is occluded, the information in the target area will not contain the target information, and the learning of the filter will have a negative effect. In terms of multi-scale, existing methods always have to calculate a large number of candidate area responses, and the processing speed is slow. Summary of the invention

[0004] In view of this, the main purpose of the present invention is to provide a tracking method based on multi-scale spatial constraints and anti-occlusion.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a tracking method based on multi-scale spatial constraint anti-occlusion, the method comprising:

[0007] Step 1, input the first frame image, and initialize the feature filter, scale filter and color histogram model;

[0008] Step 2, input the t-th frame image, first extract the gradient histogram (HOG) features of the candidate target area, and use the PCA method to reduce the dimension of the HOG features to extract the fused gradient direction histogram (FHOG) features of the candidate target area; then extract the color name (CN) features and grayscale features of the 5 layers of the candidate target area, t>1;

[0009] Step 3, determine the response result of each channel in the gradient histogram (HOG) feature, fused gradient direction histogram (FHOG) feature, color name (CN) feature and grayscale feature of the t-th frame image, and determine the second channel weight; determine the first channel weight according to the response result of each single channel in the t-1-th frame image; obtain the final single channel weight according to the first channel weight and the second channel weight, perform response fusion through the single channel weight, and determine the final feature response result of the t-th frame image;

[0010] Step 4: determine the target prediction position of the t-th frame image according to the maximum value of the final feature response result, and determine the target scale size of the t-th frame image using the ternary tree scale acceleration method according to the target prediction position;

[0011] Step 5: extract basic samples through the foreground and background of the target prediction position in the current frame and related information, and process the samples through the step space constraint model.

[0012] Step 6, determining an adaptive learning rate according to the second weight of the current frame channel obtained in step 3, and determining a search range of the t+1th frame image by using the learning rate;

[0013] Step 7, training and updating the scale filter model, the correlation filter model, and the color histogram model;

[0014] Step 8: Repeat steps 2 to 7 until the tracking is completed.

[0015] In the above scheme, the first frame image is input to initialize the feature filter, the scale filter and the color histogram model, specifically including:

[0016] The target area of ​​the first frame image has been manually marked. The target area is used as the basic sample and a training sample is constructed by cyclic shifting.

[0017] Using the ridge regression method, the initialization filter model ω is obtained through the training samples. Where a represents the basis vector corresponding to the sample, b represents the standard response corresponding to the sample, ω represents the filter model parameter to be solved, and λ represents the regularization coefficient;

[0018] The kernel function method is used to optimize the filter model ω, and the optimized solution of the filter model is obtained as In the formula, Represents the calculation of the correlation between the sample and its own kernel, and I represents the unit matrix;

[0019] Obtaining an initialized color histogram model using the target area as a basic sample;

[0020] An initialized scale filter model is obtained using the target area as a basic sample.

[0021] In the above scheme, the input t-th frame image first extracts the gradient histogram (HOG) features of the candidate target area, including the following specific steps:

[0022] The RGB image of the candidate area is converted into a grayscale image by the following formula;

[0023] Perform Gamma correction on the grayscale converted image;

[0024] By using [-1,0,1] as the horizontal gradient operator, [-1,0,1] T As a vertical gradient operator, determine the horizontal image gradient vector and the vertical image gradient vector of the corrected image;

[0025] Determine the gradient amplitude |G(x,y)| and the gradient direction θ(x,y) according to the horizontal image gradient vector and the vertical gradient vector;

[0026] The gamma-corrected image is divided into small cell units, and the gradient direction in each cell is divided into 9 parts from 0° to 180°. Each 20° direction is represented by a direction area, and Z1 to Z9 represent 9 direction areas respectively. The gradients of the 9 parts are weighted statistically to obtain the gradient direction histogram of the cell, that is, the 9-length feature vector;

[0027] Several adjacent cells are synthesized and merged into a block. In each block, the feature vectors of all cells in it are connected in series to form the HOG feature of the current block. The gradient histogram in the block is normalized to obtain the normalized gradient amplitude.

[0028] The HOG features in all blocks are integrated to obtain 36-dimensional HOG features.

[0029] In the above scheme, the HOG feature is reduced in dimension by the PCA method to extract the fused directional gradient histogram (FHOG) feature of the candidate target area, specifically including:

[0030] Divide every 20° in 360° into the same category, and determine the direction-insensitive feature F1 and direction-sensitive feature F2 corresponding to each pixel point (x, y);

[0031] Define the pixel-level feature map V and determine the gradient histogram of each pixel;

[0032] Determine the feature vector of each Cell according to the size of the image;

[0033] Determine the gradient energy of each Cell according to the feature vector of each Cell;

[0034] The feature vector is normalized by the gradient energy E of each Cell to obtain direction-sensitive features with a size of W×H and a depth of 18, direction-insensitive features with a size of W×H and a depth of 9, and texture features with a size of W×H and a depth of 4, finally forming a FHOG feature with a size of W×H and a depth of 31.

[0035] In the above scheme, the step of extracting the color name (CN) features of the 5 layers of the candidate target area specifically includes:

[0036] Convert the candidate target area image from RGB space to color space;

[0037] The candidate target area image converted to the color space by the PCA method is subjected to feature dimensionality reduction;

[0038] The eigenvector matrix corresponding to the five largest eigenvalues ​​in the candidate target area image after dimensionality reduction is extracted to form a 5-dimensional CN feature.

[0039] In the above solution, the step of re-extracting the grayscale features of the candidate target area specifically includes:

[0040] Convert the candidate target area RGB image into a grayscale image;

[0041] The converted grayscale image is normalized to control the pixel value to [-0.5, 0.5] to obtain the final grayscale feature.

[0042] In the above scheme, the first channel weight ω is determined according to the response result of each single channel in the t-1th frame image. d1 , specifically including:

[0043] Channel first weight ω d1 The maximum peak value of the response corresponding to the t-1th frame image detection of the feature channel In the formula, f d Indicates the features of the previous frame, h d Represented as the filter coefficients of the t-1th frame.

[0044] In the above scheme, the response result of each channel in the gradient histogram (HOG) feature, fused gradient direction histogram (FHOG) feature, color name (CN) feature and grayscale feature of the t-th frame image is determined, and the second channel weight ω is determined d2 , specifically including: obtaining the characteristic response map of the tth frame, processing the response map through a non-maximum suppression algorithm with a window size of 3×3, and obtaining the value of the first peak ρ d,max1 and the value of the second peak ρ d,max2 ,according to Determine the second channel weight ω d2 .

[0045] In the above scheme, the final single channel weight is obtained according to the first channel weight and the second channel weight, and the response fusion is performed through the single channel weight to determine the final response result, which specifically includes: according to Determine the single channel weight ω d ; In the formula, i represents the channel number is the i-th, and n represents the total number of feature channels;

[0046] according to Determine the final response result r es .

[0047] In the above scheme, the target prediction position is determined according to the maximum value of the final response result, and the target scale size of the t-th frame image is determined using the ternary tree scale acceleration method according to the target prediction position, specifically including

[0048] The final response result r es The coordinates (x, y) corresponding to the maximum value point are the predicted position of the target in the current frame;

[0049] According to the predicted target position, the target scale of the current frame is determined using the ternary tree scale acceleration method.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) To address the problem of insufficient fusion of multi-feature responses, a channel weight fusion method is used to train filters for each layer of features separately. Through adaptive fusion weights, the response effect of the effective feature layer is made more prominent. In addition, the HOG features, CN features, and grayscale features of the candidate region are extracted, and the PCA method is used to reduce the dimensionality of the HOG features and CN features to speed up the calculation efficiency.

[0052] (2) In order to solve the problem that the spatial constraint model has limited constraint effect, a step space constraint method is proposed. It is optimized based on the color space constraint model. It can not only effectively retain the target area, but also prevent the constraint model from erroneously covering the target information when the color features are not obvious.

[0053] (3) To address the problem that when the target is occluded, the filter learns a large amount of irrelevant information and the target is easily lost, an anti-occlusion strategy is designed. Through adaptive learning rate and diffusion search method, the irrelevant information learned by the filter is reduced and the tracking accuracy when the target is occluded is improved.

[0054] (4) In order to solve the problem of excessive computational complexity in existing multi-scale methods, a ternary tree scale acceleration method is proposed. The scale filter is introduced into the algorithm of this chapter, and based on the scale filter, the parallel structure of the scale filter is transformed into a ternary tree classification structure to improve the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to disclose a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0056] Figure 1 This is the Bird1 sequence tracking result.

[0057] Figure 2 It is the BlurOwl sequence tracking result.

[0058] Figure 3 It is the Box sequence tracking result.

[0059] Figure 4 It is the Panda sequence tracking result.

[0060] Figure 5 This is the DragonBaby sequence tracking result.

[0061] Figure 6 It is the Human6 sequence tracking result.

[0062] Figure 7 It is the Human7 sequence tracking result.

[0063] Figure 8 It is the Ironman sequence tracking result.

[0064] Fig. 9 It is the accuracy and success rate curve after summarizing the data of each algorithm. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0066] The present invention provides a tracking method based on multi-scale spatial constraint anti-occlusion, and the implementation steps are as follows:

[0067] Step 1: Input the first frame image and initialize the feature filter, scale filter and color histogram model.

[0068] Step 2: Input the t(t>1)th frame image, first extract the gradient histogram (HOG) features of the candidate target area, and then extract the fused gradient direction histogram (FHOG) features of the candidate target area through PCA method; then extract the color name (CN) features and grayscale features of the 5 layers of the candidate target area.

[0069] Step 3, calculate the response results of each channel of all features separately, obtain the first channel weight according to the response results of each single channel in the previous frame image; obtain the second channel weight according to the response results of each single channel in the current image; obtain the final single channel weight according to the first channel weight and the second channel weight, perform response fusion through the single channel weight, and obtain the final feature response result of the tth frame image.

[0070] Step 4: Determine the target prediction position according to the maximum value of the final feature response result, and use the ternary tree scale acceleration method to determine the target scale of the current frame according to the target prediction position.

[0071] Step 5: extract basic samples through the foreground and background of the target prediction position in the current frame and related information, and process the samples through the step space constraint model.

[0072] Step 6: determine an adaptive learning rate according to the second weight of the current frame channel obtained in step 3, and determine the search range of the t+1th frame image by using the learning rate.

[0073] Step 7: Train and update the scale filter model, correlation filter model, and color histogram model.

[0074] Step 8: Repeat steps 2 to 7 until the tracking is completed.

[0075] In step 1, the feature filter, scale filter and color histogram model are initialized. The specific steps are as follows:

[0076] Step 1.1: The target area of ​​the first frame image has been manually marked. The target area is used as the basic sample and a training sample is constructed by cyclic shift. The specific operation method is as follows:

[0077] (1) Assume that the size of the input image I is i×j, expressed as

[0078]

[0079] (2) Construct permutation matrices P and Q to enable the image to be quickly translated. The matrices P and Q are in the form of

[0080]

[0081] The sizes of P and Q are both i×j

[0082] (3) Matrix P is responsible for up and down translation. Matrix P is multiplied by I once on the left, and I is shifted down one row. Matrix Q is responsible for left and right translation. Matrix Q is multiplied by I once on the right, and I is shifted right one column. The formula is expressed as

[0083] {P l IQ r|l∈[1,i],r∈[1,j]}

[0084] In the formula, l and r represent the number of up and down translation and left and right translation. The basic sample is constructed into a circulant matrix by the above cyclic shift operation as a training sample.

[0085] Step 1.2, using the ridge regression method, initialize the filter model ω through the training samples obtained in step 1.1. The specific operation method is as follows:

[0086] (1) Let the training sample matrix obtained in step 1.1 be A, x be the unknown vector, and Ax be the model prediction value; let b be the standard response corresponding to the sample, that is, the real observation data. The ideal relationship between the training sample and the standard response is

[0087] Ax=b

[0088] (2) In the actual tracking process, there is an error between the model prediction value and the observed data. In order to minimize the sum of squared errors, a correction vector Δb is introduced, and the solution of the original equation is converted to the minimum value of the correction vector Δb. At this time, the solution to the equation Ax = b is equivalent to

[0089]

[0090] (3) Adding a regular term to the above equation makes it have a unique analytical solution and prevents overfitting. The objective function of ridge regression is:

[0091]

[0092] In the formula, f(·) represents the regression function, a represents the basis vector corresponding to the sample, b represents the standard response corresponding to the sample, ω represents the filter model parameter to be solved, and λ represents the regularization coefficient. Taking the minimum value and expanding it to the complex domain, we can get

[0093] w=(A H A+λI) -1 A H b

[0094] Where H represents the conjugate transpose.

[0095] Use Fourier diagonalization to H A is calculated and we get Substituting into the above formula, the filter model parameter ω is

[0096]

[0097] Perform Fourier transform on both sides of the above equation and simplify it using the diagonal matrix property to obtain

[0098]

[0099] Step 1.3, use the kernel function method to optimize the filter model. The specific operation method is as follows:

[0100] (1) The parameter to be solved ω is transformed into the dual space α (α={α1,α2,...,α N}),have to

[0101]

[0102] At this time, the result variable of the solution changes from ω to α, α N represents the mapping weight of the Nth sample in the low-dimensional space in the high-dimensional space, a N Represents the basis vector corresponding to the sample

[0103] (2) The objective function can be rewritten from the above formula as

[0104]

[0105] In the formula, K(z,a N ) represents the training samples in high-dimensional space and The regression function becomes

[0106]

[0107] The calculated filter model optimization solution is

[0108]

[0109] In the formula, Represents the calculation of the kernel correlation between the sample and itself, and I represents the identity matrix.

[0110] Step 1.4, using the target area as the basic sample, initialize the color histogram model. The specific operation method is as follows:

[0111] (1) The foreground area of ​​the target area is O, and the surrounding space is the background area S. The foreground area O and the background area S together constitute the target candidate area I. According to the Bayesian statistical formula, the likelihood probability of pixel point x belonging to the target is

[0112]

[0113] In the formula, b represents the classification interval of the color histogram, b x Represents the combination of RGB values ​​of pixel x in interval b.

[0114] (2) Estimation of likelihood term for likelihood probability is performed using the following method:

[0115]

[0116] In the formula, Indicates the number of pixels in the b classification interval in the O region in I, represents the number of pixels in the b classification interval in the S region of I, |O| represents the area of ​​the foreground area, and |S| represents the area of ​​the background area

[0117] (3) According to the likelihood estimation, the likelihood probability that the pixel point x in the target candidate area belongs to the target can be expressed as

[0118]

[0119] Where 0.5 represents the estimated response value. If the RGB combination of pixel x is x , which does not appear in O and S, has a probability of 0.5.

[0120] Step 1.5, using the target area as the basic sample, initialize the scale filter model. The specific operation method is as follows:

[0121] Constructing the scale pyramid C s , as a scale filter, used to calculate the target scale, scale pyramid C s The formula is expressed as

[0122]

[0123] Where N c Represents the number of scales in the scale pyramid, N c The value is 33, σ c represents the scale factor, σ c The value is 1.02.

[0124] In step 2, the t(t>1)th frame image is input, the gradient histogram (HOG) feature of the image is first extracted, and the fused gradient direction histogram (FHOG) feature of the image is extracted by dimensionality reduction using the PCA method; then the color name (CN) feature and grayscale feature of the 5 layers of the image are extracted. The specific steps are as follows:

[0125] Step 2.1, extract the HOG features of the candidate target area. The specific operation method is as follows:

[0126] (1) Image grayscale

[0127] The RGB image of the candidate area is converted into a grayscale image by the following formula

[0128] I gray =0.3×R c +0.59×G c +0.11×B c

[0129] In the formula, I gray Represents the converted grayscale image, R c , G c , B c Represent the red, green and blue components of the original image respectively.

[0130] (2) Gamma Correction

[0131] Gamma correction is performed on the grayscale converted image to normalize the pixel values ​​of the image. The correction formula is:

[0132]

[0133] In the formula, γ g represents the compression factor, which is generally set to 0.5. (x, y) represents the pixel point. H g (x,y) represents the correction result image

[0134] (3) Calculate image gradient

[0135] Use [-1,0,1] as the horizontal gradient operator, [-1,0,1] T As a vertical gradient operator, the image gradient is calculated. The horizontal gradient of the image is calculated by the following formula

[0136] G x (x,y)=H g(x+1,y) -H g(x-1,y)

[0137] In the formula, G x (x, y) represents the horizontal gradient vector. The vertical gradient of the image is calculated by the following formula

[0138] G y (x,y)=H(x,y+1)-H(x,y-1)

[0139] In the formula, G y (x,y) represents the vertical gradient vector.

[0140] The gradient magnitude |G(x,y)| and the gradient direction θ(x,y) are calculated by the following two equations respectively:

[0141]

[0142] (4) Constructing gradient direction histogram

[0143] The gamma-corrected image is divided into small cell units (also called cells), and the gradient direction in each cell is divided into 9 parts from 0° to 180°. Each 20° direction is represented by a direction area, and Z1 to Z9 represent 9 direction areas respectively. The gradients of the 9 parts are weighted and counted to obtain the gradient direction histogram of the cell, that is, the 9-length feature vector.

[0144] (5) Synthesize Cell into Block

[0145] Several adjacent cells are combined into a block. In each block, the feature vectors of all cells in it are concatenated to form the HOG feature of the current block. The gradient histogram in the block is normalized. The normalized gradient amplitude is

[0146]

[0147] In the formula, g i represents the initial value of the gradient, and ε represents a very small constant to prevent the denominator from being zero.

[0148] (6) Integrate HOG features

[0149] The HOG features in all blocks are integrated to obtain 36-dimensional HOG features.

[0150] Step 2.2: Perform PAC dimensionality reduction on the HOG features obtained in step 2.1 to obtain FHOG features. The specific operation method is as follows:

[0151] (1) Considering the directional gradient, every 20° in 360° is divided into the same category, and the direction-insensitive feature F1 and direction-sensitive feature F2 corresponding to each pixel point (x, y) are calculated. The calculation formula is:

[0152]

[0153] In the formula, p represents the number of classes, and its value is 18.

[0154] (2) Define the pixel-level feature map V and calculate the gradient histogram of each pixel. The mapping formula is:

[0155]

[0156] In the formula, b={1,2,…,p}.

[0157] (3) Aggregate the features V of all pixels

[0158] Assuming the image size is W×H, the feature vector of each Cell is

[0159]

[0160] In the formula, S hc Indicates the size of the cell

[0161] The gradient energy of each Cell is calculated according to the following formula

[0162]

[0163] In the formula, α and β belong to {-1,1}, and ε represents a very small number but not zero.

[0164] (4) Synthesize FHOG features

[0165] The feature vector is normalized by the gradient energy E of each Cell to obtain direction-sensitive features with a size of W×H and a depth of 18, direction-insensitive features with a size of W×H and a depth of 9, and texture features with a size of W×H and a depth of 4, finally forming a FHOG feature with a size of W×H and a depth of 31.

[0166] Step 2.3, extract the color name features of the candidate target area. The specific operation method is as follows:

[0167] (1) Convert the candidate target area image from RGB space to color space

[0168] Let the transformation matrix be L, expressed as

[0169]

[0170] In the formula, i=11, the image is converted from RGB space to color space through matrix L, and the calculation process is expressed as

[0171] P cn =(R,G,B)L

[0172] Where P cn is the feature matrix, which represents the 11-dimensional color feature after conversion. The i-th color feature p i Expressed as

[0173] p i =m 1i R+m 2i G+m 3i B

[0174] (2) Feature dimensionality reduction using PCA

[0175] Feature Matrix P cn The covariance matrix P cov for

[0176]

[0177] P is calculated by the following formula cov The eigenvalue λ and eigenvector x

[0178] (P cov -λE)x=0

[0179] Where E represents the unit matrix.

[0180] (3) Extract the eigenvector matrix corresponding to the five largest eigenvalues ​​to form a 5-dimensional CN feature.

[0181] Step 2.4, extract the grayscale features of the candidate target area. The specific operation method is as follows:

[0182] (1) Convert the RGB image of the candidate target area into a grayscale image. The conversion formula is:

[0183] I gray =0.3×R+0.59×G+0.11×B

[0184] In the formula, I gray Represents the converted image, R represents the size of the red component, G represents the size of the green component, and B represents the size of the blue component.

[0185] (2) Normalize the image and control the pixel value to [-0.5, 0.5], which is expressed as

[0186]

[0187] In the formula, F gray Represents the final grayscale feature.

[0188] In step 3, the response results are calculated separately for each channel of all features of the previous frame image and the current image. According to the response results of each single channel of the previous frame image, the first weight of the channel is obtained; according to the response results of each single channel in the current image, the second weight of the channel is obtained; according to the first weight of the channel and the second weight of the channel, the final single channel weight is obtained; the response fusion is performed through the single channel weight to obtain the final response result. The following specific steps are included:

[0189] Step 3.1, calculate the first weight ω of each feature channel of the previous frame image d1 , the specific operation method is as follows:

[0190] Channel first weight ω d1 The maximum peak value of the response corresponding to the previous frame detection of the feature channel is calculated using the following formula:

[0191]

[0192] In the formula, f d Indicates the features of the previous frame, hd Represents the filter coefficient of the previous frame. When the first frame is initialized, ω d1 Set the value to 1.

[0193] Step 3.2, calculate the second weight ω of each feature channel of the current frame image d2 , the specific operation method is as follows:

[0194] Obtain the characteristic response map of the current frame, and process the response map through the non-maximum suppression algorithm with a window size of 3×3 to obtain the value of the first peak ρ d,max1 and the value of the second peak ρ d,max2 , set the second channel weight ω d2 =1-ρ d,max2 / ρ d,max1 , in order to prevent the value from being too small, the threshold is set to 0.5. Therefore, the second channel weight ω d2 It can be expressed as

[0195]

[0196] Step 3.3, calculate the single channel weight ω according to the first channel weight and the second channel weight obtained in steps 3.1 and 3.2 d , the specific operation method is as follows:

[0197] ω d can be calculated using the following formula

[0198]

[0199] In the formula, i represents the channel number is the i-th, and n represents the total number of feature channels.

[0200] Step 3.4, calculate the current frame feature fusion response r es , the specific operation method is as follows:

[0201] The feature fusion response of the current frame is

[0202]

[0203] In the formula, r es Indicates the final response of the current frame.

[0204] In step 4, the target prediction position is determined according to the maximum value of the final response result; according to the target prediction position, the target scale size of the current frame is determined using the ternary tree scale acceleration method. The specific steps are as follows:

[0205] Step 4.1, determine the target predicted position, the specific operation method is as follows:

[0206] The fused feature response r obtained in step 3 esThe coordinates (x, y) corresponding to the maximum value point are the predicted position of the target in the current frame.

[0207] Step 4.2, according to the target predicted position, use the ternary tree scale acceleration method to determine the target scale of the current frame. The specific operation method is as follows:

[0208] (1) Constructing a scale filter

[0209] After determining the target position of the current frame, construct the scale pyramid C s Construct multiple candidate target scales and calculate the target scale, scale pyramid C s The formula is expressed as

[0210]

[0211] Where N c Represents the number of scales in the scale pyramid, N c The value is 33, σ c represents the scale factor, σ c The value is 1.02. Assuming the target size is W×H, the i-th target scale constructed by the scale pyramid is

[0212] (2) Using the ternary tree classification structure to obtain the target scale

[0213] Using the ternary tree classification structure, N c The value is 33, and the first level calculates the characteristic responses R7, R 17 , R 27 , take n1 corresponding to the maximum response value, and calculate the characteristic response R when n is n1-4 and n1+4 at the second level n1-4 , R n1+4 , and compare these two responses with R n1 By comparison, we take n2 corresponding to the maximum response value, and calculate the response R corresponding to n2-2, n2-1, n2+1, and n2+2 at the third level. n2-2 , R n2-1 , R n2+1 , R n2+2 , these four responses are compared with R n2 By comparison, we take n3 corresponding to the maximum response value. When n=n3, the corresponding image block scale is the target scale of the frame.

[0214] In step 5, basic samples are extracted, and a step space constraint model is constructed through the foreground and background of the target prediction results and related information. The specific steps are as follows:

[0215] Step 5.1: construct training samples based on the target candidate area as the basic sample. The specific operation method is as follows:

[0216] (1) Assume that the size of the target candidate region image I is i×j, expressed as

[0217]

[0218] (2) Construct permutation matrices P and Q to enable the image to be quickly translated. The matrices P and Q are in the form of

[0219]

[0220] The sizes of P and Q are both i×j

[0221] (3) Construct training samples according to the following formula

[0222] {P l IQ r |l∈[1,i],r∈[1,j]}

[0223] In the formula, l and r represent the number of up and down translation and left and right translation. The basic sample is constructed into a circulant matrix by the above cyclic shift operation as a training sample.

[0224] Step 5.2, calculate the color space constraint model, the specific operation method is as follows:

[0225] Assume the color space constraint model is m c , which can be expressed as

[0226] m c,i =p(y i )

[0227] In the formula, i represents the i-th pixel in the training area, m c,i Represents the model value corresponding to point i. The color information of the i-th pixel in the training area is represented as r i , the coordinate information is represented by x i , we can get the joint probability p(y i ) is defined as

[0228]

[0229] In the formula, p(r i |m ci =j) represents the appearance possibility, p(x i |m ci =j) represents the spatial possibility, p(m ci =j) represents the prior probability of foreground and background.

[0230] Appearance probability p(r i |m ci=j) is calculated by the Bayesian criterion of the target foreground and background color model. For a given foreground area O, its surrounding space is the background area S. The foreground area O and the background area S together constitute the target candidate area I. According to the Bayesian statistical formula, the appearance possibility can be calculated as follows

[0231]

[0232] In the formula, b represents the classification interval of the color histogram, b x Represents the combination of RGB values ​​of pixel x in interval b.

[0233] Spatial possibility p(x i |m ci =j) means the position reliability of each pixel, and the calculation method is

[0234]

[0235] In the formula, r represents the distance from the pixel to the center pixel, σ i Indicates the length of the diagonal of the corresponding bounding box. After solving the result, it is limited to [0.5, 0.9]. When the value is greater than 0.9, it is taken as 0.9, and when the value is less than 0.5, it is taken as 0.5.

[0236] The prior probability of foreground and background p(m ci =j) is the ratio between the foreground and background histogram extraction area sizes, and the calculation method is:

[0237]

[0238] In the formula, S f Indicates the size of the foreground area, S b Represents the size of the background area, and the final color space constraint model m can be obtained c

[0239] Step 5.3, calculate the step space constraint model m l , the specific operation method is as follows:

[0240] Construct a Hanning window of the same size as the model. The discrete coefficient formula of the Hanning window is

[0241]

[0242] In the formula, the number of discrete points is N+1, and two one-dimensional Hanning window vectors can be determined by the model width and height, expressed as

[0243] h anW =[h anW,0 ,h anW,1 ,…,h anW,n ,…,hanW,w ]n=0,1,2,…,w

[0244] h anH =[h anH,0 ,h anH,1 ,…,h anH,n ,…,h anH,h ]n=0,1,2,…,h

[0245] The two-dimensional Hanning window h can be obtained an2 for

[0246]

[0247] Afterwards, the color space constraint model m c With the two-dimensional Hanning window h an2 We perform weighted fusion and linear stretching to limit the value of each point in the model to the interval [0,1], thus obtaining the step space constraint model m l , the formula is

[0248]

[0249] In the formula, i represents the i-th pixel in the model area, j represents the number of pixels in the model area, and m i Represents the value of the i-th pixel in the step space constraint model.

[0250] In step 6, the adaptive learning rate is calculated according to the response of each channel of the current frame, and the candidate target area of ​​the next frame image is determined by the diffusion search method through the learning rate. The specific steps are as follows.

[0251] Step 6.1, calculate the learning rate parameter of the current frame feature response. The specific operation method is as follows:

[0252] (1) Use mean APCE as the APCE learning rate parameter ρ d,max1 Assuming that the current image is the tth frame, the APCE learning rate parameter can be expressed as

[0253]

[0254] (2) The second channel weight ρ d,max2 Introduced into the learning rate, making it the channel learning rate parameter β ω , the formula is expressed as

[0255]

[0256] (3) Multiplying the two learning rate parameters, we can get the final learning rate parameter β:

[0257] β=β APCEβ ω

[0258] Assuming that the basic learning rate set by the algorithm is a, the final adaptive learning rate a β for

[0259] a β =a·β

[0260] Train the filter separately for each feature channel. Assuming there are n channels, we will get n adaptive learning rates, which are a β1 ,a β2 ,...,a βn .

[0261] Step 6.2: Use diffusion search to determine the candidate target area of ​​the next frame image. The specific operation method is as follows:

[0262] Assume that the size of the original detection area S is w×h, and set the diffusion judgment threshold λ s is 0.3, when the adaptive learning rate a β Less than the diffusion judgment threshold λ s When, according to the adaptive learning rate a β The size of the detection area is adjusted by the size of the expansion coefficient α s It can be expressed as

[0263]

[0264] When the adaptive learning rate a β When it is less than 0.3, the detection area size of the next frame is (αw)×(αh). When the adaptive learning rate a in the process of continuous n-frame detection is β When it is less than 0.3, the size of the detection area of ​​the nth frame can be expressed as

[0265]

[0266] In the formula, w s and h s Indicates the width and height of the expanded detection area, which is the size of the updated detection area for the next frame; if the adaptive learning rate a of the next frame β If it is greater than 0.3, the detection area size of the next frame will be restored to the original size.

[0267] In step 7, the scale filter model, the correlation filter model, and the color histogram model are trained and updated. The specific processing steps are as follows:

[0268] Step 7.1, construct training samples using the next frame target candidate area obtained in step 6 as the basic sample. The specific steps are as follows.

[0269] (1) Assume that the size of the target candidate region image I is i×j, expressed as

[0270]

[0271] (2) Construct permutation matrices P and Q to enable the image to be quickly translated. The matrices P and Q are in the form of

[0272]

[0273] The sizes of P and Q are both i×j

[0274] (3) Construct training samples according to the following formula

[0275] {P l IQ r |l∈[1,i],r∈[1,j]}

[0276] In the formula, l and r represent the number of up and down translation and left and right translation. The basic sample is constructed into a circulant matrix by the above cyclic shift operation as a training sample.

[0277] Step 7.2: The training samples obtained in step 7.1 and the target candidate regions form a kernel matrix. The calculation method is as follows:

[0278] K z =C(k az )

[0279] In the formula, K z represents the kernel matrix composed of training samples and target candidate regions, k az Represents the kernel correlation calculation between the training sample a and the sample z in the area to be detected.

[0280] Substitute the above formula into the objective function expression The predicted response R(z) in time domain can be obtained, which is expressed as

[0281] R(z)=K z α

[0282] In the formula, K z represents the kernel matrix composed of training samples and target candidate regions, α represents the correlation filter model coefficient of the previous frame image, and the target position is the position corresponding to the maximum value of R(z).

[0283] Step 7.3, update the color filter model x, the scale filter model C, and the coefficient α of the correlation filter. The specific method is as follows:

[0284] x 1:t =(1-η)x 1:t-1 +ηx t

[0285] α1:t =(1-η)α 1:t-1 +ηα t

[0286] C 1:t =(1-η)C 1:t-1 +ηC t

[0287] In the formula, η represents the learning rate, which is set to 0.02. t represents the target sample vector of the current frame, x1:t represents the target sample vector learned from the beginning to time t; α t represents the correlation filter coefficient of the current frame, α1:t represents the correlation filter coefficient learned from the beginning to time t; C t Represents the scale filter coefficient of the current frame, and C1:t represents the scale filter coefficient learned from the beginning to time t.

[0288] In step 8, steps 2 to 7 are repeated until the tracking is completed.

[0289] The effects of the present invention are further described below in conjunction with simulation experiments.

[0290] Figures 1 to 8 The experimental results of the algorithm proposed in this paper and the comparison algorithm in each sequence are shown. The algorithm in this chapter is represented by SCAOMT (Spatial Constraints Against Occlusion Multiscale Tracking). The selected comparison algorithms are all correlation filter tracking algorithms, including KCF, SAMF, SRDCF, BACF and LCT.

[0291] The Human9 sequence results are as follows Figure 1 As shown in the figure, the targets in this sequence have the characteristics of illumination change, scale change, deformation and rapid movement. In the 80th frame, the target moves forward and its scale becomes smaller. Since the target frame of the previous LCT algorithm is large in scale and more background information is learned, the LCT tracking drifts. In the 140th frame, although the remaining 5 algorithms can track the target, the KCF, SAMF and BACF algorithms cannot adapt to the change of the target scale, and the model information will also be polluted to a certain extent. By the 395th frame, the target is further reduced, and only the algorithm in this chapter can continue to maintain tracking accuracy, while the other algorithms have a certain degree of drift.

[0292] The Skiing sequence results are as follows Figure 2As shown in the figure, the targets in this sequence have low resolution, in-plane rotation, out-of-plane rotation, and deformation. In the 7th frame, the skier occupies very few pixels and rotates in the air. The KCF, LCT, SAMF, and BACF algorithms all drift to a certain extent. In the 11th frame, the skier in red clothes is moving fast in the air, and the distinction between him and the woods behind him is not high. Only the algorithm in this paper and the SRDCF algorithm continue to track him. By the 25th frame, only the algorithm in this chapter continues to track him.

[0293] MotorRlling sequence results are as follows Figure 3 As shown in the figure, the targets in this sequence have characteristics such as background clutter, fast motion, in-plane rotation and motion blur. In the 10th frame, the motorcycle and the driver continue to move upward. The other algorithms all regard the lights as part of the target. Since the algorithm in this chapter uses a step space constraint model, it is more effective to learn the target area, so the tracking is more accurate. In the 26th frame, the motorcycle and the driver fall down, and all algorithms track them again. In the 35th frame, the motorcycle and the driver continue to fall. Since the other comparison algorithms have learned a lot of wrong information before, the correct information of a few frames cannot change the wrong models of these algorithms. Therefore, only the algorithm in this chapter can track them accurately.

[0294] Matrix sequence results are as follows Figure 4 As shown, the targets in this sequence have characteristics such as illumination changes, background clutter, rapid motion, and in-plane rotation. In the 47th frame, the scene becomes brighter and the person's head moves forward. The SRDCF algorithm does not adapt to the rapid changes in features and drifts to the person's legs. The other algorithms also drift to a certain extent. In the 53rd frame, the person leans back and the face becomes brighter. The KCF, LCT, SAMF, and BACF algorithms all fail to track the person. The tracking target of the algorithm in this chapter also shifts from the task head to the ear. By the 88th frame, the algorithm in this chapter continues to track the person's ear. Although it fails to adapt to the scale change of the target, it still maintains a certain accuracy.

[0295] The Lemming sequence results are as follows Figure 5 As shown in the figure, the targets in this sequence have characteristics such as occlusion, out of view, scale change, and out-of-plane rotation. In frame 370, the target emerges from the state of being occluded by the lighter, and the SRDCF, KCF, LCT, and BACF algorithms fail to follow the target. In frame 920, the target becomes larger and passes through the target area again. The KCF and LCT algorithms return to normal tracking, but fail to adapt to the target scale. By frame 1003, although the KCF and SAMF algorithms can continue tracking, the scale cannot be adapted. Only the algorithm in this chapter maintains a good level in scale and accuracy.

[0296] KiteSurf sequence results are as follows Figure 6As shown in the figure, the target in this sequence has the characteristics of illumination change, occlusion, in-plane rotation and out-of-plane rotation. In the 29th frame, the surfer rises rapidly due to being dragged, and the KCF algorithm fails to track due to the small background area during detection. The SRDCF and LCT algorithms drift to a certain extent. In the 34th frame, the target moves to the upper left of the image, and the SAMF and BACF algorithms also regard the TV channel icon information as part of the target. By the 51st frame, the SAMF and BACF algorithms switch to tracking the TV channel icon, and all the remaining algorithms except the algorithms in this chapter fail to track.

[0297] The Human2 sequence results are as follows Figure 7 As shown in the figure, the targets in this sequence have the characteristics of illumination change, scale change and out-of-plane rotation. In the 221st frame, the target walks behind the chair on the right and turns to move to the left. The BACF algorithm tracks the target and changes it to a chair. The other algorithms continue to track it, but the KCF, LCT and SAMF algorithms track it at a scale that is too large. In the 382nd frame, the target moves quickly to the left and is partially blocked. The LCT algorithm loses the target, and the other algorithms continue to track it. By the 703rd frame, the KCF, LCT, SAMF and BACF algorithms all fail to track. Although the SRDCF algorithm continues to track, the scale is obviously inconsistent with the target size. Only the algorithm in this chapter maintains a good level in scale and accuracy.

[0298] The Girl2 sequence results are as follows Figure 8 As shown in the figure, the target in this sequence has characteristics such as occlusion, deformation, motion blur, and scale change. In frame 107, the left side of the target is partially occluded by pedestrians, and the SRDCF and KCF algorithms drift. In frame 113, the target is completely occluded, and the KCF algorithm drifts greatly, while the other algorithms are in a reasonable area. By frame 120, the target reappears. During the period when the target is occluded, the other algorithms continue to learn wrong information. The algorithm in this chapter maintains the model accuracy by controlling the learning rate, so only the algorithm in this chapter can continue tracking, and all other algorithms fail to track.

[0299] Fig. 9 CLE and OR curves of the proposed algorithm and the comparison algorithm in batch simulation on the OTB100 dataset.

[0300] Table 1 shows the performance improvement rate of the algorithm proposed in this invention compared with other algorithms. Compared with KCF, SAMF and BACF algorithms, the algorithm proposed in this invention has a decrease in speed and an increase in accuracy, with AUC increased by 38.51%, 8.35 and 13.83% respectively, and precision increased by 63.64%, 9.76% and 7.88% respectively. Compared with SRDCF and LCT algorithms, all aspects have been improved, with AUC increased by 7.94% and 5.55%, precision increased by 9.45% and 6.96%, and FPS increased by 434.35% and 53.12%.

[0301] Table 2 shows the quantitative indicators of CLE and OR obtained by simulating the algorithm proposed in the present invention and the comparative algorithm. The first one is indicated by bold and underline, and the second one is indicated by bold. In all the batch simulation results of OTB100, the CLE and OR curves of the algorithm proposed in the present invention are at the top. In the AUC ranking, the algorithm proposed in the present invention ranks first with a score of 0.856, and the LCT algorithm ranks second with a score of 0.811. In the precision ranking, the algorithm in this chapter ranks first with a score of 0.630, and the LCT algorithm ranks second with a score of 0.589.

[0302] Table 1 Performance improvement rate of the algorithm proposed in this invention

[0303] SCAOMT SRDCF KCF LCT SAMF BACF AUC - 7.94％ 38.51％ 5.55％ 8.35 13.83％ precision - 9.45％ 63.64％ 6.96％ 9.76％ 7.88％ FPS - 434.35％ -87.21％ 53.12％ -18.94％ -11.89％

[0304] Table 2 Performance indicators of each algorithm after data summary

[0305] SCAOMT SRDCF KCF LCT SAMF BACF AUC <![CDATA[ 0.856 ]]> 0.793 0.618 0.811 0.790 0.752 precision <![CDATA[ 0.630 ]]> 0.573 0.385 0.589 0.574 0.584 FPS 15.71 2.94 <![CDATA[ 122.82 ]]> 10.26 19.38 17.83

[0306] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A tracking method based on multi-scale spatial constraint anti-occlusion, characterized in that: The method includes: Step 1, input the first frame image, and initialize the feature filter, scale filter and color histogram model; Step 2, input the t-th frame image, first extract the gradient histogram (HOG) features of the candidate target area, and use the PCA method to reduce the dimension of the HOG features to extract the fused gradient direction histogram (FHOG) features of the candidate target area; then extract the color name (CN) features and grayscale features of the 5 layers of the candidate target area, t>1; Step 3, determine the response result of each channel in the gradient histogram (HOG) feature, fused gradient direction histogram (FHOG) feature, color name (CN) feature and grayscale feature of the t-th frame image, and determine the second channel weight; determine the first channel weight according to the response result of each single channel in the t-1-th frame image; obtain the final single channel weight according to the first channel weight and the second channel weight, perform response fusion through the single channel weight, and determine the final feature response result of the t-th frame image; Step 4: determine the target prediction position of the t-th frame image according to the maximum value of the final feature response result, and determine the target scale size of the t-th frame image using the ternary tree scale acceleration method according to the target prediction position; Step 5, extract basic samples through the foreground and background of the target prediction position in the current frame and related information, and process the samples through the step space constraint model; Step 6, determining an adaptive learning rate according to the second weight of the current frame channel obtained in step 3, and determining a search range of the t+1th frame image by using the learning rate; Step 7, training and updating the scale filter model, the correlation filter model, and the color histogram model; Step 8: Repeat steps 2 to 7 until the tracking is completed.

2. The tracking method based on multi-scale joint feature adaptive fusion according to claim 1, characterized in that: The first frame of image is input, and the feature filter, the scale filter and the color histogram model are initialized, specifically including: The target area of ​​the first frame image has been manually marked. The target area is used as the basic sample and a training sample is constructed by cyclic shifting. Using the ridge regression method, the initialization filter model ω is obtained through the training samples. Where a represents the basis vector corresponding to the sample, b represents the standard response corresponding to the sample, ω represents the filter model parameter to be solved, and λ represents the regularization coefficient; The kernel function method is used to optimize the filter model ω, and the optimized solution of the filter model is obtained as follows: In the formula, Represents the calculation of the correlation between the sample and its own kernel, and I represents the identity matrix; Obtaining an initialized color histogram model using the target area as a basic sample; An initialized scale filter model is obtained using the target area as a basic sample.

3. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 1 or 2, characterized in that: The input t-th frame image first extracts the gradient histogram (HOG) features of the candidate target area, including the following specific steps: The RGB image of the candidate area is converted into a grayscale image by the following formula; Perform Gamma correction on the grayscale converted image; By using [-1,0,1] as the horizontal gradient operator, [-1,0,1] T As a vertical gradient operator, determine the horizontal image gradient vector and the vertical image gradient vector of the corrected image; Determine the gradient amplitude |G(x,y)| and the gradient direction θ(x,y) according to the horizontal image gradient vector and the vertical gradient vector; The gamma-corrected image is divided into small cell units Cell l, and the gradient direction in each cell is divided into 9 parts from 0° to 180°. Every 20° direction is represented by a direction area. Z1 to Z9 represent 9 direction areas respectively. The gradients of the 9 parts are weighted statistically to obtain the gradient direction histogram of Cell l, that is, the 9-length feature vector; Several adjacent cells are synthesized and merged into a block. In each block, the feature vectors of all cells in it are connected in series to form the HOG feature of the current block. The gradient histogram in the block is normalized to obtain the normalized gradient amplitude. The HOG features in all blocks are integrated to obtain 36-dimensional HOG features.

4. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 3 is characterized in that: The HOG feature is reduced in dimension by the PCA method to extract the fused directional gradient histogram (FHOG) feature of the candidate target area, specifically including: Divide every 20° in 360° into the same category, and determine the direction-insensitive feature F1 and direction-sensitive feature F2 corresponding to each pixel point (x, y); Define the pixel-level feature map V and determine the gradient histogram of each pixel; Determine the feature vector of each Cell according to the size of the image; Determine the gradient energy of each Cell l according to the feature vector of each Cell; The feature vector is normalized by the gradient energy E of each Cell to obtain direction-sensitive features with a size of W×H and a depth of 18, direction-insensitive features with a size of W×H and a depth of 9, and texture features with a size of W×H and a depth of 4, finally forming a FHOG feature with a size of W×H and a depth of 31.

5. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 4 is characterized in that: The step of extracting the color name (CN) features of the five layers of the candidate target area specifically includes: Convert the candidate target area image from RGB space to color space; The candidate target area image converted to color space by PCA method is subjected to feature dimensionality reduction; The eigenvector matrix corresponding to the five largest eigenvalues ​​in the candidate target area image after dimensionality reduction is extracted to form a 5-dimensional CN feature.

6. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 5, characterized in that: The step of re-extracting the grayscale features of the candidate target area specifically includes: Convert the RGB image of the candidate target area into a grayscale image; The converted grayscale image is normalized to control the pixel value to [-0.5, 0.5] to obtain the final grayscale feature.

7. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 6, characterized in that: According to the response result of each single channel in the t-1th frame image, the first channel weight ω is determined d1 , specifically including: Channel first weight ω d1 The maximum peak value of the response corresponding to the t-1th frame image detection of the feature channel In the formula, f d Indicates the features of the previous frame, h d Represented as the filter coefficients of the t-1th frame.

8. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 7, characterized in that: The step of determining the response result of each channel in the gradient histogram (HOG) feature, the fused gradient direction histogram (FHOG) feature, the color name (CN) feature and the grayscale feature of the t-th frame image, and determining the second channel weight ω d2 , specifically including: obtaining the characteristic response map of the tth frame, processing the response map through a non-maximum suppression algorithm with a window size of 3×3, and obtaining the value of the first peak ρ d,max1 and the value of the second peak ρ d,max2 ,according to Determine the second channel weight ω d2 .

9. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 8, characterized in that: The method further comprises: obtaining a final single-channel weight according to the first channel weight and the second channel weight, performing response fusion through the single-channel weight, and determining a final response result, specifically comprising: Determine the single channel weight ω d ; In the formula, i represents the channel number is the i-th, and n represents the total number of feature channels; according to Determine the final response result r es .

10. The tracking method based on multi-scale spatial constraint anti-occlusion according to claim 9, characterized in that: The target prediction position is determined according to the maximum value of the final response result, and the target scale size of the t-th frame image is determined using the ternary tree scale acceleration method according to the target prediction position. Specifically, the final response result r es The coordinates (x, y) corresponding to the maximum value point are the predicted position of the target in the current frame; According to the predicted target position, the target scale of the current frame is determined using the ternary tree scale acceleration method.

Images