Target tracking method based on multi-scale joint feature adaptive fusion

Abstract

The application discloses a target tracking method based on multi-scale joint feature adaptive fusion, a first frame image is inputted, a model is initialized; a tth frame image is inputted, a final candidate frame is obtained, a candidate target region is extracted from the final candidate frame; features of each candidate target region are extracted, HOG features, color histogram features, Haar local features and LBP feature responses of each candidate target region are determined; adaptive fusion parameters are calculated, adaptive response fusion is carried out, and a final response result is obtained; according to a maximum value of the final response result, a final target prediction candidate region is determined, a candidate target region corresponding to a maximum response is returned, a target region scale is determined as a target scale of a current frame; a final target prediction candidate region of the tth frame image is used to construct a basic training sample, the model is trained and updated, and a filter model and a color histogram model of a next frame image are obtained.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically relating to a target tracking method based on multi-scale joint feature adaptive fusion. Background Technology

[0002] In recent decades, the field of computer vision has developed rapidly, and computers have acquired various image processing capabilities, including object detection, object recognition, object tracking, image segmentation, image super-resolution, and image fusion. Among these, object tracking has been widely applied in video surveillance and security, autonomous driving technology, smart homes and the Internet of Things, medical image processing, and many other fields, playing a vital role in each. In-depth research into the principles and methods of object tracking technology is of great significance for promoting technological innovation and application in related fields.

[0003] In practical applications of existing target tracking algorithms, targets often possess various attributes such as fuzziness, deformation, rotation, and low resolution.

[0004] Traditional tracking algorithms have three main problems: First, the feature extraction methods are limited. Gray-scale feature extraction is simple but contains less information, color features only work well when there is a large color difference between the target and the background, and HOG features are not robust to tracking blurred images. Second, feature fusion uses a fixed ratio method and cannot adaptively adjust feature weights according to the scene. Third, due to the existence of boundary effects, the filtering template is prone to learning irrelevant information. Summary of the Invention

[0005] In view of this, the main objective of the present invention is to provide a target tracking method based on multi-scale joint feature adaptive fusion.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] This invention provides a target tracking method based on multi-scale joint feature adaptive fusion, the method comprising:

[0008] Step 1: Input the first frame image and initialize the feature filter, scale filter, and color histogram model;

[0009] Step 2: Input the image of frame t, perform contour scale detection, extract and filter candidate bounding boxes, perform exhaustive scaling on the filtered candidate bounding boxes to obtain the final candidate bounding boxes, and extract candidate target regions from the final candidate bounding boxes.

[0010] Step 3: Extract features for each candidate target region and determine the HOG features, color histogram features, Haar local features, and LBP feature responses for each candidate target region;

[0011] Step 4, calculating adaptive fusion parameters, performing adaptive response fusion, obtaining a final response result; determining a final target prediction candidate region according to a maximum value of the final response result, returning a candidate target region corresponding to the maximum response, and determining a target scale of the candidate target region as a target scale of the current frame;

[0012] Step 5, using the final target prediction candidate region of the tth frame image to construct a basic training sample, training and updating the filter model and the color histogram model, and obtaining a filter model and a color histogram model of a next frame image;

[0013] Step 6, repeating steps 2 to 5 until the tracking ends.

[0014] In the above scheme, the first frame image is input, and a feature filter, a scale filter and a color histogram model are initialized, specifically including:

[0015] The target region of the first frame image has been manually labeled, and a training sample is constructed by cyclically displacing the target region as a basic sample;

[0016] A ridge regression method is adopted to obtain an initialized filter model through the training sample , wherein a represents a basic vector corresponding to the sample, b represents a standard response corresponding to the sample, represents a filter model parameter to be solved, represents a regularization coefficient;

[0017] A kernel function method is used to optimize the filter model , and an optimized solution of the filter model is obtained as ; in the formula, represents a kernel correlation calculation of the sample with itself, and I represents a unit matrix;

[0018] An initialized color histogram model is obtained by taking the target region as a basic sample;

[0019] An initialized scale filter model is obtained by taking the target region as a basic sample.

[0020] In the above scheme, the tth frame image is input, contour scale detection is performed, a candidate bounding box is extracted and screened, the screened candidate bounding box is subjected to exhaustive scale scaling, a final candidate bounding box is obtained, and a candidate target region is extracted from the final candidate bounding box, specifically including:

[0021] Contour extraction is performed on the target region of the (t-1)th frame image. Edge calculation is performed on each pixel to obtain the edge response value of each point. Then, a non-maximum suppression algorithm is used to obtain the contour sparse response map. Each pixel in the contour sparse response map... a All have contour value size l a and contour direction β a Contour value for each point l a Make a judgment. l a If the value is greater than 0.15, the point is considered a contour and is retained.

[0022] Contour clustering is performed on the sparse contour response map, and the contour responses of the 8 connected regions surrounding each pixel in the target region are compared. If the contour directions of two pixels are... β a If the difference is less than 90°, they are determined to be the same type of contour; otherwise, they are not the same type of contour.

[0023] The correlation of the contour clustering results is determined, for the center coordinates are ( x i , y i Contour class e i With the center coordinates as ( x j , y j Contour class e j ,pass Determine the correlation between two contour classes. E c ( e i , e j The ) indicates the correlation between two contour classes. β ij coordinates ( x i , y i ) and coordinates ( x j , y j The angle between ) α c Indicates the relevant parameters, Δ d This represents the shortest distance between two classes;

[0024] Determine each contour class ei weight value , ; S eb Indicates candidate borders b wh Contour Class e i The area of ​​the shared region S e Represents outline class e i The area included;

[0025] According to each contour class e i weight value Determine each contour class e i Candidate bounding boxes b Final confidence level p b ;

[0026] Extract candidate bounding boxes and retain the top 35% of candidate bounding boxes with the highest confidence level;

[0027] The area of ​​the candidate bounding box with a confidence level of the top 35% is compared with the area of ​​the target region determined by the image in frame t-1. Candidate bounding boxes with a target region area greater than 0.6 times and less than 1.4 times that of the image in frame t-1 are retained.

[0028] The extracted candidate target regions are preserved using exhaustive scaling. The center point of each candidate region is marked as (x, y). The target width and height determined in the (t-1)th frame are respectively... w and h Each candidate region is scaled down by a certain size to extract... i Candidate target image regions at various scales.

[0029] In the above scheme, the input number t For each frame image, the gradient histogram (HOG) features of the candidate target region are extracted, including the following specific steps:

[0030] Convert the RGB image of the candidate region to a grayscale image using the following formula;

[0031] Perform Gamma correction on the image after grayscale conversion;

[0032] Using [-1, 0, 1] as the horizontal gradient operator, [-1, 0, 1] T The vertical gradient operator is used to determine the horizontal and vertical gradient vectors of the corrected image.

[0033] The magnitude of the gradient is determined based on the horizontal and vertical image gradient vectors. G ( x , y | with gradient direction ;

[0034] The corrected image is divided into small cell units. The gradient direction within each cell is divided into 9 parts from 0° to 180°. Each 20° direction is represented by a directional region. Z1 to Z9 represent the 9 directional regions respectively. The gradients of the 9 parts are weighted and statistically analyzed to obtain the gradient direction histogram of the cell, which is a feature vector of length 9.

[0035] Several adjacent cells are combined into a block. The feature vectors of all cells in each block are concatenated to obtain the HOG feature of the current block. The gradient histogram within the block is normalized to obtain the normalized gradient magnitude.

[0036] The HOG features from all blocks are integrated to obtain 36-dimensional HOG features.

[0037] In the above scheme, determining the color histogram features of each candidate target region specifically includes:

[0038] According to Bayesian statistics, the likelihood probability that pixel x belongs to the target is:

[0039] ;

[0040] In the formula, O represents the foreground region of the candidate target region, and its surrounding space is the background region S. I represents the target candidate region formed by the foreground region O and the background region S. b This represents the category intervals of the color histogram. b x express b pixels within the range x A combination of RGB values;

[0041] Estimate the likelihood term for the likelihood probability. In the formula, express I middle O within the area b The number of pixels in the classification interval. express I middle S within the area b The number of pixels in the classification interval, where |O| represents the area of ​​the foreground region and |S| represents the area of ​​the background region;

[0042] pixels xThe target likelihood probability is transformed into In the formula, 0.5 represents the estimated response value.

[0043] In the above scheme, determining the Haar local features of each candidate target region specifically includes:

[0044] The local Haar features include edge Haar features, linear Haar features, center Haar features, and diagonal Haar features;

[0045] Constructing an integral graph Among them, the candidate region location ( x , y pixel value at ) ii ( x , y ), is the original image ( x , y Gray values ​​of all pixels in the top left direction i ( x' , y' ) and , s ( x , y () represents the cumulative sum in the row direction, initialized s ( x , -1) = 0, ii ( x , y () represents an integral image, initialized ii (-1, y ) = 0;

[0046] Scan the image line by line and recursively calculate each pixel. x , y The sum of the rows s ( x , y ) and integral image ii ( x , y The values ​​of ) are respectively , ;

[0047] Traversing the candidate target region, when the bottom right corner pixel of the image is scanned, the image integral image is generated. ii ( x , y Construction complete;

[0048] Calculate the Haar feature values ​​within the candidate target region based on the integral image.

[0049] The Haar features include two edge Harr features, one center Harr feature, four linear Harr features and one diagonal Harr feature;

[0050] The edge Harr feature needs to find 6 integral graphs, the linear Harr feature needs to find 8 integral graphs, the center Harr feature needs to find 8 integral graphs, and the diagonal Harr feature needs to find 9 integral graphs; the feature extraction is in the form of a sliding window, and each time the window slides by one pixel unit; for each sliding window, the eight Haar features in the window are calculated.

[0051] In the above scheme, the LBP feature of each candidate target region is determined, specifically including:

[0052] A 3*3 neighborhood of each pixel in the candidate target region is selected, and the gray value of the center pixel is taken as a reference to compare with the gray values of the pixels in the neighborhood; if the value of the neighborhood pixel is greater than or equal to the value of the center pixel, the position is marked as 1, otherwise as 0, and the LBP feature calculation formula is represented as ; in the formula, x c , y c ) represents the center pixel coordinate, p represents the first p pixel in the neighborhood, i c represents the center pixel gray value, i p represents the neighborhood pixel gray value, and s(·) is a sign function, represented as .

[0053] In the above scheme, the adaptive fusion parameter is calculated, the adaptive response fusion is performed, and the final response result is obtained; according to the maximum value of the final response result, the final target prediction candidate region is determined, specifically including:

[0054] The response weight of the four features is evaluated by the average peak correlation energy APCE, ; in the formula, F max represents the maximum value in the response graph, F min represents the minimum value in the response graph, w , h ) represents the response coordinate; step 4 obtains the response of each feature, which is respectively represented as r es1 , r es2 , r es3 , r es4, the APCE of each feature response is calculated respectively as P APCE1 , P APCE2 , P APCE3 , P APCE4 ;

[0055] The mapping function is constructed to make the response APCE ratio float in a reasonable range, and the mapping function is ; wherein, represents the pre-fusion parameter of the response, is a hyperparameter, and by controlling the size of the hyperparameter, the parameter can be controlled in a reasonable range;

[0056] The parameter of each target region image is determined and normalized to obtain the fusion parameter ; ;

[0057] According to the fusion parameter , the final fusion response of each target region four features is ; wherein, r es represents the final response result, the candidate target region corresponding to the maximum response result is returned as the final target candidate region, and the size w i , h i of the target region is determined as the target size of the current frame.

[0058] In the above scheme, the base training sample is constructed, the filter model and the color histogram model are trained and updated to obtain the filter model and the color histogram model of the next frame image; specifically including:

[0059] The final target candidate region is taken as a base sample to construct a training sample,

[0060] The training sample and the target candidate region form a kernel matrix, ; wherein, represents the kernel matrix formed by the training sample and the target candidate region, represents the kernel correlation calculation of the base vector a corresponding to the sample and the sample z to be detected;

[0061] According to the kernel matrix, the target function expression is determined to obtain the prediction response R(z) in the time domain, ; wherein, This represents the kernel matrix formed by the training samples and the target candidate regions. This represents the correlation filter model coefficients of the previous frame image; the target location is... The position corresponding to the maximum value This represents the fundamental vector corresponding to the Nth sample;

[0062] For color histogram model x Correlation filter model coefficients with the previous frame image Update , In the formula, Indicates the learning rate. x t This represents the target sample vector of the current frame. x 1:t Indicates from the beginning to t The target sample vector learned at each step, This represents the filter coefficients of the current frame. This represents the filter coefficients learned from the beginning up to time t.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] (1) To address the problem of single feature extraction methods and the inability to highlight important features during feature fusion, a joint feature extraction method is proposed. By introducing Haar features from the face detection field, Haar local features are proposed, and an adaptive feature fusion method is designed to adaptively fuse HOG features, color histogram features, Haar local features and LBP features to highlight important features and suppress features with low discriminative power, thereby further improving the algorithm's ability to represent target features.

[0065] (2) To address the problem that filter templates are prone to learning irrelevant background information, a spatial constraint model is introduced to reduce the impact of boundary effects. Due to the introduction of the spatial constraint model, the ridge regression least squares method cannot meet the optimization requirements of the algorithm. Therefore, the alternating direction multiplier method is selected for optimization.

[0066] (3) To address the issue of scale changes during tracking, the contour exhaustive scale processing method is used. The contour scale detection method is introduced into the tracking field and combined with the exhaustive scale processing method to improve the algorithm's ability to process multi-scale targets. Attached Figure Description

[0067] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and, together with their descriptions, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0068] Figure 1 is the Bird1 sequence tracking result;

[0069] Figure 2 is the BlurOwl sequence tracking result;

[0070] Figure 3 is the Box sequence tracking result;

[0071] Figure 4 is the Panda sequence tracking result;

[0072] Figure 5 is the DragonBaby sequence tracking result;

[0073] Figure 6 is the Human6 sequence tracking result;

[0074] Figure 7 is the Human7 sequence tracking result;

[0075] Figure 8 is the Ironman sequence tracking result;

[0076] Figure 9 is the precision and success rate curve after data summary of each algorithm;

[0077] Figure 10 is the schematic diagram of 8 Haar feature blocks. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0079] The present application provides a target tracking method based on multi-scale joint feature adaptive fusion, and the implementation steps are as follows:

[0080] Step 1, input the first frame image of the video sequence, initialize the filter model and the color histogram model.

[0081] Step 2, input the first frame image of the video sequence, initialize the filter model and the color histogram model. t ( t >1) frame image, first perform contour scale detection, extract and screen candidate bounding boxes, then perform exhaustive scale zoom to obtain the final candidate bounding boxes, and extract the candidate target region

[0082] Step 3, extract features for each candidate target region obtained in step 2, calculate the HOG feature, color histogram feature, Haar local feature and LBP feature response of each candidate target region;

[0083] Step 4, calculating adaptive fusion parameters, performing adaptive response fusion on the four feature responses obtained in step 4 to obtain a final response result; determining a final target prediction candidate region according to a maximum value of the response result; returning the candidate target region corresponding to the maximum response and determining the target region scale as the target scale of the current frame;

[0084] Step 5, using the final target prediction candidate region of the tth frame image to construct a basic training sample, training and updating the filter model and the color histogram model to obtain the filter model and the color histogram model of the next frame image;

[0085] Step 6, repeating steps (2) to (5) until the tracking ends.

[0086] In step 1, inputting a first frame image of a video sequence, initializing the filter model and the color histogram model, including the following specific steps:

[0087] Step 1.1, the target region of the first frame image has been manually labeled before inputting into the algorithm, and the training sample is constructed by cyclically displacing the target region as a basic sample, and the specific operation method is as follows:

[0088] (1) Let the input image I be of size i × j , denoted as

[0089]

[0090] (2) Construct the permutation matrix P , Q so that the image is quickly translated, and the forms of the matrices P , Q are all

[0091] P , Q The size of i × j

[0092] (3) The matrix P is responsible for up-down translation, and the matrix P is multiplied by I once, I moving down one row, and the matrix Q is responsible for left-right translation, and the matrix Q is multiplied by I once, I moving right one column, and the formula is

[0093]

[0094] In the formula,l with r representing the number of up-down and left-right translations, the basis samples are constructed as a circulant matrix by the above cyclic shift operation, as the training samples.

[0095] Step 1.2, take the ridge regression method, through the training samples obtained in step 1.1, initialize the filter model , the specific operation method is as follows:

[0096] (1) assuming that the training sample matrix obtained in step 1.1 is A , x is an unknown vector, Ax is the model prediction value; assuming b represents the standard response corresponding to the sample, that is, the real observation data. The ideal relationship between the training sample and the standard response is

[0097]

[0098] (2) in the actual tracking process, there is an error between the model prediction value and the observation data, in order to minimize the error square sum, the correction vector b is introduced, and the original equation is converted to solve the minimum value of the correction vector b At this time, the solution of the equation is equivalent to

[0099]

[0100] (3) add a regularization term to the above formula, so that the above formula has a unique analytical solution and prevents overfitting, and the target function of ridge regression is obtained as

[0101]

[0102] In the formula, represents the regression function, a represents the basis vector corresponding to the sample, b represents the standard response corresponding to the sample, represents the filter model parameter to be solved, represents the regularization coefficient. The minimum value is solved and extended to the complex domain, and

[0103]

[0104] In the formula, H represents the conjugate transpose.

[0105] Use Fourier diagonalization to calculate , and get , which is brought into the above formula to get the filter model parameter is

[0106]

[0107] The Fourier transform is applied to both sides of the above equation, and the diagonal matrix property is used to simplify the equation

[0108]

[0109] Step 1.3, the filter model is optimized by using the kernel function method, and the specific operation method is as follows:

[0110] (1) The to-be-solved parameters Transform to the dual space ,

[0111]

[0112] At this time, the result variable of the solution changes from to , represents the mapping weight of the low-dimensional space N th sample in the high-dimensional space, represents the basis vector corresponding to the Nth sample;

[0113] (2) The objective function can be rewritten as

[0114]

[0115] In the formula, represents the inner product of the training sample and in the high-dimensional space. represents the basis vector corresponding to the Nth sample; the regression function becomes

[0116]

[0117] The filter model optimization solution is calculated as

[0118]

[0119] In the formula, represents the kernel correlation calculation of the sample and itself, and I represents the unit matrix.

[0120] Step 1.4, the target area is taken as the basis sample, and the color histogram model is initialized, and the specific operation method is as follows:

[0121] (1) The foreground area of the target area is O , and the surrounding space is the background area S , and the foreground area O and the background area S together constitute the target candidate areaI According to the Bayesian statistical formula, the likelihood probability of the pixel point x belonging to the target is

[0122]

[0123] In the formula, b x represents b the combination of the RGB values of the pixels in the interval. x

[0124] (2) Likelihood item estimation is performed on the likelihood probability, and the estimation method is

[0125]

[0126] In the formula, represents I the number of pixel points in the O classification interval in the region, b represents the number of pixel points in the I classification interval in the region, S | represents the area of the foreground region, b | represents the area of the background region O S (3) According to the likelihood item estimation, the likelihood probability of the pixel point x in the target candidate region belonging to the target can be represented as

[0127]

[0128] In the formula, 0.5 represents the estimated response value, and if the RGB combination of the pixel point

[0129] x b x , the probability of belonging to the target is 0.5. O S

[0130] In step 2, the first t ( t >1) frame image is input, contour scale detection is performed, candidate frames are extracted and screened, exhaustive scale scaling is performed, and finally the candidate target region is extracted. The specific operation method is as follows:

[0131] Step 2.1, according to the contour extraction of the last frame target region. The edge of each pixel point of the input image is calculated to obtain the edge response value of each point, and then the contour sparse response graph is obtained by the non-maximum suppression algorithm. Each pixel point a in the contour sparse response graph has a contour value size l ​​​​​​a and contour direction β a Contour value for each point l a Make a judgment. l a If the value is greater than 0.15, the point is considered a contour and is retained.

[0132] Step 2.2: Perform contour clustering on the contour responses obtained in Step 2.1. Compare the contour responses of the 8 connected regions surrounding each pixel in the target region. If the contour directions of two pixels are... β a If the difference is less than 90°, the contours are considered to be of the same type; otherwise, they are not.

[0133] Step 2.3: Perform correlation determination on the contour classes obtained in Step 2.2. For the center coordinates ( x i , y i Contour class e i With the center coordinates as ( x j , y j Contour class e j The correlation between two contour classes is calculated using the following formula.

[0134]

[0135] E c ( e i , e j The ) indicates the correlation between two contour classes. β ij coordinates ( x i , y i ) and coordinates ( x j , y j The angle between ) α c Indicates the relevant parameters, Δ d This represents the shortest distance between two classes.

[0136] Step 2.4, calculate the contour class weights. Calculate the weights for each contour class according to the following formula. e i weight value

[0137]

[0138] S eb representing the candidate bounding box b wh with the contour class e i the area of the region contained in common, S e representing the contour class e i the area contained

[0139] Step 2.5, calculating the confidence of the candidate bounding box, the specific operation method is as follows:

[0140] (1) according to the following formula, the sum of the contour value of all pixel points in each contour class e i obtained in step 2.2 a m i

[0141]

[0142] (2) according to the following formula, the confidence of the candidate bounding box of each contour class e i b p bb

[0143]

[0144] b w and b h respectively represent the width and height of the candidate bounding box b , the value is set to 1.5.

[0145] (3) the central contour class is suppressed, and the outer contour class is highlighted, and the final confidence of the candidate bounding box of the contour class e i is obtained according to the following formula

[0146]

[0147] In the formula, b in the width and height of b w / 2 and b h / 2 respectively.

[0148] ​​​​Step 2.6, extract the candidate bounding box, keep the top 35% of the bounding box with the confidence size.

[0149] Step 2.7, candidate bounding box screening. Calculate the area of the candidate bounding box region, compare it with the target region area determined in the last frame, and keep the candidate bounding box whose area is greater than 0.6 times and less than 1.4 times the target region area determined in the last frame.

[0150] Step 2.8, use the exhaustive scale to extract the candidate target region. The center point of each candidate box marked in step 2.7 is (x, y), and the width and height of the target determined in the last frame are w and h Scale each candidate region to a certain scale to extract i candidate target image regions. The center of these images is still (x, y), and the width and height are w 1~ w i and h 1~ h i .

[0151] In step 3, features are extracted from each candidate target region obtained in step 2, and HOG features, color histogram features, Haar local features, and LBP feature responses of each candidate target region are calculated. The specific steps include:

[0152] Step 3.1, extract the HOG features of the candidate target region, the specific operation steps are as follows:

[0153] (1) Image graying

[0154] Convert the RGB image of the candidate region to a grayscale image by the following formula

[0155]

[0156] In the formula, I gray , indicates the converted grayscale image, R c , G c , B c , and respectively represent the red, green, and blue components of the original image.

[0157] (2) Gamma correction

[0158] Gamma correction is performed on the grayscale converted image to normalize the pixel value of the image, and the correction formula is

[0159]

[0160] wherein, is the compression factor, generally taking a value of 0.5, x , y represents a pixel point, H g x , y represents a corrected result image

[0161] (3) Calculate image gradient

[0162] Use [-1, 0, 1] as a horizontal gradient operator, and [-1, 0, 1] T as a vertical gradient operator to calculate the image gradient. The image horizontal direction gradient is calculated by the following formula

[0163]

[0164] wherein, G x x , y represents a horizontal direction gradient vector. The image vertical direction gradient is calculated by the following formula

[0165]

[0166] wherein, G y x , y represents a vertical direction gradient vector.

[0167] The amplitude of the gradient G , x | and the gradient direction y are respectively calculated by the following two formulas

[0168]

[0169]

[0170] (4) Construct a gradient direction histogram

[0171] Divide the corrected image into small cell units (also known as Cells), and divide the gradient direction in each unit from 0° to 180° into 9 parts, with each 20° direction represented by a direction area, and Z1 to Z9 representing the 9 direction areas. Perform weighted statistics on the gradients of the 9 parts to obtain the gradient direction histogram of the Cell, i.e., a 9-length feature vector.

[0172] (5) Synthesize Cells into Blocks

[0173] ​​​​Several adjacent cells are merged into a single block. Within each block, the feature vectors of all its cells are concatenated to form the HOG feature of that block. The gradient histogram within the block is then normalized; the normalized gradient magnitude is...

[0174]

[0175] In the formula, g i This represents the initial value of the gradient. This represents a very small constant to prevent the denominator from being zero.

[0176] (6) Integrating HOG features

[0177] The HOG features from all blocks are integrated to obtain multidimensional HOG features.

[0178] Step 3.2: Calculate the color histogram features of the candidate target regions. The specific operation method is as follows:

[0179] (1) Let O be the foreground region of the candidate target region, and S be the surrounding space. The foreground region O and the background region S together form the target candidate region I. According to Bayesian statistics, the likelihood probability that pixel x belongs to the target is:

[0180]

[0181] In the formula, b x express b pixels within the range x A combination of RGB values.

[0182] (2) Estimate the likelihood term for the likelihood probability using the following method:

[0183]

[0184] In the formula, express I middle O within the area b The number of pixels in the classification interval. express I middle S within the area b The number of pixels in the classification interval, where |O| represents the area of ​​the foreground region and |S| represents the area of ​​the background region.

[0185] (3) Pixels x The target likelihood probability is transformed into

[0186]

[0187] In the formula, 0.5 represents the estimated response value, if the pixel point x RGB combination b x ,exist O and S None of them appeared, so the probability that it belongs to the target is 0.5.

[0188] Step 3.3: The integral image method is used to calculate four types of Haar local features of the candidate target region, namely edge Haar features, linear Haar features, center Haar features, and diagonal Haar features. The specific operation steps are as follows:

[0189] (1) Construct an integral graph. The definition formula for the elements in the integral graph is:

[0190]

[0191] The above formula represents: the location of the candidate region ( x , y Pixel value at ) ii ( x , y ), is the original image ( x , y Gray values ​​of all pixels in the top left direction i ( x' , y' The sum of ) . Using s ( x , y () represents the cumulative sum in the row direction, initialized s ( x (, -1) = 0. Using ii ( x , y () represents an integral image, initialized ii (-1, y = 0.

[0192] (2) Scan the image line by line and recursively calculate the pixel count. x , y The sum of the rows s ( x , y ) and integral image ii ( x , y The values ​​of ) are respectively

[0193]

[0194]

[0195] When the candidate target region is traversed, the integral image ii x y is constructed.

[0196] (3) Calculate Haar feature values in the candidate target region according to the integral image. Eight kinds of Haar features are selected, including two edge Harr features, one center Harr feature, four linear Harr features and one diagonal Harr feature.

[0197] The calculation method of the Harr feature is to subtract the sum of the pixel values of the black region from the sum of the pixel values of the white region. Since the integral image is the sum of the gray values of all pixels in the upper left direction of a pixel, the sum of the pixel values in any rectangular region of the image can be calculated through the integral image, and then any Haar feature value can be calculated.

[0198] Six integral images are needed to calculate the edge Harr feature, eight integral images are needed to calculate the linear Harr feature, eight integral images are needed to calculate the center Harr feature, and nine integral images are needed to calculate the diagonal Harr feature. The sliding window form is adopted for feature extraction, and each time the window is slid by one pixel unit. For each sliding window, eight Haar features in the window are calculated

[0199] Step 3.4, calculate the LBP feature of the candidate target region.

[0200] The LBP feature adopts a vector histogram to represent the feature, and the calculation method is as follows: select a 3x3 neighborhood of each pixel in the candidate target region, and compare the gray value of the center pixel with the gray values of the eight pixels in the neighborhood. If the value of the neighborhood pixel is greater than or equal to the value of the center pixel, mark the position as 1, otherwise mark it as 0. The LBP feature calculation formula is represented as

[0201]

[0202] In the formula, ( x c y c ) represents the coordinates of the center pixel, p represents the p th pixel in the neighborhood, i c represents the gray value of the center pixel, i p represents the gray value of the neighborhood pixel, and s(·) is the sign function, which is represented as

[0203]

[0204] ​​​In step 4, the adaptive fusion parameters of the four features obtained in step 3 are calculated, and adaptive response fusion is performed to obtain the final response result of each target region. According to the maximum value of the response result, the final target prediction candidate region is determined, and the specific operation method is as follows:

[0205] Step 4.1, the average peak correlation energy APCE is used to evaluate the response weight of the four features, and the APCE calculation formula is

[0206]

[0207] In the formula, F max The maximum value in the response map is represented by max, F min The minimum value in the response map is represented by min, w , h The response coordinates are represented by (x, y). The response of each feature obtained in step 4 is denoted as r es1 , r es2 , r es3 , r es4 The APCE of each feature response is calculated as P APCE1 , P APCE2 , P APCE3 , P APCE4

[0208] Step 4.2, construct a mapping function to make the APCE of each response proportional within a reasonable range, and the mapping function is

[0209]

[0210] In the formula, represents the pre-fusion parameter of the response, is a hyperparameter, and by controlling the size of the hyperparameter, the parameter can be controlled within a reasonable range.

[0211] Step 4.3, calculate the value of the pre-fusion parameter of the response of each target region image, and normalize it to obtain the fusion parameter The formula is

[0212]

[0213] Step 4.4, according to the fusion parameter The final fusion response of the four features of each target region is

[0214]

[0215] In the formula, r es The final response result is represented, and the candidate target region corresponding to the maximum response result is returned as the final target candidate region. The size of the target region obtained in step 2.8 is determined as the target size of the current frame. w i h i

[0216] In step 5, the basic training sample is constructed, and the filter model and the color histogram model are trained and updated to obtain the filter model and the color histogram model of the next frame image. The specific processing steps are as follows:

[0217] Step 5.1, the final target candidate region obtained in step 4.4 is used as the basic sample to construct the training sample, and the specific steps are as follows.

[0218] (1) Let the input image I be of size i × j , denoted as

[0219]

[0220] (2) Construct the permutation matrix P , Q to complete the fast translation of the image. The forms of the matrices P , Q are all

[0221] P , Q The size of i × j

[0222] (3) The matrix P is responsible for up and down translation. The matrix P is multiplied by I once, I moving down one row. The matrix Q is responsible for left and right translation. The matrix Q is multiplied by I once, I moving right one column. The formula is represented as

[0223]

[0224] In the formula, l and r ​​The number of up-down and left-right translations is represented, and the basic samples are constructed as a cyclic matrix by the above cyclic shift operation, as training samples.

[0225] Step 5.2, the training samples obtained in step 5.1 and the target candidate region constitute a kernel matrix, the calculation method is as follows

[0226]

[0227] In the formula, The kernel matrix constituted by the training samples and the target candidate region, The basic vector corresponding to the sample a And the kernel correlation calculation of the to-be-detected region sample z

[0228] Substitute the above formula into the objective function expression The prediction response R(z) in the time domain form can be obtained, which is represented as

[0229]

[0230] In the formula, The kernel matrix constituted by the training samples and the target candidate region, The correlation filter model coefficient of the previous frame image, and the target position is The position corresponding to the maximum value, The basic vector corresponding to the Nth sample;

[0231] Step 5.3, update the color filter model x And the correlation filter model coefficient of the previous frame image The specific way is

[0232]

[0233]

[0234] In the formula, Learning rate, x t The target sample vector of the current frame, x 1:t The target sample vector learned from the beginning to t The moment, The filter coefficient of the current frame, The filter coefficient learned from the beginning to t moment.

[0235] In step 6, steps (2) to (5) are repeated until the tracking ends.

[0236] ​The effect of the present application is further described below in combination with simulation experiments.

[0237] Figures 1 to 8 The experimental results of the algorithm proposed in the present application and comparative algorithms in various sequences are shown, wherein the algorithm of the present application is represented by JFAFMT (Joint Feature Adaptive Fusion Multiscale Tracking). The selected comparative algorithms are all correlation filtering type tracking algorithms, and the comparative algorithms include CSK, CN2, DSST, IBCCF and Staple.

[0238] The Bird1 sequence result is shown in Figure 1 The target in the sequence has the characteristics of deformation, fast motion and occlusion. At the 15th frame, the CSK and CN2 algorithms have tracking drift, and the bird's wings are taken as the target for learning. Subsequently, the bird flapping its wings causes the two algorithms to fail in tracking. At the 126th frame, heavy fog appears, the target is largely occluded by the fog, the target becomes blurred and the features are not obvious, the DSST, IBCCF and Staple algorithms start to have slight tracking drift. At the 182nd frame, only the algorithm of the present application tracks correctly, and the rest of the algorithms all lose the tracking target.

[0239] The BlurOwl sequence result is shown in Figure 2 The target in the sequence has the characteristics of in-plane rotation, motion blur, fast motion and scale change. At the 48th frame, the target starts to move downward quickly, and the target starts to become blurred, the CN2 algorithm cannot correctly track the target, and the detection box does not follow the target movement. At the 383rd frame, the target starts to shake more violently and is very blurred, all algorithms have a certain degree of drift, but the algorithm of the present application has a smaller degree of drift. At the 438th frame, the shaking of the target is weakened, only the algorithm of the present application accurately tracks the target again, and the rest of the algorithms all stay in place and cannot re-track the target area.

[0240] The Box sequence result is shown in Figure 3 The target in the sequence has the characteristics of occlusion, out-of-plane rotation, background clutter and illumination change. At the 119th frame, the IBCCF algorithm learns too many pop can features, the sample model is polluted, and the target is gradually lost. At the 225th frame, the target moves to the upper left corner area, the left side of the book has similar color features to the target, and the CN2 algorithm loses the target and stays on the left side of the book. At the 484th frame, the target passes behind the caliper with occlusion, the white and yellow color of the target is occluded, and the caliper is black in color, the lower area is also black due to darkening of the light, and except for the algorithm of the present application, the CSK, DSST and Staple algorithms all have a certain degree of drift.

[0241] The Panda sequence result is shown in Figure 4As shown in the figure, the target in the sequence has the characteristics of low resolution, scale change, deformation and in-plane rotation. In frame 143, the panda turns around suddenly when it reaches the right side, and the DSST and IBCCF algorithms cannot adjust in time, leaving only the panda's head in the detection box, and the model is contaminated. In frame 413, the panda moves to the upper left corner and no longer faces the camera sideways, and the CSK algorithm shifts to the left whiteboard area. Since the models of the DSST and IBCCF algorithms are contaminated, they no longer follow the target movement. By frame 594, the panda has returned to the right side, and the Staple algorithm can only track the head, and only the algorithm of the present application still tracks all parts of the target.

[0242] The results of the DragonBaby sequence are shown in Figure 5 The target in the sequence has the characteristics of in-plane rotation, out-of-plane rotation, fast motion and out-of-view. In frame 28, the child in the yellow clothes turns around, and the CSK and DSST algorithms drift and lose the tracking target. In frame 41, the CN2 algorithm has drifted to the yellow clothes area, the child is hit hard by the monster dragon, and the target starts to move quickly backward, and the IBCCF algorithm also partially loses the target. By frame 48, the child raises his hands and kicks the monster with his feet, and the target is partially occluded, at which time only the algorithm of the present application can continue to track the target.

[0243] The results of the Human6 sequence are shown in Figure 6 The target in the sequence has the characteristics of scale change, occlusion, deformation and fast motion. In frame 232, the target is enlarged, and the CSK, CN2 and IBCCF algorithms cannot adapt to the scale change of the target and drift, losing the tracking target. In frame 357, the pedestrian is occluded by a street sign, and the target disappears, and the DSST, Staple and algorithm of the present application begin to learn the features of the street sign. By frame 598, the target moves to other areas, and the DSST algorithm stays on the street sign, and only the algorithm of the present application and the Staple algorithm can accurately track.

[0244] The results of the Human7 sequence are shown in Figure 7 The target in the sequence has the characteristics of illumination change, scale change, motion blur and occlusion. In frame 58, all algorithms can track the target, but the CSK, CN2 and IBCCF cannot fully adapt to the scale change of the target, and the model information is contaminated. In frame 121, the target continues to walk forward and the scale becomes smaller, at which time the CN2, DSST and IBCCF algorithms all lose the target. By frame 250, the target moves to the left side of the stairs, and the CSK algorithm also loses the target, and only the algorithm of the present application and the Staple algorithm can accurately track the target.

[0245] The results of the Ironman sequence are shown in Figure 8The target in the sequence has illumination variation, in-plane rotation, occlusion and background clutter, etc. In frame 26, the overall environment changes from bright to dark, and Iron Man rotates to the right, causing Staple algorithm to drift and track the upper arm instead of the head region. CSK and IBCCF algorithms also drift to different degrees. In frame 44, Iron Man continues to rotate, and the tracked face region turns to the back. CSK, CN2, DSST and IBCCF algorithms all lose the target, and Staple algorithm contains a small piece of the target region. In frame 101, an explosion occurs in the video, and the image suddenly becomes bright. At this time, only the algorithm in this chapter can accurately track the target.

[0246] Figure 9 The accuracy and success rate curves of the proposed algorithm and the comparative algorithms in the batch simulation of OTB100 dataset. Table 1 is the quantitative indicators of the accuracy and success rate obtained by the simulation of the algorithms in this chapter and the comparative algorithms. The first ranking is represented in bold and underlined, and the second ranking is represented in bold.

[0247] Table 1 Performance indicators of data summary of each algorithm

[0248]

[0249] From Table 1, in the overall batch simulation results of OTB100, the CLE and OR curves of the proposed algorithm are located at the top. In the AUC ranking, the algorithm in this chapter ranks first with a score of 0.808, and the IBCCF algorithm ranks second with a score of 0.783. In the precision ranking, the algorithm in this chapter ranks first with a score of 0.598, and the IBCCF algorithm ranks second with a score of 0.594.

[0250] Table 2 is the performance improvement rate of the algorithm in this chapter compared with other algorithms. Compared with CSK, CN2, DSST and Staple algorithms, the algorithm in this chapter has a slight decrease in speed and an increase in accuracy. The AUC is improved by 49.35%, 121.98%, 8.31% and 3.86% respectively, and the precision is improved by 54.92%, 136.36%, 7.55% and 4.55% respectively. Compared with the IBCCF algorithm, the algorithm in this chapter has improvements in all aspects, with an AUC improvement of 3.19%, a precision improvement of 0.67%, and an FPS improvement of 620.22%.

[0251] Table 2 Performance improvement rate of the algorithm in this chapter

[0252]

[0253] In summary, when the target appears various types of changes, the algorithm has higher accuracy compared with other algorithms, and can cope with complex scenes. The above merely describes the preferred embodiments of the present application, but not used to limit the protection scope of the present application.

Claims

1. A target tracking method based on multi-scale joint feature adaptive fusion, characterized in that, The method comprises: Step 1, inputting a first frame of image, initializing a feature filter, a scale filter and a color histogram model; Step 2, input the t-th frame image, perform contour scale detection, extract and screen candidate bounding boxes, perform exhaustive scale zooming on the screened candidate bounding boxes to obtain final candidate bounding boxes, and extract a candidate target region from the final candidate bounding boxes, specifically comprising: performing contour extraction on the target region of the (t-1)-th frame image, performing edge calculation on each pixel point to obtain an edge response value of each point, and then obtaining a contour sparse response graph through a non-maximum suppression algorithm; each pixel point in the contour sparse response graph has a contour value size and a contour direction a l a β a , and the contour value of each point is determined l a l a greater than 0.15 is determined as a contour and is retained.​​​ The profile sparse response map is clustered, and the profile responses of 8-connected domains around each pixel point in the target region are compared. β a If the difference is less than 90°, it is determined that the profiles are in the same class, otherwise, the profiles are not in the same class. The correlation between the two contour classes is determined by x i , y i ) and the contour class with the center coordinates of e i x j , y j ) and the contour class with the center coordinates of e j , E c e i , e j ) represents the correlation between the two contour classes, β ij is the angle between the coordinates of x i , y i ) and the coordinates of x j , y j , α c represents the correlation parameter, Δ d represents the shortest distance between the two classes;​​​ determining a weight value for each contour class e i determining a weight value for each contour class , ; S eb representing the area of the region jointly contained by the candidate bounding box and the contour class b wh representing the area of the region jointly contained by the candidate bounding box and the contour class e i representing the area of the region jointly contained by the candidate bounding box and the contour class S e representing the area of the region jointly contained by the candidate bounding box and the contour class e i representing the area of the region jointly contained by the candidate bounding box and the contour class According to each contour class e i weight value determine each contour class e i candidate bounding box b final confidence p b ; extracting a candidate bounding box, and reserving candidate bounding boxes with a confidence value in a top 35% range; comparing area of the candidate bounding boxes with a confidence value in the top 35% range with target area determined from a (t-1)th frame of image, and reserving candidate bounding boxes with target area greater than 0.6 times and less than 1.4 times the target area of the (t-1)th frame of image; Using the extraction candidate target region reserved by the extreme scale scaling, each candidate frame marks the candidate region center point as (x, y), and the target width and height determined by the t-1th frame image are respectively w and h Scale a certain scale for each candidate region, and extract i candidate target image regions of the scale Step 3, extracting features of each candidate target area, and determining HOG features, color histogram features, Haar local features and LBP features of each candidate target area; Step 4, calculating adaptive fusion parameters, performing adaptive response fusion, and obtaining a final response result; determining a final target prediction candidate area according to a maximum value of the final response result, and returning a candidate target area corresponding to a maximum response, and determining a target scale of the candidate target area as a target scale of a current frame; the calculating adaptive fusion parameters, the performing adaptive response fusion, and the obtaining a final response result; and the determining a final target prediction candidate area according to a maximum value of the final response result specifically comprise: The four characteristic response weights are evaluated by average peak correlation energy APCE, ; wherein, F max represents the maximum value in the response graph, F min represents the minimum value in the response graph, w h represents the response coordinate; step 4 obtains the response of each characteristic, respectively denoted as r es1 , r es2 , r es3 , r es4 The APCE of each characteristic response is calculated as P APCE1 , P APCE2 , P APCE3 , P APCE4 ;​ The mapping function is constructed so that each response APCE proportion is within a reasonable range, and the mapping function is ; wherein, represents a pre-fusion parameter of the response, is a hyper parameter, and by controlling the size of the hyper parameter, the parameter can be controlled within a reasonable range; determining parameters for each target region image and normalizing them to obtain fused parameters for ; According to the fusion parameters The final fusion response of the four features of each target region is determined as ; in the formula, r es The final response result is represented, and the candidate target region corresponding to the maximum response result is returned as the final target candidate region. The scale of the target region is determined as the target scale of the current frame w i , h i ​ Step 5, using the final target prediction candidate area of the tth frame of image to construct a basic training sample, training and updating the filter model and the color histogram model, and obtaining a filter model and a color histogram model of a next frame of image; Step 6, repeating steps 2 to 5 until tracking is completed. 2.The tracking method based on multi-scale joint feature adaptive fusion according to claim 1, characterized in that, The initializing the feature filter, the scale filter and the color histogram model based on the input of the first frame of image specifically comprises: a target area of the first frame of image has been manually labeled, and a training sample is constructed by cyclically shifting the target area as a basic sample; Taking a ridge regression method, obtaining an initialized filter model through the training sample , wherein represents a basis vector corresponding to the sample, b represents a standard response corresponding to the sample, represents a filter model parameter to be solved, represents a regularization coefficient; Optimizing filter model using kernel method , the optimized solution of filter model is ; in which, represents the kernel correlation calculation of sample and itself, and I represents unit matrix. an initialization color histogram model is obtained based on the target area as the basic sample; an initialization scale filter model is obtained based on the target area as the basic sample. 3.The target tracking method based on multi-scale joint feature adaptive fusion according to claim 1, characterized in that, The input first t The gradient histogram HOG features of the candidate target region are extracted from the frame image, including the following specific steps: an RGB image of a candidate area is converted into a grayscale image by the following formula: the grayscale-converted image is subjected to Gamma correction; determining horizontal and vertical direction image gradient vectors of the corrected image by [-1, 0, 1] as a horizontal gradient operator, [-1, 0, 1] as a vertical gradient operator T determining horizontal and vertical direction image gradient vectors of the corrected image by [-1, 0, 1] as a horizontal gradient operator, [-1, 0, 1] as a vertical gradient operator determining a magnitude of the gradient from the horizontal and vertical direction gradient vectors G ( x , y )| and the gradient direction ; the corrected image is divided into small cell units Cell, a gradient direction in each unit is divided into 9 parts from 0° to 180°, each 20° direction is represented by a direction area, Z1 to Z9 represent the 9 direction areas respectively, and gradient directions of the 9 parts are weighted and counted to obtain a gradient direction histogram of the Cell, that is, a 9-length feature vector; a plurality of adjacent Cells are combined into a Block, and a feature vector of all Cells in each Block is concatenated to obtain a HOG feature of the current Block; HOG features in all Blocks are integrated to obtain a 36-dimensional HOG feature.

4. The target tracking method based on adaptive fusion of multi-scale joint features according to claim 3, characterized in that, The determining color histogram features of each candidate target area specifically comprises: according to a Bayesian statistical formula, a likelihood probability of a pixel point x belonging to a target is ； In the formula, O is a foreground region of a candidate target region, a surrounding space of O is a background region S, and I is a target candidate region composed of the foreground region O and the background region S, b a classification interval representing a color histogram, b x a classification interval representing a color histogram, b a combination of RGB values of pixels in the interval, x a combination of RGB values of pixels in the interval, likelihood term estimation is performed on the likelihood probability, wherein, represents I in O within the region b the pixel number of the classification interval, represents I in S within the region b the pixel number of the classification interval, |O| represents the area of the foreground region, and |S| represents the area of the background region. The pixel point x is converted into a target likelihood probability ; wherein 0.5 represents an estimated response value.

5. The target tracking method based on adaptive fusion of multi-scale joint features according to claim 4, characterized in that, The determining Haar local features of each candidate target area specifically comprises: The Haar local features include edge Haar features, linear Haar features, center Haar features and diagonal Haar features; Constructing integral image ; wherein the pixel value at the candidate region position x , y ) is the sum of the gray value of all pixels in the direction of the upper left corner of the original image ii x , y ), x y , i x' , y' , s x , y ) represents the cumulative sum in the row direction, initialized s x ,-1) = 0, ii x , y ) represents an integral image, initialized ii -1, y ) = 0;​​​​​ The image is scanned row by row, and the sum of the values of each pixel x , y ) in the row direction is calculated recursively s ( x , y ) and the integral image ii ( x , y ) respectively 、 ; traversing the candidate target region, when the image integral image is scanned to the pixel in the right lower corner of the image ii ( x , y ) is constructed The Haar feature values in the candidate target region are calculated according to the integral graph; The Haar features include two edge Harr features, one center Harr feature, four linear Harr features and one diagonal Harr feature; The edge Harr features need to search the integral graph for six times, the linear Harr features need to search the integral graph for eight times, the center Harr features need to search the integral graph for eight times, and the diagonal Harr features need to search the integral graph for nine times; the feature extraction is in the form of a sliding window, and each time the window slides by one pixel unit; for each sliding window, eight Haar features in the window are calculated.

6. The target tracking method based on adaptive fusion of multi-scale joint features according to claim 5, characterized in that, The LBP features of each candidate target region are determined, and specifically include: For each pixel in the candidate target region, a 3×3 neighborhood is selected. Using the grayscale value of the center pixel as a baseline, the grayscale values ​​of the pixels in its eight surrounding neighborhoods are compared. If the value of a neighboring pixel is greater than or equal to the value of the center pixel, that location is marked as 1; otherwise, it is marked as 0. The LBP feature calculation formula is expressed as follows: In the formula, ( x c , y c ) represents the center pixel coordinates. p The neighborhood number is 1. p 1 pixel, i c Indicates the grayscale value of the center pixel. i p Represents the grayscale value of neighboring pixels, where s(·) is the sign function, expressed as .

7. The target tracking method based on adaptive fusion of multi-scale joint features according to claim 1, characterized in that, The base training sample is constructed, the filter model and the color histogram model are trained and updated, and the filter model and the color histogram model of the next frame of image are obtained; specifically, the following steps are included: The final target candidate region is taken as a base sample to construct a training sample, The training samples and the target candidate regions are used to construct a kernel matrix. In the formula, This represents the kernel matrix formed by the training samples and the target candidate regions. Represents the fundamental vector corresponding to the sample a Samples from the area to be detected z Kernel-related calculations; determining a target function expression according to the kernel matrix , obtaining a prediction response R(z) in time domain, , wherein, the kernel matrix is composed of the training sample and the target candidate region, , wherein, the target position is the position corresponding to the maximum value, , wherein, the basis vector corresponding to the Nth sample is represented by​​ For color histogram model x Correlation filter model coefficients with the previous frame image Update , In the formula, Indicates the learning rate. x t This represents the target sample vector of the current frame. x 1:t Indicates from the beginning to t The target sample vector learned at each step, This represents the filter coefficients of the current frame. This represents the filter coefficients learned from the beginning up to time t.

Images