Video target tracking method and system

Abstract

The present invention discloses a video target tracking method and system, comprising: obtaining a pre-built spatiotemporal correlation filter model combined with second-order features; optimizing and solving the model to obtain a sub-problem f The computational model and sub-problems g Computational model and sub-problems h The computational model is used to extract target features, detect targets and detect scales, and optimize and update each frame in turn, and the target area is calculated and drawn until the target area of ​​the last frame of the video is calculated and drawn; the tracking video and video tracking rate are output. Advantages: It effectively alleviates the filter degradation problem caused by the lack of second-order and high-order features in the spatiotemporal correlation filter, which leads to approximate underfitting and improves the accuracy and robustness of video target tracking. Compared with traditional methods, it makes full use of the second-order feature information to fit the matching degree of the two distribution signals.

Description

Technical Field

[0001] The invention relates to a video target tracking method and system, and belongs to the technical field of image processing. Background Art

[0002] Video object tracking originated in 1955, when Wax first proposed the basic theory of object tracking. After 60 years of rapid development, object tracking technology has become a relatively mature technology today.

[0003] With the continuous improvement of computer performance and the improvement of the cost-effectiveness of video cameras, the demand for automatic video analysis and processing is also growing, which makes the research of target tracking have broad prospects at present. At present, target tracking has been widely used in video surveillance, military strikes, intelligent robot navigation and positioning, human-computer interaction, intelligent transportation and other fields, bringing scientific and technological progress to life, industry, transportation, military and other fields.

[0004] However, video target tracking is also very complex, so it is very difficult to solve all existing problems with one algorithm, and there is still no algorithm that can achieve this. And because there are large differences between different fields of target tracking, any good algorithm cannot be adapted to all fields, and cannot even solve all problems in the field it is adapted to. Therefore, today there are still a large number of researchers engaged in the research of video target tracking, constantly promoting its development.

[0005] Currently, video target tracking has been applied in video surveillance, face recognition, human-computer interaction and other fields. Although there have been many successful research results in video target tracking, and in recent years, regularized correlation filters have shown good performance in video target tracking tasks, it is still difficult to achieve high robustness and accuracy of video target tracking due to environmental factors such as occlusion, in-plane / out-of-plane rotation of the target, fast motion, blur, lighting and deformation. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a video target tracking method and system.

[0007] In order to solve the above technical problems, the present invention provides a video target tracking method, comprising:

[0008] Obtain a pre-built spatiotemporal correlation filter model that incorporates second-order features;

[0009] The model is optimized and solved to obtain the computational model of subproblem f, the computational model of subproblem g, and the computational model of subproblem h, where f is the correlation filter, and g and h are auxiliary variables in the process of solving the correlation filter f;

[0010] Acquire a first frame image in the video data and a target to be tracked in the first frame image, and extract target features in the first frame image;

[0011] Calculate the correlation filter of the first frame image according to the target features extracted from the first frame image;

[0012] Acquire a second frame image in the video data, extract target features of the second frame image, and perform target detection and scale detection on the second frame image according to the target features of the second frame image and the correlation filter calculated from the first frame image;

[0013] According to the target features of the second frame image and the value of the correlation filter calculated in the first frame, the calculation model of sub-problem f, the calculation model of sub-problem g and the calculation model of sub-problem h are used to optimize and iteratively update f, g and h, and the output result is Indicates the new correlation filter after this update is completed;

[0014] Calculate and draw the target area of ​​the second frame according to the results of target detection and scale detection of the second frame image;

[0015] Perform target feature extraction, target detection and scale detection, as well as optimization and update iterations on each frame in turn, and calculate and draw the target area until the target area of ​​the last frame of the video is calculated and drawn;

[0016] After the target area of ​​the last frame is drawn, the tracking video and the video tracking rate FPS are output.

[0017] Furthermore, the spatiotemporal correlation filter model combining the second-order features is:

[0018]

[0019] In the above formula, is a first-order data item, is the second-order data item, is the spatial regularization term, is the temporal regularization term;

[0020] Sample x t represents the result of feature extraction for each frame t, It includes D feature maps of size M*N. Represents sample x t The d-th M*N feature map, d=1,…,D, y represents the Gaussian shape label, w and f are the spatial regularization matrix and the related filter, F represents the F norm, x t 、f t 、f t-1 The matrix dimensions of the real number space of x and y satisfyt ∈R M ×N×D ,f t ∈R M×N×D , f t-1 ∈R M×N×D , y∈R M×N , λ is the penalty coefficient of the second-order feature, f t-1 is the output value of the previous frame, and μ is the penalty coefficient of the temporal regularization term.

[0021] Furthermore, the process of optimizing and solving the model to obtain subproblems f, g and h includes:

[0022] f satisfies f = g, and the augmented Lagrangian form of the second-order spatiotemporal correlation filter model is obtained as follows:

[0023]

[0024] s is the Lagrange multiplier, γ is the step size parameter, and g is the auxiliary variable;

[0025] h Satisfy According to the augmented Lagrangian form of the second-order spatiotemporal correlation filter model, the optimization function is obtained:

[0026]

[0027] The ADMM algorithm is used to divide the optimization function into the following sub-problems:

[0028]

[0029] f (i+1) Indicates the filter after this update is completed, g (i+1) 、h (i+1) Represents the updated filter f (i+1) The auxiliary variables h obtained in the ADMM algorithm solution process (i) It represents the auxiliary variable obtained by solving the last updated correlation filter through the ADMM algorithm;

[0030] Solving these sub-problems gives:

[0031]

[0032] In the above formula, represents the discrete Fourier transform of the filter f, represents the output of the jth channel of the filter f in D channels, T represents the matrix transpose operation, I represents the identity matrix, It represents the result obtained by Parseval's theorem through extracting image features from sample x. express The output of the jth channel obtained by fast Fourier transform, represents the result of Gaussian shape label y obtained by Parseval's theorem and fast Fourier transform, and It represents the output of the jth channel of the auxiliary variables g and h in the updated filter f through Parseval's theorem and fast Fourier transform, and W represents a diagonal matrix of size DMN*DMN formed by concatenating D diagonal matrices Diag(w).

[0033] Furthermore, the process of performing target detection and scale detection includes:

[0034] The grayscale features, color features and directional gradient histogram features of the new frame image are extracted to obtain the target appearance features, and the convolution cross-correlation calculation is performed on the target appearance features and the correlation filter updated in the previous frame. The Gaussian mixture model is processed based on the cross-correlation calculation results to obtain the target detection result;

[0035] The scale detection is completed according to the target appearance features, the correlation filter updated in the previous frame and the scale adaptive tracking method to obtain the scale detection result.

[0036] Furthermore, the convolution cross-correlation calculation process includes:

[0037] The target appearance features are subjected to cosine window processing, and the target appearance features after window processing and the correlation filter updated in the previous frame are subjected to fast discrete Fourier transform. Next, the calculation results are subjected to fast discrete Fourier inverse transform to obtain the calculation results of cross-correlation.

[0038] Furthermore, the process of completing scale detection according to the target appearance feature and the correlation filter updated in the previous frame and the scale adaptive tracking method to obtain the scale detection result includes:

[0039] By learning to independently discriminate the previous frame correlation filter for translation and scale estimation, learning the appearance changes caused by the target scale change, and performing a learned scale filter on the target position, an accurate target scale estimation is obtained.

[0040] A video target tracking system, comprising:

[0041] An acquisition module, used to acquire a pre-built spatiotemporal correlation filter model combined with second-order features;

[0042] An optimization solution module is used to optimize and solve the model to obtain a calculation model of sub-problem f, a calculation model of sub-problem g, and a calculation model of sub-problem h, where f is a correlation filter, and g and h are auxiliary variables in the process of solving the correlation filter f;

[0043] A first extraction module, used to obtain a first frame image in the video data and a target to be tracked in the first frame image, and extract target features in the first frame image;

[0044] A first calculation module, used for calculating a correlation filter of the first frame image according to the target features extracted from the first frame image;

[0045] The target and scale detection module is used to obtain a second frame image in the video data, extract target features of the second frame image, and perform target detection and scale detection on the second frame image according to the target features of the second frame image and the correlation filter calculated from the first frame image;

[0046] The update module is used to optimize and iteratively update f, g, and h based on the target features of the second frame image and the value of the correlation filter calculated in the first frame using the calculation model of subproblem f, the calculation model of subproblem g, and the calculation model of subproblem h, and output the result Indicates the new correlation filter after this update is completed;

[0047] A single-frame drawing module, used for calculating and drawing the target area of ​​the second frame according to the results of target detection and scale detection of the second frame image;

[0048] The overall drawing module is used to perform target feature extraction, target detection and scale detection, as well as optimization and update iterations for each frame in turn, and calculate and draw the target area until the target area of ​​the last frame of the video is calculated and drawn;

[0049] The output module is used to output the tracking video and the video tracking rate FPS after completing the drawing of the target area of ​​the last frame image.

[0050] Furthermore, the target and scale detection module includes:

[0051] The target detection module is used to extract the grayscale features, color features and directional gradient histogram features of a new frame of image to obtain the target appearance features, perform convolution cross-correlation calculation on the target appearance features and the correlation filter updated in the previous frame, and perform Gaussian mixture model processing based on the cross-correlation calculation results to obtain the target detection results;

[0052] The scale detection module is used to complete the scale detection according to the target appearance features and the correlation filter updated in the previous frame and the scale adaptive tracking method to obtain the scale detection result.

[0053] Furthermore, the target detection module includes a convolution cross-correlation calculation module,

[0054] It is used to perform cosine window processing on the target appearance features, perform fast discrete Fourier transform on the target appearance features after window processing and the correlation filter updated in the previous frame, and then perform fast discrete Fourier inverse transform on the calculation results to obtain the cross-correlation calculation results.

[0055] Furthermore, the scale detection module includes a scale adaptive tracking module,

[0056] It is used to perform translation and scale estimation by learning independent discrimination of the previous frame correlation filter, learning the appearance changes caused by target scale changes, and performing learned scale filtering on the target position to obtain accurate target scale estimation.

[0057] The beneficial effects achieved by the present invention are:

[0058] The present invention can effectively alleviate the filter degradation problem caused by the lack of second-order and high-order features in the spatiotemporal correlation filter, which leads to approximate underfitting, thereby improving the accuracy and robustness of video target tracking. Compared with traditional methods, the second-order feature information is fully utilized to fit the matching degree of two distribution signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the method of the present invention;

[0060] Figure 2 A comparison chart of the tracking accuracy of the method of the present invention and the original spatiotemporal correlation filter;

[0061] Figure 3 This is a comparison chart of the quantitative analysis of the success rate of the method of the present invention in the OTB100 data set;

[0062] Figure 4 This is a comparison chart of the accuracy of the method of the present invention in the OTB100 data set;

[0063] Figure 5 It is a comparison chart of the quantitative analysis of the success rate of the method of the present invention in the OTB50 data set;

[0064] Figure 6 This is a comparison chart of the accuracy of the method of the present invention in the OTB50 data set;

[0065] Figure 7 This is a comparison chart of the success rate quantitative analysis of the method of the present invention in the CVPR2013 data set;

[0066] Figure 8 This is a comparison chart of the accuracy quantitative analysis of the method of the present invention in the CVPR2013 data set. DETAILED DESCRIPTION

[0067] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The present invention is achieved by Figure 1 The method shown in the figure realizes the tracking and display of video targets in video images. Its tracking accuracy is compared with the STRCF (Learning Spatial-Temporal RegularizedCorrelation Filters) method based on temporal-temporal correlation filter tracking. Figure 2 The specific steps are as follows:

[0069] Step 1: Construct a spatiotemporal correlation filter (SSCF) model combined with second-order features, and use the ADMM method to optimize and solve the SSCF model;

[0070] 1. Construct a spatiotemporal correlation filter (SSCF) model combining second-order features:

[0071] The training set of multiple images is used Each sample It includes D feature maps of size M*N. y represents the label of Gaussian shape. w and f are the spatial regularization matrix and the related filter. For simplicity, let x∈R M×N×D ,f k ∈R M×N×D , f k-1 ∈R M×N×D , y∈R M×N , λ is the penalty coefficient of the second-order feature, f t-1 is the output value of the previous frame, and μ is the penalty coefficient of the temporal regularization term. Here, we get a simplified form:

[0072]

[0073] In the above formula, is a first-order data item, is the second-order data item, is the spatial regularization term, is the temporal regularization term.

[0074] 2. Solve the SSCF model using the ADMM method:

[0075] (1) Since the model is convex, it can be solved using the ADMM method. We first introduce an auxiliary variable g and a step size parameter γ, and let f = g. The augmented Lagrangian form of the above equation is:

[0076]

[0077] Here, s is the Lagrange multiplier.

[0078] make We can get:

[0079]

[0080] (2) Using the ADMM algorithm to optimize the function, the above equation is divided into the following sub-problems:

[0081]

[0082] (3) ADMM solution results:

[0083] ① For sub-problem f:

[0084] Using Parseval's theorem, we can rewrite the first equation in the above equation using FFT:

[0085]

[0086] In the above formula, represents the discrete Fourier transform (DFT) of filter f. The above equation can be decomposed into M*N sub-problems, and the j-th sub-problem is the j-th element of all D channels in filter f. Let Represents the output of the filter on the jth channel among D channels, then:

[0087]

[0088] It can be obtained that the solution to subproblem f is:

[0089]

[0090] here,

[0091] Rewritten using the Sherman-Morrison formula, the final result is:

[0092]

[0093] ② For sub-problem g:

[0094] The closed form solution of subproblem g is calculated independently:

[0095] g=(W T W+γI)-1 (γf+γh)

[0096] Here, W represents a diagonal matrix of size DMN*DMN formed by concatenating D diagonal matrices Diag(w).

[0097] ③ For sub-problem h:

[0098] From the above formula, we can get h (i+1) =h (i) +f (i+1) -g (i+1) .

[0099] ④ For the step size parameter γ:

[0100] The step size parameter is updated as follows:

[0101] γ (i+1) =min(γ max ,ργ (i) )

[0102] Step 2: Input the first frame image and tracking target, and initialize

[0103] 1. Input image in RGB format

[0104] 2. In the default first frame image, the target position to be tracked is known. The target area is obtained by giving the coordinates (p, q) of the target center point of the first frame image and the length and width (m, n) of the target, and the sub-image block of the target is extracted.

[0105] 3. Extract the features of the target sub-image block. In the correlation filtering tracking algorithm, there are three types of features that can be extracted. A brief introduction is as follows:

[0106] (1) Grayscale features

[0107] Grayscale features extract features through the grayscale of the tracking target sub-image in the image, which can preliminarily realize the tracking of general objects in black and white images.

[0108] (2) Color characteristics

[0109] For color images, pixel-level features such as RGB color features and CN color features are proposed. CN color features are a special type of color feature that uses colors that are closest to human vision. Experiments have shown that CN color space is better than other color features such as RGB features and HSV features.

[0110] (3) Histogram of Oriented Gradients (HOG) feature

[0111] The Histogram of Directed Gradients (HOG) feature is also a common feature in correlation filtering video target tracking. It is a feature formed by calculating and counting the gradient histogram of the local area of ​​the image. The essence of this method is due to the statistical information of the gradient, and the gradient mainly exists at the edge. It happens that target tracking needs to distinguish the edge of the tracked target and the background image.

[0112] 4. After feature extraction, the extracted target features need to be windowed, usually with a cosine window. The purpose of this is to mitigate the boundary effect as much as possible. By multiplying the original pixel of the target with a cosine window, the pixel value close to the edge is close to zero. At the same time, adding a cosine window can also highlight the weight of the target to achieve the effect of suppressing the background. This method can reduce the impact of background noise and improve tracking accuracy.

[0113] Step 3: Input the next frame image for target detection and scale detection

[0114] 1. Object Detection

[0115] (1) Extract features from a new frame of image according to the feature extraction method introduced in step 2

[0116] (2) Calculate the convolution cross-correlation between the target appearance features and the relevant filter. The cross-correlation calculation formula is:

[0117]

[0118] Among them, f * represents the complex conjugate of f.

[0119] (3) The target characteristics with cosine window added and the correlation filter are subjected to discrete Fourier transform (DFT) and calculated by fast discrete Fourier transform (FFT), thereby improving the calculation speed.

[0120] (4) According to the cross-correlation theory, the template and image can be matched through Fourier transform:

[0121] C(u,v)=X(u,v)·H * (u,v)

[0122] Here, X(u,v) represents the Fourier transform of the input image; h(x,y) represents the trained spatial domain template, H(u,v) represents the Fourier transform of the spatial domain template, and H * (u,v) represents the complex conjugate of H(u,v); c(x,y) is used to represent the correlation output, and C(x,y) represents the Fourier transform of the correlation output.

[0123] (5) Since there may be multiple peak positions in the image, it is necessary to use the Gaussian mixture model (GMM) to find the global optimum rather than the local optimum. A mixture model is a probability model that can be used to represent a total distribution containing K sub-distributions. Therefore, the Gaussian mixture model (GMM) can be regarded as a model composed of K single Gaussian models. These K sub-models are the hidden variables of the mixture model. The Gaussian mixture model (GMM) is used because the Gaussian distribution has good mathematical properties and good computational performance. It can find the maximum peak in the response in most cases, thereby avoiding the error of only finding the local optimum.

[0124] Through target detection, the target center position (p, q) is obtained, which represents the result of target detection.

[0125] 2. Scale detection

[0126] Most state-of-the-art methods use exhaustive scale search to estimate the object size, but they are computationally expensive and cannot cope with significant changes. Therefore, we choose a scale-adaptive tracking method that learns independent discriminant correlation filters for translation and scale estimation, learning the appearance changes caused by the object scale changes. Then, the learned scale filter is applied to the object position to obtain an accurate object size estimate.

[0127] Through scale detection, the scale adaptive method is used to obtain the target length and width (m, n), which represents the result of scale detection.

[0128] Step 4: Perform K optimization iterations on the SSCF algorithm and update the relevant filter model using the corresponding update formula

[0129] 1. Input: y, μ, γ0, ρ, K

[0130] 2. Initialization: f(0)=g(0)=h(0)=0; i=0;

[0131] 3. When (i≤K), iterate:

[0132] (1) Through the formula

[0133]

[0134] Update f.

[0135] (2) Through the formula

[0136] g=(W T W+γI) -1 (γf+γh)

[0137] Update g.

[0138] (3) Through the formula

[0139] h (i+1) =h (i) +f (i+1) -g (i+1)

[0140] Update h.

[0141] (4) i = i + 1

[0142] 4. Output results

[0143] Step 5: Calculate and draw a new target area

[0144] 1. Based on the above calculation results, the coordinates of the target center point (p, q) and the target length and width (m, n) of the new frame image are obtained.

[0145] 2. Based on the coordinates of the target center point (p, q) and the target length and width (m, n), draw the target area with a rectangular frame in the new frame image.

[0146] 3. Output the image containing the rectangular target area

[0147] Step 6: If the video has not reached the last frame, return to step 3; if the video has reached the last frame, output the tracking video and video tracking rate FPS, where the video tracking rate FPS is the number of frames per second that complete tracking. Figure 3-Figure 8 The comparative results of tracking success rate and accuracy of our algorithm in different single-target video target tracking datasets are shown.

[0148] The algorithm proposed in the present invention makes full use of the second-order and high-order features, and can better utilize the second-order and high-order features to train samples, rather than using only the first-order features. Its technical effects include: First, in response to the current problem of insufficient fitting of first-order features, we use first-order features combined with second-order features to alleviate and improve the fitting degree of the model; second, in response to the problem of inconsistent orders of magnitude that may be caused by high-order features, we use a penalty coefficient to constrain high-order features so that they can work together with first-order features in model learning; third, in order to ensure the effectiveness of the model, we use regularization methods and add regularization terms to avoid changes in the model during the optimization process, thereby greatly improving the stability of the model; finally, we have demonstrated the effectiveness of our method by conducting experiments on OTB50, OTB100 and CVPR2013 datasets. The experimental results are as follows Figure 3-Figure 8 shown.

[0149] In summary, it can be seen that the spatiotemporal correlation filter video target tracking method combined with second-order features proposed by the present invention has higher accuracy and robustness, and is less complex, simple to implement, and has a faster computing speed.

[0150] The present invention also provides a video target tracking system, comprising:

[0151] An acquisition module, used to acquire a pre-built spatiotemporal correlation filter model combined with second-order features;

[0152] An optimization solution module is used to optimize and solve the model to obtain a calculation model of sub-problem f, a calculation model of sub-problem g, and a calculation model of sub-problem h, where f is a correlation filter, and g and h are auxiliary variables in the process of solving the correlation filter f;

[0153] A first extraction module, used to obtain a first frame image in the video data and a target to be tracked in the first frame image, and extract target features in the first frame image;

[0154] A first calculation module, used for calculating a correlation filter of the first frame image according to the target features extracted from the first frame image;

[0155] The target and scale detection module is used to obtain a second frame image in the video data, extract target features of the second frame image, and perform target detection and scale detection on the second frame image according to the target features of the second frame image and the correlation filter calculated from the first frame image;

[0156] The update module is used to optimize and iteratively update f, g, and h based on the target features of the second frame image and the value of the correlation filter calculated in the first frame using the calculation model of subproblem f, the calculation model of subproblem g, and the calculation model of subproblem h, and output the result Indicates the new correlation filter after this update is completed;

[0157] A single-frame drawing module, used for calculating and drawing the target area of ​​the second frame according to the results of target detection and scale detection of the second frame image;

[0158] The overall drawing module is used to perform target feature extraction, target detection and scale detection, as well as optimization and update iterations for each frame in turn, and calculate and draw the target area until the target area of ​​the last frame of the video is calculated and drawn;

[0159] The output module is used to output the tracking video and the video tracking rate FPS after completing the drawing of the target area of ​​the last frame image.

[0160] The target and scale detection module includes:

[0161] The target detection module is used to extract the grayscale features, color features and directional gradient histogram features of a new frame of image to obtain the target appearance features, perform convolution cross-correlation calculation on the target appearance features and the correlation filter updated in the previous frame, and perform Gaussian mixture model processing based on the cross-correlation calculation results to obtain the target detection results;

[0162] The scale detection module is used to complete the scale detection according to the target appearance features and the correlation filter updated in the previous frame and the scale adaptive tracking method to obtain the scale detection result.

[0163] The target detection module includes a convolution cross-correlation calculation module,

[0164] It is used to perform cosine window processing on the target appearance features, perform fast discrete Fourier transform on the target appearance features after window processing and the correlation filter updated in the previous frame, and then perform fast discrete Fourier inverse transform on the calculation results to obtain the cross-correlation calculation results.

[0165] Furthermore, the scale detection module includes a scale adaptive tracking module,

[0166] It is used to perform translation and scale estimation by learning independent discrimination of the previous frame correlation filter, learning the appearance changes caused by target scale changes, and performing learned scale filtering on the target position to obtain accurate target scale estimation.

[0167] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0168] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video target tracking method, characterized in that: include: Obtain a pre-built spatiotemporal correlation filter model that incorporates second-order features; The model is optimized and solved to obtain the computational model of subproblem f, the computational model of subproblem g, and the computational model of subproblem h, where f is the correlation filter, and g and h are auxiliary variables in the process of solving the correlation filter f; Acquire a first frame image in the video data and a target to be tracked in the first frame image, and extract target features in the first frame image; Calculate the correlation filter of the first frame image according to the target features extracted from the first frame image; Acquire a second frame image in the video data, extract target features of the second frame image, and perform target detection and scale detection on the second frame image according to the target features of the second frame image and the correlation filter calculated from the first frame image; According to the target features of the second frame image and the value of the correlation filter calculated in the first frame, the calculation model of sub-problem f, the calculation model of sub-problem g and the calculation model of sub-problem h are used to optimize and iteratively update f, g and h, and the output result is Indicates the new correlation filter after this update is completed; Calculate and draw the target area of ​​the second frame according to the results of target detection and scale detection of the second frame image; Perform target feature extraction, target detection and scale detection, as well as optimization and update iterations on each frame in turn, and calculate and draw the target area until the target area of ​​the last frame of the video is calculated and drawn; After the target area of ​​the last frame is drawn, the tracking video and video tracking rate FPS are output; The spatiotemporal correlation filter model combining second-order features is: In the above formula, is a first-order data item, is the second-order data item, is the spatial regularization term, is the temporal regularization term; Sample x t represents the result of feature extraction for each frame t, It includes D feature maps of size M*N. Represents sample x t The d-th M*N feature map, d=1,…,D, y represents the Gaussian shape label, w and f are the spatial regularization matrix and the related filter, F represents the F norm, x t 、f t 、f t-1 The matrix dimensions of the real number space of x and y satisfy t ∈R M×N×D ,f t ∈R M×N×D , f t-1 ∈R M×N×D , y∈R M×N , λ is the penalty coefficient of the second-order feature, f t-1 is the output value of the previous frame, and μ is the penalty coefficient of the temporal regularization term.

2. The video target tracking method according to claim 1, characterized in that: The process of optimizing and solving the model to obtain subproblems f, g and h includes: f satisfies f = g, and the augmented Lagrangian form of the second-order spatiotemporal correlation filter model is obtained as follows: s is the Lagrange multiplier, γ is the step size parameter, and g is the auxiliary variable; h Satisfy According to the augmented Lagrangian form of the second-order spatiotemporal correlation filter model, the optimization function is obtained: The ADMM algorithm is used to divide the optimization function into the following sub-problems: f (i+1) Indicates the filter after this update is completed, g (i+1) 、h (i+1) Represents the updated filter f (i+1) The auxiliary variables h obtained in the ADMM algorithm solution process (i) It represents the auxiliary variable obtained by solving the last updated correlation filter through the ADMM algorithm; Solving these sub-problems, the results are: In the above formula, represents the discrete Fourier transform of the filter f, represents the output of the jth channel of the filter f in D channels, T represents the matrix transpose operation, I represents the identity matrix, It represents the result obtained by Parseval's theorem through extracting image features from sample x. express The output of the jth channel obtained by fast Fourier transform, represents the result of Gaussian shape label y obtained by Parseval's theorem and fast Fourier transform, and It represents the output of the jth channel of the auxiliary variables g and h in the updated filter f through Parseval's theorem and fast Fourier transform, and W represents a diagonal matrix of size DMN*DMN formed by concatenating D diagonal matrices Diag(w).

3. The video target tracking method according to claim 1, characterized in that: The process of performing target detection and scale detection includes: The grayscale features, color features and directional gradient histogram features of the new frame image are extracted to obtain the target appearance features, and the convolution cross-correlation calculation is performed on the target appearance features and the correlation filter updated in the previous frame. The Gaussian mixture model is processed based on the cross-correlation calculation results to obtain the target detection result; The scale detection is completed according to the target appearance features, the correlation filter updated in the previous frame and the scale adaptive tracking method to obtain the scale detection result.

4. The video target tracking method according to claim 3, characterized in that: The convolution cross-correlation calculation process includes: The target appearance features are subjected to cosine window processing, and the target appearance features after window processing and the correlation filter updated in the previous frame are subjected to fast discrete Fourier transform. Next, the calculation results are subjected to fast discrete Fourier inverse transform to obtain the calculation results of cross-correlation.

5. The video target tracking method according to claim 3, characterized in that: The process of completing scale detection according to the target appearance features, the correlation filter updated in the previous frame, and the scale adaptive tracking method to obtain the scale detection result includes: By learning to independently discriminate the previous frame correlation filter for translation and scale estimation, learning the appearance changes caused by the target scale change, and performing a learned scale filter on the target position, an accurate target scale estimation is obtained.

6. A video target tracking system, characterized in that: include: An acquisition module, used to acquire a pre-built spatiotemporal correlation filter model combined with second-order features; An optimization solution module is used to optimize and solve the model to obtain a calculation model of sub-problem f, a calculation model of sub-problem g, and a calculation model of sub-problem h, where f is a correlation filter, and g and h are auxiliary variables in the process of solving the correlation filter f; A first extraction module, used to obtain a first frame image in the video data and a target to be tracked in the first frame image, and extract target features in the first frame image; A first calculation module, used for calculating a correlation filter of the first frame image according to the target features extracted from the first frame image; The target and scale detection module is used to obtain a second frame image in the video data, extract target features of the second frame image, and perform target detection and scale detection on the second frame image according to the target features of the second frame image and the correlation filter calculated from the first frame image; The update module is used to optimize and iteratively update f, g, and h based on the target features of the second frame image and the value of the correlation filter calculated in the first frame using the calculation model of subproblem f, the calculation model of subproblem g, and the calculation model of subproblem h, and output the result Indicates the new correlation filter after this update is completed; A single-frame drawing module, used for calculating and drawing the target area of ​​the second frame according to the results of target detection and scale detection of the second frame image; The overall drawing module is used to perform target feature extraction, target detection and scale detection, as well as optimization and update iterations for each frame in turn, and calculate and draw the target area until the target area of ​​the last frame of the video is calculated and drawn; An output module is used to output the tracking video and the video tracking rate FPS after completing the drawing of the target area of ​​the last frame image; The spatiotemporal correlation filter model combining second-order features is: In the above formula, is a first-order data item, is the second-order data item, is the spatial regularization term, is the temporal regularization term; Sample x t represents the result of feature extraction for each frame t, It includes D feature maps of size M*N. Represents sample x t The d-th M*N feature map, d=1,…,D, y represents the Gaussian shape label, w and f are the spatial regularization matrix and the related filter, F represents the F norm, x t 、f t 、f t-1 The matrix dimensions of the real number space of x and y satisfy t ∈R M×N×D ,f t ∈R M×N×D , f t-1 ∈R M×N×D , y∈R M×N , λ is the penalty coefficient of the second-order feature, f t-1 is the output value of the previous frame, and μ is the penalty coefficient of the temporal regularization term.

7. The video target tracking system according to claim 6, characterized in that: The target and scale detection module includes: The target detection module is used to extract the grayscale features, color features and directional gradient histogram features of a new frame of image to obtain the target appearance features, perform convolution cross-correlation calculation on the target appearance features and the correlation filter updated in the previous frame, and perform Gaussian mixture model processing based on the cross-correlation calculation results to obtain the target detection results; The scale detection module is used to complete the scale detection according to the target appearance features and the correlation filter updated in the previous frame and the scale adaptive tracking method to obtain the scale detection result.

8. The video target tracking system according to claim 7, characterized in that: The target detection module includes a convolution cross-correlation calculation module, It is used to perform cosine window processing on the target appearance features, perform fast discrete Fourier transform on the target appearance features after window processing and the correlation filter updated in the previous frame, and then perform fast discrete Fourier inverse transform on the calculation results to obtain the cross-correlation calculation results.

9. The video target tracking system according to claim 7, characterized in that: The scale detection module includes a scale adaptive tracking module, It is used to perform translation and scale estimation by learning independent discrimination of the previous frame correlation filter, learning the appearance changes caused by target scale changes, and performing learned scale filtering on the target position to obtain accurate target scale estimation.

Images