Siamese network target tracking method based on deformable convolution and template updating

Abstract

The present invention discloses a twin network target tracking method based on deformable convolution and template update, which combines the deformable convolution network and the template update strategy for the first time. Deformable convolution is used to learn the adaptive offset of feature data in different directions to enhance the feature extraction capability of the backbone network. In addition, although deep features are rich in semantic information, they lack location information, and the positioning accuracy is improved through multi-layer deformable cross-correlation fusion. Finally, a high-confidence template update strategy is proposed, that is, the peak sidelobe ratio and the maximum value of the response map are calculated once every fixed frame, and based on this, a weighted method is used to fuse features to update the template. It not only ensures effective updates, but also prevents the template from being updated in special circumstances such as occlusion. The present invention proposes a new end-to-end deep model, which effectively improves the tracking performance of the scheme.

Description

Technical Field

[0001] The present invention relates to a twin network target tracking method based on deformable convolution and template updating, and belongs to the technical field of target tracking in computer vision. Background Art

[0002] Object tracking is widely used in video analysis, intelligent transportation, virtual reality and other fields. Its core task is to automatically estimate the position of the target in subsequent frames after the initial position of the target in the first frame of the video. However, the tracking process is easily disturbed by a series of complex factors such as occlusion, blur, deformation, and illumination changes. Therefore, it is of great research and practical value to design a tracking method that can run stably and efficiently in complex scenes.

[0003] At present, the existing tracking algorithms can be roughly divided into two categories. One category applies the idea of ​​correlation filtering to the tracking field and proposes an error minimum square filter target tracking algorithm: the solution of the tracker template is converted from complex time domain operations to Fourier domain point multiplication calculations, the amount of calculation is greatly reduced, and the speed of the tracker is greatly improved. Although this algorithm is very fast, the accuracy is not ideal. In order to achieve a balance between accuracy and speed, the twin network tracking algorithm based on deep learning has become increasingly popular. The twin network mainly consists of a template branch and a search branch that share parameters in the feature extraction process. Using the similarity learning method, a fully convolutional twin network tracking algorithm is proposed. By calculating the similarity value between the target template and the deep features of the search area, the tracking problem is converted into a similar matching problem. The algorithm can use a large amount of data for end-to-end offline training, which can obtain high accuracy without online adjustment. By introducing the region proposal network in the detection field into the fully convolutional twin network target tracking algorithm, the method of distinguishing foreground and background and regressing bounding boxes is used to achieve flexible response to the target scale and obtain a more accurate bounding box. Experimental results show that the tracking effect of this combined model is significantly better than the correlation filtering algorithm and the fully convolutional twin network algorithm.

[0004] However, most target tracking algorithms based on Siamese networks have two major shortcomings:

[0005] (1) The recognition ability of the tracking algorithm based on the twin network is largely limited by the weak feature extraction capability of the backbone network in the fully convolutional twin network tracking algorithm.

[0006] (2) No effective online template update strategy is considered. When the target undergoes significant changes during the tracking process, the useful information of the original template will be reduced, and the tracker will drift and cannot be adjusted in time.

[0007] Therefore, in response to the above pain points, the present invention proposes a twin network target tracking method based on deformable convolution and template updating. Summary of the invention

[0008] Purpose: In order to overcome the shortcomings of the prior art, the present invention provides a twin network target tracking method based on deformable convolution and template updating, which can track the target more accurately; moreover, the present invention can effectively solve the problems of weak feature extraction capability of the twin backbone network and model drift caused by non-updating templates during the tracking process, and proposes a new end-to-end deep model, which effectively improves the tracking performance of the solution.

[0009] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A method for twin network target tracking based on deformable convolution and template updating includes the following steps:

[0011] Step S1, crop the first frame image of the data set or the camera capture image with the target as the center to obtain a 127×127×3 image as the template image, and during the tracking process, crop the i+1th frame with the target position of the i-th frame as the center to obtain a 255×255×3 image as the search image, input the template image data and the search image data into the template branch and the search branch respectively, and obtain the template feature data w1 and the search feature data x respectively.

[0012] Step S2, input the template feature data w1 and the search feature data x into the deformable convolutional network, learn the adaptive offset of each feature data in different directions, output the template offset kernel and the search offset feature, and cross-correlate the template offset kernel and the search offset feature to obtain the offset map c.

[0013] Step S3, the template feature data w1, the search feature data x and the offset map c are subjected to deformable cross-correlation to obtain a final response map.

[0014] Step S4, calculate the sidelobe peak ratio of the final response graph of the i-th frame and the maximum sidelobe peak ratio of the final response graphs of the 1st to i-th frames every fixed frame, compare the sidelobe peak ratio and the maximum sidelobe peak ratio with the historical sidelobe peak ratio, and if the former two are both greater than the historical sidelobe peak ratio, update the template.

[0015] Step S5, when the template is determined to be updated, the i+1 frame captures an image of size 127×127×3 with the i-frame image as the center to obtain the target, and the image is extracted with new feature data through the template branch template. The new feature data is weightedly fused with the first frame template feature data to replace the original template feature data to become a new template for target tracking.

[0016] As a preferred solution, in step S1, the template branch and the search branch share parameters in the process of extracting feature data respectively, wherein the feature extraction process is shown in formula (1):

[0017]

[0018] Where I is the input image matrix, I((x a ,y a ),(x b ,y b )) is a submatrix of I, with (x a ,y a ) and (x b ,y b ) are the coordinates of the upper left corner and the lower right corner of the submatrix in the rectangular coordinate system, k is the step size for extracting feature data, and f(·) is the feature extraction function.

[0019] As a preferred solution, the specific steps of step S2 are as follows:

[0020] 2-1: Use a regular grid R to sample the search feature data x to obtain a sampling value;

[0021] 2-2: Use the template data feature w1 as the sum of the weighted sampling values ​​of the convolution kernel w to obtain the offset map c.

[0022] The calculation formula of regular grid R is as follows:

[0023] R={(0,0),(0,1),...,(k-1,k),(k,k)} (2)

[0024] A kernel of size k×k with dilation of 1 and padding of 0 is defined to act as a regular grid.

[0025] In the cross-correlation operation, for each position a on the output offset map c, the calculation formula is as follows:

[0026]

[0027] Among them, p n To enumerate all positions on R, w(*) is the convolution kernel, x(*) is the sampled value of the search feature data, and y(*) is the data on the offset map c.

[0028] As a preferred solution, the specific steps of step S3 are as follows:

[0029] 3-1: The data y(a) on the offset map c is taken as the offset set {Δpn|n=1,2,...,N}, N=|R|, R represents a regular grid;

[0030] 3-2: Using the template feature data w1, search for the feature data x and the offset set, calculate each position a on the output final response map, the calculation formula is as follows:

[0031]

[0032] Among them, y1(*) is the data on the final response graph.

[0033] In formula (4), the samples are taken at irregular and offset positions p n +Δp n , offset Δp n Usually a decimal, formula (4) can be implemented by bilinear interpolation:

[0034]

[0035] Where p′ represents a+p n +Δp n ,q * Enumerate all spatial positions in the search feature data x. G(*) is a bilinear interpolation kernel. Here G is a two-dimensional kernel, which can be divided into two one-dimensional kernels:

[0036]

[0037] Among them, g(a,b)=max(0,1-|ab|), Respectively represent the integer space position and fractional space position in the search feature data x; p x ′,p′ y Respectively represent a+p n +Δp n The integer spatial position and fractional spatial position of .

[0038] As a preferred solution, the peak sidelobe value ratio F of the final response diagram is calculated as follows:

[0039]

[0040] Among them, A max , A min Represent the maximum and minimum values ​​of the data on the final response graph, A i,j Represents the data on the final response graph (i, j), and mean(*) represents the mean.

[0041] The maximum value of the peak-to-sidelobe value ratio of the final response graph is the maximum value of the peak-to-sidelobe value ratios of all the final response graphs, which is recorded as M1.

[0042] Historical Sidelobe Peak Ratio The average of the Sidelobe Peak Ratios of all final response graphs.

[0043] As a preferred solution, the calculation formula for template update is:

[0044] fi=(1-η)f1(z)+ηf i (z) (10)

[0045] Among them, η is a hyperparameter, f1(z) is the first frame template feature data, and f i (z) is the new feature data extracted from the i-th frame image.

[0046] Beneficial effects: The method of twin network target tracking based on deformable convolution and template update provided by the present invention uses a deformable convolution network to effectively enhance the feature extraction capability of the backbone network, and uses a high-confidence template update strategy to ensure effective update. For the first time, the deformable convolution network and the template update strategy are combined. Deformable convolution is used to learn the adaptive offset of feature data in different directions to enhance the feature extraction capability of the backbone network. In addition, although deep features are rich in semantic information, they lack position information, and the positioning accuracy is improved through multi-layer deformable cross-correlation fusion. Finally, a high-confidence template update strategy is proposed, that is, the peak sidelobe ratio and maximum value of the response map are calculated once every fixed frame, and based on this, a weighted method is used to fuse features to update the template. It not only ensures effective update, but also prevents the template from being updated in special circumstances such as occlusion.

[0047] Its beneficial effects are as follows:

[0048] (1) The use of deformable convolution can effectively enhance the feature extraction capability of the twin backbone network and extract richer deep semantic features.

[0049] (2) By learning adaptive offsets of multi-layer data, the target positioning accuracy can be effectively improved through deformable cross-correlation fusion.

[0050] (3) Using an effective template update strategy can prevent the model drift phenomenon caused by the original template gradually losing its effectiveness due to large deformation of the target during long-term tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flow chart of the method.

[0052] Figure 2 This is the model framework diagram of the model. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with specific embodiments.

[0054] like Figure 1-2 As shown, a method for twin network target tracking based on deformable convolution and template updating includes the following steps:

[0055] Step S1, the twin network is mainly divided into a template branch and a search branch, which is used to extract deep features of image data. The first frame image of the data set or the camera capture screen is cropped with the target as the center to form an image of 127×127×3 as the template image. During the tracking process, the i+1 frame is cropped with the target position of the i-th frame as the center to form an image of 255×255×3 as the search image. Then, the template image data and the search image data are input into the template branch and the search branch respectively to obtain the template feature data w1 and the search feature data x respectively.

[0056] In step S1, the template image data and the search image data are input into the template branch and the search branch to obtain the template feature data w1 and the search feature data x respectively. The two branches share parameters in the process of extracting feature data respectively. The feature extraction process is shown in formula (1):

[0057]

[0058] Where I is the input image matrix, I((x a ,y a ),(x b ,y b )) is a submatrix of I, with (x a ,y a ) and (x b ,y b ) are the coordinates of the upper left corner and the lower right corner of the submatrix in the rectangular coordinate system, k is the step size for extracting feature data, and f(·) is the feature extraction function.

[0059] Step S2, input the template feature data w1 and the search feature data x into the deformable convolutional network, learn the adaptive offset of each feature data in different directions, output the template offset kernel and the search offset feature, and cross-correlate the template offset kernel and the search offset feature to obtain the offset map c.

[0060] The specific steps of step S2 are as follows:

[0061] 2-1: Use a regular grid R to sample the search feature data x to obtain a sampling value;

[0062] 2-2: Use the template data feature w1 as the sum of the weighted sampling values ​​of the convolution kernel w to obtain the offset map c.

[0063] The regular grid R defines the receptive field size and expansion, calculated as follows:

[0064] R={(0,0),(0,1),...,(k-1,k),(k,k)} (2)

[0065] A kernel of size k×k with dilation of 1 and padding of 0 is defined to act as a regular grid.

[0066] In the cross-correlation operation, for each position a on the output offset map c, the calculation is as follows:

[0067]

[0068] Among them, p n To enumerate all positions on R, w(*) is the convolution kernel, x(*) is the sampled value of the search feature data, and y(*) is the data on the offset map c.

[0069] Step S3, the template feature data w1, the search feature data x and the offset map c are subjected to deformable cross-correlation to obtain a final response map.

[0070] The specific steps of step S3 are as follows:

[0071] 3-1: Take the data y(a) on the offset graph c as the offset set {Δp n |n=1,2,...,N}, N=|R|, R represents a regular grid;

[0072] 3-2: Using the template feature data w1, search for the feature data x and the offset set, calculate each position a on the output final response map, the calculation formula is as follows:

[0073]

[0074] Among them, y1(*) is the data on the final response graph.

[0075] In formula (4), the samples are taken at irregular and offset positions p n +Δp n , offset Δp n Usually a decimal, formula (4) can be implemented by bilinear interpolation:

[0076]

[0077] Where p′ represents a+pn+Δpn, q * Enumerate all spatial positions in the search feature data x. G(*) is a bilinear interpolation kernel. Here G is a two-dimensional kernel, which can be divided into two one-dimensional kernels:

[0078]

[0079] Among them, g(a,b)=max(0,1-|ab|), Respectively represent the integer space position and fractional space position in the search feature data x; p′ x ,p′ yRespectively represent a+p n +Δp n The integer spatial position and fractional spatial position of .

[0080] Step S4, calculate the sidelobe peak ratio of the final response graph of the i-th frame and the maximum sidelobe peak ratio of the final response graphs of the 1st to i-th frames every fixed frame, compare the sidelobe peak ratio and the maximum sidelobe peak ratio with the historical sidelobe peak ratio, and if the former two are both greater than the historical sidelobe peak ratio, update the template.

[0081] The specific steps of step S4 are as follows:

[0082] 4-1: Calculate the peak sidelobe ratio and the maximum peak sidelobe ratio of the final response diagram. The peak sidelobe ratio F is calculated as follows:

[0083]

[0084] Among them, A max , A min Represent the maximum and minimum values ​​of the data on the final response graph, A i,j Represents the data on the final response graph (i, j), and mean(*) represents the mean.

[0085] The maximum value of the peak-to-sidelobe value ratio is the maximum value of the peak-to-sidelobe value ratios of all final response diagrams, which is recorded as M1.

[0086] Historical Sidelobe Peak Ratio The average of the Sidelobe Peak Ratios of all final response graphs.

[0087] Step S5, when the template is determined to be updated, the i+1 frame captures an image of size 127×127×3 with the i-frame image as the center to obtain the target, and the image is subjected to new feature data extracted by the template branch template. The new feature data is weightedly fused with the first frame template feature data to replace the original template feature data to become a new template for target tracking, in order to cope with tracking failures caused by large changes in the target.

[0088] The calculation formula for template update is:

[0089] fi=(1-η)f1(z)+ηf i (z) (10)

[0090] Among them, η is a hyperparameter, f1(z) is the first frame template feature data, and f i (z) is the new feature data extracted from the i-th frame image.

[0091] The reason why the first frame template feature is used is that the first frame template is in a non-occluded state and its semantic information is the most complete and rich. Since the information gap between adjacent frames is very small, frequent update operations will cause excessive time cost. Then, the above-mentioned high-confidence update strategy is selected, which only takes less time resources and enables the template information to be updated in a timely and accurate manner, so that the response map obtained in the subsequent cross-correlation operation is more accurate.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for target tracking using a twin network based on deformable convolution and template updating, characterized in that: The steps include: Step S1, crop the first frame image of the data set or the camera capture image with the target as the center to obtain a 127×127×3 image as the template image, crop the i+1th frame with the target position of the i-th frame as the center to obtain a 255×255×3 image as the search image during the tracking process, input the template image data and the search image data into the template branch and the search branch respectively, and obtain the template feature data w1 and the search feature data x respectively; Step S2, input the template feature data w1 and the search feature data x into the deformable convolutional network, learn the adaptive offset of each feature data in different directions, output the template offset kernel and the search offset feature, and cross-correlate the template offset kernel and the search offset feature to obtain the offset map c; Step S3, performing deformable cross-correlation on the template feature data w1, the search feature data x and the offset map c to obtain a final response map; Step S4, calculating the sidelobe peak ratio of the final response graph of the i-th frame and the maximum value of the sidelobe peak ratios of the final response graphs of the first to i-th frames every fixed frame, comparing the sidelobe peak ratio and the maximum value of the sidelobe peak ratio with the historical sidelobe peak ratio, and if the former two are both greater than the historical sidelobe peak ratio, updating the template; Step S5, when the template is determined to be updated, the i+1 frame captures an image of size 127×127×3 with the i-frame image as the center to obtain the target, and the image is extracted with new feature data through the template branch template. The new feature data is weightedly fused with the first frame template feature data to replace the original template feature data to become a new template for target tracking.

2. The method for target tracking using a twin network based on deformable convolution and template updating according to claim 1, characterized in that: In step S1, the template branch and the search branch share parameters in the process of extracting feature data respectively. The feature extraction process is shown in formula (1): Where I is the input image matrix, I((x a ,y a ),(x b ,y b )) is a submatrix of I, with (x a ,y a ) and (x b ,y b ) are the coordinates of the upper left corner and the lower right corner of the submatrix in the rectangular coordinate system, k is the step size for extracting feature data, and f(·) is the feature extraction function.

3. The method for target tracking using a twin network based on deformable convolution and template updating according to claim 1, characterized in that: The specific steps of step S2 are as follows: 2-1: Use a regular grid R to sample the search feature data x to obtain a sampling value; 2-2: Take the template data feature w1 as the sum of the weighted sampling values ​​of the convolution kernel w to obtain the offset map c; The calculation formula of regular grid R is as follows: R={(0,0),(0,1),...,(k-1,k),(k,k)} (2) A kernel of size k×k with expansion of 1 and filling of 0 is defined to act as a regular grid; In the cross-correlation operation, for each position a on the output offset map c, the calculation formula is as follows: Among them, p n To enumerate all positions on R, w(*) is the convolution kernel, x(*) is the sampled value of the search feature data, and y(*) is the data on the offset map c.

4. The method for target tracking using a twin network based on deformable convolution and template updating according to claim 1, characterized in that: The specific steps of step S3 are as follows: 3-1: Take the data y(a) on the offset graph c as the offset set {Δp n |n=1,2,...,N}, N=|R|, R represents a regular grid; 3-2: Using the template feature data w1, search for the feature data x and the offset set, calculate each position a on the output final response map, the calculation formula is as follows: Among them, y1(*) is the data on the final response graph.

5. The method for twin network target tracking based on deformable convolution and template updating according to claim 4, characterized in that: In formula (4), the samples are taken at irregular and offset positions p n +Δp n , offset Δp n Usually a decimal, formula (4) can be implemented by bilinear interpolation: Where p′ represents a+p n +Δp n ,q * Enumerate all spatial positions in the search feature data x. G(*) is a bilinear interpolation kernel. Here G is a two-dimensional kernel, which can be divided into two one-dimensional kernels: Among them, g(a,b)=max(0,1-|ab|), Respectively represent the integer space position and fractional space position in the search feature data x; p′ x ,p′ y Respectively represent a+p n +Δp n The integer spatial position and fractional spatial position of .

6. The method for target tracking using a twin network based on deformable convolution and template updating according to claim 1, characterized in that: The peak-to-sidelobe ratio F of the final response diagram is calculated as follows: Among them, A max , A min Represent the maximum and minimum values ​​of the data on the final response graph, A i,j Represents the data on the final response graph (i, j), and mean(*) represents the mean; The maximum value of the peak-to-sidelobe value ratio is the maximum value of the peak-to-sidelobe value ratios of all final response graphs, which is recorded as M1; Historical Sidelobe Peak Ratio The average of the Sidelobe Peak Ratios of all final response graphs.

7. The method for target tracking using a twin network based on deformable convolution and template updating according to claim 1, characterized in that: The calculation formula for template update is: f i =(1-η)f1(z)+ηf i (z) (10) Among them, η is a hyperparameter, f1(z) is the first frame template feature data, and f i (z) is the new feature data extracted from the i-th frame image.

Images