Twin network target tracking method based on multi-scale response diagram fusion
By adopting a twin network method of multi-scale response graph fusion in target tracking, the misjudgment problem caused by the similarity of target objects to background features in the prior art is solved, the tracking accuracy and robustness are improved, and efficient tracking in complex environments is achieved.
Patent Information
- Application Number
- CN202411947098.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-25
AI Technical Summary
Existing deep learning-based target tracking algorithms are prone to misjudgment when dealing with scenes where the target object is too similar to the background features, and using only high-level semantic features is not enough to ensure the tracking success rate.
Using a twin network target tracking method based on multi-scale response graph fusion, by constructing a twin network model, the output features of each layer are deeply convolutionized to obtain the response graph, and the response graph is fused on multiple scales to enhance the discriminant ability of the model.
It improves the accuracy and robustness of target tracking, enhances the model's ability to discriminate targets and backgrounds, effectively reduces residuals, and maintains efficient tracking performance in complex environments.
Smart Images
Figure CN119888568A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer vision technology, and in particular to a twin network target tracking method based on multi-scale response graph fusion. Background Art
[0002] Target tracking plays a very important role in the field of computer vision. It is a multidisciplinary cross-disciplinary technology that integrates the technologies of machine learning, computer vision, pattern recognition and other disciplines, and has been widely used in many fields such as security monitoring, interactive systems, smart transportation and assisted driving systems. In the past few decades, the research on target tracking algorithms has been very active in the field of computer vision. From the earliest particle filter framework to the correlation filter, the performance of the tracking algorithm has gradually improved.
[0003] The tracking method based on deep learning is another important branch of target tracking methods. It takes advantage of the end-to-end training of deep convolutional networks to allow the model to automatically learn the appearance and motion characteristics of the target to achieve high-quality robust tracking. Most target tracking algorithms based on deep learning are discriminative methods, which can regard tracking as a classification problem, classify the target object and the background, and distinguish between the foreground and the background.
[0004] Combining the twin network with other advanced machine vision processing modules is also an important development direction. The twin network structure has been studied by scholars at home and abroad for a long time, and has been widely used in tasks such as face recognition, single character recognition, key point descriptor learning and target tracking. Luca Bertinertto proposed a relatively new algorithm based on deep learning tracking, SiamFC (Siamese Fully-Convolutional), which uses a fully convolutional twin neural network for target tracking. In SiamFC, the network model uses a pre-trained network to obtain rich semantic features, which is conducive to the classification of target objects. However, the spatial information of shallow features is discarded during the training process. However, some inherent factors still affect the tracking effect. If the features of the target object are too similar to those of some objects in the background, it may cause the algorithm to misjudge. Using only high-level semantic features of the network cannot meet the tracking success rate. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a twin network target tracking method based on multi-scale response graph fusion in view of the defects in the prior art.
[0006] The technical solution adopted by the present invention to solve the technical problem is: a twin network target tracking method based on multi-scale response graph fusion, comprising the following steps:
[0007] 1) Obtain the template image z of the first target frame from the video sequence, and the image x to be searched of the candidate frame in the current frame of the video sequence;
[0008] 2) Constructing a twin network target tracking model, performing deep convolution on the output features in each layer to obtain a response map of each layer; the network model includes two upper and lower branches, and the two branches have two identical five-convolutional layer network structures. The input of the upper branch is the template image z, and the input of the lower branch is the image to be searched x. The upper and lower branches are then connected to two fully connected layers, and the output of the last fully connected layer is a response map, which is a network structure defined on a finite grid D∈Z 2 Similarity score matrix on ;
[0009] After each layer of convolution, the template image and the search image undergo similarity learning, as follows:
[0010]
[0011] Among them, f i It represents the response graph obtained by similarity learning after the i-th convolution layer; * indicates that the similarity metric is realized by using deep convolution, and the features obtained by comparing the upper and lower branches in the cross-correlation layer; represents convolutional feature extraction;
[0012] 3) After downsampling the response map of the first layer, the different depth response maps obtained after convolution of each layer are fused to obtain F, as shown in formula (2).
[0013] F=[f 1 ,f 2 ,f 3 ,f 4 ,f 5 ] (2)
[0014] 4) Send F to two fully connected layers to train the optimal response map and obtain the final response map. The maximum value point in the response map is the target area.
[0015] According to the above scheme, the step 1) also includes the step of image preprocessing.
[0016] For the template image, the target area is expanded so that the target area framed by the bounding box plus the context boundary is a fixed area, as shown in formula (3):
[0017] s×(w+2p)×s(h+2h)=A (3)
[0018] The width of the target bounding box is w, the height is h, and the context boundary p is half of the average dimension of the width and height, that is, A is the area of the template image, s is the scale factor, and the bounding box is scaled. The transformation method of s is to fill the pixel mean of each RGB channel of the original image without changing the original size to obtain the processed image template image and the image to be searched.
[0019] According to the above scheme, after obtaining the picture from the video sequence in step 1), the template picture and the search area of each frame are extracted offline.
[0020] The beneficial effects produced by the present invention are:
[0021] 1) The present invention improves the tracking accuracy and enhances the model's ability to distinguish between targets and backgrounds by constructing a twin network target tracking model;
[0022] 2) The method of the present invention uses an offline learning and online tracking method to effectively reduce the residual error. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0024] Figure 1 is a method flow chart of an embodiment of the present invention;
[0025] Figure 2 is a model structure diagram of a twin network of an embodiment of the present invention;
[0026] Figure 3 Schematic diagram of the accuracy curve of the embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0028] like Figure 1 As shown, a twin network target tracking method based on multi-scale response graph fusion includes the following steps:
[0029] 1) Obtain the template image z of the first target frame from the video sequence, and the image x to be searched of the candidate frame in the current frame of the video sequence;
[0030] For the template image, the target area is expanded so that the target area framed by the bounding box plus the context boundary is a fixed area, as shown in the following formula:
[0031] s×(w+2p)×s(h+2h)=A
[0032] The width of the target bounding box is w, the height is h, and the context boundary p is half of the average dimension of the width and height, that is, A is the area of the template image, s is the scale factor, and the bounding box is scaled. The transformation method of s is to fill the pixel mean of each RGB channel of the original image without changing the original size. The processed image template image and the image to be searched are obtained.
[0033] 2) Constructing a twin network target tracking model, performing deep convolution on the output features in each layer to obtain a response map of each layer; the network model includes two upper and lower branches, and the two branches have two identical five-convolutional layer network structures, the input of the upper branch is the template image z, and the input of the lower branch is the image to be searched x, and the upper and lower branches are subsequently connected to two fully connected layers, and the output of the last fully connected layer is the response map;
[0034] After each layer of convolution, the template image and the search image undergo similarity learning, as follows:
[0035]
[0036] Among them, f i It represents the response graph obtained by similarity learning after the i-th convolution layer; * indicates that the similarity metric is realized by using deep convolution, and the features obtained by comparing the upper and lower branches in the cross-correlation layer; represents convolutional feature extraction;
[0037] 3) Since the response map obtained after the first layer of convolution is relatively large, we downsample the response map of the first layer and then fuse the different depth response maps obtained after each layer of convolution;
[0038] After downsampling the response map of the first layer, the different depth response maps obtained after convolution of each layer are fused to obtain F, as shown in the following formula:
[0039] F=[f 1 ,f 2 ,f 3 ,f 4 ,f 5 ]
[0040] 4) Send F to two fully connected layers to train the optimal response map and obtain the final response map. The maximum value point in the response map is the target area.
[0041] The optimization method used during training is batch stochastic gradient descent (SGD). Eight samples are used in each training to calculate the gradient and update the parameters. The learning rate ranges from 10 -2 to 10-5 In descending order.
[0042] Experimental verification of the effectiveness of the method of the present invention:
[0043] Step 1: Select experimental data
[0044] In the present embodiment, the video sequence for target detection in the ILSVRC2015 (ImageNet Large Scale Visual Recognition Challenge) that has been sorted is used as the data set for training, i.e., the ImageNet data set. The ILSVRC2015 VID data set includes 30 different categories of animals, objects such as vehicles, and the training and verification sets include 4500 videos in total, and each frame of the video also has groundtruth manually marked, accumulating more than 1 million target frames with labels. The sorted data set is used during the training of the present embodiment, and some videos in the ILSVRC2015 VID are removed, because some targets of these video sequences are too small or too large, and some targets are too close to the edge, and it is not easy to obtain the overall outline of the target. Compared with the previous algorithm, the model is trained on small-scale data sets such as OTB50, OTB100, VOT, etc., and the types and quantity of training samples are increased, which can effectively prevent overfitting.
[0045] In order to avoid adjusting the image size during training, the template image and search area of each frame are extracted offline. For the input image, since the template image size is 127x127 and the search image size is 255x255, the target area needs to be expanded first, so that the target area framed by the bounding box plus the context boundary is a fixed area, as shown in the following formula:
[0046] s×(w+2p)×s(h+2h)=A
[0047] The width of the target bounding box is w, the height is h, and the context boundary p is half of the average dimension of the width and height, that is, A=127 2 , that is, the area of the template image, s is the scale factor, and the bounding box is scaled. The transformation method of s is to fill the pixel mean of each RGB channel of the original image without changing the original size. The processed image template image and the image to be searched are obtained.
[0048] In order to verify the comprehensive performance of the algorithm, ten videos with more complex scenes were extracted from OTB50 to analyze the target tracking algorithm based on the twin network proposed in this paper, namely BlurBody, CarDark, Couple, Crowds, Deer, Panda, Shaking, Surfer, Walking, and Woman. The data set contains 11 influencing factors: Illumination Variation (IV), Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Motion Blur (MB), Fast Motion (FM), In-Plane Rotation (IPR), Out-of-Plane Rotation (OPR), Out-of-View (OV), Background Clutters (BC) and Low Resolution (LR).
[0049] Step 2: Training Methods
[0050] The optimization method used in model training is batch stochastic gradient descent (SGD). Eight samples are used in each training to calculate the gradient and update the parameters. The learning rate ranges from 10 -2 to 10 -5 In descending order.
[0051] Step 3: Quantitative analysis;
[0052] The overall performance of the proposed method and the TLD, MUSTer, and STRUCK algorithms was evaluated using the overlap success rate (OS) and distance precision (DP) under the condition of one pass evaluation (OPE). The precision curve was evaluated using DP, and the success rate curve was compared using the area under the curve (AUC).
[0053] like Figure 3 As shown, Figure 3(a) is a test graph of the success rate curve, which represents the size of the overlap rate between the estimated target rectangular box and the real target rectangular box, that is, the area of the intersection of the two rectangular boxes divided by the area of the combined parts of the two rectangular boxes. Its horizontal axis is a continuous threshold from 0 to 1, and the vertical axis is the percentage of all frames with an overlap rate greater than the threshold in the total frames of the sequence. The area under the curve is used to represent the evaluation score. The success rate corresponding to each tracker when 0.5 is used. It can be seen from the curve that the method of the present invention is 0.08% higher than the MUSTer algorithm, 3.93% higher than the STRUCK algorithm, and 11.11% higher than the TLD algorithm, and the effect is quite obvious. Figure 3 (b) is a precision curve test, which measures the Euclidean distance error between the estimated target center and the true target center. Its horizontal axis is a series of different distance thresholds (pixel units), and the vertical axis is the percentage of all frames whose error distance of the estimated target center position is less than the threshold in the total frames of the sequence. The percentage corresponding to the threshold of 11 pixels is the evaluation score. It can be seen that the algorithm accuracy of the method of the present invention reaches 98% under complex video, which is improved to varying degrees compared with TLD, MUSTer, and STRUCK algorithms.
[0054] Step 4: Qualitative Analysis
[0055] Ten video sequences containing multiple influencing factors were selected from the OTB50 video library to compare the tracking performance of TLD and the method of the present invention. Table 2 shows the comparison of the accuracy of the influencing factor algorithms in these sequences. It can be seen that under any influencing factor, the method of the present invention performs better than TLD.
[0056] Table 1 Accuracy comparison
[0057]
[0058] According to the above analysis, it can be seen that the target tracking method based on the twin network proposed in the present invention can effectively cope with complex environments, while ensuring tracking accuracy and taking into account robustness.
[0059] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A twin network target tracking method based on multi-scale response graph fusion, characterized in that: The following steps are involved: 1) Obtain the template image z of the first target frame from the video sequence, and the image x to be searched of the candidate frame in the current frame of the video sequence; 2) Construct a twin network target tracking model, perform deep convolution on the output features in each layer to obtain the response map of each layer; the network model includes two upper and lower branches, and the two branches have two identical five-convolutional layer network structures. The input of the upper branch is the template image z, and the input of the lower branch is the image to be searched x. The upper and lower branches are then connected to two fully connected layers, and the output of the last fully connected layer is the response map, which is a network structure defined on a finite grid D∈Z 2 Similarity score matrix on ; After each layer of convolution, the template image and the search image undergo similarity learning, as follows: Among them, f i It represents the response graph obtained by similarity learning after the i-th convolution layer; * indicates that the similarity metric is realized by using deep convolution, and the features obtained by comparing the upper and lower branches in the cross-correlation layer; Represents the convolution operation; 3) After downsampling the response map of the first layer, the different depth response maps obtained after convolution of each layer are fused to obtain F, as shown in formula (2). F=[f1,f2,f3,f4,f5] (2) 4) The fused response map F is sent to two fully connected layers to train the optimal response map and obtain the final response map. The maximum value point in the response map is the target area.
2. The twin network target tracking method based on multi-scale response graph fusion according to claim 1 is characterized in that: After obtaining the picture from the video sequence in step 1), the template picture and the search area of each frame are extracted offline.
3. The twin network target tracking method based on multi-scale response graph fusion according to claim 1 or 2, characterized in that: The step 1) also includes the step of image preprocessing. For the template image, the target area is expanded so that the target area framed by the bounding box plus the context boundary is a fixed area, as shown in formula (3): s×(w+2p)×s(h+2h)=A (3) The width of the target bounding box is w, the height is h, and the context boundary p is half of the average dimension of the width and height, that is, A is the area of the template image, s is the scale factor, and the bounding box is scaled. The transformation method of s is to fill the pixel mean of each RGB channel of the original image without changing the original size to obtain the processed image template image and the image to be searched.
4. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 3.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.