A target tracking method based on a twin network

Abstract

The application discloses a target tracking method based on a twin network, constructs a target tracking network based on a convolutional neural network, then sends two images before and after into the network to acquire feature information, outputs a target feature vector feature map, completes cross correlation operation, and obtains classification information; the effective classification information is increased through channel entropy value weighting; the application improves the accuracy of the classification information through the target tracking network based on the convolutional neural network, obtains a corrected classification network result, and finally obtains accurate coordinates of the target.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a target tracking method based on Siamese networks. Background Technology

[0002] Target tracking is a crucial function in vision tasks, widely used in areas such as autonomous driving, situational awareness of unmanned platforms like drones and autonomous vehicles, and security monitoring. In recent years, with the in-depth research into deep learning methods, target tracking algorithms based on convolutional neural networks (CNNs) have demonstrated excellent performance in terms of both tracking accuracy and robustness. Compared to methods based on centroid algorithms and correlation filtering, CNN-based target tracking methods offer higher tracking accuracy and better robustness, making them particularly suitable for target tracking applications in complex environments.

[0003] Siamese network architecture has been extensively studied and applied due to its time-consuming target feature extraction process, which involves sharing network weight parameters, and its ability to correlate spatial locations with temporal information from consecutive frames. Common target tracking methods based on Siamese networks directly output target or background information, providing only classification and location regression information. When the network design is inadequate or the output classification information is unreliable, inaccurate target coordinate information can be obtained, leading to tracking failure.

[0004] In complex environments, the classification subnetwork in the target tracking method based on Siamese networks may produce unreliable output classification information due to unreasonable network structure design or weak feature extraction capabilities, leading to incorrect prediction results in the target tracking algorithm. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this patent application proposes a target tracking method based on entropy information weighting (score information weighting) and Siamese networks.

[0006] The technical solution adopted by this invention to solve its technical problem is: a target tracking method based on Siamese networks, comprising the following steps:

[0007] S1, Construct a target tracking network based on a convolutional neural network: It mainly consists of two feature extraction backbone networks with the same weight, a cross-correlation module, a classification sub-network with multiple channels, a coordinate position regression network, and a score confidence part containing classification information and score information. The feature extraction backbone network mainly consists of convolutional layers, batch normalization layers, non-linear activation layers, and max pooling layers. The classification sub-network and coordinate position regression network mainly consist of convolutional layers, batch normalization layers, and non-linear activation layers.

[0008] S2, take the first frame of input image as the previous frame, manually select or select the target to be tracked by automatic detection algorithm, send it into the feature extraction backbone network to obtain feature information, and use it as the template for the second frame processing;

[0009] S3, take the second frame input image as the current frame and send it into the feature extraction backbone network;

[0010] S4, output the target feature vector feature map from the two frames of images before and after the shared feature extraction backbone network weights to the cross-correlation module to complete the cross-correlation operation: in Let z represent the two frames of images respectively, and f(z,x) represent the cross-correlation feature map;

[0011] S5, the cross-correlation feature maps f(z,x) are fed into the classification sub-network and the coordinate position regression network, respectively;

[0012] S6, the classification information and score information output by the classification sub-network, and the position information output by the coordinate position regression network are fed into the sigmoid function to calculate the classification confidence, thus obtaining the classification information:

[0013] S5, the results of the cross-correlation operation are fed into the classification sub-network and the coordinate position regression network, respectively;

[0014] S6. The classification branch output of the target tracking network based on the convolutional neural network, namely the classification information and score information output by the classification sub-network, and the position information output by the coordinate position regression network are fed into the sigmoid function to calculate the classification confidence and obtain the classification information. Since it is a target tracking network, it is a binary classification network, namely target or background information.

[0015] S7 increases effective classification information by replacing simple channel overlay with channel entropy weighting (entropy information weighting).

[0016] Furthermore, the convolutional layer of the classification sub-network contains a 3×3 convolutional kernel, and the convolutional layer of the coordinate position regression network contains a 3×3 convolutional kernel and a 1×1 convolutional kernel.

[0017] Furthermore, the specific calculation process of step S7 is as follows:

[0018] S71, Calculate the entropy of the feature map of the target feature vector for each channel of the classification subnetwork:

[0019]

[0020] Where max and min represent the maximum and minimum values ​​of the numerical values ​​in the feature map, p i,j This indicates the probability of a certain value appearing in the feature map;

[0021] The entropy value is used as the weighting weight of the channel. The feature map of each channel is weighted, and then the weights of each pixel on the feature maps of multiple channels are accumulated and merged into one channel as the score information.

[0022] S72 uses the score information as a penalty term for the classification information. Each pixel on the feature map is multiplied, and a probability value in the range of 0 to 1 is finally output as a reliable classification result.

[0023] S73, each element in the output feature map of the coordinate position regression network is a vector containing (△x, △y) results, which is represented as the offset of the horizontal and vertical coordinates of each pixel in the feature map of the previous frame.

[0024] Furthermore, in the classification results obtained in step S72, the feature map coordinate position (i,j) corresponding to the item with the largest value is obtained. The result at (i,j) is found from the target feature vector output by the coordinate position regression network in step S73 and used as the target coordinate offset for the current frame. This offset is added to the target coordinate information of the previous frame to obtain the absolute value of the target coordinates predicted for the current frame.

[0025] The present invention has the following characteristics: The method of the present invention is based on a classification network weighted by score information, corrects unreliable classification information, improves the accuracy of target tracking algorithm, improves the accuracy of classification information, and obtains the accurate coordinates of the target through the corrected classification network results, which can effectively improve the target tracking accuracy of Siamese networks. Attached Figure Description

[0026] Figure 1 This is a network diagram of the present invention.

[0027] The figures are labeled as follows: 1—feature extraction backbone network, 2—cross-correlation module, 3—classification sub-network, 4—coordinate position regression network. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to specific embodiments in conjunction with the accompanying drawings. These embodiments are for illustrative purposes only and do not constitute a limitation thereof.

[0029] This invention discloses a target tracking method based on Siamese networks, comprising the following steps:

[0030] S1, Construct a target tracking network based on a convolutional neural network, such as Figure 1As shown, it mainly consists of two feature extraction backbone networks 1 with equal weights, a cross-correlation module 2, a classification sub-network 3 with multiple channels, a coordinate position regression network 4, and a score confidence part containing classification information and score information. Among them, the feature extraction backbone network 1 mainly consists of convolutional layers, batch normalization layers, non-linear activation layers, and max pooling layers. The classification sub-network 3 mainly consists of convolutional layers (containing 3×3 convolutional kernels), batch normalization layers, and non-linear activation layers. The coordinate position regression network 4 mainly consists of convolutional layers (containing 3×3 convolutional kernels and 1×1 convolutional kernels), batch normalization layers, and non-linear activation layers.

[0031] S2 takes the first frame of input image as the previous frame, selects the target to be tracked manually or through an automatic detection algorithm, and sends it to the feature extraction backbone network 1 to obtain feature information, which is then used as a template for the next frame (second frame) processing.

[0032] S3, take the second frame input image as the current frame and send it into the feature extraction backbone network 1.

[0033] S4, extract the shared features from the weights of backbone network 1 in the two consecutive frames of images. The target feature vector feature map is output to the cross-correlation module 2 respectively to complete the cross-correlation operation and obtain the cross-correlation feature map f(z,x):

[0034] S5, the results of the cross-correlation operation (cross-correlation feature map) are fed into the classification sub-network 3 and the coordinate position regression network 4, respectively.

[0035] The classification information and score information output by the classification branch of the convolutional neural network-based target tracking network (i.e., the classification sub-network 3), and the position information output by the coordinate position regression network 4 are fed into the sigmoid function to calculate the classification confidence score, thus obtaining the classification information f. cls Since it is a target tracking network, it is a binary classification network, that is, a target or background information network:

[0036] S7. Since the classification subnetwork 3 is composed of multiple channels, due to the black box effect of neural networks, it is difficult to determine the importance of multiple channels to the calculation results. If the classification accuracy is improved by simply superimposing channels, it may amplify invalid information and reduce effective information. Entropy information is a good parameter indicator to reflect the complexity of information. Therefore, in this patent, effective classification information is increased by replacing simple channel superposition with channel entropy weighting.

[0037] The specific calculation process is as follows:

[0038] S71, calculate the entropy value of the feature map of the target feature vector of each channel of the classification subnetwork 3, use it as the weight of each channel, weight the result (feature map) corresponding to each channel, and then accumulate the corresponding elements (each pixel on the feature map) of multiple channels, and merge them into one channel as the score information.

[0039] S71, calculate the entropy value of the feature map of the target feature vector for each channel of the classification sub-network 3, and use it as the weighting weight for each channel. Weight the results (feature maps) for each channel, then accumulate the corresponding elements (each pixel on the feature map) of multiple channels, and merge them into one channel as the score information; where the entropy is calculated as follows: Where max and min represent the maximum and minimum values ​​of the numerical values ​​in the feature map, p i,j This indicates the probability of a certain value appearing in the feature map.

[0040] S72 outputs the score information as a penalty term for the classification information. Each pixel on the feature map is multiplied, and finally a probability value in the range of 0 to 1 is output as a reliable classification result.

[0041] S73, the target feature vector output by the coordinate position regression network 4. Each element in the feature map is a vector containing (△x, △y) results, which is represented as the x-coordinate and y-coordinate offset from the corresponding element in the previous frame.

[0042] In the classification results obtained in step S71, the target feature vector feature map coordinate position (i,j) corresponding to the item with the largest value is obtained. The result (output feature vector value) at (i,j) is found from the target feature vector output by the coordinate position regression network 4 in step S73. This is the target coordinate offset predicted in the current frame. Adding it to the target coordinate information of the previous frame, the absolute value of the target coordinate predicted in the current frame can be obtained.

[0043] The above description is merely for illustrative purposes and not intended to limit the scope of the invention. Any person skilled in the art may make changes or modifications to the disclosed technical content to create equivalent embodiments. Those skilled in the art should understand that any modifications or equivalent substitutions that do not depart from the spirit and scope of the invention are covered within the scope of the claims of the invention.

Claims

1. A target tracking method based on Siamese networks, characterized in that: Includes the following steps S1, Construct a target tracking network based on a convolutional neural network: It consists of two feature extraction backbone networks (1) with the same weight, a cross-correlation module (2), a classification sub-network (3) with multiple channels, a coordinate position regression network (4), and a score confidence part containing classification information and score information. The feature extraction backbone network (1) consists of convolutional layers, batch normalization layers, non-linear activation layers, and max pooling layers. The classification sub-network (3) and the coordinate position regression network (4) are both composed of convolutional layers, batch normalization layers, and non-linear activation layers. S2, take the first frame input image as the previous frame, select the target to be tracked, and send it into the feature extraction backbone network (1) to obtain feature information, which is used as the template for the second frame processing; S3, take the second frame input image as the current frame and send it into the feature extraction backbone network (1). S4, output the target feature vector feature map to the cross-correlation module (2) respectively for the two frames of images with shared feature extraction backbone network (1) weights, and complete the cross-correlation operation: ,in , These represent the two frames of the image, respectively. Represents the cross-correlation feature map; S5, Cross-correlation Feature Map The data are fed into the classification subnetwork (3) and the coordinate position regression network (4), respectively. S6, the classification information and score information output by the classification subnetwork (3), and the position information output by the coordinate position regression network (4) are fed into the sigmoid function to calculate the classification confidence, and the classification information is obtained: ; S7 increases effective classification information by weighting the channel entropy values.

2. The target tracking method based on Siamese networks according to claim 1, characterized in that, The specific steps of step S7 are as follows: S71, Calculate the entropy of the feature map of the target feature vector for each channel of the classification subnetwork (3): Where max and min represent the maximum and minimum values ​​of the numerical values ​​in the feature map, Indicates coordinate position i,j Frequency of occurrence in this feature map; The entropy value is used as the weighting weight of the channel. The feature map of each channel is weighted, and then the weights of each pixel on the feature maps of multiple channels are accumulated and merged into one channel as the score information. S72 uses the score information as a penalty term for the classification information. Each pixel on the feature map is multiplied, and a probability value in the range of 0 to 1 is finally output as a reliable classification result. S73, the coordinate position regression network (4) outputs the target feature vector. Each element in the feature map is represented as the offset of the horizontal and vertical coordinates of each pixel in the previous frame feature map.

3. The target tracking method based on Siamese networks according to claim 2, characterized in that, In the classification results obtained in step S72, the feature map coordinates corresponding to the item with the largest value are obtained. i , j ), find the target feature vector output by the coordinate position regression network (4) in step S73 () i , j The result at () is used as the target coordinate offset predicted in the current frame. It is added to the target coordinate information of the previous frame to obtain the absolute value of the target coordinates predicted in the current frame.

4. A target tracking method based on a Siamese network according to claim 1, 2, or 3, characterized in that, The convolutional layer of the classification subnetwork (3) contains a 3×3 convolutional kernel, and the convolutional layer of the coordinate position regression network (4) contains a 3×3 convolutional kernel and a 1×1 convolutional kernel.

Images