A quantization-friendly twin network target tracking method and device

Abstract

The present invention relates to a quantization-friendly twin network target tracking method and device. With respect to the feature extraction backbone network, the accuracy and quantization friendliness of various existing lightweight feature extraction backbone networks are experimentally compared, and the feature extraction backbone network GhostNet that is most suitable for the low-precision fixed-point twin network target tracking method is selected; with respect to the feature fusion method, the quantization error and tracking accuracy of various lightweight feature fusion methods are compared, and a new lightweight and quantization-friendly feature extraction method GhostAttention is proposed; with respect to the branch prediction method, the quantization error and tracking accuracy of the existing network structure design methods are also compared, and the best branch prediction structure corner point heat map is selected. Finally, a simple, lightweight, and quantization-friendly twin network target tracking method is proposed.

Description

Technical Field

[0001] The present invention belongs to the field of visual target tracking and relates to a quantization-friendly twin network target tracking method and device. Background Art

[0002] Visual object tracking is a hot research area in computer vision, widely used in intelligent driving, security monitoring, military attack and defense, satellite remote sensing, and multimedia entertainment. Many of these applications rely on mobile computing platforms. However, mobile computing chips have limited on-chip area, relatively weak computing power, and strict power consumption constraints. Researchers have designed various lightweight network architectures for mobile platforms. Furthermore, they are also changing the computational methods of computing chips, optimizing matrix calculations and low-precision computations through neural network quantization to further reduce the computational overhead of neural network inference on mobile computing chips. In many practical applications, neural networks are designed to be lightweight and quantized before deployment. Neural network quantization is the process of mapping neural network parameters and input values ​​from high-precision data types to low-precision data types, accelerating the network's inference process. Common deep neural network platforms use 32-bit floating-point data types, while common mobile AI computing platforms primarily support 8-bit integer data types. The data range that can be represented by the 32-bit floating-point data type is [-3.4×1038, +3.4×1038]. However, the parameters and input values ​​of deep neural networks are only a small part of this range. Therefore, the data range required to map low-precision data does not completely cover all the values ​​that can be represented by high-precision data types.

[0003] Most current mainstream algorithms are based on the Siamese network target tracking algorithm framework. The network architecture of a Siamese network target tracking algorithm generally consists of three components: a feature extraction backbone network, a feature fusion method, and a branch prediction network. The feature extraction backbone network extracts deep features from the image, the feature fusion method combines template features with search region features, and the branch prediction network outputs a state estimate of the tracked target object. The breakthrough performance improvements achieved in Siamese network-based target tracking algorithms in recent years are the result of in-depth research into their various components, particularly the operators within the feature fusion architecture. Lightweight network designs have also been developed for mobile devices. However, these lightweight networks are designed to reduce computational complexity and serve merely as lightweight alternatives to large-scale models. They are still targeted at high-precision floating-point platforms and lack performance analysis of quantized low-precision fixed-point operations. When deployed on low-precision platforms, these network structures often suffer from significant quantization losses, significantly reducing algorithm accuracy. Currently, research on the quantization-friendliness of the various operators in Siamese network-based target tracking algorithms is lacking. This has become a major limitation for the deployment of visual target tracking algorithms on mobile chips.

[0004] Therefore, a twin network target tracking method with a lightweight and deployment-friendly design is needed to improve tracking accuracy while ensuring that the processing speed meets real-time requirements. Summary of the Invention

[0005] In response to the problems existing in the prior art, the present invention provides a quantization-friendly twin network target tracking method, which specifically includes the following steps:

[0006] (1) Based on the initial target annotation, the target template is captured from the initial frame of the video as the template. According to the labeled target bounding box information, the template training data is scaled to 2 and resized to 112×112;

[0007] (2) In the subsequent frames of the template, the search area is captured. The scale of the search area is 4.3 times that of the target object. For the search area training data, the labeled target bounding box information is jittered, including scale scaling and position translation. Then, based on this jittered annotation, the scale is scaled to 5 and the image is resized to 320×320.

[0008] (3) The search area and template are input into the target tracking twin network, which includes a feature extraction backbone network, a feature fusion structure, and a branch prediction structure; wherein the feature extraction backbone network adopts GhostNet to obtain the template feature f z and search area f x , the feature fusion structure adopts GhostAttention, fusion f z and f x Get the fusion feature map f zx , the branch prediction structure uses a corner heat map method to output an estimate of the target bounding box bb pred and the confidence score cls for this estimate pred ;

[0009] (4) The confidence score ranges from [0,1]. The larger the confidence score, the higher the possibility that the tracking target object described by the template exists in the search area. pred ≥90%, bb pred The target tracking Siamese network estimates the target template bounding box in the search area; otherwise, the tracking target object described by the template does not exist in the search area;

[0010] (5) Return to step (2) and process the next frame until all video frames are processed.

[0011] Among them, according to the estimation of the target bounding box bb pred and the confidence score cls for this estimatepred , calculate the bounding box estimation loss and confidence loss:

[0012] The bounding box estimation loss is: L reg (bb pred ,bb gt )=λ giou L giou (bb pred ,bb gt )+λ L1 L L1 (bb pred ,bb gt ). Among them, L giou and L L1 They are GIoU loss and L1 loss, λ giou and λ L1 are the constant coefficients of GIoU loss and L1 loss respectively.

[0013] The confidence loss is: L cls (cls pred ,cls gt )=BCE(cls pred ,cls gt ), and use the binary cross entropy function BCE to calculate its loss. Among them, the true confidence cls gt In the training sample, the search area is used to determine whether the tracking target object described by the template exists. If so, cls gt =1, otherwise cls gt =0.

[0014] For neural networks, the data that requires quantization includes model parameters, input values, and activation values. Model parameters, including weights and biases, are fixed after model training and have a limited data range. Quantization mapping can be performed offline. Input values ​​are the input data used during model inference. For some data network structures, the input value range is uncertain. For computer vision, the numerical range of the input RGB image is limited. Activation values ​​are intermediate results generated during model inference and represent the data flow during the network's forward propagation. Therefore, their range is relatively fluid. Generally, the quantization of input and activation values ​​is performed online. A common method for determining the quantization step size is post-quantization, which involves calculating the quantization step size after model training is complete. Since the numerical range and distribution of model parameters are already determined, the optimal quantization step size can be calculated without additional data. For input and activation values, a set of example input samples is required to calculate the range and distribution of the input and activation values, and thus the optimal step size. A minmax approach is typically used. These samples are called calibration datasets.

[0015] The above quantitative data all have quantitative loss, which is defined as follows:

[0016] For an 8-bit integer quantization method symmetrical around 0, the quantization step parameter s is set. The data range that can be represented by the 8-bit integer is [-127×s, 127×s]. For the original high-precision data x, its quantization function mapped to the 8-bit integer is:

[0017]

[0018] For an 8-bit integer value y, the dequantization function value mapped back to the original high-precision data is:

[0019] DQ(y,s)=y×s,

[0020] The quantized value of high-precision data is:

[0021] The quantization loss SNR is thus defined as:

[0022] This paper analyzes the quantization-friendliness and tracking performance of the three components of a target tracking twin network: the feature extraction backbone network, the feature fusion structure, and the branch prediction structure. It then summarizes a quantization-friendly design approach for a twin network target tracking algorithm. Based on this design approach, a lightweight and quantization-friendly feature fusion method is proposed, resulting in a quantization-friendly twin network target tracking method.

[0023] (1) The feature extraction backbone network uses GhostNet. The feature extraction backbone network is to obtain the template image after intercepting the RGB image. and search area image Extract the deep semantic information and get the template features and search area Here, stride is the stride of the feature extraction backbone network. The approach is: use a dataset for image classification to train a converged image classification model, and use the intermediate features in the inference process of this model as the deep semantic features of the image. GhostNet believes that there is redundancy in the feature layers in the trained network, and feature maps of different depths may have a certain degree of similarity. These feature maps are called "shadow" feature maps. Therefore, during the forward propagation of the network, GhostNet explicitly retains some channels of the previous layer, and other channels also use cheap operations with low computational complexity. Since redundancy in feature maps may be an important feature of the success of deep neural networks, GhostNet tends to receive redundant feature maps in a cost-effective manner. Therefore, GhostNet has higher classification accuracy, stronger feature extraction capabilities, and smaller quantization loss SNR.

[0024] The feature fusion structure adopts GhostAttention. This structure is proposed by the present invention. It is a feature fusion structure based on the design of GhostNet and combined with the Attention mechanism, with low quantization loss. On the one hand, compared with convolution-based feature fusion methods such as splicing and FiLM, this structure has stronger modeling capabilities; on the other hand, compared with the Attention structure, this structure has lower computational complexity under the same number of channels. Its specific structure is: perform depth-separable convolution on the search area features and template features respectively to generate corresponding output features Figure 1 and output features Figure 2 For the output features Figure 1 , differentiated into two branches, namely Figure 1 Branch 1 and branch 2 are shown. In branch 2, CrossAttention is used to fuse the output features. Figure 1 and output features Figure 2 , that is, the output feature Figure 1 As query, output features Figure 2 As key and value, by calculating the similarity between query and key, we get the weight coefficient of key corresponding to value, and then perform weighted summation on the value to get the fusion feature Figure 2 In branch 1, the “shadow” feature of the search area feature extracted by the depthwise separable convolution structure is retained, and this “shadow” feature is combined with the fusion feature through the splicing operation. Figure 2 Directly fuse in the channel dimension to obtain the final feature fusion result feature fusion Figure 1 , which is the fusion feature map f zx .

[0025] The branch prediction structure uses the corner heat map method. The input of the branch prediction structure is the fusion feature map f obtained by the feature fusion structure. zx The output is the state estimation of the target object described by the template in the search area. The state estimation includes the bounding box estimation of the target object and the confidence of this estimation. zx Depthwise separable convolution is performed to reduce the number of channels, and then a fully connected layer is used to directly estimate the target bounding box. A corner heatmap approach estimates the heatmaps of the top left and bottom right corners of the target within the search area. The final corner positions are calculated using argsoftmax, resulting in an estimated bounding box. This approach demonstrates excellent performance in both pre- and post-quantization target tracking.

[0026] The present invention also provides a quantization-friendly twin network target tracking device, comprising the following modules:

[0027] The target template capture module is used to capture the target template from the initial frame of the video according to the initial target annotation as a template;

[0028] The search area interception module is used to expand the scope of the search area in subsequent frames according to the state of the target in the previous frame;

[0029] The target tracking module is used to input the search area and template into the pre-trained target tracking twin network, which includes a feature extraction backbone network, a feature fusion structure and a branch prediction structure; wherein the feature extraction backbone network adopts GhostNet, the feature fusion structure adopts GhostAttention, and the branch prediction structure adopts a corner point heat map method to output an estimate of the target bounding box. pred and the confidence score cls for this estimate pred ;

[0030] The state estimation module is used to calculate the state of the target template in the current frame based on the confidence score;

[0031] Conventional confidence score cls pred When bb is greater than a certain threshold, pred The target tracking Siamese network estimates the target template bounding box in the search area; otherwise, the tracking target object described by the template does not exist in the search area;

[0032] The output module repeats the processing of the search area interception module, the target tracking module and the state estimation module in the next frame until all video frames are processed.

[0033] The present invention implements a simple, lightweight, and quantization-friendly twin network target tracking method through the design of feature extraction backbone network, feature fusion structure, and branch prediction structure. With respect to the feature extraction backbone network, the present invention experimentally compares the accuracy and quantization-friendliness of various existing lightweight feature extraction backbone networks, and selects the feature extraction backbone network GhostNet that is most suitable for the low-precision fixed-point twin network target tracking method; with respect to the feature fusion method, the present invention compares the quantization loss and tracking accuracy of various lightweight feature fusion methods, and proposes a new lightweight and quantization-friendly feature extraction method GhostAttention; with respect to the branch prediction method, the present invention also compares the quantization loss and tracking accuracy of existing network structure design methods, and selects the best branch prediction structure corner point heat map. Finally, a simple, lightweight, and quantization-friendly twin network target tracking algorithm is proposed. The algorithm has low parameter and computational complexity, is quantization-friendly, and has a small difference in performance indicators before and after quantization, making it suitable for deployment on mobile computing platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a structural diagram of GhostAttention in an embodiment of the present invention.

[0035] Figure 2 This is a framework diagram of the target tracking method in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] In order to illustrate the design of quantization friendliness, the present invention compares and analyzes the quantization friendliness and tracking performance of the three components of the twin network target tracking framework, namely the feature extraction backbone network, the feature fusion method and the branch prediction method. The present invention is further explained below with reference to charts and specific embodiments.

[0038] Table 1: Accuracy of common lightweight image classification networks after quantization

[0039]

[0040]

[0041] Table 1 shows the quantized accuracy of the current mainstream lightweight feature extraction backbone networks. To fairly evaluate the performance and quantization loss of different lightweight feature extraction backbone networks, the number of multiplication-add-accumulation operations (MACs) is used as the computational cost estimation standard, and the width scaling factor is used to scale each model channel. Each backbone network is trained using the ImageNet dataset. The training process is 60 cycles in total, with a learning rate of 1e -3 , after 42 cycles it dropped to 1e -4 The network weights are quantized using the minmax method, and the network activation values ​​are quantized using the percentile method. For image classification tasks, the final result is one-hot, and the quantitative confidence obtained by the network estimation is converted into a qualitative classification result. Therefore, quantization loss does not necessarily affect the quantized inference results. Visual object tracking is a numerical estimation of the state, and quantization loss will directly affect the inference results. Therefore, when selecting the feature extraction backbone network, it is also necessary to analyze the quantization loss of feature extraction.

[0042] Table 2: Quantization loss SNR of feature extraction backbone network at each feature layer

[0043]

[0044] Table 2 shows the quantization loss of the above-mentioned backbone network at each feature layer. Generally speaking, the deep features extracted from the backbone network include four layers: Layer1, Layer2, Layer3, and Layer4. For the twin network visual target tracking algorithm, the Layer3 feature layer is often used. The quantization loss of MobileNet v2, MobileNet v3, and ShuffleNet v1 at Layer3 exceeds 0.1. Among them, the quantization loss of the ShuffleNet v1 network increases sharply in the convolution layer before each channel shuffling operation, so it is believed that the channel shuffling related structure has low quantization friendliness. In summary, GhostNet has higher classification accuracy, indicating that its feature extraction capability is stronger, and GhostNet has smaller quantization loss at the Layer3 feature layer, so GhostNet is selected as the feature extraction backbone network of the quantization-friendly twin network target tracking algorithm proposed in the present invention.

[0045] The GhostAttention structure designed by the present invention is used as a feature fusion structure. Figure 1 As shown, the specific structural diagram is as follows: the search area features and template features are subjected to depthwise separable convolution respectively to generate the corresponding output features Figure 1 and output features Figure 2 For the output features Figure 1 , differentiated into two branches, namely Figure 1 Branch 1 and branch 2 are shown. In branch 2, Cross Attention is used to fuse the output features. Figure 1 and output features Figure 2 , that is, the output feature Figure 1 As query, output features Figure 2 As key and value, by calculating the similarity between query and key, we get the weight coefficient of key corresponding to value, and then perform weighted summation on the value to get the fusion feature Figure 2 In branch 1, the “shadow” feature of the search area feature extracted by the depthwise separable convolution structure is retained, and this “shadow” feature is combined with the fusion feature through the splicing operation. Figure 2 Directly fuse in the channel dimension to obtain the final feature fusion result feature fusion Figure 1 , which is the fusion feature map f zx .

[0046] Table 3: Performance comparison of different feature fusion structures before and after quantization on the LaSOT dataset

[0047]

[0048] As shown in Table 3, the designed feature fusion structures are Concatenate, DW-Correlation, Attention, FiLM, and GhostAttention target tracking networks. The training is performed for a total of 90 cycles, each cycle includes 60,000 random samplings. The learning rate is 1e -3 , after 30 cycles it drops to 2e -4 , after 60 cycles it dropped to 5e -5 The network weight quantization adopts the minmax method, the network activation value quantization adopts the percentile method, and the test set selects LaSOT.

[0049] Parameters P, AUC, P in the table norm The three dimensions required to evaluate algorithm performance on the LaSOT test set are explained as follows:

[0050] P is the precision, which refers to the percentage of video frames with an error distance D less than 20 in the total number of frames:

[0051] Where F represents all video frames, F i represents the i-th video frame, and card(·) represents the number of elements in the set. When using accuracy as a metric, the difference in target size is not taken into account.

[0052] P norm Normalized accuracy is the accuracy after normalization based on the target size. Before calculating the normalized accuracy, the error distance D needs to be normalized. Normalized error distance D norm for: Where diag(·) represents the length of the diagonal. A graph based on the normalized error distance is plotted, with the x-axis representing the threshold and the y-axis representing the percentage of video frames with a normalized error distance below this threshold. The area under the curve for x∈[0,0.5] represents the normalized accuracy.

[0053] AUC is the success rate, which refers to the area under the curve of the success graph in x∈[0,1]. The success graph is drawn based on the overlap ratio between the target bounding box estimated by the algorithm and the true label bounding box, that is, IoU, and is calculated as follows: Among them, B pred and B gt where |·| represents the area of ​​the region, or the number of pixels within it. The x-axis of the success graph represents the overlap threshold, while the y-axis represents the percentage of video frames with an overlap above this threshold, also known as overlap accuracy.

[0054] These feature fusion structures are relatively simple, and the quantization loss is at a low level. And from the perspective of AUC, P, P norm The reduction in the three dimensions shows that these feature fusion structures are at similar levels. However, compared to convolution-based feature fusion methods such as splicing and FiLM, GhostAttention has stronger modeling capabilities. Compared to the Attention structure, this structure also requires less computational effort with the same number of channels. Therefore, GhostAttention was selected as the feature fusion structure for the quantization-friendly Siamese network object tracking algorithm proposed in this paper.

[0055] Table 4: Performance comparison of two branch prediction structures before and after quantization on the LaSOT dataset

[0056]

[0057] As shown in Table 4, two target tracking networks with different branch prediction structures are designed. One uses a direct estimation of the target bounding box, denoted as GhostNet+Attn+BB, and the other uses a corner heat map, denoted as GhostNet+Attn+HM. The training lasts for 90 cycles, each cycle includes 6000 random samples. The learning rate is 1e -3 , after 30 cycles it drops to 2e -4 , after 60 cycles it dropped to 5e -5 . The network weight change adopts the minmax method, and the network activation value quantization adopts the percentile method. Both branch prediction structures are relatively simple, the network depth is shallow, and the quantization loss is low. Generally speaking, in the twin network target tracking network, there is no operator that explicitly models the scale or offset, and it is difficult to fit when estimating the target offset or scale information. The corner heat map method responds to the key points in the image and does not require the specific scale information of the target. In the performance comparison of the two branch prediction structures before and after quantization on the LaSOT dataset, it can be found that the target tracking performance of the corner heat map method before and after quantization is better than that of the direct bounding box estimation method. Therefore, the corner heat map is selected as the branch prediction structure of the quantization-friendly twin network target tracking algorithm proposed in the present invention.

[0058] like Figure 2 As shown in the figure, it is a framework diagram of the twin network target tracking algorithm in the present invention, which has good quantization friendliness. Its reasoning steps are as follows:

[0059] (1) Based on the initial target annotation, the target template is captured from the initial frame of the video as the template. According to the labeled target bounding box information, the template training data is scaled to 2 and resized to 112×112;

[0060] (2) In the subsequent frames of the template, the search area is captured. The scale of the search area is 4.3 times that of the target object. For the search area training data, the labeled target bounding box information is jittered, including scale scaling and position translation. Then, based on this jittered annotation, the scale is scaled to 5 and the image is resized to 320×320.

[0061] (3) The search area and template are input into the target tracking twin network, which includes a feature extraction backbone network, a feature fusion structure, and a branch prediction structure; wherein the feature extraction backbone network adopts GhostNet to obtain the template feature f z and search area f x , the feature fusion structure adopts GhostAttention, fusion f z and f x Get the fusion feature map f zx , the branch prediction structure uses a corner heat map method to output an estimate of the target bounding box bb pred and the confidence score cls for this estimate pred ;

[0062] (4) The confidence score ranges from [0,1]. The larger the confidence score, the higher the possibility that the tracking target object described by the template exists in the search area. pred ≥90%, bb pred The target tracking Siamese network estimates the target template bounding box in the search area; otherwise, the tracking target object described by the template does not exist in the search area;

[0063] (5) Return to step (2) and process the next frame until all video frames are processed.

[0064] Among them, according to the estimation of the target bounding box bb pred and the confidence score cls for this estimate pred , calculate the bounding box estimation loss and confidence loss:

[0065] The bounding box estimation loss is: L reg (bb pred ,bb gt )=λ giou L giou (bb pred ,bb gt )+λ L1 L L1 (bb pred ,bb gt ). Among them, L giou and L L1 They are GIoU loss and L1 loss, λ giou and λL1 are the constant coefficients of GIoU loss and L1 loss respectively.

[0066] The confidence loss is: L cls (cls pred ,cls gt )=BCE(cls pred ,cls gt ), and use the binary cross entropy function BCE to calculate its loss. Among them, the true confidence cls gt In the training sample, the search area is used to determine whether the tracking target object described by the template exists. If so, cls gt =1, otherwise cls gt =0.

[0067] The detailed design of the deep neural network used is as follows:

[0068] The feature extraction backbone network uses the pre-trained GhostNet with a width scaling factor of 0.6. Layer 3 deep features are obtained, meaning the network stride is 16. The feature fusion network uses the GhostAttention approach. The branch prediction architecture uses corner heatmap estimation. For the top left and bottom right corners, a five-depth separable convolutional layer generates heatmaps.

[0069] The hyperparameters are set to: the scale of the template is twice that of the target object, the scale of the search area is 4.3 times that of the target object, the resolution of the template is 112×112, and the resolution of the search area is 240×240.

[0070] The training used the LaSOT dataset for object tracking, the GOT-10k dataset, and the COCO dataset for object detection. The network was trained for 90 cycles, each cycle consisting of 60,000 sample units, and the optimizer used was AdamW. The learning rate of the feature extraction backbone network was 1e -3 , which decreased to 2e after 30 cycles -4 , reduced to 4e after 60 cycles -5 The learning rate of the feature fusion structure and branch prediction structure is 1e -4 , which decreased to 2e after 30 cycles -5 , which decreased to 4e after 60 cycles -6 .

[0071] Neural network quantization uses 8-bit integer quantization, and the target platform is Qualcomm SNPE DSP. All operators are quantized. Network weights are quantized using the minmax method, and network activation values ​​are quantized using the percentile method. The correction dataset contains 1024 sample units, and the dataset used for quantization loss analysis also contains 1024 sample units.

[0072] To demonstrate the quantization-friendliness of this invention, we selected mainstream object tracking algorithms on the LaSOT test set. These included the classic SiamRPN algorithm proposed by Li et al., the lightweight LightTrack algorithm proposed by Yan et al., and the lightweight STARK-Lightning algorithm proposed by Yan et al. The SiamRPN algorithm has two versions, denoted as SiamRPN AlexNet and SiamRPN MobileNetV2, using AlexNet and MobileNet v2 as the backbone feature extraction networks.

[0073] Table 5: Evaluation indicators of mainstream algorithms on the LaSOT dataset

[0074]

[0075]

[0076] The number of parameters and computational complexity of other algorithms used for comparison far exceeds the quantization-friendly twin network target tracking algorithm proposed in this invention. As shown in Table 5, the quantization losses of LightTrack and STARK-Lightning in the feature extraction stage are 2.743 and 0.693, respectively, which seriously affect the subsequent reasoning process of the algorithm. The quantization loss of SiamRPN is relatively low. The quantization losses of SiamRPN AlexNet in the three stages of feature extraction backbone network, feature fusion structure and branch prediction structure are 0.060, 0.046 and 0.059, respectively. The quantization losses of SiamRPN MobileNetV2 in the three stages are 0.107, 0.067 and 0.084, respectively. Therefore, the SiamRPN algorithm can still maintain approximate tracking capabilities after quantization. The quantization losses of the quantization-friendly twin network target tracking algorithm proposed in this invention are 0.056, 0.044 and 0.025 in the three stages, respectively. The quantization loss of the algorithm is not obvious, and it is quantization-friendly. The performance of the lightweight LightTrack and STARK-Lightning before quantization exceeds the method proposed in this invention. However, their tracking performance is seriously reduced after quantization, and the algorithm fails. The classic algorithm SiamRPN has a simple structure and a small performance index difference before and after quantization, but its parameter and computational complexity are large, and the tracking performance is poor. The quantization-friendly twin network target tracking algorithm proposed in this invention has a 2.5% decrease in AUC, a 3.0% decrease in P, and a 0.5% decrease in P after quantization. norm It dropped by 2.8, but is still usable, and most performance indicators before and after quantization exceed those of the classic algorithm SiamRPN.

[0077] The above are the comparison and implementation steps of the quantization-friendly twin network target tracking method involved in the present invention.

[0078] The present invention also provides a quantization-friendly twin network target tracking device, comprising the following modules:

[0079] The target template capture module is used to capture the target template from the initial frame of the video according to the initial target annotation as a template;

[0080] The search area interception module is used to expand the scope of the search area in subsequent frames according to the state of the target in the previous frame;

[0081] The target tracking module is used to input the search area and template into the pre-trained target tracking twin network, which includes a feature extraction backbone network, a feature fusion structure and a branch prediction structure; wherein the feature extraction backbone network adopts GhostNet, the feature fusion structure adopts GhostAttention, and the branch prediction structure adopts a corner point heat map method to output an estimate of the target bounding box.pred and the confidence score cls for this estimate pred ;

[0082] The state estimation module is used to calculate the state of the target template in the current frame based on the confidence score;

[0083] Conventional confidence score cls pred When bb is greater than a certain threshold, pred The target tracking Siamese network estimates the target template bounding box in the search area; otherwise, the tracking target object described by the template does not exist in the search area;

[0084] The output module repeats the processing of the search area interception module, the target tracking module and the state estimation module in the next frame until all video frames are processed.

[0085] The specific implementation method of each module is the same as that of each step and will not be described in detail in the present invention.

[0086] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A quantization-friendly twin network target tracking method, characterized by: The following steps are involved: (1) Based on the initial target annotation, the target template is captured from the initial frame of the video as a template; (2) In subsequent frames, the search area is expanded based on the target’s state in the previous frame; (3) The search area and template are input into the pre-trained target tracking twin network, which includes a feature extraction backbone network, a feature fusion structure and a branch prediction structure; wherein the feature extraction backbone network adopts GhostNet, the feature fusion structure adopts GhostAttention, and the branch prediction structure adopts a corner point heat map method to output an estimate of the target bounding box and the confidence score for this estimate ; The processing process of the feature fusion structure is as follows: the search area feature and the template feature are subjected to depthwise separable convolution respectively to generate the corresponding output feature map 1 and output feature map 2. For the output feature map 1, two branches are differentiated, namely branch 1 and branch 2. In branch 2, Cross Attention is used to fuse the output feature map 1 and the output feature map 2, that is, the output feature map 1 is used as the query, and the output feature map 2 is used as the key and value. By calculating the similarity between the query and the key, the weight coefficient of the key corresponding to the value is obtained, and then the value is weighted and summed to obtain the fused feature map 2; in branch 1, the "shadow" feature of the search area feature extracted by the depthwise separable convolution structure is retained, namely the output feature map 1, and this "shadow" feature is directly fused with the fused feature map 2 in the channel dimension through the splicing operation to obtain the final feature fusion result, feature fusion map 1; (4) Calculate the state of the target template in the current frame based on the confidence score; Agreed confidence score When it is greater than a certain threshold, The target tracking Siamese network estimates the target template bounding box in the search area; otherwise, the tracking target object described by the template does not exist in the search area; (5) Repeat steps (2)-(4) to process the next frame until all video frames are processed.

2. The quantization-friendly twin network target tracking method according to claim 1, characterized in that: The processing process of the feature extraction backbone network is as follows: After intercepting the RGB image, the template image is obtained and search area image Extract the deep semantic information and get the template features and search area characteristics , where stride is the stride of the feature extraction backbone network, Hz and W z Indicates the height and width of the template image. H x and W x Represents the height and width of the search area image, and C represents the number of feature channels.

3. The quantization-friendly twin network target tracking method according to claim 1, characterized in that: The branch prediction structure uses a corner heat map method to output an estimate of the target bounding box. and the confidence score for this estimate ; During the pre-training target tracking twin network, the target bounding box is estimated based on the target bounding box. and the confidence score for this estimate , calculate the bounding box estimation loss and confidence loss: The bounding box estimation loss is: ,in, and They are GIoU loss and L1 loss, and are the constant coefficients of GIoU loss and L1 loss respectively; The confidence loss is: , using the binary cross entropy function BCE to calculate its loss, where the true confidence In the training sample, the search area is used to determine whether the tracking target object described by the template exists. If so, ,on the contrary .

4. The quantization-friendly twin network target tracking method according to claim 1, characterized in that: During the pre-training of the target tracking twin network, the LaSOT dataset, the GOT-10k dataset, and the COCO dataset for target detection were used. The network was trained for 90 cycles, each cycle included 60,000 sample units, and the optimizer used was AdamW. The learning rate of the feature extraction backbone network was , after 30 cycles it decreased to , after 60 cycles it decreased to ; The learning rate of the feature fusion structure and branch prediction structure is , after 30 cycles it decreased to , after 60 cycles it decreased to .

5. The quantization-friendly twin network target tracking method according to claim 1, characterized in that: In step 1, the scale of the template is twice that of the target object, the scale of the search area is 4.3 times that of the target object, the resolution of the template is 112×112, and the resolution of the search area is 240×240.

6. The quantization-friendly twin network target tracking method according to claim 1, characterized in that: During the pre-training of the target tracking twin network, the model parameters, input values, and activation values ​​are quantized, where the model parameters include weights and biases. 8-bit integer quantization is used, the quantization platform is Qualcomm SNPE's DSP, the weight quantization uses the minmax method, and the activation value quantization uses the percentile method.

7. A quantization-friendly twin network target tracking device, characterized in that: Includes the following modules: The target template capture module is used to capture the target template from the initial frame of the video according to the initial target annotation as a template; The search area interception module is used to expand the scope of the search area in subsequent frames according to the state of the target in the previous frame; The target tracking module is used to input the search area and template into the pre-trained target tracking twin network, which includes a feature extraction backbone network, a feature fusion structure, and a branch prediction structure; wherein the feature extraction backbone network adopts GhostNet, the feature fusion structure adopts GhostAttention, and the branch prediction structure adopts a corner point heat map method to output an estimate of the target bounding box and the confidence score for this estimate ; The processing process of the feature fusion structure is as follows: the search area feature and the template feature are subjected to depthwise separable convolution respectively to generate the corresponding output feature map 1 and output feature map 2. For the output feature map 1, two branches are differentiated, namely branch 1 and branch 2. In branch 2, Cross Attention is used to fuse the output feature map 1 and the output feature map 2, that is, the output feature map 1 is used as the query, and the output feature map 2 is used as the key and value. By calculating the similarity between the query and the key, the weight coefficient of the key corresponding to the value is obtained, and then the value is weighted and summed to obtain the fused feature map 2; in branch 1, the "shadow" feature of the search area feature extracted by the depthwise separable convolution structure is retained, namely the output feature map 1, and this "shadow" feature is directly fused with the fused feature map 2 in the channel dimension through the splicing operation to obtain the final feature fusion result, feature fusion map 1; The state estimation module is used to calculate the state of the target template in the current frame based on the confidence score; Agreed confidence score When it is greater than a certain threshold, The target tracking Siamese network estimates the target template bounding box in the search area; otherwise, the tracking target object described by the template does not exist in the search area; The output module repeats the processing of the search area interception module, the target tracking module and the state estimation module in the next frame until all video frames are processed.

Images