Target tracking method and system for deep twin network

Abstract

The invention discloses a target tracking method and system for a deep twin network, and relates to the technical field of computer vision and artificial intelligence, and the target tracking method comprises the following steps: obtaining a to-be-tracked video sequence and a target template image in an initial frame, a target template image and a search area image of a current frame are input into a lightweight backbone network, multi-scale feature maps are extracted respectively, the lightweight backbone network is improved, a dynamic channel pruning module is embedded, cross-layer fusion is performed on the multi-scale feature maps, and a module is updated through a self-adaptive template. In combination with a dynamic channel pruning module, redundant channels can be closed in a self-adaptive manner, the model parameter quantity and the calculation quantity are reduced, a pruning threshold is dynamically optimized through meta-learning, and it is ensured that key feature information is reserved while complexity is reduced.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of computer vision and artificial intelligence, and particularly relates to a target tracking method and system based on deep twin networks. BACKGROUND

[0002] In recent years, lightweight networks and dynamic optimization strategies have become research hotspots for improving target tracking performance. However, existing lightweight designs often sacrifice feature expression ability, resulting in decreased tracking accuracy for small targets or low-contrast scenes. On the other hand, dynamic template updating mechanisms often rely on fixed thresholds or empirical rules, and are difficult to adaptively adjust the weights of historical information and current frames, and are easily disturbed by noise. In addition, traditional multi-scale fusion methods usually use simple feature concatenation or addition, and fail to fully exploit the complementarity of shallow details and deep semantics. In view of the above problems, there is an urgent need for a target tracking scheme that balances efficiency and accuracy, reduces computational resource consumption while improving tracking stability in complex scenes through lightweight backbone networks, adaptive feature fusion and dynamic template updating. SUMMARY

[0003] The application aims to provide a target tracking method and system based on deep twin networks.

[0004] To achieve the above-mentioned purpose, the application provides the following technical scheme: a target tracking method based on deep twin networks, the target tracking method comprising the following steps:

[0005] (1) obtaining a target template image in an initial frame and a video sequence to be tracked;

[0006] (2) inputting the target template image and a search region image of a current frame into a lightweight backbone network to extract multi-scale feature maps, respectively; the lightweight backbone network is an improved MobileNetV3 embedded with a dynamic channel pruning module (DCP), and a pruning threshold of the dynamic channel pruning module is optimized through meta-learning, and a calculation formula is as follows:

[0007] T prune =σ(W·F avg +b)

[0008] Wherein, F avg is a global average pooling result of a feature map, W and b are learnable parameters, and sigma is a Sigmoid function;

[0009] (3) performing cross-layer fusion on the multi-scale feature maps to generate enhanced target features and search region features;

[0010] (4) dynamically updating a target template according to a tracking confidence through an adaptive template updating module (ATU), and an update formula is as follows:

[0011] Tnew = a*T current + (1-a)*T history

[0012] wherein, a is a dynamic weighting coefficient based on the peak intensity of the response map, T current and T history respectively represent the current frame template and the historical template;

[0013] The network is trained end-to-end by using a hybrid loss function, which includes GIoU Loss, FocalLoss and a channel attention regularization term, and the expression is:

[0014]

[0015] wherein, λ1, λ2 and λ2 are weight coefficients, and θ SE is the parameter of the lightweight SE module.

[0016] As a further scheme of the application: the dynamic channel pruning module (DCP) in step (2) specifically comprises:

[0017] (a) performing global average pooling on the output feature map of each convolutional layer to generate a channel importance score;

[0018] (b) according to the score ranking, dynamically closing the channels with scores lower than a threshold T prune , and the closing ratio is adaptively adjusted according to the complexity of the input image.

[0019] As a further scheme of the application: the multi-scale feature fusion in step (3) specifically comprises:

[0020] Feature maps are extracted at the Conv3, Conv5 and Conv7 layers of MobileNetV3 respectively, up-sampled to the same resolution by bilinear interpolation, and cross-layer fusion is performed by element-wise addition, and the formula is:

[0021]

[0022] wherein, Conv3, Conv5 and Conv7 correspond to the 3rd, 5th and 7th inverted residual blocks of the MobileNetV3 network respectively.

[0023] As a further scheme of the application: the tracking confidence is calculated by the ratio of the peak value intensity to the average intensity of the response map, and the formula is:

[0024]

[0025] wherein, R is the cross-correlation response map, max(·) is the peak value in the response map, and mean(·) is the average value of all positions in the response map.

[0026] As a further scheme of the present application: the calculation formula of the GIoU Loss is:

[0027]

[0028] Wherein, A is a prediction box, B is a real box, and C is a minimum closed region.

[0029] As a further scheme of the present application: the training data enhancement includes:

[0030] (a) Simulated occlusion: randomly add a rectangular mask in the target area, and the mask area accounts for 10%-40%;

[0031] (b) Motion blur: generate a blurred image by Gaussian kernel convolution, and the kernel size is randomly selected as 3x3 to 7x7.

[0032] The present application also proposes a target tracking system based on deep twin network, comprising the following modules:

[0033] (1) Data input module, used for acquiring a video sequence to be tracked and an initial target template image;

[0034] (2) Lightweight feature extraction module, based on improved MobileNetV3 network, integrating dynamic channel pruning unit (DCP), which automatically closes redundant channels according to the complexity of input features, and the pruning threshold calculation method is consistent with step (2);

[0035] (3) Multi-scale feature fusion module, which aggregates shallow detail features and deep semantic features through cross-layer connection and up-sampling operation, and the fusion method corresponds to step (3) of claim 1;

[0036] (4) Adaptive template update module (ATU), which dynamically adjusts the weight of historical template based on response map confidence, and the update formula is consistent with step (4);

[0037] (5) Target positioning module, which calculates the target position by weighted cross-correlation, and outputs the final tracking box by non-maximum suppression (NMS).

[0038] As a further scheme of the present application: in the lightweight feature extraction module, the inverted residual structure (Inverted Residual) of MobileNetV3 is embedded with Lite-SE module, and the channel compression ratio is 4:1.

[0039] As a further scheme of the present application: the adaptive template update module (ATU) supports short-term memory cache, which retains the high-confidence templates of the last n frames for re-identification after target occlusion.

[0040] As a further scheme of the present application: the target positioning module adopts a hierarchical prediction strategy:

[0041] (a) Coarse positioning: screening high response areas through a lightweight SE module;

[0042] (b) Fine tuning: pixel-level regression in the coarse positioning area, with the formula being:

[0043] (x, y, w, h) = MLP (F crop )

[0044] Where F crop is the cropped local feature.

[0045] Compared with the prior art, the beneficial effects of the present application are:

[0046] 1. The present application, through the improved MobileNetV3 lightweight backbone network combined with the dynamic channel pruning module (DCP), can adaptively close redundant channels, reduce model parameter quantity and calculation quantity, and dynamically optimize the pruning threshold through meta-learning, ensuring that key feature information is retained while reducing complexity;

[0047] 2. The present application adopts a multi-scale feature fusion technology to aggregate shallow detail features and deep semantic information, effectively dealing with target deformation, occlusion and scale changes, and the adaptive template update module (ATU) dynamically adjusts the historical template weight based on the response map confidence, maintaining the timeliness of the template through an exponential moving average strategy when the target is temporarily occluded or the appearance changes;

[0048] 3. The present application combines a hybrid loss function of GIoU Loss, Focal Loss and channel attention regularization, taking into account target positioning accuracy and classification robustness, alleviating the imbalance between positive and negative samples, and the hierarchical prediction strategy screens high response areas through a lightweight SE module, then performs pixel-level regression on local features, realizing step-by-step optimization from coarse positioning to fine tuning. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of the target tracking method of the deep twin network of the present application;

[0050] Figure 2 is a flowchart of the target tracking system of the deep twin network of the present application. DETAILED DESCRIPTION

[0051] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings, and it should be noted that the description of these embodiments is used to help understand the present application and does not constitute a limitation on the present application.

[0052] Furthermore, the technical features involved in each embodiment of the application described below can be combined with each other as long as there is no conflict.

[0053] Please refer to the accompanying Figure 1 The target tracking method of the application is a deep twin network, and the target tracking method comprises the following steps:

[0054] (1) obtaining a target template image in an initial frame and a video sequence to be tracked;

[0055] (2) inputting the target template image and the search region image of the current frame into a lightweight backbone network to extract multi-scale feature maps respectively; the lightweight backbone network is an improved MobileNetV3 embedded with a dynamic channel pruning module (DCP), and the pruning threshold of the dynamic channel pruning module (DCP) is optimized through meta-learning, and the calculation formula is:

[0056] T prune =σ(W·F avg +b)

[0057] Wherein, F avg is the global average pooling result of the feature map, W and b are learnable parameters, and sigma is a Sigmoid function;

[0058] (3) performing cross-layer fusion on the multi-scale feature maps to generate enhanced target features and search region features;

[0059] (4) dynamically updating the target template according to the tracking confidence through an adaptive template updating module (ATU), and the update formula is:

[0060] T new =α·T current +(1-α)·T history

[0061] Wherein, alpha is a dynamic weighting coefficient based on the peak intensity of the response map, T current and T history represent the current frame template and the historical template respectively;

[0062] The network is trained end-to-end using a hybrid loss function, and the loss function includes GIoU Loss, Focal Loss and a channel attention regularization term, and the expression is:

[0063]

[0064] Wherein, lambda1, lambda2 and lambda2 are weight coefficients, and theta SE is the parameter of the lightweight SE module.

[0065] In one embodiment of the application, the dynamic channel pruning module (DCP) in step (2) comprises:

[0066] (a) Global average pooling on the output feature map of each convolutional layer to generate channel importance scores;

[0067] (b) According to the score ranking, dynamically close the channel whose score is lower than the threshold T prune , and the closing ratio is adaptively adjusted by the input image complexity.

[0068] In an embodiment of the present application: the multi-scale feature fusion in step (3) is specifically:

[0069] Extract the feature maps at the Conv3, Conv5 and Conv7 layers of MobileNetV3 respectively, upsample them to the same resolution by bilinear interpolation, and perform cross-layer fusion by element-wise addition, with the formula being:

[0070]

[0071] Wherein, Conv3, Conv5 and Conv7 correspond to the 3rd, 5th and 7th inverted residual blocks of the MobileNetV3 network.

[0072] In an embodiment of the present application: the tracking confidence is calculated by the ratio of the peak intensity to the average intensity of the response map, with the formula being:

[0073]

[0074] Wherein, R is the cross-correlation response map, max(·) is the peak value in the response map, and mean(·) is the average value of all positions in the response map.

[0075] In an embodiment of the present application: the calculation formula of GIoU Loss is:

[0076]

[0077] Wherein, A is the predicted box, B is the real box, and C is the minimum closed region.

[0078] In an embodiment of the present application: the training data enhancement includes:

[0079] (a) Simulated occlusion: randomly add a rectangular mask in the target area, and the mask area accounts for 10%-40%;

[0080] (b) Motion blur: generate a blurred image by Gaussian kernel convolution, and the kernel size is randomly selected from 3x3 to 7x7.

[0081] Please refer to the accompanying Figure 2 , the present application also proposes a target tracking system based on deep twin network, which includes the following modules:

[0082] (1) data input module, for obtaining a video sequence to be tracked and an initial target template image;

[0083] (2) lightweight feature extraction module, based on an improved MobileNetV3 network, integrating a dynamic channel pruning unit (DCP), which adaptively closes redundant channels according to the complexity of input features, and the pruning threshold calculation method is consistent with step (2);

[0084] (3) multi-scale feature fusion module, which aggregates shallow detail features and deep semantic features through cross-layer connection and up-sampling operation, and the fusion method corresponds to step (3) of claim 1;

[0085] (4) adaptive template update module (ATU), which dynamically adjusts the weight of the historical template based on the response map confidence, and the update formula is consistent with step (4);

[0086] (5) target positioning module, which calculates the target position using weighted cross-correlation and outputs the final tracking frame through non-maximum suppression (NMS).

[0087] In an embodiment of the present application: in the lightweight feature extraction module, the Inverted Residual structure of MobileNetV3 is embedded with a Lite-SE module, and the channel compression ratio is 4:1.

[0088] In an embodiment of the present application: the adaptive template update module (ATU) supports short-term memory caching, retaining the high-confidence templates of the last n frames for re-identification after target occlusion.

[0089] In an embodiment of the present application: the target positioning module adopts a hierarchical prediction strategy:

[0090] (a) coarse positioning: selecting high response areas through a lightweight SE module;

[0091] (b) fine tuning: pixel-level regression within the coarse positioning area, the formula is:

[0092] (x,y,w,h)=MLP(F crop )

[0093] Where F crop is the cropped local feature.

[0094] Example one, vehicle tracking in urban traffic monitoring:

[0095] 1. Data input: obtain the urban intersection monitoring video sequence, and select the white car to be tracked in the initial frame as the target template (size 128x128 pixels).

[0096] 2. Feature extraction:

[0097] The improved MobileNetV3 network (embedded dynamic channel pruning module DCP) is used to input the template image and the search area (256x256 pixels) of the current frame.

[0098] The DCP module pruning threshold is optimized by meta-learning, the initial pruning ratio is 30%, and the number of reserved channels is 64.

[0099] 3. Multi-scale fusion:

[0100] The feature maps of Conv3, Conv5 and Conv7 layers (resolution is 32x32, 16x16 and 8x8 respectively) are extracted, upsampled to 32x32 by bilinear interpolation, and element-wise added.

[0101] The number of channels of the fused feature map is 256, which is used to calculate the cross-correlation response map.

[0102] 4. Template update:

[0103] The ratio of peak intensity to average intensity of the response map is 4.2 (confidence > 3.5), triggering template update, dynamic weighting coefficient α = 0.8.

[0104] Update formula: T new = 0.8xT current + 0.2xT history .

[0105] 5. Target positioning:

[0106] The highest response area (coordinate offset error < 5 pixels) is screened out by coarse positioning, and the final tracking box (GIoU = 0.92) is obtained by the fine-tuning module through MLP regression.

[0107] 6. Training enhancement:

[0108] Data augmentation includes 30% area occlusion and 5x5 Gaussian blur, and the loss function weight is set as λ1 = 1.0 (GIoU Loss) and λ2 = 0.5 (Focal Loss).

[0109] Example two, moving pedestrian tracking in UAV aerial photography:

[0110] 1. Dynamic pruning optimization:

[0111] The input video resolution is 640x480, and the target template is a dynamic pedestrian (96x96 pixels).

[0112] The DCP module adjusts the pruning ratio (15%-40%) adaptively according to the image complexity, and the pruning threshold W and b are updated by online meta-learning.

[0113] 2. Multi-scale feature enhancement:

[0114] Fusion layer increases Conv9 features (resolution 4x4), up-sampling and concatenating with Conv3, Conv5, and Conv7 features (channel number 384).

[0115] Introducing a lightweight SE module (compression ratio 4:1) to enhance channel attention.

[0116] 3. Anti-occlusion strategy:

[0117] When the response map confidence drops to 2.8 (peak / mean ratio < 3.0), enable short-term memory cache, load the last 5 frames of high-confidence templates.

[0118] Update coefficient adjustment to α = 0.6 to improve historical template weight.

[0119] 4. Positioning accuracy verification:

[0120] In fast motion scenarios, coarse positioning error < 10 pixels, refined GIoU improves to 0.85, and FPS reaches 45 (NVIDIA Jetson Xavier platform)

[0121] Example Three, Wildlife Tracking in Low Light Environment:

[0122] 1. Low light adaptability:

[0123] Input infrared camera video, target template is a deer active at night (160x160 pixels).

[0124] Dynamic pruning module closes redundant channel ratio to 50%, retaining key texture features (channel number 32).

[0125] 2. Hybrid loss optimization:

[0126] Introducing channel attention regularization term (λ3 = 0.1) to constrain lightweight SE module parameters θ SE Prevent overfitting.

[0127] Stack motion blur (7x7 Gaussian kernel) and 40% occlusion enhancement during training.

[0128] 3. Cross-layer fusion improvement:

[0129] Weighted fusion (Conv3 weight 0.5, Conv5 weight 0.3, Conv7 weight 0.2) to improve shallow layer detail contribution.

[0130] Response map peak / mean ratio stable above 3.5, tracking box jitter rate reduced to 2%.

[0131] 4. System performance:

[0132] In the tree shelter scene, the target is successfully re-identified and the tracking is resumed (the confidence returns to 4.0), and the average tracking accuracy (Success Rate) reaches 87.6% (OTB-100 benchmark test).

[0133] Although the present application is disclosed with reference to the preferred embodiments, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, any modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solutions of the present application, fall within the protection scope defined by the claims of the present application.

Claims

1. A method for object tracking in a deep siamese network, the method comprising: The target tracking method comprises the following steps: (1) obtaining a target template image in an initial frame and a video sequence to be tracked; (2) inputting the target template image and a search region image of a current frame into a lightweight backbone network to respectively extract multi-scale feature maps; the lightweight backbone network is an improved MobileNetV3 embedded with a dynamic channel pruning module, and a pruning threshold of the dynamic channel pruning module is optimized through meta-learning, and a calculation formula of the pruning threshold is: T prune = σ(W·F avg +b) where F avg is the global average pooling result of the feature map, W and b are learnable parameters, and σ is a sigmoid function. (3) performing cross-layer fusion on the multi-scale feature maps to generate enhanced target features and search region features; (4) dynamically updating the target template according to a tracking confidence through an adaptive template updating module, and an update formula is: T new = a · T current + (1 - a) · T history wherein a is a dynamic weighting coefficient based on the peak intensity of the response map, T current and T history respectively represent the current frame template and the historical template; An end-to-end training is performed on the network by using a hybrid loss function, and the loss function comprises a GIoU Loss, a Focal Loss and a channel attention regularization term, and an expression of the loss function is: wherein λ1, λ2, and λ2are weight coefficients, θ SE are parameters of the light-weight SE module. 2.The target tracking method of deep twin networks according to claim 1, characterized in that: The dynamic channel pruning module in the step (2) comprises the following steps: (a) performing global average pooling on output feature maps of each convolutional layer to generate channel importance scores; (b) dynamically close channels with scores below a threshold T according to the ranking, the proportion of closure being adaptively adjusted by the input image complexity. prune of the channels with scores below a threshold T according to the ranking, the proportion of closure being adaptively adjusted by the input image complexity. 3.The target tracking method of deep twin networks according to claim 1, characterized in that: The multi-scale feature fusion in the step (3) comprises the following steps: Feature maps of Conv3, Conv5 and Conv7 in the MobileNetV3 are extracted, up-sampling is performed on the feature maps by using a bilinear interpolation to the same resolution, and cross-layer fusion is performed by using element-wise addition, and a formula is: Wherein, Conv3, Conv5 and Conv7 respectively correspond to the third, fifth and seventh inverted residual blocks of the MobileNetV3 network. 4.The target tracking method of deep twin networks according to claim 1, characterized in that: The tracking confidence is calculated by using a ratio of a peak value intensity to an average intensity of a response map, and a formula is: Wherein, R is a cross-correlation response map, max(·) is a peak value in the response map, and mean(·) is an average value of all positions in the response map.

5. The method of claim 1, wherein: A calculation formula of the GIoU Loss is: Wherein, A is a predicted frame, B is a real frame, and C is a minimum closed region.

6. The method of claim 1, wherein: The training data enhancement comprises the following steps: (a) simulating occlusion: a rectangular mask is randomly added to a target region, and an area ratio of the mask is 10%-40%; (b) motion blur: a blurred image is generated by using a Gaussian kernel convolution, and a kernel size is randomly selected as 3*3 to 7*7.

7. A deep siamese network based object tracking system for implementing the object tracking method of claim 1, characterized by, The system comprises the following modules: (1) a data input module, used for obtaining a video sequence to be tracked and an initial target template image; (2) a lightweight feature extraction module, based on an improved MobileNetV3 network, integrated with a dynamic channel pruning unit, the dynamic channel pruning unit is used for adaptively closing redundant channels according to complexity of input features, and a calculation manner of a pruning threshold of the dynamic channel pruning unit is consistent with the step (2); (3) a multi-scale feature fusion module, used for aggregating shallow detail features and deep semantic features by using cross-layer connection and up-sampling operations, and a fusion manner is consistent with the step (3) of claim 1; (4) an adaptive template updating module, used for dynamically adjusting a historical template weight based on a response map confidence, and an update formula is consistent with the step (4); (5) a target positioning module, used for calculating a target position by using weighted cross-correlation, and outputting a final tracking frame by using non-maximum suppression. 8.The target tracking system based on deep twin network according to claim 7, characterized in that: In the lightweight feature extraction module, an inverted residual structure of the MobileNetV3 is embedded with a lightweight SE module, and a channel compression ratio of the lightweight SE module is 4:

1. 9.The target tracking system based on deep twin network according to claim 7, characterized in that: The adaptive template updating module supports short-term memory cache, and retains the last n frames of high-confidence templates for re-identification after target occlusion. 10.The target tracking system based on deep twin network according to claim 7, characterized in that: The target positioning module adopts a hierarchical prediction strategy: (a) Coarse positioning: filtering high-response areas through a lightweight SE module; (b) Fine tuning: pixel-level regression within the coarse positioning area, with the formula being: (x, y, w, h) = MLP(F crop ) where F crop is the cropped local feature.

Images