Dynamic computation allocation method for visual object tracking based on scene complexity perception

Abstract

The application discloses a visual target tracking dynamic calculation distribution method based on scene complexity perception, and relates to the technical field of computer vision and artificial intelligence. The method first constructs a multi-layer visual Transformer backbone network containing a fixed layer and dynamic layers. A scene complexity analyzer is activated after the first dynamic layer to pool, calculate the similarity of and enhance the features of a template and a search area, and to predict the exit scores of the dynamic layers. Then, whether to terminate the inference in advance is dynamically determined according to the comparison between the scores and a preset threshold. Meanwhile, a hierarchical distillation method is adopted to transfer the knowledge of a teacher model to multiple dynamic layers of a student model to improve the prediction accuracy of the early layers. The application realizes adaptive perception of scene complexity and dynamic distribution of computing resources, achieves a balance between high accuracy and high real-time performance on resource-limited devices, and significantly improves the inference efficiency of visual target tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, and particularly relates to a visual target tracking dynamic calculation allocation method based on scene complexity perception, which is suitable for real-time visual target tracking scenes of resource-limited devices (such as embedded terminals, unmanned aerial vehicles, and monitoring cameras). BACKGROUND

[0002] Visual object tracking (VOT) is a basic task in the field of computer vision, which aims to estimate the position and state of the target in the subsequent frames in real time and accurately according to the boundary box label of the target in the initial frame of the video. As a key supporting technology in many fields such as automatic driving, intelligent monitoring, augmented reality, and unmanned aerial vehicle inspection, the technology has irreplaceable application value in industrial production, public safety, and consumer electronics scenes.

[0003] In the development history of visual object tracking technology, researchers have always been committed to solving core challenges such as large occlusion, fast motion, target severe deformation, and similar object interference, while pursuing the balance between high precision and high real-time performance. Early tracking technologies are represented by Siamese Network, which uses relatively shallow network architectures such as AlexNet and VGG as the backbone network. Although this kind of method can achieve high inference speed, it is limited by the limited feature expression ability of shallow networks, and is prone to tracking drift in complex scenes, making it difficult to meet the demand for high-precision tracking.

[0004] To improve tracking accuracy, subsequent research turned to deep convolutional neural networks (CNN), such as ResNet and MobileNet, which enhanced the hierarchical nature and abstract ability of features through the stacking of residual blocks and other structures, significantly improving tracking performance in complex scenes. However, the hierarchical stacking design of deep networks leads to an exponential increase in computational overhead, a significant increase in inference delay, and a serious decline in real-time performance, making it difficult to adapt to the deployment needs of resource-limited devices.

[0005] In recent years, tracking based on Transformer (such as TransT, STARK, and MixFormer) has further broken through the tracking accuracy bottleneck in complex scenes with the powerful long-range dependency capturing ability of the self-attention mechanism. However, the computational complexity of the self-attention mechanism grows quadratically with the number of input tokens ( , For the number of tokens), and this kind of model usually uses a multi-layer stacking design of more than 12 layers, resulting in huge overall computational overhead, far exceeding the hardware carrying capacity of resource-limited devices such as embedded terminals, unmanned aerial vehicles, and monitoring cameras, greatly limiting their widespread application in real-world scenarios.

[0006] To solve the problem of efficient calculation, dynamic inference technology (such as BranchyNet and MSDNet) emerges as the times require. This kind of technology dynamically allocates computing resources in a data-dependent manner, and terminates the inference in advance for "simple" inputs through an "early exit" mechanism, thereby improving the calculation efficiency while ensuring the accuracy. However, the existing dynamic inference technology mainly targets tasks with simple output forms such as image classification, and is difficult to directly adapt to the compound output requirements of "classification + regression" in the visual target tracking task, especially unable to design a reliable early exit judgment standard for the bounding box regression task, and cannot be directly applied to the visual target tracking scene.

[0007] In summary, the existing visual target tracking technology has the following core problems: on the one hand, although deep networks (including CNN and Transformer) can improve tracking accuracy, the "one-size-fits-all" calculation mode (all network layers are activated regardless of scene complexity) under the static calculation framework leads to serious calculation redundancy and insufficient real-time performance; on the other hand, the existing dynamic inference technology cannot adapt to the compound task characteristics of visual target tracking, and lacks an effective early exit mechanism for bounding box regression. Therefore, how to design an efficient tracking method that can adaptively perceive scene complexity, dynamically adjust computing resource allocation, and adapt to the bounding box regression task, to achieve a balance between high accuracy and real-time performance on resource-constrained devices, has become a key technical problem to be solved in the current visual target tracking field. SUMMARY

[0008] To this end, an embodiment of the present application provides a visual target tracking dynamic calculation allocation method based on scene complexity perception, which is used to solve the problems in the prior art that the visual target tracker uses a static calculation framework, leading to calculation redundancy and insufficient real-time performance, and the dynamic inference technology cannot adapt to the "classification + regression" requirements of visual target tracking and lacks a reliable early exit standard, thereby failing to achieve a balance between high accuracy and real-time performance on resource-constrained devices.

[0009] To solve the above technical problems, an embodiment of the present application provides a visual target tracking dynamic calculation allocation method based on scene complexity perception, which comprises the following steps:

[0010] Step S1: obtaining a target template image of a first frame and a search area image of a subsequent frame in a video sequence, the template image being obtained by cropping from the center of the target with a fixed expansion ratio, and the search area image being obtained by expanding from the center of the tracking result of the last frame; performing image block embedding processing on the template image and the search area image to obtain template tokens and search area tokens, and setting four special learnable prediction tokens, the prediction tokens corresponding to the top, left, bottom and right coordinates of the target bounding box, respectively;

[0011] Step S2: a backbone network based on a multi-layer visual Transformer is constructed, the backbone network is composed of a plurality of mixed attention layers, the mixed attention layers are divided into fixed layers and dynamic layers, the fixed layers unconditionally process all input tokens to extract low-level basic features, the dynamic layers are arranged after the fixed layers and are used to realize selective termination of an inference process;

[0012] Step S3: the mixed attention layers process template tokens, search region tokens and prediction tokens through two operations of target aggregated self-attention and template and search region fusion cross-attention, and complete feature interaction and enhancement.

[0013] Step S4: after the first dynamic layer is calculated, a scene complexity analyzer is activated, spatial dimension pooling, similarity calculation and feature enhancement processing are performed on template features and search region features, and an exit score corresponding to each dynamic layer is output through a multi-layer perception, the exit score quantifies the applicability of the output result of the corresponding dynamic layer.

[0014] Step S5: the exit score corresponding to each dynamic layer is compared with a preset threshold, if the exit score is greater than or equal to the preset threshold, the inference is immediately terminated, and the output of the current dynamic layer is taken as a tracking result; if the exit score is less than the preset threshold, the next dynamic layer is continued to be executed, and the score judgment process is repeated until the threshold condition is met or all dynamic layers are executed.

[0015] Step S6: a hierarchical distillation method is used to transfer the knowledge of the last layer of a teacher model to a plurality of dynamic layers of a student model, the student model is optimized through logical loss, relationship loss and feature imitation loss, and the prediction accuracy of the dynamic layers is improved.

[0016] Step S7: the output of the dynamic layer meeting the condition is embedded into an input of a lightweight multi-layer perception prediction head to obtain target bounding box coordinates, the center of a search region of the next frame is updated, and visual target tracking of continuous frames is realized.

[0017] Preferably, in step S1, the size of the template image is , and the size of the search region image is , wherein represents the height dimension of the template image, represents the width dimension of the template image, represents the height dimension of the search region image, represents the width dimension of the search region image, and 3 represents the number of channels; the prediction token performs feature interaction with the template token and the search region token through a cross-attention mechanism, and the output of the prediction token is independently embedded into an input of a shared lightweight multi-layer perception prediction head.

[0018] Preferably, in step S3, the target aggregated self-attention operation is specifically:

[0019] The template token is projected into the query, key and value space, and the internal dependency feature of the template token is calculated by an attention formula, and the attention formula is:

[0020] ,

[0021] wherein, , , are the projection results of the query, key and value matrices of the template token, is the transpose matrix of , is the feature dimension, is the attention function, is the normalization function.

[0022] Preferably, in step S3, the cross-attention operation of the template and the search area is specifically:

[0023] The template token is taken as a key-value pair, and the search area token and the predicted token are projected into the query, key and value space, and the target clue injection is realized by a cross-attention formula, and the cross-attention formula is:

[0024] ,

[0025] wherein, , are the projection results of the search area token and the predicted token, is the key-value pair matrix of the template token, , are the search area features and the predicted token features after cross-attention processing, is the cross-attention function.

[0026] Preferably, in step S4, the activation scene complexity analyzer performs spatial dimension pooling, similarity calculation and feature enhancement processing on the template features and the search area features, and the process of outputting the exit scores of the corresponding dynamic layers through a multi-layer perception includes:

[0027] Step S41: calculating the template features and the search area features through the first dynamic layer:

[0028] ,

[0029] wherein, For dynamic layer computing units, For template token, For the search area token;

[0030] Step S42: For template features Spatial dimension pooling is performed to obtain the template global features. :

[0031] ,

[0032] in, This represents a spatial dimension pooling operation;

[0033] Step S43: Calculate global features of the template using dot product Features of the search area Similarity graph Furthermore, feature enhancement of the search region is achieved through element-wise multiplication, resulting in enhanced search region features. :

[0034] , ;

[0035] in, Represents the dot product. This represents element-wise multiplication.

[0036] Step S44: Enhance the features of the search region Input multilayer perceptron, output The range is in Exit score:

[0037] ,

[0038] in, This represents the set of exit scores corresponding to each dynamic layer; The number of dynamic layers; It is a multilayer perceptron.

[0039] Preferably, in step S6, the logistic loss is calculated using KL divergence to align the bounding box prediction distributions of the teacher model and the student model. The loss formula is:

[0040] ,

[0041] in, For logical loss; This represents the KL divergence, used to calculate the difference between two probability distributions; This is the output of the last layer of the teacher model; For the student model Each dynamic layer output This represents the number of dynamic layers.

[0042] Preferably, in step S6, the relationship loss is constrained by KL divergence to determine the QKV relationship between the student model and the teacher model. The formula for calculating the QKV relationship of the teacher model is:

[0043] ,

[0044] The formula for calculating the QKV relationship in the student model is:

[0045] ,

[0046] The formula for relationship loss is:

[0047] ,

[0048] in, and The attention relationship matrix representing the teacher model and the student model, with superscript... and The following table represents the teacher model and the student model, respectively. The table below indicates the type of attention relationship. and These represent the teacher models. The first layer and student model layer; This is the normalization function; and The header size represents the relationship between the teacher model and the student model; The teacher model is represented by the first... In the layer, the type is Query / key / value matrix; The teacher model is represented by the first... layer Transpose of a type matrix; The student model is represented by the first... In the layer, the type is Query / key / value matrix; The student model is represented by the first... layer Transpose of a type matrix; Indicates a loss of relationship; This represents the KL divergence, used to calculate the difference between two probability distributions; A set of layers for teacher-student matching.

[0049] Preferably, in step S6, the feature imitation loss uses L2 loss, which is applied only to the last layer of the teacher and student models, and the loss formula is as follows: ,in is a feature imitation loss, is an L2 loss, is a feature vector of the layer of the student model, is a feature vector of the layer of the teacher model; a model with different numbers of layers is obtained through model pruning distillation, and the loss of pruning distillation includes feature loss , relationship loss and logic loss :

[0050] ,

[0051] ,

[0052] ,

[0053] wherein, is a set of layers matched by the teacher and the student; is a feature vector of the layer of the student model; represents the KL divergence, which is used to calculate the difference between two probability distributions; represents an attention relationship matrix of type in the layer of the pruned student model; represents an attention relationship matrix of type in the layer of the teacher model; represents a bounding box prediction distribution output by the layer of the pruned student model; represents a bounding box prediction distribution output by the layer of the teacher model.

[0054] The embodiment of the application also provides a visual target tracking dynamic calculation allocation system based on scene complexity perception, which is used to realize the visual target tracking dynamic calculation allocation method based on scene complexity perception.

[0055] An input preprocessing and token construction module is used to obtain a target template image of a first frame and a search area image of a subsequent frame in a video sequence, the template image is obtained by cutting from the center of the target through a fixed expansion ratio, and the search area image is obtained by expanding from the center of the last tracking result; the template image and the search area image are subjected to image block embedding processing to obtain template tokens and search area tokens, and four special learnable prediction tokens are set, the prediction tokens correspond to the top, left, bottom and right coordinates of the target bounding box respectively;

[0056] A dynamic backbone network construction module is configured to construct a backbone network based on a multi-layer visual Transformer, the backbone network is composed of a plurality of mixed attention layers, the mixed attention layers are divided into fixed layers and dynamic layers, the fixed layers unconditionally process all input tokens to extract low-level basic features, and the dynamic layers are arranged after the fixed layers and are configured to realize selective termination of an inference process;

[0057] A mixed attention feature interaction module is configured to process template tokens, search region tokens and prediction tokens through two operations of target aggregated self-attention and template and search region fusion cross-attention of the mixed attention layers, so as to complete feature interaction and enhancement.

[0058] A scene complexity analysis and exit score prediction module is configured to activate a scene complexity analyzer after the first dynamic layer is calculated, perform spatial dimension pooling, similarity calculation and feature enhancement processing on template features and search region features, and output exit scores of each dynamic layer through a multi-layer perception, the exit scores quantize the applicability of the output results of the corresponding dynamic layers.

[0059] A dynamic inference termination judgment module is configured to compare the exit scores of each dynamic layer with a preset threshold, if the exit score is greater than or equal to the preset threshold, the inference is immediately terminated, and the output of the current dynamic layer is taken as a tracking result, if the exit score is less than the preset threshold, the next dynamic layer is continued to be executed, and the score judgment process is repeated until the threshold condition is met or all dynamic layers are executed.

[0060] A hierarchical distillation model optimization module is configured to use a hierarchical distillation method to transfer the knowledge of the last layer of a teacher model to a plurality of dynamic layers of a student model, optimize the student model through logical loss, relationship loss and feature imitation loss, and improve the prediction accuracy of the dynamic layers.

[0061] A tracking result output and search region update module is configured to embed the output of the dynamic layer meeting the condition into a lightweight multi-layer perception prediction head to obtain target bounding box coordinates and update the center of the search region of the next frame, so as to realize visual target tracking of consecutive frames.

[0062] The embodiment of the present application also provides an electronic device, which comprises a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is configured to store instructions, and the processor is configured to execute the instructions stored in the memory to realize the scene complexity perception based visual target tracking dynamic calculation and distribution method.

[0063] From the above technical solutions, the present application has the following beneficial effects:

[0064] (1) Significantly improve the reasoning speed and reduce the computational overhead: By dividing the backbone network into fixed layers and dynamic layers, and introducing a scene complexity analyzer to predict the exit score in real time, the system can terminate redundant calculations in simple scenes and achieve adaptive computation allocation. Experiments show that, under the premise of maintaining the accuracy basically unchanged, compared with the static baseline model, the method can achieve a maximum speedup of 55% on GPU (such as DCATrack-L12 reaching 171 FPS), greatly improving the real-time performance of tracking.

[0065] (2) Maintain the tracking accuracy basically unchanged and ensure the robustness in complex scenes: During dynamic reasoning, the teacher model deep knowledge is transferred to multiple dynamic layers of the student model through hierarchical distillation method, and combined with logical loss, relationship loss and feature imitation loss for joint optimization, which effectively enhances the feature representation and prediction ability of early layers. On public datasets such as LaSOT, while significantly improving the speed, the tracking accuracy (AUC) decreases by a small amount (≤0.7%), and the tracking performance remains stable in complex scenes.

[0066] (3) Provide flexible and scalable multi-version models to adapt to different resource constraints: Support the generation of lightweight versions (such as 8 layers, 4 layers) from full models (such as 12 layers) through model pruning distillation strategy, forming a multi-level speed-accuracy balanced model series. Different versions of the model can be used for high, medium and low power devices (such as servers, embedded terminals, mobile devices), and can achieve significant efficiency improvement on GPU and CPU, with good actual deployment flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly described below. The features and advantages of the present application can be more clearly understood by referring to the drawings, which are schematic and should not be understood as any limitation to the present application. Those skilled in the art can obtain other drawings according to these drawings without creative labor. Among them:

[0068] Figure 1 is a flowchart of a visual target tracking dynamic computation allocation method based on scene complexity perception provided by the present application;

[0069] Figure 2 is an architectural schematic diagram of the visual target tracking dynamic computation allocation method based on scene complexity perception of the present application;

[0070] Figure 3 is a hierarchical distillation architectural schematic diagram of the present application;

[0071] Figure 4is a block diagram of a visual target tracking dynamic calculation allocation system based on scene complexity perception provided by the present application. DETAILED DESCRIPTION

[0072] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0073] Embodiment one: in order to solve the problems that the visual target tracker in the prior art adopts a static calculation framework, leading to calculation redundancy and insufficient real-time performance, and that the dynamic reasoning technology is difficult to adapt to the "classification + regression" requirements of visual target tracking and lacks reliable early exit criteria, and thus cannot achieve a balance between high precision and real-time performance on resource-limited devices, as shown in Figure 1 and Figure 2 The present application proposes a visual target tracking dynamic calculation allocation method based on scene complexity perception, which comprises the following steps:

[0074] Step S1: obtaining a target template graph of a first frame and a search area graph of a subsequent frame in a video sequence, the template graph being obtained by fixedly expanding and cutting from the center of the target, and the search area graph being obtained by expanding from the center of the last frame tracking result; performing image block embedding processing on the template graph and the search area graph to obtain template tokens and search area tokens, and setting four special learnable prediction tokens, the prediction tokens respectively corresponding to the top, left, bottom and right coordinates of the target bounding box;

[0075] Step S2: constructing a backbone network based on a multi-layer visual Transformer, the backbone network being composed of multiple mixed attention layers, the mixed attention layers being divided into fixed layers and dynamic layers, the fixed layers unconditionally processing all input tokens to extract low-level basic features, and the dynamic layers being arranged after the fixed layers to realize selective termination of the reasoning process;

[0076] Step S3: the mixed attention layer processes the template tokens, the search area tokens and the prediction tokens through two operations of target aggregation self-attention and template and search area fusion cross-attention, to complete feature interaction and enhancement;

[0077] Step S4: after the first dynamic layer calculation is completed, activate the scene complexity analyzer, perform spatial dimension pooling, similarity calculation and feature enhancement processing on the template features and search area features, output the exit score corresponding to each dynamic layer through the multi-layer perception, and the exit score quantifies the applicability of the output result of the corresponding dynamic layer;

[0078] Step S5: compare the exit score corresponding to each dynamic layer with the preset threshold value, if the exit score is greater than or equal to the preset threshold value, terminate the reasoning immediately, and take the output of the current dynamic layer as the tracking result; if the exit score is less than the preset threshold value, continue to execute the next dynamic layer, and repeat the score judgment process until the threshold condition is met or all dynamic layers are executed;

[0079] Step S6: adopt a hierarchical distillation method to transfer the knowledge of the last layer of the teacher model to multiple dynamic layers of the student model, optimize the student model through logical loss, relationship loss and feature imitation loss, and improve the prediction accuracy of the dynamic layer;

[0080] Step S7: embed the output of the dynamic layer meeting the condition into the input of the lightweight multi-layer perception prediction head to obtain the target bounding box coordinates, update the search area center of the next frame, and realize continuous frame visual target tracking.

[0081] From the above technical solution, the present application proposes a visual target tracking dynamic calculation allocation method based on scene complexity perception. First, the Token representation of the template and the search area is generated through preprocessing, and a special prediction Token is introduced to lay the foundation for boundary box regression. Then, a backbone network containing fixed layers and dynamic layers is constructed, the fixed layers extract stable basic features, and the dynamic layers support subsequent flexible calculation. The self-attention and cross-attention mechanisms are used to effectively fuse the feature information of the template and the search area, and to enhance the target representation. The core innovation is that the scene complexity analyzer is activated after the first dynamic layer, which predicts the exit score of each subsequent layer through feature analysis, thereby quantifying the tracking reliability of the current features. Based on the real-time comparison of the score and the threshold value, the system can adaptively decide to terminate the reasoning in the dynamic layer that meets the conditions, realizing the intelligent calculation allocation of "early exit in simple scene and deep calculation in complex scene". To improve the prediction accuracy of the early exit layer, the hierarchical distillation technology is used to migrate the deep knowledge of the teacher model to multiple dynamic layers of the student model, ensuring that the outputs of each layer maintain high accuracy. Finally, the target position is output by the lightweight prediction head and the search area is updated, completing efficient and adaptive continuous tracking. This method realizes dynamic optimization allocation of computing resources as a whole, while maintaining high tracking accuracy and significantly improving the reasoning speed.

[0082] The embodiment discloses a visual target tracking dynamic calculation allocation method based on scene complexity perception (hereinafter referred to as "DCATrack"), the core of which is to build an integrated tracking system of "dynamic calculation allocation framework + scene complexity analyzer + hierarchical distillation method". The system adaptively perceives the scene complexity of the video frame, dynamically adjusts the allocation of network computing resources, saves redundant calculation through an early exit mechanism in a simple scene, maintains the tracking robustness of the deep network in a complex scene, and finally realizes the balance between high precision and real-time performance on a resource-limited device.

[0083] Step S1: input preprocessing and token construction. The core function of this step is to obtain valid input data and convert it into a token form that can be processed by the network. The specific implementation is as follows:

[0084] (1) Input image acquisition:

[0085] Template image: obtained by cropping from the first frame of the video sequence with the target center as the reference and a fixed expansion ratio (1.2 expansion ratio is used in this embodiment). The size is fixed at , where , (or optionally 112x112), 3 is the RGB three-channel, where represents the height dimension of the template image, represents the width dimension of the template image.

[0086] Search area image: the center of the bounding box of the tracking result of the last frame is taken as the reference, and it is expanded outward to cover the potential motion range of the target. The size is fixed at , where , (or optionally 224x224), represents the height dimension of the search area image, represents the width dimension of the search area image.

[0087] (2) Token construction:

[0088] Image block embedding processing: 16x16 size image block segmentation is adopted, and the template image and the search area image are respectively converted into template token ( ) and search area token ( ) through a linear projection layer. The feature dimension after projection is .

[0089] Prediction token setting: define four special learnable prediction tokens ( ), respectively corresponding to the top, left, bottom, right coordinates of the target bounding box. The prediction token interacts with the template token and the search region token through cross-attention mechanism, and the output embedding of the prediction token is independently input into the shared lightweight MLP prediction head (the hidden layer dimension is 256, and the output dimension is 1).

[0090] Step S2: Constructing a dynamic backbone network. This step constructs a backbone network based on a multi-layer visual Transformer (ViT), and the core is to divide fixed layers and dynamic layers to realize selective inference, which is implemented as follows:

[0091] Network infrastructure: ViT-MAE-Base is used as the backbone network (ImageNet-1K self-supervised pre-training weight initialization), which is composed of 12 mixed attention layers. The first 2 layers are set as fixed layers, and the last 10 layers are set as dynamic layers (n=10).

[0092] Fixed layer: unconditionally processes all input tokens (template token, search region token, prediction token), and the main function is to extract low-level basic features (such as edges, textures, etc.), providing a feature basis for subsequent dynamic layers.

[0093] Dynamic layer: set after the fixed layer, each layer has independent feature output capability, and can decide whether to terminate inference through scene complexity analysis results, realizing dynamic allocation of computing resources.

[0094] Step S3: Perform mixed attention feature interaction. The mixed attention layer realizes feature interaction and enhancement through self-attention and cross-attention operations, and the specific calculation formula and implementation details are as follows:

[0095] (1) Self-attention operation of target aggregation:

[0096] The role is to capture the dependency relationship within the template token, and the specific process is as follows: project the template token ( ) into the query (Q), key (K), and value (V) spaces (the projection matrix dimension is 768x768, represented as ), and calculate by the following formula:

[0097] ,

[0098] Among them, , , are the query, key, and value matrices after projecting the template token, , , , 、 、 is a learnable projection matrix, is the transpose matrix of , is the feature dimension, is an attention function, is a normalization function used to normalize the attention weight. The final output is the processed template feature , where denotes a self-attention function.

[0099] (2) Cross-attention operation for template and search region fusion:

[0100] The role is to inject the target clues in the template token ( ) into the search region token ( ), realizing cross-modal feature fusion, and the specific formula is:

[0101] ,

[0102] wherein, , are the projection results of the search region token and the predicted token respectively (the projection matrix dimension is 768x768), is the key-value pair matrix of the template token, , are the search region feature and the predicted token feature processed by cross-attention respectively, cross-attention function. The cross-attention function realizes the key attention to the target related region by calculating the similarity between the search region query and the template key, and outputs the enhanced search region feature and the predicted token feature .

[0103] Step S4: scene complexity analysis and exit score prediction. This step is the core of dynamic calculation and allocation, which realizes scene complexity quantization by analyzing feature similarity, and outputs the exit score of each dynamic layer. The specific implementation steps are as follows:

[0104] Step S41, obtain the initial dynamic layer feature: obtain the template feature and the search region feature by the first dynamic layer calculation:

[0105] ,

[0106] wherein, is a dynamic layer calculation unit, which includes self-attention, cross-attention, layer normalization and residual connection operation; Template token, Search region token.

[0107] Step S42, template global feature extraction: the template feature is subjected to spatial dimension pooling (global average pooling) to obtain the template global feature with a dimension of 1x768, and the formula is:

[0108] ,

[0109] wherein, represents the spatial dimension pooling operation.

[0110] Step S43, search region feature enhancement:

[0111] (1) Similarity map calculation: the similarity map between the template global feature and the search region feature is calculated by dot product, and the formula is:

[0112] ,

[0113] wherein, represents the dot product operation, and the dimension is consistent with , which is used to represent the similarity degree of each position of the search region to the target.

[0114] (2) Feature enhancement: the element-wise product is calculated between and to highlight the region similar to the target in the search region, and the enhanced search region feature is obtained:

[0115] ;

[0116] wherein, represents the element-wise product operation.

[0117] Step S44, exit score prediction: the enhanced search region feature is input into a multi-layer perceptron (MLP) to output exit scores ranging from :

[0118] ,

[0119] wherein, represents the exit score set corresponding to each dynamic layer; is the number of dynamic layers. For multi-layer perception, it contains two layers of full connection layer (hidden layer dimension 512, activation function ReLU, output layer dimension 10, activation function Sigmoid), and the score closer to 1 indicates that the corresponding dynamic layer output result meets the tracking accuracy requirement.

[0120] Further, in training the scene complexity analyzer, the predicted token features of each dynamic layer are input into the prediction head to obtain the bounding box , calculate the intersection over union (IoU) with the real bounding box , supervise the accuracy of the exit score prediction through mean square error loss (MSE Loss), and the formula is:

[0121] ,

[0122] ,

[0123] wherein, is the exit score prediction loss function, is the mean square error loss function, is the Sigmoid function, which ensures that the score is mapped to the range (0, 1).

[0124] Step S5: Perform dynamic inference termination judgment. According to the comparison between the exit score and the preset threshold, it is determined whether the inference is terminated, and the specific implementation is as follows:

[0125] (1) Threshold setting: preset exit threshold (determined on the training set through cross-validation, balancing accuracy and speed).

[0126] (2) Judgment logic:

[0127] 1. After executing the first dynamic layer, obtain the exit score set ;

[0128] 2. After executing the th dynamic layer (from 1 to ), extract the corresponding score ;

[0129] 3. If , terminate the inference immediately, and input the predicted token features of the current dynamic layer into the prediction head to output the tracking result;

[0130] 4. If , continue to execute the next dynamic layer and repeat the score judgment process.​​

[0131] 5. If the threshold condition is still not met after all dynamic layers are executed, the output of the last dynamic layer is taken as the tracking result.

[0132] Step S6: Perform hierarchical distillation model optimization. The hierarchical distillation architecture is as shown in Figure 3 To improve the prediction accuracy of the dynamic layer, the hierarchical distillation method is used to transfer the knowledge of the teacher model to the student model, which is implemented as follows:

[0133] (1) Teacher-student model setting:

[0134] The teacher model uses MixFormerV2-B (12 layers of mixed attention layers, parameter amount 86.6M), and the student model is three versions of DCATrack (DCATrack-L12: 12 layers, DCATrack-L8: 8 layers, DCATrack-L4: 4 layers).

[0135] (2) Three kinds of distillation loss functions:

[0136] ① Logical loss ( ):

[0137] Align the bounding box prediction distribution of the last layer of the teacher model with each dynamic layer of the student model through KL divergence. The formula is:

[0138] ,

[0139] where represents the KL divergence, which is used to calculate the difference between two probability distributions; is the output of the last layer of the teacher model; is the output of the th dynamic layer of the student model is the number of dynamic layers.

[0140] ② Relationship loss ( ):

[0141] Constrain the QKV attention relationship matrix of the student model to be consistent with that of the teacher model, enhance the feature representation ability, and the QKV relationship matrix calculation formula of the teacher and the student is:

[0142] ,

[0143] ,

[0144] The relationship loss formula is:

[0145] ,

[0146] where and The attention relationship matrix representing the teacher model and the student model, with superscript... and The following table represents the teacher model and the student model, respectively. The table below indicates the type of attention relationship. and These represent the teacher models. The first layer and student model layer; This is the normalization function; and Indicates the size of the relationship header for the teacher model and the student model (both are set to 64); The teacher model is represented by the first... In the layer, the type is Query / key / value matrix; The teacher model is represented by the first... layer Transpose of a type matrix; The student model is represented by the first... In the layer, the type is Query / key / value matrix; The student model is represented by the first... layer Transpose of a type matrix; Indicates a loss of relationship; This represents the KL divergence, used to calculate the difference between two probability distributions; A set of layers for teacher-student matching.

[0147] ③ Feature imitation loss ( ):

[0148] Applied only to the last layer of the teacher and student models, it maintains the consistency of deep features through L2 loss, as shown in the formula:

[0149] ,

[0150] in, For L2 loss, For the student model The feature vector of the layer, For the teacher model The feature vector of the layer.

[0151] (3) Pruning distillation to achieve:

[0152] To obtain lightweight models with different numbers of layers, a stepwise pruning and distillation strategy is adopted:

[0153] With DCATrack-L12 (trained by hierarchical distillation) as the teacher model, 4 dynamic layers are pruned to obtain DCATrack-L8, and 300 epochs are trained;

[0154] With DCATrack-L8 as the teacher model, 4 dynamic layers are pruned to obtain DCATrack-L4, and 300 epochs are trained;

[0155] The loss function of pruning distillation includes feature loss , relationship loss and logical loss , and the formula is:

[0156] ,

[0157] ,

[0158] ,

[0159] Among them, represents the attention relationship matrix of the type in the layer of the pruned student model; represents the attention relationship matrix of the type in the layer of the teacher model; represents the bounding box prediction distribution output by the prediction head of the layer of the pruned student model; represents the bounding box prediction distribution output by the prediction head of the layer of the teacher model.

[0160] Step S7: output tracking results and update search area. This step converts the dynamic layer output that meets the conditions into tracking results and updates the search area of the next frame, and the specific implementation is as follows:

[0161] Bounding box prediction: input the prediction token feature (4 tokens, each dimension 768) output by the dynamic layer of the termination reasoning into the lightweight MLP prediction head, and output the top, left, bottom and right coordinates of the bounding box respectively. The coordinate value is converted into pixel coordinates by inverse normalization (the normalization range is [0, 1], and the inverse normalization is multiplied by the search area image size).

[0162] Search area update: take the center of the predicted bounding box of the current frame as the benchmark, expand outward according to the fixed expansion ratio (consistent with the initial search area expansion ratio), and determine the search area position of the next frame to ensure that the target is always within the search range.

[0163] To verify the effectiveness of the present application, the embodiment is tested on four public data sets (LaSOT, LaSOT_ext, TrackingNet, GOT10K) and compared with advanced high real-time trackers (ECO, ATOM, SiamFC++, LightTrack, HCAT, E.T.Track, HiT-B, MixformerV2, FERMT) in recent years. Table 1 is a brief overview of the tracker network. The test environment includes GPU (NVIDIA RTX 3090) and CPU (Intel i7-12700K).

[0164] Table 1 Brief overview of the tracker network

[0165]

[0166] The training and testing of the embodiment are both performed on an NVIDIA RTX 3090 GPU.

[0167] In the training phase: the backbone network of DCATrack is initialized using ViT-MAE-Base (ImageNet-1K self-supervised pre-training weights). In hierarchical distillation training, MixFormerV2-B is used as the teacher model, and DCATrack-L12 is used as the student to train for 500 epochs. The learning rate is 1e-4 for the first 400 epochs and is attenuated to 1 / 10 for the last 100 epochs. In deep pruning distillation, DCATrack-L12 is used as the teacher, 4 layers are pruned to obtain DCATrack-L8, and trained for 300 epochs; then DCATrack-L8 is used as the teacher, 4 layers are pruned to obtain DCATrack-L4, and trained for 300 epochs. In the scene complexity analyzer training, all parameters except the scene complexity analyzer are frozen, and the learning rate is trained for 40 epochs. (backbone network), (prediction head), and the learning rate is attenuated to 1 / 10 for the last 100 epochs. In deep pruning distillation, DCATrack-L12 is used as the teacher, 4 layers are pruned to obtain DCATrack-L8, and trained for 300 epochs; then DCATrack-L8 is used as the teacher, 4 layers are pruned to obtain DCATrack-L4, and trained for 300 epochs. In the scene complexity analyzer training, all parameters except the scene complexity analyzer are frozen, and the learning rate is trained for 40 epochs.

[0168] In the inference phase: the initial frame is input, the template (128x128 / 112x112) is cropped, and the prediction Token is initialized; for each subsequent frame, the search area (288x288 / 224x224) is cropped, and the dynamic calculation allocation framework is input; the fixed layer is executed first, and the basic feature is extracted; after executing the first dynamic layer, the scene complexity analyzer is activated, and the exit score is generated ; if , the first ​The output of the first dynamic layer predicts the bounding box; otherwise, the next dynamic layer is executed, and the score judgment is repeated until the threshold is met or all dynamic layers are executed; the bounding box is output, and the center of the search region in the next frame is updated.

[0169] Experiment 1: Results compared with baseline model

[0170] The test results and comparisons with the baseline model are shown in Table 2, with the two best results marked in bold and underline. The DCATrack of this invention achieves a balance between speed and accuracy compared to the baseline. DCATrack-L12 achieves 171 FPS (GPU) on the LaSOT dataset, a 55% speedup compared to the baseline model (Baseline-L12, 110 FPS), while maintaining an AUC of 69.0% (only a 0.4% decrease). DCATrack-L8 achieves 211 FPS (GPU), a 46 FPS speedup compared to the baseline model (Baseline-L8, 165 FPS), with an AUC of 67.9% (only a 0.7% decrease). DCATrack-L4 achieves 353 FPS (GPU) and 72 FPS (CPU), a significant improvement over the Baseline-L4's 275 FPS (GPU) and 56 FPS (CPU), while maintaining an AUC of 58.8% (a 0.3% improvement).

[0171] Table 2 Comparison results with baseline model

[0172]

[0173] Experiment 2: Results compared with advanced trackers

[0174] As shown in Table 3, the DCATrack of this invention achieved consistent speed and accuracy advantages compared to other high-speed trackers on multiple datasets. For example, compared to FERMT, DCATrack-L8 achieved a significant accuracy advantage on the LaSOT dataset (67.9% AUC vs 65.1% AUC) at comparable speeds (211 FPS vs 225 FPS).

[0175] Table 3 Comparison results with advanced trackers

[0176]

[0177] Example 2: Figure 4 As shown, this invention provides a scene complexity-aware dynamic calculation and allocation system for visual target tracking. This system is used to implement the scene complexity-aware dynamic calculation and allocation method for visual target tracking described in Embodiment 1 above, specifically including:

[0178] The input preprocessing and token construction module 100 is used to obtain a target template image of a first frame and a search area image of a subsequent frame in a video sequence, the template image is obtained by fixedly expanding and cutting from a target center, and the search area image is obtained by expanding and centering a last frame tracking result; image block embedding processing is performed on the template image and the search area image to obtain template tokens and search area tokens, and four special learnable prediction tokens are set, the prediction tokens correspond to top, left, bottom and right coordinates of a target bounding box respectively;

[0179] The dynamic backbone network construction module 200 is used to construct a backbone network based on a multi-layer visual Transformer, the backbone network is composed of a plurality of mixed attention layers, the mixed attention layers are divided into fixed layers and dynamic layers, the fixed layers unconditionally process all input tokens to extract low-level basic features, and the dynamic layers are arranged after the fixed layers and are used to realize selective termination of an inference process;

[0180] The mixed attention feature interaction module 300 is used for the mixed attention layers to process the template tokens, the search area tokens and the prediction tokens through two operations of target aggregated self-attention and template and search area fused cross-attention, so as to complete feature interaction and enhancement;

[0181] The scene complexity analysis and exit score prediction module 400 is used to activate a scene complexity analyzer after the first dynamic layer is calculated, to perform spatial dimension pooling, similarity calculation and feature enhancement processing on template features and search area features, to output exit scores corresponding to each dynamic layer through a multi-layer perception, and to quantize the exit scores to the applicability of the output results of the corresponding dynamic layers;

[0182] The dynamic inference termination judgment module 500 is used to compare the exit scores corresponding to each dynamic layer with a preset threshold, if the exit score is greater than or equal to the preset threshold, the inference is immediately terminated, and the output of the current dynamic layer is taken as a tracking result; if the exit score is less than the preset threshold, the next dynamic layer is continued to be executed, and the score judgment process is repeated until the threshold condition is met or all dynamic layers are executed;

[0183] The hierarchical distillation model optimization module 600 is used to transfer the knowledge of the last layer of a teacher model to a plurality of dynamic layers of a student model by using a hierarchical distillation method, to optimize the student model through logical loss, relationship loss and feature imitation loss, and to improve the prediction accuracy of the dynamic layers;

[0184] The tracking result output and search area update module 700 is used to embed the output of the dynamic layer meeting the condition into a lightweight multi-layer perception prediction head to obtain target bounding box coordinates, to update a search area center of a next frame, and to realize visual target tracking of continuous frames.

[0185] The scene complexity perception based visual target tracking dynamic calculation allocation system of the embodiment is used for implementing the scene complexity perception based visual target tracking dynamic calculation allocation method, and therefore the specific embodiments in the scene complexity perception based visual target tracking dynamic calculation allocation system can be seen from the embodiment part of the scene complexity perception based visual target tracking dynamic calculation allocation method, for example, the input preprocessing and token construction module 100, the dynamic backbone network construction module 200, the mixed attention feature interaction module 300, the scene complexity analysis and exit score prediction module 400, the dynamic reasoning termination judgment module 500, the hierarchical distillation model optimization module 600, and the tracking result output and search region update module 700 are respectively used for implementing steps S1, S2, S3, S4, S5, S6, and S7 in the scene complexity perception based visual target tracking dynamic calculation allocation method, and therefore, the specific embodiments can refer to the description of the respective embodiment part, and details are not described herein again to avoid redundancy.

[0186] Embodiment three: the embodiment provides an electronic device, the electronic device includes a processor, a memory, and a bus system, the processor and the memory are connected through the bus system, the memory is used for storing instructions, and the processor is used for executing the instructions stored in the memory to implement the scene complexity perception based visual target tracking dynamic calculation allocation method.

[0187] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to disk storage, CD-ROM, optical storage, etc.).

[0188] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system), and computer program product of the embodiment. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a machine that implements the function specified in the flowchart and / or block diagram. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the function specified in one block or multiple blocks.

[0189] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks of the block or blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flow or flows and / or blocks of the block or blocks. Figure 1 function specified in the flow or flows and / or blocks of the block or blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flow or flows and / or blocks of the block or blocks. Figure 1 function specified in the flow or flows and / or blocks of the block or blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flow or flows and / or blocks of the block or blocks. Figure 1 function specified in the flow or flows and / or blocks of the block or blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flow or flows and / or blocks of the block or blocks.

[0190] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Based on the above description, other different forms of changes or modifications can be made by those skilled in the art. Here, it is not necessary or possible to exhaust all embodiments. The obvious changes or modifications derived therefrom are still within the scope of the present application.

Claims

1. A method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness, characterized in that, Includes the following steps: Step S1: Obtain the target template map of the first frame and the search region map of subsequent frames in the video sequence. The template map is obtained by cropping from the center of the target with a fixed expansion ratio, and the search region map is obtained by expanding from the tracking result of the last frame. Perform image patch embedding processing on the template map and the search region map to obtain the template token and the search region token, and set four dedicated learnable prediction tokens. The prediction tokens correspond to the top, left, bottom and right coordinates of the target bounding box, respectively. Step S2: Construct a backbone network based on a multi-layer visual Transformer. The backbone network consists of multiple hybrid attention layers. The hybrid attention layers are divided into fixed layers and dynamic layers. The fixed layers unconditionally process all input tokens to extract low-level basic features. The dynamic layers are set after the fixed layers to achieve selective termination of the inference process. Step S3: The hybrid attention layer processes the template token, search region token, and predicted token respectively through two operations: self-attention of target aggregation and cross-attention of template and search region fusion, to complete feature interaction and enhancement. Step S4: After the first dynamic layer is calculated, the scene complexity analyzer is activated to perform spatial dimension pooling, similarity calculation and feature enhancement processing on the template features and search region features. The exit score corresponding to each dynamic layer is output through the multilayer perceptron. The exit score quantifies the applicability of the output result of the corresponding dynamic layer. Step S5: Compare the exit score corresponding to each dynamic layer with a preset threshold. If the exit score is greater than or equal to the preset threshold, the inference is terminated immediately, and the output of the current dynamic layer is used as the tracking result. If the exit score is less than the preset threshold, the next dynamic layer is executed, and the score judgment process is repeated until the threshold condition is met or all dynamic layers are executed. Step S6: Using a hierarchical distillation method, the knowledge of the last layer of the teacher model is transferred to multiple dynamic layers of the student model. The student model is optimized through logistic loss, relational loss, and feature imitation loss to improve the prediction accuracy of the dynamic layers. Step S7: Embed the dynamic layer output that meets the conditions into the input of the lightweight multilayer perceptron prediction head to obtain the target bounding box coordinates, update the search region center of the next frame, and realize visual target tracking in continuous frames.

2. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S1, the size of the template image is The size of the search area map is ,in This indicates the height dimension of the template diagram. This indicates the width dimension of the template image. This indicates the height dimension of the search area map. The width dimension of the search region map is represented by 3, and the number of channels is represented by 3. The prediction token interacts with the template token and the search region token through a cross-attention mechanism. The output of the prediction token is embedded into the shared lightweight multilayer perceptron prediction head as an independent input.

3. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S3, the self-attention operation for target aggregation is specifically as follows: The template token is projected onto the query, key, and value space, and the internal dependency features of the template token are calculated using an attention formula, which is: ， in, , , These are the query, key, and value matrices projected from the template token. for The transpose of the matrix, For feature dimension, Let be the attention function. This is the normalization function.

4. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S3, the cross-attention operation of fusing the template and the search region is specifically as follows: Using the template token as a key-value pair, the search region token and the predicted token are projected onto the query, key, and value space. Target clue injection is achieved through a cross-attention formula, which is: ， in, , These are the projection results of the search region token and the predicted token, respectively. This is a key-value pair matrix of template tokens. , These are the search region features and predicted token features after cross-attention processing, respectively. Cross-attention function.

5. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S4, the process of activating the scene complexity analyzer, performing spatial dimension pooling, similarity calculation, and feature enhancement on template features and search region features, and outputting the exit scores corresponding to each dynamic layer through a multilayer perceptron includes: Step S41: Calculate template features through the first dynamic layer and search region features : ， in, For dynamic layer computing units, For template token, For the search area token; Step S42: For template features Spatial dimension pooling is performed to obtain the template global features. : ， in, This represents a spatial dimension pooling operation; Step S43: Calculate global features of the template using dot product Features of the search area Similarity graph Furthermore, feature enhancement of the search region is achieved through element-wise multiplication, resulting in enhanced search region features. : ， ； in, Represents the dot product. This represents element-wise multiplication. Step S44: Enhance the features of the search region Input multilayer perceptron, output The range is in Exit score: ， in, This represents the set of exit scores corresponding to each dynamic layer; The number of dynamic layers; It is a multilayer perceptron.

6. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S6, the logistic loss is calculated using KL divergence to align the bounding box prediction distributions of the teacher model and the student model. The loss formula is as follows: ， in, For logical loss; This represents the KL divergence, used to calculate the difference between two probability distributions; This is the output of the last layer of the teacher model; For the student model Each dynamic layer output This represents the number of dynamic layers.

7. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S6, the relationship loss is constrained by KL divergence to establish the QKV relationship between the student model and the teacher model. The formula for calculating the QKV relationship of the teacher model is as follows: ， The formula for calculating the QKV relationship in the student model is: ， The formula for relationship loss is: ， in, and The attention relationship matrix representing the teacher model and the student model, with superscript... and The following table represents the teacher model and the student model, respectively. The table below indicates the type of attention relationship. and These represent the teacher models. The first layer and student model layer; This is the normalization function; and The header size represents the relationship between the teacher model and the student model; The teacher model is represented by the first... In the layer, the type is Query / key / value matrix; The teacher model is represented by the first... layer Transpose of a type matrix; The student model is represented by the first... In the layer, the type is Query / key / value matrix; The student model is represented by the first... layer Transpose of a type matrix; Indicates a loss of relationship; This represents the KL divergence, used to calculate the difference between two probability distributions; A set of layers for teacher-student matching.

8. The method for dynamic calculation and allocation of visual target tracking based on scene complexity awareness according to claim 1, characterized in that, In step S6, the feature imitation loss uses L2 loss, which is applied only to the last layer of the teacher and student models. The loss formula is as follows: ,in For feature imitation loss, For L2 loss, For the student model The feature vector of the layer, For the teacher model The feature vectors of each layer; models with different numbers of layers are obtained through model pruning and distillation. The loss of pruning and distillation includes feature loss. Relationship loss and logical loss : ， ， ， in, A set of layers for teacher-student matching; For the student model The feature vector of the layer; This represents the KL divergence, used to calculate the difference between two probability distributions; This indicates the student model after pruning. In the layer, the type is Attention relationship matrix; The teacher model is represented by the first... In the layer, the type is Attention relationship matrix; This indicates the student model after pruning. The predicted distribution of bounding boxes output by the layer prediction head; The teacher model is represented by the first... The bounding box prediction distribution output by the layer prediction head.

9. A dynamic calculation and allocation system for visual target tracking based on scene complexity awareness, characterized in that, The system is used to implement the scene complexity-aware visual target tracking dynamic calculation and allocation method according to any one of claims 1 to 8, specifically including: The input preprocessing and token construction module is used to obtain the target template map of the first frame and the search region map of subsequent frames in the video sequence. The template map is obtained by cropping from the center of the target with a fixed expansion ratio, and the search region map is obtained by expanding from the tracking result of the last frame. The template map and the search region map are subjected to image patch embedding processing to obtain the template token and the search region token, and four dedicated learnable prediction tokens are set. The prediction tokens correspond to the top, left, bottom and right coordinates of the target bounding box, respectively. A dynamic backbone network construction module is used to construct a backbone network based on a multi-layer visual Transformer. The backbone network consists of multiple hybrid attention layers, which are divided into fixed layers and dynamic layers. The fixed layers unconditionally process all input tokens to extract low-level basic features, and the dynamic layers are set after the fixed layers to achieve selective termination of the inference process. The hybrid attention feature interaction module is used by the hybrid attention layer to process template tokens, search region tokens and prediction tokens through two operations: self-attention of target aggregation and cross-attention of template and search region fusion, respectively, to complete feature interaction and enhancement. The scene complexity analysis and exit score prediction module is used to activate the scene complexity analyzer after the first dynamic layer calculation is completed, perform spatial dimension pooling, similarity calculation and feature enhancement processing on template features and search region features, and output the exit score corresponding to each dynamic layer through a multilayer perceptron. The exit score quantifies the applicability of the output result of the corresponding dynamic layer. The dynamic reasoning termination judgment module is used to compare the exit score corresponding to each dynamic layer with a preset threshold. If the exit score is greater than or equal to the preset threshold, the reasoning is terminated immediately, and the output of the current dynamic layer is used as the tracking result. If the exit score is less than the preset threshold, the next dynamic layer is executed, and the score judgment process is repeated until the threshold condition is met or all dynamic layers are executed. The hierarchical distillation model optimization module is used to transfer the knowledge of the last layer of the teacher model to multiple dynamic layers of the student model using the hierarchical distillation method. It optimizes the student model through logistic loss, relational loss and feature imitation loss to improve the prediction accuracy of the dynamic layers. The tracking result output and search region update module is used to embed the dynamic layer output that meets the conditions into the input of the lightweight multilayer perceptron prediction head to obtain the target bounding box coordinates, update the search region center of the next frame, and realize visual target tracking in continuous frames.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the dynamic calculation and allocation method for visual target tracking based on scene complexity awareness as described in any one of claims 1 to 8.

Images