A size-aware visual inspection tracking device and method for mobile terminals

Abstract

The application discloses a size perception visual detection tracking device and method for a mobile terminal, and the device comprises a video acquisition unit, a detection unit, a tracking unit, a detection-tracking connection unit, a size perception module, a template management unit and an offline training unit; the video acquisition unit is connected with the detection-tracking connection unit; the detection-tracking connection unit is connected with the tracking unit and the detection unit respectively; the tracking unit is connected with the size perception module; the size perception module is connected with the tracking unit; and the tracking unit is connected with the template management unit; and the application solves the problem of the prior art in a multi-scale target tracking scene.

Description

Technical Field

[0001] This invention belongs to the field of size-aware visual inspection and tracking for mobile devices, and specifically relates to a size-aware visual inspection and tracking device and method for mobile devices. Background Technology

[0002] Existing target tracking technologies typically include basic steps such as target initialization, feature extraction, template matching, position prediction, and template updating. In practical applications, targets are often affected by changes in motion trajectory, shooting distance, or zoom operations, resulting in significant size changes. This places higher demands on the tracking system in terms of feature modeling, attention allocation, and stability control.

[0003] Currently, several deep learning-based visual tracking methods and devices have been publicly disclosed or implemented, such as:

[0004] The paper "Scalekd: Distilling scale-aware knowledge in small object detector" (Zhu Y, Zhou Q, Liu N, et al., CVPR 2023) proposes a method to improve the detection performance of small objects by utilizing knowledge distillation. This method transfers size-related knowledge information from a teacher network to a student network, thereby improving detection performance in small object scenarios. While this type of method has achieved some success in detection tasks, its research focus is mainly on object detection, with less emphasis on end-to-end visual tracking tasks. Furthermore, distillation strategies often employ fixed weights, making it difficult to dynamically adjust the knowledge transfer method according to changes in object size, and limiting its adaptability to lightweight mobile models and tracking scenarios.

[0005] Existing technologies also disclose some lightweight single-target tracking training schemes based on knowledge distillation. For example, patent CN115100238B proposes a distillation training method to improve the operating efficiency of single-target trackers. While these schemes reduce model complexity and improve mobile inference efficiency to some extent, they do not adequately consider changes in target size during the design process and fail to systematically incorporate size-aware mechanisms into the distillation process and tracking strategy. Therefore, in scenarios involving rapid changes in target scale or cross-scale tracking, insufficient tracking stability remains a problem.

[0006] Patent CN120853002A proposes a photoelectric pod target recognition and tracking system based on a multi-scale attention mechanism, which improves target recognition and tracking capabilities by fusing local and global features. However, such solutions typically employ a fixed-parameter attention structure, resulting in a large model size and a relatively rigid computing power allocation method. This makes it difficult to achieve efficient deployment on resource-constrained mobile devices, limiting its practicality in mobile application scenarios.

[0007] In summary, existing mobile visual detection and tracking technologies still have shortcomings in dealing with changes in target size, especially in the lack of a unified and effective solution for size perception mechanisms and the collaborative design of lightweight model training and tracking processes, making it difficult to simultaneously meet the requirements of tracking accuracy, stability and real-time performance.

[0008] The purpose of this invention is to overcome the problem of insufficient adaptability of existing mobile visual detection and tracking systems when the target size changes significantly. Due to the limitations of mobile devices in terms of computing power and energy consumption, existing tracking methods mostly adopt fixed feature modeling and attention allocation strategies. When the target undergoes significant changes in scale due to changes in viewing distance, zooming operations, or scene changes during movement, problems such as decreased tracking accuracy and insufficient stability are prone to occur.

[0009] To address this, this invention proposes a size-aware visual detection and tracking device and method for mobile devices. By introducing a size-aware mechanism, the system can adaptively adjust the feature modeling process based on changes in target size, thereby improving the tracking capability for targets of different scales. Furthermore, by combining detection and tracking in a collaborative processing manner and introducing a size-aware training strategy suitable for lightweight models, the system's tracking accuracy and robustness are improved in complex scenarios such as significant changes in target scale and frequent changes in distance, while ensuring real-time operating efficiency on mobile devices.

[0010] Through the above methods, the present invention aims to provide a mobile visual detection and tracking solution that balances accuracy, stability and real-time performance, in order to overcome the shortcomings of existing technologies in multi-scale target tracking scenarios.

[0011] The purpose of this invention is to overcome the problem of insufficient adaptability of existing mobile visual detection and tracking systems when the target size changes significantly. Due to the limitations of mobile devices in terms of computing power and energy consumption, existing tracking methods mostly adopt fixed feature modeling and attention allocation strategies. When the target undergoes significant changes in scale due to changes in viewing distance, zooming operations, or scene changes during movement, problems such as decreased tracking accuracy and insufficient stability are prone to occur.

[0012] To address this, this invention proposes a size-aware visual detection and tracking device and method for mobile devices. By introducing a size-aware mechanism, the system can adaptively adjust the feature modeling process based on changes in target size, thereby improving the tracking capability for targets of different scales. Furthermore, by combining detection and tracking in a collaborative processing manner and introducing a size-aware training strategy suitable for lightweight models, the system's tracking accuracy and robustness are improved in complex scenarios such as significant changes in target scale and frequent changes in distance, while ensuring real-time operating efficiency on mobile devices.

[0013] Through the above methods, the present invention aims to provide a mobile visual detection and tracking solution that balances accuracy, stability and real-time performance, in order to overcome the shortcomings of existing technologies in multi-scale target tracking scenarios. Summary of the Invention

[0014] To address the aforementioned technical problems, this invention proposes a size-aware visual detection and tracking device and method for mobile devices, which solves the shortcomings of existing technologies in multi-scale target tracking scenarios.

[0015] The specific technical solution of the present invention is as follows:

[0016] A size-aware visual inspection and tracking device for mobile devices includes a video acquisition unit, a detection unit, a tracking unit, a detection-tracking connection unit, a size-aware module, a template management unit, and an offline training unit.

[0017] The video acquisition unit is connected to the detection-tracking connection unit, which is connected to the tracking unit and the detection unit respectively. The tracking unit is connected to the size perception module, which is connected to the tracking unit. The tracking unit is connected to the template management unit.

[0018] The offline training unit includes a teacher network, a student network, and a deployment to tracking unit;

[0019] The teacher network connects to the student network, and the student network connection is deployed to the tracking unit.

[0020] The video capture unit is used to acquire video frames captured by the mobile device's camera;

[0021] The detection unit is used to perform target detection on the input video frame and output target candidate boxes and detection confidence scores.

[0022] The tracking unit is used to predict the target location based on historical templates and the current search area;

[0023] The detection-tracking connection unit is used for scheduling and fusion between the detection unit and the tracking unit according to the target state;

[0024] The size-aware module is used to dynamically modulate feature modeling and attention allocation based on target size information;

[0025] The template management unit is used to maintain static and dynamic templates of various sizes and to perform template updates;

[0026] The offline training unit is used to perform joint optimization training of size-aware focusing loss supervision and size-aware knowledge distillation on the lightweight model during the offline training phase.

[0027] On the other hand, a size-aware visual detection and tracking method for mobile devices includes the following steps:

[0028] Step S1: The system takes the video stream captured by the mobile device as input and adaptively switches between target detection, target tracking, and target detection and target tracking fusion according to the target state: when in the initialization or loss recovery state, the detection module is triggered to perform target detection on the current frame; when in the stable tracking state, tracking prediction is performed and the detection result and tracking prediction result are correlated and fused through the detection-tracking connection unit when needed to update the target state and position;

[0029] Step S2: During tracking, the system activates a size-aware masking mechanism to optimize attention allocation based on changes in target size. In the offline training unit, size-aware focusing loss is used to supervise student network attention, and size-aware knowledge distillation is combined to enhance the lightweight representation ability of the student network learning the teacher model on targets of different sizes. In the online inference phase, size-aware focusing loss is not calculated; only size-aware knowledge distillation is used.

[0030] Size-aware masking strategies dynamically modulate attention to improve tracking stability and positioning accuracy.

[0031] Furthermore, step S1 includes the following steps:

[0032] Step S11: Set up the state decision maker;

[0033] Step S12: During the initialization phase of the detection and tracking fusion process, the target correlator is used to match the detection results with the tracking prediction results.

[0034] Step S13: Used in the TRACKING stage. In each frame tracking stage, the system automatically selects the corresponding size template based on the size of the current detection box or prediction box, and simultaneously matches it with the dynamic template to realize a dual template candidate mechanism, thereby further improving the matching stability in appearance change scenarios.

[0035] Furthermore, step S2 includes the following steps:

[0036] Step S21: Construct the Vision Transformer architecture diagram;

[0037] Step S22: Dynamically adjust the attention range and inhibition intensity by the target size, so that the Transformer network can adopt differentiated feature focusing strategies for targets of different sizes;

[0038] Step S23: Transmit multi-level and multi-dimensional knowledge information to the lightweight student model through a large-scale teacher model, and adaptively adjust the distillation intensity according to the target size, so that the student network can obtain supervision signals that match the task difficulty under different target size conditions, thereby improving tracking accuracy and stability while ensuring real-time performance.

[0039] Compared with existing technologies, the beneficial effects of the size-aware visual detection and tracking device and method for mobile devices of the present invention are as follows:

[0040] 1. This invention improves the adaptability and reliability of target tracking to size changes. Through a size-aware masking mechanism, the model can dynamically adjust the attention area according to changes in target size. Combined with a dual-template matching method using both static and dynamic templates, it maintains good tracking stability in scenarios with changing appearance.

[0041] 2. This invention enhances the recovery capability and tracking continuity in target loss scenarios. By introducing a state decision mechanism that includes INIT, TRACKING, and LOST states, and combining it with confidence smoothing control, it effectively reduces frequent state switching, improves the relocation capability after target loss, and keeps the tracking results continuous and stable.

[0042] 3. This invention balances tracking accuracy and real-time performance under limited computing power. It employs a lightweight VisionTransformer combined with a size-aware knowledge distillation strategy, enabling the model to maintain effective tracking performance for multi-scale targets while controlling parameter scale and computational complexity, making it suitable for real-time mobile applications.

[0043] 4. This invention enables the system to adaptively process changes in target size through a size-aware correlation mechanism, making it suitable for complex application environments where target size changes significantly and viewing distance changes considerably.

[0044] 5. This invention reduces unnecessary repetitive calculations through reasonable state management and detection triggering mechanisms, which helps to reduce the computational burden and energy consumption of the system and improve the feasibility of engineering deployment. Attached Figure Description

[0045] To more clearly illustrate the purpose, design concept, and innovation of the size-sensing visual inspection and tracking device and method for mobile devices proposed in this invention, the invention will be described in detail below with reference to the accompanying drawings and tables.

[0046] Figure 1 This is a schematic diagram of the functional units and their collaborative relationships of the size-sensing visual detection and tracking device of the present invention.

[0047] Figure 2 This is the tracking state machine diagram of the present invention.

[0048] Figure 3 This is a diagram of the Vision Transformer architecture of the present invention.

[0049] Figure 4 This is a diagram illustrating the teacher-student distillation framework of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0051] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0052] The entire invention is divided into two phases: the inference phase (real-time operation phase) and the training phase (offline optimization phase). The inference phase includes a detection-tracking connection module and a size-aware module, while the training phase includes a size-aware knowledge distillation module.

[0053] The present invention also provides a size-sensing visual detection and tracking device for mobile devices, such as... Figure 1As shown, the device takes a mobile video stream as input and includes: (1) a video acquisition unit for acquiring video frames captured by a mobile camera; (2) a detection unit for performing target detection on the input video frames and outputting target candidate boxes and detection confidence; (3) a tracking unit for predicting the target position based on historical templates and the current search area; (4) a detection-tracking connection unit for scheduling and fusing between the detection unit and the tracking unit according to the target state; (5) a size-aware module for dynamically modulating feature modeling and attention allocation based on target size information; (6) a template management unit for maintaining multi-size static templates and dynamic templates and performing template updates; and (7) an offline training unit for performing joint optimization of size-aware focusing loss supervision and size-aware knowledge distillation on the lightweight model during the offline training phase. The data flow and control relationships between the units are as follows: Figure 1 As shown.

[0054] Figure 1 This is a schematic diagram illustrating the functional units and their collaborative relationships of a size-sensing visual detection and tracking device for mobile devices proposed in this invention. Figure 1 As shown, the device takes a video stream from a mobile device as input, performs image domain preprocessing, and then enters the tracking state machine. Based on the current state, it selects to call either the target detection module or the target tracking module. The target detection module performs target detection on video frames during initialization or when the target is lost. The target tracking module performs continuous position prediction of the target during stable tracking and, in conjunction with a size-aware Vision Transformer tracking network, outputs the target bounding box, confidence score, and target size category information. The detection and tracking results are scheduled and fused in the detection-tracking connection module, ultimately outputting the target position information and tracking status.

[0055] In this method, the system takes the video stream captured by the mobile device as input and adaptively switches between target detection, target tracking, and fusion of the two according to the target state: when in the initialization or loss recovery state, the detection module is triggered to perform target detection on the current frame; when in the stable tracking state, tracking prediction is performed; and when necessary, the detection-tracking connection module is used to associate and fuse the detection results and tracking prediction results to update the target state and position.

[0056] During tracking, the system employs a size-aware masking mechanism to optimize attention allocation based on changes in target size. In the offline training phase, size-aware focusing loss is used to supervise student network attention, and a size-aware knowledge distillation method is combined to enhance the lightweight student network's ability to represent teacher models on targets of different sizes. In the online inference phase, size-aware focusing loss is not calculated; instead, a size-aware masking strategy is used to dynamically modulate attention, thereby improving tracking stability and localization accuracy.

[0057] This overall process forms a closed-loop, adaptive detection and tracking system that can be deployed on mobile devices.

[0058] like Figure 1 As shown, this technology performs detection based on the moving input video stream, and then combines detection and tracking through a tracking state machine and a tracking detection connection module. Furthermore, to address size changes caused by zooming during tracking, a size-aware masking strategy and size-aware distillation are implemented to optimize the tracking algorithm. Technical details are as follows:

[0059] 1. Detection-Tracking Connection Module

[0060] The detection-tracking connection module is used to enable collaborative work between the detection module and the tracking module. Its core function is to dynamically determine the system's operating mode based on the target's current tracking status and historical information, so as to avoid the computational overhead caused by frequent detection and improve the recovery capability after the target is lost.

[0061] 1.1 State Decision Maker

[0062] The state decision unit is used to describe the tracking state of the target in consecutive frames, and its structure is as follows: Figure 2 As shown.

[0063] Figure 2 This is a schematic diagram of the state transition of the tracking state machine in this invention. For example... Figure 2 As shown, the tracking state machine includes the INIT (initialization) state, the TRACKING (tracking in progress) state, and the LOST (lost) state. The system enters the INIT state after the first frame or after target loss, completing target initialization through the detection module. After successful target detection and template establishment, it enters the TRACKING state to perform continuous tracking. When the overall tracking confidence falls below a threshold or the target fails to be reliably located for multiple consecutive frames, the system transitions to the LOST state and triggers a re-detection process. This state transition mechanism enables adaptive switching between detection and tracking, improving system stability and reducing unnecessary detection overhead.

[0064] The state set is defined as:

[0065] The set of states S = {INIT, TRACKING, LOST};

[0066] in:

[0067] INIT: Perform full-image detection and initialize the tracking target.

[0068] TRACKING: Performs tracing and directly outputs the tracing results.

[0069] LOST: Full map detection, reinitialize target.

[0070] To achieve stable state transitions, the system defines a comprehensive tracking confidence level. :

[0071]

[0072] in, Let be the overall tracking confidence score for frame t, and α be the weighting coefficient between the current frame confidence score and the historical confidence scores. The confidence level is tracked for the current frame (frame t), where N is the size of the history window, representing the number of frames used to calculate the historical average confidence level. Let be the overall tracking confidence score for frame tk, and k be the historical frame index. A confidence smoothing mechanism is also introduced. The smoothed tracking confidence score is used for state transition determination.

[0073]

[0074] Where β is the smoothing coefficient, which controls the fusion ratio of the confidence scores of the current frame and the previous frame. The overall tracking confidence for the current frame. The smoothing confidence level is the level of the previous frame. This smoothing mechanism is used to avoid frequent state switching due to short-term fluctuations.

[0075] 1.2 Target Correlator

[0076] During the detection and tracking fusion process, specifically the INIT (Initialization) phase, the target correlator is used to match the detection results with the tracking prediction results.

[0077] First, calculate the IoU distance between the predicted bounding box and the detected bounding box:

[0078]

[0079]

[0080] Where A and B represent the first bounding box region and the second bounding box region, respectively. It is the intersection area of ​​bounding boxes A and B. It is the area of ​​the union of bounding boxes A and B. It is the IoU distance between the i-th detection box and the predicted box, and its value ranges from [0, 1]. It is the tracking prediction bounding box from the previous frame. This is the i-th detection box in the current frame, where i is the index of the detection box. Then, the cosine similarity distance of the corresponding feature vectors is calculated. , representing the feature distance between the i-th detection box and the predicted box:

[0081]

[0082] in, To track the appearance feature vector corresponding to the predicted bounding box, This represents the appearance feature vector corresponding to the i-th detection box in the current frame, and is used in the comprehensive matching cost function. Defined as:

[0083]

[0084] in, This is the weighting coefficient for the IoU distance, with a value of 0.6. This is the weighting coefficient for the feature distance, with a value of 0.4. It's the IoU distance. It is the minimum feature distance. When the minimum matching cost is less than a preset threshold, the detection and tracking results are fused.

[0085] 1.3 Template Manager - Dual Template Mechanism

[0086] Primarily used in the TRACKING stage, during each frame of the tracking phase, the system automatically selects the corresponding size template based on the size of the current detection box or prediction box, and simultaneously matches it with the dynamic template to achieve a dual template candidate mechanism, thereby further improving the matching stability in scenarios with changing appearance.

[0087] The template library structure includes static templates for various sizes and one dynamic template, specifically: small-size template (SMALL), medium-size template (MEDIUM), large-size template (LARGE), and dynamic template (DYNAMIC). Each template contains the following information:

[0088] T={I,F,S,t}

[0089] Where I represents the template image, F represents the template feature vector obtained by the feature extraction network, S represents the target size range corresponding to the template, and t represents the timestamp of template generation or the most recent update.

[0090] The specific steps are as follows:

[0091] Step 1: Size Classification and Static Template Selection: During each frame of tracking, the system classifies the current detection box or prediction box based on its size. Calculate the target size, classify the target into SMALL, MEDIUM, or LARGE categories, and select the corresponding static template denoted as Ts.

[0092] Step 2: Dual Template Matching

[0093] During the template matching phase, the system utilizes the selected static templates respectively. With dynamic templates The current search region is matched, and the matching score between the static template and the search region is calculated. Matching score between dynamic template and search area :

[0094]

[0095]

[0096] Where Search is the search region image of the current frame, and Match(·) is the matching score between template T and search region Search, defined as follows:

[0097]

[0098] in, It is the feature vector of template T. It is the feature vector of the search region. It is the L2 norm of the vector.

[0099] Step 3: Compare the matching scores of the static template and the dynamic template, and select the template with the higher score for target localization in the current frame. The template selected for target localization in the current frame:

[0100]

[0101] Here, argmax is the template that maximizes the matching score. To prevent template pollution caused by mismatches, the dynamic template is only updated when the following conditions are met.

[0102]

[0103] in, This indicates the confidence level of the currently tracked target. The template update threshold can be set to 0.5-0.8. This indicates the time interval between the current frame and the last update time of the dynamic template. The minimum update interval is set to 5 frames here.

[0104] When the update conditions are met, the features of the dynamic template are updated using a smooth weighted method:

[0105]

[0106] in, This represents the features extracted from the target region in the current frame. This represents the dynamic template features of the previous time step, and k is the update coefficient. This is used to update the slope control parameters, which are used to adjust the rate at which changes in confidence affect the template update weights.

[0107]

[0108] in, For the current smoothed confidence level, This is the template update threshold, and exp is the natural exponential function. As the confidence level increases, The larger the confidence level, the more likely the system is to use the appearance of the current frame as the new template; when the confidence level is low, the system maintains the stability of the original template to prevent drift.

[0109] 2. Size sensing module

[0110] The size-aware module is applied to the target tracking process to dynamically adjust the feature modeling range and attention allocation strategy when the target size changes, thereby improving tracking stability and positioning accuracy under conditions of large zoom ratios, changes in target distance, and complex backgrounds. The size-aware module is implemented based on a lightweight Vision Transformer architecture and introduces size-aware masks and size-weighted constraints into the self-attention mechanism.

[0111] 2.1 Vision Transformer Architecture Diagram

[0112] like Figure 3 As shown, a lightweight Vision Transformer (ViT) is used as the backbone network for feature modeling. Its overall structure includes an input embedding layer, a multi-layer Transformer encoder, and a target localization prediction head. The input image is segmented into several image patches and mapped to a feature sequence: X={x1,x2,…,x…} N},x i ∈R d Where N is the number of tokens and d is the embedding dimension. After multi-layer self-attention encoding, the output features are used for subsequent target location regression and confidence prediction.

[0113] Figure 3 This is a schematic diagram of the size-aware Vision Transformer network structure used for target tracking in this invention. Figure 3As shown, the template image and the search image are each processed through patch embedding to generate a token sequence. Position encoding and size embedding information are then introduced after token concatenation. The size-aware encoder modulates the self-attention calculation process based on the target size information, enabling the network to dynamically adjust the feature modeling range for targets of different sizes. After multi-layer Transformer encoding, the prediction head outputs the target bounding box coordinates and corresponding confidence information for target location prediction and tracking.

[0114] 2.2 Size-Aware Mask Generation Method

[0115] Size-aware mask generation methods dynamically adjust the attention scope and suppression intensity based on target size, enabling Transformer networks to employ differentiated feature focusing strategies for targets of different sizes. The methods include the following steps:

[0116] Step 1: Size Calculation and Classification

[0117] Use x c ,y c These represent the x and y coordinates of the center point of the bounding box, respectively. Based on the target bounding box bbox=(x c ,y c Calculate the target size (w,h). :

[0118]

[0119] Where w and h represent the bounding box width and height, respectively. Based on the target size, the targets are divided into three categories, with the size category represented by... express:

[0120]

[0121] Where τs is the small size threshold, with a default value of 32 pixels, τl is the large size threshold, with a default value of 96 pixels, SMALL represents the small size target category, MEDIUM represents the medium size target category, and LARGE represents the large size target category.

[0122] Step 2: Adaptive parameter calculation

[0123] Different context expansion radii are set for targets of different size categories to control the spatial extent of attention. The context expansion radius is defined as follows: :

[0124]

[0125] Here, `size` represents the target size. Small targets (SMALL) expand the context area to supplement semantics, while large targets (LARGE) shrink the range to enhance focus. Simultaneously, to control the intensity of attention suppression, a size-dependent mask intensity coefficient is introduced. :

[0126]

[0127] SMALL targets small-sized objects. Smaller (0.5) targets are masked using a soft mask, preserving some background information; MEDIUM targets are medium-sized targets, and LARGE targets are large-sized targets. The value is relatively large (1.0), and a hard mask is used to strongly shield background interference.

[0128] Step 3: Constructing the size-aware mask matrix

[0129] Based on the target bounding box location and context expansion radius Determine the set of tokens relevant to the target, and use the set of token indices that the attention needs to focus on. express:

[0130]

[0131] Where d(i,bbox) represents the spatial distance between the i-th token and the center of the target region, r(s) is the context spread radius, and i is the token index, based on which a size-aware mask matrix is ​​generated:

[0132]

[0133] in, Let be the value of the size-aware mask matrix at position (i, j). It is a set of target-related tokens, where i and j represent the Query Token index and Key Token index, respectively. For the target-related token set, The intensity mask coefficient is a scalar parameter related to the target size s, used to control the degree of attention suppression on the background token.

[0134] The mask matrix is ​​then incorporated into the Transformer's self-attention calculation process, where Attention represents the attention output.

[0135]

[0136] Q, K, and V represent the Query matrix, Key matrix, and Value matrix, respectively, obtained by linear transformation of the input features.T It is the transpose of the key matrix, d k M represents the dimension of the key vector, used to scale the dot product. size It is a size-aware mask matrix, and softmax is a normalized exponential function that converts attention scores into a probability distribution.

[0137] By introducing a size-aware suppression matrix, attention is focused on the target area, and the response of the background token is dynamically suppressed.

[0138] Step 4: Size-aware focusing loss function (offline stage)

[0139] This step is only used during offline training to supervise the attention allocation of the student network in order to learn focusing strategies for targets of different sizes; during the online inference phase, this loss is not calculated, and only steps 1–3 are performed to generate masks and incorporate self-attention.

[0140] To improve attention accuracy, a size-aware focusing loss mechanism is constructed. :

[0141]

[0142]

[0143] Where s is the target size, and the size weighting coefficient is defined as follows: , This is the attention supervision term. η is the weighting strength parameter, with a default value of 2.0. This is the reference size, with a default value of 32 pixels. exp is the natural exponential function.

[0144] Attention monitoring items Defined as:

[0145]

[0146] Among them, A j This represents the attention weight of the target Query Token to the j-th Token in the Transformer's self-attention mechanism. The first term represents the ideal attention distribution within the target region; the second term is the target region attention guidance term, used to encourage the model to focus its attention on the target region tokens; the third term is the background attention suppression regularization term, used to suppress the model's ineffective attention to background tokens. is the regularization coefficient, with a value of 0.1.

[0147] 2.3 Size-Aware Distillation Strategy

[0148] The size-aware distillation strategy transmits multi-level and multi-dimensional knowledge information to a lightweight student model through a large-scale teacher model, and adaptively adjusts the distillation intensity according to the target size. This allows the student model to obtain supervision signals that match the task difficulty under different target size conditions, thereby improving tracking accuracy and stability while ensuring real-time performance.

[0149] like Figure 4 As shown, an offline teacher-online student distillation paradigm is adopted, consisting of a teacher network T and a student network S. The teacher network T is a large-scale Vision Transformer used to provide high-quality features, attention, and structural information, while the student network S uses a lightweight Vision Transformer for real-time inference on mobile devices. The teacher network parameters are frozen during the distillation stage and used only for forward feature extraction; the student network is jointly optimized using distillation loss and tracking task loss.

[0150] 1. Feature Dimension Alignment Mechanism

[0151] Because the feature dimensions of the teacher network and the student network differ, direct distillation would lead to a mismatch in feature spaces. Therefore, a linear projection layer is introduced to unify the dimensions. Let the teacher feature be... Student characteristics are Then the aligned student characteristics are defined as follows:

[0152]

[0153] Among them, F T F S These are the feature matrices output by the teacher network and the student network, respectively, where N is the number of tokens and d... T， d s These represent the feature dimensions of the teacher network and the student network, respectively. It is a linear projection function. Represents the projection weight matrix. This represents the projection bias vector.

[0154] 2. Global Semantic Distillation Loss

[0155] Global semantic distillation is used to constrain the student model to approximate the teacher model at the overall token representation level, where N is the number of tokens and i is the token index. Its loss function is defined as follows: :

[0156]

[0157] Where N is the number of tokens, and i is the token index. It is the feature vector of the i-th token output by the teacher network. This is the feature vector of the i-th token output by the aligned student network. It is the square of the L2 norm.

[0158] 3. Target region structural distillation loss

[0159] To enhance the student model's ability to model the internal structural relationships of the target region, a target-related token set is used. Structural distillation is performed on top.

[0160] (1) Feature extraction of target region

[0161]

[0162]

[0163] in, , Let represent the feature sets of the teacher network and the student network within the target region, respectively, and j be the token index within the target region. For the target-related token set.

[0164] (2) Constructing the Gram matrix

[0165]

[0166] in, , Let represent the feature sets of the teacher network and the student network within the target region, respectively. The number of tokens in the target area, used for normalization. This is the matrix transpose. G T With G S These represent the structural relationship matrices of the teacher network and the student network within the target region, respectively, used to reflect the correlation distribution among different token features.

[0167] (3) Structural distillation loss

[0168]

[0169] in G represents the number of tokens in the target area. T With G S The matrices represent the structural relationships between the teacher network and the student network within the target region, respectively. The sum of squares of the matrix elements. The loss is used to maintain the consistency of the relative relationships between tokens within the target area, enhancing robustness under scenarios of size changes and deformation.

[0170] 4. Loss of attention distillation

[0171] To guide the student model in learning the attention distribution pattern of the teacher model, attention distillation loss is introduced. Let the attention matrices of the teacher and student at the *i*th and *j*th attention heads be respectively... and ,but

[0172]

[0173] The KL divergence is defined as follows, used to measure the difference in attention distribution.

[0174]

[0175] This is the KL divergence, used to measure the degree of difference between probability distributions P and Q. The smaller the value, the closer the two distributions are. When the two distributions are completely identical, the KL divergence is 0. P and Q are the attention distributions of the teacher network and student network, respectively. （i） Q （i） represents the attention weight of the Query Token to the i-th Key Token on the teacher network and student network, respectively, where i is the Key Token index.

[0176] 5. Size-sensing distillation weight adjustment

[0177] To avoid underfitting or overconstraining issues caused by applying a uniform distillation intensity to targets of different sizes, a size-aware distillation weighting function is introduced. Let the target size be s, and define:

[0178]

[0179] Wherein, the reference size s0 = 32 pixels; the global semantic distillation base weights Set to 1.0, the basic weight for structural distillation. Set the base weight to 1.0 for attention distillation. Set the value to 0.5. This distillation adjustment allows small targets to focus on learning local structural relationships and attention allocation, avoiding tracking drift due to insufficient features. Large targets focus on learning global semantics, enhancing the distinction between the target and the background, while medium-sized targets balance global semantics and local structure.

[0180] 6. Total distillation loss function

[0181] Based on the above-mentioned size-aware distillation weight adjustment mechanism, the total loss function of size-aware knowledge distillation is defined as:

[0182]

[0183] in, Size-aware weights for global semantic distillation For global semantic distillation loss, Size-perceived weights for structural distillation For structural distillation losses, Size-perceived weights for attention distillation Let s be the attention distillation loss, and s be the target size.

[0184] Distillation loss and target tracking task loss are used together for joint optimization of the student model.

[0185] The invention adopts a two-stage implementation approach that combines online inference and offline training: the online inference stage is deployed on mobile devices to complete target detection, target tracking, state decision-making, and size-aware processing in real time; the offline training stage uses a size-aware distillation strategy to train and optimize the lightweight student model, and after training, only the student model is deployed for online inference.

[0186] Example 1

[0187] Online inference detection and tracking loop closure implementation for video streams on smartphones

[0188] System Composition

[0189] The detection-tracking device in this embodiment includes: a video acquisition unit, a target detection unit, a target tracking unit, a detection-tracking connection unit, a state decision unit, a size perception module, and a template management unit. The collaborative relationship between the functional modules is as follows: Figure 1 As shown.

[0190] (II) Specific Implementation Steps

[0191] Step 1: Video Acquisition and Target Detection

[0192] The mobile camera captures a video stream at a resolution of 1280×720 pixels and a frame rate of 30 frames per second, using consecutive video frames as system input. The object detection unit employs a lightweight detection network (e.g., EfficientNet-Lite0) to perform object detection on the current video frame, outputting candidate object bounding boxes (bbox = (xc, yc, w, h), detection confidence, and corresponding appearance feature vectors.

[0193] Step 2: Detection-Tracking Connections and State Decisions

[0194] Based on the detection results, a detection-tracking state decision-making mechanism is constructed to determine the current state of the target. In this embodiment, the specific parameters of the state decision-maker are set as follows: weight coefficient α = 0.7, smoothing coefficient β = 0.8, historical window size N = 5, and confidence threshold = 0.5.

[0195] The overall tracking confidence level is calculated as follows:

[0196]

[0197] Further time smoothing is applied to the comprehensive tracking confidence score to obtain the smoothed confidence score:

[0198]

[0199] Based on the smoothing confidence level, the following state transition judgment is made: when... When the value is >0.5, the target remains in or enters the TRACKING state. When the value is ≤0.5, the target enters the LOST state, which triggers a global re-detection. If the target is in the LOST state for 5 consecutive times, it returns to the INIT state. Through the above state decision mechanism, frequent state jitter caused by short-term occlusion or noise can be effectively avoided.

[0200] Step 3: Fusion of target association and detection results

[0201] In the INIT state, a target association operation is performed between the detection results and historical tracking results. In this embodiment, the target association parameter is set as: IoU distance weight. =0.6, feature distance weight =0.4, matching cost threshold =0.5

[0202] The formula for calculating the overall matching cost is:

[0203]

[0204] When the minimum matching cost When the value is less than 0.5, the detected target and the currently tracked target are determined to be the same target, and the detection result and the tracking result are weighted and fused to improve the positioning stability.

[0205] Step 4: Dual Template Matching and Template Management

[0206] To adapt to changes in target scale, this embodiment introduces a dual-template management mechanism. The template manager's parameters are set as follows: small size threshold. =32 pixels, large size threshold =96 pixels; Template update threshold =0.5, minimum template update interval is 5 frames, update coefficient control parameter κ=10.

[0207] In a specific experiment: when the target size was 25 pixels, it was classified as SMALL and a small-sized static template was used; when the target size was 60 pixels, it was classified as MEDIUM and a medium-sized static template was used; when the target size was 120 pixels, it was classified as LARGE and a large-sized static template was used. During the dynamic template update phase, the update coefficient was calculated using the following formula:

[0208]

[0209] when =0.8, the update factor is approximately 0.95, and the system tends to use the current frame appearance to update the template; when When the value is 0.5, the update coefficient is 0.5, and the weights of the old and new templates are the same, thus ensuring the stability of the template.

[0210] Step 5: Size-Aware Mask Generation

[0211] A size-aware mask is generated based on the target size. In this embodiment, the mask generation parameters are set as follows: basic expansion radius r = 16 pixels; size weighting parameter η = 2.0; reference size... =32 pixels; regularization coefficient =0.1. The specific context expansion radius is: 24 pixels for the SMALL class, 16 pixels for the MEDIUM class, and approximately 11 pixels for the LARGE class. By setting the mask as described above, smaller targets receive higher weights during training and tracking, thereby improving the stable tracking performance of small targets.

[0212] Step Six: Output Tracking Results

[0213] Finally, the system outputs the target bounding box coordinates bbox=(xc,yc,w,h) and the corresponding tracking confidence score. This information is provided for subsequent control or application modules to call. Verified through the complete experimental process described above, this embodiment can stably achieve real-time detection and tracking in a mobile environment, and exhibits higher robustness in small-sized target scenarios.

[0214] Example 2

[0215] A size-aware knowledge distillation training strategy

[0216] like Figure 4 As shown in the figure, this embodiment discloses a size-aware knowledge distillation training method for training a lightweight target tracking model. By introducing target size information to modulate the distillation process, the tracking performance of the student model under target conditions of different scales is improved.

[0217] (a) Construction of teacher-student network

[0218] In this embodiment, a distillation training framework consisting of a teacher network and a student network is constructed. The teacher network uses a large-scale Vision Transformer (ViT-Base) with frozen parameters as the feature extraction backbone network to provide high-quality intermediate feature representations, attention distributions, and prediction results; the student network uses a lightweight Vision Transformer (ViT-Tiny) as the backbone network to learn the representation capabilities of the teacher network while ensuring a small model size.

[0219] (II) Training Data Preparation

[0220] A target tracking dataset containing targets at multiple scales was selected as the training samples. During training, the template image was cropped to 128×128 pixels, the search region was cropped to 256×256 pixels, and data augmentation operations such as random flipping, color perturbation, and scale changes were performed on the training samples to improve the model's adaptability to scale changes.

[0221] (III) Teacher and Student Network Parameter Settings

[0222] The teacher network employs a Vision Transformer architecture with the following parameters: embedding dimension of 768, attention heads of 12, and Transformer layers of 12. The student network uses a lightweight Vision Transformer architecture with the following parameters: embedding dimension of 256, attention heads of 4, and Transformer layers of 6. Feature dimension alignment is achieved by mapping the intermediate layer features output from the teacher network to the feature space of the student network through a linear projection layer.

[0223] (iv) Size-aware distillation strategy and weight setting

[0224] During training, a size-aware distillation strategy is introduced based on the actual size information of the target in the current sample, and different distillation constraints are weighted and modulated. In this embodiment, the reference target size is set to 32 pixels: when the target size is 24 pixels, the weights of the target region structure distillation term and the attention distillation term are increased, so that the student model focuses on learning the local structural relationships and attention distribution features of small-sized targets; when the target size is 96 pixels, the weight of the global semantic distillation term is increased, so that the student model focuses on learning the overall semantic representation ability of large-sized targets. Through the above method, the distillation process can be adaptively modulated according to the changes in target scale.

[0225] (v) Training and optimization process

[0226] In this embodiment, the training parameters are set as follows: the initial learning rate is 1×10⁻ 4 The batch size was 32, the training epochs were 100, the optimizer used was AdamW, and the learning rate scheduling method was cosine annealing. During training, the student network was optimized by combining the target tracking task loss and the size-aware modulation distillation loss, so that the student model could still maintain high target tracking accuracy with a significant reduction in the number of parameters.

[0227] Example 3

[0228] Mobile tracking application examples in multi-size target scenarios

[0229] In this embodiment, various video scenes containing small, medium, and large targets are selected as test samples, and the detection-tracking device described in Embodiment 1 is deployed on a smartphone for comparative testing.

[0230] In the comparative experiment, the target tracking performance was observed with and without the size-aware masking mechanism enabled. The experimental results show that in small target scenarios, enabling the size-aware masking mechanism significantly improves the tracking stability of the target under long-distance and fast-moving conditions, and significantly reduces tracking box jitter. In medium and large target scenarios, the system can maintain stable tracking and avoid positioning offset caused by background interference.

[0231] The above embodiments demonstrate that the size sensing mechanism proposed in this invention has good adaptability under different target scale conditions and can operate stably in real-time mobile application environments.

[0232] Figure 4 This is a schematic diagram of the size-aware knowledge distillation training framework in this invention. (See diagram below.) Figure 4 As shown, the distillation framework comprises a teacher network and a student network. The teacher network employs a large-scale Vision Transformer with frozen parameters to extract high-quality feature representations, structural relationships, and attention distributions. The student network uses a lightweight Vision Transformer structure for real-time inference on mobile devices. During training, both the teacher and student networks extract features from the input image. Through a feature alignment module and a size-aware distillation module, global semantic distillation, target region structural distillation, and attention distillation constraints are introduced. The distillation loss weights are adjusted according to the target size, thereby guiding the student network to learn effective representation capabilities for targets of different sizes.

[0233] This invention addresses the problem of insufficient adaptability to changes in target size during mobile target detection and tracking, proposing the following key technical points:

[0234] (1) Overall architecture for size-aware detection and tracking for mobile devices. A size-aware visual detection and tracking device and method for mobile devices is proposed. The target detection, target tracking, state decision, template management and size perception mechanism are designed in a coordinated manner to achieve stable detection and continuous tracking of targets of different sizes while ensuring the real-time performance of mobile devices.

[0235] (2) A state decision maker with three states, INIT, TRACKING and LOST, was designed, and a comprehensive tracking confidence and its smoothing mechanism were introduced as a unified judgment basis to realize adaptive scheduling between detection and tracking, reduce unnecessary detection calls, and improve recovery stability in target loss scenarios.

[0236] (3) Establish a template library containing static and dynamic templates of multiple sizes. Match the static and dynamic templates simultaneously during each frame tracking and select the template with higher score for target localization. Update the dynamic templates using a confidence-driven smooth weighting method to improve the matching stability in appearance change scenarios.

[0237] (4) A size-aware masking strategy is proposed, which dynamically adjusts the context extension radius and mask strength coefficient according to the target size, so that different size targets adopt differentiated feature modeling methods, thereby improving the tracking stability in the scenario of size change.

[0238] (5) In the offline training stage, a size-aware knowledge distillation method is introduced. The weights of global semantic distillation, structural distillation and attention distillation are adaptively adjusted according to the target size, so that the lightweight model can obtain feature expression capabilities that are more in line with the characteristics of different target sizes while keeping the model size under control.

Claims

1. A size-sensing visual detection and tracking device for mobile devices, characterized in that, It includes a video acquisition unit, a detection unit, a tracking unit, a detection-tracking connection unit, a size perception module, a template management unit, and an offline training unit; The video acquisition unit is connected to the detection-tracking connection unit, which is connected to both the tracking unit and the detection unit. The tracking unit is connected to the size perception module, which is connected to the tracking unit. The tracking unit is connected to the template management unit.

2. The size-sensing visual detection and tracking device for mobile terminals according to claim 1, characterized in that, The offline training unit includes a teacher network, a student network, and a deployment to a tracking unit; The teacher network is connected to the student network, and the student network is connected to the tracking unit.

3. The size-sensing visual detection and tracking device for mobile terminals according to claim 1 or 2, characterized in that, The video acquisition unit is used to acquire video frames captured by the mobile terminal camera; The detection unit is used to perform target detection on the input video frame and output target candidate boxes and detection confidence scores. The tracking unit is used to predict the target location based on historical templates and the current search area; The detection-tracking connection unit is used for scheduling and fusion between the detection unit and the tracking unit according to the target state; The size perception module is used to dynamically modulate feature modeling and attention allocation based on target size information; The template management unit is used to maintain static and dynamic templates of various sizes and to perform template updates. The offline training unit is used to perform joint optimization training of size-aware focusing loss supervision and size-aware knowledge distillation on the lightweight model during the offline training phase, and is also used to perform joint optimization of size-aware focusing loss supervision and size-aware knowledge distillation on the lightweight model during the offline training phase.

4. A size-aware visual detection and tracking method for mobile devices, characterized in that, Includes the following steps: Step S1: The system takes the video stream captured by the mobile device as input and adaptively switches between target detection, target tracking, and target detection and target tracking fusion according to the target status: when in the initialization or loss recovery state, the detection module is triggered to perform target detection on the current frame; When in a stable tracking state, tracking prediction is performed, and when necessary, the detection results and tracking prediction results are correlated and fused through the detection-tracking connection unit to update the target state and position; Step S2: During tracking, the system activates a size-aware masking mechanism to optimize attention allocation based on changes in target size. In the offline training unit, size-aware focusing loss is used to supervise student network attention, and size-aware knowledge distillation is combined to enhance the lightweight representation ability of the student network learning the teacher model on targets of different sizes. In the online inference phase, size-aware focusing loss is not calculated; only size-aware knowledge distillation is used. Size-aware masking strategies dynamically modulate attention to improve tracking stability and positioning accuracy.

5. The size-aware visual detection and tracking method for mobile devices according to claim 4, characterized in that, Step S1 includes the following steps: Step S11: Set up the state decision maker; Step S12: During the initialization phase of the detection and tracking fusion process, the target correlator is used to match the detection results with the tracking prediction results. Step S13: Used in the TRACKING stage. In each frame tracking stage, the system automatically selects the corresponding size template based on the size of the current detection box or prediction box, and simultaneously matches it with the dynamic template to realize a dual template candidate mechanism, thereby further improving the matching stability in appearance change scenarios.

6. The size-aware visual detection and tracking method for mobile devices according to claim 4, characterized in that, Step S2 includes the following steps: Step S21: Construct the Vision Transformer architecture diagram; Step S22: Dynamically adjust the attention range and inhibition intensity by the target size, so that the Transformer network can adopt differentiated feature focusing strategies for targets of different sizes; Step S23: Transmit multi-level and multi-dimensional knowledge information to the lightweight student model through a large-scale teacher model, and adaptively adjust the distillation intensity according to the target size, so that the student network can obtain supervision signals that match the task difficulty under different target size conditions, thereby improving tracking accuracy and stability while ensuring real-time performance.

Images