An unmanned aerial vehicle real-time target tracking method based on asynchronous semantic correction

Abstract

This invention discloses a real-time target tracking method for unmanned aerial vehicles (UAVs) based on asynchronous semantic correction. It employs a spatiotemporally decoupled architecture of a high-frequency geometric main thread and an asynchronous semantic correction thread. The main thread estimates the target's geometric state frame-by-frame using sparse feature tracking and filtering models, outputting a target prediction bounding box. The asynchronous thread calls a lightweight semantic segmentation model at low frequency according to trigger conditions to generate a target semantic mask and uses the motion trajectory within the delay period for spatiotemporal alignment compensation. The system performs dual reliability checks on semantic input and output, updating the target state through a hierarchical write-back mechanism. When the main thread's geometric propagation fails, an emergency fallback tracker takes over the output to ensure tracking continuity. This invention enables category-independent, real-time, robust target tracking in open scenes on edge platforms.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and UAV intelligent perception technology, and particularly relates to a real-time target tracking method for UAVs based on asynchronous semantic correction. Background Technology

[0002] Unmanned aerial vehicle (UAV) platforms, due to their high mobility, flexible deployment, wide observation range, and low operating costs, have been widely applied in scenarios such as security patrols, disaster search and rescue, traffic monitoring, border patrols, and low-altitude target reconnaissance. In these applications, video target tracking technology is used to output target position and contour information in real-time and stably within continuous video sequences, serving as a crucial foundational technology for UAV situational awareness, autonomous path planning, and subsequent mission decision-making. As UAV applications expand from structured environments to open, unconstrained environments, higher demands are placed on the robustness of tracking methods under complex conditions such as drastic changes in target scale, large-viewpoint disturbances, short-term occlusion, and sudden changes in illumination.

[0003] Existing video target tracking technologies can be broadly categorized into two types. The first type is based on traditional computer vision methods, represented by discriminative correlation filtering, generative sparse optical flow, and Kalman filtering and particle filtering. These methods estimate target location by manually designed features combined with online update mechanisms. The second type is based on deep learning methods, represented by Siamese network matching frameworks and Transformer-based feature extraction and attention tracking frameworks. These methods pre-train offline on large-scale tracking datasets and output target bounding boxes using feature matching or end-to-end regression. Furthermore, in recent years, cue-driven semantic segmentation models such as SAM and EfficientSAM have emerged, capable of outputting pixel-level foreground segmentation results through box cues or point cues, providing new technical means for target contour constraints and spatial localization.

[0004] However, the above methods still have the following significant limitations in the actual deployment of UAV airborne edge platforms: First, traditional correlation filtering and deep learning tracking methods are prone to a decline in generalization ability when the target category or scene distribution exceeds the pre-training coverage, leading to target drift, tracking box jumps, or even loss of lock. Second, if the semantic segmentation model is directly applied frame by frame to the real-time video stream, its high inference overhead makes it difficult to meet the real-time requirements of the airborne edge platform. Although using asynchronous threads to perform semantic inference can alleviate the main thread blocking, there is a temporal deviation between the asynchronously returned historical semantic results and the current frame system state. The lack of an effective delay motion compensation mechanism can easily cause tracking box jumps and state inconsistencies. Third, when the tracking box experiences local drift, using the current tracking box as the semantic segmentation input prompt will induce incorrect segmentation. If the low-quality segmentation results lack multi-dimensional quality evaluation and graded write-back mechanism, directly writing back the system state will cause state pollution and drift accumulation. In addition, when the main thread geometric propagation becomes unstable for a short time and the asynchronous semantic correction results have not yet been returned or are temporarily unavailable, the existing solutions lack a lightweight fallback mechanism, which can easily create a short-term output gap. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a real-time target tracking method for UAVs based on asynchronous semantic correction. It aims to solve the timing misalignment problem by using asynchronous semantic correction and delayed motion compensation, avoid state contamination by combining multi-dimensional quality assessment, and introduce a lightweight fallback mechanism to ensure output continuity, so as to achieve real-time, robust, and category-independent target tracking in open scenarios on UAV airborne edge platforms.

[0006] Technical solution: The real-time target tracking method for UAVs based on asynchronous semantic correction described in this invention includes the following steps:

[0007] S1. Obtain the first frame image of the UAV video sequence and the initial bounding box of the target. Use a lightweight cue-driven semantic segmentation model to generate an initial semantic mask of the target based on the initial bounding box of the target. Construct an initial compact anchor box of the target based on the initial semantic mask of the target, and initialize the target mask constraint region, the target feature point set and the target appearance template. At the same time, establish a filtering model for target geometric state estimation, and establish a historical transformation sequence for recording the geometric transformation of adjacent frames and an emergency fallback tracker for abnormal scene takeover.

[0008] S2. In the main thread, based on the target geometric state of the previous frame, a frame-by-frame high-frequency geometric propagation is performed by combining sparse feature tracking with the filtering model, outputting the target prediction box of the current frame, and using the target prediction box as the system's default tracking result and the prompt input for the asynchronous semantic branch.

[0009] S3. In an asynchronous thread, the semantic segmentation task is triggered at a low frequency according to the preset triggering conditions. The target prediction box output by the main thread is used as the prompt input. The lightweight prompt-driven semantic segmentation model is called to perform non-blocking inference and generate the target semantic mask and the corresponding confidence information corresponding to the semantic segmentation result.

[0010] S4. Based on the target motion trajectory output by the main thread within the delay period corresponding to semantic reasoning, perform spatiotemporal alignment compensation on the target semantic mask generated by the asynchronous thread, map the semantic results of historical moments to the current frame coordinate system, and obtain the aligned semantic mask and bounding box.

[0011] S5. Perform input reliability verification on the prompt input in the asynchronous thread, and perform output reliability verification on the semantic output result corresponding to the prompt input. Only use the semantic output result that passes the verification for system state update, and perform hierarchical write-back update on the target geometric state, target mask constraint region, target feature point set and target appearance template according to the quality of the semantic output result.

[0012] S6. Monitor the health status of the main thread geometric propagation in real time. When the main thread geometric propagation fails or the propagation reliability is continuously lower than the preset threshold, start the emergency fallback tracker to take over the output, maintain continuous output at the target position, and switch back to the main thread tracking results after the main thread recovers.

[0013] This solution effectively addresses the temporal misalignment between semantic segmentation and main-thread geometric propagation in UAV tracking through asynchronous semantic correction and delayed motion compensation, ensuring that high-quality semantic information from historical moments is accurately aligned to the current frame. Combined with multi-dimensional input / output reliability verification and a hierarchical write-back update mechanism, it mitigates the contamination of the system state by low-quality or erroneous semantic results, guaranteeing the robust evolution of target geometric state, feature points, and appearance templates. Simultaneously, a lightweight emergency fallback tracker is introduced to seamlessly take over the output when main-thread geometric propagation fails, maintaining continuous tracking capability. Overall, this method achieves real-time, category-independent, and highly robust target tracking in open scenarios on a UAV-borne edge platform.

[0014] Preferably, step S1 includes:

[0015] S1.1 Acquire the first frame image of the video sequence captured by the UAV's onboard camera, and preprocess the first frame image, including image decoding, size normalization, color space conversion, and tensor input, to obtain a standardized first frame image; S1.2 Receive the initial bounding box of the target provided by the user on the standardized first frame image; S1.3 Use the standardized first frame image and the initial bounding box of the target as box cue, input them into a lightweight cue-driven semantic segmentation model, generate an initial binary segmentation mask for the target, and perform post-processing on the initial binary segmentation mask, including morphological opening and closing operations, connected component analysis, maximum connected component preservation, hole filling, and small... S1.4. Area noise regions are removed to obtain the initial semantic mask of the target; S1.5. The minimum bounding rectangle of the target foreground region is calculated based on the initial semantic mask of the target to generate the target compact anchor frame; S1.6. The target geometry is initialized based on the target compact anchor frame, the target geometry includes the target center x-coordinate, the target center y-coordinate, the target width and the target height, and a linear Kalman filter model under the assumption of constant velocity motion is established as the filter model for the target geometry estimation, and the process noise covariance matrix Q, the observation noise covariance matrix R and the initial error covariance matrix of the linear Kalman filter model are initialized; S1.7. Based on the initial semantic mask of the target or the target compact anchor frame, the minimum bounding rectangle of the target foreground region is calculated to generate the target compact anchor frame; S1.8. The target geometry is initialized based on the initial semantic mask of the target or the target compact anchor frame. Within the bounded area, extract Shi-Tomasi corner feature points, retaining only those located inside the target foreground mask with corner response values ​​greater than a preset threshold, and construct an initial feature point set; S1.7, Based on the standardized first frame image and the target compact anchor box, reuse the image embedding features output by the lightweight cue-driven semantic segmentation model encoder in step S1.3, and extract the target appearance descriptor through region cropping, adaptive average pooling, flattening, and normalization processing to construct a dual-track appearance template. The dual-track appearance template includes a frozen baseline template and a dynamic template. During initialization, the same target appearance descriptor is simultaneously assigned to the frozen baseline template and the dynamic template; S1.8, A historical transformation sequence is established to record the geometric transformations between adjacent frames. The maximum length of the historical transformation sequence matches the maximum allowable delay frame number for asynchronous semantic correction. At the same time, an emergency fallback tracker is initialized, using the target compact anchor frame as the initial target position input, so that the emergency fallback tracker is in hot standby mode. S1.9 After completing the initialization of steps S1.1 to S1.8, the system enters the online tracking stage. The main thread performs high-frequency geometric propagation based on the target compact anchor frame, the target geometric state, the initial feature point set, the dual-track appearance template, and the initial state corresponding to the linear Kalman filter model. The asynchronous thread performs low-frequency semantic correction according to a preset fixed trigger period.

[0016] The optimized initialization process described above generates a relatively compact and pure target semantic mask and anchor box through lightweight prompt-driven segmentation and post-processing, reducing background noise interference. On this basis, reliable feature points are extracted by combining Shi-Tomasi corner points with mask constraints, and encoder features are reused to construct a dual-track appearance template, improving the accuracy of initial features and appearance representation. At the same time, Kalman filtering, historical transformation stack, and hot standby emergency tracker are established to provide a relatively stable initial state and fault-tolerant foundation for the subsequent coordinated operation of high-frequency geometric propagation and low-frequency semantic correction.

[0017] Preferably, step S2, which involves combining sparse feature tracking with the filtering model to perform frame-by-frame high-frequency geometric propagation, includes:

[0018] S2.1. Based on the target geometry, target mask constraint region, and predicted state of the filtering model after gating confirmation in the previous frame, determine the local tracking region in the current frame; S2.2. Within the local tracking region, extract the Shi-Tomasi corner feature point set based on the target mask constraint region of the previous frame, perform inter-frame feature point propagation using the pyramid Lucas-Kanade sparse optical flow method, and eliminate mismatched points through forward and backward consistency checks to obtain a set of valid matching point pairs; wherein, the response value R of the Shi-Tomasi corner point is defined as two eigenvalues ​​of the local neighborhood structure tensor M of the image. , The minimum value in, i.e. A corner point is included in the feature point set only when R is greater than a preset corner point threshold and the corner point is located within the foreground region of the target mask; S2.3, a random sampling consensus algorithm is used to fit a two-dimensional similarity transformation matrix between the previous frame and the current frame to the set of valid matching point pairs. The two-dimensional similarity transformation matrix It includes at least a translation component and a uniform scale variation factor; during the fitting process, the reprojection error of each effective matching point is calculated, and effective matching points with reprojection errors less than a preset reprojection error threshold are determined as inliers. A set of inliers that satisfy model consistency is retained, and the geometric consistency score of the current frame is calculated based on the set of inliers.

[0019]

[0020] in Let be the number of elements in the set of interior points. The geometric consistency score represents the total number of matching point pairs. Used to characterize the reliability of the local geometry propagation results in the current frame.

[0021] The aforementioned preferred sparse feature tracking process reduces background feature interference by combining target mask constraints to extract Shi-Tomasi corner points; it alleviates feature mismatch problems caused by rapid inter-frame motion or occlusion by employing pyramid Lucas-Kanade optical flow and forward / backward consistency verification; and it further quantifies the credibility of local geometric propagation results to a certain extent by fitting a two-dimensional similarity transformation through random sampling consistency and calculating geometric consistency scores, providing a basis for monitoring the health status of subsequent geometric propagation.

[0022] Preferably, the output of the target prediction box in the current frame in step S2 includes:

[0023] According to the two-dimensional similarity transformation matrix A geometric transformation is performed on the target bounding box of the previous frame to obtain the initial propagation box of the current frame. The center coordinates of the initial propagation box are used as the observation vector and input into the filtering model. A prediction step and an update step are executed, and the filtered and smoothed target geometric state is output to generate the target prediction box of the current frame. The filtering model adopts a two-dimensional uniform motion model, and its state vector is defined as follows: ,in, , The image coordinates of the target center. , Let the target's inter-frame motion velocity be in the horizontal and vertical directions; the prediction and update steps of the filtering model satisfy:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] in, For prior state estimation, Here is the state transition matrix. For the posterior state estimation of the previous frame, Let be the prior error covariance matrix. The posterior error covariance matrix of the previous frame. For matrix transpose, The process noise covariance matrix is... For Kalman gain, For the observation matrix, To observe the noise covariance matrix, For posterior state estimation, This is the observation vector output by the geometric propagation of the current frame. Let be the posterior error covariance matrix. The identity matrix is ​​used; after obtaining the target prediction bounding box of the current frame, the internal state of the main thread is updated. The internal state of the main thread includes the target geometric state of the current frame, the set of effective feature points, the two-dimensional similarity transformation matrix, and the geometric consistency score; the two-dimensional similarity transformation matrix is ​​then used to... The attached frame number and timestamp are pushed into the historical transformation sequence, and the number of valid feature points in the current frame is counted. The number of valid feature points and the geometric consistency score are jointly output as the main thread propagation reliability index.

[0030] The above-mentioned preferred propagation box generation process uses the initial propagation box obtained by the two-dimensional similarity transformation as the observation input of the filtering model. Combined with the prediction update mechanism under the assumption of uniform motion, it can suppress the disturbance of the target box by the single-frame feature matching error to a certain extent. At the same time, the two-dimensional similarity transformation matrix is ​​pushed into the history stack, and the propagation reliability index is formed by combining the number of effective feature points and the geometric consistency score. This provides a quantifiable basis for the triggering of subsequent asynchronous semantic correction and the decision of abnormal takeover.

[0031] Preferably, step S3 includes:

[0032] S3.1, according to the preset fixed trigger cycle A separate asynchronous thread is awakened, recording the frame number, timestamp, target prediction bounding box output by the main thread, current target geometric state, and main thread propagation reliability index at the current trigger moment as context input for the semantic correction task. After the asynchronous thread starts, the main thread continues to execute the high-frequency geometric propagation in step S2 without waiting for the asynchronous thread to return a result. The asynchronous thread employs a mutual exclusion request mechanism, retaining only one valid asynchronous inference task at any given time. If the previous round of asynchronous requests has not yet been completed, no new inference request is submitted. If the asynchronous inference duration exceeds twice the trigger cycle... Then, the semantic correction task is forcibly terminated and the corresponding semantic observation results are marked as invalid results; S3.2, in the asynchronous thread, the target prediction box output by the main thread is used as the initial cue box, and the image embedding features output by the encoder of the lightweight cue-driven semantic segmentation model are reused to perform cue self-correction: with the initial cue box as the center, a set of candidate cue positions is constructed in the neighborhood, and for each candidate cue position, the corresponding region features are extracted from the image embedding feature graph, and then subjected to adaptive average pooling, vectorization and... After normalization, the feature vector of the candidate region is obtained. Calculate separately Features of the frozen baseline template Dynamic template features cosine similarity And obtain the comprehensive similarity through weighted fusion. :

[0033]

[0034]

[0035] in, Indicates the features of the reference template. The weighting coefficient for cosine similarity is used; the candidate position with the highest comprehensive similarity and that meets the preset threshold condition is determined as the center coordinate of the corrected prompt box, while keeping the width and height of the prompt box unchanged. If none of the candidate positions meet the threshold condition, the initial prompt box is retained as the default prompt box; S3.3, using the corrected prompt box as the boundary prompt input, the lightweight prompt-driven semantic segmentation model is called to perform asynchronous inference to obtain the target binary semantic mask corresponding to the trigger time, and the target binary semantic mask is post-processed by performing morphological opening and closing operations, connected component analysis and maximum connected region preservation, and the minimum bounding rectangle of the post-processed mask is extracted as the initial semantic correction box.

[0036] The aforementioned preferred asynchronous semantic correction process reduces the blocking impact of semantic segmentation inference on the high-frequency geometric propagation of the main thread through independent asynchronous threads and mutual exclusion request mechanisms. The introduction of a prompt self-correction strategy based on dual-track appearance templates before model inference can alleviate the prompt bias caused by slight drift of the main thread prediction box to a certain extent, thereby improving the input quality of semantic segmentation. At the same time, the combination of candidate position similarity evaluation and preset threshold judgment only affects the semantic output when the correction result is reliable, which helps to maintain the stability of the system in complex scenarios.

[0037] Preferably, step S4 includes:

[0038] S4.1 After the asynchronous thread completes the inference in step S3.3 and returns the target binary semantic mask and the initial semantic correction box to the main thread, it reads the two-dimensional similarity transformation matrix of all adjacent frames from the trigger time to the current time from the historical transformation sequence in step S2, and calculates the cumulative transformation matrix by multiplying them in chronological order. ,in For the asynchronous thread triggering time, The current moment when the asynchronous result is returned. For the first The two-dimensional similarity transformation matrix corresponding to the frame; using the cumulative transformation matrix Perform a similarity transformation mapping on the pixel coordinates of the target binary semantic mask at the trigger time and the vertex coordinates of the initial semantic correction box, transforming the pixel coordinates and vertex coordinates from the coordinate system at the trigger time to the coordinate system at the current time, to obtain the temporally aligned target semantic mask and the temporally aligned semantic correction box; S4.2, generate the semantic observation result of the current asynchronous trigger cycle and send it back to the main thread. The semantic observation result includes at least the temporally aligned target semantic mask, the temporally aligned semantic correction box, the main thread propagation reliability index corresponding to the trigger time, and the comprehensive similarity information obtained during the self-correction phase. If the asynchronous thread of the current asynchronous trigger cycle does not obtain a valid semantic mask, or the historical geometric transformation sequence required for delayed motion compensation is incomplete, or the highest comprehensive similarity is lower than the preset minimum threshold, then the semantic observation result of the current asynchronous trigger cycle is marked as an invalid result, and the main thread is notified to skip the semantic write-back in the current asynchronous trigger cycle.

[0039] The aforementioned preferred delay compensation and alignment process obtains the cumulative transformation by reading the transformation matrices of adjacent frames in the historical transformation sequence and multiplying them together. This process can map the asynchronously triggered semantic mask and correction box from the historical time coordinate system to the current time coordinate system, thereby alleviating the spatiotemporal misalignment problem caused by semantic reasoning delay. At the same time, it combines the integrity of historical transformation, propagation reliability indicators, and prompt self-correction similarity to determine validity. For cases that do not meet the conditions, they are actively marked as invalid results and skipped from writing back, reducing the risk of delayed or low-quality semantic information negatively affecting the main thread state.

[0040] Preferably, step S5, which involves performing input reliability verification on the prompt input in the asynchronous thread, includes:

[0041] S5.1 Receive the aligned semantic mask and bounding box obtained in step S4 as the semantic observation result. If step S3 has marked the semantic observation result of the current asynchronous triggering cycle as invalid, directly reject this semantic write-back and keep the current target state unchanged; otherwise, based on the aligned semantic bounding box, extract candidate observation region features from the embedded feature map output by the lightweight cue-driven semantic segmentation model encoder of the current frame, and perform adaptive average pooling, vectorization, and... Normalization process yields the observed feature vector. S5.2 Calculate the appearance consistency score. Semantic confidence score Conformity score :

[0042]

[0043]

[0044]

[0045] in, The first weighting coefficient for the frozen baseline template; The cosine similarity function has a mathematical range of values. ; The weighting coefficients for the mask quality score. The weighting coefficient for the proportion of effective prospects. The weighting coefficients for the intersection-union ratio of the bounding boxes. And satisfy , To optimize the quality score of mask prediction IoU output by the lightweight cue-driven semantic segmentation model, This represents the percentage of the effective foreground region after post-processing. This is the intersection-over-union ratio (IoU) between the semantically corrected bounding box and the real-time tracking box in the current main thread. The center coordinates of the aligned semantic bounding box. The target center coordinates are predicted by the filtering model. For motion consistency scale parameters; S5.3, read the geometric consistency score output from step S2. Calculate the overall quality score :

[0046]

[0047] in, The weighting coefficients for appearance consistency scores. These are the weighting coefficients for the semantic confidence score. The weighting coefficients for the geometric consistency score. The weighting coefficients for the motion consistency score are: And satisfy .

[0048] The aforementioned preferred input reliability verification process, through multi-dimensional weighted fusion of appearance consistency, semantic confidence, geometric consistency, and motion consistency, can quantify the credibility of semantic observation results from different perspectives. At the same time, combined with the mechanism of directly rejecting invalid results, it alleviates the direct write-back of low-quality or abnormal semantic information to the main thread state. The calculation of the comprehensive quality score provides a relatively comprehensive basis for judging whether the main thread should perform state updates subsequently, which helps to maintain the stability of the tracking system in complex scenarios.

[0049] Preferably, step S5, which involves using only the validated semantic output results for system state updates, includes:

[0050] Based on the comprehensive quality score The three-state gating decision is executed, which divides the hierarchical write-back into three states: full update, partial update, and frozen update; wherein: when When, it enters full update state; when When, it enters a partial update state; when At that time, it enters a frozen update state; among which, , For a preset threshold and .

[0051] The aforementioned preferred three-state gating decision mechanism divides semantic write-back into three states—full update, partial update, and frozen update—by setting two thresholds. It can adaptively adjust the intensity of system state updates based on the interval in which the comprehensive quality score is located. Compared to a simple binary accept or reject strategy, this hierarchical approach helps to fully absorb semantic information when the observation is of high quality, cautiously update key states when the observation is of medium quality, and postpone updates when the observation is of low quality, thereby reducing the risk of state contamination caused by a single unreliable semantic output to a certain extent.

[0052] Preferably, step S5, which involves performing a graded write-back update on the target geometric state, target mask constraint region, target feature point set, and target appearance template based on the quality of the semantic output results, includes:

[0053] When the gating state is full update, the aligned semantic bounding box is written back to the target geometry state, the aligned semantic mask is written back to the target mask constraint region, and the Shi-Tomasi corner feature point set is re-extracted or supplemented within the target mask constraint region; simultaneously, based on the observed feature vector... Update the dynamic template using an exponential moving average method:

[0054]

[0055] in, The historical template momentum coefficient, For the dynamic template features before the update, The updated dynamic template features are used. When the gating state is partial update, only the aligned semantic mask is used to update the target mask constraint region to assist in subsequent feature point selection, without updating the target geometry and target appearance template. When the gating state is frozen update, all write-back operations of the current semantic observation results are rejected, and the target geometry, target mask constraint region, target feature point set and target appearance template remain unchanged.

[0056] The aforementioned preferred hierarchical write-back strategy adaptively adjusts the degree of semantic information injection based on the three-state gating decision results: in the full update state, the geometric state, mask constraints, feature point set, and dynamic template are updated simultaneously, which helps to fully absorb high-quality semantic observation results; in the partial update state, only the mask constraint region is updated, which can optimize subsequent feature point selection without changing the geometric state and appearance model; in the frozen update state, write-back is completely rejected, maintaining the stability of the existing system state. This hierarchical mechanism balances the relationship between semantic information utilization and anomaly observation protection to a certain extent, reducing the risk of system performance degradation due to unreliable semantic observations.

[0057] Preferably, step S6 includes:

[0058] S6.1 Continuously read the geometric consistency score output from step S2. Number of effective feature points and the current target geometry; when Below the preset geometric failure threshold ,or Below the preset feature point failure threshold When this happens, it is determined that the propagation quality of the main thread in the current frame has deteriorated; a continuous failure counter is set. :

[0059]

[0060] in, Indicates the current frame number, when When the main thread is determined to be in a failed state, an emergency fallback tracker is triggered to take over the output. To preset the threshold for the number of consecutive failure frames; S6.2, the emergency fallback tracker maintains a hot standby state throughout the normal operation of the main thread, receiving the current image and the valid target box confirmed by the gating at the previous moment for background updates in each frame; and outputting candidate boxes. Then, appearance consistency verification is performed: the embedded feature map output by the lightweight cue-driven semantic segmentation model encoder is reused from the region corresponding to the candidate box, and then subjected to adaptive average pooling, vectorization, and... Normalization yields the backup observation feature vector Calculate backup tracking verification score :

[0061]

[0062] in, The second weighting coefficient for freezing the baseline template is used to strengthen long-term identity consistency constraints. To freeze the baseline template features, For dynamic template features, Represents the cosine similarity function; only when Higher than the preset verification threshold When the current output of the emergency backup tracker is deemed valid; S6.3, when step S6.1 determines that the main thread has entered a failure state and the backup tracking verification score in step S6.2 is... When the emergency backup tracker takes over the system output, the candidate box is activated. Output as a temporary target location; if the main thread has failed but... If the previous frame's output remains stable, wait for the asynchronous semantic correction result to return; S6.4, During the takeover period, continuously monitor the geometric consistency score of the main thread. With the number of effective feature points And set a continuous recovery counter. :

[0063]

[0064] in, To preset the geometric recovery threshold, Set a preset feature point recovery threshold; when Furthermore, when the most recent asynchronous semantic observation is determined by the gate in step S5 to be a full update or a partial update, a main thread recovery success flag is generated, in which... To preset a continuous recovery frame count threshold; S6.5, when the main thread recovery condition is met, the system outputs a switchback from the result of the emergency fallback tracker to the current geometric propagation result of the main thread, and uses the target state updated by the most recent valid semantic observation as the shared reference state; after the switchback is completed, the emergency fallback tracker exits the takeover state and returns to the hot standby mode, while resetting the continuous failure counter. Continuous recovery counter And a fallback takeover flag; if the takeover duration exceeds a preset takeover duration threshold, the emergency fallback tracker is reinitialized using the target state corresponding to the latest valid semantic correction box.

[0065] The aforementioned preferred emergency fallback mechanism, by continuously monitoring geometric consistency scores and the number of effective feature points, and setting continuous failure judgment conditions, can more smoothly identify the quality decline trend of the main thread's geometric propagation, reducing frequent takeovers caused by occasional fluctuations. The emergency tracker in hot standby mode only participates in system output after appearance consistency verification, which helps to alleviate the accumulation of errors when tracking drifts or mismatches. During the takeover, a differentiated strategy is adopted based on the verification score, maintaining stable output of the previous frame or waiting for semantic correction results. The recovery process combines the achievement of consecutive frame count and recent effective semantic updates for joint judgment. After switching back, the relevant states are reset and the emergency tracker returns to hot standby mode, which improves the continuity and robustness of system output in complex scenarios.

[0066] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. This invention uses a spatiotemporal decoupling architecture of a high-frequency geometric main thread and an asynchronous semantic correction thread, utilizes delayed motion compensation to solve the timing misalignment problem, combines multi-dimensional quality assessment and three-state gating write-back to avoid state pollution, and introduces an emergency fallback tracker to ensure output continuity, achieving real-time, robust, and category-independent target tracking on the UAV edge platform for open scenarios; 2. This invention designs a prompt self-correction strategy, effectively correcting the prompt box offset caused by geometric propagation cumulative error through neighborhood candidate position optimization and dual-template weighted similarity matching, improving the semantic segmentation input quality from the source; 3. This invention constructs a two-stage reliability gating mechanism, comprehensively considering appearance, semantics, geometry, and motion consistency scores, and writes back the semantic observation results in stages as full, partial, or frozen updates, avoiding low-quality segmentation from polluting the system state and ensuring long-term tracking stability; 4. This invention sets up a low-overhead emergency fallback tracker, which seamlessly takes over the output and performs appearance consistency verification when the main thread health status monitoring fails, alleviating the tracking interruption problem in scenarios such as occlusion and rapid movement, and achieving continuous and uninterrupted tracking in complex dynamic environments. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0068] Figure 2 This is the center position error (CLE) curve of the entire sequence in this invention;

[0069] Figure 3 This is a schematic diagram of the tracking result corresponding to the 100th frame of the sequence of the present invention, showing the tracking box output of the target in a normal driving state without obstruction;

[0070] Figure 4 This is a schematic diagram of the tracking result corresponding to the 390th frame of the sequence of the present invention, showing the tracking box output of the target before it is obscured by the road sign;

[0071] Figure 5 This is a schematic diagram of the tracking result corresponding to the 415th frame of the sequence of the present invention, showing the tracking box output of the target during the period when the road sign is obscured;

[0072] Figure 6 This is a schematic diagram of the tracking result corresponding to the 1000th frame of the sequence of the present invention, showing the target tracking box output in a multi-vehicle intersection interference scenario;

[0073] Figure 7 This is a schematic diagram of the tracking result corresponding to the 1200th frame of the sequence of the present invention, showing the tracking box output when the target is turning at the intersection. Detailed Implementation

[0074] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0075] The real-time target tracking method for unmanned aerial vehicles provided by this invention has the following overall process: Figure 1 As shown, the method includes the following steps executed sequentially: system initialization step S1, main thread high-frequency geometric propagation step S2, asynchronous thread low-frequency semantic correction step S3, multi-dimensional gating update and state protection step S4, and emergency fallback tracking step S5, which runs in parallel throughout the process. Steps S2 and S5 are executed in parallel (state detection and fallback output), step S3 is triggered asynchronously and non-blockingly with the main thread, and step S4 performs a state update based on the semantic result output of step S3.

[0076] The following provides a detailed description of the specific implementation methods for each step:

[0077] S1 system initialization, acquiring target initial reference information.

[0078] Step S1 is used to construct the initial semantic representation, geometric representation, and motion state representation of the target before video tracking begins, and to provide a unified initial state for subsequent high-frequency geometric propagation in the main thread, low-frequency semantic correction in the asynchronous thread, gating updates, and anomaly fallback. Step S1 specifically includes the following sub-steps:

[0079] S1.1 Obtain the first frame image of the video sequence

[0080] The first frame of a video sequence captured by the UAV's onboard camera is designated as the initial frame image. This initial frame image includes, but is not limited to, images captured in real-time by the onboard camera, video stream frames output from the electro-optical pod, and offline stored video sequence frames. To ensure consistency in subsequent feature extraction and semantic segmentation, the initial frame image can be preprocessed. This preprocessing includes image decoding, size normalization, color space conversion, and tensor input, thereby obtaining standardized image data suitable for subsequent model calculations.

[0081] S1.2 Receive the initial bounding box of the target.

[0082] The system receives an initial bounding box (BOC) of the target provided by the user on the initial frame image, denoted as the initial target box. The initial target box indicates the approximate location and spatial extent of the target within the initial frame. The generation methods for the initial target box include, but are not limited to, manual selection, interactive point selection, and automatic output from the upstream target detection module. In this embodiment, the initial target box primarily serves as input cues for the subsequent prompt-driven semantic segmentation model, rather than directly as the final geometric reference for the entire online tracking phase, to avoid background redundancy caused by the user's coarse selection interfering with subsequent state propagation.

[0083] S1.3 Obtain the initial mask of the target based on the initial bounding box.

[0084] The initial frame image is input into a lightweight cue-driven semantic segmentation model, and the initial target bounding box is input into the model as a box cue to obtain an initial binary segmentation mask for the target. The semantic segmentation model is preferably EfficientSAM, but other cue-driven semantic segmentation models suitable for edge deployment can also be used. This initial target mask is used to characterize the target foreground region at the pixel level. Compared to using only rectangular boxes to represent the target, it can more accurately describe the target contour and reduce the influence of the background region on subsequent feature sampling and appearance modeling.

[0085] To improve the usability of the initial target mask, post-processing can be performed after obtaining the initial mask. This post-processing includes, but is not limited to, morphological opening and closing operations, connected component analysis, maximum connected region preservation, hole filling, and small-area noise region removal. Specifically, morphological opening operations are used to eliminate isolated noise points, and morphological closing operations are used to fill small holes within the target region. Through these processes, discrete noise points and false foreground regions can be suppressed, improving the stability of semantic representation during the initialization stage.

[0086] S1.4 Generates the target compact anchor frame based on the initial mask.

[0087] Based on the post-processed initial target mask, the minimum bounding rectangle of the target foreground region is calculated to generate a compact anchor frame. This compact anchor frame has a higher spatial fit to the actual target region compared to the user-provided initial target bounding box, and can more accurately reflect the target's center position and scale range. The compact anchor frame serves both as the basis for the target's geometric state during the initialization phase and as the spatial constraint region for subsequent feature point extraction and initial appearance template construction.

[0088] S1.5 Initialize the target geometry and motion model

[0089] The target geometry is initialized based on the compact anchor frame. The target geometry includes at least the x-coordinate of the target center, the y-coordinate of the target center, the target width, and the target height. In one embodiment, a linear Kalman filter model under the assumption of constant velocity motion is also established based on the target geometry for state smoothing and short-term motion prediction in the subsequent main thread. During initialization, the target center position can be set as the position component of the Kalman state vector, and the velocity component can be initialized to zero or a preset default value, thereby completing the transformation from a static initial frame to a structured motion state.

[0090] Furthermore, the process noise covariance matrix Q, observation noise covariance matrix R, and initial error covariance matrix of the linear Kalman filter model are initialized simultaneously. The relevant parameters can be calibrated based on the measured data of the UAV airborne platform, thereby improving the stability and reproducibility of subsequent state estimation.

[0091] S1.6 Initialize the mask-constrained feature point set

[0092] An initial set of feature points is extracted within the region defined by the initial target mask or the compact anchor frame. These feature points are preferably Shi-Tomasi corner feature points suitable for sparse optical flow propagation. Preferably, only feature points located within the target foreground mask and whose response intensity meets a threshold condition are retained to mitigate geometric propagation errors caused by background texture intrusion. This initial set of feature points is used to support inter-frame optical flow propagation, robust geometric estimation, and propagation reliability assessment in subsequent main threads.

[0093] S1.7 Initialize Dual-Track Appearance Template

[0094] Based on the initial frame image and the compact anchor boxes, target appearance features are extracted, and a dual-track appearance template is constructed. The appearance features preferably reuse the image embedding features already output by the semantic segmentation model encoder in step S1.3, without requiring additional model forward inference. Specifically, the target appearance descriptor can be obtained by mapping the compact anchor boxes to the encoder feature map coordinate system, cropping the target region features, and then performing adaptive average pooling, flattening, and normalization processing.

[0095] In this embodiment, the dual-track appearance template includes at least a frozen baseline template and a dynamic template. The frozen baseline template is used to maintain the target's initial identity information long-term and is not updated during online tracking. The dynamic template is used to perform smooth updates under subsequent high-confidence observations to adapt to reasonable changes in the target's scale, attitude, and local appearance. During the initialization phase, the same initial appearance descriptor can be assigned to both the frozen baseline template and the dynamic template simultaneously, providing a unified reference for subsequent self-correction prompts, appearance gating, and candidate identity verification.

[0096] S1.8 Initialization of Historical Transformation Sequence and Emergency Backup Tracker

[0097] A historical transformation sequence is established to store geometric transformation information between subsequent adjacent frames. During the initialization phase, this historical transformation sequence can be an empty stack or pre-set as a set of unit transformations. Preferably, the maximum length of the historical transformation sequence is set to match the maximum allowable delay frame number for asynchronous semantic correction, to avoid unlimited stack growth and excessive onboard memory consumption. As the main thread's tracing process progresses, the similarity transformation matrix of each frame is continuously pushed onto the historical transformation sequence. When the asynchronous semantic segmentation result is returned, the cumulative transformation matrix can be calculated based on the historical transformation sequence from the trigger time to the current time, thereby achieving delay motion compensation of the asynchronous semantic result to the current time coordinate system.

[0098] Simultaneously, a lightweight backup tracker is initialized to provide short-term continuous output in case of main thread geometric propagation instability. In one embodiment, the backup tracker may employ a CSRT correlation filter tracker or other lightweight backup trackers suitable for edge deployment. During initialization, the compact anchor frame is used as its initial target position input, putting it in a hot standby state. After initialization, the backup tracker synchronously receives images and performs background hot standby updates in each subsequent frame, taking over system output only when main thread propagation fails, thereby avoiding initialization delays and tracking interruptions caused by temporary startup.

[0099] S1.9 completes system initialization and enters the online tracing phase.

[0100] After completing the initial target mask, tight anchor frame, target geometry, feature point set, dual-track appearance template, historical transformation sequence, linear Kalman filter model, and asynchronous semantic correction trigger cycle... After initializing the emergency fallback tracker, the system enters the online tracking phase. At this time, the main thread performs high-frequency geometric propagation based on the initial state, while the asynchronous thread follows a preset fixed trigger cycle. Perform low-frequency semantic correction to form a unified starting state for subsequent real-time target tracking processes.

[0101] S2 constructs a high-frequency geometric propagation branch for the main thread and outputs real-time tracking results frame by frame.

[0102] Step S2 is used to perform high-frequency geometric propagation of the current frame based on the effective target state of the previous time step without blocking the real-time output of the system. This process obtains the real-time tracking box, geometric consistency score, number of effective feature points, and geometric transformation matrix between adjacent frames for the current frame. These results are then used as the basis for the system's default output and subsequent asynchronous semantic correction, gating updates, and emergency fallback. Step S2 specifically includes the following sub-steps.

[0103] S2.1 Determine the local tracking region of the current frame based on the target state of the previous time step.

[0104] When processing the current frame image, the effective target state confirmed by gating in the previous time step is first read, including the target geometry, the target mask constraint region, the Shi-Tomasi corner feature point set, and the predicted state of the linear Kalman filter model under the constant velocity motion assumption. Based on the target center position, target scale, and motion prediction results in the previous time step, a local tracking region is determined in the current frame to constrain the calculation range of subsequent feature point propagation and geometric estimation, thereby reducing the interference of irrelevant background regions on real-time propagation.

[0105] The local tracking region includes, but is not limited to, the following two generation methods:

[0106] (1) It is obtained by expanding the target bounding box outward according to a preset ratio from the previous moment;

[0107] (2) The search range is adaptively adjusted by combining the prediction covariance of the linear Kalman filter model, so as to keep the computational cost controllable and retain the necessary propagation margin when the target is translated, scaled and slightly disturbed.

[0108] S2.2 Extract the set of mask-constrained feature points of the current frame and perform inter-frame propagation.

[0109] Within the local tracking area, the set of feature points currently participating in the propagation is extracted based on the spatial constraints of the effective target mask from the previous moment. The feature points are Shi-Tomasi corner feature points that were established and gated and retained in the previous moment; when the number of effective feature points is lower than a preset lower limit, new Shi-Tomasi corner points can be re-detected within the target mask constraint area from the previous moment.

[0110] The Shi-Tomasi corner response value is calculated as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Take its two eigenvalues , The smaller value in the range is used as the corner response value, that is:

[0111]

[0112] Only when the response value A point is included in the set of valid feature points only when its value is greater than a preset threshold and its spatial location is within the foreground area of ​​the target mask.

[0113] Subsequently, the Lucas-Kanade sparse optical flow method is used to perform inter-frame propagation from the previous frame to the current frame on the feature point set, obtaining the corresponding position of each feature point in the current frame. To improve the reliability of the propagation results, a forward and backward consistency check is preferably performed: that is, the propagated points in the current frame are mapped back to the previous frame, and the forward and backward position deviations are calculated; when the deviation is greater than a preset pixel threshold, the corresponding feature point is determined to be a mismatched point and is removed, finally obtaining a set of valid matching point pairs.

[0114] S2.3 Solving similarity transformations and calculating geometric consistency scores based on robust geometric estimation

[0115] For the valid matching point pairs retained after forward and backward consistency verification, a robust estimation method of Random Sample Consensus (RANSAC) is used to fit a two-dimensional similarity transformation matrix between the previous frame and the current frame. The similarity transformation matrix is ​​used to describe the local geometric motion relationship of the target between adjacent frames, including at least translation components and a uniform scale change factor, and rotation components can be added according to scene requirements. Considering that the short-term motion of the target in the UAV scene is mainly manifested as translation and scale change, the rotation component can be set to zero to simplify the model and further reduce the computational cost.

[0116] During the fitting process, the geometric model is fitted by randomly sampling matching points using RANSAC iteration, and the reprojection error of each matching point is calculated. When the reprojection error is less than a preset threshold, the corresponding matching point is determined to be an interior point, and the transformation solution with the most interior points is finally selected as the optimal similarity transformation matrix. And retain the set of interior points that satisfy model consistency. .

[0117] Based on the final set of inliers, the geometric consistency score of the current frame is calculated. The calculation formula is:

[0118]

[0119] in, Let be the number of elements in the set of interior points. This represents the total number of matching point pairs. The range of the geometric consistency score is... The score is used to characterize the reliability of the local geometric propagation results in the current frame. The higher the score, the more reliable the propagation results are, and it serves as the core quantitative input for subsequent three-state gating updates and emergency takeover judgments.

[0120] S2.4 Updates the target geometry and generates the real-time tracking box for the current frame.

[0121] According to the optimal similarity transformation matrix A geometric transformation is performed on the target compact anchor box from the previous time step or the current effective target bounding box to obtain the initial propagation box for the current frame. Further, the initial propagation box is fused with the prediction result of the linear Kalman filter model to obtain the smoothed target geometric state, and a real-time tracking box for the current frame is generated.

[0122] The Kalman filter model adopts a two-dimensional uniform motion model, and its state vector is defined as follows:

[0123]

[0124] in, , The image coordinates of the target center. , The inter-frame motion speed of the target in the horizontal and vertical directions.

[0125] The target center coordinates obtained from optical flow geometric propagation are used as the observation vector and input into the Kalman filter model to execute the complete prediction-update recursive process:

[0126] Prediction steps:

[0127]

[0128]

[0129] Update steps:

[0130]

[0131]

[0132]

[0133] In the formula, Here is the state transition matrix. For the observation matrix, The process noise covariance matrix is... To observe the noise covariance matrix, For Kalman gain, This is the observation vector output by the geometric propagation of the current frame.

[0134] Finally, the target center position is determined using the smoothed posterior state output by the Kalman filter. The target width and height directly inherit the geometric propagation results, generating the real-time tracking bounding box for the current frame. This real-time tracking bounding box serves as the system's default output at the current moment, and also as the initial reference for asynchronous thread prompt self-correction and semantic segmentation prompt input.

[0135] S2.5 updates the main thread state and outputs propagation reliability metrics.

[0136] After obtaining the real-time tracking bounding box of the current frame, the internal state of the main thread is updated, including the target geometric state of the current frame, the set of effective feature points, the geometric transformation matrices of adjacent frames, and the geometric consistency score. Subsequently, the geometric transformation matrix of the current frame, along with the corresponding frame number and timestamp, is pushed into the historical transformation sequence to provide a temporal alignment basis for the delayed motion compensation of subsequent asynchronous semantic results. At the same time, a sliding window is set for the historical transformation sequence, retaining only the transformation data within the maximum delay window of asynchronous semantic correction, and automatically cleaning up historical data exceeding the window to avoid unlimited stack growth and occupation of onboard memory.

[0137] Simultaneously, the number of valid feature points in the current frame is counted, and the number of valid feature points, together with the geometric consistency score, is output as the main thread propagation reliability index. When the reliability index continues to decline, an instability warning can be sent to the emergency fallback tracking mechanism in step S5. When the asynchronous thread is triggered, the real-time tracking box, geometric consistency score, and number of valid feature points in the current frame can also be used as inputs for the self-correction prompt in step S3 and the gating judgment in step S4. If the number of valid matching points is 0 after forward and backward verification, the geometric propagation of the current frame is directly determined to be failed, and a trigger signal is sent to the emergency fallback tracking mechanism.

[0138] S3 constructs an asynchronous thread for low-frequency semantic correction, acquiring semantic observations in a non-blocking manner.

[0139] Step S3 is used to maintain high-frequency geometric propagation and real-time output on the main thread, while following a preset fixed triggering cycle. An independent asynchronous thread is awakened to periodically acquire low-frequency semantic observation results. The availability and temporal consistency of semantic observations are improved through cue-based self-correction and delayed motion compensation. The asynchronous thread runs in parallel with the main thread, and the real-time output of the main thread is not blocked during asynchronous inference. This achieves spatiotemporal decoupling between high-frequency geometric propagation and low-frequency semantic correction, and mitigates the impact of semantic segmentation model inference latency on the real-time output of the edge platform from an architectural perspective. Step S3 specifically includes the following sub-steps.

[0140] S3.1 Wake up asynchronous threads at fixed trigger cycles and record trigger information.

[0141] Once the system enters the online tracking phase, it will follow the fixed triggering cycle initialized in step S1. Wake up an independent asynchronous thread. The fixed triggering period... It is preferred to define it by frame period, but it can also be defined by time interval. Its value is set comprehensively based on the computing power budget of the airborne platform, the inference latency of the semantic segmentation model, and the target's motion speed.

[0142] When the asynchronous thread is triggered, the frame number, timestamp, real-time tracking box of the main thread, current target geometric state, and main thread propagation reliability index corresponding to the current trigger moment are recorded and used as the context input for this semantic correction task. After the asynchronous thread starts, the main thread continues to execute the high-frequency geometric propagation in step S2 without waiting for the asynchronous thread to return the result, thereby ensuring that the system output frame rate is not affected by the inference latency of the semantic segmentation model.

[0143] To avoid semantic results expiring due to asynchronous queue backlog, asynchronous threads employ a mutual exclusion request mechanism: only one valid asynchronous inference task is maintained at any given time. If the previous round of asynchronous requests has not yet completed, no new inference request is submitted, thus ensuring that asynchronous threads always run around the most recently valid state and reducing the risk of state pollution caused by writing expired semantic results back to the system. Furthermore, if the asynchronous inference duration exceeds twice the trigger cycle... If the result is not found, the asynchronous task will be forcibly terminated and marked as invalid.

[0144] S3.2 Multiplexed encoder feature execution prompt self-calibration

[0145] In the asynchronous thread, the real-time tracking box currently output by the main thread is first extracted as the initial prompt box. To reduce the risk of erroneous prompts caused by local drift in the main thread, preferably, a prompt self-correction is performed on the initial prompt box before calling the semantic segmentation model to perform inference.

[0146] The self-correction mechanism fully reuses the image embedding features output by the encoder of the lightweight prompt-driven semantic segmentation model, eliminating the need for a separate deep feature extraction network and avoiding additional model inference overhead. Specifically, a set of candidate prompt locations is constructed within the neighborhood of the real-time tracking box output by the main thread. For each candidate prompt location, corresponding region features are extracted from the encoder's embedded feature map, and then subjected to adaptive average pooling, vectorization, and... After normalization, the feature vector of the candidate region is obtained. Subsequently, the features of the candidate region and the features of the frozen baseline template are calculated respectively. Dynamic template features Cosine similarity:

[0147]

[0148] in, This represents the features of the reference template. Furthermore, the two types of similarity are weighted and fused to obtain the comprehensive similarity of the candidate positions. :

[0149]

[0150] in, The weighting coefficients for cosine similarity are used. The candidate position with the highest overall similarity that meets the preset threshold is determined as the center coordinate of the corrected tooltip, while keeping the width and height of the tooltip unchanged. If none of the candidate positions meet the threshold, the real-time tracking box currently output by the main thread is retained as the default tooltip.

[0151] Through the above-mentioned self-correction of prompts, the local drift of the main thread can be pre-corrected before semantic segmentation inference, reducing the risk of incorrect segmentation caused by prompt offset, blocking the cascading failure link of prompt offset - segmentation error - state pollution - further drift, thereby improving the effectiveness of asynchronous semantic observation.

[0152] S3.3 Performs asynchronous semantic segmentation inference based on the correction prompt box.

[0153] Using the corrected prompt box as the boundary prompt input, the EfficientSAM lightweight semantic segmentation model is called to perform asynchronous inference, obtaining the target binary semantic mask corresponding to the trigger time, and the initial semantic correction box generated by the semantic mask.

[0154] The target semantic mask is a binary foreground mask used to describe the target's foreground region at the pixel level at the trigger time. After obtaining the original semantic mask, post-processing is performed on it, including morphological opening and closing operations, connected component analysis, and maximum connected component preservation. Then, the minimum bounding rectangle of the post-processed mask is extracted to obtain the initial semantic correction box. Due to the inherent inference latency when asynchronous threads call the semantic segmentation model, the target semantic mask and the initial semantic correction box correspond to the target observation results at the trigger time, rather than the instantaneous observation results at the asynchronous result return time.

[0155] S3.4 Perform delayed motion compensation based on historical geometric transformation sequence

[0156] After the asynchronous semantic result is returned to the main thread, read the two-dimensional similarity transformation matrices of all adjacent frames between the trigger time and the current time from the historical transformation sequence maintained in step S2, and calculate the cumulative transformation matrix by multiplying them in chronological order:

[0157]

[0158] in, For the asynchronous thread triggering time, The current moment when the asynchronous result is returned. For the first The two-dimensional similarity transformation matrix corresponding to the frame.

[0159] Subsequently, the cumulative transformation matrix is ​​used to perform a similarity transformation mapping on the target semantic mask pixel coordinates and the initial semantic correction box vertex coordinates at the trigger time, transforming them from the coordinate system at the trigger time to the coordinate system at the current time, resulting in the temporally aligned target semantic mask and the temporally aligned semantic correction box. Through the above-mentioned delayed motion compensation, the temporal misalignment caused by asynchronous semantic inference delay can be effectively compensated, avoiding the problem of tracking box back-jumping and state inconsistency caused by directly writing the semantic observations of historical moments back to the current state.

[0160] S3.5 outputs the time-aligned semantic observations and sends them back to the main thread.

[0161] After completing the delayed motion compensation, the semantic observation results for the current asynchronous cycle are generated. These semantic observation results include at least the temporally aligned target semantic mask, the temporally aligned semantic correction box, the main thread propagation reliability index corresponding to the trigger time, and the comprehensive similarity information obtained during the self-correction phase. Subsequently, the semantic observation results are sent back to the main thread as direct input for step S4, multi-dimensional gating update and state protection.

[0162] In one embodiment, an invalid result masking mechanism is set up: if the asynchronous thread fails to obtain a valid semantic mask within the current cycle, the historical geometric transformation sequence required for delayed motion compensation is incomplete, or the highest comprehensive similarity during the self-correction phase is lower than a preset minimum threshold, then the current asynchronous semantic observation is marked as an invalid result, and the main thread is notified to skip the semantic write-back in the current cycle. This invalid result masking mechanism further prevents low-quality semantic observations from polluting the system state, providing advance protection for subsequent gating updates.

[0163] S4 performs multi-dimensional gating updates and state protection to suppress state pollution.

[0164] Step S4 performs a multi-dimensional quality assessment of the temporal alignment semantic observation results returned in Step S3, and executes a three-state gating update strategy based on the assessment results to suppress state pollution caused by writing erroneous semantic observations back to the system state. Together with the self-correction prompts in Step S3, it forms a two-layer state protection system of "input-end correction and output-end gatekeeping." Step S4 takes the aforementioned main thread geometry propagation results, frozen baseline template, dynamic template, and semantic observation results returned by the asynchronous thread as input, and outputs the updated target geometry state, target mask constraint region, feature point set, and dynamic template. Step S4 specifically includes the following sub-steps.

[0165] S4.1 receives semantic observation results and extracts candidate observation features.

[0166] The main thread receives the current asynchronous periodic semantic observation results returned in step S3. The semantic observation results include at least the temporally aligned target semantic mask, the temporally aligned semantic correction box, the main thread propagation reliability index corresponding to the trigger time, and the comprehensive similarity information obtained during the self-correction phase. If step S3 has marked the current asynchronous semantic observation as an invalid result, the main thread directly rejects the semantic write-back for this period and keeps the current target state unchanged.

[0167] If the current asynchronous semantic observation is a valid result, then based on the temporally aligned semantic correction box, candidate observation region features are extracted from the embedded feature map output by the encoder of the lightweight cue-driven semantic segmentation model in the current frame, and then subjected to adaptive average pooling, vectorization, and... Normalization is performed to obtain the feature vector corresponding to the current semantic observation. This is used for subsequent appearance consistency evaluation.

[0168] S4.2 Calculate appearance consistency score, semantic confidence score, and motion consistency score.

[0169] First, based on candidate observation feature vectors Features of the frozen baseline template Dynamic template features Calculate cosine similarity and fuse the results to obtain an appearance consistency score. :

[0170]

[0171] in, The first weighting coefficient of the frozen benchmark template; The cosine similarity function has a mathematical range of values. Under the normalized semantic embedding feature conditions used in this embodiment, its actual effective range is mainly concentrated in... The appearance consistency score is used to characterize the degree of consistency between the current semantic observation and the target's historical identity representation; a higher value indicates a better identity matching degree.

[0172] Secondly, calculate the semantic confidence score. In one embodiment, the semantic confidence score is derived from the mask prediction IoU quality score output by the lightweight cue-driven semantic segmentation model. The percentage of effective foreground area after post-processing and the intersection-over-union ratio between the semantic correction bounding box and the real-time tracking box of the current main thread. Weighted fusion determination:

[0173]

[0174] in And satisfy The semantic confidence score ranges from [value missing]. A higher score indicates that the current semantic observation is more likely to correspond to the real target region.

[0175] Next, read the geometric consistency score of the current frame output in step S2. The motion consistency score is calculated by combining the Kalman filter prediction results. In one embodiment, the center of the temporally aligned semantic correction box is set to... The target center predicted by Kalman filter is The motion consistency score is defined as follows:

[0176]

[0177] in, This is a parameter for motion consistency. The range of values ​​for the motion consistency score is... This is used to characterize the consistency between semantic observation results and main thread motion predictions; a higher value indicates better motion continuity.

[0178] S4.3 Calculate the overall quality score and execute the three-state gating decision.

[0179] Based on appearance consistency score Semantic confidence score Geometric consistency score and motor consistency score Calculate the overall quality score of the current asynchronous semantic observations. :

[0180]

[0181] in, And satisfy A higher overall quality score indicates greater reliability of the current asynchronous semantic observation.

[0182] Based on overall quality score The three-state gating decision is executed, and the decision logic and state transition are as follows:

[0183] (1) When At that time, it enters the FULL full update state;

[0184] (2) When At that time, it enters the PARTIAL section update state;

[0185] (3) When At that time, it enters the FREEZE freeze update state.

[0186] By using the above three-state gating decision, we can avoid writing low-quality semantic observations directly back to the internal state of the system, while allowing high-quality semantic observations to effectively correct drift propagation in the main thread, thus blocking the cascading failure chain of prompt offset—segmentation error—state pollution—further drift.

[0187] S4.4 Execute state write-back and template update according to the three-state gating strategy

[0188] When the gating state is FULL full update, the temporally aligned semantic correction bounding box is written back to the target geometric state, the temporally aligned target semantic mask is written back to the target mask constraint region, and the Shi-Tomasi corner feature point set is regenerated or supplemented within this mask constraint region. Simultaneously, based on the current semantic observation feature vector... The dynamic template is updated using an exponential moving average method.

[0189]

[0190] in, The historical template momentum coefficient is frozen and remains unchanged throughout the tracking process to continuously preserve the target's initial identity information and provide a stable benchmark for long-term identity verification.

[0191] When the gating state is PARTIAL (partial update), only the spatial constraint information of the current temporally aligned semantic mask on the target foreground region is retained to assist in subsequent main thread feature point selection and local tracking region limitation, without updating the target geometry and appearance template. This partial update preserves the spatial constraint value when the semantic observation quality is at an intermediate level, mitigating feature point dissipation and avoiding potential geometric jumps and identity contamination that might result from direct write-back.

[0192] When the gating state is FREEZE (Freeze Update), all write-back operations of the current asynchronous semantic observation are rejected, and the target geometry, target mask constraint region, feature point set, and dynamic template remain unchanged. Freeze update maintains the stability of the system's internal state when the semantic observation quality is insufficient, preventing erroneous segmentation results from causing a chain reaction on subsequent tracking.

[0193] S4.5 completes state protection and waits for the next asynchronous cycle.

[0194] After completing the three-state gating update for this cycle, the updated target geometry, target mask constraint region, feature point set, and dynamic template are written back to the main thread shared state for use in subsequent high-frequency geometry propagation in step S2 and subsequent self-correction prompts in step S3. If the semantic observation for this cycle is determined to be an invalid result or enters the FREEZE frozen update state, only the gating result for the current cycle is recorded, and the shared state is not modified.

[0195] At the same time, the current period's gating status and overall quality score will be... The scores for each item are written to the status log for subsequent system diagnostics and parameter tuning. After the gating update is complete, the asynchronous thread enters a sleep state, waiting to be awakened again when the next fixed trigger cycle arrives.

[0196] S5 executes a parallel emergency backup tracking mechanism to ensure tracking continuity.

[0197] Step S5 is used to maintain the continuity of system output through a lightweight emergency fallback tracking mechanism when the main thread's high-frequency geometric propagation becomes unstable for a short time and the asynchronous semantic correction results have not yet been returned in time or are temporarily unavailable. This avoids interruption of target position output and fills the tracking vacuum period between the failure of the main thread observation and the asynchronous semantic update. Step S5 runs in parallel with the main thread's step S2, and takes the aforementioned main thread propagation reliability indicators, emergency fallback tracker status, frozen baseline template, dynamic template, and the latest valid target status as inputs. It outputs the target position results during the temporary takeover period and the back-switch control signal after the main thread recovers. Step S5 specifically includes the following sub-steps.

[0198] S5.1 monitors the reliability of main thread propagation in real time and determines failure status.

[0199] During the online operation of the system, the geometric consistency score output in step S2 is continuously read. Number of effective feature points In addition to the current target geometric state, a two-dimensional criterion for the reliability of main thread propagation is constructed. When the geometric consistency score falls below a preset failure threshold... Or the number of valid feature points is lower than the preset failure threshold. When this happens, it is determined that the propagation quality of the main thread in the current frame has deteriorated.

[0200] To suppress false triggering caused by random fluctuations in a single frame and motion blur, a continuous failure counter is set. Its update method is as follows:

[0201]

[0202] in, Indicates the current frame number. When When the main thread is in a failed state, the emergency fallback tracker is triggered to take over the output.

[0203] S5.2 maintains emergency backup tracker hot standby updates and performs visual verification.

[0204] The emergency fallback tracker has completed its initialization synchronously with the main thread during the system initialization phase in step S1, and remains in a hot standby state throughout the normal operation phase of the main thread. In the hot standby state, the emergency fallback tracker receives the current image and the valid target bounding boxes confirmed by gating in each frame, performs lightweight local tracking updates on the candidate target positions, but its output is only cached in the background by default and does not directly overwrite the results of the main thread.

[0205] The emergency backup tracker preferably employs a CSRT discriminative correlation filter tracker, but other lightweight backup trackers suitable for edge platform deployment can also be used. To prevent the backup tracker from drifting in complex backgrounds and scenarios with interference from similar targets, its output candidate boxes... Next, appearance consistency verification is performed. Specifically, the embedded feature map output by the EfficientSAM encoder of the current frame is reused from the region corresponding to the candidate box, and then subjected to adaptive average pooling, vectorization, and... Extracting observed feature vectors after normalization and respectively with the frozen baseline template features and dynamic template features Calculate the cosine similarity and perform weighted fusion to obtain the backup tracking verification score. :

[0206]

[0207] in, The second weighting coefficient of the baseline template is frozen to strengthen long-term identity consistency constraints; Let be the cosine similarity function. Only when Higher than the preset verification threshold Only then is the current output of the backup tracker considered valid.

[0208] When the main thread fails in S5.3, a fallback mechanism is initiated and a temporary location is output.

[0209] When step S5.1 determines that the main thread has entered a failure state, and the backup trace verification score output in step S5.2 meets the threshold condition, the emergency fallback tracer is activated to take over the system output. At this time, the candidate boxes output by the backup tracer... Replaces the current result of the main thread and serves as the system's temporary target output position at the current moment.

[0210] In one embodiment, during the takeover, the main thread's step S2 continues to run, but its output is not directly published externally; it is only used for subsequent recovery determination. The asynchronous thread's step S3 still performs semantic correction according to a preset trigger cycle, and step S4 synchronously performs gating updates, without being interrupted by the fallback takeover. In this way, the system can still maintain continuous position output during the main thread's failure, while preserving the complete state information of the main thread and asynchronous thread for subsequent recovery and switchback.

[0211] If the main thread has been determined to be invalid, but the backup tracking verification score is high. If the threshold is not reached, the stable output result from the previous moment is maintained, while waiting for the asynchronous semantic correction result to return, thereby preventing the low-reliability backup output from further amplifying the tracking drift.

[0212] S5.4 continuously assesses the main thread's recovery conditions during the fallback takeover period.

[0213] During the emergency fallback tracker takeover, the propagation reliability of main thread step S2 and the effectiveness of the asynchronous semantic correction results output by steps S3 and S4 are continuously monitored. When the main thread geometric consistency score exceeds the recovery threshold for multiple consecutive frames... The number of effective feature points exceeds the recovery threshold for multiple consecutive frames. If the most recent asynchronous semantic observation is determined by step S4 to be a FULL update or a PARTIAL update, then the system is considered to have the conditions to switch back to the main thread.

[0214] To avoid frequent jittering switching between the main thread and the fallback branch, a continuous recovery counter is set. Its update method is as follows:

[0215]

[0216] when And when the most recent asynchronous semantic correction result is valid, a main thread recovery success flag is generated.

[0217] S5.5 performs output cutback and resets the fallback state.

[0218] Once the main thread recovery conditions are met, the system output smoothly switches back from the backup tracker results to the current geometry propagation results of the main thread. The target state updated by the most recent valid semantic observation is used as the new reference state shared by the main thread and the emergency fallback tracker. After the switchback is complete, the emergency fallback tracker exits the takeover state and returns to hot standby mode, continuing to perform low-overhead synchronous updates in the background, awaiting the next possible exception trigger.

[0219] During the switchback process, the continuous failure counter is reset synchronously. Continuous recovery counter The system also includes a fallback takeover flag and records the duration of the takeover, the time of the switchback, and key state parameters before and after the switchback for subsequent system diagnosis and parameter tuning. For scenarios where the takeover duration exceeds 10 frames, the target state corresponding to the latest valid semantic correction box can be used to perform on-demand reinitialization of the CSRT emergency fallback tracker to prevent model degradation of the backup tracker after a long-term takeover.

[0220] The real-time target tracking method for UAVs provided in this embodiment achieves a balance between high-frequency real-time output and low-frequency semantic correction on the UAV's onboard edge platform through a spatiotemporally decoupled asynchronous dual-track architecture. Furthermore, it significantly improves tracking robustness in complex scenarios through cue-based self-correction, three-state gating updates, and a hot-standby emergency backup mechanism. To verify the effectiveness of the above method, the car4 sequence from the publicly available UAV aerial photography dataset UAV123 is used as an example. This sequence contains typical challenges such as target scale variations, interference from similar targets, and low resolution. The continuous tracking results of this method on this sequence are as follows: Figures 2-7 As shown, where: Figure 2 The center position error (CLE) curve for the entire sequence visually reflects the change in the target's position deviation during the entire tracking process, providing quantitative support for tracking stability; Figure 3 The image shows the tracking result for the 100th frame of the sequence. At this point, the target is in a normal driving state without any obstruction. The tracking box stably fits the target outline without drifting or misalignment, demonstrating the basic tracking accuracy of the method in normal scenarios. Figure 4 The tracking result is for frame 390 of the sequence. At this time, the target is about to pass the road sign. The tracking box still accurately locks onto the target and is not affected by the background road sign elements, which verifies the method's ability to resist background clutter. Figure 5 The tracking result is for frame 415 of the sequence. At this time, the target is in a road sign occlusion scene. The main thread geometric propagation experienced a short-term fluctuation due to local occlusion. The emergency fallback tracking mechanism and asynchronous semantic correction worked together to ensure that the tracking box could still maintain stable output and the target was not lost due to occlusion. Figure 6 The tracking result is for the 1000th frame of the sequence. At this time, the target is in a multi-vehicle intersection scene with interfering vehicles that look similar. The tracking box can still accurately distinguish the target from the interfering objects through the dual-track appearance template and geometric consistency check, and maintain a stable lock on the target. Figure 7 The image shows the tracking result for frame 1200 of the sequence. At this point, the target is turning at an intersection, and its motion state changes significantly. The combination of Kalman filter prediction and geometric propagation allows the tracking box to adaptively adjust with the target's motion without any jumps or loss, verifying the method's adaptability to changes in the target's motion state. Figure 2 As can be seen, the tracking box of this method remains stably attached to the target throughout the entire sequence of operation, and the center position error is maintained at a low level. There are no problems such as tracking loss, tracking box jump or output interruption, which meets the engineering requirements of real-time tracking of UAVs.

Claims

1. A method for real-time target tracking of a UAV based on asynchronous semantic correction, characterized in that, Includes the following steps: S1. Obtain the first frame image of the UAV video sequence and the initial bounding box of the target. Use a lightweight cue-driven semantic segmentation model to generate an initial semantic mask of the target based on the initial bounding box of the target. Construct an initial compact anchor box of the target based on the initial semantic mask of the target, and initialize the target mask constraint region, the target feature point set and the target appearance template. At the same time, establish a filtering model for target geometric state estimation, and establish a historical transformation sequence for recording the geometric transformation of adjacent frames and an emergency fallback tracker for abnormal scene takeover. S2. In the main thread, based on the target geometric state of the previous frame, a frame-by-frame high-frequency geometric propagation is performed by combining sparse feature tracking with the filtering model, outputting the target prediction box of the current frame, and using the target prediction box as the system's default tracking result and the prompt input for the asynchronous semantic branch. S3. In an asynchronous thread, the semantic segmentation task is triggered at a low frequency according to the preset triggering conditions. The target prediction box output by the main thread is used as the prompt input. The lightweight prompt-driven semantic segmentation model is called to perform non-blocking inference and generate the target semantic mask and the corresponding confidence information corresponding to the semantic segmentation result. S4. Based on the target motion trajectory output by the main thread within the delay period corresponding to semantic reasoning, perform spatiotemporal alignment compensation on the target semantic mask generated by the asynchronous thread, map the semantic results of historical moments to the current frame coordinate system, and obtain the aligned semantic mask and bounding box. S5. Perform input reliability verification on the prompt input in the asynchronous thread, and perform output reliability verification on the semantic output result corresponding to the prompt input. Only use the semantic output result that passes the verification for system state update, and perform hierarchical write-back update on the target geometric state, target mask constraint region, target feature point set and target appearance template according to the quality of the semantic output result. S6. Monitor the health status of the main thread geometric propagation in real time. When the main thread geometric propagation fails or the propagation reliability is continuously lower than the preset threshold, start the emergency fallback tracker to take over the output, maintain continuous output at the target position, and switch back to the main thread tracking results after the main thread recovers.

2. The method of claim 1, wherein, Step S1 includes: S1.1 Acquire the first frame image of the video sequence captured by the UAV's onboard camera, and preprocess the first frame image, including image decoding, size normalization, color space conversion, and tensor input, to obtain a standardized first frame image; S1.2 Receive the initial bounding box of the target provided by the user on the standardized first frame image; S1.3 Use the standardized first frame image and the initial bounding box of the target as box cue, input them into a lightweight cue-driven semantic segmentation model, generate an initial binary segmentation mask for the target, and perform post-processing on the initial binary segmentation mask, including morphological opening and closing operations, connected component analysis, maximum connected component preservation, hole filling, and small... S1.

4. Area noise regions are removed to obtain the initial semantic mask of the target; S1.

5. The minimum bounding rectangle of the target foreground region is calculated based on the initial semantic mask of the target to generate the target compact anchor frame; S1.

6. The target geometry is initialized based on the target compact anchor frame, the target geometry includes the target center x-coordinate, the target center y-coordinate, the target width and the target height, and a linear Kalman filter model under the assumption of constant velocity motion is established as the filter model for the target geometry estimation, and the process noise covariance matrix Q, the observation noise covariance matrix R and the initial error covariance matrix of the linear Kalman filter model are initialized; S1.

7. Based on the initial semantic mask of the target or the target compact anchor frame, the minimum bounding rectangle of the target foreground region is calculated to generate the target compact anchor frame; S1.

8. The target geometry is initialized based on the initial semantic mask of the target or the target compact anchor frame. Within the bounded area, extract Shi-Tomasi corner feature points, retaining only those located inside the target foreground mask with corner response values ​​greater than a preset threshold, and construct an initial feature point set; S1.7, Based on the standardized first frame image and the target compact anchor box, reuse the image embedding features output by the lightweight cue-driven semantic segmentation model encoder in step S1.3, and extract the target appearance descriptor through region cropping, adaptive average pooling, flattening, and normalization processing to construct a dual-track appearance template. The dual-track appearance template includes a frozen baseline template and a dynamic template. During initialization, the same target appearance descriptor is simultaneously assigned to the frozen baseline template and the dynamic template; S1.8, A historical transformation sequence is established to record the geometric transformations between adjacent frames. The maximum length of the historical transformation sequence matches the maximum allowable delay frame number for asynchronous semantic correction. At the same time, an emergency fallback tracker is initialized, using the target compact anchor frame as the initial target position input, so that the emergency fallback tracker is in hot standby mode. S1.9 After completing the initialization of steps S1.1 to S1.8, the system enters the online tracking stage. The main thread performs high-frequency geometric propagation based on the target compact anchor frame, the target geometric state, the initial feature point set, the dual-track appearance template, and the initial state corresponding to the linear Kalman filter model. The asynchronous thread performs low-frequency semantic correction according to a preset fixed trigger period.

3. The method of claim 1, wherein, Step S2, which involves combining sparse feature tracking with the filtering model to perform frame-by-frame high-frequency geometric propagation, includes: S2.

1. Based on the target geometry, target mask constraint region, and predicted state of the filtering model after gating confirmation in the previous frame, determine the local tracking region in the current frame; S2.

2. Within the local tracking region, extract the Shi-Tomasi corner feature point set based on the target mask constraint region of the previous frame, perform inter-frame feature point propagation using the pyramid Lucas-Kanade sparse optical flow method, and eliminate mismatched points through forward and backward consistency checks to obtain a set of valid matching point pairs; wherein, the response value R of the Shi-Tomasi corner point is defined as two eigenvalues ​​of the local neighborhood structure tensor M of the image. , The minimum value in, i.e. A corner point is included in the feature point set only when R is greater than a preset corner point threshold and the corner point is located within the foreground region of the target mask; S2.3, a random sampling consensus algorithm is used to fit a two-dimensional similarity transformation matrix between the previous frame and the current frame to the set of valid matching point pairs. The two-dimensional similarity transformation matrix It includes at least a translation component and a uniform scale variation factor; during the fitting process, the reprojection error of each effective matching point is calculated, and effective matching points with reprojection errors less than a preset reprojection error threshold are determined as inliers. A set of inliers that satisfy model consistency is retained, and the geometric consistency score of the current frame is calculated based on the set of inliers. ;in Let be the number of elements in the set of interior points. The geometric consistency score represents the total number of matching point pairs. Used to characterize the reliability of the local geometry propagation results in the current frame.

4. The method according to claim 3, characterized in that, Step S2, which involves outputting the target prediction box for the current frame, includes: According to the two-dimensional similarity transformation matrix A geometric transformation is performed on the target bounding box of the previous frame to obtain the initial propagation box of the current frame. The center coordinates of the initial propagation box are used as the observation vector and input into the filtering model. A prediction step and an update step are executed, and the filtered and smoothed target geometric state is output to generate the target prediction box of the current frame. The filtering model adopts a two-dimensional uniform motion model, and its state vector is defined as follows: ,in, , The image coordinates of the target center. , Let the target's inter-frame motion velocity be in the horizontal and vertical directions; the prediction and update steps of the filtering model satisfy: ; ; ; ; ;in, For prior state estimation, Here is the state transition matrix. For the posterior state estimation of the previous frame, Let be the prior error covariance matrix. The posterior error covariance matrix of the previous frame. For matrix transpose, The process noise covariance matrix is... For Kalman gain, For the observation matrix, To observe the noise covariance matrix, For posterior state estimation, This is the observation vector output by the geometric propagation of the current frame. Let be the posterior error covariance matrix. The identity matrix is ​​used; after obtaining the target prediction bounding box of the current frame, the internal state of the main thread is updated. The internal state of the main thread includes the target geometric state of the current frame, the set of effective feature points, the two-dimensional similarity transformation matrix, and the geometric consistency score; the two-dimensional similarity transformation matrix is ​​then used to... The attached frame number and timestamp are pushed into the historical transformation sequence, and the number of valid feature points in the current frame is counted. The number of valid feature points and the geometric consistency score are jointly output as the main thread propagation reliability index.

5. The method according to claim 4, characterized in that, Step S3 includes: S3.1, according to the preset fixed trigger cycle A separate asynchronous thread is awakened, recording the frame number, timestamp, target prediction bounding box output by the main thread, current target geometric state, and main thread propagation reliability index at the current trigger moment as context input for the semantic correction task. After the asynchronous thread starts, the main thread continues to execute the high-frequency geometric propagation in step S2 without waiting for the asynchronous thread to return a result. The asynchronous thread employs a mutual exclusion request mechanism, retaining only one valid asynchronous inference task at any given time. If the previous round of asynchronous requests has not yet been completed, no new inference request is submitted. If the asynchronous inference duration exceeds twice the trigger cycle... Then, the semantic correction task is forcibly terminated and the corresponding semantic observation results are marked as invalid results; S3.2, in the asynchronous thread, the target prediction box output by the main thread is used as the initial cue box, and the image embedding features output by the encoder of the lightweight cue-driven semantic segmentation model are reused to perform cue self-correction: with the initial cue box as the center, a set of candidate cue positions is constructed in the neighborhood, and for each candidate cue position, the corresponding region features are extracted from the image embedding feature graph, and then subjected to adaptive average pooling, vectorization and... After normalization, the feature vector of the candidate region is obtained. Calculate separately Features of the frozen baseline template Dynamic template features cosine similarity And obtain the comprehensive similarity through weighted fusion. : ; ;in, Indicates the features of the reference template. The weighting coefficient for cosine similarity is used; the candidate position with the highest comprehensive similarity and that meets the preset threshold condition is determined as the center coordinate of the corrected prompt box, while keeping the width and height of the prompt box unchanged. If none of the candidate positions meet the threshold condition, the initial prompt box is retained as the default prompt box; S3.3, using the corrected prompt box as the boundary prompt input, the lightweight prompt-driven semantic segmentation model is called to perform asynchronous inference to obtain the target binary semantic mask corresponding to the trigger time, and the target binary semantic mask is post-processed by performing morphological opening and closing operations, connected component analysis and maximum connected region preservation, and the minimum bounding rectangle of the post-processed mask is extracted as the initial semantic correction box.

6. The method according to claim 5, characterized in that, Step S4 includes: S4.1 After the asynchronous thread completes the inference in step S3.3 and returns the target binary semantic mask and the initial semantic correction box to the main thread, it reads the two-dimensional similarity transformation matrix of all adjacent frames from the trigger time to the current time from the historical transformation sequence in step S2, and calculates the cumulative transformation matrix by multiplying them in chronological order. ,in For the asynchronous thread triggering time, The current moment when the asynchronous result is returned. For the first The two-dimensional similarity transformation matrix corresponding to the frame; using the cumulative transformation matrix Perform a similarity transformation mapping on the pixel coordinates of the target binary semantic mask at the trigger time and the vertex coordinates of the initial semantic correction box, transforming the pixel coordinates and vertex coordinates from the coordinate system at the trigger time to the coordinate system at the current time, to obtain the temporally aligned target semantic mask and the temporally aligned semantic correction box; S4.2, generate the semantic observation result of the current asynchronous trigger cycle and send it back to the main thread. The semantic observation result includes at least the temporally aligned target semantic mask, the temporally aligned semantic correction box, the main thread propagation reliability index corresponding to the trigger time, and the comprehensive similarity information obtained during the self-correction phase. If the asynchronous thread of the current asynchronous trigger cycle does not obtain a valid semantic mask, or the historical geometric transformation sequence required for delayed motion compensation is incomplete, or the highest comprehensive similarity is lower than the preset minimum threshold, then the semantic observation result of the current asynchronous trigger cycle is marked as an invalid result, and the main thread is notified to skip the semantic write-back in the current asynchronous trigger cycle.

7. The method according to claim 6, characterized in that, Step S5, which involves performing input reliability verification on the prompt input in the asynchronous thread, includes: S5.1 Receive the aligned semantic mask and bounding box obtained in step S4 as the semantic observation result. If step S3 has marked the semantic observation result of the current asynchronous triggering cycle as invalid, directly reject this semantic write-back and keep the current target state unchanged; otherwise, based on the aligned semantic bounding box, extract candidate observation region features from the embedded feature map output by the lightweight cue-driven semantic segmentation model encoder of the current frame, and perform adaptive average pooling, vectorization, and... Normalization process yields the observed feature vector. S5.2 Calculate the appearance consistency score. Semantic confidence score Conformity score : ; ; ;in, The first weighting coefficient of the frozen benchmark template; The cosine similarity function has a mathematical range of values. ; The weighting coefficients for the mask quality score. The weighting coefficient for the proportion of effective prospects. The weighting coefficients for the intersection-union ratio of the bounding boxes. And satisfy , To optimize the quality score of mask prediction IoU output by the lightweight cue-driven semantic segmentation model, This represents the percentage of the effective foreground region after post-processing. This is the intersection-over-union ratio (IoU) between the semantically corrected bounding box and the real-time tracking box in the current main thread. The center coordinates of the aligned semantic bounding box. The target center coordinates are predicted by the filtering model. For motion consistency scale parameters; S5.3, read the geometric consistency score output from step S2. Calculate the overall quality score : ;in, The weighting coefficients for appearance consistency scores. These are the weighting coefficients for the semantic confidence score. The weighting coefficients for the geometric consistency score. The weighting coefficients for the motion consistency score are: And satisfy .

8. The method according to claim 7, characterized in that, Step S5, which involves using only the validated semantic output results for system state updates, includes: Based on the comprehensive quality score The three-state gating decision is executed, which divides the hierarchical write-back into three states: full update, partial update, and frozen update; wherein: when When, it enters full update state; when When, it enters a partial update state; when At that time, it enters a frozen update state; among which, , For a preset threshold and .

9. The method according to claim 8, characterized in that, Step S5, which involves performing a graded write-back update on the target geometric state, target mask constraint region, target feature point set, and target appearance template based on the quality of the semantic output results, includes: When the gating state is full update, the aligned semantic bounding box is written back to the target geometry state, the aligned semantic mask is written back to the target mask constraint region, and the Shi-Tomasi corner feature point set is re-extracted or supplemented within the target mask constraint region; simultaneously, based on the observed feature vector... Update the dynamic template using an exponential moving average method: ;in, The historical template momentum coefficient, For the dynamic template features before the update, The updated dynamic template features are used. When the gating state is partial update, only the aligned semantic mask is used to update the target mask constraint region to assist in subsequent feature point selection, without updating the target geometry and target appearance template. When the gating state is frozen update, all write-back operations of the current semantic observation results are rejected, and the target geometry, target mask constraint region, target feature point set and target appearance template remain unchanged.

10. The method according to claim 9, characterized in that, Step S6 includes: S6.1 Continuously read the geometric consistency score output from step S2. Number of effective feature points and the current target geometry; when Below the preset geometric failure threshold ,or Below the preset feature point failure threshold When this happens, it is determined that the propagation quality of the main thread in the current frame has deteriorated; a continuous failure counter is set. : in, Indicates the current frame number, when When the main thread is determined to be in a failed state, an emergency fallback tracker is triggered to take over the output. To preset the threshold for the number of consecutive failure frames; S6.2, the emergency fallback tracker maintains a hot standby state throughout the normal operation of the main thread, receiving the current image and the valid target box confirmed by the gating at the previous moment for background updates in each frame; and outputting candidate boxes. Then, appearance consistency verification is performed: the embedded feature map output by the lightweight cue-driven semantic segmentation model encoder is reused from the region corresponding to the candidate box, and then subjected to adaptive average pooling, vectorization, and... Normalization yields the backup observation feature vector Calculate backup tracking verification score : ;in, To freeze the second weighting coefficient of the benchmark template, in order to strengthen the long-term identity consistency constraint, To freeze the baseline template features, For dynamic template features, Represents the cosine similarity function; only when Higher than the preset verification threshold When the current output of the emergency backup tracker is deemed valid; S6.3, when step S6.1 determines that the main thread has entered a failure state and the backup tracking verification score in step S6.2 is... When the emergency backup tracker takes over the system output, the candidate box is activated. Output as a temporary target location; if the main thread has failed but... If the previous frame's output remains stable, wait for the asynchronous semantic correction result to return; S6.4, During the takeover period, continuously monitor the geometric consistency score of the main thread. With the number of effective feature points And set a continuous recovery counter. : ;in, To preset the geometric recovery threshold, Set a preset feature point recovery threshold; when Furthermore, when the most recent asynchronous semantic observation is determined by the gate in step S5 to be a full update or a partial update, a main thread recovery success flag is generated, in which... To preset a continuous recovery frame count threshold; S6.5, when the main thread recovery condition is met, the system outputs a switchback from the result of the emergency fallback tracker to the current geometric propagation result of the main thread, and uses the target state updated by the most recent valid semantic observation as the shared reference state; after the switchback is completed, the emergency fallback tracker exits the takeover state and returns to the hot standby mode, while resetting the continuous failure counter. Continuous recovery counter And a fallback takeover flag; if the takeover duration exceeds a preset takeover duration threshold, the emergency fallback tracker is reinitialized using the target state corresponding to the latest valid semantic correction box.

Images