A long-term tracking method incorporating spatiotemporal constraints and neighboring region re-detection

Abstract

The present invention discloses a long-term tracking method that incorporates spatiotemporal constraints and adjacent area re-detection, and relates to the field of computer vision technology. Based on the ToMP tracker, the present invention proposes a long-term tracking algorithm that incorporates spatiotemporal constraints and adjacent area re-detection. Long-term tracking needs to focus on the robustness of the tracker's re-detection and the robustness to visual deformation. First, the confidence evaluation function proposed by the present invention is used to accurately evaluate the tracking quality of the tracker. According to the confidence, the target state and whether re-detection is required are judged. If the target is lost, a neighboring area search is performed on the current frame. Secondly, a dynamic spatiotemporal constraint is proposed to offset the distortion of the re-detected target in spatiotemporal information, and adaptively punish re-detected targets that are farther away. At the same time, an adaptive update strategy is used to update the tracking model, while ensuring efficiency, avoiding the introduction of negative samples, i.e., similar semantic objects.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a long-term tracking method integrating spatiotemporal constraints and adjacent area re-detection. Background Art

[0002] Visual object tracking is a fundamental research problem in computer vision. It involves specifying an arbitrary target in the first frame and accurately locating it in subsequent video sequences. In recent years, visual tracking has been widely used in fields such as public security, video surveillance, robotics, and human-computer interaction. Currently, most trackers predict the target's position under the assumption of smooth and minimal changes in position and scale. This approach fails to account for tracking drift or target loss in complex environments due to occlusion or loss of the target's field of view.

[0003] With advances in deep learning technology, a growing number of deep trackers are emerging. These trackers are not only effective for short-term tracking (seconds) but are also widely used for long-term tracking (minutes). In recent years, most deep trackers have focused on developing robust appearance models to achieve accurate target localization. For example, the TrDiMP tracker uses an attention mechanism to enhance target features and improve the robustness of the target template. The tracker SiamRPN++ uses a hierarchical cross-correlation-based feature aggregation structure to generate multiple similarity graphs with different semantics to build an accurate and robust target tracking model. A feature alignment module is introduced to learn object-aware features from the predicted bounding box, ensuring that feature sampling is performed within the predicted region to adapt to changes in target scale and position.

[0004] The aforementioned methods focus on building robust target features, but they are effectively limited by the quality and discriminative power of appearance models, making it difficult to distinguish between the tracked target and distractors. Furthermore, within their tracking frameworks, once the target is lost, the tracker struggles to redetect it. The SiamDW_LT tracker proposes an online correction model to improve tracking results in the event of target disappearance or occlusion. However, after target loss, similar semantic objects are easily introduced, causing model contamination. SiamR-CNN utilizes dynamic programming to develop a dynamic tracking strategy based on visual deformation, but requires traversing each candidate region, resulting in high time complexity. The ToMP tracker uses the Transformer to capture global features with a small inductive bias, while simultaneously encoding target position and range information, fully utilizing target information. However, it does not account for long-term tracking scenarios, such as when the target disappears from view or is occluded by similar semantic objects, and lacks adaptability to unpredictable mutations.

[0005] Therefore, a new solution to the above problems needs to be proposed. Summary of the Invention

[0006] The object of the present invention is to provide a long-term tracking method that incorporates spatiotemporal constraints and adjacent area re-detection.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a long-term tracking method incorporating spatiotemporal constraints and adjacent area re-detection, comprising at least the following steps:

[0008] Taking the ToMP tracker as the base tracker, an adaptive model update method is designed. Whether the model is updated is jointly determined based on the ASP and APCE values, thereby achieving long-term tracking stability confidence.

[0009] Design a nearby area search detection method. After the target is lost, a nearby search area is constructed. With the search area as the center, nearby windows are adaptively added around the search area. The size of the search area changes adaptively with the number of detection frames until the tracking target is detected. By adaptively adjusting the number of nearby windows, it is controlled within the image range.

[0010] A dynamic spatiotemporal constraint method is designed to penalize situations where objects disappear and suddenly reappear at a long distance. In terms of spatial constraints, when the distance between the new tracked object and the original tracked object is large, the score will be penalized. In terms of temporal constraints, as time goes by and the new tracked object is not found, the distance penalty will be compensated by the time penalty.

[0011] Furthermore, the design adaptive model updating method includes at least the following steps:

[0012] The APCE and ASP proposed in the LMCF algorithm are used as the basis for model update.

[0013] The expression of APCE is as follows:

[0014]

[0015] In the formula , and are the highest response value of the i-th frame, the lowest response value of the i-th frame, and the Response value at position;

[0016] The ASP is combined with historical tracking information to evaluate the tracking quality. The expression of ASP is as follows:

[0017]

[0018] In the formula To avoid small constants with denominators equal to 0, and They represent the maximum response value of the i-th frame and the average maximum response value before the i-th frame respectively;

[0019] Whether the model is updated is determined jointly based on the values ​​of ASP and APCE. The confidence level of long-term tracking stability is as follows:

[0020]

[0021]

[0022] In the formula is the long-term tracking stability confidence, is the scale parameter, because and The values ​​differ greatly and need to be controlled at the same level. is the control parameter, To update the judgment parameters, is the update threshold;

[0023] The model update expression is as follows:

[0024]

[0025] In the formula and Respectively represent the last updated tracking model and the current updated tracking model, is the learning rate, set to 0.01;

[0026] ASP and APCE are combined into a confidence joint judgment formula to obtain the long-term tracking stability confidence , Multiply by the ratio When it is greater than the update threshold T, the model is updated. Add to the mean set.

[0027] Furthermore, the design of the adjacent area search detection method includes at least the following steps:

[0028]

[0029] Where: For the The number of frames adjacent to the window, is the search area for the i-th frame, To track images;

[0030] Set the size of the adjacent window to 0.5 times the search area, and adjust the position of the adjacent window according to the area of ​​the search area in the image;

[0031] judge When the quantity By default, 12 adjacent windows are added to complete the judgment, that is, the search area of ​​12 adjacent windows is added;

[0032] If the target is not detected in the adjacent search area for a long time, the target may be moving behind an obstruction and appearing somewhere outside the search area.

[0033] During the re-detection process, the size of the search area determines the search range, so the size of the search area needs to be adaptively adjusted to achieve the relocation target;

[0034] make represents the search area expansion metric, is the current frame number, and its expression is as follows:

[0035]

[0036] The method for designing dynamic spatiotemporal constraints comprises at least the following steps:

[0037] The confidence expression combined with dynamic spatiotemporal constraints is as follows:

[0038]

[0039] Where, and Represent the position vectors of the new and old positions respectively, represents the diagonal length of the image, , Respectively represent the current frame number and the number of lost frames;

[0040] Depend on get;

[0041] After obtaining the confidence score of the dynamic spatiotemporal constraint, combined with Judge, finally get the re-detection target and complete the re-detection.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This paper proposes a peak absolute stability index and a new long-term tracking stability confidence level. Combined with the average peak correlation energy, this method realizes the adaptive update of the tracking model, avoids the contamination of the model by similar targets, and simultaneously judges the status of the tracked target.

[0044] 2. When the target is lost, the method of the present invention adaptively constructs adjacent areas to obtain target candidate areas, thereby achieving target re-detection;

[0045] 3. The method of the present invention imposes dynamic spatiotemporal constraints on the search target, restricting long-distance changes in the spatiotemporal information of the re-detected target;

[0046] 4. This invention proposes a new long-term tracking algorithm by integrating spatiotemporal constraints and adjacent area re-detection, which effectively solves the challenges caused by target disappearance and occlusion during long-term tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A schematic diagram of the tracking algorithm flow of the present invention;

[0049] Figure 2 Schematic diagram of the visualization of IoU and re-detection between LTMP and ToMP of the present invention;

[0050] Figure 3 Schematic diagram showing the qualitative results of the six tracking algorithms of the present invention on some representative video sequences. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0052] See also Figure 1 - Figure 3 A long-term tracking method incorporating spatiotemporal constraints and neighboring area re-detection includes at least the following steps:

[0053] Based on the ToMP tracker, an adaptive template updating method is designed;

[0054] Design a nearby area search and detection method;

[0055] Design dynamic spatiotemporal constraint methods.

[0056] The ToMP tracker is a leading correlation filter tracker. It integrates a target model prediction module, uses the Transformer to capture global features, and constructs a minimization objective function to achieve precise global inference. During tracking, ToMP uses the first annotated frame and previously tracked frames as the training set (Strain). Interference from objects with similar semantics can easily contaminate the tracking model.

[0057] In ToMP model updates, if the threshold is above a certain value, the model is updated. When the target is occluded, continuing to update the template will pollute the tracking model. When there are areas with similar semantic objects around the target, the highest peak may be an interference peak (similar semantic objects). An adaptive template update method is designed, combining the APCE proposed in the LMCF algorithm and the ASP proposed in this paper as the basis for judging model updates.

[0058] The expression of APCE is as follows:

[0059]

[0060] In the formula , and are the highest response value of the i-th frame, the lowest response value of the i-th frame, and the Response value at position;

[0061] The basic method for the tracker to locate the target is to locate the position of the maximum peak in the response graph. If the update is simply determined based on the response value of the current frame information, once the tracking model is contaminated, the true target will be completely lost. Therefore, combining the peak value of the historical frame to evaluate the tracking quality of the current frame is the only way to achieve a balanced result.

[0062] The ASP is combined with historical tracking information to evaluate the tracking quality. The expression of ASP is as follows:

[0063]

[0064] In the formula To avoid small constants with denominators equal to 0, and They represent the maximum response value of the i-th frame and the average maximum response value before the i-th frame respectively;

[0065] Whether the model is updated is determined jointly based on the values ​​of ASP and APCE. The confidence level of long-term tracking stability is as follows:

[0066]

[0067]

[0068] In the formula is the long-term tracking stability confidence, is the scale parameter, because and The values ​​differ greatly and need to be controlled at the same level. is the control parameter, To update the judgment parameters, is the update threshold;

[0069] The model update expression is as follows:

[0070]

[0071] In the formula and Respectively represent the last updated tracking model and the current updated tracking model, is the learning rate, set to 0.01;

[0072] ASP and APCE are combined into a confidence joint judgment formula to obtain the long-term tracking stability confidence , Multiply by the ratio When it is greater than the update threshold T, the model is updated. Add to the mean set;

[0073] General short-term trackers can only find targets within the search area. When the target disappears from the search area or moves at a faster speed, the target will be lost. To solve this problem, the present invention proposes a neighboring area search detection method. After the target is lost, a neighboring search area is constructed, and with the search area as the center, neighboring windows (of the same size) are adaptively added around the search area. Next, the size of the search area changes adaptively with the number of detection frames until the tracking target is detected. In order to avoid the search area being too large, introducing too many interferences, and preventing the search window from exceeding the image range, this paper adaptively adjusts the number of neighbors to control it within the image range. The designed neighboring area search detection method includes at least the following steps:

[0074]

[0075] Where: For the The number of frames adjacent to the window, is the search area for the i-th frame, To track images;

[0076] judge When the quantity By default, 12 adjacent windows are added to complete the judgment, that is, the search area of ​​12 adjacent windows is added;

[0077] If the target is not detected in the adjacent search area for a long time, the target may be moving behind an obstruction and appearing somewhere outside the search area.

[0078] During the re-detection process, the size of the search area determines the search range, so the size of the search area needs to be adaptively adjusted to achieve the relocation target;

[0079] make represents the search area expansion metric, is the current frame number, and its expression is as follows:

[0080] .

[0081] Most tracking algorithms do not dynamically judge the target's spatiotemporal information during the re-detection process, resulting in the detection of other similar objects or background interferences after the target is lost. When the confidence level is higher than the detection threshold, the object is relocated as the tracking target, causing tracking drift and contamination of the tracking model. The present invention proposes a dynamic spatiotemporal constraint method to penalize the disappearance of objects and their sudden appearance at a long distance. In terms of spatial constraints, when the distance between the new tracked object and the original tracked object is relatively large, the score will be penalized. Conversely, in terms of time constraints, as time goes by, if the new tracked object is not found, the penalty on distance will be compensated by the time penalty. Because the target may reappear at a farther distance as the time the tracked object is blocked increases.

[0082] The confidence expression combined with dynamic spatiotemporal constraints is as follows:

[0083]

[0084] Where, and Represent the position vectors of the new and old positions respectively, represents the diagonal length of the image, , Respectively represent the current frame number and the number of lost frames;

[0085] Depend on get;

[0086] After obtaining the confidence score of the dynamic spatiotemporal constraint, combined with Judge, finally get the re-detection target and complete the re-detection.

[0087] To validate the re-detection and tracking performance of this algorithm, five representative video sequences (group2, car2_s, person10, uav6, and parachute) were selected from the dataset. These sequences contain various complex variations, including partial occlusion, full occlusion, out-of-view, and similar targets. This was used to qualitatively verify the tracking performance of the improved algorithm. First, the proposed method performed a visual qualitative analysis of the Intersection over Union (IoU) values ​​and target re-detection, selecting the sequences group2 and parachute. Because targets in these two sequences often disappear from view for extended periods, robust tracking requires a tracking algorithm with strong re-detection capabilities. Therefore, these two sequences effectively validate the algorithm's re-detection capabilities.

[0088] like Figure 2 As shown in (a), in the 97th frame of the group2 sequence, the target is completely blocked and disappears from the field of view, which lasts until the 142th frame. When the target appears and can be detected again, the proposed algorithm can re-detect the target in a short time, while the ToMP algorithm does not re-detect the tracking target until the 1243th frame. This shows that the proposed algorithm has strong robustness. Figure 2 In the parachute sequence (b), the target disappears from view for a long time during the tracking process, and this is constantly accompanied by camera changes, posing a significant challenge to the algorithm's re-detection capabilities. After 1285 frames, the ToMP algorithm fails to detect the tracked target. However, the proposed algorithm uses a confidence metric to determine the model update mechanism, avoiding erroneous model updates under occlusion. The model is updated only when the target re-enters the field of view.

[0089] The proposed algorithm (LTMP) is compared with the five best tracking algorithms on the data set. The tracking results are as follows: Figure 3 shown.

[0090] Figure 3 (a) In sequence person10, the tracking target is a walking pedestrian. In frame 269, after the target disappears from view and reappears, only our algorithm is able to quickly lock onto the target within a few frames, and accurately control its scale. At frame 295, the RBO and TransT tracking algorithms completely lose the target. In frame 691, when two semantically similar objects approach, most tracking algorithms misjudge the target. Only our algorithm and TransT are able to track the target, and our algorithm also has a better judgment of the target's scale.

[0091] Figure 3 (b) Sequence car2_s tracks a simulated car. The target undergoes a significant scale change and briefly disappears from view. The RBO algorithm's scale adaptability is clearly insufficient, and it fails to match the target's scale during tracking. At frame 181, only our proposed algorithm accurately tracks the target. Compared to other tracking algorithms, our proposed method of temporally constrained re-detection limits distortion in the target's temporal and spatial information.

[0092] Figure 3 (c) Sequence uav6: The target is a small, fast-flying drone. The target is small, has few features, and moves rapidly. Furthermore, the background contains numerous interfering targets. Most of the compared algorithms failed to redetect the target after losing it. At frame 103, most algorithms failed to keep up with the target's rapid motion. Only our proposed algorithm was able to accurately and continuously track the target. All other algorithms failed to track the target, mistaking the interfering target for the target.

[0093] The proposed algorithm, LTMP, compares mAo and mSR with the currently advanced tracking algorithms SiamFC, STARK, Ocean, DiMP, TransT, SAOT, TrDimp, and ToMP on the GOT-10K dataset. Table 2 shows the comparison results of mAo, mSR50, and mSR75 with these tracking algorithms. As shown in the table, after improving the ToMP algorithm, the proposed algorithm, LTMP, achieves the best results, with mAo, mSR50, and mSR75 reaching 72.9%, 84.9%, and 66.2%, respectively. Furthermore, mAo, mSR50, and mSR75 surpass the baseline-TOMP tracking algorithm by 5.1%, 3.9%, and 3.7%, respectively.

[0094] surface Comparison of mAO (%) and mSR (%) on the GOT-10k test set

[0095]

[0096] Based on the ToMP tracker, a long-term tracking algorithm that incorporates spatiotemporal constraints and neighboring area re-detection is proposed. Long-term tracking needs to focus on the robustness of the tracker's re-detection and the robustness to visual deformation. First, the confidence evaluation function proposed in this invention is used to accurately evaluate the tracking quality of the tracker. According to the confidence, the target state and whether re-detection is required are judged. If the target is lost, a neighboring area search is performed on the current frame. Secondly, a dynamic spatiotemporal constraint is proposed to offset the distortion of the re-detected target in spatiotemporal information, and adaptively penalize re-detected targets with a longer distance. At the same time, an adaptive update strategy is used to update the tracking model to ensure efficiency while avoiding the introduction of negative samples, i.e., similar semantic objects.

[0097] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A long-term tracking method that incorporates spatiotemporal constraints and adjacent area re-detection, characterized by: At least the following steps are included: Taking the ToMP tracker as the base tracker, an adaptive model update method is designed. Whether the model is updated is jointly determined based on the ASP and APCE values, thereby achieving long-term tracking stability confidence. Design a nearby area search detection method. After the target is lost, a nearby search area is constructed. With the search area as the center, nearby windows are adaptively added around the search area. The size of the search area changes adaptively with the number of detection frames until the tracking target is detected. By adaptively adjusting the number of nearby windows, it is controlled within the image range. A dynamic spatiotemporal constraint method is designed to penalize situations where objects disappear and suddenly reappear at a long distance. In terms of spatial constraints, when the distance between the new tracked object and the original tracked object is large, the score will be penalized. In terms of temporal constraints, as time goes by and the new tracked object is not found, the distance penalty will be compensated by the time penalty.

2. The long-term tracking method incorporating spatiotemporal constraints and adjacent area re-detection according to claim 1, characterized in that: The design adaptive model updating method comprises at least the following steps: The APCE and ASP proposed in the LMCF algorithm are used as the basis for model update. The expression of APCE is as follows: ； In the formula , and are the highest response value of the i-th frame, the lowest response value of the i-th frame, and the Response value at position; The ASP is combined with historical tracking information to evaluate the tracking quality. The expression of ASP is as follows: ； In the formula To avoid small constants with denominators equal to 0, and They represent the maximum response value of the i-th frame and the average maximum response value before the i-th frame respectively; Whether the model is updated is determined jointly based on the values ​​of ASP and APCE. The confidence level of long-term tracking stability is as follows: ； ； In the formula is the long-term tracking stability confidence, is the scale parameter, because and The values ​​differ greatly and need to be controlled at the same level. is the control parameter, To update the judgment parameters, is the update threshold; The model update expression is as follows: ； In the formula and Respectively represent the last updated tracking model and the current updated tracking model, is the learning rate, set to 0.01; ASP and APCE are combined into a confidence joint judgment formula to obtain the long-term tracking stability confidence , Multiply by the ratio When it is greater than the update threshold T, the model is updated. Add to the mean set.

3. The long-term tracking method incorporating spatiotemporal constraints and adjacent area re-detection according to claim 2, characterized in that: The design of the adjacent area search detection method comprises at least the following steps: ； Where: For the The number of frames adjacent to the window, is the search area for the i-th frame, To track images; Set the size of the adjacent window to 0.5 times the search area, and adjust the position of the adjacent window according to the area of ​​the search area in the image; judge When the quantity By default, 12 adjacent windows are added to complete the judgment, that is, the search area of ​​12 adjacent windows is added; If the target is not detected in the adjacent search area for a long time, the target may be moving behind an obstruction and appearing somewhere outside the search area. During the re-detection process, the size of the search area determines the search range, so the size of the search area needs to be adaptively adjusted to achieve the relocation target; make represents the search area expansion metric, is the current frame number, and its expression is as follows: ； The method for designing dynamic spatiotemporal constraints comprises at least the following steps: The confidence expression combined with dynamic spatiotemporal constraints is as follows: ； Where, and Represent the position vectors of the new and old positions respectively, represents the diagonal length of the image, , Respectively represent the current frame number and the number of lost frames; Depend on get; After obtaining the confidence score of the dynamic spatiotemporal constraint, combined with Judge, finally get the re-detection target and complete the re-detection.

Images