Long-term single target tracking method, system and device based on sequence modeling reinforcement learning

Abstract

The long-term single target tracking method, system and device based on sequence modeling reinforcement learning belong to the field of target tracking technology and solve the problem of low performance of trackers in long-term tracking environments. The method of the present invention includes: constructing a long-term tracker based on sequence modeling reinforcement learning, including a perception layer and a decision layer, both of which adopt a Transformer-based structure; the visual Transformer encoding information of the perception layer is used as the input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer; the long-term tracker is improved in combination with the reinforcement learning based on sequence modeling, and the baseline short-term tracker is adaptively selected. Decisions are made by analyzing the content in the memory sequence; a single short-term tracker affects the overall tracking result, and the overall tracking result is jointly determined by the visual encoder and the tracking method; the decision layer dynamically optimizes the search area position to achieve target tracking. The present invention is suitable for target tracking in long-term single target tracking scenarios.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to long-term single target tracking. Background Art

[0002] Long-term single object tracking aims to achieve robust tracking over extended video sequences, even when the object disappears from view, is occluded for long periods of time, or undergoes abrupt changes. Currently, the dominant paradigm for long-term tracking is to ensemble multiple baseline short-term trackers to accommodate a variety of scenarios. This paradigm separates the long-term tracking problem into two key dimensions. First, a short-term tracker predicts the location of the object, which is then verified to generate a confidence score. Second, the best short-term tracker is selected from multiple models based on a logical criterion, and its prediction is applied accordingly. These methods are simple to implement because they decompose long video sequences into manageable segments for model optimization and do not require the design of an additional object tracking model.

[0003] However, baseline short-term trackers are not specifically designed for challenging scenarios and show performance degradation in long-term tracking environments. In particular, appearance-based short-term trackers tend to fail when the target is occluded for a long time (e.g. Figure 1 (as shown in Figure 2). In this case, a steering motion model is relied upon; however, its predictions are based on the assumption of uniform motion and often fail to capture transient target appearances. Furthermore, long-term trackers lacking re-detection capabilities struggle to reacquire the target after sudden position changes, contradicting their primary design intent.

[0004] The aforementioned paradigm guides the selection of short-term trackers in long sequences, relying on logical rule-based judgments. Recent advances have leveraged Transformer-based visual encoders for object tracking. Transformers are favored due to their ability to effectively model a wide range of sequences and diverse data distributions. The challenges of modeling long sequences extend beyond object tracking and encompass a wide range of sequential decision-making problems in fields such as reinforcement learning. Therefore, Transformer-based methods for trajectory modeling are urgently needed to replace traditional reinforcement learning methods. Summary of the Invention

[0005] The purpose of the present invention is to solve the technical problem mentioned above, the problem that the existing tracker has low performance in a long-term tracking environment, and to provide a long-term single-target tracking method, system and device based on sequence modeling reinforcement learning.

[0006] The present invention is achieved through the following technical solutions. In one aspect, the present invention provides a long-term single target tracking method based on sequence modeling reinforcement learning, the method comprising:

[0007] Step 1: Get the original image of the current video frame and the position coordinates of the target of interest in the initial frame;

[0008] Step 2: Build a long-term tracker based on sequence modeling reinforcement learning. The long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer. Both the perception layer and the decision layer adopt a Transformer-based structure. The information encoded by the visual Transformer of the perception layer serves as the observation input of the decision Transformer. The action sequence generated by the decision layer is fed back to the perception layer.

[0009] The perception layer includes: building a tracking pool, which includes several trackers; setting a search area and a target template; building a state space;

[0010] The decision layer includes: constructing action space and expected rewards;

[0011] Step 3: Training the long-term tracker based on sequence modeling reinforcement learning, specifically including:

[0012] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after visual Transformer processing as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame;

[0013] Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length;

[0014] Divide the trajectory τ into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one;

[0015] Calculate the cumulative return at each time step

[0016] The modified trajectory Input to the decision Transformer decoder in the decision layer, the output of the causal Transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result.

[0017] Step 4: Track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

[0018] Furthermore, the plurality of trackers include a short-term tracking method ViTTrack, a short-term tracking method KeepTrack and a motion module.

[0019] Furthermore, the setting of the search area and the target template specifically includes:

[0020] For a given target bounding box (x, y, w, h) in the initial frame, the slice size is determined according to the search area coefficient sf crop :

[0021]

[0022] Determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the padding part, where λ represents the scaling factor;

[0023] Forming a filled slice area

[0024] x1=[x+λw-λsize crop ]

[0025] x2=x1+size crop

[0026] y1=[y+λh-λsize crop ]

[0027] y2=y1+size crop

[0028]

[0029] Where W and H are the width and height of the image.

[0030] Furthermore, the constructing of the state space specifically includes:

[0031] Separate the target template from the search area in the original video frame;

[0032] After the raw video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the visual transformer.

[0033] S=(F search ,F template ,F attn )

[0034] Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector, and S represents the state space sampling

[0035] The visual Transformer architecture serves as the backbone network, extracting features from the embedded target template and search region, generating a feature tensor that encapsulates information related to the target template and search region, and reflects the relationship between the target frames.

[0036] Furthermore, the constructing of the action space specifically includes:

[0037] Optimize the search area using nine dimensions:

[0038] A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t)

[0039] Among them, (l, r, u, d) are used for horizontal and vertical translation (left, right, up, down), (l free ,r free ,u free ,d free ) four for free translation in the same direction, sf is used to adjust the scale factor of the search area, t is used to indicate whether the target exists, and A represents the action space sampling;

[0040] When a translation action is initiated, the search area will be translated by αsize crop , where α represents the translation scale factor; after activating the free translation action, the free value will be adjusted in the corresponding direction;

[0041] When the decision layer judges that the deviation of the search area is large, free translation will be activated; conversely, when the deviation is small, only quantitative translation will be used;

[0042] The scale factor sf of the search area is adjusted by multiplying the continuous action value output by the decision Transformer by a fixed coefficient.

[0043] Furthermore, the expected reward is constructed, specifically including: target score, intersection over union (IoU) ratio compared with the ground truth IoU , characteristic distance ap disand local outlier factor lof dis ;

[0044]

[0045] Among them, R is the reward record, represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

[0046] Furthermore, the loss function of the long-term tracker based on sequence modeling reinforcement learning is specifically:

[0047]

[0048] in, To predict actions, For prediction rewards.

[0049] In a second aspect, the present invention provides a long-term single target tracking system based on sequence modeling reinforcement learning, the system comprising:

[0050] The image and position acquisition module is used to obtain the original image of the current video frame and the position coordinates of the target of interest in the initial frame;

[0051] A tracker building module for building a long-term tracker based on sequence modeling reinforcement learning. The long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer. Both the perception layer and the decision layer adopt a Transformer-based architecture. The information encoded by the visual Transformer of the perception layer serves as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer.

[0052] The perception layer includes: building a tracking pool, which includes several trackers; setting a search area and a target template; building a state space;

[0053] The decision layer includes: constructing action space and expected rewards;

[0054] A tracker training module is used to train the long-term tracker based on sequence modeling reinforcement learning, specifically including:

[0055] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after visual Transformer processing as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame;

[0056] Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length;

[0057] Divide the trajectory τ into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one;

[0058] Calculate the cumulative return at each time step

[0059] The modified trajectory Input to the decision Transformer decoder in the decision layer, the output of the causal Transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result.

[0060] The target tracking module is used to track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

[0061] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the steps of a long-term single target tracking method based on sequence modeling reinforcement learning as described above are executed.

[0062] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a plurality of computer instructions are stored in the computer-readable storage medium, and the plurality of computer instructions are used to enable a computer to execute a long-term single-target tracking method based on sequence modeling reinforcement learning as described above.

[0063] Beneficial effects of the present invention:

[0064] The present invention is based on the currently common long-term tracker paradigm, combined with a reinforcement learning method based on sequence modeling for improvement, adaptively selecting a baseline short-term tracker and improving its performance. The rule-based scoring judgment solidifies the pattern of the long-term tracking model. The intelligent agent decision layer based on sequence modeling proposed in the present invention can make more intelligent decisions by analyzing the content in the memory sequence, thereby improving the versatility of the long-term tracking model in various scenarios without relying on thresholds. The performance of a single short-term tracker will significantly affect the overall tracking results, which are jointly determined by the visual encoder and the tracking method. Most existing methods use a fixed ratio to define the search area, requiring the tracker to pay attention to the foreground and background within the area. However, a well-trained intelligent agent decision layer can dynamically adjust the ratio and optimize the search area position, because using an inefficient search area is also inefficient when the tracker prediction is inaccurate.

[0065] Within the popular long-term tracker paradigm, this paper primarily utilizes a short-term tracking model based on the Vision Transformer, supplemented by a KeepTrack model and a motion model, to construct a comprehensive long-term tracking model. Furthermore, the decision-making Transformer serves as a robust baseline for the present invention's sequential decision model. The agent's decision layer utilizes the output of the Vision Transformer encoder as a state space containing combined features of target samples and search regions to generate model selection results, optimize search region variables, and determine target presence.

[0066] In view of the common requirement for sequential modeling between long-term target tracking and sequential decision-making, this paper proposes an application of a reinforcement learning method based on sequence modeling in long-term target tracking scenarios, which will serve as a new paradigm to solve challenging problems in long-term target tracking scenarios.

[0067] The present invention is based on the cross-application of the fields of computer vision single target tracking and reinforcement learning, and is suitable for target tracking in long-term single target tracking scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 Schematic diagram of the main challenges and differences of the present invention;

[0070] Figure 2 A network block diagram of the method of the present invention;

[0071] Figure 3Schematic diagram of the specific optimization operations fed back to the tracking model. DETAILED DESCRIPTION

[0072] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but are not to be construed as limiting the present invention.

[0073] Embodiment 1: A long-term single target tracking method based on sequence modeling reinforcement learning, the method comprising:

[0074] Step 1: Get the original image of the current video frame and the position coordinates of the target of interest in the initial frame;

[0075] Step 2: Build a long-term tracker based on sequence modeling reinforcement learning. The long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer. Both the perception layer and the decision layer adopt a Transformer-based structure. The information encoded by the visual Transformer of the perception layer serves as the observation input of the decision Transformer. The action sequence generated by the decision layer is fed back to the perception layer.

[0076] The perception layer includes: building a tracking pool, which includes several trackers; setting a search area and a target template; building a state space;

[0077] The decision layer includes: constructing action space and expected rewards;

[0078] Step 3: Training the long-term tracker based on sequence modeling reinforcement learning, specifically including:

[0079] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after visual Transformer processing as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame;

[0080] Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i∈R, T is the sequence length; each trajectory is generated according to the high-performance long-term tracker in the tracking pool, and the action record is obtained by comparing it with the true value, so the offline data constructed in this embodiment is closer to the expert data.

[0081] Divide the trajectory τ into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one;

[0082] Calculate the cumulative return at each time step

[0083] The modified trajectory Input to the decision Transformer decoder in the decision layer, the output of the causal Transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result.

[0084] Step 4: Track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

[0085] This embodiment uses a Transformer-based method for trajectory modeling to replace the traditional reinforcement learning method. In view of the common requirement for sequential modeling between long-term target tracking and sequential decision-making, a reinforcement learning method based on sequence modeling is proposed for application in long-term target tracking scenarios. An integrated single target tracking model is adopted, which integrates the visual Transformer (ViT) of the perception layer and the decision Transformer (DT) of the decision layer. The information encoded by the ViT of the perception layer serves as the observation input of the decision layer DT. In addition, the action sequence generated by the decision layer is fed back to the perception layer to promote tracker selection and improve tracking performance.

[0086] Implementation 2: This implementation further defines the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation, the multiple trackers are further defined, specifically including:

[0087] The plurality of trackers include a short-term tracking method ViTTrack, a short-term tracking method KeepTrack and a motion module.

[0088] The goal of this embodiment is to unify the network architecture of the visual perception layer and the decision layer by adopting a Transformer-based structure for these two layers. This approach allows direct interaction between Transformers performing different functions without the need for additional mapping layers. For this reason, ViTTrack is used to build a ViT-based tracker. ViTTrack uses ViT as its backbone network and implements a corner prediction head to determine the location of the target. It connects the template and search area into the spatial channel after the original embedding block and then uses ViT to extract features. At the same time, this embodiment also combines other single-target trackers to enhance the tracking pool. KeepTrack adopts an alternative strategy to track distractors, in stark contrast to most methods that aim to suppress distractors through more powerful appearance models. In addition, the tracking pool includes a simple but effective motion module that predicts the target state based on previous temporal trajectory information and is activated when the target is considered to be moving abnormally.

[0089] Implementation method 3: This implementation method further limits the long-term single target tracking method based on sequence modeling reinforcement learning described above. In this implementation method, the setting of the search area and the target template is further limited, specifically including:

[0090] The setting of the search area and the target template specifically includes:

[0091] For a given target bounding box (x, y, w, h) in the initial frame, the slice size is determined according to the search area coefficient sf crop :

[0092]

[0093] Determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the padding part, where λ represents the scaling factor;

[0094] Forming a filled slice area

[0095] x1=[x+λw-λsize crop ]

[0096] x2=x1+size crop

[0097] y1=[y+λh-λsize crop ]

[0098] y2=y1+size crop

[0099]

[0100] Where W and H are the width and height of the image.

[0101] In this embodiment, since the decision-making layer in this application mainly achieves effective tracking and re-detection by optimizing the search area, the technology of how to obtain the search area and template will be provided to optimize the position and size of the overall search area to cope with the challenges of target changes that occur during long-term tracking.

[0102] Implementation 4: This implementation further limits the long-term single target tracking method based on sequence modeling reinforcement learning described above. In this implementation, the construction of the state space is further limited, specifically including:

[0103] The constructing state space specifically includes:

[0104] Separate the target template from the search area in the original video frame;

[0105] After the raw video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the visual transformer.

[0106] S=(F search ,F template ,F attn )

[0107] Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector, and S represents the state space sampling

[0108] The visual Transformer architecture serves as the backbone network, extracting features from the embedded target template and search region, generating a feature tensor that encapsulates information related to the target template and search region, and reflects the relationship between the target frames.

[0109] This implementation effectively preserves scene understanding relevant to perceptual intelligence tasks, and due to the inherent feature richness in perceptual intelligence algorithms, it can enhance the decision-making process of reinforcement learning agents.

[0110] Implementation 5: This implementation further limits the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation, the construction of the action space is further limited, specifically including:

[0111] The constructing of the action space specifically includes:

[0112] Optimize the search area using nine dimensions:

[0113] A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t)

[0114] Among them, (l, r, u, d) are used for horizontal and vertical translation (left, right, up, down), (l free ,r free ,u free ,d free ) four for free translation in the same direction, sf is used to adjust the scale factor of the search area, t is used to indicate whether the target exists, and A represents the action space sampling;

[0115] When a translation action is initiated, the search area will be translated by αsize crop , where α represents the translation scale factor; after activating the free translation action, the free value will be adjusted in the corresponding direction;

[0116] When the decision layer judges that the deviation of the search area is large, free translation will be activated; conversely, when the deviation is small, only quantitative translation will be used;

[0117] The scale factor sf of the search area is adjusted by multiplying the continuous action value output by the decision Transformer by a fixed coefficient.

[0118] In this implementation, the search area is divided based on the extension of the target's position. As the target's position changes, the search area adjusts accordingly. However, if the target becomes blurred, obscured, or the intelligent perception model fails, the target's position may become unclear or nonexistent. This can cause the intelligent perception model to misunderstand the current frame, resulting in a decrease in focus on the target.

[0119] While defining the action space by optimizing the search region is logically sound, in practical design, specific parameters of the search region cannot be used as a subspace of the action space. This limitation arises because such a definition would destroy the decision direction within the action space, potentially hindering the convergence of offline reinforcement learning agents. Therefore, this implementation uses nine dimensions to represent the optimization of the search region.

[0120] Implementation 6: This implementation further limits the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation, the construction of the expected reward is further limited, specifically including:

[0121] The expected reward is constructed, specifically including: target score, intersection over union (IoU) ratio compared to the ground truth IoU, characteristic distance ap dis and local outlier factor lof dis ;

[0122]

[0123] Among them, R is the reward record, represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

[0124] In this embodiment, since the numerical spaces of the characteristic distance and the local outlier factor are not uniform, in order to balance the contributions of the two when recording the reward, the two are normalized.

[0125] In this application's sequential modeling-based reinforcement learning model, the interpretation of the reward function differs from traditional reinforcement learning methods. Rather than maximizing the reward function, this approach strategically plans the endpoint of the task by establishing a target reward (default value 1.0). This embodiment proposes a reward mechanism derived from a tracking model.

[0126] Implementation method seven: This implementation method further limits the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation method, the loss function of the long-term tracker based on sequence modeling reinforcement learning is further limited, specifically including:

[0127] The loss function of the long-term tracker based on sequence modeling reinforcement learning is specifically:

[0128]

[0129] in, To predict actions, For prediction rewards.

[0130] The loss function of this implementation adopts two minimization supervision losses: predicting action and prediction rewards This implementation is based on the following two aspects: First, the predicted action is an essential attribute of the decision-making Transformer and serves as the main guide for optimizing the parameters of the decision layer. In addition, the source of the predicted action is the difference between the long-term tracker's trajectory prediction results and the true value, which is consistent with the task requirements of the long-term tracking model. Second, the predicted reward acts as a regularization term to regulate overfitting of the decision layer. Relying solely on predicted actions will produce an effect closer to imitation learning, which is detrimental to the versatility of the decision layer. The predicted reward term can effectively circumvent this problem.

[0131] Embodiment 8: This embodiment is an example of a long-term single target tracking method based on sequence modeling reinforcement learning as described above, specifically including:

[0132] The overall network structure of the long-term tracker based on sequence modeling reinforcement learning is as follows Figure 2 As shown in Figure 1, it can be mainly divided into two layers: the perception layer and the decision layer. An integrated single-target tracking model is adopted, which integrates the visual transformer (ViT) in the perception layer and the decision transformer (DT) sequence modeling in the decision layer. The information encoded by the ViT in the perception layer serves as the observation input of the DT in the decision layer. In addition, the action sequence generated by the decision layer is fed back to the perception layer to facilitate tracker selection and improve tracking performance.

[0133] Step 1: Build a tracking pool, which includes several trackers:

[0134] By adopting a Transformer-based structure for the visual perception layer and the decision layer to unify the network architecture of the two layers, this approach allows direct interaction between Transformers performing different functions without the need for additional mapping layers. Based on this insight, a ViT-based tracker (ViTTrack) is constructed. ViTTrack uses ViT as its backbone network and implements a corner prediction head to determine the location of the target. It links the template and search area into the spatial channel after the original embedding block and then uses ViT to extract features. In addition, this embodiment also combines other single-target trackers to enhance the tracking pool. KeepTrack adopts an alternative strategy to track distractors, in stark contrast to most methods that aim to suppress distractors through more powerful appearance models. In addition, the tracking pool includes a simple but effective motion module that predicts the target state based on previous temporal trajectory information and is activated when the target is considered to be moving abnormally.

[0135] Step 2: Get the search area and target template

[0136] Since the decision layer in this invention mainly achieves effective tracking and re-detection by optimizing the search area, the technology of how to obtain the search area and template will be provided. For the target bounding box (x, y, w, h) given in the initial frame, the slice size size is first determined according to the search area coefficient sf crop :

[0137]

[0138] Then determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the filling part, where λ represents the scaling factor, which is set to 0.5 by default. Finally, the filled slice area is formed by the copyMakeBorder function of OpenCV

[0139] x1=[x+λw-λsize crop ]

[0140] x2=x1+size crop

[0141] y1=[y+λh-λsize crop ]

[0142] y2=y1+size crop (2-2)

[0143]

[0144] These fill parts work with the copyMakeBorder function to divide the search area and the template, such as Figure 3 As shown in Figure 2. Where W and H are the width and height of the image. One of the decision-making parts of this work is to optimize the location and size of the overall search area to cope with the challenge of target changes that occur during long-term tracking.

[0145] Step 3: State Space Construction

[0146] This step aims to minimize the loss of valuable information within the video frame while effectively embedding the implicit correlation between consecutive frames. Figure 2 As shown in Figure 2, the process starts with the target sampling module, which separates the target template from the search region within the original video frame. Next, after the original video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the Vision Transformer.

[0147] S=(F search ,F template ,F attn ) (3-1)

[0148] Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector.

[0149] Subsequently, the Vision Transformer architecture serves as the backbone network to further extract features from the embedded target template and search region. Ultimately, a feature tensor of size (1,721,768) is generated, encapsulating all information related to the target template and search region and reflecting the relationship between the target frames. This approach effectively preserves scene understanding related to perceptual intelligence tasks and, due to the inherent feature richness of perceptual intelligence methods, can enhance the decision-making process of reinforcement learning agents.

[0150] Step 4: Action Space Construction

[0151] The optimization of the search area is represented by nine dimensions: (l, r, u, d) four for horizontal and vertical translation (left, right, up, down), (l free ,r free ,u free ,d free ) four free translations in the same direction, sf is used to adjust the scale factor of the search area, and t is used to indicate whether the target is present. These parameters are encoded using a one-hot representation, where left-right and up-down movements are mutually exclusive, ensuring that opposite directions are not activated simultaneously.

[0152] A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t) (4-1)

[0153] When a translation action is initiated, the search area will be translated by αsize crop , where α represents the translation scale factor, and the default value is 0.05. After the free translation action is activated, the free value will be adjusted in the corresponding direction. When the agent decision layer determines that the deviation of the search area is large, the free translation will be activated; conversely, when the deviation is small, only quantitative translation is used. In order to adjust the scale coefficient sf of the search area, the present invention multiplies the continuous action value output by the decision converter by a fixed coefficient for transformation. The size of the constant value indicates the degree of change of the scale coefficient, while ensuring that the minimum value of the scale coefficient is maintained at 2. In addition, the agent decision layer can also output whether the target exists in the search area, which helps to realize the re-detection long-term tracker.

[0154] Step 5: Expected Reward Construction

[0155] In sequence-based reinforcement learning models, the interpretation of the reward function is different from that of traditional reinforcement learning methods. Instead of maximizing the reward function, this method strategically plans the end point of the task by establishing a target reward (default value 1.0). A reward mechanism derived from the tracking model is proposed, which consists of four main components: the target score, the intersection-over-union (IoU) ratio compared to the ground truth, and the target score. IoU ), feature distance (ap dis ) and local outlier factor (lof dis ).

[0156]

[0157] Among them, R is the reward record. Since the numerical space of feature distance and local outlier factor is not uniform, in order to balance the contribution of the two when making reward records, the two are normalized. represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

[0158] The target score reflects the tracker's assessment of the presence of the target in the current frame after adjusting the search area, and is used as a reward component together with the IoU. The feature distance quantifies the difference between the slice features of the target in the current frame and the slice features of the standard template, and serves as a penalty component in the reward function. The outlier factor and the feature distance are comparable indicators for evaluating the difference between the target in the current frame and the standard template. The difference is calculated using the outlier operator and serves as a penalty component in the reward function. Given that the value ranges of these four components are different, this embodiment uses normalization technology to adjust them according to their respective value ranges to ensure that all components are within the interval [0,1]. The normalized reward function is expressed as formula (5-1).

[0159] Step 6: Perform model training

[0160] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after Vision Transformer processing (as state S), the search area adjustment method and target existence judgment (as action A), and the reward record R obtained by the reward function for each frame. Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i∈R, T is the sequence length. Existing long-term trackers generate each trajectory and obtain action records by comparing it with the true value, so the offline data constructed in this embodiment is closer to expert data.

[0161]

[0162] First, the trajectory τ is divided into several segments, each of which is of length K. It is represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K ), but this example modifies these trajectories rather than inputting them directly, specifically, computing the cumulative reward for each time step It encourages the converter module to generate operations that can reduce this value as close to 1.0 as possible. Then the modified trajectory Input to the causal Transformer decoder in the decision layer. The output of the decision Transformer is a sequence vector Where d is the dimension of the embedding space, and the vector is consistent with the action space constructed in step 4. This sequence vector feeds the output search area optimization action back to the tracking model, thereby optimizing the tracking result. In terms of loss function, two minimization supervision losses are used to predict the action a t , predicted reward r t , the loss function is shown in Formula (6-1). The complete training process is summarized in the following Algorithm 1.

[0163]

[0164] The proposed method can be tested on standard single-object tracking datasets and can be plug-and-play optimized on some existing methods. To ensure a fair comparison and demonstrate the superiority of the proposed method, the proposed method can leverage existing publicly available short-term trackers and motion models. Specifically, the proposed method uses the same tracker selections as the Multi-Model Hybrid Long-Term Tracker (VITKT_M): ViTTrack, KeepTrack, and motion model. The state space size corresponds to the output of the combined features generated by the Vision Transformer. Furthermore, the proposed method separately passes information from the attention layer and integrates it with the combined features, resulting in a total state space dimension of 1,107,456, calculated as (2×721×768). Since the original decision Transformer can only produce continuous values ​​as action decisions, the proposed method estimates the mapping for the portion that must produce discrete values, consistent with the action space defined in this invention. The mapping layer and the GPT model are trained simultaneously. The proposed method sets K = 20 as the length of each trajectory segment. The GPT model uses a five-layer Transformer architecture with an embedding dimension of 128. The dropout rate of the MLP layer is 0.1. Training was performed 1,000 times in a warmup phase, followed by 10,000 iterations per epoch, with 10 epochs for each scene (representing a complete sequence). The hardware consisted of a single RTX4090 GPU and a 12th Generation Intel(R) Core(TM) i7-12700K CPU running at 3.60GHz.

[0165] The sequence modeling agents of the present invention can effectively address the long-standing challenges of long-term tracking, especially disappearing targets and position mutations. These agents are adaptively adjusted according to the current target position and region of interest. The method proposed in this invention significantly improves the tracking recall rate by automatically selecting a suitable tracking model after training, adjusting the position and size of the region of interest, and evaluating whether the target exists in the search area. In addition, the method is very effective in alleviating re-detection failures caused by sudden changes in the target position. Extensive experiments have demonstrated the strong generalization ability of this method across various trackers and scenarios. Using widely used short-term tracking models, the method achieves performance that surpasses the state-of-the-art methods. The present invention hopes that the characteristics of the proposed method will help to re-examine the long-term tracking challenges.

Claims

1. A long-term single target tracking method based on sequence modeling reinforcement learning, characterized in that: The method comprises: Step 1: Get the original image of the current video frame and the position coordinates of the target of interest in the initial frame; Step 2: Build a long-term tracker based on sequence modeling reinforcement learning. The long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer. Both the perception layer and the decision layer adopt a Transformer-based structure. The information encoded by the visual Transformer of the perception layer serves as the observation input of the decision Transformer. The action sequence generated by the decision layer is fed back to the perception layer. The perception layer includes: building a tracking pool, which includes several trackers; setting a search area and a target template; building a state space; The decision layer includes: constructing action space and expected rewards; Step 3: Training the long-term tracker based on sequence modeling reinforcement learning, specifically including: Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after visual Transformer processing as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame; Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length; Divide the trajectory τ into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one; Calculate the cumulative return at each time step The modified trajectory Input to the decision Transformer decoder in the decision layer, the output of the causal Transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result. Step 4: Track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

2. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 1, characterized in that: The plurality of trackers include a short-term tracking method ViTTrack, a short-term tracking method KeepTrack and a motion module.

3. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 2, characterized in that: The setting of the search area and the target template specifically includes: For a given target bounding box (x, y, w, h) in the initial frame, the slice size is determined according to the search area coefficient sf Determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the padding part, where λ represents the scaling factor; Forming a filled slice area x1=[x+λw-λsize crop ] x2=x1+size crop <h2 style=";text-align:left;direction:ltr">y1=[y+λh-λsize<h2 style=";text-align:left;direction:ltr"> crop <h2 style=";text-align:left;direction:ltr"> ] y2=y1+size crop Where W and H are the width and height of the image.

4. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 3 is characterized in that: The constructing state space specifically includes: Separate the target template from the search area in the original video frame; After the raw video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the visual transformer. S=(F search ,F template ,F attn ) Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector, and S represents the state space sampling The visual Transformer architecture serves as the backbone network, extracting features from the embedded target template and search region, generating a feature tensor that encapsulates information related to the target template and search region, and reflects the relationship between the target frames.

5. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 4 is characterized in that: The constructing of the action space specifically includes: Optimize the search area using nine dimensions: A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t) Among them, (l, r, u, d) are the four horizontal and vertical translations, (l free ,r free ,u free ,d free ) are four free translations in the same direction, sf is used to adjust the scale factor of the search area, t is used to indicate whether the target exists, and A represents the action space sampling; When a translation action is initiated, the search area will be translated by αsize crop , where α represents the translation scale factor; after activating the free translation action, the free value will be adjusted in the corresponding direction; When the decision layer judges that the deviation of the search area is large, free translation will be activated; conversely, when the deviation is small, only quantitative translation will be used; The scale factor sf of the search area is adjusted by multiplying the continuous action value output by the decision Transformer by a fixed coefficient.

6. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 5, characterized in that: The expected reward is constructed, specifically including: target score, intersection over union (IoU) ratio compared to the ground truth IoU , characteristic distance ap dis and local outlier factor lof dis ; Among them, R is the reward record, represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

7. The long-term single target tracking method based on sequential modeling reinforcement learning according to any one of claims 1 to 6, characterized in that: The loss function of the long-term tracker based on sequence modeling reinforcement learning is specifically: in, To predict actions, For prediction rewards.

8. A long-term single target tracking system based on sequence modeling reinforcement learning, characterized in that: The system comprises: The image and position acquisition module is used to obtain the original image of the current video frame and the position coordinates of the target of interest in the initial frame; A tracker building module for building a long-term tracker based on sequence modeling reinforcement learning. The long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer. Both the perception layer and the decision layer adopt a Transformer-based architecture. The information encoded by the visual Transformer of the perception layer serves as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer. The perception layer includes: building a tracking pool, which includes several trackers; setting a search area and a target template; building a state space; The decision layer includes: constructing action space and expected rewards; A tracker training module is used to train the long-term tracker based on sequence modeling reinforcement learning, specifically including: Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after visual Transformer processing as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame; Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length; Divide the trajectory τ into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one; Calculate the cumulative return at each time step The modified trajectory Input to the decision Transformer decoder in the decision layer, the output of the causal Transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result. The target tracking module is used to track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor runs the computer program stored in the memory, the steps of the method according to any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of computer instructions, and the plurality of computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images