Long-term single target tracking method, system and equipment based on sequence modeling reinforcement learning

Abstract

The invention discloses a long-term single-target tracking method, system and equipment based on sequence modeling reinforcement learning, belongs to the technical field of target tracking, and solves the problem of low performance of a tracker in a long-term tracking environment. The method comprises the following steps: constructing a long-term tracker based on sequence modeling reinforcement learning, wherein the long-term tracker comprises a sensing layer and a decision-making layer, and the sensing layer and the decision-making layer adopt structures based on Transform; visual Transform coding information of the perception layer is used as input of a decision Transform, and an action sequence generated by the decision layer is fed back to the perception layer; a long-term tracker is combined with reinforcement learning improvement based on sequence modeling, and a baseline short-term tracker is adaptively selected. Making a decision by analyzing the content in the memory sequence; a single short-term tracker affects an overall tracking result, and the overall tracking result is jointly determined by a visual encoder and a tracking method; and the decision-making layer dynamically optimizes the position of the search region to realize target tracking. The method is suitable for target tracking in a long-term single-target tracking scene.

Description

Technical Field

[0001] The present application relates to the technical field of target tracking, and in particular to long-term single target tracking. Background Art

[0002] Long-term single object tracking aims to achieve robust tracking over extended video sequences, even when the object disappears from the field of view, is occluded for long periods of time, or changes abruptly. Currently, the dominant paradigm for long-term tracking is to ensemble multiple baseline short-term trackers to accommodate a variety of scenarios. This paradigm separates the long-term tracking problem into two key dimensions. First, a short-term tracker predicts the location of the object, which is then verified to generate a confidence score. Second, the best short-term tracker is selected from multiple models based on a logical criterion and its prediction is applied accordingly. These methods are simple to implement because they decompose long video sequences into manageable segments for model optimization and do not require additional design of object tracking models.

[0003] However, baseline short-term trackers are not specifically designed for challenging scenarios and show performance degradation in long-term tracking environments. In particular, appearance-based short-term trackers tend to fail when the target is occluded for a long time (e.g. Figure 1 In this case, a steering motion model is relied upon; however, its predictions are based on the assumption of uniform motion and usually fail to capture transient target appearances. Moreover, long-term trackers lacking re-detection capabilities have difficulty reacquiring targets after abrupt position changes, contradicting the main intention of their design.

[0004] The above paradigm guides the selection of short-term trackers in long sequences, relying on logical rule-based judgments. Recent advances have utilized Transformer-based visual encoders to implement object tracking methods. Transformers are favored because they can effectively model a wide range of sequences and diverse data distributions. The challenges of modeling long sequences are not limited to object tracking, but also cover various sequential decision-making problems in fields such as reinforcement learning. Therefore, Transformer-based methods are urgently needed for trajectory modeling to replace traditional reinforcement learning methods. Summary of the invention

[0005] The purpose of the present invention is to solve the technical problem mentioned above, the problem that the existing tracker has low performance in a long-term tracking environment, and to provide a long-term single-target tracking method, system and device based on sequence modeling reinforcement learning.

[0006] The present invention is implemented by the following technical solutions. On the one hand, the present invention provides a long-term single target tracking method based on sequence modeling reinforcement learning, and the method comprises:

[0007] Step 1: Get the original image of the current video frame and the position coordinates of the target of interest in the initial frame;

[0008] Step 2: construct a long-term tracker based on sequence modeling reinforcement learning, wherein the long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer; both the perception layer and the decision layer adopt a Transformer-based structure; the information encoded by the visual Transformer of the perception layer is used as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer;

[0009] The perception layer includes: constructing a tracking pool, the tracking pool includes a plurality of trackers; setting a search area and a target template; constructing a state space;

[0010] The decision layer includes: constructing action space and expected rewards;

[0011] Step 3: Training the long-term tracker based on sequence modeling reinforcement learning, specifically including:

[0012] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination processed by the visual Transformer as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame;

[0013] Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length;

[0014] The trajectory τ is divided into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one;

[0015] Calculate the cumulative return at each time step

[0016] The modified trajectory The decision transformer decoder is input to the decision layer, and the output of the causal transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result.

[0017] Step 4: Track the target using the trained sequence modeling reinforcement learning based long-term tracker.

[0018] Furthermore, the plurality of trackers include a short-term tracking method ViTTrack, a short-term tracking method KeepTrack and a motion module.

[0019] Furthermore, the setting of the search area and the target template specifically includes:

[0020] For a given target bounding box (x, y, w, h) in the initial frame, determine the slice size size according to the search area coefficient sf crop :

[0021]

[0022] Determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the padding part, where λ represents the scaling factor;

[0023] Forming a filled slice area

[0024] x1=[x+λw-λsize crop ]

[0025] x2=x1+size crop

[0026] y1=[y+λh-λsize crop ]

[0027] y2=y1+size crop

[0028]

[0029] Where W and H are the width and height of the image.

[0030] Furthermore, the constructing of the state space specifically includes:

[0031] Separating the target template from the search area within the original video frame;

[0032] After the raw video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the visual transformer.

[0033] S=(F search ,F template ,F attn )

[0034] Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector, and S represents the state space sampling

[0035] The visual Transformer architecture serves as the backbone network, extracting features from the embedded target template and search region, generating feature tensors that encapsulate information related to the target template and search region, and reflecting the relationship between target frames.

[0036] Furthermore, the constructing of the action space specifically includes:

[0037] Use nine dimensions to optimize the search area:

[0038] A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t)

[0039] Among them, (l, r, u, d) are used for horizontal and vertical translation (left, right, up, down), (l free ,r free ,u free ,d free ) four for free translation in the same direction, sf is used to adjust the scale factor of the search area, t is used to indicate whether the target exists, and A represents the action space sampling;

[0040] When a translation action is initiated, the search area is translated by αsize crop , where α represents the translation scale factor; after activating the free translation action, the free value will be adjusted in the corresponding direction;

[0041] When the decision layer judges that the deviation of the search area is large, free translation is activated; conversely, when the deviation is small, only quantitative translation is used;

[0042] The scale factor sf of the search area is adjusted by multiplying the continuous action value output by the decision Transformer by a fixed factor.

[0043] Furthermore, the expected reward is constructed, specifically including: target score score, intersection over union ratio (IoU) ratio compared with the ground truth IoU , characteristic distance ap disand local outlier factor lof dis ;

[0044]

[0045] Among them, R is the reward record, represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

[0046] Furthermore, the loss function of the long-term tracker based on sequence modeling reinforcement learning is specifically:

[0047]

[0048] in, To predict actions, Reward for prediction.

[0049] In a second aspect, the present invention provides a long-term single target tracking system based on sequence modeling reinforcement learning, the system comprising:

[0050] The image and position acquisition module is used to obtain the original image of the current video frame and the position coordinates of the target of interest in the initial frame;

[0051] A tracker building module, used to build a long-term tracker based on sequence modeling reinforcement learning, wherein the long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer; both the perception layer and the decision layer adopt a Transformer-based structure; the information encoded by the visual Transformer of the perception layer is used as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer;

[0052] The perception layer includes: constructing a tracking pool, the tracking pool includes a plurality of trackers; setting a search area and a target template; constructing a state space;

[0053] The decision layer includes: constructing action space and expected rewards;

[0054] A tracker training module is used to train the long-term tracker based on sequence modeling reinforcement learning, specifically including:

[0055] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination processed by the visual Transformer as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame;

[0056] Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length;

[0057] The trajectory τ is divided into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one;

[0058] Calculate the cumulative return at each time step

[0059] The modified trajectory The decision transformer decoder is input to the decision layer, and the output of the causal transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result.

[0060] The target tracking module is used to track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

[0061] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the steps of a long-term single target tracking method based on sequence modeling reinforcement learning as described above are executed.

[0062] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a plurality of computer instructions are stored in the computer-readable storage medium, and the plurality of computer instructions are used to enable a computer to execute a long-term single target tracking method based on sequence modeling reinforcement learning as described above.

[0063] Beneficial effects of the present invention:

[0064] The present invention is based on the current common long-term tracker paradigm, combined with the reinforcement learning method based on sequence modeling for improvement, adaptively selecting a baseline short-term tracker and improving its performance. The rule-based scoring judgment solidifies the pattern of the long-term tracking model. The intelligent agent decision layer based on sequence modeling proposed in the present invention can make more intelligent decisions by analyzing the content in the memory sequence, thereby improving the versatility of the long-term tracking model in various scenarios without relying on thresholds. The performance of a single short-term tracker will significantly affect the overall tracking results, which are jointly determined by the visual encoder and the tracking method. Most existing methods use a fixed ratio to define the search area, requiring the tracker to pay attention to the foreground and background within the area. However, a well-trained intelligent agent decision layer can dynamically adjust the ratio and optimize the search area location, because it is also inefficient to use an inefficient search area when the tracker prediction is inaccurate.

[0065] In the popular long-term tracker paradigm, the present invention mainly utilizes the short-term tracking model based on the vision transformer, supplemented by the keep tracking model (KeepTrack) and the motion model, to build a comprehensive long-term tracking model. At the same time, the decision transformer can be used as a robust baseline for the sequence decision model of the present invention. The agent decision layer uses the output of the vision transformer encoder as the state space, which contains the combined features of the target sample and the search area to generate the model selection result, optimize the search area variable, and determine whether the target exists.

[0066] In view of the common requirement for sequential modeling between long-term target tracking and sequential decision-making, this paper proposes an application of a reinforcement learning method based on sequence modeling in long-term target tracking scenarios, which will serve as a new paradigm to solve challenging problems in long-term target tracking scenarios.

[0067] The present invention is based on the cross-application of the field of computer vision single target tracking and the field of reinforcement learning, and is suitable for target tracking in long-term single target tracking scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 Schematic diagram of the main challenges and differences of the present invention;

[0070] Figure 2 A network block diagram of the method of the present invention;

[0071] Figure 3Schematic diagram of the specific optimization operations fed back to the tracking model. DETAILED DESCRIPTION

[0072] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0073] Embodiment 1: A long-term single target tracking method based on sequence modeling reinforcement learning, the method comprising:

[0074] Step 1: Get the original image of the current video frame and the position coordinates of the target of interest in the initial frame;

[0075] Step 2: construct a long-term tracker based on sequence modeling reinforcement learning, wherein the long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer; both the perception layer and the decision layer adopt a Transformer-based structure; the information encoded by the visual Transformer of the perception layer is used as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer;

[0076] The perception layer includes: constructing a tracking pool, the tracking pool includes a plurality of trackers; setting a search area and a target template; constructing a state space;

[0077] The decision layer includes: constructing action space and expected rewards;

[0078] Step 3: Training the long-term tracker based on sequence modeling reinforcement learning, specifically including:

[0079] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination processed by the visual Transformer as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame;

[0080] Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i∈R, T is the sequence length; each trajectory is generated according to the high-performance long-term tracker in the tracking pool, and the action record is obtained by comparing it with the true value, so the offline data constructed in this implementation is closer to the expert data.

[0081] The trajectory τ is divided into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one;

[0082] Calculate the cumulative return at each time step

[0083] The modified trajectory The decision transformer decoder is input to the decision layer, and the output of the causal transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result.

[0084] Step 4: Track the target using the trained sequence modeling reinforcement learning based long-term tracker.

[0085] This embodiment uses a Transformer-based method for trajectory modeling to replace the traditional reinforcement learning method. In view of the common requirement for sequential modeling between long-term target tracking and sequential decision-making, a reinforcement learning method based on sequence modeling is proposed for application in long-term target tracking scenarios. An integrated single target tracking model is adopted, which integrates the visual Transformer (ViT) of the perception layer and the decision Transformer (DT) of the decision layer. The information encoded by the ViT of the perception layer serves as the observation input of the decision layer DT. In addition, the action sequence generated by the decision layer is fed back to the perception layer to facilitate tracker selection and improve tracking performance.

[0086] Implementation method 2: This implementation method is a further limitation of the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation method, the plurality of trackers are further limited, specifically including:

[0087] The plurality of trackers include a short-term tracking method ViTTrack, a short-term tracking method KeepTrack and a motion module.

[0088] The goal of this implementation is to unify the network architecture of the visual perception layer and the decision layer by adopting a Transformer-based structure for these two layers. This approach allows direct interaction between Transformers performing different functions without the need for additional mapping layers. For this reason, ViTTrack is used to build a ViT-based tracker. ViTTrack uses ViT as its backbone network and implements a corner prediction head to determine the location of the target. It connects the template and search area into the spatial channel after the original embedding block, and then uses ViT to extract features. At the same time, this implementation also combines other single-target trackers to enhance the tracking pool. KeepTrack adopts an alternative strategy to track distractors, in stark contrast to most methods that aim to suppress distractors through more powerful appearance models. In addition, the tracking pool includes a simple but effective motion module that predicts the target state based on previous temporal trajectory information and is activated when the target is considered to be moving abnormally.

[0089] Implementation method three: This implementation method is a further limitation of the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation method, the setting of the search area and the target template is further limited, specifically including:

[0090] The setting of the search area and the target template specifically includes:

[0091] For a given target bounding box (x, y, w, h) in the initial frame, determine the slice size size according to the search area coefficient sf crop :

[0092]

[0093] Determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the padding part, where λ represents the scaling factor;

[0094] Forming a filled slice area

[0095] x1=[x+λw-λsize crop ]

[0096] x2=x1+size crop

[0097] y1=[y+λh-λsize crop ]

[0098] y2=y1+size crop

[0099]

[0100] Where W and H are the width and height of the image.

[0101] In this embodiment, since the decision-making layer in this application mainly achieves effective tracking and re-detection by optimizing the search area, the technology of how to obtain the search area and template will be provided to optimize the position and size of the overall search area to cope with the challenges of target changes occurring during long-term tracking.

[0102] Implementation mode 4: This implementation mode is a further limitation on the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation mode, the construction of the state space is further limited, specifically including:

[0103] The constructing state space specifically includes:

[0104] Separating the target template from the search area within the original video frame;

[0105] After the raw video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the visual transformer.

[0106] S=(F search ,F template ,F attn )

[0107] Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector, and S represents the state space sampling

[0108] The visual Transformer architecture serves as the backbone network, extracting features from the embedded target template and search region, generating feature tensors that encapsulate information related to the target template and search region, and reflecting the relationship between target frames.

[0109] This implementation effectively preserves the scene understanding relevant to perceptual intelligence tasks, and due to the inherent feature richness in perceptual intelligence algorithms, it can enhance the decision-making process of reinforcement learning agents.

[0110] Implementation mode 5: This implementation mode is a further limitation of the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this implementation mode, the construction of the action space is further limited, specifically including:

[0111] The constructing of the action space specifically includes:

[0112] Use nine dimensions to optimize the search area:

[0113] A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t)

[0114] Among them, (l, r, u, d) are used for horizontal and vertical translation (left, right, up, down), (l free ,r free ,u free ,d free ) four for free translation in the same direction, sf is used to adjust the scale factor of the search area, t is used to indicate whether the target exists, and A represents the action space sampling;

[0115] When a translation action is initiated, the search area is translated by αsize crop , where α represents the translation scale factor; after activating the free translation action, the free value will be adjusted in the corresponding direction;

[0116] When the decision layer judges that the deviation of the search area is large, free translation is activated; conversely, when the deviation is small, only quantitative translation is used;

[0117] The scale factor sf of the search area is adjusted by multiplying the continuous action value output by the decision Transformer by a fixed factor.

[0118] In this embodiment, the division of the search area is derived based on the extension of the target position. As the target position changes, the search area will also be adjusted accordingly. However, when the target becomes blurred, blocked, or the perception intelligence model fails, the position of the target may become unclear or non-existent. This situation will cause the perception intelligence model to deviate from the understanding of the current frame, resulting in a decrease in the focus on the target.

[0119] Although it is logically reasonable to define the action space by optimizing the search region, in practical design, the specific parameters of the search region cannot be used as a subspace of the action space. This limitation occurs because such a definition destroys the decision direction within the action space, which may hinder the convergence of offline training reinforcement learning agents. Therefore, this embodiment represents the optimization of the search region through nine dimensions.

[0120] Embodiment 6: This embodiment further limits the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this embodiment, the construction of the expected reward is further limited, specifically including:

[0121] The expected reward is constructed, specifically including: target score score, intersection over union ratio ratio compared to the ground truth IoU, characteristic distance ap dis and local outlier factor lof dis ;

[0122]

[0123] Among them, R is the reward record, represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

[0124] In this implementation, since the numerical spaces of the feature distance and the local outlier factor are not uniform, in order to balance the contributions of the two when recording the reward, the two are normalized.

[0125] In the sequence modeling-based reinforcement learning model of this application, the interpretation of the reward function is different from that of the traditional reinforcement learning method. This method does not maximize the reward function, but strategically plans the end point of the task by establishing a target reward (default value 1.0). This embodiment proposes a reward mechanism derived from the tracking model.

[0126] Embodiment 7: This embodiment further limits the long-term single target tracking method based on sequence modeling reinforcement learning as described above. In this embodiment, the loss function of the long-term tracker based on sequence modeling reinforcement learning is further limited, specifically including:

[0127] The loss function of the long-term tracker based on sequence modeling reinforcement learning is specifically:

[0128]

[0129] in, To predict actions, Reward for prediction.

[0130] The loss function of this implementation adopts two-term minimization supervision loss: predicting action and prediction rewards This implementation is based on the following two aspects. First, the predicted action is the essential attribute of the decision transformer, and serves as the main guide for optimizing the parameters of the decision layer. In addition, the source of the predicted action is the difference between the long-term tracker trajectory prediction result and the true value, which is consistent with the task requirements of the long-term tracking model. Second, the predicted reward acts as a regular term to regulate the overfitting of the decision layer. Relying solely on the predicted action will produce an effect closer to imitation learning, which is detrimental to the versatility of the decision layer. The predicted reward item can circumvent this problem well.

[0131] Embodiment 8: This embodiment is an example of a long-term single target tracking method based on sequence modeling reinforcement learning as described above, and specifically includes:

[0132] The overall network structure of the long-term tracker based on sequence modeling reinforcement learning is as follows Figure 2 As shown in Figure 1, it can be mainly divided into two levels: perception layer and decision layer. An integrated single target tracking model is adopted, which integrates the visual Transformer (ViT) of the perception layer and the decision Transformer (DT) sequence modeling of the decision layer. The information encoded by the ViT of the perception layer is used as the observation input of the decision layer DT. In addition, the action sequence generated by the decision layer is fed back to the perception layer to facilitate tracker selection and improve tracking performance.

[0133] Step 1: Build a tracking pool, which includes several trackers:

[0134] By adopting a Transformer-based structure for the visual perception layer and the decision layer to unify the network architecture of these two layers, this approach allows direct interaction between Transformers performing different functions without the need for additional mapping layers. Based on this insight, a ViT-based tracker (ViTTrack) is constructed. ViTTrack uses ViT as its backbone network and implements a corner prediction head to determine the location of the target. It links the template and search area into the spatial channel after the original embedding block, and then uses ViT to extract features. In addition, this embodiment also combines other single-target trackers to enhance the tracking pool. KeepTrack adopts an alternative strategy to track distractors, in stark contrast to most methods that aim to suppress distractors through more powerful appearance models. In addition, the tracking pool includes a simple but effective motion module that predicts the target state based on previous temporal trajectory information and is activated when the target is considered to be moving abnormally.

[0135] Step 2: Get the search area and target template

[0136] Since the decision layer in the present invention mainly achieves effective tracking and re-detection by optimizing the search area, the technology of how to obtain the search area and template will be provided. For the target bounding box (x, y, w, h) given in the initial frame, first determine the slice size size according to the search area coefficient sf crop :

[0137]

[0138] Then determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the filled part, where λ represents the scaling factor, which is set to 0.5 by default. Finally, the filled slice area is formed by the copyMakeBorder function of OpenCV.

[0139] x1=[x+λw-λsize crop ]

[0140] x2=x1+size crop

[0141] y1=[y+λh-λsize crop ]

[0142] y2=y1+size crop (2-2)

[0143]

[0144] These fill parts work with the copyMakeBorder function to divide the search area and the template, such as Figure 3 As shown in Figure 2. Where W and H are the width and height of the image. One of the decision-making parts of this work is to optimize the location and size of the overall search area to cope with the challenges of target changes that occur during long-term tracking.

[0145] Step 3: State Space Construction

[0146] This step aims to minimize the loss of valuable information within the video frame while effectively embedding the implicit correlation between consecutive frames. Figure 2 As shown in Figure 2, the process starts with a target sampling module that separates the target template from the search region within the original video frame. Next, after the original video frame is processed, the target template and the search region are embedded using a patch embedding module to prepare for input into the Vision Transformer.

[0147] S=(F search ,F template ,F attn ) (3-1)

[0148] Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector.

[0149] Subsequently, the Vision Transformer architecture is used as the backbone network to further extract features from the embedded target template and search region. Finally, a feature tensor of size (1,721,768) is generated, encapsulating all information related to the target template and search region, and reflecting the relationship between the target frames. This approach effectively preserves scene understanding related to perceptual intelligence tasks, and due to the inherent feature richness in perceptual intelligence methods, it can enhance the decision-making process of reinforcement learning agents.

[0150] Step 4: Action Space Construction

[0151] The optimization of the search area is represented by nine dimensions: (l, r, u, d) four for horizontal and vertical translation (left, right, up, down), (l free ,r free ,u free ,d free ) four free translations in the same direction, sf is used to adjust the scale factor of the search area, and t is used to indicate whether the target is present. These parameters are encoded using a one-hot representation, where left-right and up-down movements are mutually exclusive, ensuring that opposite directions are not activated at the same time.

[0152] A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t) (4-1)

[0153] When a translation action is initiated, the search area is translated by αsize crop , where α represents the translation scale factor, and the default value is 0.05. After the free translation action is activated, the free value will be adjusted in the corresponding direction. When the agent decision layer determines that the deviation of the search area is large, the free translation will be activated; conversely, when the deviation is small, only quantitative translation is used. In order to adjust the scale coefficient sf of the search area, the present invention transforms the continuous action value output by the decision converter by multiplying it by a fixed coefficient. The size of the constant value indicates the degree of change of the scale coefficient, while ensuring that the minimum value of the scale coefficient is maintained at 2. In addition, the agent decision layer can also output whether the target exists in the search area, which helps to realize the re-detection long-term tracker.

[0154] Step 5: Expected Reward Construction

[0155] In the sequence modeling-based reinforcement learning model, the interpretation of the reward function is different from that of traditional reinforcement learning methods. Instead of maximizing the reward function, this method strategically plans the end point of the task by establishing a target reward (default value 1.0). A reward mechanism derived from the tracking model is proposed, which contains four main components: the target score, the intersection over union (IoU) ratio compared to the ground truth, and the reward function. IoU ), characteristic distance (ap dis ) and local outlier factor (lof dis ).

[0156]

[0157] Among them, R is the reward record. Since the numerical space of feature distance and local outlier factor is not uniform, in order to balance the contribution of the two when making reward records, the two are normalized. represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

[0158] The target score reflects the tracker's assessment of the presence of the target in the current frame after adjusting the search area, and is used as a reward component together with IoU. The feature distance quantifies the difference between the slice features of the target in the current frame and the slice features of the standard template, and is used as a penalty component in the reward function. The outlier factor and feature distance are comparable indicators for evaluating the difference between the target in the current frame and the standard template. The difference is calculated using the outlier operator and is also used as a penalty component in the reward function. In view of the different value ranges of these four components, this embodiment uses normalization technology to adjust them according to their respective value ranges to ensure that all components are in the interval [0,1]. The normalized reward function is expressed as formula (5-1).

[0159] Step 6: Perform model training

[0160] Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination after Vision Transformer processing (as state S), the search area adjustment method and target existence judgment (as action A), and the reward record R obtained by the reward function for each frame. Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i∈R, T is the sequence length. The existing long-term tracker generates each trajectory and obtains the action record by comparing it with the true value, so the offline data constructed in this embodiment is closer to the expert data.

[0161]

[0162] First, the trajectory τ is divided into several segments, each of which is of length K. It is represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K ), but this embodiment modifies these trajectories rather than inputting them directly, specifically, computing the cumulative reward for each time step It encourages the converter module to generate operations that reduce this value as close to 1.0 as possible. Then the modified trajectory Input to the causal Transformer decoder in the decision layer. The output of the decision Transformer is a sequence vector Where d is the dimension of the embedding space, and the vector is consistent with the action space constructed in step 4. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result. In terms of the loss function, two terms are used to minimize the supervision loss and predict the action a t , predict reward r t , the loss function is shown in formula (6-1). The complete training process is summarized in the following Algorithm 1.

[0163]

[0164] The proposed method can be tested on standard single target tracking datasets and can be optimized in a plug-and-play manner on some existing methods. In order to ensure a fair comparison and prove the superiority of the proposed method, the present invention can use existing publicly available short-term trackers and motion models. Specifically, the present invention adopts the same tracker selection ViTTrack, KeepTrack and motion model as the multi-model hybrid long-term tracker (VITKT_M). The state space size corresponds to the output of the combined features generated by the Vision Transformer. In addition, the present invention passes the information from the attention layer separately and integrates it with the combined features, so as to obtain a total state space dimension of 1,107,456, which is calculated as (2×721×768). Since the original decision Transformer can only produce continuous values ​​as action decisions, the proposed method of the present invention maps and estimates the part that must produce discrete values, which is consistent with the action space defined in the present invention. The mapping layer and the GPT model are trained simultaneously. The proposed method sets K=20 as the length of each trajectory segment. The GPT model adopts a five-layer Transformer architecture with an embedding dimension of 128. The dropout rate of the MLP layer is 0.1. Training was performed 1,000 times in a warm-up phase, followed by 10,000 iterations per epoch, and 10 epochs per scene (representing a complete sequence). The hardware consisted of a single RTX4090 GPU and a 12th Generation Intel(R) Core(TM) i7-12700K CPU running at 3.60GHz.

[0165] The sequence modeling agents of the present invention can effectively solve the long-standing challenges of long-term tracking, especially disappearing targets and position mutations. These agents are adaptively adjusted according to the current position of the target and the region of interest. The method proposed in the present invention significantly improves the tracking recall rate by automatically selecting a suitable tracking model after training, adjusting the position and size of the region of interest, and evaluating whether the target exists in the search area. In addition, the method is very effective in alleviating re-detection failures caused by sudden changes in the target position. Extensive experiments have demonstrated the strong generalization ability of the method across a variety of trackers and scenarios. Using widely used short-term tracking models, the method achieves performance that exceeds the state-of-the-art methods. The present invention hopes that the characteristics of the proposed method will help to re-examine the long-term tracking challenges.

Claims

1. A long-term single target tracking method based on sequence modeling reinforcement learning, characterized in that: The method comprises: Step 1: Get the original image of the current video frame and the position coordinates of the target of interest in the initial frame; Step 2: construct a long-term tracker based on sequence modeling reinforcement learning, wherein the long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer; both the perception layer and the decision layer adopt a Transformer-based structure; the information encoded by the visual Transformer of the perception layer is used as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer; The perception layer includes: constructing a tracking pool, the tracking pool includes a plurality of trackers; setting a search area and a target template; constructing a state space; The decision layer includes: constructing action space and expected rewards; Step 3: Training the long-term tracker based on sequence modeling reinforcement learning, specifically including: Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination processed by the visual Transformer as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame; Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length; The trajectory τ is divided into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one; Calculate the cumulative return at each time step The modified trajectory The decision transformer decoder is input to the decision layer, and the output of the causal transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result. Step 4: Track the target using the trained sequence modeling reinforcement learning based long-term tracker.

2. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 1 is characterized in that: The plurality of trackers include a short-term tracking method ViTTrack, a short-term tracking method KeepTrack and a motion module.

3. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 2 is characterized in that: The setting of the search area and the target template specifically includes: For a given target bounding box (x, y, w, h) in the initial frame, the slice size is determined according to the search area coefficient sf Determine the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the slice area, including the padding part, where λ represents the scaling factor; Forming filled slice areas x1=[x+λw-λsize crop ] x2=x1+size crop <h2 style=";text-align:left;direction:ltr">y1 = [y+λh-λsize<h2 style=";text-align:left;direction:ltr"> crop <h2 style=";text-align:left;direction:ltr"> ] y2=y1+size crop Where W and H are the width and height of the image.

4. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 3 is characterized in that: The constructing state space specifically includes: Separating the target template from the search area within the original video frame; After the raw video frame is processed, the target template and search region are embedded using the patch embedding module to prepare for input into the visual transformer. S=(F search ,F template ,F attn ) Among them, F search represents the search area feature vector, F template represents the target template feature vector, F attn represents the global attention feature vector, and S represents the state space sampling The visual Transformer architecture serves as the backbone network, extracting features from the embedded target template and search region, generating feature tensors that encapsulate information related to the target template and search region, and reflecting the relationship between target frames.

5. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 4 is characterized in that: The constructing of the action space specifically includes: Use nine dimensions to optimize the search area: A=(l,r,u,d,l free ,r free ,u free ,d free ,sf,t) Among them, (l, r, u, d) are the four horizontal and vertical translations, (l free ,r free ,u free ,d free ) are four free translations in the same direction, sf is used to adjust the scale factor of the search area, t is used to indicate whether the target exists, and A represents the action space sampling; When a translation action is initiated, the search area is translated by αsize crop , where α represents the translation scale factor; after activating the free translation action, the free value will be adjusted in the corresponding direction; When the decision layer judges that the deviation of the search area is large, free translation is activated; conversely, when the deviation is small, only quantitative translation is used; The scale factor sf of the search area is adjusted by multiplying the continuous action value output by the decision Transformer by a fixed factor.

6. The long-term single target tracking method based on sequence modeling reinforcement learning according to claim 5, characterized in that: The expected reward is constructed, specifically including: target score score, intersection over union ratio ratio compared to the ground truth IoU , characteristic distance ap dis and local outlier factor lof dis ; Among them, R is the reward record, represents the normalized representation of feature distance, Represents the normalized representation of the local outlier factor.

7. A long-term single target tracking method based on sequence modeling reinforcement learning according to any one of claims 1 to 6, characterized in that: The loss function of the long-term tracker based on sequence modeling reinforcement learning is specifically: in, To predict actions, Reward for prediction.

8. A long-term single target tracking system based on sequence modeling reinforcement learning, characterized in that: The system comprises: The image and position acquisition module is used to obtain the original image of the current video frame and the position coordinates of the target of interest in the initial frame; A tracker building module, used to build a long-term tracker based on sequence modeling reinforcement learning, wherein the long-term tracking model based on sequence modeling reinforcement learning includes a perception layer and a decision layer; both the perception layer and the decision layer adopt a Transformer-based structure; the information encoded by the visual Transformer of the perception layer is used as the observation input of the decision Transformer, and the action sequence generated by the decision layer is fed back to the perception layer; The perception layer includes: constructing a tracking pool, the tracking pool includes a plurality of trackers; setting a search area and a target template; constructing a state space; The decision layer includes: constructing action space and expected rewards; A tracker training module is used to train the long-term tracker based on sequence modeling reinforcement learning, specifically including: Using a series of video sequences Seq train Construct a 2D tracker environment where each video sequence task seq i ∈Seq train It includes: the feature combination processed by the visual Transformer as the state S, the search area adjustment method and target existence judgment as the action A, and the reward record R obtained by the reward function for each frame; Each trajectory τ=(s0,a0,r0,…,s L ,a L ,r L ), where s i ∈S,a i ∈A,r i ∈R, T is the sequence length; The trajectory τ is divided into several segments, each of which is of length K, represented as the input sequence τ t+1:t+K =(s t+1 ,a t+1 ,r t+1 ,…,s t+K ,a t+K ,r t+K )one; Calculate the cumulative return at each time step The modified trajectory The decision transformer decoder is input to the decision layer, and the output of the causal transformer decoder is a sequence vector Where d is the dimension of the embedding space. The sequence vector feeds the output search area optimization action back to the tracking model to optimize the tracking result. The target tracking module is used to track the target using the trained long-term tracker based on sequence modeling reinforcement learning.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor runs the computer program stored in the memory, the steps of the method according to any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of computer instructions, and the plurality of computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images