State-action-reward representation extraction method based on Transform
By using a Transformer-based 'state-action-reward' representation extraction method, and by fusing local image information with sequence temporal information, the performance of deep reinforcement learning algorithms is improved, and the problem of insufficient fusion of image feature extraction and sequence encoding in existing technologies is solved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-03-27
AI Technical Summary
Existing deep reinforcement learning methods have shortcomings in sample efficiency, image feature extraction, reward design and exploration, and the fusion of sequence coding and image coding, resulting in poor algorithm performance.
We employ a Transformer-based 'state-action-reward' representation extraction method. By performing patch embedding on the image, we preserve fine-grained spatial features and merge sequence tokens and image tokens into fused features. We then utilize the cross-connection of Step Transformer and Sequence Transformer, combined with causal masks, to perform autoregressive modeling.
It improves the autoregressive capability of the encoder, enhances the correlation between image and sequence features, and improves the performance of deep reinforcement learning algorithms.
Smart Images
Figure CN121745209A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep reinforcement learning technology, specifically relating to a Transformer-based method for extracting "state-action-reward" representations. Background Technology
[0002] Existing deep reinforcement learning methods mainly consider three aspects: Sample efficiency Low sample efficiency in reinforcement learning is a widely concerned problem. Common solutions include adding experience replay or episodic control functions to increase the number of times historical trajectories are used for updates.
[0003] Indicate extraction Deep reinforcement learning directly uses pixel images as input, requiring the ability to extract features from both trajectory sequences and images. The quality of the feature extraction method significantly impacts the algorithm's performance. Commonly used methods in this area include various image feature extraction techniques and text feature encoding methods.
[0004] Reward Design and Exploration For some particularly complex scenarios, manually designing reliable reward functions is extremely difficult. However, simply setting a binary reward for task completion can lead to algorithm convergence problems. Common solutions include imitation learning, which uses expert trajectories to pre-train the agent's policy, or exploration methods specifically designed for sparse reward scenarios.
[0005] Deficiencies of existing technology Using the entire image to extract features lacks attention to local information, resulting in weak correlation between the obtained features and image information, and poor autoregressive ability of the encoder.
[0006] The trajectory of reinforcement learning has temporal characteristics, while the attention mechanism in text processing is bidirectional, which leads to the association between the current token and future tokens, generating perturbation information and reducing the accuracy of trajectory prediction.
[0007] Insufficient fusion of sequence coding and image coding, and lack of proper alignment between the two, resulted in poor performance of the final model. Summary of the Invention
[0008] (a) Technical problems to be solved The technical problem to be solved by this invention is to improve the autoregressive capability of the encoder by focusing on local image information and sequence temporal information, thereby improving the final performance of the algorithm.
[0009] (II) Technical Solution To address the aforementioned technical problems, this invention provides a Transformer-based "state-action-reward" representation extraction method. This Transformer-based "state-action-reward" representation extraction method is a deep reinforcement learning representation extraction method. The goal of deep reinforcement learning is to find a strategy that maximizes rewards, and the quality of the selected representation extraction method directly affects the final performance of the algorithm. This method improves the autoregressive ability of the encoder by focusing on local image information and sequence temporal information, ultimately enhancing the performance of the deep reinforcement learning algorithm.
[0010] The method involves performing patch embedding on the image to preserve fine-grained spatial features and improve the correlation between features and local image information.
[0011] The method merges sequence tokens and image tokens into a fusion feature that preserves temporal information, and uses a causal mask to ensure that the token at time t cannot be associated with future tokens (>t).
[0012] The method implements a trajectory sequence encoder that makes better use of local image information and trajectory temporal information, obtains better autoregressive ability, and can effectively improve the performance of deep reinforcement learning algorithms.
[0013] The method involves two interconnected components: a Step Transformer and a Sequence Transformer. The Step Transformer learns local representations, namely StAR representations, by using self-attention tokens within a single time step window; here, image states are encoded as ViT-like patches, preserving fine-grained spatial information; then, the Sequence Transformer combines the StAR representations with pure image state representations from the entire sequence to model the entire input trajectory; where the pure image state representations are extracted as convolutional features.
[0014] (1) First, briefly introduce RL for sequence modeling, and then introduce in detail the two Transformer components and their corresponding token embeddings in the Transformer-based "state-action-reward" representation extraction method: The following is a detailed introduction to RL used for sequence modeling: Consider a Markov Decision Process (MDP) using tuples. It means that, among them Indicates state, Indicates an action, P represents the reward, and P represents the reward. The given transition dynamics; in MDP, the trajectory Defined as the agent's past experience, it is a sequence of states, actions, and rewards arranged in chronological order as follows:
[0015] Sequence modeling in RL predicts actions based on past experience:
[0016] Existing techniques attempt to use the existing Transformer architecture for RL, with the formula described above; where state s, action a, and reward r are considered as input tokens, while causal masks are used to ensure the generation of autoregressive output sequences, i.e., in the formula... In this context, a token can access any of its preceding tokens through self-attention; To explicitly focus on tokens with strong causal relationships while also paying attention to long-term relationships, a method is adopted to decompose the trajectory into small combinations of state-action-reward tuples, namely s, a, and r. It learns the local relationships within each group of tokens through self-attention and then performs long-term sequence modeling.
[0017] (2) Regarding Step Transformer, the details are as follows: ⑤ Grouping of Status-Action-Reward The intuition for grouping is to explicitly model strong local relationships; to do this, firstly, the trajectories... The set is divided into groups, each group consisting of previous actions. ,award and current state Composition; each element in a group has a strong causal relationship with the other elements; ⑥ Status-Action-Reward Token Embeddings In Step Transformer, each input image state is divided into a set of non-overlapping spatial patches along its spatial dimension. Each input image state is tokenized; the motivation for using patch embeddings is to create fine-grained state embeddings, which allows the Step Transformer to model the relationship between actions and rewards using local regions of the state; for action and reward tokens, a linear layer is used for embedding; after obtaining a set of state, action, and reward embeddings, this is used as input to the initial Step Transformer layer, which is given by the following equation:
[0018] Each trajectory has T sets of such tokens, which are processed simultaneously by a Step Transformer with shared parameters; ⑦ Step Transformer layer The Step Transformer layer uses a traditional Transformer design; it originates from the previous layer. Each group of tokens consists of a mapping Step Transformer layer transformation to ; ⑧ StAR indicates In each Step Transformer layer At the output location, tokens are output through aggregation. To obtain the State-Action-Reward representation, i.e., the StAR representation. :
[0019] This represents the chain of tokens within each group. This represents the time position embeddings for each time step; finally, the output StAR representation is... The data is fed into the corresponding Sequence Transformer layer for long dependency sequence modeling.
[0020] (3) The Sequence Transformer is described in detail below: The Sequence Transformer models long dependency sequences by looking at learnable StAR representations and pure state tokens over the entire trajectory; ③ Pure State Token Embeddings In addition to the patch-level token embeddings in the Step Transformer, the input image state Embed as a whole to create pure state tokens Each such token represents a single state representation, globally describing the state in space; to achieve this, a CNN encoder is used to process each state, as the convolutional layers spatially mix features:
[0021] in, Represents the time position embeddings for each time step, and the embeddings added to... They are exactly the same; ④ Sequence Transformer layer Similar to the Step Transformer, the Sequence Transformer is implemented using a traditional Transformer layer design; Sequence Transformer layer The input consists of representations from two sources: ① the StAR representation learned from the corresponding Step Transformer layer. ② Output from the previous Sequence Transformer layer ;Set here The tokens are represented as pure states; the two types of tokens are merged to form a single sequence, preserving their chronological order.
[0022] Located in origin of Previously, because Includes previous actions The information, it precedes in the trajectory Furthermore, a causal mask is applied in the Sequence Transformer to ensure that the token at time t does not concern itself with any future tokens, i.e., >t. The Sequence Transformer uses the StAR representation generated from each intermediate Step Transformer layer, rather than the final StAR representation after all Step Transformer layers; in this way, the model gains the ability to view the StAR representation at multiple levels of abstraction; furthermore, the Sequence Transformer computes an intermediate set of output tokens, including: Then, select Tokens at even indices are considered pure state tokens. Then it is fed into the next Sequence Transformer layer.
[0023] Among them, the Even indices start from 1.
[0024] (III) Beneficial Effects Compared with the prior art, the present invention has the following beneficial effects: (1) This invention provides a Transformer-based “state-action-reward” representation extraction method, which improves the autoregression rate of the encoder by fusing image and sequence features, thereby improving the performance of deep reinforcement learning models.
[0025] (2) The present invention performs patch embedding on the image to preserve fine-grained spatial features and improves the correlation between features and local information of the image.
[0026] (3) The present invention ensures that the token at time t cannot pay attention to future tokens (>t) by using causal masking, which can prevent incorrect attention from interfering with the trajectory prediction capability.
[0027] (4) This invention implements a general deep reinforcement learning trajectory sequence encoder. This encoder has better autoregressive capability and can effectively improve the performance of deep reinforcement learning algorithms. Attached Figure Description
[0028] Figure 1 This is a diagram of the overall architecture of the present invention.
[0029] Figure 2 This is a schematic diagram of an existing approach to using Transformer to model RL as a sequence.
[0030] Figure 3 This diagram illustrates how to use Transformer to improve RL as a sequence modeling approach.
[0031] Figure 4 This is a schematic diagram of the attention mapping between action tokens and pixel state patches.
[0032] Figure 5 This is a schematic diagram of the Step Transformer.
[0033] Figure 6 This is a schematic diagram of a Sequence Transformer. Detailed Implementation
[0034] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.
[0035] Example 1 This embodiment provides a Transformer-based "state-action-reward" representation extraction method, which is a deep reinforcement learning representation extraction method. The goal of deep reinforcement learning is to find a policy that maximizes rewards, and the quality of the selected representation extraction method directly affects the final performance of the algorithm. This method improves the autoregressive ability of the encoder by focusing on local image information and sequence temporal information, and ultimately improves the performance of the deep reinforcement learning algorithm. The method performs patch embedding on the image to preserve fine-grained spatial features and improve the correlation between features and local image information; The method merges sequence tokens and image tokens into a fusion feature that preserves temporal information, and uses a causal mask to ensure that the token at time t cannot be associated with future tokens (>t). The method implements a trajectory sequence encoder, which makes better use of local image information and trajectory temporal information, obtains better autoregressive ability, and can effectively improve the performance of deep reinforcement learning algorithms. Figure 1 The overall architecture diagram of this invention shows that the "state-action-reward" representation extraction method based on Transformer proposed in this invention includes two intersecting components: a Step Transformer and a Sequence Transformer, which are cross-connected. The Step Transformer learns local representations, namely StAR representations, by using self-attention tokens within a single time step window; here, image states are encoded as ViT-like patches, preserving fine-grained spatial information; then, the Sequence Transformer combines the StAR representations with pure image state representations from the entire sequence to model the entire input trajectory; where the pure image state representations are extracted as convolutional features.
[0036] The following section first provides a brief introduction to RL used for sequence modeling, and then details the two Transformer components and their corresponding token embeddings in the Transformer-based "state-action-reward" representation extraction method: (1) RL for sequence modeling Consider a Markov Decision Process (MDP) using tuples. It means that, among them Indicates state, Indicates an action, P represents the reward, and P represents the reward. The given transition dynamics; in MDP, the trajectory Defined as the agent's past experience, it is a sequence of states, actions, and rewards arranged in chronological order as follows:
[0037] Sequence modeling in RL predicts actions based on past experience:
[0038] Existing technologies attempt to use existing Transformer architectures for RL (such as...) Figure 2 The formula is as described above; where state s, action a, and reward r are considered as input tokens, and a causal mask is used to ensure the generation of the autoregressive output sequence, i.e., in the formula... In this context, a token can access any of its preceding tokens through self-attention; To explicitly focus on tokens with (potential) strong causal relationships while simultaneously considering long-term relationships, a method is employed that decomposes the trajectory into small combinations of state-action-reward tuples, namely s, a, and r. This method learns local relationships within each group of tokens through self-attention and then performs long-term sequence modeling (see [link to documentation]). Figure 3 and Figure 4 ); (2) Step Transformer 9. Grouping of Status-Action-Reward The intuition for grouping is to explicitly model strong local relationships; to do this, firstly, the trajectories... The set is divided into groups, each group consisting of previous actions. ,award and current state Composition (e.g.) Figure 5 Each element in a group has a strong causal relationship with the other elements; ⑩ Status-Action-Reward Token Embeddings In Step Transformer, each input image state is divided into a set of non-overlapping spatial patches along its spatial dimension. Each input image state is tokenized; the motivation for using patch embeddings is to create fine-grained state embeddings, which allows the Step Transformer to model the relationship between actions and rewards using local regions of the state; for action and reward tokens, a linear layer is used for embedding; after obtaining a set of state, action, and reward embeddings, this is used as input to the initial Step Transformer layer, which is given by the following equation:
[0039] Each trajectory has T sets of such tokens, which are processed simultaneously by a Step Transformer with shared parameters; Step Transformer layer The Step Transformer layer uses a traditional Transformer design; it originates from the previous layer. Each group of tokens consists of a mapping Step Transformer layer transformation to ; StAR stated In each Step Transformer layer At the output location, tokens are output through aggregation. To obtain the Status-Action-Reward representation (StAR representation). :
[0040] This represents the chain of tokens within each group. This represents the time position embeddings for each time step; finally, the output StAR representation is... The data is fed into the corresponding Sequence Transformer layer for long dependency sequence modeling. (3) Sequence Transformer The Sequence Transformer models long-dependent sequences by looking at learnable StAR representations and pure state tokens over the entire trajectory (e.g., ...). Figure 6 ); ⑤ Pure State Token Embeddings In addition to the patch-level token embeddings in the Step Transformer, the input image state Embed as a whole to create pure state tokens Each such token represents a single state representation, globally describing the state in space; to achieve this, a CNN encoder is used to process each state, as the convolutional layers spatially mix features:
[0041] in, Represents the time position embeddings for each time step, and the embeddings added to... They are exactly the same; ⑥ Sequence Transformer layer Similar to the Step Transformer, the Sequence Transformer is implemented using a traditional Transformer layer design; Sequence Transformer layer The input consists of representations from two sources: ① the StAR representation learned from the corresponding Step Transformer layer. ② Output from the previous Sequence Transformer layer ;Set here The tokens are represented as pure states; the two types of tokens are merged to form a single sequence, preserving their chronological order.
[0042] Located in origin of Previously, because Includes previous actions The information, it precedes in the trajectory Furthermore, a causal mask is applied in the Sequence Transformer to ensure that the token at time t does not concern itself with any future tokens (i.e., > t). The Sequence Transformer uses the StAR representation generated from each intermediate Step Transformer layer, rather than the final StAR representation after all Step Transformer layers; in this way, the model gains the ability to view the StAR representation at multiple levels of abstraction; furthermore, the Sequence Transformer computes intermediate sets of output tokens, such as: Then, select Tokens at even indices (starting from 1) are considered pure state tokens. Then it is fed into the next Sequence Transformer layer.
[0043] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A Transformer-based method for extracting "state-action-reward" representations, characterized in that, The Transformer-based "state-action-reward" representation extraction method is a deep reinforcement learning representation extraction method. The goal of deep reinforcement learning is to find a policy that maximizes rewards, and the quality of the selected representation extraction method directly affects the final performance of the algorithm. This method improves the autoregressive ability of the encoder by focusing on local image information and sequence temporal information, thereby improving the performance of the deep reinforcement learning algorithm.
2. The Transformer-based "state-action-reward" representation extraction method as described in claim 1, characterized in that, The method performs patch embedding on the image to preserve fine-grained spatial features and improve the correlation between features and local image information.
3. The Transformer-based "state-action-reward" representation extraction method as described in claim 1, characterized in that, The method merges sequence tokens and image tokens into a fusion feature that preserves temporal information, and uses a causal mask to ensure that the token at time t cannot be associated with future tokens (>t).
4. The Transformer-based "state-action-reward" representation extraction method as described in claim 1, characterized in that, The method implements a trajectory sequence encoder that makes better use of local image information and trajectory temporal information, obtains better autoregressive ability, and can effectively improve the performance of deep reinforcement learning algorithms.
5. The Transformer-based "state-action-reward" representation extraction method as described in claim 1, characterized in that, The method involves two interconnected components: a Step Transformer and a Sequence Transformer. The Step Transformer learns local representations, namely StAR representations, by using self-attention tokens within a single time step window; here, image states are encoded as ViT-like patches, preserving fine-grained spatial information; then, the Sequence Transformer combines the StAR representations with pure image state representations from the entire sequence to model the entire input trajectory; where the pure image state representations are extracted as convolutional features.
6. The Transformer-based "state-action-reward" representation extraction method as described in claim 5, characterized in that, First, we'll briefly introduce RL for sequence modeling, and then delve into the two Transformer components and their corresponding token embeddings in the Transformer-based "state-action-reward" representation extraction method: The following is a detailed introduction to RL used for sequence modeling: Consider a Markov Decision Process (MDP) using tuples. It means that among them Indicates state, Indicates an action, P represents the reward, and P represents the reward. The given transition dynamics; in MDP, the trajectory Defined as the agent's past experience, it is a sequence of states, actions, and rewards arranged in chronological order as follows: Sequence modeling in RL predicts actions based on past experience: Existing techniques attempt to use the existing Transformer architecture for RL, with the formula described above; where state s, action a, and reward r are considered as input tokens, while causal masks are used to ensure the generation of autoregressive output sequences, i.e., in the formula... In this context, a token can access any of its preceding tokens through self-attention; To explicitly focus on tokens with strong causal relationships while also paying attention to long-term relationships, a method is adopted to decompose the trajectory into small combinations of state-action-reward tuples, namely s, a, and r. It learns the local relationships within each group of tokens through self-attention and then performs long-term sequence modeling.
7. The Transformer-based "state-action-reward" representation extraction method as described in claim 6, characterized in that, The Step Transformer is described in detail below: ① Grouping of Status-Action-Reward The intuition for grouping is to explicitly model strong local relationships; to do this, firstly, the trajectories... The set is divided into groups, each group consisting of previous actions. ,award and current state Composition; each element in a group has a strong causal relationship with the other elements; ② Status-Action-Reward Token Embeddings In Step Transformer, each input image state is divided into a set of non-overlapping spatial patches along its spatial dimension. Each input image state is tokenized; the motivation for using patch embeddings is to create fine-grained state embeddings, which allows the Step Transformer to model the relationship between actions and rewards using local regions of the state; for action and reward tokens, a linear layer is used for embedding; after obtaining a set of state, action, and reward embeddings, this is used as input to the initial Step Transformer layer, which is given by the following equation: Each trajectory has T sets of such tokens, which are processed simultaneously by a Step Transformer with shared parameters; ③ Step Transformer layer The Step Transformer layer uses a traditional Transformer design; it originates from the previous layer. Each group of tokens consists of a mapping Step Transformer layer transformation to ; ④ StAR indicates In each Step Transformer layer At the output location, tokens are output through aggregation. To obtain the State-Action-Reward representation, i.e., the StAR representation. : This represents the chain of tokens within each group. This represents the time position embeddings for each time step; finally, the output StAR representation is... The data is fed into the corresponding Sequence Transformer layer for long dependency sequence modeling.
8. The Transformer-based "state-action-reward" representation extraction method as described in claim 7, characterized in that, The Sequence Transformer is described in detail below: The Sequence Transformer models long dependency sequences by looking at learnable StAR representations and pure state tokens over the entire trajectory; ① Pure State Token Embeddings In addition to the patch-level token embeddings in the Step Transformer, the input image state Embed as a whole to create pure state tokens Each such token represents a single state representation, globally describing the state in space; to achieve this, a CNN encoder is used to process each state, as the convolutional layers spatially mix features: in, Represents the time position embeddings for each time step, and the embeddings added to... They are exactly the same; ② Sequence Transformer layer Similar to the Step Transformer, the Sequence Transformer is implemented using a traditional Transformer layer design; Sequence Transformer layer The input consists of representations from two sources: ① the StAR representation learned from the corresponding Step Transformer layer. ② Output from the previous Sequence Transformer layer ;Set here The tokens are represented as pure states; the two types of tokens are merged to form a single sequence, preserving their chronological order. Located in origin of Previously, because Includes previous actions The information, it precedes in the trajectory Furthermore, a causal mask is applied in the Sequence Transformer to ensure that the token at time t does not concern itself with any future tokens, i.e., > t; The Sequence Transformer uses the StAR representation generated from each intermediate Step Transformer layer, rather than the final StAR representation after all Step Transformer layers; in this way, the model gains the ability to view the StAR representation at multiple levels of abstraction; furthermore, the Sequence Transformer computes an intermediate set of output tokens, including: Then, select Tokens at even indices are considered pure state tokens. Then it is fed into the next Sequence Transformer layer.
9. The Transformer-based "state-action-reward" representation extraction method as described in claim 8, characterized in that, The Even indices start from 1.
10. The Transformer-based "state-action-reward" representation extraction method as described in claim 8, characterized in that, The method improves the autoregressive rate of the encoder by fusing image and sequence features, thereby enhancing the performance of deep reinforcement learning models.