Off-line reinforcement learning method combining generalized advantage estimation and mode decomposition interaction

Abstract

The invention belongs to off-line reinforcement learning, and particularly relates to an off-line reinforcement learning method combining generalized advantage estimation and modal decomposition interaction. In order to solve the problems of insufficient track splicing capability and intra-modal interaction and inter-modal interaction of multi-modal information in a track in an off-line reinforcement learning method based on TransFormer, the track splicing capability of a model in an off-line data set is improved by adopting generalized advantage estimation, a modal decomposition interaction module utilizes an encoder-decoder structure, and a mode decomposition interaction module utilizes an encoder-decoder structure, so that the track splicing capability of the model in the off-line data set is improved. And the encoder effectively processes time sequence information in a single mode by utilizing modal internal interaction based on ConvFormer. Relative to this, the correlation between the state and the action is captured based on the interaction between the modes of the TransFormer, so that the characterization capability of the model on the state and action sequence is enhanced. The decoder further fuses multi-modal information by using convolution and cross-attention mechanisms, thereby realizing more accurate motion prediction.

Description

Technical Field

[0001] The present invention belongs to offline reinforcement learning, and specifically relates to an offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction. Background Art

[0002] Reinforcement learning (RL) is a control learning paradigm of machine learning that aims to help intelligent agents interact with the surrounding environment through trial and error, and learn the optimal behavior strategy to maximize long-term returns based on the reward signals received during the interaction. By continuously improving and perfecting reinforcement learning, the performance and effect of intelligent agents in multiple practical applications such as resource allocation, robotics and autonomous driving can be improved.

[0003] Existing reinforcement learning can be divided into traditional reinforcement learning and deep reinforcement learning. Traditional reinforcement learning mainly uses manually designed features or tables to represent states and actions, and adopts value function-based methods or policy-based methods to learn state and action representations. Among them, value function-based methods learn policies by estimating and optimizing value functions, such as Q-learning and SARSA; policy-based methods learn optimal policies by directly parameterizing and optimizing policies, such as Policy Gradient and Actor-Critic, but the above methods will face problems such as dimensionality disaster and overfitting when dealing with high-dimensional state space and action space; while deep reinforcement learning (DRL) combines reinforcement learning with deep learning by using CNN or TranFormer to approximate value functions or policy functions, so that it can effectively avoid dimensionality disaster and overfitting when dealing with high-dimensional, continuous state and action space. For example, Imtiaz et al. used a fully convolutional network to form a deep Q network (DQN) in RL, and Qin et al. proposed a CNN-based strategy, using CNN to represent and select states and actions in high-dimensional space, but CNN has limitations when dealing with long sequence problems and historical state and action sequences. TransFormer has an advantage in capturing long-range dependencies in sequences. By applying TransFormer to RL, it is possible to model trajectories in sequence through its attention mechanism, allowing the agent to pay attention to important clues in the environment and improve the agent's decision-making. Shang et al. proposed StARTransFormer, which models the strong local relationship between state, action, and reward in visual RL, effectively improving the problem of long sequence modeling in RL; Gopalakrishnan et al. proposed SloTTAr, which combines the TransFormer encoder-decoder architecture with slot attention, learns spatiotemporal features from action observation sequences, and simplifies the decision-making and planning of complex RL problems. Therefore, TransFormer is widely used to solve reinforcement learning problems.

[0004] Transformer-based reinforcement learning algorithms can be classified according to different learning methods, namely offline reinforcement learning and online reinforcement learning. Online reinforcement learning does not require the collection of a large number of data samples in advance. Its intelligent agent obtains feedback when interacting freely with the environment, and gradually updates the strategy while collecting data generated during the interaction process. For example, Essling et al. proposed the Deep Transformer Network (DTQN), but it has the limitation that the simulation environment is unavailable or the interaction cost is very high. Offline reinforcement learning can get rid of this limitation and learn strategies from previously collected offline data sets. It does not need to interact with the environment in real time, which reduces the cost of experiments and training. For example, Xu et al. proposed the Prompt-based decision transformer, Wang et al. proposed the bootstrap Transformer (BooT), and Konan et al. proposed the contrastive decision Transformer. All of them use Transformer to model the sequence of offline data, but the above methods face the problem that Transformer does not have the ability to splice trajectories in offline data sets. Therefore, Yamagata et al. proposed the Q-learning decision Transformer, which uses Q-learning for dynamic programming to re-label offline data sets and connect multiple segments of trajectories, but ignores that its performance will also be affected by environmental randomness. Gao et al. proposed the ACT algorithm, which evaluated the quality of actions through an advantage function. The advantage value of action evaluation improved the trajectory splicing ability of TransFormer and its susceptibility to environmental randomness. However, the above method only models the data in the trajectory as a sequence for strategy learning, ignoring the importance of intra-modal and inter-modal interactions of multimodal information in the trajectory, which limits the decision-making ability and performance of the intelligent agent.

[0005] Therefore, there is a need in this field for an offline reinforcement learning method to solve the problem that the offline reinforcement learning method based on TransFormer has insufficient trajectory splicing ability, and the problem that the offline reinforcement learning method based on TransFormer ignores the intra-modal interaction and inter-modal interaction of multimodal information in the trajectory. Summary of the invention

[0006] In view of the above problems, the present invention provides an offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction. The method improves the model's trajectory splicing ability for offline data sets. In addition, the method fully learns the intra-modal and inter-modal interactions of multimodal information in the trajectory, enhances the intelligent agent's comprehensive understanding of the data, and further improves its decision-making ability.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides an offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction, comprising the following steps:

[0009] Step 1: Collect trajectory data and build an offline dataset;

[0010] Step 2, inputting the offline data set into a pre-trained offline reinforcement learning model and outputting the predicted action, wherein the offline reinforcement learning model includes an advantage estimation module and a modal decomposition interaction module;

[0011] The advantage estimation module consists of three parts: value function approximation, generalized advantage estimation and predictor. The value function approximation approximates the value function of the offline data set by an approximate dynamic programming method. The generalized advantage estimation estimates the advantage of the state in the offline data set based on the approximate value function to re-label the offline data set. The predictor evaluates the target advantage for each state.

[0012] The modal decomposition interaction module consists of a modal decomposition module, an encoder and a decoder. The modal decomposition module decomposes the information in the trajectory sequence and splits it into information of different modalities. The encoder performs intra-modal interaction based on ConvFormer and inter-modal interaction based on TransFormer on state and action information. The decoder combines convolution and cross-attention to fully fuse multi-modal information to predict actions.

[0013] The present invention uses generalized advantage estimation to improve the model's trajectory splicing ability in offline data sets, and improves the ability of the intelligent agent to learn the optimal strategy. The modal decomposition interaction module of the present invention utilizes an encoder-decoder structure to fully utilize the correlation within and between modalities in multimodal trajectory data, helping the intelligent agent to more comprehensively understand and utilize trajectory data. The encoder in the modal decomposition interaction module utilizes the ConvFormer-based modal intra-interaction to effectively process the timing information in a single modality. This method helps the model capture local correlations in state and action sequences. In contrast, the TransFormer-based inter-modal interaction captures the correlation between state and action, enhancing the model's ability to characterize state and action sequences. The decoder further uses convolution and cross-attention mechanisms to fuse multimodal information, thereby achieving more accurate action prediction.

[0014] Furthermore, the value function approximation is specifically as follows:

[0015] By parameterizing the value network V ψ To approximate the state value function to evaluate the long-term accumulated reward or return expectation in a specific state, and to update and optimize the value network V ψ To obtain the state value estimate, a three-layer MLP is used to construct independent value networks V ψ, and introduce the corresponding target network for each state

[0016] In the value network V ψ In the training of With the expected regression loss L expectile_regression Value Network V ψ Update the parameters and target network The parameters of are updated in the form of moving average, as shown in formulas (1) to (2):

[0017]

[0018] Among them, γ represents the discount factor, s' represents the next state after the state s is transferred, and ψ represents the value network V ψ Parameters, Indicates the target network at the kth step Parameters, k represents the index of the time step, α represents the smoothing coefficient, and The function represents the expected regression loss, where represents the indicator function, and σ1 represents the hyperparameter.

[0019] Furthermore, the generalized advantage estimation is specifically:

[0020] The state value function is estimated by using the generalized advantage estimation method. The generalized advantage estimation uses the pre-trained value network V ψ The estimated advantage value is shown in formula (3):

[0021]

[0022] in, represents the advantage estimate from the current time step t, l represents the number of steps along a given trajectory, γ represents the discount factor, represents the pre-trained state value function, r t represents the reward function at time step t, λ∈[0,1] represents a hyperparameter used to control the trade-off between Monte Carlo estimation and temporal difference estimation;

[0023] The offline dataset is labeled by generalized advantage estimation to obtain a new dataset in, represents the advantage estimate at the tth time step of the kth trajectory obtained by GAE.

[0024] Furthermore, the predictor is specifically:

[0025] The trained predictor is used to evaluate the target advantage value of each state. The predictor consists of a 3-layer MLP and is trained with the expected regression loss. The loss function is shown in formula (4):

[0026]

[0027] Among them, (AE t ,s t )~D AE Indicates (AE t ,s t ) is from D AE Uniformly sampled, σ2 is a hyperparameter, when σ2→1, the predictor will learn the maximum in-sample advantage value given the state.

[0028] Furthermore, the modal decomposition module is specifically:

[0029] For new dataset The trajectory information is sequence modeled, the state, action and advantage value in the trajectory are regarded as information of different modes, and the trajectory information is modally decomposed, as shown in formula (5):

[0030]

[0031] The state-action and advantage value are separated, and only the state information S and action information A are sent to the encoder for interactive learning of higher-level representations of different modal information, while the advantage value AE is directly sent to the decoder to predict the action.

[0032] Furthermore, the encoder is composed of intra-modality interaction based on ConvFormer and inter-modality interaction based on TransFormer. First, the state and action are subjected to intra-modality interaction based on ConvFormer respectively, and then the state and action are subjected to inter-modality interaction based on TransFormer.

[0033] Furthermore, the ConvFormer-based intra-modal interaction is composed of a convolutional layer and a feed-forward neural layer;

[0034] The convolution layer is composed of a normalization layer and a convolution block. The convolution block is composed of two layers of convolution. Each convolution layer is composed of a 1D convolution and a Rlue activation function. The state information S is normalized by the normalization layer. The sequence X obtained by the normalization layer is sent to the convolution block. The convolution block performs a convolution operation on each column of the time series of the sequence X to capture the inherent local correlation in the sequence. The 1D convolution uses a filter with a length of L to perform a one-dimensional convolution operation on the sequence X, as shown in formula (6):

[0035]

[0036] Where d represents the hidden dimension of sequence X, N represents the length of sequence X, and w q [n] represents the filter coefficient of the qth hidden dimension, n = 0, 1, ..., L-1, and p represents the index of the length of the sequence X;

[0037] At the same time, the input state information S is residually connected to obtain Z, as shown in formula (7):

[0038] Z=Conv(X)+S (7)

[0039] Where Conv(X) represents the output of the convolutional block;

[0040] The feedforward neural layer is composed of a normalized layer and a feedforward neural network, and the feedforward neural network is composed of two layers of MLP. Z obtains the state representation H through the feedforward neural layer. S , as shown in formula (8):

[0041] H S =FFN(LN(Z))+Z (8)

[0042] The action information A is input into the ConvFormer-based intra-modal interaction in the same way to obtain the action representation H A .

[0043] Furthermore, the TransFormer-based inter-modal interaction consists of an embedding module, an attention layer, and a feed-forward neural layer;

[0044] The embedding module consists of an embedding layer and a position encoding. and action representation Interweave into a sequence The sequence H is obtained through the embedding layer and position encoding As shown in formula (9):

[0045]

[0046] in, represents the output of the embedding module, Embed represents the embedding layer, PE represents the position encoding, and the position encoding adopts the sinusoidal position encoding method, as shown in formula (10):

[0047]

[0048] Among them, pos represents the position index, i represents the dimension index, and d represents the dimension of the embedded vector;

[0049] The attention layer consists of a normalization layer and multi-head attention. Processing sequence The correlation between any two positions in the multi-head attention pair is modeled as follows:

[0050] will sequence Perform a linear transformation to obtain the Q, K and V vectors at each position, as shown in formula (11):

[0051]

[0052] Among them, f q 、f k 、f v A learnable parameter matrix representing queries, keys, and values;

[0053] The Q, K and V vectors of each position are sent to a multi-head attention with M heads for scaled dot product attention calculation. Each head calculates the attention score based on the Q and K vectors to obtain the similarity between different positions. The attention scores are normalized by Softmax to obtain the attention weights of different positions. The V vectors are weighted and summed using the attention weights to obtain the attention values ​​of the M heads. The obtained attention values ​​of the M heads are concatenated, as shown in formula (12):

[0054] MultiHead(Q,K,V)=Concat(head1,…,head n )W o ,

[0055] where head i =Attention(QW i Q ,KW i K ,VW i V ),

[0056]

[0057] Among them, head i ∈R N*C / n represents the output of the i-th attention head, W i Q , W i K , W i V ∈R C*C / h represents the corresponding input projection weight, W 0 ∈R C*C Represents the weight matrix, which is used to aggregate the outputs of all heads, C is the channel dimension, N is the length of the sequence, d∈R C / M Represents the dimension of the vector;

[0058] At the same time, a causal mask is introduced when calculating the attention value, and a residual connection is performed with the input sequence, as shown in formula (13):

[0059]

[0060] Among them, Atten() represents the attention layer;

[0061] The feedforward neural layer is composed of a normalized layer and a feedforward neural network, and the feedforward neural network is composed of two layers of MLP, and finally the sequence representation is obtained. As shown in formula (14):

[0062]

[0063] Furthermore, the decoder is composed of a convolutional layer, a cross-attention module, a feed-forward neural layer, and a linear layer;

[0064] The convolutional layer is composed of an embedding module and a convolutional module. The embedding module is composed of an embedding layer and a position encoding. The convolutional module is composed of a normalization layer and a convolutional block. The advantage value AE is input into the convolutional layer to obtain As shown in formula (15):

[0065]

[0066] Among them, Embed represents the embedding layer, PE represents the position encoding, LN represents the normalization layer, and Conv represents the convolution block;

[0067] The cross attention module includes two groups of normalization layers and multi-head attention, which converts the output of the encoder into Decomposition into state representation and action representation They are sent to the normalization layer and multi-head attention of the two groups respectively, and the advantage value To interact, in multi-head attention, and will be used as K, V, and The multi-head attention calculation for the state and action is the same as that in the TransFormer-based inter-modal interaction, as shown in formula (16):

[0068]

[0069] For the obtained Z A and Z s Sum and send to the feedforward neural layer to get The feedforward neural layer consists of a normalized layer and a feedforward neural network, and the feedforward neural network consists of two layers of MLP. Input to the linear layer to get the predicted action As shown in formulas (17) and (18):

[0070]

[0071] Furthermore, the method also includes training of an offline reinforcement learning model, specifically:

[0072] The value network and the predictor are updated with gradients, and the dataset is relabeled based on the value network using generalized advantage estimation. The encoder in the modal decomposition interaction is pre-trained using the masked language modeling technique, and then the decoder is fine-tuned based on the action reconstruction loss, as shown in formula (19):

[0073]

[0074] in, Represents the data set D AE The calculation of the expected value on the trajectory τ sampled in , a t represents the actual action at time step t, DCDT represents the modal decomposition interaction module, θ is the parameter of the modal decomposition interaction module, and a t-1 represents the actual action at time step t-1, s t and AE t Represent the state and advantage value at time step t respectively.

[0075] Compared with the prior art, the present invention has the following advantages:

[0076] (1) The present invention mainly includes a dominance estimation module and a modal decomposition interaction module to improve the trajectory splicing capability and fully extract the interaction between multimodal information in the trajectory.

[0077] (2) In the dominance estimation module, generalized dominance estimation is used to estimate the dominance value of the state, and the dominance value is used to relabel the dataset to enhance the trajectory splicing ability of the model.

[0078] (3) The modality decomposition interaction module consists of an encoder and a decoder. In the encoder, ConvFormer-based intra-modality interaction and TransFormer-based inter-modality interaction are used to fully extract the intra-modality and inter-modality interactions of states and actions, thereby achieving comprehensive learning of high-level representations of states and actions. In the decoder, convolution and cross-attention are further used to integrate the advantage value and state-action multimodal information, thereby improving the accuracy of action prediction.

[0079] (4) The proposed method is compared with eight other offline reinforcement learning methods on the D4RL dataset. The results show that the proposed method performs well in improving the agent's decision-making ability and achieves significant improvements and progress in offline tasks. These results show that the proposed method not only outperforms other methods in performance, but also has higher effectiveness in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the modal decomposition module;

[0081] Figure 2 A schematic diagram of intra-modal interaction based on ConvFormer; (Z in the figure 1 Need to be changed to Z)

[0082] Figure 3 This is a schematic diagram of the interaction between modalities based on TransFormer; (the embedding layer in the figure needs to be changed to the embedding module, and X is changed to )

[0083] Figure 4 is a schematic diagram of a decoder;

[0084] Figure 5 This is the evaluation curve diagram of ablation experiment 1;

[0085] Figure 6 This is the evaluation curve diagram of ablation experiment 2. DETAILED DESCRIPTION

[0086] In order to further illustrate the technical solution of the present invention, the present invention is further described below through embodiments.

[0087] An offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction in this embodiment includes the following steps:

[0088] Step 1: Collect trajectory data and build an offline dataset;

[0089] Step 2: Input the offline data set into a pre-trained offline reinforcement learning model to output the predicted action. The offline reinforcement learning model includes an advantage estimation module and a modal decomposition interaction module.

[0090] The advantage estimation module consists of three parts: value function approximation, generalized advantage estimation and predictor. The value function approximation approximates the value function of the offline data set by an approximate dynamic programming method. The generalized advantage estimation estimates the advantage of the state in the offline data set based on the approximate value function to re-label the offline data set. The predictor evaluates the target advantage for each state.

[0091] 1. Value function approximation

[0092] By parameterizing the value network V ψ To approximate the state value function to evaluate the long-term accumulated reward or return expectation in a specific state, and to update and optimize the value network V ψ To obtain the state value estimate, in order to overcome the overestimation problem, this embodiment uses three layers of MLP to construct independent value networks V ψ , used to evaluate the value of the state and introduce the corresponding target network for each state

[0093] In the value network V ψ In the training of With the expected regression loss L expectile_regression Value Network V ψ Update the parameters and target network The parameters of are updated in the form of moving average, as shown in formulas (1) to (2):

[0094]

[0095] Among them, γ represents the discount factor, s' represents the next state after the state s is transferred, and ψ represents the value network V ψ Parameters, Indicates the target network at the kth step Parameters, k represents the index of the time step, α represents the smoothing coefficient, and The function represents the expected regression loss, where Represents the indicator function, and σ1 represents the hyperparameter. By adjusting the value of σ1, a balance point is found between the approximate policy value function and the optimal value function in the sample, that is, while retaining the information of the optimal value function in the sample, it can be compensated by the approximate policy value function, so that it can obtain better performance and higher convergence speed during the training process. Once converged, the parameter ψ of the value function is frozen and no further update is made to ensure that the value function estimate obtained by training is maintained.

[0096] 2. Generalized Dominance Estimation

[0097] In order to avoid model bias caused by considering only a single estimate and to improve the accuracy of predicting future returns, this embodiment uses the generalized advantage estimation (GAE) method to estimate the state value function. The generalized advantage estimation uses a pre-trained value network V ψ The estimated advantage value is shown in formula (3):

[0098]

[0099] in, represents the advantage estimate from the current time step t, l represents the number of steps along a given trajectory, γ represents the discount factor, represents the pre-trained state value function, r t represents the reward function at time step t, λ∈[0,1] represents a hyperparameter used to control the trade-off between Monte Carlo estimation and temporal difference (TD) estimation;

[0100] The offline dataset is labeled by generalized advantage estimation to obtain a new dataset in, represents the advantage estimate at the tth time step of the kth trajectory obtained by GAE.

[0101] 3. Predictor

[0102] This embodiment uses a trained predictor to evaluate the target advantage value of each state. The predictor consists of a three-layer MLP and is trained with the expected regression loss. The loss function is shown in formula (4):

[0103]

[0104] Among them, (AE t ,s t )~D AE Indicates (AE t ,s t ) is from D AE Uniformly sampled, σ2 is a hyperparameter, when σ2→1, the predictor will learn the maximum in-sample advantage value given the state.

[0105] The modal decomposition interaction module consists of a modal decomposition module, an encoder, and a decoder. The modal decomposition module decomposes the information in the trajectory sequence and splits it into information of different modalities. The encoder performs intra-modal interaction based on ConvFormer and inter-modal interaction based on TransFormer on the state and action information. The intra-modal interaction based on ConvFormer combined with TransFormer and convolution can fully capture the inherent associations within a single modality and improve the representation ability of single modal information; the inter-modal interaction based on TransFormer uses causal TransFormer to capture the correlation between state and action information to fully learn the high-level representation of state and action. The decoder combines convolution and cross attention to fully integrate multi-modal information to predict actions.

[0106] 1. Modal decomposition module

[0107] For new data sets To help the agent better integrate and utilize the information in the trajectory, the state, action and advantage value in the trajectory are regarded as information of different modes, and the trajectory information is modally decomposed, as shown in formula (5):

[0108]

[0109] In order to keep the model more adaptable when the task or strategy changes, and to better pre-train the encoder to improve the overall performance, the state-action and advantage value are separated. Only the state information S and action information A are sent to the encoder for interactive learning of different modal information and its higher-level representation, while the advantage value AE is directly sent to the decoder to predict the action, such as Figure 1 The modal decomposition module is shown in Figure 2.

[0110] 2. Encoder

[0111] The encoder consists of intra-modal interaction based on ConvFormer and inter-modal interaction based on TransFormer. First, the state and action are subjected to intra-modal interaction based on ConvFormer respectively to enhance the representation of the state and action. Then, the state and action are subjected to inter-modal interaction based on TransFormer to fully capture the interactive relationship between the state and action, thereby representing the data more comprehensively.

[0112] (1) Intra-modal interaction based on ConvFormer

[0113] The ConvFormer structure is used in intra-modal interaction, which is different from the decision TransFormer structure. The convolution module is used in the token mixer instead of the attention module, because the convolution module can integrate the timing information between adjacent tokens, which is more helpful to strengthen the feature information of the single modality. Different from the decision ConvFomer, in order to help the intelligent agent better and more accurately understand the single modal information of state and action, the present invention sends the state and action to the intra-modal interaction module based on ConvFormer to capture the unique information in its embedding.

[0114] The internal structure of the intra-modal interaction based on ConvFormer is as follows Figure 2As shown, it is composed of a convolution layer and a feedforward neural layer. The convolution layer is composed of a normalization layer (Norm) and a convolution block (Convolution Block). The convolution block is composed of two layers of convolution. Each convolution layer is composed of a 1D convolution and a Rlue activation function. The state information S is normalized by the normalization layer. The sequence X obtained by the normalization layer is sent to the convolution block. The convolution block performs a convolution operation on each column of the sequence X to capture the inherent local correlation in the sequence. The 1D convolution uses a filter with a length of L to perform a one-dimensional convolution operation on the sequence X, as shown in formula (6):

[0115]

[0116] Where d represents the hidden dimension of sequence X, N represents the length of sequence X, and w q [n] represents the filter coefficient of the qth hidden dimension, n = 0, 1, ..., L-1, and p represents the index of the length of the sequence X;

[0117] At the same time, in order to avoid gradient disappearance, the input state information S is residually connected to obtain Z, as shown in formula (7):

[0118] Z=Conv(X)+S (7)

[0119] Where Conv(X) represents the output of the convolutional block;

[0120] The feedforward neural layer consists of a normalization layer (Norm) and a feedforward neural network (Feed Forward). The feedforward neural network consists of two layers of MLP. Z obtains the state representation H through the feedforward neural layer. S , as shown in formula (8):

[0121] H S =FFN(LN(Z))+Z (8)

[0122] The action information A is input into the ConvFormer-based intra-modal interaction in the same way to accurately capture its local information and obtain the action representation H A .

[0123] (2) Intermodal interaction based on TransFormer

[0124] The TransFormer-based inter-modal interaction is used to interweave state and action representations into a sequence, and its attention mechanism is used to effectively utilize information at different positions in the sequence. Cross-modal information interaction and fusion are performed based on the correlation between them, helping the model to better understand the relationship between state and action, thereby improving the accuracy of prediction.

[0125] The inter-modal interaction based on TransFormer consists of an embedding module, an attention layer, and a feed-forward neural layer, such as Figure 3 The embedding module consists of an embedding layer (Embedding) and a positional encoding (Positional Encoding), which represents the state and action representation Interweave into a sequence The sequence H is obtained through the embedding layer and position encoding As shown in formula (9):

[0126]

[0127] in, represents the output of the embedding module, Embed represents the embedding layer, and PE represents the position encoding. In order to avoid overfitting during position encoding, the position encoding adopts the sinusoidal position encoding method, as shown in formula (10):

[0128]

[0129] Among them, pos represents the position index, i represents the dimension index, and d represents the dimension of the embedded vector;

[0130] The attention layer consists of a normalization layer (Norm) and a multi-head attention layer (Multi-HeadAttention). Processing sequence The multi-head attention model is used to model the correlation between any two positions in order to achieve the interaction of different modal information in the sequence, specifically:

[0131] will sequence Perform a linear transformation to obtain the Q, K and V vectors at each position, as shown in formula (11):

[0132]

[0133] Among them, f q 、f k 、f v A learnable parameter matrix representing queries, keys, and values;

[0134] The Q, K and V vectors of each position are sent to a multi-head attention with M heads for scaled dot-product attention calculation. Specifically, each head calculates the attention score based on the Q and K vectors to obtain the similarity between different positions. The attention scores are normalized by Softmax to obtain the attention weights of different positions. The V vectors are weighted and summed using the attention weights to obtain the attention values ​​of the M heads, so as to capture the correlation between each position in the sequence, thereby effectively realizing the interaction and fusion of different modal information in the sequence. The obtained attention values ​​of the M heads are spliced, as shown in formula (12):

[0135] MultiHead(Q,K,V)=Concat(head1,…,head n )W o ,

[0136] where head i =Attention(QW i Q ,KW i K ,VW i V ),

[0137]

[0138] Among them, head i ∈R N*C / n represents the output of the i-th attention head, W i Q , W i K , W i V ∈R C*C / h represents the corresponding input projection weight, W 0 ∈R C*C Represents the weight matrix, which is used to aggregate the outputs of all heads, C is the channel dimension, N is the length of the sequence, d∈R C / M Represents the dimension of the vector;

[0139] At the same time, a causal mask is introduced when calculating the attention value to ensure that the model does not rely on future information during generation, and a residual connection is made with the input sequence to avoid gradient disappearance, as shown in formula (13):

[0140]

[0141] Among them, Atten() represents the attention layer;

[0142] The feedforward neural layer uses the same architecture based on the intra-modal interaction of ConvFormer, which consists of a normalization layer and a feedforward neural network. The feedforward neural network consists of two layers of MLP, and finally obtains the sequence representation As shown in formula (14):

[0143]

[0144] 3. Decoder

[0145] The decoder uses convolution and cross attention to perform modal interaction between the advantage value and the state and action to improve the accuracy of action prediction. The present invention uses a convolution module to extract unimodal information from the advantage value AE to help the model better understand the advantage information. At the same time, cross attention is used to capture the association between the advantage value and the state and action high-level representation obtained by the encoder, making better use of multimodal information, thereby providing more accurate decisions.

[0146] The decoder consists of convolutional layers, cross-attention modules, feed-forward neural layers, and linear layers, such as Figure 4 As shown in the figure, the convolutional layer consists of an embedding module and a convolutional module. The embedding module has the same structure as the embedding module in the inter-modal interaction based on TransFomer, which consists of an embedding layer and a position encoding. The convolutional module adopts the structure of the convolutional layer in the intra-modal interaction based on ConvFomer, which consists of a normalization layer and a convolutional block. The advantage value AE is input into the convolutional layer to obtain As shown in formula (15):

[0147]

[0148] Among them, Embed represents the embedding layer, PE represents the position encoding, LN represents the normalization layer, and Conv represents the convolution block;

[0149] The cross attention module includes two groups of normalization layers and multi-head attention, which transforms the encoder output Decomposition into state representation and action representation They are sent to the two groups' normalization layer (Norm) and multi-head attention (Multi-HeadAttention), and the advantage value To interact, in multi-head attention, and will be used as K, V, and The multi-head attention calculation for the state and action is the same as that in the TransFormer-based inter-modal interaction, as shown in formula (16):

[0150]

[0151] For the obtained Z A and Z s Sum and send to the feedforward neural layer to get The feedforward neural layer uses the same structure as that in ConvFormer, which consists of a normalization layer and a feedforward neural network. The feedforward neural network consists of two layers of MLP. Input to the linear layer to get the predicted action As shown in formulas (17) and (18):

[0152]

[0153] In addition, the complete algorithm of the offline reinforcement learning method combining generalized advantage estimation and modal decomposition interaction of the present invention is shown in Algorithm 1, which is mainly divided into a training phase and a testing phase.

[0154] In the training phase, the value network and the predictor are gradient updated, and the dataset is relabeled based on the value network using generalized advantage estimation. The encoder in the modal decomposition interaction is pre-trained using the masked language modeling (MLM) technique, and then the decoder is fine-tuned based on the action reconstruction loss, as shown in formula (19):

[0155]

[0156] in, Represents the data set D AE The calculation of the expected value on the trajectory τ sampled in , a t represents the actual action at time step t, DCDT represents the modal decomposition interaction module, θ is the parameter of the modal decomposition interaction module, and a t-1 represents the actual action at time step t-1, s t and AE t Represent the state and advantage value at time step t respectively.

[0157] During the test phase, the predictor estimates the advantage of the current state. Then, the advantage estimate, historical information, and current state are input into the model to predict the next action. The predicted action will be used for simulation execution, and the simulation environment will transfer the state according to the predicted action, and then estimate the advantage of the state, and repeat the cycle until termination.

[0158] Algorithm 1:

[0159]

[0160]

[0161] Example 2

[0162] In order to evaluate the performance of the offline reinforcement learning method combining generalized advantage estimation and modal decomposition interaction, a large number of experiments were conducted in this embodiment. The main objectives of the experiments include: (1) testing with the offline RL dataset D4RL as a benchmark and comparing with other different offline reinforcement learning methods; (2) setting up two groups of ablation experiments to evaluate the impact of the main structure of the present invention on the overall performance.

[0163] Dataset: This embodiment evaluates the performance of the method through three different continuous control tasks: hopper, halfcheetach and walk2d in the MuJoCo simulation environment. For each different task, the present invention uses three different levels of historical datasets, namely: (1) medium dataset, which is mainly composed of the states and actions collected by the agent in previous experiments. (2) medium-expert dataset, which is mainly composed of experienced expert-level data, including high-quality state and action data of human experts. (3) medium-replay dataset, which is mainly composed of historical data recorded when the agent performs tasks in the simulation environment.

[0164] Comparison method: This embodiment selects eight different offline reinforcement learning methods for comparison with this method, mainly including: (1) offline reinforcement learning (IQL) based on implicit Q learning; (2) conservative Q learning (CQL) for offline reinforcement learning; (3) decision TransFormer (DT); (4) trajectory TransFormer (TT); (5) decision converter (DC); (6) decision sensor (DTd); (7) RvS; (8) ACT, and this method is recorded as GCM.

[0165] 1. Comparative Experiment

[0166] Table 1 Comparative experimental results on the MoJoCo motion benchmark

[0167]

[0168] The experimental results of the GCM of this method are the average and standard deviation of running 3 independent models and evaluating 10 fragments.

[0169] This experiment selected 8 different tasks of MuJoCo for testing on the benchmark of the D4RL dataset, and used the standardized score of the MuJoCo task as the final test result. At the same time, this method was compared with nine other offline reinforcement learning methods. The results of the comparative experiment are shown in Table 1, where the highest score font in Table 1 is bold. According to the results in Table 1, the present invention performed well in 5 tasks and showed superiority in offline tasks. This further proves the effective extraction ability of the present invention for intra-modal and inter-modal information in the trajectory and its ability to make the best decisions in different tasks. At the same time, compared with other methods, the present invention has improved the accuracy of the agent's prediction of actions. Although the standardized scores in the four tasks of hopper-medium, hopper-medium-replay, walker2d-medium-expert and walker2d-medium-replay are not the highest, the highest score is obtained in the average score of the eight tasks, showing that the method has certain stability and advantages in various offline tasks. These results further highlight the effectiveness of the present invention for intra-modal and inter-modal interactions and the full utilization of multimodal information. By extracting multiple modal information from the trajectory and comprehensively considering the interactions within the modalities and the complementarity between the modalities, the method proposed in the present invention can optimize the decision-making ability of the intelligent agent and improve its performance in various tasks.

[0170] 2. Ablation Experiment

[0171] In order to fully verify the effectiveness of the present invention in offline tasks, two groups of ablation experiments are set up on the MoJoCo motion benchmark to explain the effectiveness of the advantage value adopted by the present invention and the structure of the present invention.

[0172] Experiment 1: This experiment mainly conducts ablation experiments on the generalized advantage estimation used in the present invention. In order to prove that the increase of advantage value can effectively improve the model's trajectory splicing ability in offline data sets, the present invention conducts ablation experiments in the tasks of hopper, halfcheetach and walk2d without using advantage value, that is, using RTG value for training. The initial RTG value settings in different tasks are shown in Table 2.

[0173] The results of ablation experiment 1 are shown in Table 3. The present invention obtains the optimal value, which proves that the present invention effectively improves the overall performance of the model by using the advantage value. At the same time, the present invention draws the evaluation curves of the three tasks of hopper-medium, walker2d-medium-expert and halfcheetach-medium-replay in the first 300K, as shown in Figure 5As shown, in the present invention, when the advantage value is not used, the performance of the model is slightly reduced compared to when the advantage value is used, which proves that in the offline data set, if only suboptimal trajectories are included, the trajectory splicing ability of the model will be affected. By adopting the advantage value, the model can evaluate the value of each state-action pair and judge its superiority relative to other options. This is particularly important for splicing discontinuous trajectories. In the case of only suboptimal trajectories, the model may make wrong decisions during the splicing process. However, by using the advantage value, the model can better evaluate the value difference of each state-action pair, thereby more accurately performing trajectory splicing. Therefore, the experimental results of the present invention show that the use of advantage values ​​can improve the trajectory splicing ability of the model in the offline data set and improve the accuracy of the agent prediction. This is of great significance for improving the performance of the model and the decision-making ability in practical applications.

[0174] Table 2. Initial RTG value settings in MuJoCo task for ablation experiment 1

[0175]

[0176] Table 3 Results of ablation experiment 1

[0177] Datatset halfcheetah hopper walker2d No-AE 59.7 85.8 83.3 GCM 64.6 91.9 89.2

[0178] The results are the average of the standardized scores of the hopper, halfcheetach and walker2d tasks on three different levels of datasets.

[0179] Experiment 2: To demonstrate the importance of the ConvFormer structure in the modal decomposition module of the present invention in performing intra-modal and inter-modal information interaction when extracting trajectories, this experiment ablates the structure of the modal decomposition interaction of ConvFormer. In this experiment, a total of 5 ablation experiments are set up: (a) Replace the ConvFormer-based intra-modal interaction of the encoder with a pure convolutional module. (b) Replace the ConvFormer-based intra-modal interaction module and the convolutional module of the decoder with an attention structure. (c) Only remove the ConvFormer-based intra-modal interaction module of the encoder. (d) Only remove the TransFormer-based inter-modal interaction module of the encoder. (e) Only replace the cross-attention module of the decoder with an attention module.

[0180] The experimental results are shown in Table 4. The present invention achieves the best value, proving its effectiveness in information extraction and decision optimization. This is of great significance for improving the performance of intelligent agents and the decision-making ability of practical applications. Compared with the ablation experiment, the present invention performs better in the evaluation curve, such as Figure 6As shown, by ablating the intra-modal interaction based on ConvFormer, the inter-modal interaction based on TransFormer, and the cross module of the decoder, it is shown that the method of capturing multi-modal information interaction of the present invention is very critical to the decision-making process of the intelligent agent. Information in different modalities is often complementary and relevant. By comprehensively utilizing them, a more comprehensive and accurate basis for decision-making can be provided. Therefore, the results of the present invention achieving the optimal value in each task prove that it can fully extract the information in the trajectory, thereby effectively optimizing the decision-making ability of the intelligent agent.

[0181] Table 4 Results of ablation experiment 2

[0182] Datatset halfcheetah hopper walker2d a 48.2 70.3 67.5 b 61.5 88.0 84.0 c 53.1 63.9 80.2 d 48.6 77.3 70.6 e 62.2 90.1 82.3 GCM 64.6 91.9 89.2

[0183] The results are the average of the standardized scores of the hopper, halfcheetach and walker2d tasks on three different levels of datasets.

[0184] The above shows and describes the main features and advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0185] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An offline reinforcement learning method combining generalized advantage estimation and modal decomposition interaction, characterized in that: The following steps are involved: Step 1: Collect trajectory data and build an offline dataset; Step 2, inputting the offline data set into a pre-trained offline reinforcement learning model and outputting the predicted action, wherein the offline reinforcement learning model includes a dominance estimation module and a modal decomposition interaction module; The advantage estimation module consists of three parts: value function approximation, generalized advantage estimation and predictor. The value function approximation approximates the value function of the offline data set by an approximate dynamic programming method. The generalized advantage estimation estimates the advantage of the state in the offline data set based on the approximate value function to re-label the offline data set. The predictor evaluates the target advantage for each state. The modal decomposition interaction module consists of a modal decomposition module, an encoder and a decoder. The modal decomposition module decomposes the information in the trajectory sequence and splits it into information of different modalities. The encoder performs intra-modal interaction based on ConvFormer and inter-modal interaction based on TransFormer on state and action information. The decoder combines convolution and cross-attention to fully fuse multi-modal information to predict actions.

2. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 1, characterized in that: The value function approximation is specifically: By parameterizing the value network V ψ To approximate the state value function to evaluate the long-term accumulated reward or return expectation in a specific state, and to update and optimize the value network V ψ To obtain the state value estimate, a three-layer MLP is used to construct independent value networks V ψ , and introduce the corresponding target network for each state In the value network V ψ In the training of With the expected regression loss L expectile_regression Value Network V ψ Update the parameters and target network The parameters of are updated in the form of moving average, as shown in formulas (1) to (2): Among them, γ represents the discount factor, s' represents the next state after the state s is transferred, and ψ represents the value network V ψ Parameters, Indicates the target network at the kth step Parameters, k represents the index of the time step, α represents the smoothing coefficient, and The function represents the expected regression loss, where represents the indicator function, and σ1 represents the hyperparameter.

3. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 2, characterized in that: The generalized advantage estimate is specifically: The state value function is estimated by using the generalized advantage estimation method. The generalized advantage estimation uses the pre-trained value network V ψ The estimated advantage value is shown in formula (3): in, represents the advantage estimate from the current time step t, l represents the number of steps along a given trajectory, γ represents the discount factor, represents the pre-trained state value function, r t represents the reward function at time step t, λ∈[0,1] represents a hyperparameter used to control the trade-off between Monte Carlo estimation and temporal difference estimation; The offline dataset is labeled by generalized advantage estimation to obtain a new dataset in, represents the advantage estimate at the tth time step of the kth trajectory obtained by GAE.

4. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 3, characterized in that: The predictor is specifically: The trained predictor is used to evaluate the target advantage value of each state. The predictor consists of a 3-layer MLP and is trained with the expected regression loss. The loss function is shown in formula (4): Among them, (AE t ,s t )~D AE Indicates (AE t ,s t ) is from D AE Uniformly sampled, σ2 is a hyperparameter, when σ2→1, the predictor will learn the maximum in-sample advantage value given the state.

5. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 4, characterized in that: The modal decomposition module is specifically: For new data sets The trajectory information is sequence modeled, the state, action and advantage value in the trajectory are regarded as information of different modes, and the trajectory information is modally decomposed, as shown in formula (5): The state-action and advantage value are separated, and only the state information S and action information A are sent to the encoder for interactive learning of higher-level representations of different modal information, while the advantage value AE is directly sent to the decoder to predict the action.

6. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 5, characterized in that: The encoder is composed of intra-modality interaction based on ConvFormer and inter-modality interaction based on TransFormer. First, the state and action are subjected to intra-modality interaction based on ConvFormer respectively, and then the state and action are subjected to inter-modality interaction based on TransFormer.

7. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 6, characterized in that: The ConvFormer-based intra-modal interaction consists of a convolutional layer and a feed-forward neural layer; The convolution layer is composed of a normalization layer and a convolution block. The convolution block is composed of two layers of convolution. Each convolution layer is composed of a 1D convolution and a Rlue activation function. The state information S is normalized by the normalization layer. The sequence X obtained by the normalization layer is sent to the convolution block. The convolution block performs a convolution operation on each column of the time series of the sequence X to capture the inherent local correlation in the sequence. The 1D convolution uses a filter with a length of L to perform a one-dimensional convolution operation on the sequence X, as shown in formula (6): Where d represents the hidden dimension of sequence X, N represents the length of sequence X, and w q [n] represents the filter coefficient of the qth hidden dimension, n = 0, 1, ..., L-1, and p represents the index of the length of the sequence X; At the same time, the input state information S is residually connected to obtain Z, as shown in formula (7): Z=Conv(X)+S(7) Where Conv(X) represents the output of the convolutional block; The feedforward neural layer is composed of a normalized layer and a feedforward neural network, and the feedforward neural network is composed of two layers of MLP. Z obtains the state representation H through the feedforward neural layer. S , as shown in formula (8): H S =FFN(LN(Z))+Z(8) The action information A is input into the ConvFormer-based intra-modal interaction in the same way to obtain the action representation H A .

8. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 7, characterized in that: The TransFormer-based inter-modal interaction consists of an embedding module, an attention layer, and a feed-forward neural layer; The embedding module consists of an embedding layer and a position encoding. and action representation Interweave into a sequence The sequence H is obtained through the embedding layer and position encoding As shown in formula (9): in, represents the output of the embedding module, Embed represents the embedding layer, PE represents the position encoding, and the position encoding adopts the sinusoidal position encoding method, as shown in formula (10): Among them, pos represents the position index, i represents the dimension index, and d represents the dimension of the embedded vector; The attention layer consists of a normalization layer and a multi-head attention layer. The normalization layer processes H to obtain the sequence The correlation between any two positions in the multi-head attention pair is modeled as follows: will sequence Perform a linear transformation to obtain the Q, K and V vectors at each position, as shown in formula (11): Among them, f q 、f k 、f v A learnable parameter matrix representing queries, keys, and values; The Q, K and V vectors of each position are sent to a multi-head attention with M heads for scaled dot product attention calculation. Each head calculates the attention score based on the Q and K vectors to obtain the similarity between different positions. The attention scores are normalized by Softmax to obtain the attention weights of different positions. The V vectors are weighted and summed using the attention weights to obtain the attention values ​​of the M heads. The obtained attention values ​​of the M heads are concatenated, as shown in formula (12): MultiHead(Q,K,V)=Concat(head1,…,head n )W o , wherehead i =Attention(QW i Q ,KW i K ,VW i V ), Among them, head i ∈R N*C / n represents the output of the i-th attention head, W i Q , W i K , W i V ∈R C*C / h represents the corresponding input projection weight, W 0 ∈R C*C Represents the weight matrix, which is used to aggregate the outputs of all heads, C is the channel dimension, N is the length of the sequence, d∈R C / M Represents the dimension of the vector; At the same time, a causal mask is introduced when calculating the attention value, and a residual connection is performed with the input sequence, as shown in formula (13): Among them, Atten() represents the attention layer; The feedforward neural layer is composed of a normalized layer and a feedforward neural network, and the feedforward neural network is composed of two layers of MLP, and finally the sequence representation is obtained. As shown in formula (14):

9. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to claim 8, characterized in that: The decoder consists of a convolutional layer, a cross-attention module, a feed-forward neural layer, and a linear layer; The convolutional layer is composed of an embedding module and a convolutional module. The embedding module is composed of an embedding layer and a position encoding. The convolutional module is composed of a normalization layer and a convolutional block. The advantage value AE is input into the convolutional layer to obtain As shown in formula (15): Among them, Embed represents the embedding layer, PE represents the position encoding, LN represents the normalization layer, and Conv represents the convolution block; The cross attention module includes two groups of normalization layers and multi-head attention, which converts the output of the encoder into Decomposition into state representation and action representation They are sent to the normalization layer and multi-head attention of the two groups respectively, and the advantage value To interact, in multi-head attention, and will be used as K, V, and The multi-head attention calculation for the state and action is the same as that in the TransFormer-based inter-modal interaction, as shown in formula (16): For the obtained Z A and Z s Sum and send to the feedforward neural layer to get The feedforward neural layer consists of a normalized layer and a feedforward neural network, and the feedforward neural network consists of two layers of MLP. Input to the linear layer to get the predicted action As shown in formulas (17) and (18):

10. The offline reinforcement learning method combining generalized dominance estimation and modal decomposition interaction according to any one of claims 1 to 9, characterized in that: The method also includes training of an offline reinforcement learning model, specifically: The value network and the predictor are updated with gradients, and the dataset is relabeled based on the value network using generalized advantage estimation. The encoder in the modal decomposition interaction is pre-trained using the masked language modeling technique, and then the decoder is fine-tuned based on the action reconstruction loss, as shown in formula (19): in, Represents the data set D AE The calculation of the expected value on the trajectory τ sampled in , a t represents the actual action at time step t, DCDT represents the modal decomposition interaction module, θ is the parameter of the modal decomposition interaction module, and a t-1 represents the actual action at time step t-1, s t and AE t Represent the state and advantage value at time step t respectively.

Images