State feature optimization method based on hierarchical visual transformer model

Abstract

The application provides a state feature optimization method based on a hierarchical visual transformer model, and relates to the technical field of reinforcement learning. The method specifically comprises the following steps: creating an interactive environment and obtaining a picture dataset; constructing and initializing an intrinsic curiosity model DiNAT-ICM based on a hierarchical visual transformer in the interactive environment; iteratively training the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer by using the picture dataset, calculating the total reward and state feature variance of the DiNAT-ICM, and storing them into a data log storage file logger; reading the total reward and state feature variance in the data log storage file logger, calculating the average reward and average state feature variance of the DiNAT-ICM in a unit round, and then performing smoothing processing on the average reward and average state feature variance by using an exponential weighted average (EWMA) method, and finally performing visual display on the processed results to complete the state feature optimization. The method can improve the utilization efficiency of data samples and alleviate the problem of sparse rewards commonly existing in the exploration process of an intelligent agent in reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a method for optimizing state features based on a hierarchical visual transformer model. Background Technology

[0002] Feature selection in machine learning has been studied for a long time, with specific methods including minimum redundancy, maximum relevance, and heuristic algorithms. The purpose of feature selection is to reduce variance during model training by filtering out features with weak correlation. Currently, the most researched methods are based on deep learning-based relevance feature selection, the core idea of ​​which is to automatically learn representations or features from data using unsupervised or supervised learning methods. In recent years, many research methods based on deep reinforcement learning have emerged, mainly because reinforcement learning is one of the techniques for making optimal decisions. Reinforcement learning methods can be flexibly applied to any dataset and adjusted for each feature. However, current reinforcement learning (RL) methods require significant computational time and power to search a large state space.

[0003] Deep reinforcement learning has achieved remarkable success in video games since its inception, but its application to real-world decision-making and control tasks, such as navigation, robot manipulation, and robotic arm grasping, has not yielded the same impressive results. In reinforcement learning, agents learn through trial and error by interacting with the environment, selecting the optimal strategy in unknown environments to maximize cumulative rewards. During this exploratory learning process, sparse rewards from the external environment can lead to insufficient exploration. This sparse reward problem, characterized by uneven positive and negative rewards, is defined as the sparse reward problem, a core challenge faced by deep reinforcement learning in solving real-world tasks. Therefore, addressing the sparse reward problem is crucial for the exploratory learning process of agents in deep reinforcement learning.

[0004] The most pressing issue in sparse reward settings for deep reinforcement learning is improving the agent's efficiency in exploring the external environment and achieving efficient exploration decisions. Major research on sparse reward problems includes reward reshaping and design, exploration and utilization, multi-objective learning, and auxiliary tasks. Currently, intrinsic motivation-based methods are widely used in exploration and utilization, including state access counting, curiosity-driven exploration, refined random networks, set divergence, and state reachability of memory episodes. Among these, curiosity-driven exploration is a milestone in solving the sparse reward problem in deep reinforcement learning. In recent years, many research methods related to curiosity-driven exploration have emerged. These methods primarily construct intrinsic rewards by adding extra parameters, modifying the loss function, or changing the network structure, thereby incentivizing exploration and improving the agent's exploration performance. Compared to traditional random exploration methods, this approach is more inclined to explore unknown regions in the environment and can handle complex or sparse state spaces. Besides acquiring external reward signals based on the environment, curiosity-driven exploration methods can use the intrinsic reward information generated by the agent to encourage exploration of the state space, making it less prone to getting trapped in local optima. Oudeyer et al. first proposed Intelligent Adaptive Curiosity (IAC), which learns exploration by introducing a forward dynamics model, reducing prediction errors while calculating intrinsic reward signals. Pathak et al. proposed the Intrinsic Curiosity Module (ICM), which mainly consists of a forward dynamics model and a backward dynamics model. It uses the encoded state features as input to the forward dynamics model to predict the next state features. This method quantifies the prediction error of the agent's next state and action by introducing forward and backward dynamics models, using this error as an intrinsic reward signal to provide the agent with more reward signals during exploration, thus alleviating the dilemma of sparse rewards.

[0005] The Transformer model has achieved tremendous success in artificial intelligence fields such as natural language processing, computer vision, speech processing, and large-scale models, attracting widespread attention from both academia and industry. Attention mechanisms are widely used in Transformer models, including self-attention, multi-head self-attention, sparse attention, local attention, and global attention. Self-attention is the core of the Transformer model, and the introduction of multi-head self-attention has further improved its performance. However, due to its high complexity and slow training speed, local attention is often used to reduce the secondary complexity of self-attention, such as Stand-Alone Self-Attention (SASA) and Swing Window Self-Attention (SWAS). Hassani et al. first proposed the concept of Neighborhood Attention (NA), a scalable and efficient visual sliding window attention mechanism that focuses attention on the nearest surrounding pixels. Since neighborhood attention primarily focuses on a local pixel range, it suffers from two problems: reduced long-term dependence on global data information and neglect of the global receptive field. Therefore, Hassani et al. proposed Dilated Neighborhood Attention (DiNA) to address the aforementioned issues. DiNA is a natural and flexibly efficient extension of neighborhood attention, capable of exponentially expanding the receptive field without additional cost, thus successfully acquiring more global feature information. Finally, they combined Neighborhood Attention (NA) with DiNA to propose a Dilated Neighborhood Attention Transformer (DiNAT) model, which can acquire not only local information but also sparse global information.

[0006] A unified curiosity-driven approach has been proposed in existing technologies to address the exploration difficulties encountered in reinforcement learning under reward sparsity. This method primarily utilizes an attention mechanism to obtain reliable feature representations, uses state novelty to estimate the feature representations of the agent's current and next states, calculates the intrinsic reward information, and weights the exploration level estimates of three different samples in the state space to obtain the intrinsic reward before smoothing and fusion. However, this method suffers from drawbacks when the training data sample increases as the forward dynamics model interacts with new environments. Utilizing state novelty to estimate the intrinsic reward among three different samples leads to the loss of target features, increases the computational burden on deep networks, and reduces the efficiency of reward calculation.

[0007] In addition, existing technologies have proposed an automatic feature construction method based on attention mechanisms and reinforcement learning. This method first specifies a dataset for a fixed classification problem and uses a self-attention mechanism to score different types of feature combinations, i.e., the feature classification result. Secondly, it introduces a policy gradient-based method from reinforcement learning to evaluate the feature state, using meta-features from reinforcement learning as the data labels for this method. However, this method mainly uses the self-attention mechanism to classify and evaluate different types of feature groups, without considering the automatic construction of different feature weights for the same type of data. This method introduces the Monte Carlo Policy Gradient (REINFORCE) algorithm from reinforcement learning for automatic feature selection, using data combined with real label features as input to the reinforcement learning controller, and then defining the output reward value as the intrinsic reward and using it as the probability of feature selection. This leads to problems such as unstable gradient calculation and training, and large variance in the calculated reward. Summary of the Invention

[0008] To address the shortcomings of the existing technologies, this invention proposes a state feature optimization method based on a hierarchical visual transformer model, aiming to alleviate the dilemma of agents getting stuck in local optima during the exploration process, improve the utilization efficiency of limited data samples, and optimize the accuracy of state feature extraction.

[0009] This invention proposes a state feature optimization method based on a hierarchical visual transformer model, which includes the following steps:

[0010] Step 1: Create interactive environments (envs) to obtain the image dataset;

[0011] Step 2: Build and initialize the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer in the interactive environment envs;

[0012] Step 3: Iteratively train the DiNAT-ICM intrinsic curiosity model based on the hierarchical visual transformer using the image dataset, and calculate the total reward of DiNAT-ICM. total and state feature variance std And store it in the data log storage file logger;

[0013] Step 4: Read the total reward from the logger data storage file. total and state feature variance std Calculate the average reward of DiNAT-ICM within a unit round. and mean state characteristic variance Using the exponentially weighted average (EWMA) method to... and Smoothing is performed, and the processed results are visualized to complete the optimization of state features;

[0014] Step 1 further includes:

[0015] Step 1.1: Create several independently running parallel environments according to the set number of parallel environments num_envs, build an interactive environment using the created parallel environments, and set the number of random seeds in the interactive environment;

[0016] Step 1.2: Set the number of data frames n_stack and obtain an image dataset consisting of n_stack consecutive frames;

[0017] The intrinsic curiosity model DiNAT-ICM based on hierarchical visual transformer mentioned in step 2 includes: intrinsic curiosity network ICM_Network, scalable proximity attention Transformer model DiNAT_Network, and A2C algorithm network A2C_Network;

[0018] The intrinsic curiosity network ICM_Network is used to predict the intrinsic reward of the DiNAT-ICM intrinsic curiosity model based on hierarchical visual transformer, and includes the following components connected in sequence: Feature_Encoder, ForwardNet, and InverseNet.

[0019] The scalable neighbor attention Transformer model DiNAT_Network is used to optimize the intrinsic curiosity network ICM_Network, which includes the following connected in sequence: a first normalized layer LayerNorm1, a scalable sparse global attention DiNA, a second normalized layer LayerNorm2, a multilayer perceptron MLP, a linear layer Linear5, and a softmax function.

[0020] The A2C algorithm network, A2C_Network, is used to predict the discounted reward of the DiNAT-ICM intrinsic curiosity model based on hierarchical visual transformers. It includes the following connected networks in sequence: Self-AttentionNet, Actor Network, and Critic Network.

[0021] Step 2, which initializes the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer, includes setting the number of data frames n_stack, the number of parallel environments num_envs, the number of actions num_actions, the environment name envs_name, the input data size input_size, and the state feature size feature_size.

[0022] Step 3 further includes:

[0023] Step 3.1: Reset the interactive environment, obtain the current round state observation value obs of DiNAT-ICM in the interactive environment, and initialize the state observation value obs to obtain an N-dimensional array object Numpy of size [num_env, width, height, n_stack]. obs Where width is the image width and height is the image height;

[0024] Step 3.2: Convert the array object using NumPy obs Convert to a state information tensor states The state information tensor states The size is [num_env, n_stack, width, height];

[0025] Step 3.3: Based on the state information tensor states The intrinsic curiosity model DiNAT-ICM based on a hierarchical visual transformer was iteratively trained using the deep reinforcement learning A2C algorithm, and the discounted reward R of DiNAT-ICM was calculated. ext ;

[0026] Step 3.4: Clear the accumulated gradients in the DiNAT-ICM intrinsic curiosity model based on the hierarchical visual transformer and initialize them to 0;

[0027] Step 3.5: Calculate the state prediction error of DiNAT-ICM as the state feature variance of DiNAT-ICM. std and intrinsic reward R insic ;

[0028] Step 3.6: Utilize DiNAT-ICM's discount reward R ext and intrinsic reward R insic Calculate the total reward in the current iteration step. to t al ;

[0029] Step 3.7: Distribute the total reward. total and state feature variance std Store the data in the logger file and repeat steps 3.1-3.6 until the preset number of iterations is completed;

[0030] Step 3.3 further includes:

[0031] Step 3.3.1: Construct and initialize the experience recycling pool Buffer;

[0032] Step 3.3.2: Convert the state information tensor of DiNAT-ICM at the current iteration step. states As state information (states);

[0033] Step 3.3.3: Use the Feature_Encoder to encode and transform the state information states to obtain the state encoding feature, and map the state encoding into the feature space;

[0034] Step 3.3.4: Input the mapped state code into the attention network and use attention weights to filter the state information score features. score The feature for scoring state information score Softmax normalization is performed to obtain the final attention state feature. attn ;

[0035] Step 3.3.5: Extract the final attention state features. attn Input the actor network to obtain the decision π attnThen, softmax normalization is performed to obtain the current action probability distribution (Action), and the next action (action) of DiNAT-ICM is obtained by sampling using a random distribution. attn ;

[0036] Step 3.3.6: Encode the mapped state and the decision π attn Input the critique network and calculate the value V(feature, π) in the current state. attn );

[0037] Step 3.3.7: Calculate the policy loss, value function loss, and policy entropy. entropy And construct the total loss Utilizing total loss Calculate the current discount reward R of DiNAT-ICM ext and will offer discount rewards R ext State features attn action attn Action probability distribution and value V(feature,π) attn Stored in the experience recycling pool buffer;

[0038] Step 3.3.8: Reset the experience recycling pool Buffer, and repeat steps 3.3.2-3.3.7 to continue iteratively calculating the next round's step. next Discount Rewards R ext State features attn action attn Action probability distribution and value V(feature,π) attn This continues until the preset number of iterations is completed.

[0039] Step 3.5 further includes:

[0040] Step 3.5.1: Convert the state information tensor of DiNAT-ICM at the current iteration step. states As state information, `states` is encoded and transformed using the Feature Encoder to obtain a state-encoded feature. This state-encoded feature is then mapped onto the feature space to obtain the mapped state feature `feature0` and the corresponding state feature information `feature`. curt Then, slice the state feature 0 to obtain the next state feature information of DiNAT-ICM. next ;

[0041] Step 3.5.2: Obtain the state feature of round t in the current iteration step. t It also utilizes the scalable neighbor attention Transformer structure DiNAT_Network to analyze state features. t Normalization is performed, the resulting feature tensor is then increased in dimensionality, and Scalable Sparse Global Attention DiNA is used to perform residual calculation to obtain the state features after residual calculation.

[0042] Step 3.5.3: After normalizing and reducing the dimensionality of the state features calculated from the residuals, perform another softmax normalization to obtain the state features optimized by the DiNAT network. DiNAT ;

[0043] Step 3.5.4: Based on the state feature information of DiNAT-ICM curt DiNAT-ICM next state feature information next and optimized state features DiNAT The forward dynamics model ForwardNet is used to predict the next state feature encoding of DiNAT-ICM. Then, the InverseNet inverse dynamics model is used to calculate the predicted value of the current action.

[0044] Step 3.5.5: Encode the next state feature using DiNAT-ICM And the predicted value of the current action Calculate the state feature variance of DiNAT-ICM std and intrinsic reward R insi c);

[0045] The forward dynamics model and the inverse dynamics model are as follows:

[0046] feature DiNAT =DiNAT(feature) curt )

[0047]

[0048]

[0049] Where DiNAT() represents the extended proximity attention Transformer network structure; ForwardNet() represents the forward dynamics model in DiNAT-ICM; action attnThe InverseNet() method represents the inverse dynamics model in DiNAT-ICM, indicating the action obtained after optimizing the actor network through a self-attention mechanism.

[0050] Compared with existing technologies, the beneficial effects of adopting the above technical solution are as follows:

[0051] (1) The method of the present invention utilizes the expandable neighbor attention Transformer model to optimize the forward dynamics model in the intrinsic curiosity model in reinforcement learning, calculates the weight information of different state features, evaluates the state features that are more favorable to the agent's exploration and obtains better intrinsic reward signals, improves the utilization efficiency of data samples, and alleviates the sparse reward problem that is common in the exploration process of agents.

[0052] (2) The method of the present invention improves the Actor network in the Advantage Actor-Critic (A2C) algorithm by using the self-attention mechanism, extracts high-quality local state features, thereby obtaining a more favorable action space, reducing the search space for the next best action, avoiding the state space from being too sparse and improving the model performance.

[0053] (3) The method of the present invention uses the Exponential Weighted Moving Average (EWMA) method to smooth the data, thereby enhancing the interpretability of deep reinforcement learning. Attached Figure Description

[0054] Figure 1 This is a flowchart of a state feature optimization method based on a hierarchical visual transformer model in this embodiment;

[0055] Figure 2 This is a schematic diagram illustrating the principle of optimizing the forward dynamics model using DiNAT in this embodiment.

[0056] Figure 3 This is a schematic diagram of the state feature optimization method based on the hierarchical visual transformer model in this embodiment.

[0057] Figure 4Figure 1 shows a comparison of reward values ​​in different game environments on Atari 2600 in this embodiment. Figure 2 shows the reward values ​​in the KungFuMaster environment; Figure 3 shows the reward values ​​in the Pong environment; Figure 4 shows the reward values ​​in the Pitfall environment; Figure 5 shows the reward values ​​in the Solaris environment; Figure 6 shows the reward values ​​in the Gravitar environment; Figure 7 shows the reward values ​​in the BeamRider environment; Figure 8 shows the reward values ​​in the CrazyClimber environment; Figure 9 shows the reward values ​​in the Seaquest environment; Figure 10 shows the reward values ​​in the Breakout environment; and Figure 11 shows the reward values ​​in the PrivateEye environment. Detailed Implementation

[0058] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0059] The core idea of ​​this invention is to introduce the Dilated Neighborhood Attention Transformer (DiNAT) model from the field of computer vision into the field of agent exploration and utilization in deep reinforcement learning. The DiNAT model is used to optimize the intrinsic curiosity model DiNAT-ICM based on a hierarchical visual transformer in the intrinsic reward mechanism. The sparse global attention in the DiNAT model is used to calculate the weight information of different state features of DiNAT-ICM and to evaluate the importance between different state features. The DiNAT model extracts sparse global attention information from the dataset using Neighborhood Attention (NA) and the Dilated Neighborhood Attention (DiNA). The Transformer model can better obtain useful state features for different target networks, and also enables DiNAT-ICM to have better action and state feature selection capabilities. This allows DiNAT-ICM to learn more accurate and useful state information, thereby calculating a high-quality intrinsic reward signal and alleviating the exploration difficulties of DiNAT-ICM in complex or sparse state spaces. Furthermore, the method of this invention utilizes an improved Advantage Actor-Critic (A2C) algorithm to calculate the discounted reward. Since different state features in the feature space have different degrees of importance to the Actor network and the Critic network, a self-attention mechanism is introduced to improve the Actor network, enabling the agent to obtain more accurate local state information, reducing the variance in the reward calculation process, and improving the convergence stability of the model. Then, the Exponential Weighted Moving Average (EWMA) method is used to smooth the data.

[0060] This embodiment presents a state feature optimization method based on a hierarchical visual transformer model, such as... Figure 1 As shown, the method includes the following steps:

[0061] Step 1: Create interactive environments (envs) to obtain the image dataset.

[0062] Step 1.1: Create several independently running parallel environments based on the set number of parallel environments num_envs, use the created parallel environments to build interactive environments envs, and set the number of random seeds in the interactive environments.

[0063] In this implementation, the number of parallel environments num_env = 4, creating four independently running parallel environments to provide data sampling space. The created parallel environments are used to construct interactive environments envs, and the number of random seeds in the interactive environments is set to 42.

[0064] Step 1.2: Set the number of data frames n_stack and obtain an image dataset consisting of n_stack consecutive frames.

[0065] In this implementation, the Atrai game environment in the gym library is called, and four consecutive frames of images are extracted from the Atrai game environment to form an image dataset. The number of data frames is set to n_stack = 4 to provide the number of stacked frames.

[0066] Step 2: Build and initialize the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer in the interactive environment envs.

[0067] The intrinsic curiosity model DiNAT-ICM based on hierarchical visual transformer includes: intrinsic curiosity network ICM_Network, scalable proximity attention Transformer model DiNAT_Network, and A2C algorithm network A2C_Network.

[0068] The intrinsic curiosity network ICM_Network is used to predict the intrinsic reward of the DiNAT-ICM intrinsic curiosity model based on a hierarchical visual transformer. It consists of a Feature Encoder, a Forward Network, and an Inverse Network connected in sequence.

[0069] In this embodiment, such as Figure 2 and Figure 3As shown, the Feature_Encoder consists of four cascaded Conv2d convolutional layers and a Long Short-Term Memory layer. The network consists of sequentially connected Memory (LSTM) networks. The Conv2d convolutional layer has the following network parameters: input channels = 4, number of kernels num_filter = 32, kernel size filter_size = 3, stride = 2, and padding size pad = 1.5. The LSTM network has the following parameters: input dimension input_size = 288, hidden layer size hidden_size = 288. The ForwardNet forward dynamics model consists of two linear layers, Linear1 and Linear2, with the following network parameters: input dimension input_size = 288, hidden layer size hidden_size = 256, and output dimension output_size = 288. The InverseNet inverse dynamics model consists of two linear layers, Linear3 and Linear4, with the following network parameters: feature space dimension feature_space_size = 288, hidden layer size hidden_size = 256, and number of actions num_action ∈ [3, 18].

[0070] The scalable neighborhood attention Transformer model DiNAT_Network is used to optimize the intrinsic curiosity network ICM_Network, which includes the following layers connected in sequence: a first normalized layer LayerNorm1, a scalable sparse global attention layer (Dilated_Neighborhood_Attention, DiNA), a second normalized layer LayerNorm2, a multi-layer perceptron (MLP), a linear layer Linear5, and a softmax function.

[0071] In this implementation, the scalable proximity attention Transformer model DiNAT_Network improves the efficiency of sample data utilization by optimizing the intrinsic curiosity network. For example... Figure 3As shown, the input parameters of LayerNorm1 and LayerNorm2 are set to input_shape = 288. The input dimension Attn_size ∈ [291, 306], the number of heads num_head ∈ [2, 6], and the kernel size Kernel_size = 7 are set for DiNA. The input dimension in_features of the multilayer perceptron (MLP) is set to in_features = 288, the hidden layer dimension hidden_feature = 1152, and the GELU function is used as the activation function of the MLP. The input dimension input_size of the linear layer Linear5 is set to input_size = 288, and the output dimension output_size = 288.

[0072] The A2C algorithm network, A2C_Network, is used to predict discounted rewards for the DiNAT-ICM intrinsic curiosity model based on a hierarchical visual transformer. It consists of a Self-Attention Network, a Critics Actor Network, and a Critics Critic Network connected in sequence.

[0073] The initialization of the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer includes setting the number of data frames n_stack, the number of parallel environments num_envs, the number of actions num_actions, the environment name envs_name, the input data size input_size, and the state feature size feature_size.

[0074] In this embodiment, the number of data frames n_stack set in step 1.2 and the number of parallel environments num_envs set in step 1.1 are obtained. At the same time, the input data size input_size is set to 288, the state feature size feature_size is set to 256, and the numerical range of the number of actions num_actions and the environment name envs_name are set according to the created interactive environment.

[0075] Step 3: Iteratively train the DiNAT-ICM intrinsic curiosity model based on the hierarchical visual transformer using the image dataset, and calculate the total reward of DiNAT-ICM. total and state feature variance std And store it in the data log storage file logger.

[0076] In this embodiment, while performing iterative training, the data information of the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer is saved, including: creation time, environment name, number of training rounds, average reward, average state feature variance, and model weight information.

[0077] Step 3.1: Reset the interactive environment, obtain the current round state observation value obs of DiNAT-ICM in the interactive environment, and initialize the state observation value obs to obtain an N-dimensional array object Numpy of size [num_env, width, height, n_stack]. obs Where width is the image width and height is the image height.

[0078] Step 3.2: Convert the array object using NumPy obs Convert to a state information tensor states The state information tensor states The size is [num_env, n_stack, width, height].

[0079] In this implementation, the array object NumPy is used. obs Convert to a state information tensor states It is then mounted onto the Graphics Processing Unit (GPU) as a state information tensor. states The size is [4, 4, 84, 84]. These represent the number of parallel environments (num_env), the number of game frames (n_stack), the image width (width), and the image height (height), respectively.

[0080] Step 3.3: Based on the state information tensor states The intrinsic curiosity model DiNAT-ICM based on a hierarchical visual transformer was iteratively trained using the deep reinforcement learning A2C algorithm, and the discounted reward R of DiNAT-ICM was calculated. ext .

[0081] Step 3.3.1: Construct and initialize the experience recycling pool Buffer.

[0082] In this embodiment, the data information stored in the experience recycling pool Buffer includes: rewards, states, features, actions, and long-term reward values ​​for actions; the experience recycling pool Buffer is initialized by setting it to 0.

[0083] Step 3.3.2: Convert the state information tensor of DiNAT-ICM at the current iteration step. states As state information (states).

[0084] Step 3.3.3: Use the Feature_Encoder to encode and transform the state information states to obtain the state encoding feature, and then map the state encoding into the feature space.

[0085] In this embodiment, the current state information states is encoded and transformed using a state feature encoder (Feature_Encoder) to obtain a state encoding feature. The tensor size of this state encoding feature is [4,288], where 4 represents the number of parallel environments and 288 is actually the state information extracted from each environment. [4,288] can be understood as four [1,288]. By inputting this state encoding into an LSTM network, the current state encoding is mapped to the feature space.

[0086] Step 3.3.4: Input the mapped state code into the attention network and use attention weights to filter the state information score features. score The feature for scoring state information score Softmax normalization is performed to obtain the final attention state feature. attn .

[0087] Step 3.3.5: Extract the final attention state features. attn Input the actor network to obtain the decision π attn Then, softmax normalization is performed to obtain the current action probability distribution (Action), and the next action (action) of DiNAT-ICM is obtained by sampling using a random distribution. attn .

[0088] The current action probability distribution (Action) is obtained, and the next action (action) is derived by sampling using a random distribution. attn The formula for calculation is:

[0089]

[0090] Where π attn Represents the action decision optimized using self-attention; Action represents the action probability distribution improved by self-attention; action attnThis represents the agent's next action; actor() represents the policy network in the A2C algorithm; cat() represents random sampling of the action probability distribution; softmax(·) represents softmax normalization.

[0091] Step 3.3.6: Encode the mapped state and the decision π attn Input the critique network and calculate the value V(feature, π) in the current state. attn ).

[0092] Step 3.3.7: Calculate the policy loss, value function loss, and policy entropy. entropy And construct the total loss Utilizing total loss Calculate the current discount reward R of DiNAT-ICM ext and will offer discount rewards R ext The final attention state feature attn action attn Action probability distribution and value V(feature,π) attn Stored in the experience recycling pool buffer.

[0093] The formula for the total loss is shown in formula (2);

[0094]

[0095] Where A is the dominance function; R total The θ represents the total discount reward for the current number of steps. critic This indicates updating the parameters of the critic network; V(feature; θ) critic ) represents the state-value function; θ actor This indicates updating the parameters of the actor network; Loss policy Represents the policy loss, used to update the action network and measure the advantage of the current action relative to the average action; Loss value Represents the value function loss, used to update the accuracy of the critic network's estimated state values; Policy entropy Represents policy entropy, used to sum all actions within the current action space; action i denoted as the i-th action; a∈[0,1] represents the weight factor of the value function loss.

[0096] Step 3.3.8: Reset the experience recycling pool Buffer, and repeat steps 3.3.2-3.3.7 to continue iteratively calculating the next round's step. next Discount Rewards R extThe final attention state feature attn action attn Action probability distribution and value V(feature,π) attn This continues until the preset number of iterations is completed.

[0097] In this embodiment, the preset number of iteration rounds is 2.5 million.

[0098] Step 3.4: Clear the accumulated gradients in the DiNAT-ICM intrinsic curiosity model based on the hierarchical visual transformer and initialize them to 0.

[0099] Step 3.5: Calculate the state prediction error of DiNAT-ICM as the intrinsic reward R insic .

[0100] Step 3.5.1: Convert the state information tensor of DiNAT-ICM at the current iteration step. states As state information, `states` is encoded and transformed using the Feature Encoder to obtain a state-encoded feature. This state-encoded feature is then mapped onto the feature space to obtain the mapped state feature `feature0` and the corresponding state feature information `feature`. curt Then, slice the state feature 0 to obtain the next state feature information of DiNAT-ICM. next .

[0101] Step 3.5.2: Obtain the state feature of round t in the current iteration step. t And it utilizes the scalable neighbor attention Transformer structure DiNAT_Network to analyze this state feature. t Normalization is performed, the resulting feature tensor is then increased in dimensionality, and Scalable Sparse Global Attention (DiNA) is used to perform residual calculation to obtain the state features after residual calculation.

[0102] Step 3.5.3: After normalizing and reducing the dimensionality of the state features calculated from the residuals, perform another softmax normalization to obtain the state features optimized by the DiNAT network. DiNAT .

[0103] Step 3.5.4: Based on the state feature information of DiNAT-ICM curt DiNAT-ICM next state feature informationnext and optimized state features DiNAT The forward dynamics model ForwardNet is used to predict the next state feature encoding of DiNAT-ICM. Then, the InverseNet inverse dynamics model is used to calculate the predicted value of the current action.

[0104] The formulas for the forward dynamics model and the inverse dynamics model are shown in expression (3):

[0105]

[0106] Where DiNAT() represents the extended proximity attention Transformer network structure; ForwardNet() represents the forward dynamics model in DiNAT-ICM; action attn The InverseNet() method represents the inverse dynamics model in DiNAT-ICM, indicating the action obtained after optimizing the actor network through a self-attention mechanism.

[0107] Step 3.5.5: Encode the next state feature using DiNAT-ICM And the predicted value of the current action Calculate the state feature variance of DiNAT-ICM std and intrinsic reward R insic .

[0108] In this embodiment, the next state feature encoding of DiNAT-ICM is utilized. And the predicted value of the current action Calculate the loss of DiNAT-ICM icm The expression is:

[0109]

[0110] Loss fwd This indicates the magnitude of the loss in the forward dynamics model; This represents the mean square error between the state characteristics and the predicted value. Loss represents the cross-entropy between the predicted value of the current action and the current action. inv This represents the magnitude of the loss of the inverse dynamics model, where β∈[0,1] represents a scalar used to measure the loss of the forward and inverse dynamics models.

[0111] In this embodiment, the prediction error of the DiNAT-ICM state is used as the intrinsic reward R. insic And the intrinsic reward R insicThe calculation formula is shown in expression (5);

[0112]

[0113] Where λ represents a scalar used to measure the importance of policy gradient loss and learning intrinsic reward, and λ>0.

[0114] Step 3.6: Utilize DiNAT-ICM's discount reward R ext and intrinsic reward R insic Calculate the total reward in the current iteration step. total .

[0115] The total reward total The calculation formula is shown in (6);

[0116]

[0117] in This represents the extrinsic reward in round t. Let represent the intrinsic reward in round t; a is the weighting factor of the intrinsic reward, representing the relative contribution of the intrinsic reward to the total reward; γ is the discount factor; t represents the round t; T is the maximum number of time steps in that round.

[0118] Step 3.7: Reward the total reward for the current round. total and state feature variance std Store the data in the logger file and repeat steps 3.1-3.6 until the preset number of iterations is completed.

[0119] In this implementation, the total reward will be... total and state feature variance std Store the data in the logger file and then continue iterating up to 2.5 million times.

[0120] Step 4: Read the total reward from the logger data storage file. total and state feature variance std Calculate the average reward of DiNAT-ICM within a unit round. and mean state characteristic variance Using the exponentially weighted average (EWMA) method to... and Smoothing is performed, and the processed results are visualized to complete the optimization of state features.

[0121] In this embodiment, the data log storage file `logger` is read, and the average reward is calculated in units of 100 rounds. and mean state characteristic variance The data is smoothed using the exponentially weighted average (EWMA) method, and the processed results are visualized to achieve feature optimization.

[0122] In this embodiment, the game frame rate on Atari 2600 is used as the optimized DiNAT-ICM dataset. This embodiment is compared with the baseline Rational Curiosity Model (RCM) and the existing method Random Network Distillation (RND) in the Atari 2600 game benchmark environment proposed by OpenAI. The experimental results are as follows: Figure 4 As shown, through Figure 4 Analysis reveals a significant improvement in the maximum average reward metric (MMR) across four sparse reward environments: Pitfall, Solaris, Gravitar, and PrivateEye (i.e., at the highest peak of the curve). Furthermore, this metric is improved across the other six dense reward environments. Therefore, it can be concluded that the intrinsic reward signal and state feature variance values ​​obtained by this method are superior to the baseline and existing methods, particularly for the four sparse reward environments of Pitfall, Solaris, Gravitar, and PrivateEye. The state feature optimization method of this invention can indeed improve model performance.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A state feature optimization method based on a hierarchical visual transformer model, characterized in that, The method includes the following steps: Step 1: Create interactive environments (envs) to obtain the image dataset; Step 2: Build and initialize the intrinsic curiosity model DiNAT-ICM based on the hierarchical visual transformer in the interactive environment envs; The intrinsic curiosity model DiNAT-ICM based on hierarchical visual transformer includes: intrinsic curiosity network ICM_Network, scalable proximity attention Transformer model DiNAT_Network, and A2C algorithm network A2C_Network. The intrinsic curiosity network ICM_Network is used to predict the intrinsic reward of the DiNAT-ICM intrinsic curiosity model based on hierarchical visual transformer, and includes the following components connected in sequence: Feature_Encoder, ForwardNet, and InverseNet. The scalable neighbor attention Transformer model DiNAT_Network is used to optimize the intrinsic curiosity network ICM_Network, which includes the following connected in sequence: a first normalized layer LayerNorm1, a scalable sparse global attention DiNA, a second normalized layer LayerNorm2, a multilayer perceptron MLP, a linear layer Linear5, and a softmax function. The A2C algorithm network, A2C_Network, is used to predict the discounted reward of the DiNAT-ICM intrinsic curiosity model based on hierarchical visual transformers. It includes the following connected networks in sequence: Self-AttentionNet, Actor Network, and Critic Network. Step 3: Iteratively train the DiNAT-ICM intrinsic curiosity model based on the hierarchical visual transformer using the image dataset, and calculate the total reward of DiNAT-ICM. and state characteristic variance And store it in the data log storage file logger; Step 4: Read the total reward from the data log storage file logger. and state characteristic variance Calculate the average reward of DiNAT-ICM within a unit round. and mean state characteristic variance Using the exponentially weighted average (EWMA) method to... and Smoothing is performed, and the processed results are visualized to complete the optimization of state features.

2. The state feature optimization method based on a hierarchical visual transformer model according to claim 1, characterized in that, Step 1 further includes: Step 1.1: Create several independently running parallel environments according to the set number of parallel environments num_envs, build an interactive environment using the created parallel environments, and set the number of random seeds in the interactive environment; Step 1.2: Set the number of data frames n_stack and obtain an image dataset consisting of n_stack consecutive frames.

3. The state feature optimization method based on a hierarchical visual transformer model according to claim 1, characterized in that, Step 2, which initializes the intrinsic curiosity model DiNAT-ICM based on a hierarchical visual transformer, includes setting the number of data frames n_stack, the number of parallel environments num_envs, the number of actions num_actions, the environment name envs_name, the input data size input_size, and the state feature size feature_size.

4. The state feature optimization method based on a hierarchical visual transformer model according to claim 3, characterized in that, Step 3 further includes: Step 3.1: Reset the interactive environment, obtain the current round state observation value obs of DiNAT-ICM in the interactive environment, and initialize the state observation value obs to obtain an N-dimensional array object of size [num_env, width, height, n_stack]. Where width is the image width and height is the image height; Step 3.2: Transfer the array object Convert to state information tensor The state information tensor The size is [num_env, n_stack, width, height]; Step 3.3: Based on the state information tensor The intrinsic curiosity model DiNAT-ICM based on a hierarchical visual transformer was iteratively trained using the deep reinforcement learning A2C algorithm, and the discounted reward of DiNAT-ICM was calculated. ; Step 3.4: Clear the accumulated gradients in the DiNAT-ICM intrinsic curiosity model based on the hierarchical visual transformer and initialize them to 0; Step 3.5: Calculate the state prediction error of DiNAT-ICM as the state characteristic variance of DiNAT-ICM. and intrinsic rewards ; Step 3.6: Utilize DiNAT-ICM's discount rewards and intrinsic rewards Calculate the total reward in the current iteration step. ; Step 3.7: Total Rewards and state characteristic variance Store the data in the logger file and repeat steps 3.1-3.6 until the preset number of iterations is completed.

5. The state feature optimization method based on a hierarchical visual transformer model according to claim 4, characterized in that, Step 3.3 further includes: Step 3.3.1: Construct and initialize the experience recycling pool Buffer; Step 3.3.2: Convert the state information tensor of DiNAT-ICM at the current iteration step. As state information (states); Step 3.3.3: Use the Feature_Encoder to encode and transform the state information states to obtain the state encoding feature, and map the state encoding into the feature space; Step 3.3.4: Input the mapped state code into the attention network and use attention weights to filter the state information scores. Score the status information Softmax normalization is performed to obtain the final attention state features. ; Step 3.3.5: Extract the final attention state features Input the actor network to obtain the decision. Then, softmax normalization is performed to obtain the current action probability distribution (Action), and the next action of DiNAT-ICM is obtained by sampling using a random distribution. ; Step 3.3.6: Encode the mapped state and make a decision. Input the critical network and calculate the value in the current state. ; Step 3.3.7: Calculate the policy loss, value function loss, and policy entropy. And construct the total loss Utilizing total losses Calculate the current discount reward of DiNAT-ICM and discount rewards State characteristics ,action Action probability distribution and value Stored in the experience recycling pool buffer; Step 3.3.8: Reset the experience recycling pool Buffer, and repeat steps 3.3.2-3.3.7 to continue iteratively calculating the next round. Discount Rewards State characteristics ,action Action probability distribution and value This continues until the preset number of iterations is completed.

6. The state feature optimization method based on a hierarchical visual transformer model according to claim 5, characterized in that, Step 3.5 further includes: Step 3.5.1: Convert the state information tensor of DiNAT-ICM at the current iteration step. As state information, states are encoded and transformed using the Feature Encoder to obtain state encoded features. These state encoded features are then mapped onto the feature space to obtain the mapped state features. and corresponding state feature information Then, regarding the state features Slice the data to obtain the next state feature information of DiNAT-ICM. ; Step 3.5.2: Obtain the state features of round t in the current iteration step. It also utilizes the scalable neighbor attention Transformer structure DiNAT_Network to analyze state features. Normalization is performed, the resulting feature tensor is then increased in dimensionality, and Scalable Sparse Global Attention DiNA is used to perform residual calculation to obtain the state features after residual calculation. Step 3.5.3: After normalizing and reducing the dimensionality of the state features calculated from the residuals, perform another softmax normalization to obtain the state features optimized by the DiNAT network. ; Step 3.5.4: Based on the state characteristic information of DiNAT-ICM Next state feature information of DiNAT-ICM and optimized state features The forward dynamics model ForwardNet is used to predict the next state feature encoding of DiNAT-ICM. Then, the InverseNet inverse dynamics model is used to calculate the predicted value of the current action. ; Step 3.5.5: Encode the next state feature using DiNAT-ICM And the predicted value of the current action Calculate the state characteristic variance of DiNAT-ICM and intrinsic rewards .

7. The state feature optimization method based on a hierarchical visual transformer model according to claim 6, characterized in that, The forward dynamics model and the inverse dynamics model are as follows: ； in This indicates an extended proximity attention Transformer network structure; Represents the forward dynamics model in DiNAT-ICM; This represents the action obtained after optimizing the actor network using a self-attention mechanism: This represents the inverse dynamics model in DiNAT-ICM.

Images