A Reinforcement Learning Model Environment Adaptation Method Based on Zero-Shot Generalization

Abstract

The present invention discloses a method for environment adaptation of a reinforcement learning model based on zero-shot generalization. First, two sets of data are randomly selected from the training data obtained by the interaction between the model and the environment. One set of state images is subjected to data augmentation. Then, semantic encoding is performed on the augmented and non-augmented original state images respectively. Style-independent semantic information and key content semantic information are extracted through the IBN module and the attention module. Then, prediction values are obtained through the Q function respectively. Finally, the encoder of the augmented data is updated by combining the prediction errors of the two sets of data.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and particularly relates to a method for adapting a reinforcement learning model to an environment. Background Art

[0002] Vision-based deep reinforcement learning has achieved success in various control tasks and demonstrated superhuman performance, such as in tasks like Atari games, Go games, robot control, and autonomous driving. However, the training of deep reinforcement learning models relies on frequent interactions with a specific environment. Once the environment changes, the model is prone to making unreasonable decisions and its performance significantly degrades. Different from the fixed state space of game environments, the real environment where edge agents are located is complex and constantly changing dynamically. It is difficult for training data to cover all environmental states. Therefore, the ability to adapt to the environment is crucial for the large-scale deployment and application of deep reinforcement learning models.

[0003] Existing researchers have explored the environmental adaptation technology of the model through methods such as zero-shot generalization, transfer learning, and continual learning. Among them, zero-shot generalization is widely applied in most scenarios where the environment changes slightly. Zero-shot generalization means training the model in the source environment so that the model performs well in multiple unseen target environments without the need for the model to evolve in the new environment and without computational requirements for the agent. There are mainly methods based on data augmentation, methods based on state representation, and methods based on policy learning. The method based on data augmentation means enhancing the robustness of the model by randomly augmenting the training data. Representative methods include Random convolution, Mixup, Cutmix, Attentionneuron, etc. Using such methods alone is likely to hinder training and lead to suboptimal performance, so they are usually used in combination with the latter two methods. The method based on state representation means training an encoder with generalization ability to enhance the overall generalization ability of the model, such as PIE-G, LUSR, MoDE, DBC, etc. The method based on policy learning means training the overall generalization ability of the deep reinforcement learning policy model, such as Secant, SEVA, RCAN, netrand. Summary of the Invention

[0004] To overcome the deficiencies of the prior art, the present invention provides a method for adapting a reinforcement learning model to an environment based on zero-shot generalization. First, two sets of data are randomly selected from the training data obtained by the interaction between the model and the environment. One set of state images is augmented with data. Then, semantic encoding is performed on the augmented and non-augmented original state images respectively. Style-independent semantic information and key content semantic information are extracted through the IBN module and the attention module. Then, prediction values are obtained through the Q function respectively. Finally, the encoder of the augmented data is updated by combining the prediction errors of the two sets of data.

[0005] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0006] Step 1: Data collection;

[0007] The agent is controlled by a reinforcement learning model to interact with the environment according to the policy π, and collect the (s t , a t , r t , s t+1 ) sequence as training data; during the training process, data collection and model parameter update are carried out alternately; s t , a t , r t , s t+1 respectively represent the state image, the action taken, the reward obtained, and the state image at the next moment of the agent;

[0008] Step 2: Data augmentation;

[0009] Randomly select two groups of data at time t and time h from the training data B, (s t , a t , r t , s t+1 ), (s h , a h , r h , s h+1 ), crop the state images s t and s h into the size of A×A, perform data augmentation on the s t state image to obtain s′ t ; use the interference picture I to perform mixup linear data augmentation on the state picture s t :

[0010] f(s t ) = α st + (1 - α)I

[0011] In the formula, f(s t ) represents the enhanced state image, and α represents the weight coefficient;

[0012] Step 3: Data encoding;

[0013] Encode the state images s′ t and s h , and extract style-independent semantic information and key content semantic information through the IBN module and the attention module; the IBN module includes an IN layer and a BN layer, the IN layer can learn features invariant to appearance changes, that is, domain-independent features, and the BN layer can learn content information;

[0014] Step 3-1: Low-level feature extraction; respectively downsample the state pictures s′ t and s h to extract the low-level features F t and F h of the images at high resolution;

[0015] Step 3-2: Style-agnostic semantic feature extraction; respectively extract style-agnostic semantic features from F t and F h through the IBN module to obtain F′ t and F′ h ;

[0016] Step 3-3: Key information semantic feature extraction; respectively extract the key information semantic features from F′ t and F′ h through the CBAM attention module to obtain F″ t and F″ h ; The principle of the extraction process is as follows:

[0017] Step 3-3-1: The channel attention sub-module uses average pooling and max pooling operations on the input feature F to aggregate the spatial information of the feature map, generating two different spatial context descriptors and representing the average pooling feature and the max pooling feature respectively, forwarding the two descriptors to a shared network to generate the channel attention map M c ; The shared network consists of a multi-layer perceptron MLP and a hidden layer; after applying the shared network to each descriptor, the output feature vectors are summed element-wise and merged as F′:

[0018] M c (F) = σ(MLP(AveragePool(F)) + MLP(MaxPool(F)))

[0019] where σ represents the sigmoid function;

[0020] Step 3-3-2: The spatial attention sub-module applies average pooling and max pooling operations on the feature F along the channel axis, generating two 2D maps and Connecting and convolving this data through a standard convolutional layer to generate the 2D spatial attention map M s :

[0021] M s (F) = a(f 7×7 ([Averagepool(F); Maxpool(F)]))

[0022] Step 3-3-3: The sequential combination of the CBAM module, i.e., the channel attention sub-module and the spatial attention sub-module. The overall attention process is as follows:

[0023]

[0024]

[0025] Among them, denotes element-wise multiplication, and F″ is the finally output feature. During the multiplication operation, the attention values are correspondingly extracted: the channel attention sub-module extracts the key information content, and the spatial attention sub-module extracts the key information location.

[0026] Step 4: Q-function prediction;

[0027] The state-action pair (F″ t and action a t formed by combining them) and the state-action pair (F″ t , a t ) at time h are input into the Q-function to obtain the Q-value. The Q-function is defined as: h and action a h formed by combining them) and the state-action pair (F″ h , a h ) are input into the Q-function to obtain the Q-value. The Q-function is defined as:

[0028] q t = Q θ (F″ t , a t )

[0029] The target state-action value function is defined as:

[0030]

[0031] Among them is the exponential moving average of θ, and γ represents the weight coefficient;

[0032] Step 5: Joint optimization of the Q-function;

[0033] The Q-function is jointly optimized using the enhanced data stream (s′ t , a t , r t , s t+1 ) and the non-enhanced data stream (s h , a h , r h , s h+1 ). The optimization objective of the Q-function is:

[0034]

[0035]

[0036] Among them, α and β are the weights of the enhanced data and the unenhanced data in optimization.

[0037] Preferably, the A = 84.

[0038] Preferably, the downsampling layer used in the downsampling contains 11 convolutional layers, the convolutional kernel size is 3×3, and the stride is 1.

[0039] Preferably, the interference image comes from the DAVIS17 dataset.

[0040] The beneficial effects of the present invention are as follows:

[0041] The present invention enhances the generalization ability of the model while the accuracy does not decrease significantly, achieving the effect of environmental adaptation. Description of the Drawings

[0042] Figure 1 It is the method model diagram of the present invention. Detailed Embodiments

[0043] The present invention will be further described below in conjunction with the drawings and embodiments.

[0044] The present invention proposes a method for environmental adaptation of a reinforcement learning model based on zero-shot generalization. Using the following principles: environmental changes lead to style changes and content changes in the state image. By jointly training a deep reinforcement learning model containing an IBN module and an attention module with the original data and enhanced data, the environmental adaptation ability of the model can be enhanced while the accuracy does not decrease significantly: (1) The model trained with the original data has high accuracy, and the model trained with the enhanced data has strong generalization ability. Jointly training the model with the original data and the enhanced data combines accuracy and generalization ability; (2) The IBN module can learn the invariance of the image appearance style; (3) The channel attention module identifies the important information content of the image, and the spatial attention module identifies the position of the important information of the image.

[0045] A method for environmental adaptation of a reinforcement learning model based on zero-shot generalization includes the following steps:

[0046] Step 1: Data collection. The agent is controlled by the reinforcement learning model to take actions and interact with the environment according to a certain policy π, and collect (s t , a t , r t , s t+1 ) sequences as training data. During the training process, data collection and model parameter update are carried out alternately.

[0047] Step 2: Data enhancement. Randomly select two groups of data at time t and time h from the training data B (s t , a t , rt , s t+1 ), (s h , a h , r h , s h+1 ), crop the status images s t and s h to a size of 84×84, and perform data augmentation on the status image s t to obtain s′ t . Use the interference picture I to perform the mixup linear data augmentation method on the status picture s t as follows:

[0048] f(s t ) = α st + (1 - α)I

[0049] Step 3: Data encoding. Encode the status images s′ t and s h to extract style-agnostic semantic information and key content semantic information through the IBN module and the attention module.

[0050] (1) Low-level feature extraction. Downsample the status pictures s′ t and s h respectively, and extract the low-level features F t and F h of the images at high resolution. Among them, the downsampling layer contains 11 convolutional layers, the kernel size of the convolutional kernels is all 3×3, and the stride is 1.

[0051] (2) Style-agnostic semantic feature extraction. Extract style-agnostic semantic features from F t and F h respectively through the IBN layer to obtain F′ t and F′ h . The IN (Instance Normalization) layer can learn features that are invariant to appearance changes (domain-agnostic), and the BN (Batch Normalization) layer can effectively learn content information. The IBN layer is an integration of IN and BN, which can improve the model accuracy and generalization ability.

[0052] (3) Key information semantic feature extraction. Extract the semantic features of key information from F′ t and F′ h respectively through the CBAM attention module to obtain F″ t and F″ h . The principle of the extraction process is as follows:

[0053] The channel attention sub-module uses average pooling and max pooling operations on the input feature F to aggregate the spatial information of the feature map, generating two different spatial context descriptors and represent the average pooling feature and the max pooling feature respectively. Forward the two descriptors to the shared network to generate the channel attention map M c . The shared network consists of a multi-layer perceptron (MLP) and a hidden layer. After applying the shared network to each descriptor, the output feature vectors F′ are combined by element-wise summation.

[0054] M c (F)==σ(MLP(AveragePool(F))+MLP(MaxPool(F)))

[0055] The spatial attention sub-module applies average pooling and max pooling operations on the feature F along the channel axis, generating two 2D maps and Connect and convolve this data through a standard convolutional layer to generate the 2D spatial attention map M s .

[0056] M s (F)=σ(f 7×7 ([Averagepool(F);MaxPool(F)]))

[0057] The channel attention sub-module extracts what the key information is, and the spatial attention sub-module extracts where the key information is. The two are complementary. Therefore, using the two modules together can achieve a better key feature extraction effect. The CBAM module is the sequential combination of the channel attention sub-module and the spatial attention sub-module. The overall attention process is as follows:

[0058]

[0059]

[0060] Among them, represents element-wise multiplication, and f″ is the finally output feature. During the multiplication operation, the attention values are extracted accordingly: the channel attention sub-module extracts the content of the key information, and the spatial attention sub-module extracts the location of the key information.

[0061] Step 4: Q-function prediction. Respectively combine the state-action pair (F″ t and the action a t ) composed of the state feature F″ at time t t and the action a t ) with the state-action pair (F″ h and the action a h ) composed of the state feature F″ at time h h, a h ) Input it into the Q function to obtain the Q value. The Q function is defined as:

[0062] q t = Q θ (F″ t , a t )

[0063] The target state-action value function is defined as:

[0064]

[0065] where is the exponential moving average of θ.

[0066] Step 5: Jointly optimize the Q function. Use the enhanced data stream (s′ t , a t , r t , s t+1 ) and the unenhanced data stream (s h , a h , r h , s h+1 ) to jointly optimize the Q function. The optimization objective of the Q function is:

[0067]

[0068]

[0069] where α and β are the weights of the enhanced data and the unenhanced data in the optimization. Specific embodiments:

[0071] Step 1: Data collection. In the Atari game environment, control the agent to interact with the environment according to a certain policy π through the reinforcement learning model, and collect the (s t , a t , r t , s t+1 ) sequence as training data. During the training process, data collection and model parameter update are carried out alternately.

[0072] Step 2: Data augmentation. Randomly select two groups of data at time t and time h from the training data B, (s t , a t , r t , s t+1 ), (s h , a h , r h , s h+1 ), crop the state images s t and s h to the size of 84×84, and for st The state image is enhanced to obtain s′ t Use the interference image I to the state image s t Perform mixup linear data enhancement:

[0073] f(s t )=0.5s t +0.5I

[0074] The interference images are from the DAVIS17 dataset.

[0075] Step 3: Data encoding. For the state image s′ t and h Encoding is performed, and style-independent semantic information and key content semantic information are extracted through the IBN module and the attention module.

[0076] (1) Low-level feature extraction. t and h Downsample and extract low-level features F of the image at high resolution t and F h The downsampling layer contains 11 convolutional layers, the convolution kernel size is 3×3, and the step size is 1.

[0077] (2) Style-independent semantic feature extraction. Through the IBN layer, F t and F h Extract style-independent semantic features and obtain F′ t and F′ h .

[0078] (3) Extraction of key information semantic features. Through the CBAM attention module, F′ t and F′ h Extract the semantic features of key information and obtain F″ t and F″ h .

[0079] Step 4: Q function prediction. The state feature F″ at time t is t and action a t The state-action pair (F″ t ,a t ) and the state characteristics F″ at time h h and action a h The state-action pair (F″ h ,a h ) is input into the Q function to get the Q value.

[0080] Step 5: Jointly optimize the Q function. Using the enhanced data flow (s′ t ,a t ,rt ,s t+1 ), and the unenhanced data stream (s h ,a h ,r h ,s h+1 ), jointly optimize the Q function. The weights of the enhanced data and the unenhanced data in the optimization are α = β, and thus the Q function optimization objective is:

[0081]

Claims

1. A method for environment adaptation of a reinforcement learning model based on zero-shot generalization, characterized in that It includes the following steps: Step 1: Data acquisition; The agent is controlled by a reinforcement learning model to interact with the environment by taking actions according to policy π, collecting the (s t , a t , r t , s t+1 ) sequence as training data; During the training process, data collection and model parameter update are carried out alternately; s t , a t , r t , s t+1 represent the state image, action taken, reward obtained, and state image at the next moment of the agent at the current moment, respectively; Step 2: Data augmentation; Randomly select two sets of data at time t and time h from the training data B (s t , a t , r t , s t+1 ), (s h , a h , r h , s h+1 ). Crop the state images s t and s h to the size of A×A, perform data augmentation on the s t state image to obtain s′ t ; Use the interference picture I to perform mixup linear data augmentation on the state picture s t : f(s t ) = αs t + (1 - α)I where f(s t ) represents the enhanced state image, and α represents the weight coefficient; Step 3: Data encoding; Encode the state images s′ t and s h to extract style-agnostic semantic information and key content semantic information through the IBN module and the attention module; the IBN module includes an IN layer and a BN layer. The IN layer can learn features invariant to appearance changes, i.e., domain-agnostic features, and the BN layer can learn content information; Step 3-1: Low-level feature extraction; Downsample the status images s′ t and s h respectively, and extract the low-level features F t and F h ; Step 3-2: Style-independent semantic feature extraction; respectively extract style-independent semantic features from F t and F h through the IBN module to obtain F' t and F' h ; Step 3-3: Extraction of semantic features of key information; respectively extract the semantic features of key information from F′ t and F′ h through the CBAM attention module to obtain F″ t and F″ h ; the principle of the extraction process is as follows: Step 3-3-1: The channel attention sub-module uses average pooling and max pooling operations on the input feature F to aggregate the spatial information of the feature map, generating two different spatial context descriptors and representing the average pooling feature and the max pooling feature respectively, forwarding the two descriptors to the shared network to generate the channel attention map M c ; The shared network consists of a multi-layer perceptron MLP and a hidden layer; After applying the shared network to each descriptor, the output feature vectors F′ are merged by element-wise summation: M c (F) = σ(MLP(AveragePool(F)) + MLP(MaxPool(F))) Wherein, σ represents the S-shaped function; Step 3-3-2: The spatial attention sub-module applies average pooling and max pooling operations to the feature F along the channel axis to generate two 2D maps and These data are concatenated and convolved through a standard convolutional layer to generate a 2D spatial attention map M s : M s (F) = σ(f 7×7 ([AveragePool(F); MaxPool(F)])) Step 3-3-3: The sequential combination of the CBAM module, i.e., the channel attention sub-module and the spatial attention sub-module. The overall attention process is: Among them, represents element-wise multiplication, and F″ is the finally output feature; during the multiplication operation, the attention values are correspondingly extracted: the channel attention sub-module extracts the key information content, and the spatial attention sub-module extracts the key information location; Step 4: Q-function prediction; The state feature F″ at time t t and the action a t The state-action pair (F″ t , a t ) and the state feature F″ at time h h and the action a h The state-action pair (F″ h , a h ) are input into the Q function to obtain the Q value; the Q function is defined as: q t = Q θ (F″ t , a t ) The target state-action value function is defined as: Among them is the exponential moving average of θ, and γ represents the weight coefficient; Step 5: Joint optimization of the Q function; Using an enhanced data stream (s′ t , a t , r t , s t+1 ) and an unenhanced data stream (s h , a h , r h , s h+1 ) to jointly optimize the Q function; the Q function optimization objective is: Wherein, α and β are the weights of the augmented data and the non-augmented data in the optimization.

2. The environment adaptation method of the reinforcement learning model based on zero-shot generalization according to claim 1, wherein Said A = 84.

3. A method for environment adaptation of a reinforcement learning model based on zero-shot generalization according to claim 1, characterized in that, The downsampling layer used in the downsampling includes 11 convolutional layers, the convolutional kernel size of each is 3×3, and the stride is 1.

4. A method for environment adaptation of a reinforcement learning model based on zero-shot generalization according to claim 1, characterized in that, The interference image comes from the DAVIS17 dataset.

Images