Offline reinforcement learning method and device for pixel-by-pixel Q-value estimation based on data enhancement

Abstract

The present application relates to an offline reinforcement learning method and device for pixel-by-pixel Q-value estimation based on data enhancement. The method comprises: sampling a small batch of original input observations from an offline data set, performing data enhancement on the small batch of original input observations through a common image transformation algorithm, and regularizing the Q-value of the input observation, and finally training a Q-value network for decision-making. The present invention expands the training data set by using a standard incremental method, avoids overestimating the data near the data distribution of the offline data set by regularizing the Q-value of the input observation, and significantly improves the generalization of the algorithm; the method of combining data enhancement with an offline RL algorithm based on pixel observation does not require additional modification to the underlying RL algorithm, making the method easy to implement and scalable to other offline RL algorithms based on Q-value estimation, with strong scalability and good practicality.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning technology, and in particular to an offline reinforcement learning method and device for pixel-by-pixel Q-value estimation based on data enhancement. Background Art

[0002] Reinforcement learning (RL) has recently made significant progress in a wide range of fields, including autonomous driving, video games, and assisted medical care. RL uses real-time interaction with the environment to collect the latest data through trial and error, and iteratively learn optimal policies. However, the application of online RL in real-world environments faces challenges: First, the trial-and-error interactions between agents and the real environment can have devastating consequences. For example, autonomous vehicles exploring roads can pose a serious threat to pedestrian safety. Second, accumulating and collecting data through interaction with the environment is expensive and prone to resource waste, which restricts agents from frequently and unlimitedly collecting data. Finally, the quality of the data collected online during agent interaction with the environment is often variable. In particular, the data generated by a large amount of ineffective exploration is of low value, resulting in low data utilization during the agent's learning process.

[0003] Offline RL provides a method for learning from existing datasets without further interaction with the environment, allowing algorithm training to avoid high-risk trial-and-error and significantly conserving computational resources. Previous offline RL algorithms, such as behavior cloning (BC), batch-constrained deep Q-learning (BCQ), and conservative Q-learning (CQL), learn from large, previously collected datasets and do not update them during training. These algorithms facilitate learning policies that are both efficient and data-efficient. Despite the significant advantages of offline RL, agents can easily fall into suboptimal policies when training offline RL due to two main reasons: (a) the quality and quantity of datasets are often limited; and (b) extrapolation error due to out-of-distribution (OOD) data increases. Consequently, existing techniques suffer from high requirements for offline data quality and low data utilization during learning. Summary of the Invention

[0004] Based on this, it is necessary to provide an offline reinforcement learning method, device, computer equipment and storage medium for pixel-by-pixel Q-value estimation based on data enhancement, which can improve the performance of offline reinforcement learning methods in response to the above technical problems.

[0005] A pixel-by-pixel Q-value estimation offline reinforcement learning method based on data enhancement, the method comprising:

[0006] Based on a pre-acquired offline dataset of agent visual control, sampling a small batch of raw input observations from the offline dataset;

[0007] The small batch of original input observations is data-augmented by a common image transformation algorithm; each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batches of data, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback obtained by the agent after taking action a at the current moment s; each of the image stacks contains a number of original observation images stacked in successive frames; after image stacks s and s′ are data-augmented K times and M times, respectively, a first augmented sample and a second augmented sample are obtained; the first augmented sample includes K input observations, and the second augmented sample includes M input observations; the parameters of the image transformation algorithm in each image stack are set consistently, and the parameters of the image transformation algorithm of each image stack are set randomly;

[0008] Obtaining Q values ​​of K input observations in the first expanded sample through a preset Q-value estimation network, and taking the average Q value of the K input observations as the predicted Q value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation;

[0009] Obtaining Q values ​​of the M input observations in the second expanded sample through a preset Q value target network, and calculating a target Q value corresponding to the original input observation based on the TD-error according to the Q values ​​of the M input observations; the Q value target network and the Q value estimation network use the same network architecture;

[0010] According to the predicted Q value and the target Q value, the parameters of the Q value estimation network are updated by minimizing the MSE mean square error loss function;

[0011] Soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network until the updating stops at a preset time step;

[0012] The updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

[0013] In one embodiment, the further step includes: obtaining a trained learning network as an expert strategy or a behavior strategy;

[0014] Determine the action with the highest value and the corresponding reward corresponding to the input observation according to the expert strategy or behavioral strategy;

[0015] The current image stack s, the corresponding action a, the corresponding reward r, and the next image stack s′ constitute an offline data tuple (s, a, r, s′);

[0016] After multiple sampling, a large number of offline data tuples are obtained, which constitute the offline dataset for the visual control of the intelligent agent.

[0017] In one embodiment, the further step further includes: applying a random image transformation algorithm to the small batch of original input observations to obtain a transformed image;

[0018] Using a bilinear interpolation algorithm on the transformed image, padding each side with 2 pixels to obtain a padded image;

[0019] Random cropping is performed on the padded image to obtain a data-enhanced image with the same specifications as the original image.

[0020] In one embodiment, the method further includes: obtaining Q values ​​of K input observations in the first expanded sample through a preset Q value estimation network;

[0021] Calculate the average Q value of K input observations as:

[0022]

[0023] Where i is the index of the original input observation, θ is the parameter of the Q-value estimation network, (s i ,a i ) is the state-action pair of the original input observation, and f(s,v) is the transformation function that transforms the original input observation by adjusting v, where f retains the Q value of the state-action pair unchanged, that is, Q θ (s,a)=Q θ (f(s,ν),a);

[0024] The average Q value Q θ (s i ,a i ) as the predicted Q-value corresponding to the original input observation.

[0025] In one embodiment, the method further includes: obtaining Q values ​​of the M input observations in the second expanded sample through a preset Q value target network;

[0026] According to the Q values ​​of the M input observations, the target Q value corresponding to the original input observation is calculated based on the TD-error:

[0027]

[0028] Among them, θ′ is the parameter of the Q value target network, r i is the reward value of the original input observation, γ∈(0,1] is the discount coefficient, λ is the adjustment coefficient, To adjust A transformation function that transforms the original input observations.

[0029] In one embodiment, the method further includes: updating the parameters of the Q-value estimation network by minimizing the mean square error (MSE) loss function based on the predicted Q-value and the target Q-value, taking the Q-value target network as the target network:

[0030]

[0031] Where N is the batch size of the mini-batch, α is the learning rate, represents the expectation of the Q-value estimate.

[0032] In one embodiment, the method further includes: soft-updating the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network:

[0033] θ'←τθ+(1-τ)θ'

[0034] Among them, τ is the update coefficient.

[0035] A device for offline reinforcement learning of pixel-by-pixel Q-value estimation based on data enhancement, comprising:

[0036] A raw input observation acquisition module is used to sample small batches of raw input observations from a pre-acquired offline dataset of the agent visual control;

[0037] A data augmentation module is configured to perform data augmentation on the small batch of original input observations using a common image transformation algorithm. Each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batches of data, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback obtained by the agent after taking action a at the current moment s. Each image stack contains a number of original observation images stacked in successive frames. After performing data augmentation K times and M times on image stacks s and s′, respectively, a first augmented sample and a second augmented sample are obtained. The first augmented sample includes K input observations, and the second augmented sample includes M input observations. The parameters of the image transformation algorithm within each image stack are set consistently, and the parameters of the image transformation algorithm for each image stack are set randomly.

[0038] a predicted Q-value calculation module, configured to obtain the Q-values ​​of the K input observations in the first expanded sample through a preset Q-value estimation network, and use the average Q-value of the K input observations as the predicted Q-value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation;

[0039] a target Q-value calculation module, configured to obtain the Q-values ​​of the M input observations in the second expanded sample through a preset Q-value target network, and calculate the target Q-value of the corresponding original input observation based on the TD-error according to the Q-values ​​of the M input observations; the Q-value target network and the Q-value estimation network use the same network architecture;

[0040] A Q-value estimation network parameter updating module is used to update the parameters of the Q-value estimation network by minimizing the MSE mean square error loss function according to the predicted Q value and the target Q value;

[0041] The Q-value target network parameter update module is used to soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network until the updating is stopped at a preset time step; the updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

[0042] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Based on a pre-acquired offline dataset of agent visual control, sampling a small batch of raw input observations from the offline dataset;

[0044] The small batch of original input observations is data-augmented by a common image transformation algorithm; each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batches of data, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback obtained by the agent after taking action a at the current moment s; each of the image stacks contains a number of original observation images stacked in successive frames; after image stacks s and s′ are data-augmented K times and M times, respectively, a first augmented sample and a second augmented sample are obtained; the first augmented sample includes K input observations, and the second augmented sample includes M input observations; the parameters of the image transformation algorithm in each image stack are set consistently, and the parameters of the image transformation algorithm of each image stack are set randomly;

[0045] Obtaining Q values ​​of K input observations in the first expanded sample through a preset Q-value estimation network, and taking the average Q value of the K input observations as the predicted Q value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation;

[0046] Obtaining Q values ​​of the M input observations in the second expanded sample through a preset Q value target network, and calculating a target Q value corresponding to the original input observation based on the TD-error according to the Q values ​​of the M input observations; the Q value target network and the Q value estimation network use the same network architecture;

[0047] According to the predicted Q value and the target Q value, the parameters of the Q value estimation network are updated by minimizing the MSE mean square error loss function;

[0048] Soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network until the updating stops at a preset time step;

[0049] The updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0051] Based on a pre-acquired offline dataset of agent visual control, sampling a small batch of raw input observations from the offline dataset;

[0052] The small batch of original input observations is data-augmented by a common image transformation algorithm; each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batches of data, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback obtained by the agent after taking action a at the current moment s; each of the image stacks contains a number of original observation images stacked in successive frames; after image stacks s and s′ are data-augmented K times and M times, respectively, a first augmented sample and a second augmented sample are obtained; the first augmented sample includes K input observations, and the second augmented sample includes M input observations; the parameters of the image transformation algorithm in each image stack are set consistently, and the parameters of the image transformation algorithm of each image stack are set randomly;

[0053] Obtaining Q values ​​of K input observations in the first expanded sample through a preset Q-value estimation network, and taking the average Q value of the K input observations as the predicted Q value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation;

[0054] Obtaining Q values ​​of the M input observations in the second expanded sample through a preset Q value target network, and calculating a target Q value corresponding to the original input observation based on the TD-error according to the Q values ​​of the M input observations; the Q value target network and the Q value estimation network use the same network architecture;

[0055] According to the predicted Q value and the target Q value, the parameters of the Q value estimation network are updated by minimizing the MSE mean square error loss function;

[0056] Soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network until the updating stops at a preset time step;

[0057] The updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

[0058] The above-mentioned pixel-by-pixel Q-value estimation offline reinforcement learning method, device, computer equipment and storage medium based on data enhancement, by sampling a small batch of original input observations from the offline data set, data enhancement of the small batch of original input observations through a common image transformation algorithm, to obtain a first expanded sample and a second expanded sample, obtaining the Q values ​​of K input observations in the first expanded sample through a Q-value estimation network, taking the average Q value of the K input observations as the Q value of the corresponding original input observation, obtaining the Q values ​​of M input observations through a Q-value target network, calculating the Q value target corresponding to the original input observation based on the Q values ​​of the M input observations based on TD-error, updating the parameters of the Q-value estimation network by minimizing the loss function, and then soft-updating the parameters of the Q-value target network based on the parameters of the updated Q-value estimation network, and finally training to obtain the Q-value estimation network. The present invention uses a standard incremental method to expand the training data set through the above process. By regularizing the Q-value function of the input observation, the data near the data in the data set but not in the data set has a Q-value consistent with the observation value in the data set, avoiding overestimation of static data in the offline data set and significantly improving the generalization of the algorithm. The method of the present invention combines data augmentation with offline RL based on pixel observations, which does not require additional modifications to the underlying RL algorithm, making the method easy to implement and scalable to other algorithms. It has strong scalability and good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 1 is a flow chart of an offline reinforcement learning method for pixel-by-pixel Q-value estimation based on data enhancement in one embodiment;

[0060] Figure 2 A pipeline diagram of an offline reinforcement learning method for pixel-by-pixel Q-value estimation based on data augmentation in one embodiment;

[0061] Figure 3 An enhanced transformation method used in offline reinforcement learning for pixel-by-pixel Q-value estimation based on data enhancement in one embodiment;

[0062] Figure 4t-SNE results of Breakout game samples after being processed by the feature extraction network layer using the DAQ and BCQ algorithms in one embodiment, wherein (a) is the t-SNE result of the Breakout game sample after being processed by the feature extraction network layer using the DAQ algorithm, and (b) is the t-SNE result of the Breakout game sample after being processed by the feature extraction network layer using the BCQ algorithm;

[0063] Figure 5 1 is a block diagram of an offline reinforcement learning device for pixel-by-pixel Q-value estimation based on data enhancement in one embodiment;

[0064] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0066] In one embodiment, Figure 1 As shown, an offline reinforcement learning method for pixel-by-pixel Q-value estimation based on data enhancement is provided, comprising the following steps:

[0067] Step 102: based on a pre-acquired offline dataset of the agent visual control, sample a small batch of original input observations from the offline dataset.

[0068] Each image stack contains several consecutive frames of the original observation image. The reason for using multiple consecutive frames is that a single frame alone is insufficient to capture the true state information. Conventional wisdom in visual control tasks is to use three or four consecutive frames to maximize the capture of true information about the current state. For example, a single frame alone would make it difficult to capture information such as displacement and velocity in a game or robotic scene. Using consecutive frames as input allows the agent to capture this information.

[0069] In offline reinforcement learning (offlineRL), from the behavior policy π β The sampled dataset is used as the offline dataset. Represents a mini-batch of data in an offline dataset. In offline reinforcement learning, the agent cannot interact with the environment.

[0070] The present invention proposes an offline reinforcement learning method (DAQ) based on pixel-by-pixel Q-value estimation based on data enhancement. Pixel-by-pixel means that the input of the control task that the intelligent agent wants to solve is pixels (pixels or images), and the corresponding one is state (vector or array). The method of the present invention is an offline reinforcement learning method framework that combines data enhancement and Q-value estimation. It performs pixel-by-pixel data enhancement on the image input of the visual control task and regularizes the Q-value estimation to improve the generalization and data utilization of the offline reinforcement learning algorithm. This is the first time that an offline reinforcement learning algorithm with data enhancement has been used on visual control tasks in discrete action spaces. This method makes full use of the invariance properties of image data such as rotation and translation, so that when using offline reinforcement learning algorithms on visual control tasks in discrete action spaces, the data set of pixel-level image inputs is effectively expanded. The main difficulty in using data enhancement methods on pixel-level offline RL is the need to overcome the problem of over-estimation of out-of-distribution data. To this end, we propose a method using regularized Q-values ​​to effectively alleviate this problem.

[0071] like Figure 2 As shown, DAQ applies a standard transformation to augment input observations sampled from a pre-collected dataset and regularizes the Q-value function of the input observations.

[0072] In this embodiment, a small batch of original input observations is sampled from an offline dataset, where the original input observations are stacks of several consecutive images.

[0073] Step 104 , perform data augmentation on the small batch of original input observations using a common image transformation algorithm.

[0074] Among them, each image stack at the current moment and the next moment is subjected to K and M data enhancements respectively to obtain the first expanded sample and the second expanded sample respectively; the parameter settings of the image transformation algorithm in each image stack are consistent, and the parameters of the image transformation algorithm of each image stack are randomly set; the first expanded sample includes K input observations, and the second expanded sample includes M input observations.

[0075] It is worth noting that the data augmentation operation here only targets the original input observations (i.e., s and s′) in the data tuple, without changing the action a and reward r in the data tuple.

[0076] Consider a transformation function f:S×Γ→S, where Γ is a set of parameters that control the transformation. In particular, suppose there exists e∈Γ, and for any f(s,e)=s holds. The present invention assumes that the transition function f preserves the Q value of the state-action pair unchanged, that is:

[0077] Q θ (s,a)=Qθ (f(s,ν),a) (1)

[0078] This holds true for any ν∈Γ, where θ is the parameter of the Q-value function.

[0079] Random transformations are selected from common image transformations to adjust the input observations sampled from the offline dataset, such as random flipping, cropping, and rotation. First, the random transformation image enhancement algorithm is applied to the original observation images stacked with several consecutive frames. The data enhancement method is randomly applied in a batch of data, but a unified enhancement method is used on each data stack, which can maintain the time domain information within the stack image (such as speed, displacement, rotation angle, etc.). The training data is obtained by using a bilinear interpolation method on the transformed image, padding 2 pixels on each side, and randomly cropping the image to generate an image that is transformed by ±4 pixels based on the original image. Data enhancement methods such as Figure 3 shown.

[0080] We use traditional data augmentation methods to diversify image observations to effectively estimate the value of data in static datasets without modifying the underlying offline RL algorithm.

[0081] Step 106 : Obtain the Q values ​​of the K input observations in the first expanded sample through a preset Q value estimation network, and use the average Q value of the K input observations as the predicted Q value of the corresponding original input observation.

[0082] The Q-value estimation network can be any offline reinforcement learning network that uses Q-value estimation.

[0083] Specifically, for any state-action pair (s i ,a i ,r i ,s i′ ), from Γ according to (Enhanced data includes the original state) uniformly performs K transformations To reduce the variance of the value function estimate, use To estimate Q θ (s i ,a i ). That is, the Q values ​​of the K input observations in the first expanded sample are obtained through the preset Q value estimation network;

[0084] Calculate the average Q value of K input observations as:

[0085]

[0086] Where i is the index of the original input observation, θ is the parameter of the Q-value estimation network, (s i ,a i) is the state-action pair of the original input observation, and f(s,v) is the transformation function that transforms the original input observation by transforming v, where f retains the Q value of the state-action pair unchanged, that is, Q θ (s,a)=Q θ (f(s,ν),a);

[0087] The average Q value Q θ (s i ,a i ) as the predicted Q-value corresponding to the original input observation.

[0088] Here, since the augmented states are assumed to be in the approximate distribution of the original states and they should have similar actions under the same policy, no new actions are generated for the augmented states to save computation.

[0089] Step 108: Obtain the Q values ​​of the M input observations in the second expanded sample through a preset Q value target network, and calculate the target Q value corresponding to the original input observation based on the TD-error according to the Q values ​​of the M input observations.

[0090] The Q-value target network and the Q-value estimation network use the same network architecture.

[0091] Specifically, when the offline reinforcement learning algorithm is BCQ, the Q-values ​​of the M input observations in the second expanded sample are obtained through the Q-value target network;

[0092] The target Q value corresponding to the original input observation is calculated based on the Q value of the M input observations based on the TD-error:

[0093]

[0094] Among them, θ′ is the parameter of the Q value target network, r i is the reward value of the original input observation, γ∈(0,1] is the discount coefficient, λ is the adjustment coefficient, To adjust A transformation function that transforms the original input observations.

[0095] Step 110 : Based on the predicted Q value and the target Q value, the parameters of the Q value estimation network are updated by minimizing the mean square error (MSE) loss function.

[0096] Specifically, according to the predicted Q value and the target Q value, the Q value target network is used as the target network, and the parameters of the Q value estimation network are updated by minimizing the loss function:

[0097]

[0098] Where N is the batch size of the mini-batch, α is the learning rate, represents the expectation of the Q-value estimate.

[0099] Step 112, soft-update the parameters of the Q-value target network according to the parameters of the updated Q-value estimation network until the preset time step is reached and the updating is stopped, so that the updated Q-value estimation network is finally learned as the Q-value estimation reinforcement learning network.

[0100] Specifically, the parameters of the Q-value target network are soft-updated according to the parameters of the updated Q-value estimation network:

[0101] θ'←τθ+(1-τ)θ'

[0102] Among them, τ is the adjustment coefficient.

[0103] In the above-mentioned pixel-by-pixel Q-value estimation offline reinforcement learning method based on data enhancement, a small batch of original input observations is sampled from the offline data set, and the data of the small batch of original input observations is enhanced by a common image transformation algorithm to obtain a first expanded sample and a second expanded sample. The Q-value of the K input observations in the first expanded sample is obtained by the Q-value estimation network, and the average Q-value of the K input observations is used as the Q-value estimate of the corresponding original input observation. The Q-value of the M input observations in the second expanded sample is obtained by the Q-value target network. The Q-value target corresponding to the original input observation is calculated based on the TD-error according to the Q-value of the M input observations. The parameters of the Q-value estimation network are updated by minimizing the loss function, and then the parameters of the Q-value target network are soft-updated according to the parameters of the updated Q-value estimation network, and finally the Q-value estimation network is trained. The present invention uses a standard augmentation method to expand the training dataset through the above process. By regularizing the Q-value function of the input observation, the data near the data in the dataset, but not in the dataset, has a Q-value consistent with the observation in the dataset, avoiding overestimation of static data in the offline dataset and significantly improving the generalization of the algorithm. The present invention combines data augmentation with offline RL based on pixel observations, without requiring additional modifications to the underlying RL, making the method easy to implement and scalable to other algorithms. It has strong scalability and excellent practicality.

[0104] In one embodiment, the method further includes: obtaining a trained learning network as an expert strategy or a behavioral strategy; determining the action with the highest value and the corresponding reward corresponding to the input observation based on the expert strategy or the behavioral strategy; forming an offline data tuple (s, a, r, s′) from the current moment image stack s, the corresponding action a, the corresponding reward r, and the next moment image stack s′; and performing multiple sampling to obtain a large number of offline data tuples to form an offline data set for the visual control of the intelligent agent.

[0105] In one embodiment, the method further includes: applying a random image transformation algorithm to a small batch of original input observations to obtain a transformed image; using a bilinear interpolation algorithm on the transformed image to pad each side with 2 pixels to obtain a padded image; and performing random cropping on the padded image to obtain a data-enhanced image with the same specifications as the original image.

[0106] The reason why bilinear interpolation is used instead of simply copying the pixels at the edge of the image is that the former method replaces each pixel value with the average of the two nearest pixel values, which can obtain smoother information.

[0107] In another embodiment, the pseudo code of the DAQ algorithm is provided as follows:

[0108]

[0109] In one specific embodiment, BCQ was used as the backbone offline RL algorithm, and randomized transformations were used for data augmentation to form the DAQ algorithm. The DAQ algorithm was evaluated on Atari games and solved discrete control tasks in images. The performance, generalization, and sample efficiency of the DAQ algorithm were compared with those of the BCQ algorithm.

[0110] To obtain full information of the observation, the input observation of the real application is a stack of 4 consecutive images along the channel dimension, where the pixels are 84×84 and are rendered by the environment. Image conversion is selected as the image transformation method. For the Q-value estimation network and the Q-value target network, the number of transformations is [K=4, M=4] respectively. During training, a deep Q-learning network DQN trained for 1 million time steps is used as the behavior policy to collect 1 million (s, a, r, s′, d) tuples as an offline dataset. The DAQ method of the present invention uses the offline dataset to train the intelligent agent throughout the training process.

[0111] To generate a static replay buffer, we need to interact with the environment to gather experience through a behavioral policy. Given that observation inputs must be raw images, we chose Atari games as the environment, and trained a deep Q-learning network (DQN) as the behavioral policy to adapt to a discrete action space. After 1 million training steps, we obtained a value network that ultimately converged to stable performance. The trained network was used to generate data tuples by selecting the highest-valued action for each observation. These tuples contain the action, observation, and completion (the end of an episode in the game). The next observation can be obtained from the observation block through a few simple operations. To increase data diversity, random actions are randomly sampled from the action space within a certain number of time steps. For the remaining time steps, actions are selected using ε-greedy, with ε linearly annealed from 1 to 0.01 at the final step.

[0112] Evaluate performance:

[0113] Comparison with Offline Reinforcement Learning Methods: For comparison with the BCQ algorithm, all hyperparameters of the DAQ algorithm remain the same as those of BCQ. For reproducibility, no modifications were made to the original environment or reward function. As usual, performance was achieved using policy steps rather than ground truth steps (ground truth steps are four times as many as policy steps because each action is repeated four times). Both algorithms were trained for 1 million steps, with an episode corresponding to 1,000 steps, and a reward in the range [0, 1] per step. BCQ and CQL are canonical versions of the original algorithms applied to discrete action spaces with image-based input observations. Offline DQN is a DQN agent trained on an offline dataset without interacting with the environment. Buffer performance is the performance on the dataset generated by the policy. Each evaluation is the average performance over five episodes. Results show that the proposed DAQ algorithm outperforms or matches state-of-the-art offline RL methods on eight tasks, achieving significant gains on three tasks (DemonAttack, JamesBond, and SeaQuest).

[0114] Improved Generalization: To validate the generalization ability of the DAQ method, trained models of the DAQ and BCQ algorithms were applied to both raw and augmented data to obtain features for each input image stack. First, 20 image stacks were sampled from the dataset as the original input observations. Second, each stack was augmented four times using random transformations, generating 80 image stacks. Finally, the image processing layers of the previously trained models (including those from the BCQ and DAQ models) were used to extract intrinsic features from the observations. t-SNE was used to visualize the clustering of features to compare the generalization abilities of the two methods. Figure 4 t-SNE results of a Breakout game sample processed using the feature extraction layers of the DAQ and BCQ algorithms, respectively. The data used in the figure is expanded from the four original sampled input observations in batch transformation. All but the original four data points are generated through transformation augmentation. The figure shows that even with many unseen images, DAQ (Figure a) can better cluster them, indicating that DAQ outperforms BCQ (Figure b) in generalization. The figure shows that the DAQ algorithm clusters the augmented image stack better than BCQ in the feature map. This suggests that data augmentation enables the network to learn more generalizable representations.

[0115] Ablation experiment:

[0116] Data Efficiency: Unlike online reinforcement learning, which measures sample efficiency by the number of interactions with the environment, offline reinforcement learning evaluates data efficiency by comparing performance under limited data training. To evaluate the data efficiency of DAQ, we trained the model using 10% and 50% of the dataset and then evaluated model performance. The experiments in Table 1 show that in many cases, DAQ achieves comparable sample efficiency to BCQ when trained on the 50% dataset. Furthermore, BCQ outperforms DAQ on most tasks when trained on the 10% dataset.

[0117] Table 1 Average episode returns for different Atari games using 10% and 50% of the dataset

[0118]

[0119] It should be understood, although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0120] In one embodiment, Figure 5 As shown, a pixel-by-pixel Q-value estimation offline reinforcement learning device based on data enhancement is provided, comprising: an original input observation acquisition module 502, a data enhancement module 504, a predicted Q-value calculation module 506, a target Q-value calculation module 508, a Q-value estimation network parameter updating module 510, and a Q-value target network parameter updating module 512, wherein:

[0121] The original input observation acquisition module 502 is used to sample a small batch of original input observations from the offline dataset based on the pre-acquired offline dataset of the agent visual control;

[0122] Data augmentation module 504 is used to perform data augmentation on small batches of original input observations using a common image transformation algorithm. Each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batches, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback received by the agent after taking action a at the current moment s. Each image stack contains a number of consecutive frames of original observation images. After performing data augmentation K times and M times on image stacks s and s′, respectively, a first augmented sample and a second augmented sample are obtained. The first augmented sample includes K input observations, and the second augmented sample includes M input observations. The parameters of the image transformation algorithm within each image stack are set consistently, and the parameters of the image transformation algorithm for each image stack are set randomly.

[0123] A predicted Q-value calculation module 506 is configured to obtain the Q-values ​​of the K input observations in the first expanded sample using a preset Q-value estimation network, and use the average Q-value of the K input observations as the predicted Q-value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation;

[0124] A target Q-value calculation module 508 is configured to obtain the Q-values ​​of the M input observations in the second expanded sample through a preset Q-value target network, and calculate the target Q-value of the corresponding original input observation based on the TD-error according to the Q-values ​​of the M input observations. The Q-value target network and the Q-value estimation network use the same network architecture.

[0125] A Q-value estimation network parameter updating module 510 is configured to update the parameters of the Q-value estimation network by minimizing the mean square error (MSE) loss function based on the predicted Q-value and the target Q-value;

[0126] The Q-value target network parameter update module 512 is used to soft-update the parameters of the Q-value target network according to the parameters of the updated Q-value estimation network until the updating stops at a preset time step; the updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

[0127] The original input observation acquisition module 502 is also used to obtain a trained learning network as an expert strategy or a behavioral strategy; according to the expert strategy or the behavioral strategy, the action with the highest value corresponding to the input observation and the corresponding reward are determined; an offline data tuple (s, a, r, s′) is formed by the image stack s at the current moment, the corresponding action a, the corresponding reward r and the image stack s′ at the next moment; multiple samplings are performed to obtain a large number of offline data tuples, which constitute an offline data set for the visual control of the intelligent agent.

[0128] The data enhancement module 504 is further configured to apply a random image transformation algorithm to a small batch of original input observations to obtain a transformed image; use a bilinear interpolation algorithm on the transformed image to pad each side with 2 pixels to obtain a padded image; and perform random cropping on the padded image to obtain a data-enhanced image with the same specifications as the original image.

[0129] The predicted Q value calculation module 506 is further configured to obtain the Q values ​​of the K input observations in the first expanded sample through a preset Q value estimation network; and calculate the average Q value of the K input observations as:

[0130]

[0131] Where i is the index of the original input observation, θ is the parameter of the Q-value estimation network, (s i ,a i ) is the state-action pair of the original input observation, and f(s,v) is the transformation function that transforms the original input observation by adjusting v, where f retains the Q value of the state-action pair unchanged, that is, Q θ (s,a)=Q θ (f(s,ν),a); the average Q value Q θ (s i ,a i ) as the predicted Q-value corresponding to the original input observation.

[0132] The target Q value calculation module 508 is further configured to obtain the Q values ​​of the M input observations in the second expanded sample through a preset Q value target network; and calculate the target Q value of the corresponding original input observation based on the TD-error according to the Q values ​​of the M input observations:

[0133]

[0134] Among them, θ′ is the parameter of the Q value target network, r i is the reward value of the original input observation, γ∈(0,1] is the discount coefficient, λ is the adjustment coefficient, To adjust A transformation function that transforms the original input observations.

[0135] The Q-value estimation network parameter updating module 510 is further configured to update the parameters of the Q-value estimation network by minimizing the mean square error (MSE) loss function based on the predicted Q-value and the target Q-value, with the Q-value target network as the target network:

[0136]

[0137] Where N is the batch size of the mini-batch, α is the learning rate, represents the expectation of the Q-value estimate.

[0138] The Q-value target network parameter updating module 512 is further configured to soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network:

[0139] θ'←τθ+(1-τ)θ'

[0140] Among them, τ is the update coefficient.

[0141] Regarding the specific limitations of the offline reinforcement learning device for pixel-by-pixel Q-value estimation based on data enhancement, please refer to the limitations of the offline reinforcement learning method for pixel-by-pixel Q-value estimation based on data enhancement above, which will not be repeated here. Each module in the above-mentioned offline reinforcement learning device for pixel-by-pixel Q-value estimation based on data enhancement can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0142] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a pixel-by-pixel Q value estimation offline reinforcement learning method based on data enhancement is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0143] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0146] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A pixel-by-pixel Q-value estimation offline reinforcement learning method based on data enhancement, characterized in that: The method comprises: Based on a pre-acquired offline dataset of agent visual control, sampling a small batch of raw input observations from the offline dataset; The small batch of original input observations is data-augmented by a common image transformation algorithm; each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batches of data, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback obtained by the agent after taking action a at the current moment s; each of the image stacks contains a number of original observation images stacked in successive frames; after image stacks s and s′ are data-augmented K times and M times, respectively, a first augmented sample and a second augmented sample are obtained; the first augmented sample includes K input observations, and the second augmented sample includes M input observations; the parameters of the image transformation algorithm in each image stack are set consistently, and the parameters of the image transformation algorithm of each image stack are set randomly; Obtaining Q values ​​of K input observations in the first expanded sample through a preset Q-value estimation network, and taking the average Q value of the K input observations as the predicted Q value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation; Obtaining Q values ​​of the M input observations in the second expanded sample through a preset Q value target network, and calculating a target Q value corresponding to the original input observation based on the TD-error according to the Q values ​​of the M input observations; the Q value target network and the Q value estimation network use the same network architecture; According to the predicted Q value and the target Q value, the parameters of the Q value estimation network are updated by minimizing the MSE mean square error loss function; Soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network until the updating stops at a preset time step; The updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

2. The method according to claim 1, characterized in that Based on pre-acquired offline datasets for agent visual control, including: Obtain the trained learning network as an expert strategy or behavior strategy; Determine the action with the highest value and the corresponding reward corresponding to the input observation according to the expert strategy or behavioral strategy; The current moment image stack s, the corresponding action a, the corresponding reward r and the next moment image stack s′ constitute an offline data tuple (s, a, r, s′); After multiple sampling, a large number of offline data tuples are obtained, which constitute the offline dataset for the visual control of the intelligent agent.

3. The method according to claim 1, characterized in that The mini-batch of raw input observations is augmented using common image transformation algorithms, including: Applying a randomized image transformation algorithm to the mini-batch of original input observations to obtain a transformed image; Using a bilinear interpolation algorithm on the transformed image, padding each side with 2 pixels to obtain a padded image; Random cropping is performed on the padded image to obtain a data-enhanced image with the same specifications as the original image.

4. The method according to claim 1, wherein Obtaining Q values ​​of K input observations in the first expanded sample through a preset Q value estimation network, and using the average Q value of the K input observations as the predicted Q value of the corresponding original input observation, including: Obtaining Q values ​​of the K input observations in the first expanded sample through a preset Q value estimation network; Calculate the average Q value of K input observations as: Where i is the index of the original input observation, θ is the parameter of the Q-value estimation network, (s i ,a i ) is the state-action pair of the original input observation, and f(s,v) is the transformation function that transforms the original input observation by adjusting v, where f retains the Q value of the state-action pair unchanged, that is, Q θ (s,a)=Q θ (f(s,ν),a); The average Q value Q θ (s i ,a i ) as the predicted Q-value corresponding to the original input observation.

5. The method according to claim 4, characterized in that The Q values ​​of the M input observations in the second expanded sample are obtained through a preset Q value target network. According to the Q values ​​of the M input observations, a target Q value corresponding to the original input observation is calculated based on the TD-error, including: Obtaining Q values ​​of the M input observations in the second expanded sample through a preset Q value target network; According to the Q value of the M input observations, the target Q value corresponding to the original input observation is calculated based on the TD-error: Among them, θ′ is the parameter of the Q value target network, r i is the reward value of the original input observation, γ∈(0,1] is the discount coefficient, λ is the adjustment coefficient, To adjust A transformation function that transforms the original input observations.

6. The method according to claim 5, characterized in that According to the predicted Q value and the target Q value, updating the parameters of the Q value estimation network by minimizing the MSE mean square error loss function includes: According to the predicted Q value and the target Q value, the parameters of the Q value estimation network are updated by minimizing the MSE mean square error loss function with the Q value target network as the target network: Where N is the batch size of the mini-batch, α is the learning rate, represents the expectation of the Q-value estimate.

7. The method according to claim 6, characterized in that Soft updating the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network includes: Soft update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network: θ'←τθ+(1-τ)θ' Among them, τ is the update coefficient.

8. A pixel-by-pixel Q-value estimation offline reinforcement learning device based on data enhancement, characterized in that: The device comprises: A raw input observation acquisition module is used to sample small batches of raw input observations from a pre-acquired offline dataset of the agent visual control; A data augmentation module is used to perform data augmentation on the original input observations of the small batch using a common image transformation algorithm; each small batch of data consists of batch data groups (s, a, r, s′), where batch is the number of small batch data, s represents the image stack at the current moment, s′ represents the image stack at the next moment, a represents the action taken by the agent at the current moment, and r represents the environmental reward feedback obtained by the agent after taking action a at the current moment s; each of the image stacks contains a number of original observation images stacked in successive frames; after image stacks s and s′ are subjected to K and M data augmentation respectively, a first augmented sample and a second augmented sample are obtained respectively; the first augmented sample includes K input observations, and the second augmented sample includes M input observations; the parameter settings of the image transformation algorithm in each image stack are consistent, and the image transformation algorithm parameters of each image stack are randomly set; a predicted Q-value calculation module, configured to obtain the Q-values ​​of the K input observations in the first expanded sample through a preset Q-value estimation network, and use the average Q-value of the K input observations as the predicted Q-value of the corresponding original input observation; the Q-value estimation network is any offline reinforcement learning network that uses Q-value estimation; a target Q-value calculation module, configured to obtain the Q-values ​​of the M input observations in the second expanded sample through a preset Q-value target network, and calculate the target Q-value of the corresponding original input observation based on the TD-error according to the Q-values ​​of the M input observations; the Q-value target network and the Q-value estimation network use the same network architecture; A Q-value estimation network parameter updating module is used to update the parameters of the Q-value estimation network by minimizing the MSE mean square error loss function according to the predicted Q value and the target Q value; The Q-value target network parameter update module is used to soft-update the parameters of the Q-value target network according to the updated parameters of the Q-value estimation network until the updating is stopped at a preset time step; the updated Q-value estimation network is used as the final learned Q-value estimation reinforcement learning network.

9. The device according to claim 8, characterized in that The data enhancement module is further configured to: Applying a randomized image transformation algorithm to the mini-batch of original input observations to obtain a transformed image; Using a bilinear interpolation algorithm on the transformed image, padding each side with 2 pixels to obtain a padded image; Random cropping is performed on the padded image to obtain a data-enhanced image with the same specifications as the original image.

10. The device according to claim 8, characterized in that The predicted Q value calculation module is also used for: Obtaining Q values ​​of the K input observations in the first expanded sample through a preset Q value estimation network; Calculate the average Q value of K input observations as: Where i is the index of the original input observation, θ is the parameter of the value function of the Q-value estimation network, (s i ,a i ) is the state-action pair of the original input observation, and f(s,v) is the transformation function that transforms the original input observation by adjusting v, where f retains the Q value of the state-action pair unchanged, that is, Q θ (s,a)=Q θ (f(s,ν),a); The average Q value Q θ (s i ,a i ) as the predicted Q-value corresponding to the original input observation.

Images