A deep reinforcement learning network training method and device for Atari games

Abstract

The application discloses a deep reinforcement learning network training method and device for Atari games, and the method comprises the following steps: acquiring game image data in an Atari game platform; inputting the game image data into a deep reinforcement learning model to extract image features, and using the extracted image features to train a model and an agent in the game, so as to obtain a trained deep reinforcement learning model and a trained agent; wherein the deep reinforcement learning model comprises a deep separable convolution with a first convolution kernel size, a deep separable convolution with a second convolution kernel size and a full connection layer comprising an H-Swish activation function. The LDQN model provided by the application not only guarantees the feature extraction capability of the image, but also further optimizes the network structure of the DQN, so that the same performance as the original DQN is achieved at a low cost, and the performance of the LDQN model is still superior to that of a human being in the Atari game.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent algorithm applied in Atari games, and particularly relates to a deep reinforcement learning network training method and device for Atari games. BACKGROUND

[0002] In recent years, deep learning (DL) technology has been developing continuously. With the excellent feature representation capability of deep neural networks, many difficult problems in academic and industrial fields have been solved and important research results have been achieved. Reinforcement learning (RL) as an important method for solving sequential decision-making gives the agent the ability of self-supervised learning, which can interact with the environment autonomously and constantly correct the strategy through the reward obtained. The introduction of deep neural networks has made great progress in reinforcement learning and has derived deep reinforcement learning (DRL). Deep reinforcement learning has made significant breakthroughs in various fields in recent years.

[0003] In the field of games: the agent in the Atari series of video games uses deep reinforcement learning algorithm to directly learn image pixels, which has surpassed the human level. AlphaGo developed by DeepMind has defeated top human players. The final version of AlphaZero has defeated AlphaGo through self-learning. The absolute enlightenment AI developed by Tencent AI Lab has defeated top human players in the game of “King Glory” and won the championship in the Kaggle football AI competition. AlphaStar of Open AI has defeated professional players in the game of “StarCraft II” with a score of 5:0, showing the excellent performance of AI in multi-agent and complex state action space. In the field of business, Facebook has opened the Horizon reinforcement learning platform for developing and deploying recommendation systems based on deep reinforcement learning. Alibaba uses deep reinforcement learning to improve user click rate in the double eleven activity. Sliver proposes to use deep reinforcement learning to build a system for customer interaction. In the field of control, it is currently possible to use DRL method to learn the control strategy of the robot from the input of the real world camera, for example, Stanford University uses DRL method to control the helicopter to complete the stunt flight, reaching the level of human beings.

[0004] Deep reinforcement learning is the combination of deep learning and reinforcement learning. Deep learning uses representation learning to extract data, without the need for feature selection, dimension compression, format conversion and other data processing methods. It has stronger feature representation ability than traditional machine learning methods. Through the combination of low-level features, it forms more abstract high-level features and realizes the distribution representation of data. Reinforcement learning originated from the optimal control theory in cybernetics. It is mainly used to solve time series decision problems. Through continuous interaction and trial and error with the environment, the optimal strategy for a specific task is obtained and the cumulative expected return of the task is maximized.

[0005] The mainstream method of traditional reinforcement learning mainly includes Monte Carlo method and time difference classification method. The former is an unbiased estimate with large variance, and the latter uses limited step bootstrap method with small variance but introduces bias. Experimental verification shows that the above methods are not ideal in high-dimensional state action space tasks, and even the algorithm is difficult to converge. The reason is that the above methods need to perform policy evaluation first to obtain state value function or action value function information, and then use the value function information to improve the current policy. The algorithm uses table type reinforcement learning method to evaluate the value function, establishes a table, and for state value function, the index is state, and for action value function, the index is state action pair. The iterative update of the value function is the update of the data in the table. For high-dimensional state action space tasks, the table method is difficult to evaluate the value function corresponding to all state action pairs. To solve the dimension disaster problem of the table method in processing high-dimensional state action space tasks, researchers propose to use function approximation method for prediction, which uses parameterized method to approximate the value function. The approximate value function is no longer represented as a table, but as a parameterized function with weight vector. By adjusting the weight, different functions can be obtained. According to the different approximation methods, it can be divided into linear approximation method and nonlinear approximation method. Linear approximation methods include polynomial basis, Fourier basis, coarse coding, tile coding and other methods. The advantage is that it can converge to the global optimum, and the disadvantage is that the representation ability is limited. Since the basis function is fixed, for complex functions, the number of basis functions is too small and the form is fixed, which cannot get good approximation effect. Nonlinear approximation methods have strong performance, including kernel function approximation, memory-based function approximation and other methods. Compared with linear approximation methods, they have made great progress, but experimental results show that the performance for complex tasks is still not good. Until the emergence of deep learning, the combination of deep neural network reinforcement learning has greatly improved the algorithm performance.

[0006] Deep reinforcement learning combines the structure of deep learning with the ideas of reinforcement learning to solve decision-making problems. It leverages the powerful representational capabilities of deep neural networks to fit any component of reinforcement learning, including state-value functions, action-value functions, policies, and models, using the weights of the deep neural network as fitting parameters. Deep reinforcement learning is primarily used to solve high-dimensional state-action space tasks, integrating the powerful understanding capabilities of deep learning in feature representation problems with the decision-making capabilities of reinforcement learning, achieving end-to-end learning. The emergence of deep reinforcement learning has made reinforcement learning technology truly practical, enabling the solution of complex problems in real-world scenarios. The most representative algorithm, DQN, was proposed in the Atari video game series, learning directly from image pixels through an end-to-end method and achieving results surpassing human players, marking the beginning of the rapid development of deep reinforcement learning. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems in the related art.

[0008] This invention proposes a deep reinforcement learning network training method for Atari games. Specifically targeting the characteristics of Atari video games, it proposes a novel lightweight, high-performance deep reinforcement learning network, Lite Deep Q Learning Network (LDQN). This model structure is based on Deep Q Learning Network and incorporates design ideas from advanced convolutional neural networks such as MobileNets, aiming to reduce the computational cost and number of parameters of the deep reinforcement learning model while ensuring efficient extraction of game video image features. Furthermore, this invention introduces the H-Swish activation function, further achieving the goal of lightweighting and low computational cost through this high-performance, low-computational-cost activation function. The proposed LDQN model not only maintains its image feature extraction capabilities but also further optimizes the DQN network structure, achieving the same performance as the original DQN at a low cost, and still exhibiting superior performance to human users in Atari games.

[0009] Another objective of this invention is to provide a deep reinforcement learning network training device for Atari games.

[0010] To achieve the above objectives, this invention proposes a method for training deep reinforcement learning networks for Atari games, comprising:

[0011] Retrieve game image data from the Atari gaming platform;

[0012] The game image data is input into a deep reinforcement learning model for image feature extraction. The extracted image features are then used to train the model and the agent in the game to obtain a trained deep reinforcement learning model and a trained agent. The deep reinforcement learning model includes a depthwise separable convolution with a first kernel size, a depthwise separable convolution with a second kernel size, and a fully connected layer containing the H-Swish activation function.

[0013] In addition, the deep reinforcement learning network training method for Atari games according to the above embodiments of the present invention may also have the following additional technical features:

[0014] Furthermore, in one embodiment of the present invention, the game image data undergoes data preprocessing, including:

[0015] The three-channel RGB game image data of the first pixel size is compressed to obtain a grayscale image of the second pixel size;

[0016] The grayscale image of the second pixel size is cropped to obtain the central game information area image of the third pixel size;

[0017] The background pixel value of the central game information area image of the third pixel size is calculated to obtain the game image data after removing the background, and the game image data after removing the background is subjected to pixel normalization processing.

[0018] Furthermore, in one embodiment of the present invention, the depth-separable convolution of the first convolution kernel size includes a depth-separable convolution of the third convolution kernel size and a pointwise convolution; the depth-separable convolution of the second convolution kernel size includes a depth-separable convolution of the fourth convolution kernel size and a pointwise convolution.

[0019] Furthermore, in one embodiment of the present invention, the computational cost expressions for the depthwise separable convolution and the standard convolution are respectively:

[0020]

[0021]

[0022] in, and This represents the computational cost of depthwise separable convolution and standard convolution. This represents the width and height of the input feature map. Represents the spatial dimension of the convolution kernel. It is the number of input channels. It refers to the number of output channels;

[0023] The computational cost ratio between the depthwise separable convolution and the standard convolution is:

[0024]

[0025] Furthermore, in one embodiment of the present invention, the expression of the H-Swish activation function is:

[0026]

[0027]

[0028] To achieve the above objectives, another aspect of the present invention provides a deep reinforcement learning network training device for Atari games, comprising:

[0029] The data acquisition module is used to acquire game image data from the Atari gaming platform;

[0030] The model training module is used to input the game image data into the deep reinforcement learning model for image feature extraction, and to use the extracted image features to train the model and the agent in the game to obtain a trained deep reinforcement learning model and a trained agent; wherein, the deep reinforcement learning model includes a depthwise separable convolution with a first kernel size, a depthwise separable convolution with a second kernel size, and a fully connected layer containing the H-Swish activation function.

[0031] The present invention discloses a method and apparatus for training deep reinforcement learning networks for Atari games, utilizing a novel lightweight, high-performance deep reinforcement learning network, Lite Deep Q Learning Network (LDQN). This model structure is based on Deep Q Learning Network and incorporates design principles from advanced convolutional neural networks such as MobileNets, aiming to reduce the computational cost and number of parameters of the deep reinforcement learning model while ensuring efficient extraction of game video image features.

[0032] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0034] Figure 1 This is a flowchart of a deep reinforcement learning network training method for Atari games according to an embodiment of the present invention;

[0035] Figure 2 This is a network structure diagram of the Lite Deep Q Learning Network according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of depth-separable convolution according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a deep reinforcement learning network training device for Atari games according to an embodiment of the present invention. Detailed Implementation

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] The following description, with reference to the accompanying drawings, describes a method and apparatus for training deep reinforcement learning networks for Atari games, according to embodiments of the present invention.

[0041] Figure 1 This is a flowchart of a deep reinforcement learning network training method for Atari games according to an embodiment of the present invention.

[0042] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0043] S1, retrieve game image data from the Atari gaming platform;

[0044] S2, input game image data into the deep reinforcement learning model for image feature extraction, and use the extracted image features to train the model and agent in the game to obtain a trained deep reinforcement learning model and a trained agent; wherein, the deep reinforcement learning model includes a depthwise separable convolution with a first convolution kernel size, a depthwise separable convolution with a second convolution kernel size, and a fully connected layer containing the H-Swish activation function.

[0045] In summary, the LDQN model proposed in this invention not only guarantees its feature extraction capability for images but also further optimizes the network structure of DQN, enabling it to achieve the same performance as the original DQN at a low cost, and still outperform human performance in Atari games.

[0046] The weakly supervised learning method based on duplex similarity measurement according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] In Atari's various video games, the standard for judging the quality of an algorithm is its game score. First, the game footage is extracted and preprocessed, including cropping and background removal, to make it suitable for input into a deep reinforcement learning network. The image processing method and quality also affect the final agent's performance. However, on some lightweight devices and devices with low computing power, the computational cost and number of parameters are relatively high.

[0048] To reduce the computational cost and number of parameters in deep reinforcement learning models, this invention proposes a lightweight, high-performance deep reinforcement learning algorithm, Lite Deep Q Learning Network, with the following specific steps:

[0049] As an example, preprocessing images in Atari video games:

[0050] Understandably, to ensure that effective visuals from Atari video games are fed into the Lite Deep Q Learning Network model for more efficient feature extraction, a series of preprocessing operations are performed on the game image data before it is input into the Lite Deep Q Learning Network. First, the game image is cropped to extract the effective region, with a size of 80×80 pixels. Then, the background of the 80×80 pixel image is removed. Finally, the 80×80 pixel image is normalized. The specific processing flow is as follows:

[0051] (1) Grayscale conversion: Compress the three-channel RGB video game screen image into a grayscale image.

[0052] (2) Downsampling: The original 210×160 pixel video game image is compressed to 110×84 pixels.

[0053] (3) Cropping: Cropping the image to obtain the central game information area containing the effective information, and finally obtaining an image with a size of 80×80 pixels.

[0054] (4) Background removal: The background pixels of the cropped video game image are calculated, and then the background pixel value is subtracted from all pixels to obtain the game image after the background is removed.

[0055] (5) Pixel normalization processing: The game image after removing the background is subjected to pixel normalization processing.

[0056] Furthermore, the above preprocessing operations on the game video images have the following technical effects:

[0057] 1) To speed up the game, calculating the Q-value is the most time-consuming step.

[0058] 2) Reduce noise: Overly dense frames can lead to a lot of repetitive information, making it easy for previous actions to be rejected.

[0059] 3) Shorten the time interval between the reward signal and the specific action to speed up training.

[0060] 4) Using four key frames as input can help the agent obtain more and more effective information for learning and training.

[0061] 5) Reward Clipping loses some information, but ensures that the reward scale is the same for different games, so that the same parameters can be used for training.

[0062] Furthermore, this invention is an improvement upon the original DQN network structure, which is lightweight. The specific steps of its Deep Q Learning algorithm are shown in Table 1:

[0063] Table 1

[0064]

[0065] Deep Q Learning Algorithm

[0066]

[0067] Initialize replay memory to capacity

[0068] Initialize action-value function with random weights

[0069] For , do

[0070] Initialize sequence and preprocessed sequenced

[0071] For , do

[0072] With probability select a random action

[0073] Otherwise select

[0074] Execute action in emulator and observe reward and image

[0075] Set , , and preprocess

[0076] Store transition minibatch of transitions from

[0077] Set

[0078] Perform a gradient descent step on according to

[0079] End for

[0080] End for

[0081]

[0082] Furthermore, to reduce the computational cost and number of parameters of the original DQN, this invention fully incorporates the deep separable convolution concept from MobileNets, an excellent lightweight convolutional neural network in the field of deep learning, into the deep reinforcement learning model DQN to construct the Lite Deep Q Learning Network proposed in this invention, such as... Figure 2 As shown.

[0083] Specifically, the first 8×8 convolutional layer in the original DQN model is modified to a depthwise separable convolution, which consists of an 8×8 depthwise convolution and a 1×1 pointwise convolution. The depthwise separable convolution performs the same function as the standard convolution, but this structure significantly reduces the number of model parameters and computational cost. The depthwise separable convolution used in this invention is as follows: Figure 3 As shown, the computational cost expressions for depthwise separable convolution and standard convolution are as follows:

[0084]

[0085]

[0086] in, and This represents the computational cost of depthwise separable convolution and standard convolution. This represents the width and height of the input feature map. This represents the spatial dimension of the convolution kernel. This refers to the number of input channels. This refers to the number of output channels.

[0087] The computational cost ratio between depthwise separable convolution and standard convolution is:

[0088]

[0089] By adopting the idea of ​​depthwise separable convolution, the first layer of the original DQN model, which is 8×8 convolution, is modified to a depthwise separable convolution. This not only reduces the computational parameters of the model and lowers the computational cost, but also speeds up its training, making it more suitable for lightweight mobile and smart devices.

[0090] The 4×4 ordinary convolution in layer 2 is modified to a depthwise separable convolution. This depthwise separable convolution consists of a 4×4 depthwise convolution and a 1×1 pointwise convolution, in order to further reduce the number of model parameters and computational cost, and reduce its CPU load and memory consumption on the application device.

[0091] Furthermore, in the modified DQN network structure, the first layer's 8×8 ordinary convolutions are changed to a depthwise convolution with an 8×8 kernel and a 1×1 pointwise convolution. This outputs a 16×20×20 effective feature layer. The second layer's 4×4 ordinary convolutions are also changed to a depthwise convolution with a 4×4 kernel and a 1×1 pointwise convolution. This outputs a 32×9×9 effective feature layer. After these two convolutional layers and a subsequent fully connected layer, a low-cost, high-performance H-Swish activation function is used. The function's computational definition is as follows:

[0092]

[0093]

[0094] Furthermore, the modified DQN model, Lite Deep Q Learning Network, was trained using a video game environment on the Atari gaming platform. The Atari gaming platform's game environment was accessed via the gym library in Python. The hyperparameters for training were set as follows: batch size of 128, learning rate of 0.001, reward decay gamma of 0.99, 3000 episodes, a maximum step size of 1500 for each episode, an initial epsilon of 1, a decay of 0.995, and a minimum value of 0.05. The trained agent outperformed human performance in Atari games.

[0095] The deep reinforcement learning network training method for Atari games according to embodiments of the present invention not only ensures its feature extraction capability for images but also further optimizes the network structure of DQN, enabling it to achieve the same performance as the original DQN at low cost and still outperform human performance in Atari games.

[0096] To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a deep reinforcement learning network training device 10 for Atari games, which includes a data acquisition module 100 and a model training module 200.

[0097] Data acquisition module 100 is used to acquire game image data from the Atari gaming platform;

[0098] The model training module 200 is used to input game image data into the deep reinforcement learning model for image feature extraction, and use the extracted image features to train the model and the agent in the game to obtain a trained deep reinforcement learning model and a trained agent. The deep reinforcement learning model includes a depthwise separable convolution with a first kernel size, a depthwise separable convolution with a second kernel size, and a fully connected layer containing the H-Swish activation function.

[0099] Furthermore, prior to the aforementioned model training module 200, the device 10 also includes a preprocessing module for:

[0100] The three-channel RGB game image data of the first pixel size is compressed to obtain a grayscale image of the second pixel size;

[0101] The grayscale image of the second pixel size is cropped to obtain the central game information area image of the third pixel size;

[0102] The background pixel value of the central game information area image of size 3 pixels is calculated to obtain the game image data after removing the background, and the game image data after removing the background is then subjected to pixel normalization processing.

[0103] Furthermore, the depthwise separable convolution of the first kernel size in the above-mentioned model training module 200 includes depthwise convolution of the third kernel size and pointwise convolution; the depthwise separable convolution of the second kernel size includes depthwise convolution of the fourth kernel size and pointwise convolution.

[0104] Furthermore, the computational cost expressions for depthwise separable convolution and standard convolution in the model training module 200 are as follows:

[0105]

[0106]

[0107] in, and This represents the computational cost of depthwise separable convolution and standard convolution. This represents the width and height of the input feature map. Represents the spatial dimension of the convolution kernel. It is the number of input channels. It refers to the number of output channels;

[0108] The computational cost ratio between depthwise separable convolution and standard convolution is:

[0109]

[0110] Furthermore, the expression for the H-Swish activation function in the aforementioned model training module 200 is:

[0111]

[0112]

[0113] The deep reinforcement learning network training device for Atari games according to embodiments of the present invention not only ensures its feature extraction capability for images but also further optimizes the network structure of DQN, enabling it to achieve the same performance as the original DQN at low cost, and still outperform human performance in Atari games.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for training deep reinforcement learning networks for Atari games, characterized in that, Includes the following steps: Retrieve game image data from the Atari gaming platform; The game image data is input into a deep reinforcement learning model for image feature extraction. The extracted image features are then used to train the model and the agent in the game to obtain a trained deep reinforcement learning model and a trained agent. The deep reinforcement learning model includes a depthwise separable convolution with a first kernel size, a depthwise separable convolution with a second kernel size, and a fully connected layer containing an H-Swish activation function. Data preprocessing of the game image data includes: The three-channel RGB game image data of the first pixel size is compressed to obtain a grayscale image of the second pixel size; The grayscale image of the second pixel size is cropped to obtain the central game information area image of the third pixel size; The background pixel value of the central game information area image of the third pixel size is calculated to obtain the game image data after removing the background, and the game image data after removing the background is subjected to pixel normalization processing. The first convolution kernel size of depth separable convolution includes the third convolution kernel size of depth convolution and pointwise convolution; the second convolution kernel size of depth separable convolution includes the fourth convolution kernel size of depth convolution and pointwise convolution. The computational cost expressions for the depthwise separable convolution and the standard convolution are as follows: in, and This represents the computational cost of depthwise separable convolution and standard convolution. This represents the width and height of the input feature map. Represents the spatial dimension of the convolution kernel. It is the number of input channels. It refers to the number of output channels; The computational cost ratio between the depthwise separable convolution and the standard convolution is: The expression for the H-Swish activation function is: 。 2. A deep reinforcement learning network training device for Atari games, characterized in that, include: The data acquisition module is used to acquire game image data from the Atari gaming platform; The model training module is used to input the game image data into the deep reinforcement learning model for image feature extraction, and use the extracted image features to train the model and the agent in the game to obtain a trained deep reinforcement learning model and a trained agent; wherein, the deep reinforcement learning model includes a depthwise separable convolution with a first convolution kernel size, a depthwise separable convolution with a second convolution kernel size, and a fully connected layer containing the H-Swish activation function; Prior to the model training module, the apparatus further includes a preprocessing module for: The three-channel RGB game image data of the first pixel size is compressed to obtain a grayscale image of the second pixel size; The grayscale image of the second pixel size is cropped to obtain the central game information area image of the third pixel size; The background pixel value of the central game information area image of the third pixel size is calculated to obtain the game image data after removing the background, and the game image data after removing the background is subjected to pixel normalization processing. The model training module includes a first kernel-size depthwise separable convolution, which includes a third kernel-size depthwise convolution and a pointwise convolution; and a second kernel-size depthwise separable convolution, which includes a fourth kernel-size depthwise convolution and a pointwise convolution. The computational cost expressions for depthwise separable convolution and standard convolution in the model training module are as follows: in, and This represents the computational cost of depthwise separable convolution and standard convolution. This represents the width and height of the input feature map. Represents the spatial dimension of the convolution kernel. It is the number of input channels. It refers to the number of output channels; The computational cost ratio between the depthwise separable convolution and the standard convolution is: The expression for the H-Swish activation function in the model training module is: 。

Images