Sequence parallel model-based reinforcement learning method

Abstract

The invention discloses a sequence parallel reinforcement learning method based on a model. On the basis of a linear attention mechanism of a linear recurrent neural network, a world model capable of being fully parallelized is constructed by using root-mean-square layer normalization and a gating unit; meanwhile, a parallel scanning algorithm is used for parallelizing the qualification trace estimation process in the strategy learning stage on the aspect of time series. Compared with the prior art, the parallel scanning algorithm is combined, and on the premise that high sample efficiency is kept, the hardware efficiency of model-based reinforcement learning is improved in the two aspects of world model training and qualification trace estimation at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning, and in particular relates to the improvement of hardware efficiency of model-based reinforcement learning. Background Art

[0002] Model-based reinforcement learning (MBRL) is widely regarded as having great potential to significantly improve sample efficiency by learning world model training strategies. Many MBRL methods proposed so far achieve the same asymptotic performance as model-free RL methods, but require less interaction data. In particular, some recent MBRL methods have achieved sample efficiency that exceeds human level in Atari video games and real-world problems such as robot control, healthcare, and advertising recommendation systems. However, such reinforcement learning algorithms consume more computing resources and have low training efficiency, so they need to be accelerated. Summary of the invention

[0003] The purpose of the present invention is to solve the defects in the prior art and provide a sequential parallel model-based reinforcement learning method.

[0004] The specific technical solutions adopted by the present invention are as follows:

[0005] In a first aspect, the present invention provides a sequential parallel model-based reinforcement learning method, which comprises the following steps:

[0006] S1. Construct a time series parallel world model to simulate the operating rules of the game environment; the world model includes an encoder network, a decoder network, a sequence modeling network, an environment dynamics prediction network and a reward prediction network, which are used to map the observation input obtained from the game environment into a low-dimensional latent space, and use the sequence modeling network and the environment dynamics prediction network in the latent space to realize future predictions;

[0007] S2. Build a reinforcement learning strategy network and value network to continuously obtain observations at different times from the game environment. t ; For each current moment t, observe the current moment t obtained in the game environment t Input the encoder network to obtain the latent space vector z at the current time t t , the latent space vector z at the current time t t The latent vector h representing the historical context information output by the sequence modeling network t-1 After splicing, it is used as the input of the strategy network and the value network at the same time. The sequence modeling network of the world model is set to adopt the time series serial mode, and the strategy network outputs the action a that controls the interaction of the characters in the game environment t The value network outputs the estimated value v of the action performed by the game character in the current game environment statet , the observation of the next moment after the game environment receives the action feedback t+1 , the single-step reward r obtained by the game character t And the signal whether the game is terminated t , and finally collect observations, actions, and single-step rewards into the experience replay cache;

[0008] S3, randomly sampling interaction sequence data from the experience replay cache to train the world model, wherein the interaction sequence data includes observation sequence, action sequence and reward sequence, wherein the sequence modeling network of the world model is trained in a time series parallel mode;

[0009] S4. Flatten the interaction sequence data randomly sampled from the experience replay cache. The data at each time t in the interaction sequence data is used as the starting point for prediction. The game character action a sampled by the policy network is used. t Interact with the world model and transform the latent space vector of the next moment observed by the world model into Prediction Rewards Save to temporary cache;

[0010] S5. The latent space vector observed in the temporary cache and historical context t-1 Input value network to get value estimate Compute the temporal difference target of the value network using parallelized eligibility trace estimation and train the value network;

[0011] S6. Use the temporal difference target to calculate the advantage estimate, use the advantage estimate to train the policy network, and use the trained policy network to control the interactive actions of the characters in the game environment.

[0012] Preferably, in S1, if the observations obtained from the game environment are image observations, the encoder network and the decoder network use a convolutional neural network including a two-dimensional convolutional output layer and a batch normalization layer.

[0013] Preferably, in S1, if the observations obtained from the game environment are vector observations, the encoder network and the decoder network use a multilayer perceptron including a fully connected output layer and a batch normalization layer.

[0014] Preferably, the strategy network and the value network both use multi-layer perceptrons.

[0015] Preferably, the environment dynamics prediction network and the reward prediction network both use multi-layer perceptrons.

[0016] Preferably, the sequence modeling network adopts a linear recurrent neural network.

[0017] Preferably, in S3, the specific steps of randomly sampling interaction sequence data from the experience replay buffer to train the world model are:

[0018] S31, randomly sample K interactive sequence data with a sequence length of T from the experience playback cache, and the data at each time t in the interactive sequence data includes the observation o corresponding to the time t 、Action a t and reward r t ;

[0019] S32, the observation o of the interaction sequence data t Input the encoder network to get the latent space vector z t ; The latent space vector z t Input the decoder network to obtain the reconstructed observation Concatenate the latent space vector z corresponding to each moment t and action a t , input the sequence modeling network in parallel in sequence order to obtain the historical context h corresponding to each moment t ; The historical context h t Input the reward prediction network to get the predicted value of the reward The historical context h t Input the environment dynamic prediction network to get the predicted value of the latent space vector

[0020] S33, according to the output of the decoder network Compute regression loss, predicting the random distribution of network outputs based on rewards Calculate the maximum log-likelihood loss and combine the two losses to calculate the prediction loss L of the world model pred t (θ):

[0021]

[0022] where θ is the neural network parameters of the world model;

[0023] S34. Model the latent space vector output by the encoder as a joint distribution of multiple category losses Model the latent space vector output by the environment dynamics prediction network as a joint distribution of multiple class losses The KL divergence with threshold clipping is used as the loss function, where the clipping threshold is set to 1, to calculate the dynamic prediction loss L dyn t (θ):

[0024]

[0025] Where: sg(·) represents gradient cutoff;

[0026] S35. Use the threshold-clipped KL divergence as the representation loss of the encoder output latent space vector and calculate the representation consistency loss L rep t (θ):

[0027]

[0028] S36. Use three weight hyperparameters β for the three loss functions pred , β rep , β dyn Weighted, calculate the total loss function L of the world model total (θ):

[0029]

[0030] S37, using the sampled K interactive sequence data to minimize the total loss function L total (θ) is the target, and the neural network parameter θ of the world model is optimized. During the training process, the sequence modeling network of the world model is trained in a time series parallel mode.

[0031] In a second aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the sequential parallel model-based reinforcement learning method as described in any one of the first aspects above.

[0032] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the serial parallel model-based reinforcement learning method as described in any one of the first aspects above can be implemented.

[0033] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0034] The memory is used to store computer programs;

[0035] The processor is used to implement the sequential parallel model-based reinforcement learning method as described in any one of the first aspects above when executing the computer program.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] Compared with the existing model-based reinforcement learning methods, the proposed time series parallel world model and parallelized eligibility trace estimation in the sequential parallel model-based reinforcement learning method of the present invention realize parallel acceleration, and can achieve significant acceleration on hardware supporting parallel computing such as GPU without affecting the calculation results. At the same time, the world model structure adopted by the present invention enables the strategy to surpass most existing similar methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of the steps of a sequential parallel model-based reinforcement learning method;

[0039] Figure 2 The overall architecture diagram of the sequential parallel model-based reinforcement learning method provided by the present invention;

[0040] Figure 3 A diagram showing the effect of a module ablation experiment provided by an embodiment of the present invention;

[0041] Figure 4 This is a diagram showing the effect of an ablation experiment on the hardware efficiency of the algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0043] like Figure 1 As shown, in a preferred embodiment of the present invention, a sequential parallel model-based reinforcement learning method is provided. The reinforcement learning framework of the entire method is as follows: Figure 2 The reinforcement learning method specifically includes the following steps:

[0044] S1. Construct a time series parallel world model to simulate the operating rules of the game environment; the world model includes an encoder network, a decoder network, a sequence modeling network, an environment dynamics prediction network and a reward prediction network, which are used to map the observation input obtained from the game environment into a low-dimensional latent space, and use the sequence modeling network and the environment dynamics prediction network in the latent space to realize future predictions.

[0045] In the present invention, the sequence modeling network can adopt a linear recurrent neural network, which has two modes: a time series serial mode and a time series parallel mode. The linear recurrent network can be implemented by using a state space model, a gated unit in the recurrent network and a residual module.

[0046] In the present invention, the observations obtained from the game environment can be image observations such as game screens, or vector observations integrated from state variables such as game character states. For different observation types, the encoder network and the decoder network can take different forms. If the observations obtained from the game environment are image observations, the encoder network and the decoder network use a convolutional neural network including a two-dimensional convolutional output layer and a batch normalization layer. If the observations obtained from the game environment are vector observations, the encoder network and the decoder network use a multilayer perceptron including a fully connected output layer and a batch normalization layer. Both the above-mentioned environment dynamics prediction network and the reward prediction network can use a multilayer perceptron.

[0047] S2. Build a reinforcement learning strategy network and value network to continuously obtain observations at different times from the game environment. t , and then collect observations, actions, and single-step rewards from the outputs of the policy network and the value network and store them in the experience replay cache. For each current moment t, the current moment t observation o obtained in the game environment is t Input the encoder network to obtain the latent space vector z at the current time t t , the latent space vector z at the current time t t The latent vector h representing the historical context information output by the sequence modeling network t-1 After splicing, it is used as the input of the strategy network and the value network at the same time. The sequence modeling network of the world model is set to adopt the time series serial mode, and the strategy network outputs the action a that controls the interaction of the characters in the game environment t The value network outputs the estimated value v of the action performed by the game character in the current game environment state t , the observation of the next moment after the game environment receives the action feedback t+1 , the single-step reward r obtained by the game character t And the signal whether the game is terminated t , and finally collect the observations, actions, and single-step rewards into the experience replay cache.

[0048] The policy network is the key training object of the present invention. In the game environment, the policy network constructs the policy model of the role agent in the game environment, integrates the observations of the current time step state information of the role in the environment and the latent variables representing the interaction history information into the agent policy model, obtains the action strategy distribution, samples the actions that can be performed by the role in the environment and applies them to the agent corresponding to the game role to make corresponding behaviors.

[0049] In the present invention, both the above-mentioned strategy network and value network can be implemented by using a multi-layer perceptron.

[0050] S3. Randomly sample interaction sequence data from the experience replay cache to train the world model, where the interaction sequence data includes observation sequence, action sequence and reward sequence, wherein the sequence modeling network of the world model is trained in a time series parallel mode.

[0051] In an embodiment of the present invention, in the above step S3, the specific steps of randomly sampling interaction sequence data from the experience replay buffer to train the world model are:

[0052] S31, randomly sample K interactive sequence data with a sequence length of T from the experience playback cache, and the data at each time t in the interactive sequence data includes the observation o corresponding to the time t 、Action a t and reward r t ;

[0053] S32, the observation o of the interaction sequence data t Input the encoder network to get the latent space vector z t ; The latent space vector z t Input the decoder network to obtain the reconstructed observation o recon t ; Splice the latent space vector z corresponding to each moment t and action a t , input the sequence modeling network in parallel in sequence order to obtain the historical context h corresponding to each moment t ; The historical context h t Input the reward prediction network to get the predicted value of the reward The historical context h t Input the environment dynamic prediction network to get the predicted value of the latent space vector

[0054] S33, according to the output of the decoder network Compute regression loss, predicting the random distribution of network outputs based on rewards Calculate the maximum log-likelihood loss and combine the two losses to calculate the prediction loss L of the world model pred t (θ):

[0055]

[0056] where θ is the neural network parameters of the world model;

[0057] In the above prediction loss L pred t(θ), reconstruct the prediction function The output of the reward prediction function is modeled as a random distribution. The maximum log-likelihood loss is used.

[0058] S34. In order to make the world model have the function of simulating the dynamics of the environment, the latent space vector output by the encoder is modeled as the joint distribution q(z t |o t ), modeling the latent space vector output by the environment dynamics prediction network as a joint distribution of multiple class losses The KL divergence with threshold clipping is used as the loss function, where the clipping threshold is set to 1, to calculate the dynamic prediction loss L dyn t (θ):

[0059]

[0060] Where: sg(·) represents gradient cutoff;

[0061] S35, the above dynamic prediction loss L dyn t The output distribution of the encoder q(z t |o t ) uses gradient truncation sg(·). Therefore, in order to maintain the consistency between the predicted output of the environment dynamic prediction network and the encoder output and prevent the prediction from being too far away from the actual distribution of latent space vectors observed in the game environment, which may cause the world model to fail, this step needs to use the threshold-clipped KL divergence as the representation loss of the encoder output latent space vector and calculate the representation consistency loss L rep t (θ):

[0062]

[0063] S36, for the above three sets of loss functions, use three sets of hyperparameters to adjust the weights of the three losses, and then use three weight hyperparameters β for the three loss functions respectively. pred , β rep , β dyn Weighted, calculate the total loss function L of the world model total (θ):

[0064]

[0065] S37, using the sampled K interactive sequence data, calculate the total loss function L through S32~S36 total (θ), and minimize the total loss function L total(θ) is the target, and the neural network parameter θ of the world model is optimized. During the training process, the sequence modeling network of the world model is trained in a time series parallel mode.

[0066] S4. Flatten the interaction sequence data randomly sampled from the experience replay cache. The data at each time t in the interaction sequence data is used as the starting point for prediction. The game character action a sampled by the policy network is used. t Interact with the world model and transform the latent space vector of the next moment observed by the world model into Prediction Rewards Save to temporary cache.

[0067] S5. The latent space vector observed in the temporary cache and historical context t-1 Input value network to get value estimate Compute the temporal difference target of the value network using parallelized eligibility trace estimation and train the value network.

[0068] In the present invention, the specific implementation process of step S5 is as follows:

[0069] The latent space vector of the observation in the temporary cache and historical context t-1 Input value network to get value estimate Calculate single-step timing difference target The formula is:

[0070]

[0071] The eligibility trace estimate is calculated using the same time series parallelization method as in S3, expressed as:

[0072]

[0073] Among them, λ t is a hyperparameter, when t = 0 At this time, we can use the time difference value target The regression loss of the value network is obtained, and the loss function of training the value network can be expressed as:

[0074]

[0075] Where: 2 Represents the square of the L2 norm.

[0076] By minimizing the above loss function The policy network can be trained.

[0077] S6. Use the temporal difference target to calculate the advantage estimate, use the advantage estimate to train the policy network, and use the trained policy network to control the interactive actions of the characters in the game environment.

[0078] In the present invention, the specific implementation process of step S6 is as follows:

[0079] The timing difference target obtained in S5 above It can also be used as an advantage estimate to evaluate the quality of an action and as the output distribution of a training strategy network. , plus the entropy of the policy distribution weighted by the hyperparameter η The maximized regularization term constructs the loss function L(η) of the training strategy network, which can be expressed as:

[0080]

[0081] By minimizing the above loss function L(ψ), the policy network can be trained. The policy network finally trained can be used to control the interactive actions of the characters in the game environment.

[0082] It should be noted that the method steps S1 to S6 in the above embodiments can essentially be implemented in the form of a computer program.

[0083] Similarly, based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the sequential parallel model-based reinforcement learning method provided in the above embodiment, which includes a memory and a processor;

[0084] The memory is used to store computer programs;

[0085] The processor is used to implement the aforementioned sequential parallel model-based reinforcement learning method when executing the computer program;

[0086] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0087] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to a serial parallel model-based reinforcement learning method, on which a computer program is stored. When the computer program is executed by a processor, the serial parallel model-based reinforcement learning method as described above can be implemented.

[0088] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the serial parallel model-based reinforcement learning method as described above.

[0089] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S6.

[0090] It is also understandable that the above storage medium may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which may store program codes.

[0091] It can also be understood that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0092] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0093] In order to better demonstrate the technical effect of the above reinforcement learning method, it is applied to specific task data through an embodiment.

[0094] Example

[0095] This example uses the Atari100K video game environment to evaluate the performance of the policy model (named PaMoRL) trained by the model-based reinforcement learning method for implementing time series parallelism shown in S1 to S6 above on this task. Atari100K is a benchmark for evaluating the sample efficiency of reinforcement learning methods. The benchmark consists of 26 video games with a discrete action dimension of up to 18 dimensions. 100K samples are equivalent to 400K actual game frames, which is equivalent to about 2 hours of real-time game time. Each action is repeated 4 times. The standardized score is defined as:

[0096]

[0097] where s human refers to the score of human operation, s random Scores from randomly sampled actions.

[0098] This embodiment follows Figure 1 The process of S1 to S6 shown in FIG. Figure 2 The reinforcement learning framework shown in Figure 1 compares the standardized scores on the Atari100K benchmark. The encoder and decoder of the world model use convolutional neural networks including a two-dimensional convolutional output layer and a batch normalization layer, and the activation function is SiLU. The environment dynamics prediction network, reward prediction network, policy network, and value network all use a three-layer MLP network with a hidden layer dimension of 512 and an activation function of SiLU. The sequence modeling network adds the RMSNorm module and the Token Mixing module on the basis of the linear recurrent network. The experimental results are shown in Table 1.

[0099] Table 1

[0100]

[0101] The existing reinforcement learning methods used for comparison in Table 1 above are specifically referred to in the following prior art documents:

[0102] 1.SPR: Schwarzer M, Anand A, Goel R, et al.Data-Efficient ReinforcementLearning with Self-Predictive Representations[C] / / International Conference onLearning Representations.

[0103] 2.SR-SPR:D'Oro P,Schwarzer M,Nikishin E,,et al.Sample-EfficientReinforcement Learning by Breaking the Replay Ratio Barrier[C] / / The EleventhInternational Conference on Learning Representations.

[0104] 3.SimPLe:Kaiser Babaeizadeh M, P,et al.Model Based ReinforcementLearning for Atari[C] / / International Conference on Learning Representations.

[0105] 4.IRIS:Micheli V,Alonso E,Fleuret F.Transformers are Sample-EfficientWorld Models[C] / / The Eleventh International Conference on LearningRepresentations.

[0106] 5.TWM:Robine J, M,Uelwer T,et al.Transformer-based WorldModels Are Happy With 100k Interactions[C] / / The Eleventh InternationalConference on Learning Representations.

[0107] 6.STORM:Zhang W,Wang G,Sun J,et al.STORM:Efficient stochastictransformer based world models for reinforcement learning[J].Advances inNeural Information Processing Systems,2024,36.

[0108] 7. DreamerV3: Hafner D, Pasukonis J, Ba J, et al. Mastering diverse domains through world models [J]. arXiv preprint arXiv: 2301.04104, 2023.

[0109] This embodiment also conducts ablation experiments on the hardware efficiency of each module of the world model and different methods. The ablation experiment results of each module of the world model are as follows: Figure 3 As shown in the figure, the experimental objects of this experiment include: the complete world model PWM and the world models in the above three cases: removing the RMSNorm module, removing the Token Mixing module, and retaining the state space model (SSM) module, as well as the performance of the benchmark method DreamerV3, showing the impact of each module in the world model on the model performance.

[0110] In the ablation experiments on each module of the world model, the three game environments "Alien", "Boxing" and "MsPacman" are suitable for methods focusing on sequence prediction; the three game environments "Amidar", "UpNDown" and "Qbert" are suitable for methods focusing on model training stability.

[0111] The experimental objects in the ablation experiment of the hardware efficiency of different algorithms include: the complete world model, the world model that only retains the state space model (SSM), and the hardware efficiency of the benchmark method DreamerV3. Figure 4 As shown, the left figure shows the memory occupied by different algorithms at different time series lengths, and the right figure shows the running time of different algorithms at different time series lengths.

[0112] In summary, compared with existing similar methods, the present invention greatly increases the hardware efficiency of the model-based reinforcement learning method while maintaining high sample efficiency.

[0113] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A sequential parallel model-based reinforcement learning method, characterized in that: The following steps are involved: S1. Construct a time series parallel world model to simulate the operating rules of the game environment; the world model includes an encoder network, a decoder network, a sequence modeling network, an environment dynamics prediction network and a reward prediction network, which are used to map the observation input obtained from the game environment into a low-dimensional latent space, and use the sequence modeling network and the environment dynamics prediction network in the latent space to realize future predictions; S2. Build a reinforcement learning strategy network and value network to continuously obtain observations at different times from the game environment. t ; For each current moment t, observe the current moment t obtained in the game environment t Input the encoder network to obtain the latent space vector z at the current time t t , the latent space vector z at the current time t t The latent vector h representing the historical context information output by the sequence modeling network t-1 After splicing, it is used as the input of the strategy network and the value network at the same time. The sequence modeling network of the world model is set to adopt the time series serial mode, and the strategy network outputs the action a that controls the interaction of the characters in the game environment t , the value network outputs the estimated value v of the action performed by the game character in the current game environment state t , the observation of the next moment after the game environment receives the action feedback t+1 , the single-step reward obtained by the game character t And the signal whether the game is terminated t , and finally collect observations, actions, and single-step rewards into the experience replay cache; S3, randomly sampling interaction sequence data from the experience replay cache to train the world model, wherein the interaction sequence data includes observation sequence, action sequence and reward sequence, wherein the sequence modeling network of the world model is trained in a time series parallel mode; S4. Flatten the interaction sequence data randomly sampled from the experience replay cache. The data at each time t in the interaction sequence data is used as the starting point for prediction. The game character action a sampled by the policy network is used. t Interact with the world model and transform the latent space vector of the next moment observed by the world model into Prediction Rewards Save to temporary cache; S5. The latent space vector observed in the temporary cache and historical context t-1 Input value network to get value estimate Compute the temporal difference target of the value network using parallelized eligibility trace estimation and train the value network; S6. Use the temporal difference target to calculate the advantage estimate, use the advantage estimate to train the policy network, and use the trained policy network to control the interactive actions of the characters in the game environment.

2. The sequential parallel model-based reinforcement learning method according to claim 1, characterized in that: In S1, if the observation obtained from the game environment is an image observation, the encoder network and the decoder network use a convolutional neural network including a two-dimensional convolutional output layer and a batch normalization layer.

3. The sequential parallel model-based reinforcement learning method according to claim 1, characterized in that: In S1, if the observation obtained from the game environment is a vector observation, the encoder network and the decoder network use a multilayer perceptron including a fully connected output layer and a batch normalization layer.

4. The sequential parallel model-based reinforcement learning method according to claim 1, characterized in that: The strategy network and the value network both use multi-layer perceptrons.

5. The sequential parallel model-based reinforcement learning method according to claim 1, characterized in that: The environment dynamics prediction network and the reward prediction network both use multi-layer perceptrons.

6. The sequential parallel model-based reinforcement learning method according to claim 1, characterized in that: The sequence modeling network adopts a linear recurrent neural network.

7. The sequential parallel model-based reinforcement learning method according to claim 1, characterized in that: In S3, the specific steps of randomly sampling interaction sequence data from the experience replay cache to train the world model are: S31, randomly sample K interactive sequence data with a sequence length of T from the experience playback cache, and the data at each time t in the interactive sequence data includes the observation o corresponding to the time t 、Action a t and reward r t ; S32, the observation o of the interaction sequence data t Input the encoder network to get the latent space vector z t ; The latent space vector z t Input the decoder network to obtain the reconstructed observation Concatenate the latent space vector z corresponding to each moment t and action a t , input the sequence modeling network in parallel in sequence order to obtain the historical context h corresponding to each moment t ; The historical context h t Input the reward prediction network to get the predicted value of the reward The historical context h t Input the environment dynamic prediction network to get the predicted value of the latent space vector S33, according to the output of the decoder network Compute regression loss, predicting the random distribution of network outputs based on rewards Calculate the maximum log-likelihood loss and combine the two losses to calculate the prediction loss L of the world model pred t (θ): where θ is the neural network parameters of the world model; S34, model the latent space vector output by the encoder as the joint distribution q(z t |o t ), the latent space vector output by the environment dynamics prediction network is modeled as the joint distribution of multiple category losses p(z pred t |h t ), using KL divergence with threshold clipping as the loss function, where the clipping threshold is set to 1, to calculate the dynamic prediction loss L dyn t (θ): Where: sg(·) represents gradient cutoff; S35. Use the threshold-clipped KL divergence as the representation loss of the encoder output latent space vector and calculate the representation consistency loss L rep t (θ): S36. Use three weight hyperparameters β for the three loss functions pred , β rep , β dyn Weighted, calculate the total loss function L of the world model total (θ): S37, using the sampled K interactive sequence data to minimize the total loss function L total (θ) is the target, and the neural network parameter θ of the world model is optimized. During the training process, the sequence modeling network of the world model is trained in a time series parallel mode.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, it can implement the sequential parallel model-based reinforcement learning method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the sequential parallel model-based reinforcement learning method according to any one of claims 1 to 7 can be implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the sequential parallel model-based reinforcement learning method according to any one of claims 1 to 7 when executing the computer program.

Images