A sequence-parallel model-based reinforcement learning method

Abstract

The application discloses a sequence-parallel model-based reinforcement learning method. On the basis of linear attention mechanism of a linear recurrent neural network, the application uses root mean square layer normalization and a gating unit to construct a world model which can be fully parallelized; meanwhile, a parallel scan algorithm is used to realize parallelization on a time sequence in the eligibility trace estimation process in the strategy learning stage. Compared with the prior art, the application combines the parallel scan algorithm, improves the hardware efficiency of the model-based reinforcement learning in the world model training and eligibility trace estimation on the premise of maintaining high sample efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of reinforcement learning, and particularly relates to improvement of model-based reinforcement learning in hardware efficiency. BACKGROUND

[0002] Model-based reinforcement learning (MBRL) is widely considered to have great potential to significantly improve sample efficiency by learning a world model to train a policy. Many current proposed MBRL methods achieve the same asymptotic performance as model-free RL methods and require less interactive data. In particular, some recent MBRL methods achieve sample efficiency that surpasses human-level performance in Atari video games and real-world problems such as robot control, healthcare, and advertising recommendation systems. However, such reinforcement learning algorithms consume a lot of computing resources and have low training efficiency, and need to be accelerated. SUMMARY

[0003] The present application aims to solve the defects in the prior art and provide a sequence-parallel model-based reinforcement learning method.

[0004] The specific technical solutions adopted by the present application are as follows:

[0005] In a first aspect, the present application provides a sequence-parallel model-based reinforcement learning method, comprising the following steps:

[0006] S1, constructing a time series parallelized world model to simulate the running rules of the game environment; the world model comprises an encoder network, a decoder network, a sequence modeling network, an environment dynamic prediction network, and a reward prediction network, which are used to map the observations obtained from the game environment into a low-dimensional latent space, and use the sequence modeling network and the environment dynamic prediction network to realize prediction of the future in the latent space;

[0007] S2, constructing a reinforcement learning policy network and a value network, and constantly obtaining observations o t from the game environment at different times; for each current time t, input the current time t observation o t from the game environment into the encoder network to obtain the latent space vector z t of the current time t, concatenate the latent space vector z t of the current time t with the hidden vector h t-1 output by the sequence modeling network to represent the historical context information, and input the concatenated vectors into the policy network and the value network as inputs, set the sequence modeling network of the world model to adopt a time series serial mode, output the action a t controlled by the policy network to interact with the game environment, and output the value estimate v of the action performed by the game character in the state of the current game environment by the value network.t , the next time of observation o fed back after the game environment receives the action t+1 , the single-step reward r obtained by the game character t , and the signal d indicating whether the game is terminated t , and finally the observation, action, and single-step reward are collected into an experience replay buffer;

[0008] S3. Training a world model from the interaction sequence data randomly sampled from the experience replay buffer, the interaction sequence data including an observation sequence, an action sequence, and a reward sequence, wherein the sequence modeling network of the world model is trained in a time series parallel mode;

[0009] S4. Performing a flattening operation on the interaction sequence data randomly sampled from the experience replay buffer, and using the game character action a sampled by the policy network as the starting point of prediction for each time t in the interaction sequence data t interacting with the world model, and saving the latent space vector of the next time observation predicted by the world model , the predicted reward to a temporary buffer;

[0010] S5. Inputting the latent space vector of the observation and the historical context h t-1 to the value network to obtain a value estimate , calculating the time difference target of the value network using the parallelized eligibility trace estimate, and training the value network;

[0011] S6. Calculating an advantage estimate using the time difference target, training the policy network using the advantage estimate, and using the trained policy network to control the actions of the character interacting in the game environment.

[0012] Preferably, in S1, if the observation obtained from the game environment is an image observation, the encoder network and the decoder network use a convolutional neural network including a two-dimensional convolution output layer and a batch normalization layer.

[0013] Preferably, in S1, if the observation obtained from the game environment is a vector observation, the encoder network and the decoder network use a multi-layer perceptron including a fully connected output layer and a batch normalization layer.

[0014] Preferably, the policy network and the value network both use a multi-layer perceptron.

[0015] Preferably, the environment dynamics prediction network and the reward prediction network both use a multi-layer perceptron.

[0016] Preferably, the sequence modeling network uses a linear recurrent neural network.

[0017] As preferred, the specific steps of training the world model from the experience replay buffer in S3 are as follows:

[0018] S31, randomly sample K pieces of interaction sequence data with sequence length T from the experience replay buffer, and the data at each time t in the interaction sequence data includes the observation o t , action a t and reward r t at the corresponding time;

[0019] S32, input the observation o t of the interaction sequence data into the encoder network to obtain the latent space vector z t ; input the latent space vector z t into the decoder network to obtain the reconstructed observation ; concatenate the latent space vector z t and action a t at each time, and input them into the sequence modeling network in sequence order in parallel to obtain the historical context h t at each time; t input the historical context h t into the reward prediction network to obtain the predicted value of the reward ; input the historical context h t into the environment dynamic prediction network to obtain the predicted value of the latent space vector ;

[0020] S33, calculate the regression loss according to the output of the decoder network , calculate the maximum log-likelihood loss according to the random distribution output by the reward prediction network , and calculate the prediction loss of the world model by combining the two losses :

[0021]

[0022] wherein is the neural network parameter of the world model;

[0023] S34, model the latent space vector output by the encoder as a joint distribution of multiple category losses , model the latent space vector output by the environment dynamic prediction network as a joint distribution of multiple category losses , and use the KL divergence with threshold clipping as the loss function, wherein the clipping threshold is set to 1, so as to calculate the dynamic prediction loss :

[0024]

[0025] In the formula: represents gradient clipping;

[0026] S35, using the threshold clipping KL divergence as the representation loss of the encoder output hidden space vector, calculating the representation consistency loss

[0027]

[0028] S36, using three weight hyperparameters to weight three loss functions respectively to calculate the total loss function of the world model

[0029]

[0030] S37, using the sampled K interaction sequence data to optimize the neural network parameters of the world model with the goal of minimizing the total loss function , and the sequence modeling network of the world model adopts a time series parallel mode for training during the training process.

[0031] In a second aspect, the present application provides a computer program product, comprising computer programs / instructions which, when executed by a processor, can implement the sequence-parallel model-based reinforcement learning method according to any one of the above first aspect.

[0032] In a third aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the sequence-parallel model-based reinforcement learning method according to any one of the above first aspect can be implemented.

[0033] In a fourth aspect, the present application provides a computer electronic device, comprising a memory and a processor;

[0034] The memory is used to store a computer program;

[0035] The processor is used to implement the sequence-parallel model-based reinforcement learning method according to any one of the above first aspect when executing the computer program.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] ​​​​Compared with existing model-based reinforcement learning methods, in the sequence-parallel model-based reinforcement learning method of the present application, the proposed time sequence parallelized world model and parallelized eligibility trace estimation achieve parallel acceleration, which can achieve substantial acceleration on hardware supporting parallel computing such as GPU without affecting the calculation results. At the same time, the world model structure adopted by the present application enables the strategy to surpass most existing methods of the same type. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A step flowchart of a sequence-parallel model-based reinforcement learning method;

[0039] Figure 2 A whole architecture diagram of the sequence-parallel model-based reinforcement learning method provided by the present application;

[0040] Figure 3 A module ablation experiment effect diagram provided by an embodiment of the present application;

[0041] Figure 4 An algorithm hardware efficiency ablation experiment effect diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application, so the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0043] As shown in Figure 1 , in a preferred embodiment of the present application, a sequence-parallel model-based reinforcement learning method is provided, and the reinforcement learning framework of the whole method is as shown in Figure 2 . The reinforcement learning method specifically includes the following steps:

[0044] S1, a time sequence parallelized world model is constructed to simulate the running rules of the game environment; the world model includes an encoder network, a decoder network, a sequence modeling network, an environment dynamic prediction network and a reward prediction network, which are used to map the observation input obtained from the game environment to a low-dimensional hidden space, and the sequence modeling network and the environment dynamic prediction network are used to realize the prediction of the future in the hidden space.

[0045] In this invention, the sequence modeling network described above can be a linear recurrent neural network, which has two modes: a serial time series mode and a parallel time series mode. The linear recurrent network can be implemented using a state-space model, gating units, and residual modules within the recurrent network.

[0046] In this invention, observations obtained from the game environment can be image observations such as game screen views, or vector observations formed by integrating state variables such as game character states. The encoder and decoder networks can adopt different forms for different observation types. If the observations obtained from the game environment are image observations, the encoder and decoder networks use a convolutional neural network containing a two-dimensional convolutional output layer and a batch normalization layer. If the observations obtained from the game environment are vector observations, the encoder and decoder networks use a multilayer perceptron containing a fully connected output layer and a batch normalization layer. Both the aforementioned environment dynamic prediction network and reward prediction network can employ a multilayer perceptron.

[0047] S2. Construct a reinforcement learning policy network and a value network to continuously obtain observations from the game environment at different times. t Then, observations, actions, and single-step rewards are collected from the outputs of the policy network and value network and stored in the experience replay cache. For each current time t, the observations o acquired from the game environment at current time t are... t Input the encoder network to obtain the latent space vector z at the current time t. t The latent space vector z at the current time t t The latent vector h, which represents historical context information, is output by the sequence modeling network. t-1 After being concatenated, the network serves as input to both the policy network and the value network. The sequence modeling network of the world model is configured to use a time-series serial mode, with the policy network outputting actions that control character interactions in the game environment. t The value network outputs a value estimate v of the action performed by the game character in the current game environment state. t The game environment receives an action and then observes the next moment's response. t+1 The single-step reward obtained by the game character t And the signal indicating whether the game has ended. t Finally, the observations, actions, and single-step rewards are collected into the experience replay cache.

[0048] The policy network is the key training object of this invention. In the game environment, the policy network constructs a policy model for the game character's agent. It integrates the observation and interaction history information of the character's current time step into latent variables, and obtains the action policy distribution. It samples the actions that the character can execute in the environment and applies them to the agent corresponding to the game character to make the corresponding behavior.

[0049] In the present application, the strategy network and the value network can both be implemented by using a multi-layer perceptron.

[0050] S3, training the world model by randomly sampling interaction sequence data from the experience replay cache, the interaction sequence data comprising an observation sequence, an action sequence and a reward sequence, wherein the sequence modeling network of the world model is trained in a time series parallel mode.

[0051] In an embodiment of the present application, the specific steps of training the world model by randomly sampling interaction sequence data from the experience replay cache in step S3 are as follows:

[0052] S31, randomly sampling K pieces of interaction sequence data with a sequence length of T from the experience replay cache, the data at each time t in the interaction sequence data comprising the observation , action and reward at the corresponding time t;

[0053] S32, inputting the observation of the interaction sequence data into the encoder network to obtain the latent space vector ; inputting the latent space vector into the decoder network to obtain the reconstructed observation ; concatenating the latent space vector and the action at each time t, and inputting them into the sequence modeling network in sequence order in parallel to obtain the historical context at each time t; inputting the historical context into the reward prediction network to obtain the predicted value of the reward ; inputting the historical context into the environment dynamic prediction network to obtain the predicted value of the latent space vector ;

[0054] S33, calculating the regression loss according to the output of the decoder network, calculating the maximum log-likelihood loss according to the random distribution output by the reward prediction network, and calculating the prediction loss of the world model by synthesizing the two losses:

[0055]

[0056] wherein is the neural network parameter of the world model;

[0057] In the prediction loss , the reconstruction prediction function ​​The output of the encoder adopts a regression loss, and the output of the reward prediction function is modeled as a random distribution , and a log-likelihood loss is used.

[0058] S34, in order to make the world model simulate the dynamic of the environment, the hidden space vector output by the encoder is modeled as a joint distribution of multiple category losses , the hidden space vector output by the environment dynamic prediction network is modeled as a joint distribution of multiple category losses , and the KL divergence with threshold clipping is used as the loss function, where the clipping threshold is set to 1, so as to calculate the dynamic prediction loss :

[0059]

[0060] In the formula: Indicates gradient clipping;

[0061] S35, the calculation formula of the above dynamic prediction loss The output distribution of the encoder uses gradient clipping , so in order to keep the consistency between the prediction output of the environment dynamic prediction network and the output of the encoder, prevent the world model from failing due to the large difference between the predicted and the actual observed hidden space vector distribution from the game environment, this step needs to use the KL divergence with threshold clipping as the representation loss of the hidden space vector output by the encoder, and calculate the representation consistency loss :

[0062]

[0063] S36, three groups of hyperparameters are used to adjust the weights of the three losses respectively, and then the three loss functions are weighted using three weight hyperparameters 、 、 to calculate the total loss function of the world model :

[0064]

[0065] S37, using K sampled interaction sequence data, the total loss function is calculated by S32-S36 , and the objective is to minimize the total loss function , and the neural network parameters of the world model are optimized , and the sequence modeling network of the world model adopts a time series parallel mode for training during the training process.

[0066] S4, performing a flattening operation on the randomly sampled interaction sequence data from the experience replay buffer, each time t of the interaction sequence data being used as a starting point for prediction, and using the policy network to sample the game agent action interacting with the world model and saving the world model predicted next time observation latent space vector , the predicted reward to the temporary buffer.

[0067] S5, inputting the observation latent space vector and the historical context into the value network to obtain a value estimate , calculating the time difference target of the value network using the parallelized eligibility trace estimate, and training the value network.

[0068] In the present application, the specific implementation process of step S5 is as follows:

[0069] inputting the observation latent space vector and the historical context into the value network to obtain a value estimate , calculating a single-step time difference target , which can be expressed by the formula:

[0070]

[0071] The eligibility trace estimate is calculated using the same time series parallelization method as in S3, which can be expressed by the formula:

[0072]

[0073] wherein, is a hyperparameter, and when , , the time difference value target can be used to obtain the regression loss of the value network, and the loss function of the value network can be expressed by the formula:

[0074]

[0075] In the formula: represents the square of the L2 norm.

[0076] The value network can be trained by minimizing the above loss function .

[0077] S6, using the advantage estimate value calculated by the time difference target to train the policy network, and using the trained policy network to control the interaction action of the game agent in the game environment.

[0078] In the present application, the specific implementation process of step S6 is as follows:

[0079] The timing difference target obtained in S5 Also can be used as the advantage estimate value of evaluating the good and bad of the action, as the output distribution of training the strategy network Weighted by the hyperparameter The regular term used to encourage the entropy of the strategy distribution to be maximized, and the loss function of the training strategy network constructed by the institution , expressed by formula as:

[0080]

[0081] By minimizing the loss function , the strategy network can be trained. The strategy network finally trained can be used to control the action of the role interaction in the game environment

[0082] It should be noted that the method steps S1-S6 in the above embodiment can be essentially realized in the form of a computer program.

[0083] Similarly, based on the same inventive concept, the present application also provides a computer electronic device corresponding to the sequence-parallel model-based reinforcement learning method provided by the above embodiment, which comprises a memory and a processor;

[0084] The memory is used to store a computer program;

[0085] The processor is used to realize the sequence-parallel model-based reinforcement learning method as described above when the computer program is executed;

[0086] In addition, the logical instructions in the above memory can be realized in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application.

[0087] Therefore, based on the same inventive concept, the present application provides a computer readable storage medium corresponding to the sequence-parallel model-based reinforcement learning method, and the storage medium stores a computer program. When the computer program is executed by a processor, the sequence-parallel model-based reinforcement learning method as described above can be realized.​

[0088] Therefore, based on the same inventive concept, the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, can implement the sequence-parallel model-based reinforcement learning method as described above.

[0089] Specifically, in the computer readable storage medium of the above three embodiments, the stored computer programs are executed by the processor to perform the steps S1-S6 as described above.

[0090] It should be understood that the storage medium can include a random access memory (RAM) and a non-volatile memory (NVM), such as at least one disk memory.

[0091] It should be understood that the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0092] It should be noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method is only a logical functional division, and there can be another division mode in actual implementation, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0093] In order to better show the technical effects of the reinforcement learning method, it is applied to specific task data through an embodiment.

[0094] Embodiment

[0095] The embodiment adopts Atari100K video game environment to evaluate the performance of the policy model (named PaMoRL) trained by the above-mentioned model-based reinforcement learning method shown in S1-S6 which can be parallel in time sequence on the task. Atari100K is a benchmark for evaluating the sample efficiency of reinforcement learning methods, which consists of 26 video games with up to 18-dimensional discrete action space. 100K samples correspond to 400K actual game frames, which is equivalent to about 2 hours of real-time game time, each action is repeated 4 times, and the standardized score is defined as:

[0096]

[0097] wherein is the score of human operation, is the score from randomly sampled action.

[0098] The embodiment compares the standardized scores on Atari100K benchmark according to the processes of S1-S6 shown in Figure 1 and the reinforcement learning framework shown in Figure 2 The encoder and decoder of the world model use convolutional neural network including two-dimensional convolution output layer and batch normalization layer, and the activation function is SiLU. The environment dynamics prediction network, reward prediction network, policy network and value network all use three-layer MLP network, and the dimension of the hidden layer is 512, and the activation function used is SiLU. The sequence modeling network increases the RMSNorm module and the Token Mixing module on the basis of the linear recurrent network. The experimental results are shown in Table 1.

[0099] Table 1

[0100]

[0101] The existing reinforcement learning methods used for comparison in the above Table 1 refer to the following prior art documents:

[0102] 1. SPR: Schwarzer M, Anand A, Goel R, et al. Data-Efficient Reinforcement Learning with Self-Predictive Representations [C] / / International Conference on Learning Representations.

[0103] 2. SR-SPR: D'Oro P, Schwarzer M, Nikishin E, et al. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier [C] / / The Eleventh International Conference on Learning Representations.

[0104] 3. SimPLe: Kaiser Ł, Babaeizadeh M, Miłos P, et al. Model Based Reinforcement Learning for Atari [C] / / International Conference on Learning Representations.

[0105] 4. IRIS: Micheli V, Alonso E, Fleuret F. Transformers are Sample-Efficient World Models [C] / / The Eleventh International Conference on Learning Representations.

[0106] 5. TWM: Robine J, Höftmann M, Uelwer T, et al. Transformer-based World Models Are Happy With 100k Interactions [C] / / The Eleventh International Conference on Learning Representations.

[0107] 6. STORM: Zhang W, Wang G, Sun J, et al. STORM: Efficient stochastic transformer based world models for reinforcement learning [J]. Advances in Neural Information Processing Systems, 2024, 36.

[0108] 7. DreamerV3: Hafner D, Pasukonis J, Ba J, et al. Mastering diverse domains through world models[J]. arXiv preprint arXiv:2301.04104, 2023.

[0109] The ablation experiment of the modules of the world model and the hardware efficiency of different methods is also carried out respectively. The ablation experiment results of the modules of the world model are shown in Figure 3 The experimental objects of the experiment include: the performance of the complete world model PWM and the world model under the conditions of only removing the RMSNorm module, only removing the Token Mixing module, and only retaining the state space model (SSM) module, and the performance of the benchmark method DreamerV3, which shows the influence of each module in the world model on the performance of the model.

[0110] In the ablation experiment of the modules of the world model, the three game environments of “Alien”, “Boxing” and “MsPacman” are suitable for methods focusing on sequence prediction; the three game environments of “Amidar”, “UpNDown” and “Qbert” are suitable for methods focusing on the stability of model training.

[0111] The experimental objects in the ablation experiment of the hardware efficiency of different algorithms include: the hardware efficiency of the complete world model, the world model with only the state space model (SSM) retained, and the benchmark method DreamerV3. The results are shown in Figure 4 The left graph shows the memory occupied by different algorithms under different time sequence lengths, and the right graph shows the running time of different algorithms under different time sequence lengths.

[0112] As can be seen from the above, compared with the same type of method, the hardware efficiency of the model-based reinforcement learning method is greatly increased while maintaining high sample efficiency.

[0113] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A sequence-parallel model-based reinforcement learning method, characterized in that, The method comprises the following steps: S1, constructing a time series parallel world model to simulate the running rules of the game environment; the world model comprises an encoder network, a decoder network, a sequence modeling network, an environment dynamic prediction network and a reward prediction network, which are used to map the observation input obtained from the game environment into a low-dimensional latent space, and the sequence modeling network and the environment dynamic prediction network are used to realize the prediction of the future in the latent space; S2, construct reinforcement learning policy network and value network, constantly obtain observation o at different time from game environment t ; for each current time t, obtain observation o at current time t from game environment t input encoder network, obtain hidden space vector z of current time t t , hidden space vector z of current time t t and the hidden vector h output by the sequence modeling network representing the historical context information t-1 After splicing, it is input into the policy network and the value network at the same time, and the sequence modeling network of the world model is set to adopt time series serial mode, the action a of the game character in the game environment is output by the policy network t , the value estimation v of the action executed by the game character in the state of the current game environment is output by the value network t , the observation o of the next time fed back by the game environment after receiving the action t+1 , the single-step reward r obtained by the game character t , and the signal d whether the game is terminated t , finally, the observation, action and single-step reward are collected into experience replay buffer; S3, training the world model by randomly sampling interaction sequence data from the experience replay buffer, wherein the interaction sequence data comprises an observation sequence, an action sequence and a reward sequence, and the sequence modeling network of the world model is trained in a time series parallel mode; S4, performing a flattening operation on the randomly sampled interaction sequence data from the experience replay cache, with each time t of the interaction sequence data as a starting point for prediction, using the policy network to sample the game agent action a t interacting with the world model and saving the world model predicted next time observation latent space vector , the predicted reward into a temporary cache; S5, the latent space vector observed in the temporary cache and historical context h t-1 input value network to obtain value estimate , the value network is calculated using the parallelized eligibility trace estimation, and the value network is trained; the specific implementation process is as follows: observed latent space vectors in temporary cache and historical context input value network to get value estimate , compute one-step temporal difference target , expressed in formula as: The eligibility trace estimation is completed using the time series parallel mode, and the formula is: where is a hyperparameter, when the temporal-difference value target is obtained, and the loss function for training the value network is given by​ wherein: denotes the square of the L2 norm; by minimizing the above loss function training the value network; S6, obtaining the advantage estimation value by using the time difference target calculation, and training the policy network using the advantage estimation value, and using the trained policy network to control the action of the character interaction in the game environment.

2. The sequence-parallel model-based reinforcement learning method of claim 1, wherein, In S1, if the observation obtained from the game environment is an image observation, the encoder network and the decoder network use a convolutional neural network comprising a two-dimensional convolution output layer and a batch normalization layer.

3. The sequence-parallel model-based reinforcement learning method of claim 1, wherein, In S1, if the observation obtained from the game environment is a vector observation, the encoder network and the decoder network use a multilayer perceptron comprising a fully connected output layer and a batch normalization layer.

4. The sequence-parallel model-based reinforcement learning method of claim 1, wherein, The policy network and the value network both use a multilayer perceptron.

5. The sequence-parallel model-based reinforcement learning method of claim 1, wherein, The environment dynamic prediction network and the reward prediction network both use a multilayer perceptron.

6. The sequence-parallel model-based reinforcement learning method of claim 1, wherein, The sequence modeling network uses a linear recurrent neural network.

7. The sequence-parallel model-based reinforcement learning method of claim 1, wherein, In S3, the specific steps of training the world model by randomly sampling interaction sequence data from the experience replay buffer are as follows: S31, randomly sample K pieces of interaction sequence data with sequence length T from the experience replay cache, the data at each time t in the interaction sequence data includes the observation o t , action a t , and reward r t at the time t S32, observing o t inputting the encoder network to obtain the latent space vector z t ; inputting the latent space vector z t inputting the decoder network to obtain the reconstructed observation ; concatenating the latent space vector z t and the action a t inputting the sequence modeling network in sequence order and in parallel to obtain the historical context h t ; inputting the historical context h t into the reward prediction network to obtain the predicted value of the reward ; inputting the historical context h t into the environment dynamics prediction network to obtain the predicted value of the latent space vector ; S33, according to the output of the decoder network computing a regression loss, according to a random distribution output by the reward prediction network computing a maximum log-likelihood loss, synthesizing the two losses to compute a prediction loss for the world model : wherein are neural network parameters of the world model; S34, modeling the latent space vector output by the encoder as a joint distribution of multiple categorical losses , modeling the latent space vector output by the environment dynamics prediction network as a joint distribution of multiple categorical losses , using the KL divergence with threshold clipping as the loss function, where the clipping threshold is set to 1, to compute the dynamics prediction loss : In the formulae: denotes a gradient truncation; S35, the KL divergence with threshold clipping is used as the representation loss of the encoder output hidden space vector, and a representation consistency loss is calculated : S36, using three weight hyperparameters , , to weight the three loss functions, calculating the total loss function of the world model : S37, utilize the sampled K interaction sequence data to minimize the total loss function a neural network parameter of the world model is optimized , and the sequence modeling network of the world model adopts a time series parallel mode for training during the training process.

8. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the sequence parallel model-based reinforcement learning method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer program is stored on the storage medium and can implement the sequence parallel model-based reinforcement learning method according to any one of claims 1-7 when executed by the processor.

10. A computer electronic device, comprising: The storage medium comprises a memory and a processor; The memory is used to store a computer program; The processor is used to implement the sequence parallel model-based reinforcement learning method according to any one of claims 1-7 when the computer program is executed.

Images