A deep reinforcement learning policy optimization method and system with multi-layer entropy structure

Abstract

The application discloses a deep reinforcement learning strategy optimization method and system with a multi-layer entropy structure, wherein the deep reinforcement learning model with the multi-layer entropy structure takes PPO as a framework, and adopts a loss function constructed according to a clipping target function with an entropy structure, a value network target function with the entropy structure and an entropy function. On one hand, the application intelligently selectively memorizes important information, reduces the dependence on a large amount of sample data, thereby speeds up the learning process, and can effectively improve sample utilization; on the other hand, the application balances the relationship between exploration and utilization through the design of the multi-layer entropy structure, optimizes strategy selection, improves the performance and stability of the algorithm, and effectively solves the problem that the traditional deep reinforcement learning method may be limited by excessive curiosity or conservatism when facing a discrete action space task, thereby leading to suboptimality of strategy optimization.

Description

Technical Field

[0001] This invention relates to a method and system for optimizing deep reinforcement learning strategies with a multi-layer entropy structure, belonging to the fields of artificial intelligence and machine learning technology. Background Technology

[0002] Deep Reinforcement Learning (DRL) is a method by which an agent learns a reward-maximizing behavioral policy by interacting with its environment and utilizing neural networks. When solving tasks in discrete action spaces, DRL faces two major challenges: low sample utilization and difficulty in balancing information exploration and utilization. Solving these problems depends not only on algorithmic improvements but also on the full utilization of hardware resources.

[0003] Discrete action space tasks typically involve high-dimensional state spaces, leading to increased sampling costs and low sample utilization. Algorithms require significant time to collect sample data, but current algorithms are often limited by hardware resources when processing large amounts of state information. The limited computing power of CPUs makes it difficult to meet the processing demands of large-scale data in complex tasks, further exacerbating the low sample utilization problem. On the other hand, balancing information exploration and utilization is crucial for the success of DRL. However, this is also a challenge to deploy in hardware, as current algorithms often require substantial computational resources to search the action space to find the optimal policy. How to effectively balance information exploration and utilization and improve performance by utilizing existing hardware resources remains to be studied.

[0004] Therefore, in order to solve the problems of low sample utilization and imbalance between information exploration and utilization in Discrete Action Space tasks, this invention is proposed. Summary of the Invention

[0005] This invention provides a method and system for optimizing deep reinforcement learning strategies with a multi-layer entropy structure. On the one hand, a multi-layer entropy structure is introduced to construct a loss function to optimize strategy selection; on the other hand, a selective memory mechanism is introduced on the basis of the multi-layer entropy structure to reduce the dependence on a large amount of sample data, thereby accelerating the learning process.

[0006] The technical solution of this invention is:

[0007] According to a first aspect of the present invention, a deep reinforcement learning policy optimization method with a multi-layer entropy structure is provided, comprising: acquiring target task RGB image data; preprocessing the target task RGB image data to obtain preprocessed target task RGB image data; using the preprocessed target task RGB image data of a preset frame as input to a deep reinforcement learning model with a multi-layer entropy structure, and providing an optimal control policy for the agent by continuously training the network parameters of the deep reinforcement learning model with a multi-layer entropy structure; wherein the deep reinforcement learning model with a multi-layer entropy structure uses PPO as a framework and employs a loss function constructed based on a pruning objective function with an entropy structure, a value network objective function with an entropy structure, and an entropy function.

[0008] The pruning objective function with entropy structure is expressed as follows:

[0009]

[0010] in, Let represent the pruning objective function with entropy structure at time t, and let clip() represent the pruning function. Let ε represent the advantage estimate at time t, ξ represent the importance sampling ratio, and α be the entropy regularization coefficient. Represents the strategy π at time t θ The gradient value.

[0011] The objective function of the value network with entropy structure is expressed as:

[0012]

[0013] in, Denotes the objective function of the value network with entropy structure at time t; θ, These are the parameters for the policy network and the value network, respectively. Let α be the state value of the value network at time t; α is the coefficient of the entropy regularization term. Represents the strategy π at time t θ The gradient value.

[0014] The loss function is expressed as follows:

[0015]

[0016] in, Let c1 and c2 be the pruning objective function with entropy structure at time t, respectively, and c1 and c2 be the value network objective functions with entropy structure. And entropy function S t (π θ The weighting coefficients of ).

[0017] The deep reinforcement learning model with a multi-layer entropy structure employs a selective memory mechanism to form priority trajectory data with selective memory from the trajectory data.

[0018] The selective memory mechanism includes: calculating the sampling probability of trajectory data based on priority weights; calculating priority sampling weights based on the sampling probability of trajectory data; calculating the temporal error of trajectory data samples; updating priority weights based on the temporal error; calculating weight change values ​​based on priority sampling weights and temporal error; updating priority network parameters based on weight change values ​​to form original priority trajectory data; acquiring forgotten trajectory data and remembered trajectory data based on the original priority trajectory data; forming new priority trajectory data based on the forgotten trajectory data and remembered trajectory data; and forming priority trajectory data with selective memory based on the original priority trajectory data and the new priority trajectory data.

[0019] The training of the deep reinforcement learning model with the multi-layer entropy structure includes: initializing the policy network, value network, and experience pool under the PPO framework; the agent interacts with the environment through the policy network to generate a preset number of trajectory data, which are then processed by a selective memory mechanism and aggregated into the priority experience pool; based on the current state, data is extracted from the priority experience pool according to the priority network parameters to determine the action to be executed, and then the advantage function is calculated; based on the advantage function, a pruning objective function with an entropy structure is calculated; based on the pruning objective function with an entropy structure, the value network objective function with an entropy structure, and the entropy function, a loss function is obtained; based on the loss function and the priority network parameters, the parameters of the policy network and value network are updated, the updated value network parameters are fixed, the updated policy network continues to interact with the environment and is processed by the selective memory mechanism to generate trajectory data and store it in the priority experience pool, until the termination condition is reached, and the training is completed.

[0020] According to a second aspect of the present invention, a deep reinforcement learning policy optimization system with a multi-layer entropy structure is provided, comprising: a preprocessing module for acquiring target task RGB image data; preprocessing the target task RGB image data to obtain preprocessed target task RGB image data; and a policy optimization module for using the preprocessed target task RGB image data of a preset frame as input to a deep reinforcement learning model with a multi-layer entropy structure, and providing an optimal control policy for the agent by continuously training the network parameters of the deep reinforcement learning model with a multi-layer entropy structure; wherein the deep reinforcement learning model with a multi-layer entropy structure uses PPO as a framework and employs a loss function constructed based on a pruning objective function with an entropy structure, a value network objective function with an entropy structure, and an entropy function.

[0021] According to a third aspect of the present invention, a terminal device is provided, including a memory and a processor, wherein the processor executes the deep reinforcement learning policy optimization method with a multi-layer entropy structure as described above during operation.

[0022] The beneficial effects of this invention are as follows: On the one hand, by intelligently and selectively memorizing important information, this invention reduces the dependence on a large amount of sample data, thereby accelerating the learning process and effectively improving sample utilization. On the other hand, through the design of a multi-layer entropy structure, this invention effectively balances the relationship between exploration and utilization, optimizes policy selection, and improves the performance and stability of the algorithm. It effectively solves the problem that traditional deep reinforcement learning methods may suffer from suboptimal policy optimization due to excessive curiosity or conservatism when facing discrete action space tasks. Furthermore, through theoretical derivation and experimental verification on different tasks, this invention verifies that the agent can learn efficient and stable policies, enabling it to continuously and efficiently cope with complex tasks in discrete action spaces. Based on the above, this invention solves the challenges faced by existing deep reinforcement learning methods in discrete action space tasks, including low sample utilization, suboptimal policy optimization caused by excessive curiosity or conservatism, and poor algorithm performance, thus having a wide range of application scenarios. Attached Figure Description

[0023] Figure 1 This is a structural block diagram of the method for introducing a selective memory mechanism provided in an embodiment of the present invention;

[0024] Figure 2 This is a structural block diagram of the system of the present invention;

[0025] Figure 3 This is a convergence visualization diagram of some discrete action space tasks in an embodiment of the present invention. Detailed Implementation

[0026] The invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited to the description.

[0027] Example 1: As Figure 1-3As shown, according to a first aspect of the present invention, a deep reinforcement learning policy optimization method with a multi-layer entropy structure is provided, comprising: acquiring target task RGB image data under a simulation platform; preprocessing the target task RGB image data to obtain preprocessed target task RGB image data; wherein, the target task is a task in a discrete action space, such as (MAPF task in a grid world, Atari task); wherein, the preprocessing sequentially includes grayscale processing, uniform size processing, standardization processing, and differential image processing; in the embodiment of the present invention, uniform size processing is to adjust the image size to 84*84*3. The preprocessed target task RGB image data of a preset frame is used as input to a deep reinforcement learning model with a multi-layer entropy structure, and the network parameters of the deep reinforcement learning model with a multi-layer entropy structure are continuously trained to provide the agent with an optimal control policy; in the embodiment of the present invention, the preset frame is 4 frames. The deep reinforcement learning model with a multi-layer entropy structure uses PPO as a framework and employs a loss function constructed based on a pruning objective function with an entropy structure, a value network objective function with an entropy structure, and an entropy function. Proximal policy optimization (PPO) algorithm.

[0028] Furthermore, the pruning objective function with entropy structure is expressed as:

[0029]

[0030] in, Let represent the pruning objective function with entropy structure at time t, and let clip() represent the pruning function. Let ε represent the advantage estimate at time t, ξ represent the importance sampling ratio, and α be the entropy regularization coefficient. Represents the strategy π at time t θ The gradient value of ξ is calculated by the following formula:

[0031]

[0032] Among them, P θ , These represent the probabilities of taking the action under the current strategy and the previous strategy, respectively.

[0033] Furthermore, the objective function of the value network with entropy structure is expressed as:

[0034]

[0035] in, Denotes the objective function of the value network with entropy structure at time t; θ, These are the parameters for the policy network and the value network, respectively. Let α be the state value of the value network at time t; α is the coefficient of the entropy regularization term. Represents the strategy π at time t θ The gradient value.

[0036] Furthermore, the entropy function is expressed as:

[0037]

[0038] Among them, S t (π θ () represents the entropy function at time t; α is the coefficient of the entropy regularization term. Represents the strategy π at time t θ The gradient value.

[0039] Furthermore, the loss function is expressed as:

[0040]

[0041] in, Let c1 and c2 be the pruning objective function with entropy structure at time t, respectively, and c1 and c2 be the value network objective functions with entropy structure. And entropy function S t (π θ The weighting coefficients of ).

[0042] Furthermore, the deep reinforcement learning model with a multi-layer entropy structure employs a selective memory processing mechanism to form priority trajectory data with selective memory from the trajectory data.

[0043] Furthermore, the selective memory mechanism includes:

[0044] Calculate trajectory data D t Sampling probability:

[0045]

[0046] Where j represents the trajectory data sample number, j~P(j) represents the sampling probability P(j) corresponding to trajectory data sample j; μ is the priority sampling factor (hyperparameter), p j The priority weight of sample j is represented (initially, the priority weight is selected between random(0,1)), and N is the total number of samples in the trajectory data.

[0047] Calculate the priority sampling weight w based on the sampling probability of the trajectory data. j :

[0048]

[0049] Among them, w jHere, β represents the priority sampling weight corresponding to trajectory data sample j, N is the total number of samples in the trajectory data, β is the priority importance sampling factor weight (hyperparameter), and m... i axw i This represents the maximum priority sampling weight (in this embodiment of the invention, it is set to 1 during the first iteration; during the second iteration, it is set to the maximum value among all priority sampling weights in the first iteration, and so on).

[0050] Calculate the temporal difference error (TD) of trajectory data sample j at time t.

[0051]

[0052] Where, r t Let γ be the reward value obtained at time t, γ be the discount factor (hyperparameter), V(*) be the value function value, and s be the reward value obtained at time t. t Let t be the state at time t.

[0053] Update priority weights based on timing errors. Based on the priority sampling weights and timing errors, calculate the weight change value Δ:

[0054]

[0055] Where Δ is initially 0, Represents the strategy π at time t θ The gradient value is θ, where θ is the policy network parameter.

[0056] Update the priority network parameters θ based on the weight changes. p ←θ+τ·Δ; where τ is the priority step size (hyperparameter).

[0057] The trajectory data D at time t up to this point t Processed as priority trajectory data Then, selective memory processing is performed on them.

[0058] Obtain the trajectory data after forgetting:

[0059]

[0060] Among them, h t-1 This represents the priority trajectory data of the previous time step at time t (initially, the priority trajectory data of the previous time step is 0). This represents the priority trajectory data at time t. f represents the priority trajectory data at time t-1. t 1The trajectory data after forgetting irrelevant information at time t-1, i.e., the trajectory data after forgetting time t-1.

[0061] Obtain the memorized trajectory data:

[0062]

[0063] Among them, i t For the trajectory data that needs to be updated, Given candidate trajectory data at time t, f t 2 This is the trajectory data after it has been memorized.

[0064] Then, for f t 1 Add new experience data f t 2 The priority trajectory data at time t-1 is updated to form new priority trajectory data C at time t. t .

[0065] C t =f t 1 +f t 2 ;

[0066] Based on the priority trajectory data at time t Priority trajectory data at time t forms priority trajectory data with selective memory.

[0067]

[0068] Among them, o t The importance of the trajectory data to be output is determined.

[0069] Furthermore, the training of the deep reinforcement learning model with a multi-layer entropy structure includes:

[0070] By utilizing the CPU's multi-process mechanism, eight processes are launched simultaneously. One process is responsible for initializing a main process, while the other seven processes collect data in parallel.

[0071] The entire system is initialized through the CPU's main process, including the parameters of the policy network and value network under the PPO framework, as well as the experience pool.

[0072] The agent's policy interacts with the environment using seven additional CPU processes. Each process independently samples data and generates trajectories. Each process is responsible for running the reinforcement learning interaction environment and generating a preset number of trajectory data. This data is selectively memorized and then aggregated into a priority experience pool. The trajectory data consists of a quadruple (S, A, P, R), where S, A, P, and R represent the set of states, the set of actions, the state transition probability distribution, and the set of reward values, respectively. When a greedy policy is used for sampling, the state s at time t... t ∈S、Action a t ∈A, the state transition probability distribution P∈[0,∞), and the action a. t And transition to the next state s t+1 The reward value R∈[r] at that time min ,r max ], r min ,r max This indicates the lower and upper limits of the reward value; the preset quantity of trajectory data is 10,000 during the initial interaction.

[0073] In the priority experience pool, the CPU periodically checks whether a preset amount of trajectory data has been provided for the GPU to train. If the data is sufficient, the CPU will instruct the GPU to begin training.

[0074] Based on the current state, data is extracted from the priority experience pool according to the priority network parameters to determine the action to be executed (the higher the priority, the earlier it is extracted), and then provided to the GPU for training the policy network and value network.

[0075] The training data drawn from the priority experience pool is divided into multiple batches, with each batch assigned to a different GPU to achieve data-parallel computing. Each GPU computes the advantage function in parallel based on its current state and the actions it is performing, ensuring efficient processing of the data generated by each process.

[0076] Based on the advantage function, and utilizing the parallel computing capabilities of the GPU, the data of each process is processed to calculate the pruning objective function with entropy structure, the value network objective function with entropy structure, and the entropy function, thereby obtaining the loss function and improving the overall computational efficiency.

[0077] Based on the loss function and priority network parameters, the CPU's main process updates the parameters of the policy network and value network. The updated value network parameters are then fixed, and the updated policy network continues to interact with the environment through the other 7 CPU processes. A selective memory mechanism is used to generate trajectory data, which is stored in the priority experience pool, until the termination condition is met, and training is complete.

[0078] Furthermore, based on the loss function and priority network parameters, the parameters of the policy network and value network are updated using the CPU's main process, as expressed in the following expression:

[0079]

[0080]

[0081] Among them, L r This represents the learning rate.

[0082] As can be seen from the above technical solution, the present invention not only improves the software, but also makes full use of hardware resources. By effectively utilizing the computing power of CPU and GPU, it accelerates the speed at which the algorithm executes discrete action space tasks, thereby converging to the optimal solution more quickly, and thus significantly improving the performance of deep reinforcement learning algorithms with multi-layer entropy structures.

[0083] According to a second aspect of the present invention, a deep reinforcement learning policy optimization system with a multi-layer entropy structure is provided, comprising: a preprocessing module for acquiring target task RGB image data; preprocessing the target task RGB image data to obtain preprocessed target task RGB image data; and a policy optimization module for using the preprocessed target task RGB image data of a preset frame as input to a deep reinforcement learning model with a multi-layer entropy structure, and providing an optimal control policy for the agent by continuously training the network parameters of the deep reinforcement learning model with a multi-layer entropy structure; wherein the deep reinforcement learning model with a multi-layer entropy structure uses PPO as a framework and employs a loss function constructed based on a pruning objective function with an entropy structure, a value network objective function with an entropy structure, and an entropy function.

[0084] According to a third aspect of the present invention, a terminal device is provided, including a memory and a processor, the processor including a CPU and a GPU, wherein the processor executes the deep reinforcement learning policy optimization method with a multi-layer entropy structure as described above during operation.

[0085] In this embodiment, we conducted ablation experiments and compared our invention with PPO, DQN, and C51. The environment settings are end-to-end reinforcement learning based on visual settings, incorporating challenges such as delayed reward allocation, partial observability, task diversification, and high-dimensional observation states. Our invention exhibits good generalization ability and does not require hyperparameter tuning for specific problems; the specific settings are shown in Table 1.

[0086] Table 1 Parameter Settings

[0087] Discount factor γ 0.99 <![CDATA[Value function weight coefficient c1]]> 0.50 <![CDATA[Entropy objective function weight coefficient c2]]> 0.01 Priority step size τ 0.25 Priority sampling factor μ 0.20 Priority importance sampling factor weight β 0.60 Clipping factor ε 0.10 Entropy regularization coefficient α 0.20 <![CDATA[Learning rate L r > <![CDATA[10 -5 ]]> Total training time step <![CDATA[10 7 ]]>

[0088] like Figure 3As shown in Table 2, the deep reinforcement learning method with a multi-layer entropy structure and selective memory proposed in this invention exhibits high efficiency and stability across various tasks, with fast convergence and optimal task performance. In different tasks, when only one improvement is applied, this invention demonstrates a significant advantage over the baseline algorithm PPO in 17 tasks. For example, in the Berzerk task, applying only the multi-layer entropy structure (Improvement 1) improves performance by over 115.52% compared to the baseline algorithm. This is because introducing an entropy function into the pruning objective and value functions increases the algorithm's consideration of unknown states or uncertain actions, thus prompting the policy to more actively explore the environment and learn new knowledge. This exploratory behavior avoids the algorithm getting trapped in local optima, helping to improve the model's robustness and generalization ability, allowing the algorithm to better adapt to different tasks and environmental changes. In the Zaxxon task, applying only the selective memory mechanism (Improvement 2) improves performance by 84.06%, because the selective memory mechanism allows the policy to focus on important experiences. During training, the algorithm may face a large number of state-action sequences, and not all experiences are equally important for the learning task. By selectively memorizing, the algorithm can prioritize storing experiences that are more critical to the current task or the overall learning process, thereby improving training efficiency. When these two improvements are adopted simultaneously, the selective memory mechanism and the multi-level entropy structure complement each other, jointly promoting the policy learning process and achieving optimal performance across 28 tasks. The selective memory mechanism helps the agent more effectively utilize limited storage resources during experience replay, while retaining experiences that are highly critical to the task, while the multi-level entropy structure enhances the algorithm's ability to handle unknowns and uncertainties. Taking the Atlantis task as an example, its performance is improved by 32.42 times compared to DQN, 37.46 times compared to C51, and 13.08 times compared to PPO. Therefore, the deep reinforcement learning method with a multi-level entropy structure and selective memory proposed in this invention can significantly improve algorithm performance and has the characteristic of optimal performance for different tasks.

[0089] Table 2 Performance Comparison of Tasks in Discrete Action Space

[0090]

[0091] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A deep reinforcement learning policy optimization method with multi-layer entropy structure, characterized in that, include: Obtain the RGB image data of the target task; The target task RGB image data is preprocessed to obtain preprocessed target task RGB image data; The preprocessed target task RGB image data of the preset frame is used as the input of a deep reinforcement learning model with a multi-layer entropy structure. By continuously training the network parameters of the deep reinforcement learning model with a multi-layer entropy structure, the optimal control strategy is provided for the agent. The deep reinforcement learning model with a multi-layer entropy structure uses PPO as its framework and employs a loss function constructed based on a pruning objective function with an entropy structure, a value network objective function with an entropy structure, and an entropy function. The deep reinforcement learning model with a multi-layer entropy structure employs a selective memory mechanism to form priority trajectory data with selective memory from the trajectory data. The selective memory mechanism includes: Calculate the sampling probability of trajectory data based on priority weights; Calculate the priority sampling weight based on the sampling probability of the trajectory data; Calculate the temporal error of the trajectory data samples; Update priority weights based on timing error; calculate weight changes based on priority sampling weights and timing error. Based on the weight changes, update the priority network parameters to form the original priority trajectory data; Based on the original priority trajectory data, obtain the trajectory data after forgetting and the trajectory data after remembering; New priority trajectory data is formed based on the trajectory data after forgetting and the trajectory data after memorization; Based on the original priority trajectory data and the new priority trajectory data, priority trajectory data with selective memory is formed.

2. The deep reinforcement learning policy optimization method with multi-layer entropy structure according to claim 1, characterized in that, The pruning objective function with entropy structure is expressed as follows: ； in, Let represent the pruning objective function with entropy structure at time t. () represents the clipping function. Let ε represent the advantage estimate at time t, ξ represent the importance sampling ratio, and α be the entropy regularization coefficient. Represents the strategy π at time t θ The gradient value.

3. The deep reinforcement learning strategy optimization method with a multi-layer entropy structure according to claim 1, characterized in that, The objective function of the value network with entropy structure is expressed as: in, Denotes the objective function of the value network with entropy structure at time t; θ, These are the parameters for the policy network and the value network, respectively. Let α be the state value of the value network at time t; α is the coefficient of the entropy regularization term. Represents the strategy π at time t θ The gradient value.

4. The deep reinforcement learning strategy optimization method with a multi-layer entropy structure according to claim 1, characterized in that, The loss function is expressed as follows: ； in, Let c1 and c2 be the pruning objective function with entropy structure at time t, respectively, and c1 and c2 be the value network objective functions with entropy structure. Sum and entropy function The weighting coefficients.

5. The deep reinforcement learning strategy optimization method with a multi-layer entropy structure according to claim 1, characterized in that, The training of the deep reinforcement learning model with the multi-layer entropy structure includes: Initialize the policy network, value network, and experience pool under the PPO framework; The agent interacts with the environment through a policy network, generates a preset amount of trajectory data, and after being processed by a selective memory mechanism, it is aggregated into a priority experience pool. Based on the current state, and by extracting data from the priority experience pool according to the priority network parameters, the action to be executed is determined, and then the advantage function is calculated. Based on the advantage function, calculate the pruning objective function with entropy structure; based on the pruning objective function with entropy structure, the value network objective function with entropy structure, and the entropy function, obtain the loss function; The parameters of the policy network and value network are updated based on the loss function and priority network parameters. The updated value network parameters are fixed, and the updated policy network continues to interact with the environment and undergoes selective memory processing to generate trajectory data which is stored in the priority experience pool until the termination condition is met, thus completing the training.

6. A deep reinforcement learning policy optimization system with a multi-layer entropy structure, characterized in that, include: The preprocessing module is used to acquire the RGB image data of the target task; The target task RGB image data is preprocessed to obtain preprocessed target task RGB image data; The strategy optimization module is used to take the preprocessed target task RGB image data of the preset frame as the input of the deep reinforcement learning model with a multi-layer entropy structure. By continuously training the network parameters of the deep reinforcement learning model with a multi-layer entropy structure, it provides the agent with the optimal control strategy. The deep reinforcement learning model with a multi-layer entropy structure is based on PPO and adopts a loss function constructed based on the pruning objective function with an entropy structure, the value network objective function with an entropy structure, and the entropy function. The deep reinforcement learning model with a multi-layer entropy structure employs a selective memory mechanism to form priority trajectory data with selective memory from the trajectory data. The selective memory mechanism includes: Calculate the sampling probability of trajectory data based on priority weights; Calculate the priority sampling weight based on the sampling probability of the trajectory data; Calculate the temporal error of the trajectory data samples; Update priority weights based on timing error; calculate weight changes based on priority sampling weights and timing error. Based on the weight changes, update the priority network parameters to form the original priority trajectory data; Based on the original priority trajectory data, obtain the trajectory data after forgetting and the trajectory data after remembering; New priority trajectory data is formed based on the trajectory data after forgetting and the trajectory data after memorization; Based on the original priority trajectory data and the new priority trajectory data, priority trajectory data with selective memory is formed.

7. A terminal device, comprising a memory and a processor, characterized in that, The processor executes the deep reinforcement learning policy optimization method with a multi-layer entropy structure as described in any one of claims 1-5.

Images