Brain-like reinforcement learning method and system based on hierarchical experience playback

Abstract

The invention specifically discloses a brain-like reinforcement learning method and system based on hierarchical experience playback, and relates to the technical field of reinforcement learning and brain-like computing. The method comprises the steps that S1, observation data are collected and preprocessed; s2, initializing an experience buffer pool, an actor network, a commentator network and a corresponding target network, and performing parameter initialization; s3, exploring noise is initialized, actions are selected from the actor network according to the current state and executed, and obtained experience samples are stored in an experience buffer pool; s4, obtaining a new sample from the experience buffer pool, and updating the short-term memory pool; s5, determining whether partial experience in the short-term memory experience pool is transferred to the long-term memory experience pool or not by using an attention discrimination module; and S6, updating parameters of the actor network, the reviewer network and the corresponding target network. By adopting the method, the experience utilization rate of the intelligent agent is improved, the reinforcement learning performance is improved, and the method has wide application potential in multiple fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning and brain-like computing, in particular to a brain-like reinforcement learning method and system based on hierarchical experience replay. BACKGROUND

[0002] Reinforcement learning, as an important branch of machine learning, aims to learn the optimal policy through the interaction between the agent and the environment to maximize the cumulative reward. In traditional reinforcement learning, the agent learns by trial and error, adjusting its behavior strategy according to the environmental feedback in the process of constantly trying different behaviors. However, this learning method often requires a large amount of sample data, and the data utilization efficiency is low. To solve this problem, the experience replay method emerged. This method stores the historical experience of the agent in the process of interacting with the environment in the experience pool, and randomly selects experience samples from the experience pool for learning in the learning process, so that the agent can reuse these experiences, effectively improving the data utilization efficiency. However, the traditional experience replay method generally uses a single experience pool to store all experiences, which has obvious limitations: On the one hand, this storage method does not fully consider the temporal hierarchy of experience. In actual application, the influence of experience generated at different times on the learning and decision-making of the agent is quite different. The experience generated recently is more relevant to the current environment and can help the agent quickly adapt to changes in the environment; while the experience generated earlier may have reduced applicability in the current environment, but some general information and long-term regularities contained therein are crucial for the agent to improve overall decision-making ability. The traditional single experience pool cannot effectively distinguish and utilize these different temporal hierarchy experiences, resulting in reduced learning efficiency and decision-making quality.

[0003] On the other hand, the traditional experience replay method does not reasonably distinguish the importance of experience. Different types of experience in the experience pool have different effects on the learning and decision-making of the agent. Analogous to the memory mode of the human brain, the information in the experience pool can also be divided into short-term memory and long-term memory. Short-term experience reflects the agent's recent behavior and immediate feedback from the environment, which can help the agent quickly respond and adapt to environmental changes; long-term experience contains more stable and in-depth information, which helps the agent make more reasonable choices in long-term decision-making. Existing methods fail to effectively distinguish and manage these experiences, making it difficult for the agent to fully exploit and utilize the value of different experiences in the learning process, limiting the performance improvement of reinforcement learning algorithms.

[0004] In summary, the existing reinforcement learning experience replay method has deficiencies in data utilization efficiency and experience management, and there is an urgent need for an innovative method and device to solve these problems to improve the performance and effectiveness of reinforcement learning. SUMMARY

[0005] The object of the present application is to propose a brain-like reinforcement learning method and system based on hierarchical experience replay, which optimizes the efficiency of experience replay by introducing brain-like mechanisms, optimizes the experience replay process using short-term memory and long-term memory thinking patterns, improves the experience utilization rate and decision-making efficiency of the agent in the learning process, and improves the learning effect and performance of reinforcement learning.

[0006] To achieve the above object, the present application proposes a brain-like reinforcement learning method based on hierarchical experience replay, comprising the following steps: Step S1, collecting observation data of the interaction between the agent and the environment and performing data preprocessing, constructing hierarchical training data for the reinforcement learning network, the observation data including state feature vector data, action encoding data, immediate reward signal data, visual observation frame and task description text; Step S2, initializing the actor network, critic network and corresponding target network, and initializing the experience buffer pool and performing parameter initialization, the parameters including discount factor γ, maximum emotion number M and maximum time step T ; Step S3, at the beginning of each training round, initializing the exploration noise, the agent obtaining the initial state of the current environment, selecting an action from the actor network according to the current state and executing it, and storing the experience samples obtained after executing the action into the experience buffer pool; Step S4, short-term memory pool update, the short-term memory experience pool obtaining new samples from the experience buffer pool, if the short-term memory experience pool is full, deleting obsolete samples according to the first-in-first-out strategy; Step S5, experience screening transfer, calculating the similarity between the new experience samples and the existing experience samples in the short-term memory experience pool using the attention discrimination module, comparing whether it exceeds the threshold, recording the number of times the experience is repeatedly evaluated as similar, and deciding whether to transfer part of the experience in the short-term memory experience pool to the long-term memory experience pool; Step S6, network parameter update, obtaining N 1 samples from the short-term memory experience pool, obtaining N 2 samples from the long-term memory experience pool, inputting these samples into the actor network and the critic network for parameter update, and performing parameter update of the target network every fixed number of network parameter update times, wherein N 1 and N 2 are positive integers.

[0007] Preferably, in step S1, the data preprocessing includes: denoising the observed data, filtering out outliers and invalid data points, normalizing or standardizing the numerical feature data, and performing one-hot encoding or embedding representation on the discrete action code.

[0008] Preferably, the short-term memory experience pool has a limited capacity first-in-first-out (FIFO) structure, and the FIFO strategy ensures that the experience in the pool always remains timely.

[0009] Preferably, the long-term memory experience pool is a full experience pool, in which the experiences are derived from the screening of the short-term memory experience pool. The attention discrimination module evaluates the long-term preservation value of the experiences in the short-term memory experience pool and decides whether to transfer them.

[0010] Preferably, the attention discrimination module uses the cosine similarity method to calculate the similarity between experience samples. By comparing the cosine similarity between the new experience sample and the feature vectors of each existing sample in the short-term memory experience pool, the degree of similarity between the two is quantified.

[0011] Preferably, in step S3, the exploration noise is Gaussian noise, and its standard deviation is set to a large value in the early stage of training to promote the agent to explore the environment extensively. As training progresses, it is gradually reduced so that the agent can focus on learning the optimal policy.

[0012] Preferably, in step S6, the parameter update of the target network adopts a delayed update strategy. Every fixed number of network parameter updates, the parameters of the actor network and the commentator network are copied to the target network to update the parameters of the target network.

[0013] This invention also provides a brain-like reinforcement learning system based on hierarchical experience playback, comprising: Initialization module: Used to initialize the actor network, critic network and corresponding target network, as well as the experience buffer pool and parameters; Interaction module: In each training round, initialize exploration noise, obtain the initial state of the environment, control the agent to select actions from the actor network to execute, and store experience samples in the experience buffer pool; Short-term memory pool management module: responsible for retrieving new samples from the experience buffer pool to update the short-term memory experience pool, and deleting outdated samples according to the FIFO strategy; Attention filtering module: Calculates the similarity of experience samples and determines whether an experience is transferred from the short-term memory experience pool to the long-term memory experience pool based on the similarity and the number of repetitions; Network update module: Samples are drawn from the short-term memory experience pool and the long-term memory experience pool, input into the actor network and the critic network to update parameters, and the parameters of the actor network and the critic network are copied into the target network at a certain update frequency to achieve delayed update of the target network.

[0014] The present invention also provides a computer device, including: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described brain-like reinforcement learning method based on hierarchical experience playback.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described brain-like reinforcement learning method based on hierarchical experience playback.

[0016] Therefore, this invention proposes a brain-like reinforcement learning method and system based on hierarchical experience playback, the beneficial effects of which are as follows: (1) Improve experience utilization: By designing a hierarchical experience pool, experiences are divided into short-term memory experiences and long-term memory experiences for separate management and utilization. The short-term memory experience pool can update experiences in a timely manner, allowing the agent to quickly acquire the latest environmental information; the long-term memory experience pool stores important experiences that have been filtered, avoiding the loss of important information. During the learning process, the agent can make full use of experiences of different time levels and importance, effectively improving the utilization rate of experience.

[0017] (2) Optimizing decision-making efficiency: The introduction of the attention discrimination module enables the agent to more accurately filter out experiences valuable for long-term decision-making. By evaluating experience similarity and repetition frequency, experiences with high long-term retention value are transferred to the long-term memory experience pool, providing strong support for the agent to make long-term and stable decisions in complex environments. This helps the agent make more reasonable decisions when facing various situations, improving decision-making efficiency and quality.

[0018] (3) Improved learning effectiveness and performance: The brain-like reinforcement learning method and system based on hierarchical experience replay proposed in this invention comprehensively considers the temporal hierarchy and importance of experience and optimizes the experience replay process. During training, the agent can learn environmental rules more efficiently and converge to the optimal policy faster, thereby improving the learning effectiveness and performance of reinforcement learning. In practical applications, this method and system can significantly improve the performance of the agent in various tasks and has broad application prospects.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a flowchart of a brain-like reinforcement learning method based on hierarchical experience playback according to the present invention. Detailed Implementation

[0021] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 like Figure 1 The diagram shows a flowchart of a brain-like reinforcement learning method based on hierarchical experience playback according to the present invention, which includes the following steps: 1. Data acquisition and preprocessing.

[0024] Collect observation data of the interaction between the agent and the environment and perform data preprocessing to construct hierarchical training data for reinforcement learning networks. The observation data includes state feature vector data, action encoding data, instant reward signal data, visual observation frames and task description text. The specific operations of data preprocessing include denoising the observed data, filtering out outliers and invalid data points to ensure the stability of the state feature vector. Numerical features (such as state feature vectors and reward signals) are normalized or standardized to improve the numerical stability of the training process. Discrete action codes are encoded using one-hot encoding or embedding representation to adapt to the input requirements of different network structures. The observed data collected in each interaction are integrated into a set of vectors as training input to the reinforcement learning network to improve the stability and efficiency of policy optimization.

[0025] 2. Network and parameter initialization.

[0026] 2.1 First, initialize the actor network. and the network of critics The parameters are set using random initialization, with weights and biases of neurons in each layer of the network randomly generated using a normal distribution. The target network is then initialized. and The parameters of the target network are set to be the same as the initial parameters of the actor network and the critic network. The parameters of the target network will be updated according to a certain delayed update strategy to improve the stability of the learning process.

[0027] 2.2 Initialize the experience buffer pool. The experience buffer pool can be implemented using data structures such as queues or lists. Set the maximum capacity of the experience buffer pool. When the experience buffer pool reaches its maximum capacity, new experience samples will overwrite the oldest experience samples.

[0028] 2.3 Set a discount factor γ, typically between 0 and 1. The discount factor measures the importance of future rewards; a larger discount factor indicates a greater emphasis on future rewards, while a smaller discount factor focuses more on current rewards.

[0029] 2.4. Set the maximum number of episodes M and maximum time steps T The maximum number of episodes determines the total number of training rounds, while the maximum number of time steps limits the duration of the agent's interaction with the environment in each training round.

[0030] 3. Training process.

[0031] 3.1 At the beginning of each training round, initialize the exploration noise. The exploration noise uses random noise such as Gaussian noise for the agent's action space. A This generates a Gaussian noise vector ϵ with a mean of 0 and a standard deviation of σ. The noise standard deviation σ can be adjusted according to the training phase. In the early stages of training, a larger σ value helps the agent explore the environment more extensively; as training progresses, the σ value is gradually decreased, allowing the agent to focus more on learning the optimal policy.

[0032] 3.2 The agent obtains the initial state of the current environment. Based on the current state Selecting actions from the actor network The actions output by the actor network can be discrete or continuous. If the actor network outputs a probability distribution of actions, then specific actions are obtained by sampling based on that probability distribution; if the output is a deterministic action, then that action is used directly.

[0033] 3.3 The agent performs actions It interacts with the environment, and the environment returns a reward after the action is performed. and the next state . Experience samples Stored in the experience buffer pool.

[0034] 4. Experience pool update and transfer.

[0035] 4.1 The short-term memory experience pool retrieves new samples from the experience buffer pool. If the short-term memory experience pool is full, the earliest sample is deleted according to the FIFO strategy to ensure that the experience pool can be updated in a timely manner.

[0036] 4.2 The attention discrimination module starts working and calculates new experience samples. Cosine similarity between the empirical sample and other empirical samples in the short-term memory experience pool. Assume that an empirical sample can be represented as a feature vector. The feature vector of the new experience sample is Then the empirical sample i Cosine similarity with new empirical samples The calculation formula is: .

[0037] 4.3 For each experience sample in the short-term memory experience pool, if it exceeds a threshold β, it is rated as similar, and the number of times it is repeatedly rated as similar is recorded. If the number of times an experience sample is repeatedly rated as similar reaches a set number... K If so, the experience sample is transferred from the short-term memory experience pool to the long-term memory experience pool.

[0038] 5. Network parameter update 5.1 Randomly draw from the short-term memory pool N 1 Each sample is randomly drawn from the long-term memory pool. N 2 One sample. N 1 and N 2 The value can be adjusted according to the actual situation. 5.2. Input the extracted samples into the network for parameter updates. First, calculate the target of the samples based on the critic network. Q Value, for a sample in the termination state, target Q Value equals reward r For samples in non-terminating states, the target Q The formula for calculating the value is as follows: ; in, For the goal Q value, For the reward function, The state at time t, for t Actions that are happening all the time Actions calculated by the target actor network. For critics' goals Q The value network calculates the next state and action. Q value; Then, based on the extracted samples and the calculated target... Q The value is used to calculate the loss function, as shown in the following formula: ; in, For loss function, For the first i The target of each sample Q value, It is the network of critics on the first i Calculated from samples Q value, s For the state of the sample, a The action selected for the sample; The parameters of the critic network are updated by minimizing the loss function.

[0039] 5.3 Update the actor network using the policy gradient method. The goal of the policy gradient is to maximize the expected return, and its basic form is: ; in, For the objective function J(θ) The gradient with respect to the policy parameter θ, In strategy The result is the probability-weighted average of all possible actions 'a'. For the target strategy.

[0040] The importance sampling method is introduced to correct the policy gradient. Furthermore, to reduce the variance of importance sampling, truncated importance weights are introduced, and the gradient update formula is modified as follows: ; in, , c To truncate the threshold, p The ratio of the probability of the target policy to the probability of the action policy is used to update the actor network parameters based on the policy gradient. ; in, This is the learning rate.

[0041] 5.4. Copy the parameters of the actor network and the critic network to the target network at a certain update frequency to achieve delayed updates of the target network.

[0042] Example 2 A brain-inspired reinforcement learning system based on hierarchical experience playback includes: Initialization module: Used to initialize the actor network, critic network and corresponding target network, as well as the experience buffer pool and parameters; Interaction module: In each training round, initialize exploration noise, obtain the initial state of the environment, control the agent to select actions from the actor network to execute, and store experience samples in the experience buffer pool; Short-term memory pool management module: responsible for retrieving new samples from the experience buffer pool to update the short-term memory experience pool, and deleting outdated samples according to the FIFO strategy; Attention filtering module: Calculates the similarity of experience samples and determines whether an experience is transferred from the short-term memory experience pool to the long-term memory experience pool based on the similarity and the number of repetitions; Network update module: Samples are drawn from the short-term memory experience pool and the long-term memory experience pool, input into the actor network and the critic network to update parameters, and the parameters of the actor network and the critic network are copied into the target network at a certain update frequency to achieve delayed update of the target network.

[0043] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0044] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0045] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0046] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0047] Therefore, this invention provides a brain-like reinforcement learning method and system based on hierarchical experience replay. By introducing a brain-like mechanism to optimize the efficiency of experience replay, and by utilizing the thinking patterns of short-term and long-term memory to optimize the experience replay process, the invention improves the experience utilization rate and decision-making efficiency of the agent in the learning process, thereby enhancing the learning effect and performance of reinforcement learning.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A brain-like reinforcement learning method based on hierarchical experience playback, characterized in that, Includes the following steps: Step S1: Collect observation data of the interaction between the agent and the environment and perform data preprocessing to construct hierarchical training data for reinforcement learning networks. The observation data includes state feature vector data, action encoding data, instant reward signal data, visual observation frames and task description text. Step S2: Initialize the actor network, critic network, and corresponding target network. Simultaneously, initialize the experience buffer pool and perform parameter initialization, including the discount factor γ and the maximum number of episodes. M and maximum time steps T ; Step S3: At the beginning of each training round, initialize the exploration noise, the agent obtains the initial state of the current environment, selects an action from the actor network based on the current state and executes it, and stores the experience samples obtained after executing the action into the experience buffer pool. Step S4: Short-term memory pool update. The short-term memory experience pool obtains new samples from the experience buffer pool. If the short-term memory experience pool is full, outdated samples are deleted according to the first-in-first-out strategy. Step S5: Experience screening and transfer. The attention discrimination module is used to calculate the similarity between the new experience sample and the existing experience samples in the short-term memory experience pool. The similarity is compared to see if it exceeds the threshold. The number of times the experience is repeatedly rated as similar is recorded. It is then decided whether to transfer some of the experience in the short-term memory experience pool to the long-term memory experience pool. Step S6: Network parameters are updated by retrieving data from the short-term memory experience pool. N 1 One sample was obtained from the long-term memory experience pool. N 2 A set of samples are input into the actor network and the critic network for parameter updates. The target network's parameters are updated at fixed intervals of network parameter updates. N 1 and N 2 It is a positive integer.

2. The brain-like reinforcement learning method based on hierarchical experience playback according to claim 1, characterized in that, In step S1, the data preprocessing includes: denoising the observed data, filtering out outliers and invalid data points, normalizing or standardizing the numerical feature data, and performing one-hot encoding or embedding representation on the discrete action encoding.

3. The brain-like reinforcement learning method based on hierarchical experience playback according to claim 1, characterized in that, The short-term memory experience pool has a limited capacity first-in-first-out (FIFO) structure, and the FIFO strategy ensures that the experience in the pool always remains timely.

4. The brain-like reinforcement learning method based on hierarchical experience playback according to claim 1, characterized in that, The long-term memory experience pool is a complete experience pool, in which the experiences are derived from the screening of the short-term memory experience pool. The attention discrimination module evaluates the long-term preservation value of the experiences in the short-term memory experience pool and decides whether to transfer them.

5. The brain-like reinforcement learning method based on hierarchical experience playback according to claim 4, characterized in that, The attention discrimination module uses the cosine similarity method to calculate the similarity between experience samples. By comparing the cosine similarity between the feature vectors of new experience samples and existing samples in the short-term memory experience pool, the degree of similarity between the two is quantified.

6. The brain-like reinforcement learning method based on hierarchical experience playback according to claim 1, characterized in that, In step S3, Gaussian noise is used for exploration noise. Its standard deviation is set to a large value in the early stage of training to promote extensive exploration of the environment by the agent. As training progresses, it is gradually reduced so that the agent can focus on learning the optimal policy.

7. The brain-like reinforcement learning method based on hierarchical experience playback according to claim 1, characterized in that, In step S6, the parameter update of the target network adopts a delayed update strategy. Every fixed number of network parameter updates, the parameters of the actor network and the critic network are copied to the target network to update the parameters of the target network.

8. A brain-like reinforcement learning system based on hierarchical experience playback, characterized in that, include: Initialization module: Used to initialize the actor network, critic network and corresponding target network, as well as the experience buffer pool and parameters; Interaction module: In each training round, initialize exploration noise, obtain the initial state of the environment, control the agent to select actions from the actor network to execute, and store experience samples in the experience buffer pool; Short-term memory pool management module: responsible for retrieving new samples from the experience buffer pool to update the short-term memory experience pool, and deleting outdated samples according to the FIFO strategy; Attention filtering module: Calculates the similarity of experience samples and determines whether an experience is transferred from the short-term memory experience pool to the long-term memory experience pool based on the similarity and the number of repetitions; Network update module: Samples are drawn from the short-term memory experience pool and the long-term memory experience pool, input into the actor network and the critic network to update parameters, and the parameters of the actor network and the critic network are copied into the target network at a certain update frequency to achieve delayed update of the target network.

9. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, it implements the steps of the brain-like reinforcement learning method based on hierarchical experience playback as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of a brain-like reinforcement learning method based on hierarchical experience playback as described in any one of claims 1-7.

Images