Self-organizing reinforcement learning method with priority experience playback function

Abstract

The invention relates to a self-organizing reinforcement learning method with priority experience playback in the technical field of robot intelligent control, and the method combines a deep neural network (DNN) and a self-organizing learning map (SOM), simulates a learning mechanism of a new cortex and a hippocampus in a brain, and carries out reinforcement learning. Wherein the deep neural network (DNN) is used for simulating the characteristics of new cortex slow learning distributed representation, and the self-organizing learning map (SOM) is used for simulating the characteristics of hippocampus rapid learning and mode separation; and dynamically regulating and controlling the balance of exploration and utilization (EET) by adopting a priority sampling strategy considering a time difference (TD) error, a Bellman (BE) error and an empirical familiarity factor index. According to the method, the problems of low data utilization rate and poor learning efficiency in deep reinforcement learning are effectively relieved, the sampling efficiency of key experience and the accuracy of strategy evaluation are improved, and the method is very effective in robot behavior decision making.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to the field of robot intelligent control technology. More specifically, the present application relates to a self-organizing reinforcement learning method with prioritized experience replay. Background Art

[0002] In the era of booming artificial intelligence, reinforcement learning, as a key technology area, has been widely and intensively researched. The core idea of reinforcement learning is to simulate the reward-driven behavioral learning process of organisms. Through continuous trial and error, intelligent agents gradually explore and approach optimal strategies. This technology has shown great application potential in a wide range of fields, including robotic control, gaming, and autonomous driving.

[0003] For researchers in the field of cognitive science, exploring the intrinsic connection between reinforcement learning theory and the brain's neural structure has always been an important and challenging research direction that has attracted much attention. In this research process, the discovery that midbrain dopaminergic neurons encode temporal difference errors has had a profound impact. These neurons are mainly concentrated in the midbrain region, where they project prediction signals to the striatum. The striatum plays a crucial role in the entire decision-making and evaluation process. It can simultaneously receive information from the cortex and hippocampus, and then integrate different types of information to provide strong support for decision-making and evaluation. This neural structure provides an important biological basis for reinforcement learning theory.

[0004] The complementary learning systems theory further reveals the brain's sophisticated mechanisms of memory and learning. It suggests that the neocortex and hippocampus each possess unique strengths in memory and learning. The hippocampus rapidly captures new information through its fast, pattern-separating memory storage, while the neocortex slowly learns distributed representations, generalizing across experiences and features to build long-term, stable knowledge structures. This collaborative approach enables organisms to respond flexibly and efficiently to complex and changing environments, processing a wide range of information.

[0005] Inspired by this biological theory, many researchers are actively exploring its application in the field of artificial intelligence, hoping to leverage this naturally evolved wisdom to improve the efficiency and adaptability of machine learning systems. Deep reinforcement learning (RL) emerged in this context. It innovatively combines traditional RL methods with deep neural networks (DNNs), leveraging the powerful representational capabilities of DNNs to approximate action-value functions in high-dimensional state spaces. Deep reinforcement learning has achieved remarkable results in numerous applications, demonstrating performance advantages over traditional methods in fields such as Go and robotic control.

[0006] However, during the development process, some problems that urgently need to be solved have emerged in deep reinforcement learning. Among them, low data efficiency and insufficient robustness to changes in the input distribution have become key factors restricting its further development and wide application. To solve these problems, researchers have turned their attention to the "hippocampus" learning mechanism in biology, attempting to gain inspiration from it.

[0007] The early Deep Q-network adopted the experience replay mechanism, which to a certain extent simulated the process of the hippocampus replaying past experiences in a dormant state. In this way, the network can break the correlation between data, improve the stability of training and the utilization rate of samples, and alleviate the problems faced by deep reinforcement learning to a certain extent. However, the uniform sampling method adopted by this mechanism does not fully consider the importance differences of different experiences, resulting in difficulty in further improving the learning efficiency.

[0008] To improve this defect, the Prioritized Experience Replay (PER) strategy came into being. The PER strategy uses the Temporal Difference (TD) error (hereinafter referred to as "TD" error) as the priority standard for sampling, enabling experiences with higher TD errors to be sampled more frequently, effectively improving the sample utilization rate and training efficiency. This strategy has been applied in multiple reinforcement learning frameworks and achieved good results. For example, M et al. applied it to the robot path planning task, and JiL et al. applied it to the field of multi-agent collaborative control, both achieving good results.

[0009] Although the PER strategy has improved the performance of deep reinforcement learning to a certain extent, its limitations have gradually emerged when facing a stochastic environment. Relying solely on TD error to measure the importance of experiences has deficiencies. Especially near the state boundary, the TD error may increase abnormally, resulting in an unstable learning process.

[0010] To address this problem of the PER strategy, some scholars have proposed integrating episodic memory into the learning process to improve the sampling efficiency of agents. For example, Lin Z et al. proposed the Episodic Memory Deep Q-Network, which uses episodic memory to supervise the training of agents and improves the learning efficiency of agents to a certain extent. However, this method stores the highest reward obtained by taking action a from state s in memory, which makes the algorithm perform poorly when generalized to new states. Srivastava S et al. improved this by proposing the Cognitive Heuristic Episodic Memory of the DQN network, which stores all the obtained return values in memory instead of only the maximum value, and improves the data efficiency of the algorithm by retaining and learning more information. At the same time, the episodic memory strategy has also been explored in practical applications. Hassani H et al. applied the prioritized memory replay strategy guided by episodic memory to the DQN framework to achieve traffic navigation tasks in autonomous driving; Liu D et al. applied it to related tasks of robotic arms, effectively improving the learning speed of robot operation skills.

[0011] In addition to the above methods, some researchers have also considered entropy-based methods and reward-value-based methods to improve the sampling efficiency of experience. However, these sampling strategies that only adopt a single criterion have obvious drawbacks. The selected experiences often fail to achieve the ideal effect and perform poorly in balancing exploration and exploitation (EET), and this problem is particularly prominent in practical application scenarios.

[0012] To overcome the limitations of single-criterion sampling strategies, researchers have begun to explore the construction of multi-criterion sampling priority systems. Brittain M et al. considered the correlation between experiences and proposed the Prioritized Order Experience Replay algorithm; Cao X et al. proposed the High-Value First Experience Replay method, which uses both the action value function value and the TD error as sampling metrics; Liu X et al. comprehensively considered four metrics: TD error, state entropy, reward, and the number of experience replays. Although these multi-criterion techniques have improved the limitations of PER to a certain extent, most methods are still limited to sorting experiences in a static framework and cannot flexibly adjust the weights of different sampling criteria according to the dynamic changes of the environment and the training process during training. In a complex and changing real environment, the optimal sampling strategy needs to evolve continuously with the progress of training and the changes of the environment, and there is still much room for improvement in current technical means.

[0013] As is well known, in reinforcement learning, the immediate value of an action in a certain state can be judged according to the TD error, and the values of all potential actions in this state can be judged through the Bellman (BE) error (hereinafter referred to as the BE error). Hassani H et al. proposed a method that uses the TD error and the Bellman (BE) error as indicators for priority sampling. By dynamically regulating the TD error and the BE error, it balances the EET, thereby improving the sampling efficiency and learning quality. However, this method has some deficiencies: it adaptively adjusts the proportion of the TD error and the BE error in the weight scoring rule according to the gradient-based optimization strategy. However, the adaptive change of this proportion changes with the increase in the number of training times. Therefore, how to define the early stage and the later stage in different tasks, as well as the ratio of the two weights in a complete training round, brings more difficulties to the design of the algorithm.

[0014] In addition, from the perspective of the complementary learning system theory, the hippocampus relies on fast memory storage with pattern separation characteristics, while the neocortex realizes generalization across experiences and features through slow learning of distributed representations. However, most traditional methods only utilize the framework of a single learning system, that is, introducing the replay memory characteristics of the "hippocampus" in biology into deep reinforcement learning, without considering the framework of complementary learning of multiple learning systems.

[0015] To sum up, although significant progress has been made in the field of reinforcement learning in past research, there are still problems such as low data utilization efficiency, poor learning efficiency, and the need to further improve the sampling efficiency of key experiences and the accuracy of policy evaluation. Therefore, it is urgent to further improve. Summary of the Invention

[0016] The object of the present invention is to simulate the learning mechanisms of the "neocortex" and the "hippocampus" and propose a self-organizing reinforcement learning method with priority experience replay (SORLPER).

[0017] Based on the above object of the invention, the present application proposes a self-organizing reinforcement learning method with priority experience replay, including the following steps:

[0018] S1. Combine a deep neural network (DNN) and a self-organizing learning map (SOM) to perform reinforcement learning by simulating the learning mechanisms of the neocortex and the hippocampus in the brain. Among them, the deep neural network (DNN) is used to simulate the characteristics of slow learning of distributed representations in the neocortex, and the self-organizing learning map (SOM) is used to simulate the characteristics of fast learning and pattern separation in the hippocampus;

[0019] S2. During the process of reinforcement learning, a prioritized sampling strategy that takes into account the temporal difference (TD) error, Bellman (BE) error, and experience familiarity factor metrics is adopted to dynamically regulate the balance between exploration and exploitation (EET), that is, a self-organizing reinforcement learning method with prioritized experience replay is obtained, denoted as SORLPER.

[0020] Preferably, in step S1, the deep neural network (DNN) and the self-organizing learning map (SOM) work in parallel. The deep neural network (DNN) is responsible for generalizing the state space, while the self-organizing learning map (SOM) focuses on storing experiences that are difficult to evaluate by the deep neural network (DNN) and has a pattern separation effect on the experiences.

[0021] Preferably, in step S1, the combination of the deep neural network (DNN) and the self-organizing learning map (SOM) includes the following three aspects:

[0022] Step S11. Combination of action selection: The SORLPER adopts the greedy policy ε-greedy and selects actions according to the weighted Q-values of both the deep neural network (DNN) and the self-organizing learning map (SOM);

[0023] Step S12. Combination of experience sampling: The SORLPER preferentially samples according to the weighted average score of the TD error and the BE error, and introduces importance sampling weights for bias correction;

[0024] Step S13. Combination of network update: The online network update of the SORLPER depends on the objective function jointly obtained by the deep neural network (DNN) and the self-organizing learning map (SOM) in the target network. The update of the neuron weights and the neighborhood function in the self-organizing learning map (SOM) also depends on the TD error generated by the deep neural network (DNN). This dynamic update process enables the SORLPER to preferentially consider and retain the state experiences with less accurate predictions in the deep neural network (DNN) part.

[0025] Preferably, in step S12, the importance sampling weight is denoted as ω(i), and its calculation formula is:

[0026]

[0027] where M represents the size of the replay buffer, p(i) is the priority of the i-th transition, and β is an adjustable hyperparameter.

[0028] Preferably, step S2 includes the following steps:

[0029] S21. Interact the decisions of the deep neural network (DNN) and the self-organizing learning map (SOM) with the environment, and store the interaction experiences in the experience replay buffer;

[0030] S22. Obtain the corresponding TD error and BE error values according to the familiarity of each transition sample by SORLPER;

[0031] S23. Obtain a score by weighted summation of the TD error and the BE error, and then obtain the sampling probability of the sample through sampling priority processing, realizing the adjustment of exploration and exploitation (EET) to accelerate the convergence speed and learning quality of the reinforcement learning process.

[0032] Preferably, in step S2: The experience familiarity factor determines the contribution degree of the deep neural network (DNN) and the self-organizing learning map (SOM) to the Q value, denoted as η, and its specific calculation formula is as follows:

[0033]

[0034] Among them, τ η is a hyperparameter. It can be seen that if the value of η is larger, the contribution of the self-organizing learning map (SOM) is greater; if the value of η is smaller, the contribution of the deep neural network (DNN) is greater; the activated neuron u of the self-organizing learning map (SOM) t is obtained according to the Euclidean distance between the state input s t and the state weight w u That is

[0035] u t = argmin u ||w u - s t || 2 .

[0036] The main working principle of this application is:

[0037] (1) In terms of action decision-making, the agent is jointly regulated by two systems, and it balances the Q value contribution degrees of the DNN and the SOM according to the familiarity with the state;

[0038] (2) Designed a priority sampling rule considering the TD error and the BE error. This rule dynamically adjusts the weight ratio of the TD error and the BE according to the familiarity with the state, thereby regulating the EET balance, and can further improve the learning speed of the agent.

[0039] It is worth mentioning that the rule considering the familiarity factor in (1) and (2) is more in line with the human episodic memory pattern;

[0040] (3) In addition to using the priority sampling rule in (2) to enable DNN and SOM to preferentially learn those more valuable state actions, thereby accelerating the learning efficiency of the agent, compared with DNN, since the learning rate of SOM is larger, the complementary learning between the two systems can further improve the data utilization rate.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. A new framework based on the parallel operation of DNN and SOM is designed, corresponding to the learning mechanisms of the neocortex and the hippocampus in the brain respectively. Among them, DNN is responsible for generalizing most regions in the state space, while SOM focuses on storing those experiences that are difficult for DNN to evaluate, and SOM has a pattern separation effect on these experiences. Through the complementary learning of these two systems, the sampling efficiency of key experiences and the accuracy of policy evaluation can be effectively improved;

[0043] 2. A new multi-criterion adaptive priority sampling technique is proposed, which fuses TD error and BE error in a unified scoring function. The agent can adaptively regulate the sampling of experiences according to the familiarity with the experiences, and this strategy can further improve the data utilization efficiency of the algorithm;

[0044] 3. It provides a new idea for the balance problem between exploration and exploitation in deep reinforcement learning, and further promotes the understanding of the role of biological neural mechanisms in intelligent decision-making.

[0045] In summary, the present invention effectively alleviates the problems of low data utilization rate and poor learning efficiency in deep reinforcement learning by simulating the learning mechanisms of the hippocampus and the neocortex, while improving the sampling efficiency of key experiences and the accuracy of policy evaluation, and is very effective in robot behavior decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects, and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0047] Figure 1 Shows the overall architecture of a self-organizing reinforcement learning method with priority experience replay proposed in an embodiment of the present invention;

[0048] Figure 2 Shows the priority update process in an embodiment of the present invention;

[0049] Figure 3 Shows the simulation experiment environment in an embodiment of the present invention, where Figure 3 (a) The shown environment 1 contains three obstacles surrounded by different obstacle points, Figure 3The environment 2 shown in (b) contains four obstacles surrounded by different obstacle points;

[0050] Figure 4 The average reward bar chart of the SORLPER algorithm under different parameters in the embodiments of the present invention is shown;

[0051] Figure 5 The average score of the SORLPER algorithm under different parameters in the embodiments of the present invention is shown;

[0052] Figure 6 The reward comparison line chart of the SORLPER algorithm, SORLPER-TD algorithm, and RLPER algorithm in the embodiments of the present invention is shown;

[0053] Figure 7 The average score bar chart of the SORLPER algorithm, SORLPER-TD algorithm, and RLPER algorithm in the embodiments of the present invention is shown;

[0054] Figure 8 The average reward comparison result chart of the SORLPER algorithm under two strategies in the embodiments of the present invention is shown;

[0055] Figure 9 The reward comparison line chart of the SORLPER algorithm, PER algorithm, and BETDQN algorithm in the embodiments of the present invention is shown;

[0056] Figure 10 The architecture of the actual navigation decision-making process of the robot in the embodiments of the present invention is shown;

[0057] Figure 11 Two environments built in the robot physical simulation software Gazebo in the embodiments of the present invention are shown;

[0058] Figure 12 The simple experimental environment in the embodiments of the present invention is shown;

[0059] Figure 13 The navigation trajectory map of the robot in the embodiments of the present invention is shown. Detailed implementation manners

[0060] Next, the technical solutions of the present invention will be described clearly and completely in conjunction with the attached Figures 1 - 13 Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] This embodiment proposes a self-organizing reinforcement learning method with prioritized experience replay, including the following steps:

[0062] S1. Combine a deep neural network (DNN) and a self-organizing learning map (SOM) to perform reinforcement learning by simulating the learning mechanisms of the neocortex and hippocampus in the brain. Among them, the deep neural network (DNN) is used to simulate the characteristics of slow learning and distributed representation of the neocortex, and the self-organizing learning map (SOM) is used to simulate the characteristics of fast learning and pattern separation of the hippocampus;

[0063] S2. During the process of reinforcement learning, adopt a priority sampling strategy considering the temporal difference (TD) error, Bellman (BE) error, and empirical familiarity factor index to dynamically regulate the balance between exploration and exploitation (EET), that is, obtain a self-organizing reinforcement learning method with priority experience replay, denoted as SORLPER.

[0064] Specifically, in step S1, the deep neural network (DNN) and the self-organizing learning map (SOM) work in parallel. Among them, the deep neural network (DNN) is responsible for generalizing the state space, while the self-organizing learning map (SOM) focuses on storing experiences that are difficult for the deep neural network (DNN) to evaluate and has a pattern separation effect on the experiences.

[0065] In the prior art, the core goal of reinforcement learning is for an agent to interact with the environment, thereby perceiving the state of the environment and selecting actions according to its own strategy to pursue the maximum future expected reward. Specifically, the agent selects an action a t in the state s t . The environment then responds to this action, generating the next state s t+1 and the reward r t+1 . This process continues until the termination state is reached. Define the cumulative reward return R t at time t as:

[0066]

[0067] where γ ∈ (0, 1) is the discount factor, making the value of immediate rewards higher than that of future rewards, r t' represents the immediate reward obtained at time t', and T is the time step when the task or episode ends. Solving the reinforcement learning problem is equivalent to learning a policy (i.e., the mapping from state to action), such that the expected return starting from each state is maximized. At this time, the problem can be transformed into finding an optimal policy π*. The problem of finding π* can be transformed into learning the optimal action-value function Q*(s, a), which represents the expected return that can be obtained by following π* after performing action a in state s, and can be expressed by the following formula:

[0068]

[0069] Once the agent has learned Q*(s,a), it can make optimal decisions by selecting the action with the maximum Q value in the current state (i.e., argmax a Q(s,a)). Among them, the optimal action-value function can be defined by the Bellman equation as follows:

[0070] Q*(s,a) = E s' [r + γmaxQ*(s',a')|s,a] (3)

[0071] As an off-policy reinforcement learning algorithm, DQN uses an "online network" to estimate future discounted rewards. In DQN, the policy π is obtained by calculating the value function Q(s,a) with parameters θ. Specifically, DQN updates the parameters of the online network through the following optimization objective to extract the final policy:

[0072]

[0073] Among them, represents the parameters of the target network. The target network parameters will be synchronized with the training network every certain number of steps to enhance the stability of the learning process.

[0074] The overall architecture of the algorithm proposed in this application is as shown in Figure 1 . First, the agent interacts with the environment through the decisions of two learning systems, and the interaction experience is stored in the experience replay buffer in the form of (s t ,a t ,r t+1 ,s t+1 ). Different from the traditional DQN algorithm that randomly selects transition samples, SORLPER calculates the corresponding TD error and BE error values according to its familiarity with each transition sample. Then, the TD error and BE are scored through the weighted summation rule, and then the sampling probability of the sample is obtained through the sampling priority process. The higher the sampling probability, the greater the possibility that the transition is selected during the training of the network. Through the above process, the adjustment of exploration and exploitation (EET) is realized to accelerate the convergence speed and learning quality of the agent learning process.

[0075] The following content details the SORLPER algorithm (abbreviation: "SORLPER algorithm"): The main core of the SORLPER algorithm is the combination of two algorithms, DNN and SOM. This combination is mainly carried out from three aspects: action selection, experience sampling, and network update. Next, the proposed SORLPER algorithm will be described from these three aspects.

[0076] (1) The combination of the two learning systems in terms of action selection

[0077] Traditional DNNs select actions based on the magnitude of Q-values. Similar to DNNs, SOMs also represent the value of an agent's action in a particular state through the state-action value Q. Specifically, in SOM's representation of states, SOM classifies states into corresponding units u, and each unit u contains the action value Q(u,a) corresponding to that state and the corresponding weight β u . When an agent using the SORLPER algorithm makes a decision, it adopts the greedy strategy ε-greedy and selects actions based on the weighted Q-values of both DNN and SOM, as shown in the following formula:

[0078]

[0079] Among them, the familiarity factor η of the agent with respect to this state determines the contribution degrees of DNN and SOM to the Q-value. Its specific calculation formula is as follows:

[0080]

[0081] Among them, τ η is a hyperparameter. It can be seen that if the value of η is larger, the contribution of SOM is greater; if the value of η is smaller, the contribution of DNN is greater. In addition, the activated neuron u of SOM t is obtained based on the Euclidean distance between the state input s t and the weight w u , that is

[0082] u t = argmin u ||w u - s t || 2 (7)

[0083] (2) Combination of the two learning systems in terms of experience sampling

[0084] In traditional experience sampling strategies, most methods sample based on TD errors. Equation (3) gives the calculation method of the optimal Q function in state s'. However, during the training process, the learned Q(s,a) function may not satisfy the Bellman equation given by Equation (3). Given that the TD error is the immediate calculation result of the difference between both sides of the Bellman equation (i.e., not including the expectation part), the TD error calculation formula in traditional DNNs is as follows:

[0085]

[0086] The calculation of formula (8) is only for a single action, that is, the action with the maximum value. The traditional BE error (the difference between both sides of formula (3) is the BE error) gives a more general case, which considers all possible actions in a specific state, as shown in the following formula:

[0087]

[0088] To ensure the balance between exploration and exploitation during the training process, the SORLPER algorithm considers the TD error, BE error, and experience familiarity factor to formulate the sampling rule. Figure 2 The update process of the sampling priority of the SORLPER algorithm is given.

[0089] In this application, the SORLPER algorithm obtains the corresponding TD error and BE error values according to its familiarity with each transition sample. Specifically, the TD error in the SORLPER algorithm can be defined as:

[0090]

[0091] The BE error in the SORLPER algorithm can be defined as:

[0092]

[0093] According to the TD error, the immediate value of an action in a certain state can be judged, and through the BE error, the values of all potential actions in this state can be judged. Based on this, the SORLPER algorithm preferentially samples according to the weighted average score of the TD error and BE error. The weighted scoring rules for these two are as follows:

[0094] σ(i) = κ1δ θ (s t ,a t ,s t+1 ) + κ2ψ θ (s t ) (10)

[0095] Among them, the parameters κ1 and κ2 are hyperparameters.

[0096] To prevent bias from being introduced during training, the processing method in PER is adopted. According to the scoring function defined in formula (10), the priorities of each transition sample are adjusted. First, the scores of each transition are normalized. To prevent division-by-zero errors, a small value e of about 10 -6 order of magnitude is added to the scores, and then normalization is performed according to the following formula:

[0097]

[0098] Among them, represents the normalized score of the i-th group of experience tuples.

[0099] Next, according to the normalized scores, the probability of each transition being sampled is calculated, and its calculation formula is:

[0100]

[0101] Among them, q is an adjustable hyperparameter used to control the degree of priority allocation.

[0102] Finally, importance sampling weights are introduced for bias correction. Prioritized sampling of transition samples will change the distribution of samples in the replay buffer, thus introducing bias in the learning process. To this end, importance sampling weights are assigned to each transition, and its calculation formula is:

[0103]

[0104] where M represents the size of the replay buffer, p(i) is the priority of the i-th transition, and β is an adjustable hyperparameter.

[0105] (3) Network update process of two learning systems

[0106] After sampling a batch of experiences according to the prioritized policy, the DNN and SOM learning systems update the network according to these experiences. The network update of DNN is similar to that of PER. After initializing the online network, the target network, and the current step t, iterations are carried out in multiple episodes. In each episode, as long as the number of transitions in the replay buffer exceeds c, training starts, and the parameters of the online network are updated using the selected batch of samples; every c' steps, the parameters of the online network are copied to the target network. Different from PER, in the SORLPER algorithm, the update of the online network DNN in the SORLPER algorithm depends on the target function y jointly obtained by the two learning systems of the target network DNN and SOM. The target function y is:

[0107]

[0108] The online network DNN performs gradient descent on (y t -Q θ (s t-1 ,a t-1 )) 2 to update the network parameters. The update of SOM depends on the TD error generated by DNN, and this error can be obtained from the target function y t and the Q value of the online network DNN. The formula is as follows:

[0109] δ = exp(|y t -Q θ (s t-1 ,a t-1 )| / τ δ ) - 1 (15)

[0110] where τ δis a constant. The update of SOM is mainly divided into two parts: one is the update of its weights, and the other is the update of the action values. Specifically, the update method of the weights is set as:

[0111]

[0112] where, w j,i represents the connection weight between the j-th neuron in the input layer and the u-th neuron in the middle layer i , j ∈ {1, 2,..., D} represents the serial number of the input neurons, D represents the size of the input dimension, i ∈ {1, 2,..., L} represents the serial number of the middle layer neuron u, and L represents the number of middle layer neurons. λ is the learning rate for the update of the weight w j,i . s j,t represents the state input at time t. represents the neighborhood function formed by the middle layer neuron u i and the activated neuron u t at time t, specifically

[0113]

[0114] where, φ is a constant. The update method of the action value Q SOM (u t , a t ) is set as:

[0115] Q SOM (u t , a t ) ← Q SOM (u t , a t ) + αη t [y t+1 - Q SOM (u t , a t )](18)

[0116] where, α is the learning rate, and η t is the familiarity degree factor of the agent for this state defined in Equation (6).

[0117] It should be noted that the update of the neuron weights and the neighborhood function in SOM also depends on the TD error generated by DNN. This dynamic update process enables the SORLPER algorithm to give priority to and retain the state experiences that are not accurately predicted by the DNN part. Through this parallel learning method of DNN and SOM, the overall learning ability of the system can be enhanced.

[0118] To further understand the proposed SORLPER algorithm, Algorithm 1 gives the working process of the algorithm. First, initialize the action value function Qθ , the target action value function Q θ , the SOM state-action value Q SOM , and the state weight w u , and then start iterating for E rounds. During each round, first initialize information such as the agent state and the number of movement steps, and then start interacting with the environment. Specifically, first observe the current state, and obtain the SOM winning neuron u according to the Euclidean distance between the SOM neuron weights and the state t , and the familiarity factor η; then, obtain the execution action according to the Q-value contribution degrees of the DNN and SOM, and then store the interaction information of the agent and the environment, that is, the experience tuple (s t , a t , r t+1 , s t+1 , done) in the experience pool. When the number of stored experiences reaches the set threshold, start training. Obtain a small batch of experiences through the scoring rule (10), and use these experiences to update the weights and action values of the online networks DNN and SOM. Every time a round passes, the target network DNN is updated once. When the agent reaches the target point or the number of movement steps is greater than the maximum exploration steps, end this round. Complete the complete training of the agent by repeating the above process, and its working process is as follows:

[0119]

[0120]

[0121] The main working principle of this application is:

[0122] (1) In terms of action decision-making, the agent is jointly regulated by two systems, and it balances the Q-value contribution degrees of the DNN and SOM according to the familiarity with the state;

[0123] (2) Designed a priority sampling rule considering the TD error and the BE error. This rule dynamically adjusts the weight ratios of the TD error and the BE according to the familiarity with the state, thereby regulating the EET balance, and can further improve the learning speed of the agent.

[0124] It is worth mentioning that the rule considering the familiarity factor in (1) and (2) is more in line with the human episodic memory pattern;

[0125] (3) In addition to using the priority sampling rule mentioned in (2) to make the DNN and SOM preferentially learn those more valuable state actions, thereby accelerating the learning efficiency of the agent, compared with the DNN, since the SOM has a larger learning rate, the complementary learning between the two systems can further improve the data utilization rate.

[0126] Experiments and analysis

[0127] To verify the effectiveness and superiority of the proposed algorithm, this embodiment will conduct verification in numerical simulation navigation experiments and physical navigation experiments.

[0128] First, as Figure 3 shown, in the numerical simulation environment, a 2D unknown environment is set as the simulation experiment environment, and the specific settings are as follows:

[0129] (1) Set the starting point and ending point of the robot to be (50, 30) and (450, 450) respectively. The length and width of the environment are 500 unit lengths each, the moving step of the robot step is 10 unit lengths, and the input information of the robot is where θ g , d g represent the angle and distance between the robot and the target respectively, and θ o , d o represent the angle and distance between the robot and the obstacle respectively. The number of actions of the robot a is 36, and the robot movement is defined as state next = state + step * (cos(a * π / 18), sin(a * π / 18)).

[0130] Among them, Environment 1 contains three obstacles surrounded by different obstacle points; Environment 2 contains four obstacles surrounded by different obstacle points.

[0131] (2) Reward function setting:

[0132] In the navigation decision experiment, the robot needs to move from the starting point to the target point. The reward r during its navigation decision task is set as follows:

[0133]

[0134] Among them, and represent the distances between the robot and the target and the obstacle at time t respectively. d arrive , d safe and d arrive are set distance constants. When the distance is less than d arrive , it indicates that the task is successful. When the distance is less than d safe , it indicates that the task fails. d warning is the distance at which a warning is issued when a collision is imminent. Among them, d arrive = 10, d safe = 10, d warning = 25. The reward form for each movement of the robot is: r = r1 + r2 + r3 + r4.

[0135] To ensure the effectiveness of the SORLPER algorithm, parameter analysis experiments were first conducted to select appropriate parameters for the algorithm. Specifically, the parameter values were set as follows: the replay experience size M = 10000, the sampling batch size m = 64, the training start threshold c = 64, the target network update frequency c' = 1, the maximum number of attempt steps steps = 300, the discount factor γ = 0.98, the exploration rate ε = 1 → 0.001, the learning rate lr of the Adam optimizer used in the DNN = 0.001, the learning rate λ of SOM weight update = 0.01, and the learning rate α of Q-value update = 0.9, and the hyperparameters τ η = 10, τ δ = 1, φ = 0.1. The DNN contains two hidden layers with Relu activation functions, each having 128 neurons, and the SOM has one hidden layer with 15 neurons 2 . In the following comparison experiments, the parameters of each algorithm were set to the same values.

[0136] To ensure the effectiveness of the proposed SORLPER algorithm in the EET balance, multiple groups of different parameter analysis experiments were conducted in this section to select appropriate values of κ1 and κ2. Figure 4 、 Figure 5 The experimental results under seven different combinations of κ1 and κ2 are given.

[0137] Figure 4 This is the bar chart of the average rewards of the SORLPER algorithm under different parameters. The average rewards of 5 navigation decision experiments were statistically analyzed, and different random number seeds were used in each experiment. Note: The larger the average reward in the figure, the better the parameter.

[0138] Figure 5 This is the average score of the SORLPER algorithm under different parameters. The average scores of 5 navigation decision experiments were statistically analyzed. This score represents the cumulative number of successful tasks of the robot. When the robot reaches the target point without collision, the cumulative score is +1, otherwise +0. The higher the cumulative score, the better the parameter.

[0139] From Figure 4 、 Figure 5 it can be seen that when κ1 = 0.01 and κ2 = 0.99, the SORLPER algorithm achieved the best results in both the average reward and average score evaluations. Therefore, this set of relatively optimal parameters will be used for the SORLPER algorithm in subsequent experiments.

[0140] To verify the complementary learning characteristics between the DNN and SOM systems utilized by the SORLPER algorithm and the effectiveness of the priority sampling strategy for adjusting samples by combining TD error, BE error, and familiarity with experience, ablation experiments were conducted. The strategies include: (1) Utilizing the complementary learning characteristics between the DNN and SOM systems to improve the learning efficiency and quality of the agent; (2) Combining TD error, BE error, and familiarity with experience to adjust the priority sampling order of samples, further improving the learning efficiency of the agent.

[0141] During the experiment, the SORLPER algorithm was compared with the algorithm after removing the SOM part (RLPER) and the algorithm after adopting the traditional priority replay rule (SORLPER-TD). Figure 6 、 7 The average rewards and cumulative score graphs of these algorithms were given respectively. Additionally, to further observe the comparison effect, the average values of these average reward and cumulative score data were statistically analyzed in Table 1.

[0142] Figure 6 It is the line graph comparing the rewards of the SORLPER algorithm, SORLPER-TD algorithm, and RLPER algorithm. The reward of each algorithm is the average of 10 experiments, and different random number seeds were used. To facilitate observing the reward changes of the algorithms, all the reward curves were smoothed. The solid line in the figure depicts the change of the average reward value, and the shaded part represents the range of the average reward ± standard deviation. The faster the reward curve rises, the higher the learning efficiency of the agent; the larger the reward value, the better the learning effect of the agent.

[0143] Figure 7 It is the bar graph of the average scores of the SORLPER algorithm, SORLPER-TD algorithm, and RLPER algorithm. The average score results of the robot in 10 navigation decision experiments were statistically analyzed. This score represents the cumulative number of successful tasks on average. When the robot reaches the target point without collision, the cumulative score is +1, otherwise +0. The higher the cumulative score, the better the method.

[0144] Table 1 Average Rewards and Scores of the SORLPER Algorithm, SORLPER-TD Algorithm, and RLPER Algorithm

[0145]

[0146] From Figure 6 it can be seen that as the number of Episodes increases, the cumulative reward of the SORLPER algorithm increases the fastest, followed by the SORLPER-TD algorithm, while the performance of the RLPER algorithm is not good; from Figure 7From the score comparison results, it can be seen that the SORLPER algorithm has the highest score, the SORLPER-TD algorithm ranks second, and the RLPER algorithm performs the worst. Specifically, as can be seen from Table 1, in Environment 1, the average reward and average score of the SORLPER algorithm are 121.791 and 307.8 respectively, the average reward and average score of the SORLPER-TD algorithm are 112 and 282.2 respectively, and the average reward and average score of the RLPER algorithm are 39.887 and 19.8 respectively; in Environment 2, the average reward and average score of the SORLPER algorithm are 91.823 and 238.6 respectively, the average reward and average score of the SORLPER-TD algorithm are 85.540 and 220.5 respectively, and the average reward and average score of the RLPER algorithm are 27.418 and 14.4 respectively.

[0147] In summary, it can be seen that the SORLPER algorithm has achieved the best results, thus proving that the method proposed in this application, namely the parallel learning strategy of two learning systems and the designed sampling priority rule, can indeed improve the learning quality and learning efficiency of the agent.

[0148] It should be noted that although the SORLPER-TD algorithm does not introduce the BE error to adjust the sampling priority, it has also achieved good results. This is mainly due to the fact that the introduced SOM will adjust the sampling priority according to the familiarity with the state, thereby improving the learning efficiency and learning quality of the agent.

[0149] To verify that the priority sampling rule of this application, which adjusts the TD error and BE weight according to the familiarity with the state, can dynamically regulate the EET balance and thus improve the learning speed of the agent, a comparison was made with the priority sampling rule of the BETDQN algorithm; the following formula gives the gradient-based optimization strategy of the BETDQN algorithm:

[0150]

[0151] dκ2(t) = -dκ1(t) (20)

[0152] κ1(t + 1) = κ1(t) - dκ1(t) × l (21)

[0153] κ2(t + 1) = κ2(t) - dκ2(t) × l (22)

[0154] Among them, ζ ∈ [0, 1] represents the target ratio of κ1 / κ2, and l is the learning rate. The dynamic parameters κ1 and κ2 satisfy κ1, κ2 ≥ 0, κ1 + κ2 ≡ 1, and are updated once per episode.

[0155] Note: The above BETDQN algorithm adaptively adjusts the weights of TD error and BE error in the weight scoring rule according to the gradient-based optimization strategy. However, the adaptive change of this weight is changed with the increase of the number of training times (the corresponding change rules are given in Eqs. (19)-(22)), that is, the purpose of the inventor is: in the initial stage of the algorithm, it focuses on the BE weight, and in the later stage, it focuses on the TD error weight. However, how to define the early stage and the later stage in different tasks, and the ratio of the two weights in a complete training round bring more difficulties to the design of the algorithm (for example, in the 2D unknown environment navigation task of this embodiment, it is defined that the algorithm needs to be trained for 500 rounds. Then, the first question: define whether the early stage is 0-150 rounds or 0-250 rounds, or even more choices; the second question: the definition of the weight ratio κ1 and κ2, it is known that κ1+κ2≡1, and the evolution of the TD weight and the BE weight ratio κ1 / κ2 from the initial stage to the final stage: …, 0.001, …, 0.01, …, → …, 1.2, …, 2.2, … There are too many ratio evolution choices. These two problems bring more uncertainties to the design of the algorithm). Different from the BETDQN algorithm, this application designs a new weighted scoring mechanism, which does not need to define the early stage and the later stage of the algorithm and the proportion of sampling in the training stage according to the experience of experts, but considers the similarity between the state in the past experience and the current state of the agent to adaptively balance the TD error and the BE error. This mechanism of calculating the weighted score according to the familiarity with the state is more in line with the characteristics of human episodic memory. Next, the effectiveness and superiority of this mechanism will be verified.

[0156] Figure 8 The average reward comparison results of the SORLPER algorithm under two strategies are given. The average reward of the algorithm under each group of parameters is obtained from the results of 5 experiments, and different random number seeds are used. a, b, and c on the X-axis respectively represent three parameter selection mechanisms of the SORLPER algorithm. Among them, for the a mechanism in the first group and the second group: the κ1 and κ2 parameters remain unchanged; for the b mechanism in the first group: ζ = 1.2, and the values of κ1 and κ2 evolve according to the rules of Eqs. (19)-(22); for the c mechanism in the second group: ζ = 2.2, and the values of κ1 and κ2 evolve according to the rules of Eqs. (19)-(22).

[0157] From Figure 8It can be seen that when the parameters of the SORLPER algorithm are selected as κ1 = 0.01 and κ2 = 0.99, the average reward of the agent is the largest. In comparison, the effect of the SORLPER algorithm using the b and c mechanisms is reduced. In addition, when the parameters are selected as κ1 = 0.1, κ2 = 0.9 and κ1 = 0.9, κ2 = 0.1, the average reward of the algorithm also shows a decreasing trend. This is mainly because the priority sampling rule of this application dynamically adjusts the TD error and BE weighted proportion according to the familiarity with the state, thereby adjusting the EET balance. The additional dynamic weight adjustment mechanism of the BETDQN algorithm will cause the two weighted calculation rules to conflict, resulting in an imbalance in the sampling priority and poor learning effect of the agent. Figure 8 Although several sets of data have shown an increasing trend (which proves that this strategy in the BETDQN algorithm does help improve the learning ability of the algorithm), there is still a certain gap compared to the effect when the parameters are fixed at κ1=0.01 and κ2=0.99. In addition, it is worth noting that in these sets of comparison results, most of the time when the initial κ1 value is large, that is, the TD error weight accounts for a large proportion, the effect of the algorithm will have an increasing trend, while when the initial κ1 value is small, the effect will have a decreasing trend. In addition, it is easy to understand from the data in this column of group a that a large proportion of TD error will affect the overall performance of the algorithm. Therefore, the improvement effect of this strategy in the BETDQN algorithm still needs to be improved. In summary, from the above results and analysis, it can be concluded that the strategy designed in this application to dynamically control the TD error and the BE weighted proportion according to the degree of familiarity with the state and then control the EET balance has certain effectiveness and superiority.

[0158] In order to verify the superiority of the SORLPER algorithm, the SORLPER algorithm is compared with the classic algorithm PER and the most advanced algorithm BETDQN. The learning rate lr is set to 5×10 -4 The average rewards of these algorithms can be found in Figure 9 .

[0159] Figure 9 This is a line chart comparing the average rewards of the SORLPER algorithm, the PER algorithm, and the BETDQN algorithm. The average reward of each algorithm was obtained from 10 experiments using different random number seeds.

[0160] Depend on Figure 9 From the reward curves of each algorithm in the two environments, it can be seen that compared with the traditional PER algorithm that only relies on one sampling indicator and the BETDQN algorithm that uses multiple sampling indicators, the SORLPER algorithm performs best, thus proving the superiority of the algorithm.

[0161] To verify that the proposed SORLPER algorithm can be applied to the physical robot navigation decision-making task, physical simulation experiments and physical experiments were carried out using the AutolaborPro.1 robot platform. Figure 10 Describes the general architecture of the robot navigation decision-making process.

[0162] Figure 10 It is the architecture of the actual robot navigation decision-making process. AP1 obtains its own position and obstacle information by subscribing to the topics of " / odom" and " / scan", realizes the coordinate system conversion through " / tf", and executes the speed command by subscribing to the topic of " / cmd_vel". After obtaining the input state, AP1 realizes action selection through the proposed SORLDN2 algorithm. During the action execution process, it is set that AP1 starts to move forward when facing the target heading angle, that is, the action selection direction. The linear velocity of AP1 is set to 0.2 m / s, and the angular velocity is set to 0.2 rad / s.

[0163] The experiment is mainly divided into two parts. The first part: Test the effectiveness of the proposed SORLPER algorithm in the physical simulation environment. The second part: Test the effectiveness of the proposed SORLPER algorithm in the real environment. Two environments were built in the robot physical simulation software Gazebo, as Figure 11 shown. Five experiments were carried out in these two environments respectively, and the specific experimental results are shown in Table 2 and Table 3.

[0164] Figure 11 The side length of each square in it is one unit length. The starting point of the robot is set as (0,0), and the navigation target of the robot is the area within 0.5 unit lengths near the coordinate point (4,4). When the distance between the center coordinate of the robot and the target point is less than 0.5, it means that the navigation task is successfully completed. The safety distance between the center point of the robot and the obstacle is set to 0.5 unit lengths. When the safety distance is less than this value, it means that the task fails.

[0165] Table 2 Number of steps the robot moves under the SORLPER algorithm

[0166]

[0167] Table 3 Analysis indicators of experimental results

[0168]

[0169]

[0170] Table 2 shows the specific number of steps for each navigation of the robot in two environments. Table 3 analyzes these data using different metrics, including the maximum number of steps, minimum number of steps, average number of steps, success rate, and number of collisions. From the data in Table 2 and Table 3, it can be clearly seen that AP1 can successfully complete the navigation task in both Environment 1 and Environment 2, thus proving the effectiveness and strong generalization ability of the trained model.

[0171] The algorithm was transplanted into a physical robot for testing, and the real environment is as Figure 12 shown. To facilitate observing the experimental effect, Figure 13 a set of robot navigation trajectory diagrams is given.

[0172] Figure 12 is a simple experimental environment. The size of the experimental environment is approximately 6.8m × 6.4m. The starting point of the robot is set at (0,0), and the navigation target of the robot is an area within 0.5 meters near the coordinate point (4,4).

[0173] Figure 13 is the robot navigation trajectory diagram. It shows six stages of a complete navigation process: (a) the first stage; (b) the second stage; (c) the third stage; (d) the fourth stage; (e) the fifth stage; (f) the sixth stage. Each stage diagram includes the position of AP1 in the real environment and the moving trajectory and position of AP1 displayed in Rviz.

[0174] From Figure 13 the robot navigation path shown, it can be seen that AP1 successfully reached the target area without hitting any obstacles. The results show that AP1 using the SORLDN2 algorithm has excellent performance in the navigation task.

[0175] To avoid contingency, multiple experiments were conducted in the real environment, and Table 4 shows the results of 10 groups of experiments.

[0176] Table 4 Moving Steps of the SORLPER Algorithm

[0177] Algorithm / Number of Experiments 1 2 3 4 5 6 7 8 9 10 SORLPER 48 45 43 46 44 43 48 49 44 40

[0178] In summary, in this embodiment, through parameter analysis experiments, sampling strategy comparison experiments, ablation experiments, and comparative experiments, the effectiveness and superiority of the proposed method are verified, and the practicability of the proposed method is verified through physical environment experiments. Therefore, the strategy proposed in this technology is a feasible method.

[0179] For those skilled in the art, it is obvious that the present application is not limited to the details of the above-mentioned exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be construed as limiting the claims involved. In addition, it is obvious that the term "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the apparatus claims can also be implemented by one unit or device through software or hardware. First, second, etc. are used to denote names and do not denote any particular order.

Claims

1. A self-organizing reinforcement learning method with prioritized experience replay, characterized in that, It includes the following steps: S1. Combine a deep neural network (DNN) and a self-organizing learning map (SOM), and simulate the learning mechanisms of the neocortex and the hippocampus in the brain for reinforcement learning. Among them, the deep neural network (DNN) is used to simulate the characteristics of slow learning and distributed representation of the neocortex, and the self-organizing learning map (SOM) is used to simulate the characteristics of fast learning and pattern separation of the hippocampus; S2. During the reinforcement learning process, adopt a priority sampling strategy considering the temporal difference (TD) error, Bellman (BE) error, and empirical familiarity factor index to dynamically regulate the balance between exploration and exploitation (EET), that is, obtain a self-organizing reinforcement learning method with priority experience replay, denoted as SORLPER.

2. The self-organizing reinforcement learning method with prioritized experience replay according to claim 1, characterized in that: In step S1, the deep neural network (DNN) and the self-organizing learning map (SOM) work in parallel. Among them, the deep neural network (DNN) is responsible for generalizing the state space, while the self-organizing learning map (SOM) focuses on storing the experiences that are difficult for the deep neural network (DNN) to evaluate, and has a pattern separation effect on the experiences.

3. The self-organizing reinforcement learning method with prioritized experience replay according to claim 2, characterized in that: In step S1, the combination of the deep neural network (DNN) and the self-organizing learning map (SOM) includes the following three aspects: Step S11. Combination of action selection: SORLPER adopts the ε-greedy strategy and screens actions according to the weighted Q-values of the deep neural network (DNN) and the self-organizing learning map (SOM); Step S12. Combination of experience sampling: SORLPER preferentially samples according to the weighted average score of the temporal difference (TD) error and the Bellman (BE) error, and introduces importance sampling weights for bias correction; Step S13. Combination of network update: The online network update of SORLPER depends on the objective function jointly obtained by the deep neural network (DNN) and the self-organizing learning map (SOM) in the target network. The update of the neuron weights and the neighborhood function in the self-organizing learning map (SOM) also depends on the temporal difference (TD) error generated by the deep neural network (DNN). This dynamic update process enables SORLPER to preferentially consider and retain the state experiences with inaccurate predictions in part of the deep neural network (DNN).

4. A self-organizing reinforcement learning method with prioritized experience replay according to claim 3, characterized in that: In step S12, the importance sampling weight is denoted as ω(i), and its calculation formula is: where M represents the size of the replay buffer, p(i) is the priority of the i-th transition, and β is an adjustable hyperparameter.

5. The self-organizing reinforcement learning method with prioritized experience replay according to claim 3, characterized in that: Step S2 includes the following steps: S21. Interact the decisions of the deep neural network (DNN) and the self-organizing learning map (SOM) with the environment, and store the interactive experiences in the experience replay buffer; S22. Obtain the corresponding temporal difference (TD) error and Bellman (BE) error according to the familiarity of SORLPER with each transition sample; S23. Calculate the score by weighted summation of the temporal difference (TD) error and the Bellman (BE) error, and then obtain the sampling probability of the sample through sampling priority processing, so as to realize the adjustment of exploration and exploitation (EET), and accelerate the convergence speed and learning quality of the reinforcement learning process.

6. A self-organizing reinforcement learning method with prioritized experience replay according to claim 5, characterized in that: In step S2: The experience familiarity factor determines the contribution of the deep neural network (DNN) and the self-organizing learning map (SOM) to the Q value, denoted as η, and its specific calculation formula is as follows: Among them, τ η is a hyperparameter. It can be seen that if the value of η is larger, the contribution of the self-organizing learning map (SOM) is greater; if the value of η is smaller, the contribution of the deep neural network (DNN) is greater. The activated neuron u of the self-organizing learning map (SOM) t is obtained according to the state input s t and the Euclidean distance between the state weight w u , that is, u t = argmin u ||w u - s t || 2 .

Images