Robot behavior decision-making method and device simulating brain learning and memory mechanism

Abstract

The application belongs to the technical field of robot intelligent control, and relates to a robot behavior decision method and equipment simulating brain learning and memory mechanism. The method combines a developmental network DN2, reinforcement learning and a self-organizing map SOM, simulates functions of a cerebellum learning system, a basal ganglia and a hippocampus learning system, and obtains a self-organizing reinforcement learning model based on the developmental network DN2. The equipment comprises one or more processors and a computer readable medium storing one or more computer readable instructions, and is used for implementing the above robot behavior decision method. The application properly utilizes functions of the cerebellum learning system, the basal ganglia and the hippocampus learning system. The fusion algorithm combines reinforcement learning and supervised learning methods, enhances robustness of the self-organizing reinforcement learning model based on the developmental network DN2, improves the ability of the algorithm to obtain an effective obstacle avoidance strategy, and can help an intelligent agent to quickly learn an efficient solution.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer, especially relates to the field of robot intelligent control technology, and more particularly relates to a robot behavior decision-making method and device simulating brain learning and memory mechanism. BACKGROUND

[0002] With the development of science and technology, constructing a reliable brain-like model to be applied in the field of intelligent robots so as to enable the robots to efficiently complete complex tasks has become an important research direction in the field of artificial intelligence and neuroscience.

[0003] In the exploration of brain cognitive functions, the key roles of the basal ganglia and cerebellum in motor control and decision-making have been confirmed by a number of neuroscience studies. Researchers have carried out a lot of work on robot learning by simulating basal ganglia and cerebellum models. In 1999, Doya et al. proposed a neural network combining different learning modules in the cerebellum, basal ganglia and cerebral cortex, respectively giving the cerebellum the function of supervised learning, the basal ganglia the function of reinforcement learning, and the cerebral cortex the function of unsupervised learning. Since then, related research has been continuously advancing, such as the development of a guidance-oriented action-dependent heuristic dynamic programming learning mechanism by Ruan et al. in 2012, which uses an Actor-Critic model to simulate the functions and cooperation mechanisms of the two; and the biologically inspired hybrid model proposed by Zhang et al. in 2024 for musculoskeletal robots, which explores the exploration and exploitation balance in reinforcement learning.

[0004] In addition to motor control, researchers have also explored the roles of the cerebellum and basal ganglia in intelligent agent behavior decision-making. For example, some researchers have simulated the functions of the two by combining association-based learning and reward-based learning using the Actor-Critic framework of reinforcement learning to complete robot goal-oriented decision-making tasks; Wang et al. used a modulatory developmental network (MDN) and a radial basis function network-based Q-learning to simulate the functions of the cerebellum and basal ganglia, respectively, demonstrating the potential of neuromodulatory models in mobile robot behavior decision-making, but also presenting certain problems.

[0005] At the same time, the important roles of brain regions such as the hippocampus and prefrontal cortex in memory encoding, retrieval and decision-making have also become clear. Related research results such as the hippocampus-striatum system constructed by Yuan et al. and the biologically inspired hippocampal sequence memory system constructed by Casanueva-Morato et al. have provided important references for brain-inspired artificial intelligence research, and the new complementary learning system theory has revealed the benefits of the complementary properties between the hippocampus and neocortex for achieving complex behaviors.

[0006] As an unsupervised learning method, the pattern separation characteristic of self-organizing map (SOM) is widely used to improve the performance of the model. For example, Notsu et al. combined it with differential learning to generate state space for reinforcement learning, Su et al. combined it with deep reinforcement learning technology to solve the problem of detecting anomalies in multivariate time series data, and Modhej et al. used it to simulate the pattern separation function of the hippocampus to solve the problem of handwriting recognition in deep learning.

[0007] However, the current brain-like model still has many deficiencies. For example, the cerebellum simulation part of the research by Wang et al. is similar to the developmental network-1 (DN1), only uses the low-level representation part (i.e. 100,001 two-class neurons in the Y layer), and has low efficiency in complex learning tasks and lacks transverse connections between hidden layer neurons; the reinforcement learning part simulating the function of the basal ganglia requires the agent to search the entire state space to perform actions, which consumes a lot of computing resources in a complex environment, resulting in slow learning and decision-making speed of the robot; in addition, the model does not consider the hippocampal learning system. Therefore, it is urgent to establish a more complete and efficient brain-like model to meet the actual application requirements, which is an important background and starting point for the development of the present patent. SUMMARY

[0008] The purpose of the present application is to provide a robot behavior decision-making method and device simulating the learning and memory mechanism of the brain, which cleverly integrates the functions of the cerebellum, basal ganglia and hippocampal learning system to provide an efficient decision-making model for the agent, helping the agent to quickly learn and obtain an optimized decision-making scheme in a complex environment, significantly improving the accuracy and efficiency of the agent's behavior decision-making, and achieving significant progress in resource utilization, convergence speed and decision-making quality compared with the prior art, having a broad application prospect and market value.

[0009] Based on one of the above purposes, the present application provides a robot behavior decision-making method simulating the learning and memory mechanism of the brain, which combines developmental network DN2, reinforcement learning and self-organizing map SOM to simulate the functions of the cerebellar learning system, basal ganglia and hippocampal learning system, and obtains a self-organizing reinforcement learning model based on developmental network DN2 (referred to as "SORLDN2");

[0010] In most methods simulating hippocampus function, hippocampus learning system cannot be built alone, and often needs to be combined with another method to realize its function. Based on the self-organizing reinforcement learning model of developmental network DN2, the present application first uses developmental network DN2 as cerebellar learning system to realize the supervised learning function of cerebellum, then combines self-organized map and reinforcement learning to describe the basal ganglia-hippocampus learning system with SORL (Self-Organized Reinforcement Learning) model, denoted as SORL, and then uses the time difference error from developmental network DN2 to train SORL, so that the self-organizing reinforcement learning model based on developmental network DN2 can store the states that developmental network DN2 cannot accurately evaluate, and then balance the decision weight between cerebellar learning system and basal ganglia-hippocampus learning system by simulating the familiarity of brain to state input, and then cooperate in evaluating action value, so that the agent can fully utilize the respective advantages of cerebellum and basal ganglia-hippocampus learning system to select actions.

[0011] Further, Zhu et al. obtained good results in navigation decision tasks using the DN1 imitating cerebellum, and then Zheng and Wu proposed the upgraded developmental network DN2. The developmental network DN2 is provided with three regions, including a perception input layer X from an external environment, a movement layer Z acting on the external environment, and a hidden layer Y organizing the interaction between the perception input layer X and the movement layer Z;

[0012] wherein the input x(t) from the perception input layer X, z(t) from the movement layer Z, and the response value y(t) of the hidden layer Y are respectively represented as vectors, the adaptive part N=(V, G), wherein V is a matrix composed of synaptic weights, and G is a vector composed of neuron activation ages, N y and N z respectively represent the adaptive part of Y neurons in the hidden layer Y and Z neurons in the movement layer Z.

[0013] The main workflow of the above developmental network DN2 can be described as follows:

[0014] (1) Initialization. At time t=0, input the environment perception input x and the supervision signal z, and the response value y(0)=0. Set the synaptic weights V y of all Y neurons in the hidden layer Y and the synaptic weights V z of all Z neurons in the movement layer Z to random values. Because no neuron is activated, their neuron ages are equal to zero.

[0015] (2) Response value and adaptive part calculation N. At time t=1, …, repeat the following two steps:

[0016] (a) Calculate the response value of all Y neurons, using the region function f y Update adaptive part N y As follows:

[0017] (y(t),N′ y )=f y (p y ,N y ) (1)

[0018] Where p y represents the tuple of [x(t-1), y(t-1), z(t-1)], which is the input vector of the update response value y(t).

[0019] (b) Guide the robot to learn. Where the motion layer Z has two working modes, namely the guidance mode and the response mode. The guidance mode is to teach the robot to learn, and the response mode is to calculate the response value of Z neurons, and use the region function f z Update adaptive part N z As follows:

[0020] (z(t),N′ z )=f z (p z ,N z ) (2)

[0021] Where p z =y(t-1) is the input vector of the update response value z(t).

[0022] The calculation process of the region function is as follows:

[0023] By calculating the pre-response value r' i of each neuron, the function f y in equation (1) and the function f z in equation (2) are determined. The process is described as follows:

[0024] (1) Select the winning neurons by Top-k competition mechanism. Only the top k larger neurons in the response value will be selected and activated, which simulates the lateral inhibition of neurons. Specifically, the pre-response value r' i of neuron i in the hidden layer Y is:

[0025] r' i =r' b,i +r' t,i +r' l,i (3)

[0026] r' b,i =x(t-1) / ||x(t-1)||·v b,i / ||vb,i || (4)

[0027]

[0028] r' l,i = y(t-1) / ||y(t-1)||·v l,i / ||v l,i || (6)

[0029] In the formula, r' b,i , and r' l,i represents the pre-response values ​​for bottom-up connections, top-down connections, and lateral connections of a neuron, respectively, while v b,i , and v l,i Let represent the weight vectors of these three connections. If neuron i is one of the larger Top-k neurons, then the response value of neuron i is set to y. i =1, activation age n i Add 1: n i ←n i +1. If not, then y i =0, neuron age remains unchanged. The calculation method for neuronal response in motor layer Z is similar to the description above.

[0030] (2) Hebbian rules update synaptic weights. For the winning neuron i, its synaptic weights are updated according to the Hebbian learning rules as follows:

[0031] v b,i ←β b,1 *v b,i +β b,2 *y b,i *x(t-1)(7)

[0032]

[0033] v l,i ←β l,1 *v l,i +β l,2 *y l,i *y(t-1)(9)

[0034] Among them, the activation age n i The determined β1 and β2 are the retention rate and learning rate of the neuron, respectively, and β1 + β2 ≡ 1, where

[0035] β1=(n i -1) / n i β2=1 / n i (10)

[0036] and β b1 , β b2 , β t1 , β t2 , β l1 , β l2 respectively represent the retention rate and learning rate of the neuron from bottom to top, from top to bottom and lateral connection.

[0037] (3) Splitting. If the pre-response value of the first winning neuron is lower than the threshold value ρ(t) corresponding to the Y neuron type, the neuron in the initial stage is transferred to the active stage, and then activated, and ρ(t) is defined as

[0038] ρ(t) = μ(t)(θ - ∈) (11)

[0039] where μ(t) is a function for adjusting the increasing speed of the number of neurons, θ is a predefined constant, θ is 1 when the Y neuron type is 100, θ is 2 when the Y neuron type is 011, and ∈ is a constant.

[0040] (4) Synapse maintenance. Keep the stable connection between the deviation of the weight and the input less than the threshold value, and prune other connections.

[0041] Further, the self-organizing reinforcement learning model based on the developmental network DN2 uses two Y neurons of the developmental network DN2 to assist the robot to complete the navigation decision task, wherein the two Y neurons include a lower-level Y neuron and a higher-level Y neuron, the lower-level Y neuron is responsible for identifying different local features in the perception input layer X, and the higher-level Y neuron is connected to the lower-level Y neuron instead of being directly connected to the perception input layer X.

[0042] Further, the basal ganglia-hippocampus learning system uses the timing difference error generated by the cerebellar learning system to update its state representation, realizes the rapid learning of the agent, and the timing difference error is a real-time adjustment factor of the learning rate and the standard deviation of the neighborhood function in the basal ganglia-hippocampus learning system.

[0043] Reinforcement learning belongs to the category of semi-supervised learning technology. When the agent performs an action, it will obtain the corresponding reward or punishment. The agent obtains the optimal solution of the task by maximizing the immediate reward of each state and evaluating the behavior that may bring greater rewards in the future.

[0044] Further, in the basal ganglia-hippocampus learning system, individual memories are allowed to be stored in a unique, pattern-separated manner, and a simple error-driven learning mechanism is used to facilitate the rapid updating of the system.

[0045] Initially, the state perception inputs are provided to the hippocampus, which will be stimulated in a pattern separation manner, and the neurons arranged in the grid are used to represent the excitement level. According to the state input s t The similarity between the weight w u and the activation neuron u t is determined, and the calculation is as follows:

[0046] u t = argmin u ||w u -s t || 2 (12)

[0047] In the formula, w u represents the weight between the neuron and the input state s t and the activation neuron u t . For each neuron in the grid, there is a corresponding action value function Q SOM (u,a) to realize learning and memory, and the update mode of the action value Q SOM (u,a) is set as:

[0048] Q SOM (u t ,a t )←Q SOM (u t ,a t )+αη t [y t+1 -Q SOM (u t ,a t )](13)

[0049] Wherein, α is the learning rate. The weighting parameter η ensures that the action value will only be updated greatly when the closest neuron u is similar to the state value, which is very consistent with the human learning method, that is, quickly learning similar things and slowly adapting to unfamiliar things. The setting rule of η is as follows:

[0050]

[0051] Wherein, the parameter τ η is used to scale the Euclidean distance between w u and s t . In addition, considering the cooperation mechanism of the cerebellum and the basal ganglia-hippocampus learning system, the calculation of the state-action value adopts the following weight average rule:

[0052] Q(s,a)=ηQ SOM (u t ,a t )+(1-η)DN2(st a) (15)

[0053] It can be inferred that the smaller the value of η, the greater the contribution of DN2, i.e. u is not familiar with s, otherwise the greater the contribution of SORL. The above weight average rule (15) guarantees a learning mode in which the early state-action mainly depends on the development network DN2, and after learning, SORL has a greater contribution in the later period. The target value y set in formula (13) is:

[0054] y = r + γmax a' Q(s,a) (16)

[0055] where r is the reward and γ is the discount factor. For learning in SORL, the target value y and the temporal difference error generated by DN2 are used to affect the update of the model. The weight of neuron i is updated as follows:

[0056]

[0057] where λ is the learning rate. The neighborhood function is scaled by the exponentially increasing parameter δ:

[0058] δ = exp(|y - DN2(s,a)| / τ δ )-1 (18)

[0059] Here, the neighborhood function is represented by a Gaussian function

[0060]

[0061] where the parameter τ δ is used to scale the temporal difference error, u t and u i represent the mapping positions of the winning neuron and the output neuron i, respectively, and the neighborhood range of the neuron is composed of parameters σ c , δ and σ.

[0062] Finally, the action of the robot is selected according to the weighted average of the values of SORL and development network DN2.

[0063] The above method combines the development network DN2, reinforcement learning and self-organizing map SOM, and simulates the functions of the cerebellum, basal ganglia and hippocampus learning system.

[0064] Further, the basal ganglia is responsible for implementing the reinforcement learning function, and the hippocampus is responsible for encoding the knowledge learned by the agent.

[0065] Further, the self-organizing reinforcement learning model based on the development network DN2 further comprises:

[0066] environment for providing state and reward information;

[0067] sensory cortex responsible for representing input states;

[0068] thalamus responsible for action selection;

[0069] motor cortex responsible for action output.

[0070] According to another aspect of the present application, there is also provided a computer readable medium having stored thereon computer readable instructions executable by a processor to cause the processor to implement the above-described robot behavior decision method.

[0071] According to another aspect of the present application, there is also provided a robot behavior decision device simulating the learning and memory mechanism of the brain, the robot behavior decision device comprising:

[0072] one or more processors; and a computer readable medium for storing one or more computer readable instructions that, when executed by the one or more processors, cause the one or more processors to implement the above-described robot behavior decision method.

[0073] The working principle of the present application is that the present application proposes a self-organizing reinforcement learning model SORLDN2 based on a developmental network DN2, uses the developmental network DN2 as a cerebellar learning system, and describes the basal ganglia-hippocampus learning system through self-organizing reinforcement learning. In particular, the cerebellar learning system provides exploratory guidance for the basal ganglia-hippocampus learning system, and alleviates the problem of slow convergence of reinforcement learning. The basal ganglia-hippocampus learning system updates its state representation using the temporal difference error generated by the cerebellar learning system. Specifically, the temporal difference error is a real-time adjustment factor for the learning rate and the standard deviation of the neighborhood function in the basal ganglia-hippocampus learning system. This dynamic adjustment process enables the basal ganglia-hippocampus learning system to prioritize and retain state memories that are less accurately predicted by the cerebellar learning system, and subsequently use these memories to enhance overall decision-making ability and learning.

[0074] In addition, for the hippocampus, since experience is stored in a separate mode, the table method representing the action value function is more consistent with the hippocampus, so that the model can use a higher learning rate. However, as the state or action space increases, the table method requires more experience to fully explore the action value of each state, and therefore requires more computing resources to store these values. In this regard, the present application utilizes the advantages of self-organizing maps SOM to help the system reduce the complexity of the state space, so that reinforcement learning can be more effectively run under limited computing resources. SORL can generalize similar states, grouping similar states together, rather than treating each state as independent, making the reinforcement learning process more efficient by reducing the number of updates required to cover the state space.

[0075] Compared with the prior art, the present application has the following beneficial effects:

[0076] (1) Supervised learning of cerebellum using developmental network DN2 can better learn complex tasks.

[0077] (2) Combining self-organizing maps and reinforcement learning can achieve fast learning of the agent.

[0078] (3) Using the time difference error from the developmental network DN2 to train the self-organizing reinforcement learning SORL, so that the self-organizing reinforcement learning model based on the developmental network DN2 can store states that the developmental network DN2 cannot accurately evaluate. This innovative combination of self-organizing reinforcement learning and developmental network DN2 highlights how the cerebellar system's time difference signals to the hippocampus- basal ganglia system, telling the hippocampus when and what memories should be stored.

[0079] (4) Simulating the brain's familiarity with state input to balance the decision weights between the cerebellar learning system and the basal ganglia-hippocampus learning system. All systems collaborate in evaluating action values, enabling the agent to take advantage of the respective strengths of the cerebellar and basal ganglia-hippocampus learning systems to select actions.

[0080] As can be seen from the above, the present application properly utilizes the functions of the cerebellar learning system, the basal ganglia and the hippocampus learning system. This fusion algorithm combines reinforcement learning and supervised learning methods, enhancing the robustness of the self-organizing reinforcement learning model based on the developmental network DN2 and improving the algorithm's ability to obtain effective obstacle avoidance strategies. It can help the agent quickly learn an efficient solution, thereby improving the agent's behavior decision-making ability. BRIEF DESCRIPTION OF DRAWINGS

[0081] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0082] Figure 1 The overall architecture of the self-organizing reinforcement learning model based on the developmental network DN2 proposed in the embodiments of the present application is shown, wherein, Figure 1 (a) is a brain region and a pathway, Figure 1 (b) is the structure of the self-organizing reinforcement learning model based on the developmental network DN2;

[0083] Figure 2 The learning process of the developmental network DN2 in the embodiments of the present application is shown;

[0084] Figure 3 The schematic diagram of the simulation environment in the embodiments of the present application is shown;

[0085] Figure 4 The navigation path of the agent in different success periods in the embodiments of the present application is shown;

[0086] Figure 5 The change diagram of the navigation decision step number of the agent in the embodiments of the present application is shown;

[0087] Figure 6 The comparison diagram of the rewards obtained by the SORL and the SORLDN2 in the navigation decision task process in the embodiments of the present application is shown;

[0088] Figure 7 The comparison diagram of the step numbers realized by the SORL, the DN2 and the SORLDN2 in the navigation decision task in the embodiments of the present application is shown;

[0089] Figure 8 The comparison diagram of the motion step numbers obtained by the agent using six algorithms in five experimental environments in the embodiments of the present application is shown;

[0090] Figure 9 The box plot of the motion step numbers obtained by the six algorithms in five environments in the embodiments of the present application is shown;

[0091] Figure 10 The process architecture of the actual navigation decision of the robot in the embodiments of the present application is shown;

[0092] Figure 11 The simple experimental environment in the embodiments of the present application is shown;

[0093] Figure 12 The process of the robot completing the navigation decision task in the embodiments of the present application is shown;

[0094] Figure 13 The dynamic environment-1 in the embodiments of the present application is shown;

[0095] Figure 14 The performance of the robot avoiding dynamic obstacles in the first dynamic experiment in the embodiments of the present application is shown;

[0096] Figure 15 This illustrates the robot's performance in avoiding dynamic obstacles during the second dynamic experiment in this embodiment of the invention;

[0097] Figure 16 The path diagram of the robot in a simple environment is shown in an embodiment of the present invention;

[0098] Figure 17 Experimental environment 2 in an embodiment of the present invention is shown;

[0099] Figure 18 The diagram illustrates the navigation path of a robot in a complex environment according to an embodiment of the present invention. Detailed Implementation

[0100] The following will be combined with the appendix Figures 1 to 18 The technical solutions of the present invention have been clearly and completely described. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] This application proposes a robot behavior decision-making method that simulates the brain's learning and memory mechanisms. This robot behavior decision-making method combines the developmental network DN2, reinforcement learning, and self-organizing map SOM to simulate the functions of the cerebellar learning system, basal ganglia, and hippocampal learning system, resulting in a self-organizing reinforcement learning model based on the developmental network DN2 (hereinafter referred to as "SORLDN2").

[0102] In most methods simulating hippocampal function, the hippocampal learning system cannot be built independently and often needs to be combined with another method to achieve its function. This application's self-organizing reinforcement learning model based on the developmental network DN2 first utilizes DN2 as the cerebellar learning system to achieve supervised learning function of the cerebellum. Then, combining self-organizing maps and reinforcement learning, a self-organized reinforcement learning (SORL) model is used to describe the basal ganglia-hippocampal learning system, denoted as SORL. The SORL is then trained using temporal difference errors from the developmental network DN2, enabling the DN2-based self-organizing reinforcement learning model to store states that are difficult for the developmental network DN2 to accurately assess. Furthermore, by simulating the brain's familiarity with state inputs, the decision weights between the cerebellar learning system and the basal ganglia-hippocampal learning system are balanced. Finally, they collaborate in evaluating action value, allowing the agent to utilize the respective strengths of the cerebellum and basal ganglia-hippocampal learning systems to select actions.

[0103] Figure 1The overall scheme of the self-organizing reinforcement learning model based on developmental network DN2 (SORLDN2) is shown in FIG. 1. Among them, Figure 1 (a) is the brain area and pathway, Figure 1 (b) is the structure of the self-organizing reinforcement learning model based on developmental network DN2. For the cerebellum, the DN2 algorithm is used to realize the characteristics of supervised learning, which can avoid the small reward caused by the decision-making behavior of the SORLDN2 model in the exploration process of the agent. The basal ganglia is responsible for realizing the function of reinforcement learning. The hippocampus encodes the knowledge learned by the agent. The SORL simulates the coordinated function of the basal ganglia and the hippocampus. The SORL not only learns knowledge from DN2 (memory A in (b)) quickly, Figure 1 but also learns knowledge from other experiences that DN2 exhibits limitations (memory B in (b)). In addition, the sensory cortex is responsible for representing the input state, the thalamus is responsible for action selection, and the motor cortex is responsible for action output. Figure 1

[0104] Specifically, Zhu et al. obtained good results in a navigation decision-making task using a DN1 that mimics the cerebellum, and then Zheng and Wu proposed an upgraded developmental network DN2. The developmental network DN2 is provided with three regions, including a perception input layer X from an external environment, a movement layer Z that acts on the external environment, and a hidden layer Y that organizes the interaction between the perception input layer X and the movement layer Z;

[0105] Among them, the input x(t) from the perception input layer X, z(t) from the movement layer Z, and the response value y(t) of the hidden layer Y are respectively represented as vectors, the adaptive part N = (V, G), wherein V is a matrix composed of synaptic weights, and G is a vector composed of neuron activation ages, N y and N z represent the adaptive part of the Y neurons in the hidden layer Y and the Z neurons in the movement layer Z, respectively.

[0106] Figure 2 The learning process of DN2 is shown in FIG. 2. The input layer X region is the sensory input of the agent. In the hidden layer Y region, there are two kinds of internal neurons equipped with different connection mechanisms, which adaptively learn features from different regions. The activated neurons are marked in red or green, and the unactivated neurons are represented in gray. The blue connection represents the top-down input from the movement layer Z region, the orange connection represents the lateral interaction within the hidden layer Y region, and the green connection corresponds to the bottom-up input from the input layer X region.

[0107] ​In this embodiment, the self-organizing reinforcement learning model based on the developmental network DN2 utilizes two types of Y neurons of the developmental network DN2 to assist the robot in completing the navigation decision task, wherein the two types of Y neurons include a lower-level Y neuron and a higher-level Y neuron, the lower-level Y neuron is responsible for identifying different local features in the perception input layer X, and the higher-level Y neuron is connected to the lower-level Y neuron, rather than being directly connected to the perception input layer X.

[0108] The following Algorithm 1 gives the pseudo code of the learning process of the developmental network DN2:

[0109]

[0110] More specifically, the basal ganglia-hippocampus learning system utilizes the temporal difference error generated by the cerebellum learning system to update its state representation, which enables the fast learning of the agent, and the temporal difference error is a real-time adjustment factor of the learning rate and the standard deviation of the neighborhood function in the basal ganglia-hippocampus learning system.

[0111] Reinforcement learning belongs to the category of semi-supervised learning techniques. When the agent performs an action, it will obtain the corresponding reward or punishment. The agent obtains the optimal solution of the task by maximizing the immediate reward of each state and evaluating the behavior that may bring greater rewards in the future. In the basal ganglia-hippocampus learning system, individual memories are stored in a unique, pattern-separated manner, and a simple error-driven learning mechanism is used to facilitate the rapid updating of the system.

[0112] The above method combines the developmental network DN2 (hereinafter referred to as “DN2”), reinforcement learning, and self-organizing map SOM (hereinafter referred to as “SOM”) to simulate the functions of the cerebellum, basal ganglia, and hippocampus learning systems. In order to clearly understand this method, Algorithm 2 describes the workflow of self-organizing reinforcement learning and DN2.

[0113]

[0114]

[0115] From the above description, it can be seen that the proposed algorithm has several obvious characteristics: first, the calculation of Q value is affected by DN2 and SORL, and their relative contribution is determined by the parameter η. This ensures that when the current state is consistent with the state stored in SORL, the prediction Q value of SORL is more important. This process is similar to retrieving episodic memory and making decisions based on the relevance of memory to the state. In addition, since SORL updates its memory storage using the time difference error generated by DN2, it can store the states that can be accurately evaluated by DN2. In theory, this design can enable SORL to store both the experience in DN2 and the knowledge outside DN2. In addition, SORL uses a higher learning rate, potentially improving the utilization of data.

[0116] Specifically, the basal ganglia is responsible for implementing reinforcement learning functions, and the hippocampus is responsible for encoding the knowledge learned by the agent. The self-organizing reinforcement learning model based on developmental network DN2 also includes an environment, a sensory cortex, a thalamus, and a motor cortex, wherein the environment is used to provide state and reward information, the sensory cortex is responsible for representing the input state, the thalamus is responsible for action selection, and the motor cortex is responsible for action output.

[0117] The working principle of the present application is that the self-organizing reinforcement learning model SORLDN2 based on developmental network DN2 is proposed, developmental network DN2 is used as a cerebellar learning system, and the basal ganglia-hippocampus learning system is described through self-organizing reinforcement learning. In particular, the cerebellar learning system provides exploratory guidance for the basal ganglia-hippocampus learning system, alleviating the problem of slow convergence of reinforcement learning. The basal ganglia-hippocampus learning system updates its state representation using the time difference error generated by the cerebellar learning system. Specifically, the time difference error is a real-time adjustment factor for the learning rate and the standard deviation of the neighborhood function in the basal ganglia-hippocampus learning system. This dynamic adjustment process enables the basal ganglia-hippocampus learning system to prioritize and retain state memories that are less accurately predicted by the cerebellar learning system, and subsequently use these memories to enhance overall decision-making ability and learning.

[0118] In addition, for the hippocampus, since the experience is stored in a separate mode, the table method representing the action value function is more consistent with the hippocampus, allowing the model to use a higher learning rate. However, as the state or action space increases, the table method requires more experience to fully explore the action value of each state, thus requiring more computational resources to store these values. To this end, the present application utilizes the advantages of self-organizing map SOM to help the system reduce the complexity of the state space, making reinforcement learning more effective with limited computational resources. SORL can generalize to similar states, grouping similar states together rather than treating each state as independent, making the reinforcement learning process more efficient by reducing the number of updates required to cover the state space.

[0119] To verify the effectiveness of the self-organizing reinforcement learning model based on the developmental network DN2 in robot behavior decision-making, simulation and physical experiments were conducted. The parameters set in the experiments were: discount factor γ = 0.99, and the neighborhood range parameter σ of the neuron. c =0.1, σ=0.1, α=0.9, λ=0.01. Let the input layer X be X. r ={cosθ g sinθ g cosθ o sinθ o ,d g / (d g +d o ),d o / (d g +d o )}, where θ g d g θ represents the angle and distance between the robot and the target, respectively. o d o These represent the angle and distance between the robot and the obstacle, respectively. The output layer consists of 36 action numbers a∈(1,36), and the robot's motion is defined as state. next = state + step * (cos(a*π / 18), sin(a*π / 18)), where step is the robot's step size. The robot needs to move from the starting point to the target point. When it reaches the target, it will receive a reward of 100; if the robot collides with an obstacle or wall, it will receive a reward of -50.

[0120] The specific reward r settings are as follows:

[0121]

[0122] in, and d represents the distance between the robot and the target and the obstacle at time t, respectively. arrive d safe and d arrive It is a set distance constant. When the distance Less than d arrive When the distance is... Less than d safe When d indicates that the task has failed warning It is the distance at which a collision warning is issued.

[0123] The reward for each movement of the robot is: r = r1 + r2 + r3 + r4.

[0124] 1. Simulation Experiment

[0125] Figure 3 The simulation environment is shown in FIG. 6. The blue dots represent the starting positions of the agents, the green pentagons represent the target positions, and the black squares represent obstacles. The size of the environment is set to 500x500 units, the movement step length of the robot is set to 10 units, the coordinates of the starting point and the target point are (20, 50) and (470, 450) respectively, and the reward evaluation parameters in the simulation are set as follows: d arrive = 10, d safe = 10, d warning = 25.

[0126] Figure 4 The navigation paths of the agent in different success periods are shown in FIG. 7. It can be intuitively seen that the path length of the robot gradually decreases as the number of interactions between the agent and the environment increases. In particular, Figure 4 (a)-(f) represent that the agent reaches the target with 100 steps, 87 steps, 85 steps, 81 steps, 80 steps and 74 steps of decision steps when completing the 1st, 10th, 20th, 30th, 40th and 50th task respectively.

[0127] In order to verify the integrity of the learning process of the robot, Figure 5 The navigation step number change diagram of 400 experiments is shown in FIG. 8. In the figure, the x-axis represents the number of experiments, and 400 experiments are performed. The y-axis represents the movement steps of the agent. The results are represented by green circle points when the agent successfully reaches the target point, and by red bifurcation points when the agent fails. The navigation decision step number of the agent finally remains at 74 steps. In Figure 5 In FIG. 7, the fewer steps mean the better decision-making ability of the agent. As mentioned before, the step number is gradually reduced. This is mainly due to the contribution of the basal ganglia, which adopts reinforcement learning and can continuously improve the learning ability through interaction with the environment.

[0128] 1.1 Ablation experiment

[0129] In order to verify the synergistic contribution of SORL and DN2, an ablation experiment is performed. The parameters of DN2 and SORL are the same as those of SORLDN2. The results of the three algorithms are shown in Figure 6 , Figure 7 .

[0130] Figure 6 The rewards obtained by SORL and SORLDN2 in the navigation decision-making task are shown in FIG. 10. SORL and SORLDN2 are run for 30 times, and each curve is smoothed with the average of 30 data. The solid line represents the average value, and the shaded area represents the standard deviation of the reward. The green curve is the SORLDN2 model proposed in the embodiment.

[0131] Figure 7The number of steps implemented by SORL, DN2 and SORLDN2 in the navigation decision task. The shaded column represents task failure, and the solid line column represents task success. The green column represents the SORLDN2 model proposed in this embodiment.

[0132] In Figure 6 , the reward curve of DN2 is not drawn because the learning of DN2 is irrelevant to the reward function, so only SORL and SORLDN2 involve reward comparison. Importantly, in order to reflect the effectiveness of combining SORL and DN2, the experimental results in five environments are given.

[0133] From Figure 6 the experimental results, it can be seen that the reward of SORLDN2 is significantly better than that of SORL. This is mainly due to the guidance of DN2. DN2 has prior knowledge of navigation task decision-making, which can provide preliminary decision guidance and help reinforcement learning to reduce unnecessary exploration paths. However, this knowledge is not perfect, so the exploration of SORL can help SORLDN2 further improve its performance. Combined Figure 7 with the experimental results of the above methods, it can be seen that SORLDN2 has the smallest number of decision steps in each environment, and thus the superiority of SORLDN2 can be derived. By combining reinforcement learning and supervised learning methods, the robustness of the SORLDN2 model is enhanced, and the ability of the algorithm to obtain effective obstacle avoidance strategies is improved. The SORLDN2 method optimizes the ability of the intelligent agent with DN2 to accumulate experience, and enables the behavior strategy learned by the intelligent agent to contribute to high-quality decisions in future tasks. Therefore, it can be concluded that the cooperative learning of SORL and DN2 can improve the performance of the intelligent agent in the navigation task.

[0134] 1.2 Comparative experiment

[0135] In order to verify the advantages of the proposed method, the experimental performance of the proposed method (SORLDN2), RLDN1, DN2, DN1, Deep Q-Network (DQN) and Double DQN (DDQN) is compared in five environments. The parameter settings of the six models are shown in Table 1.

[0136] Table 1 Parameter settings of each algorithm in the navigation decision task

[0137]

[0138] For fairness, the DN1 used in RLDN1 is ported to the algorithm in this embodiment for comparison. DN2 in Wu et al.'s work utilizes two types of Y-layer neurons (111, 100) to perform visual navigation decisions. Unlike the experimental environment of this embodiment, the second intermediate layer in this embodiment does not require state input information from the environment. Therefore, two types of Y-layer neurons (011, 100) are used. Furthermore, SORLDN2 is not suitable for comparison with DQN and DDQN because the developmental networks used contain a small training set and learn quickly, while DQN and DDQN require extensive exploration to train a good model. SORLDN2 performs well with minimal interaction with the environment. However, DQN and DDQN require extensive interaction to achieve superior performance. To address this issue, only test results are presented and analyzed in the comparative experiments. The test results of these algorithms are as follows: Figure 8 , Figure 9 As shown.

[0139] Figure 8 The number of steps taken by the agent using six different algorithms is presented as a line graph across five experimental environments. The x-axis represents the environment, and the y-axis represents the number of steps taken by the algorithm in each environment. Different colors are used to indicate the results of the six algorithms. Steps are plotted for successful experiments and not for failed experiments.

[0140] exist Figure 8 In the experiment, DN1 failed to complete the task in the second experimental environment and consistently made more decision-making actions in other environments, indicating poor learning ability. DN2 performed slightly better than DN1 because it added a new type of y-layer neuron for more refined learning. However, compared to SORLDN1 and SORLDN2, DN1 and DN2 performed worse. This is due to the lack of a large, perfect training set. The superior performance of both relies on extensive training with specialized experience, which is difficult to obtain. Thanks to SORL's contributions, SORLDN2 and SORLDN1 both improved the agent's performance based on DN2 and DN1.

[0141] Figure 9 The box plot shows the number of steps taken by six algorithms under five different environments, illustrating the overall trend. The x-axis represents the algorithm, and the y-axis represents the number of steps in a successful experiment. Each box contains all the steps taken by an algorithm in a successful experiment. It can be seen that DN2 consistently outperforms DN1 across all five environments. SORLDN1 and SORLDN2 outperform DN1 and DN2, respectively. DDQN shows better performance compared to DQN. Overall, SORLDN2 performs best.

[0142] 2 Physical Experiment

[0143] To verify the actual application effect of SOMRLDN2, a real environment experiment was conducted using Robot Operation Systems (ROS). The experimental environment was about 6.83 m x 3.84 m in size. Autolabor Pro.1 (AP1) was used as the experimental platform.

[0144] Figure 10 The process architecture for the robot's actual navigation decision was constructed. AP1 obtained its own position and obstacle information by subscribing to the “ / odom” and “ / scan” topics, implemented coordinate system conversion through “ / tf”, and executed velocity instructions by subscribing to the “ / cmd vel” topic. After obtaining the input state, AP1 implemented action selection through the proposed SORLDN2 algorithm. During action execution, AP1 was set to start moving when it was facing the target heading angle, i.e., the direction of action selection. The linear velocity of AP1 was set to 0.2 m / s, and the angular velocity was set to 0.2 rad / s.

[0145] 2.1 Simple Environment Experiment

[0146] Figure 11 The simple experiment environment was constructed. The environment shown contains three regular obstacles, and the starting position of the robot is set to (0, 0), and the target position is set to (4.0, 0.8).

[0147] Figure 12 The process of the robot completing the navigation decision task in the experiment is shown, which shows the navigation trajectory of the robot in the actual environment in four stages. Figure 12 (a) is the first stage, Figure 12 (b) is the second stage, Figure 12 (c) is the third stage, Figure 12 (d) is the fourth stage. As can be seen from the navigation path in the second and third stages, the robot AP1 avoids obstacles very well. As can be seen from the results of stage 4, AP1 successfully reached the target area and executed a very short path. The results show that AP1 using the SORLDN2 algorithm has excellent decision-making ability in the navigation decision task.

[0148] 2.2 Dynamic Environment Experiment

[0149] To verify the adaptability of the SORLDN2 algorithm, two experiments were conducted in a dynamic environment. In the first experiment, a person was used as a dynamic obstacle. Figure 13 Dynamic Environment-1, in which three obstacle areas were set within the range of AP1's intended walking path. During AP1's movement towards the target point, a person actively blocked the robot's path. Figure 14The effect diagram of the robot avoiding dynamic obstacles in the experiment. Figure 14 (a) is the first stage, Figure 14 (b) is the second stage, Figure 14 (c) is the third stage, Figure 14 (d) is the fourth stage.

[0150] Figure 14 The first row of the four figures shows the real-world navigation process, and the second row of the four figures shows the corresponding trajectory through the visualization tool Rviz in ROS. The four groups of figures show the navigation decision-making process in four stages. In order to observe the overall obstacle avoidance of AP1, after AP1 completes the task, the dynamic obstacle returns to the original position area, and the overall path of the robot and the radar scan information are combined to observe the obstacle avoidance effect as Figure 14 The four figures in the third row show only four groups of experimental process diagrams.

[0151] In the second dynamic experiment, a "puppet" AP1 controlled by a handle and a person as two dynamic obstacles in the experiment. The "puppet" robot is responsible for blocking the movement of AP1 in the middle of the road, and the person is responsible for blocking the road around the target point.

[0152] Figure 15 The test results are visualized, and the performance of the robot avoiding dynamic obstacles in the second dynamic experiment is shown. Figure 15 (a) is the first stage, Figure 15 (b) is the second stage, Figure 15 (c) is the third stage, Figure 15 (d) is the fourth stage. It can be seen that AP1 with SORLDN2 can autonomously navigate to the target without any collision.

[0153] 2.3 Comparative experiment

[0154] In the comparative experiment, tests were conducted in two unknown environments, with thirty experiments conducted respectively, and four indicators were collected to compare the performance of the six algorithms.

[0155] Figure 16 The path diagram of the robot in a simple environment, Figure 16 (a) is the DN1 algorithm, Figure 16 (b) is the DN2 algorithm, Figure 16 (c) is the DQN algorithm, Figure 16 (d) is the DDQN algorithm, Figure 16 (e) is the SORLDN1 algorithm, Figure 16 (f) is the SORLDN2 algorithm.

[0156] The numerical results are given in Table 2. The indexes include the number of times of successfully reaching the goal, the minimum number of steps in successful tasks, the maximum number of steps in successful tasks, the average number of steps in successful tasks, and the variance of the number of steps in successful tasks. It can be seen that DN2, SORLDN1 and SORLDN2 show better results, with the average number of steps being about 25-26. However, DN1, DQN and DDQN remain about 27-28 steps. It can be found that these algorithms achieve 100% success rate and show good ability in simple environment. Compared with DN1, DQN and DDQN, DN2, SORLDN1 and SORLDN2 have greater advantages in decision-making ability.

[0157] Table 2 Results of 30 experiments of 6 algorithms in simple environment

[0158]

[0159] The previous experiments verify the effectiveness and superiority of SORLDN2 in simple environment. Next, comparative experiments will be carried out in experimental environment 2 as shown in Figure 17

[0160] Figure 18 To navigate the path of the robot in complex environment, Figure 18 (a) DN1 algorithm, Figure 18 (b) DN2 algorithm, Figure 18 (c) DQN algorithm, Figure 18 (d) DDQN algorithm, Figure 18 (e) SORLDN1 algorithm, Figure 18 (f) SORLDN2 algorithm. The specific experimental results are shown in Table 3.

[0161] Table 3 Results of 30 experiments of 6 algorithms in complex environment

[0162]

[0163] From the experimental results, compared with the simple environment, the performance of these algorithms in the complex environment has declined. Due to the increase in the number of obstacles, the success rate of DN1, DQN and DDQN becomes low, and the average number of steps in successful tasks remains between 30-32, the average number of steps of DN2, SORLDN1 remains between 29-30, the average number of steps of SORLDN2 is the smallest, remains between 28-29, and the variance of the number of steps of SORLDN2 is also small. Therefore, it is proved again that SORLDN2 has better decision-making ability.

[0164] ​It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0165] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of this application.

[0166] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware.

Claims

1. A robot behavior decision-making method that simulates the brain's learning and memory mechanisms, characterized by: The robot behavior decision-making method combines developmental network DN2, reinforcement learning, and self-organizing map SOM to simulate the functions of the cerebellar learning system, basal ganglia, and hippocampal learning system, resulting in a self-organizing reinforcement learning model based on developmental network DN2. The self-organizing reinforcement learning model based on the developmental network DN2 first utilizes DN2 as a cerebellar learning system to achieve supervised learning functions of the cerebellum. Then, it combines self-organizing maps (SOM) and reinforcement learning to describe the basal ganglia-hippocampal learning system using the SORL model, denoted as SORL. The SORL is then trained using temporal difference errors from the developmental network DN2, enabling the self-organizing reinforcement learning model based on DN2 to store states that are difficult for the developmental network DN2 to accurately evaluate. Furthermore, by simulating the brain's familiarity with state inputs, the decision weights between the cerebellar learning system and the basal ganglia-hippocampal learning system are balanced. Finally, they collaborate in evaluating the value of actions, allowing the agent to utilize the respective advantages of the cerebellar learning system and the basal ganglia-hippocampal learning system to select actions. The basal ganglion-hippocampus learning system uses the temporal difference error generated by the cerebellum learning system to update its state representation, thereby enabling the agent to learn quickly. The temporal difference error is a real-time adjustment factor for the learning rate and the standard deviation of the neighborhood function in the basal ganglion-hippocampus learning system. The basal ganglion-hippocampus learning system allows individual memories to be stored in a unique, pattern-separated manner and promotes rapid system updates through a simple error-driven learning mechanism. Initially, state-sensory input is provided to the hippocampal learning system. This input stimulates the hippocampal learning system in a pattern-separated manner, with neurons arranged in a grid representing excitation levels; based on the state input... With weight Similarity judgment between neurons activates neurons The calculation is as follows: In the above formula, This indicates the neuron's input state. and activate neurons The weights between them; for each neuron in the grid, there is a corresponding action-value function. To achieve learning and memorization, set action values. The update method is as follows: in, It's the learning rate. y The target value, weighted parameters Ensure that action values ​​are only applied to the nearest neuron. u A large update will only be obtained when the value is similar to the state value. The settings rules are as follows: Among them, parameters Used for scaling and The Euclidean distance between them, taking into account the cooperative mechanism of the cerebellum and basal ganglia-hippocampus learning system, is calculated using the following weighted averaging rule: The aforementioned weighted averaging rule guarantees a learning mode in which early state-behavior primarily depends on the developmental network DN2, while SORL contributes more significantly in later stages after learning. Target value y The formula for setting it is: in, It's a reward. It is a discount factor; for learning in SORL, the target value is used. y The temporal difference error generated by the developmental network DN2 affects the update of the self-organizing reinforcement learning model based on the developmental network DN2; among which, neurons i The weights are updated according to the following rules: in, It is the learning rate, determined by the neighborhood function. Scaling exponentially increasing parameter for: Here, the neighborhood function Represented by a Gaussian function Among them, parameters Used to scale timing differential errors. and These represent the winning neuron and the output neuron, respectively. i The mapping location, the neighborhood range of the neuron is determined by the parameters. , and composition; Finally, the robot's actions are selected based on the weighted average of SORL and the developmental network DN2 value.

2. The robot behavior decision-making method according to claim 1, characterized in that: The developmental network DN2 has three regions, including a sensory input layer X from the external environment, a motor layer Z that acts on the external environment, and a hidden layer Y that organizes the interaction between the sensory input layer X and the motor layer Z. Here, the input x(t) from the perceptual input layer X, the z(t) from the motion layer Z, and the response value y(t) from the hidden layer Y are represented as vectors, respectively, and the adaptive part... ,in V It is a matrix composed of synaptic weights. G It is a vector composed of the activation ages of neurons. and These represent the adaptive portions of Y neurons in the hidden layer Y and Z neurons in the motor layer Z, respectively.

3. The robot behavior decision-making method according to claim 2, characterized in that: The self-organizing reinforcement learning model based on the developmental network DN2 utilizes two types of Y neurons in the developmental network DN2 to assist the robot in completing navigation decision-making tasks. The two types of Y neurons include lower-level Y neurons and higher-level Y neurons. The lower-level Y neurons are responsible for identifying different local features in the perceptual input layer X, while the higher-level Y neurons connect to the lower-level Y neurons, rather than directly connecting to the perceptual input layer X.

4. The robot behavior decision-making method according to claim 1, characterized in that: The basal ganglia are responsible for implementing reinforcement learning functions, while the hippocampal learning system is responsible for memorizing and encoding the knowledge learned by the intelligent agent.

5. The robot behavior decision-making method according to claim 1, characterized in that: Self-organizing reinforcement learning models based on the developmental network DN2 also include: The environment is used to provide status and reward information; The sensory cortex is responsible for representing input states; The thalamus is responsible for action selection; The motor cortex is responsible for outputting motor movements.

6. A robotic behavior decision-making device that simulates the brain's learning and memory mechanisms, characterized in that, The robot's behavior decision-making device includes: One or more processors; Computer-readable medium for storing one or more computer-readable instructions. When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the robot behavior decision method as described in any one of claims 1 to 5.

Images