Mobile robot behavior decision-making method inspired by functions of hippocampus and prefrontal cortex

Abstract

The invention provides a hippocampus and prefrontal cortex function inspired mobile robot behavior decision-making method, and relates to the technical field of mobile robot control, the current state and the target position of a mobile robot are obtained from the environment, and a model-based reinforcement learning module is constructed through the spatial cognitive function of a hippocampus; reducing the dimension of the current state into a two-dimensional discrete state, inputting the two-dimensional discrete state into a model-based reinforcement learning module, and outputting an action to a decision controller; a model-free reinforcement learning module is obtained based on the basal ganglion and hippocampus memory function, the current state is input into the model-free reinforcement learning module, an action is output to a decision controller, and the decision controller is constructed by simulating the function that the prefrontal cortex is switched between the model-based reinforcement learning module and the model-free reinforcement learning module; and the decision controller outputs an execution action to complete the behavior decision of the mobile robot, so that the mobile robot is ensured to make a more reliable decision, and the performance of the mobile robot in the aspect of optimal decision is improved.

Description

Technical Field

[0001] This invention relates to the field of mobile robot control technology, specifically to a mobile robot behavior decision-making method inspired by the functions of the hippocampus and prefrontal cortex. Background Technology

[0002] With the rapid development of technology, mobile robots have been widely used in various industries, and the application scenarios they face are becoming increasingly complex. Most tasks require mobile robots to operate without prior environmental knowledge, which poses a significant challenge to their autonomous decision-making capabilities. Therefore, how to design new control and decision-making models to endow mobile robots with a certain degree of intelligence, enabling them to make autonomous decisions to cope with various unexpected problems, has become an important research topic.

[0003] To imbue robots with intelligence, it is essential to first understand the intelligence of the brain. In recent years, with in-depth research in neuroscience, researchers have noticed that the mammalian brain, through its highly complex neural networks, is not only capable of efficiently processing external sensory information but also of making rational decisions based on intrinsic motivations and goals. Developing computational models to enable mobile robots to simulate the cognitive decision-making functions of living organisms has become a popular approach in the field of artificial intelligence research.

[0004] In the mammalian brain, the hippocampus is a crucial region involved in cognitive and decision-making processes. Khamassi et al. proposed an innovative bidirectional search model to explain the diverse offline reactivation phenomena of the hippocampus during reward-based learning. Simulation experiments showed that forward reactivation mainly occurs at decision points, while backward reactivation occurs at reward points. Furthermore, the model was able to generate imagined trajectories not experienced during task execution. These findings provide a new theoretical framework for understanding the role of the hippocampus in reinforcement learning.

[0005] Bakermans et al. proposed that the hippocampus constructs a state space through combinatorial primitives, forming combinatorial memories to support behavioral inference in new environments. Experiments revealed the crucial role of the reflexology in the formation and consolidation of combinatorial memories, providing a new framework for understanding hippocampal function and emphasizing the importance of combinatorial mechanisms in adaptive behavior. Huang et al. proposed a new spatial cognition model that simulates the collaborative mechanism between the hippocampus and striatum, simultaneously considering the influence of external environmental information and internal animal information on spatial encoding. Through a simulated water maze experiment, they verified the model's superior performance in navigation path length and demonstrated its adaptability and robustness. This research provides a new perspective for understanding the true mechanisms of animal spatial cognition, emphasizing the importance of the hippocampal-striatum network and the integration of internal and external information in spatial navigation. Wu Z, Wang D, Liu L. Integrated model of cerebellar supervised learning and basal ganglia's reinforcement learning for mobile robot behavioral decision-making[J] constructed a hybrid model combining cerebellar supervised learning and basal ganglia reinforcement learning functions, achieving good performance in navigation experiments. The aforementioned technologies, failing to organically integrate memory and spatial cognition mechanisms, result in models exhibiting low decision-making efficiency, untimely strategy updates, and unstable path selection when facing complex tasks or dynamic environments, hindering rapid adaptation and generalization. Furthermore, during task switching or environmental changes, the models often lack effective utilization of historical experience, reducing the reliability and robustness of decisions. Therefore, there is an urgent need to propose a brain-like decision-making model that integrates hippocampal memory and spatial cognition functions to enhance the autonomous decision-making ability and environmental adaptability of mobile robots in unknown environments. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a mobile robot behavior decision-making method inspired by the functions of the hippocampus and prefrontal cortex. The memory function and spatial cognition function of the hippocampus work synergistically, enabling the hippocampus to efficiently integrate memory and spatial information, optimize behavior decisions, and support flexible processing of complex cognitive tasks.

[0007] To achieve the aforementioned technical objectives, the adopted technical solution is: a mobile robot behavior decision-making method inspired by the functions of the hippocampus and prefrontal cortex. This method obtains the current state and target position of the mobile robot from the environment, constructs a model-based reinforcement learning module using the spatial cognitive function of the hippocampus, reduces the current state to a two-dimensional discrete state, inputs it into the model-based reinforcement learning module, and outputs action 'a'.MB The current state is input into the model-free reinforcement learning module, which then outputs action a, based on the memory functions of the basal ganglia and hippocampus. This input is then fed into the decision controller. MF The decision controller simulates the switching function of the prefrontal cortex between model-based reinforcement learning modules and model-free reinforcement learning modules. The decision controller outputs action 'a' to complete the behavior decision of the mobile robot.

[0008] The decision controller of the present invention makes decisions by calculating action a using action selection entropy. MB and action a MF The corresponding action selection entropy H MB (s) and H MF (s), through Choose the action to perform, where k is a constant.

[0009] The method for constructing a model-based reinforcement learning module described in this invention is as follows:

[0010] (1) Simulate the spatial cognitive function of the hippocampus to construct a cognitive map, which includes a state transition function and a reward function;

[0011] (2) The value iteration method is used for action planning and updating of the state-action value function;

[0012]

[0013] Where Q(s,a) represents the state-action value function, which represents the expected cumulative reward of the mobile robot for performing action a in the current state s; γ represents the discount factor, which is a constant between 0 and 1; and A represents the set of all executable actions of the mobile robot in the next state s′. Let T(s,a,s′) represent the maximum state-action value corresponding to the next state s′, where T(s,a,s′) is the state transition function and R(s,a,s′) is the reward function.

[0014] (3) Convert the state-action value into different action selection probabilities through the Softmax function, and send the action selection probabilities and corresponding actions to the decision controller;

[0015] (4) The mobile robot performs action a, reaches the next state s', and obtains a reward r from the environment, which is stored in the reward function to update the cognitive map;

[0016] (5) Repeat steps (2)-(4) until the preset number of training sessions is reached.

[0017] The state transition function described in this invention is

[0018]

[0019] In the formula, N(s,a) represents the cumulative number of times the mobile robot performs action a in state s, and N(s,a,s') represents the number of times the mobile robot reaches state s' after performing action a in state s. When the mobile robot performs N actions in state s... g After action a, if in N g If state s' is not reached in any of the steps, then the corresponding N(s,a,s') is reduced to half of its original value, and N(s,a) is adjusted to N(s,a)←N(s,a)-N(s,a,s') / 2.

[0020] The Softmax function described in this invention is

[0021]

[0022] in, It is the sum of the exponentially expressed Q values ​​of all possible actions, where τ represents the temperature parameter and is a positive real number; exp(Q(s,a) i ) / τ) represents the exponential Q value of the action.

[0023] The action selection entropy described in this invention is

[0024]

[0025] In the formula, H(s) represents the action selection entropy in state s, and P(a|s) represents the probability of choosing action a in state s.

[0026] The beneficial effects of this invention are as follows: Inspired by the functions of the hippocampus and prefrontal cortex, this invention proposes a hybrid reinforcement learning framework that combines model-based and model-free approaches. Using action selection entropy as an indicator, a decision controller is designed to achieve dynamic switching between model-based and model-free reinforcement learning modules. Simultaneously, it enables the mobile robot to adaptively switch between focusing on long-term and short-term goals, ensuring more reliable decisions and improving its performance in optimal decision-making. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the behavioral decision-making model of the present invention;

[0028] Figure 2 A diagram illustrating the content of a cognitive map.

[0029] Figure 3 A schematic diagram of hippocampal memory playback types;

[0030] Figure 4A schematic diagram showing the positional relationship between the mobile robot and obstacles and the target point;

[0031] Figure 5 Figure 1 shows the experimental environment for comparing the performance of different types of memory playback.

[0032] Figure 6 A graph showing the cumulative rewards for the mobile robot under different replay types;

[0033] Figure 7 This is a schematic diagram of the generalization experimental environment;

[0034] Figure 8 This is a schematic diagram of the static comparative test environment;

[0035] Figure 9 The cumulative reward graph of the mobile robot under four models in a static comparative test environment;

[0036] Figure 10 The optimal decision path for the mobile robot under four models in a static comparative test environment;

[0037] Figure 11 This is a schematic diagram of a dynamic environment.

[0038] Figure 12 Box plot of the number of times a mobile robot successfully reaches a target point in a dynamic environment;

[0039] Figure 13 This is a diagram of a static physical experimental environment.

[0040] Figure 14 The optimal path diagram for the robot over thirty rounds under four models;

[0041] Figure 15 Diagram of a dynamic physical experiment environment;

[0042] Figure 16 The first motion trajectory diagram of the robot interacting with dynamic obstacles;

[0043] Figure 17 This is the second motion trajectory diagram of the robot interacting with dynamic obstacles;

[0044] Figure 18 A diagram showing the relative positions of the robot and dynamic obstacles at different times;

[0045] Figure 19 The figures show the navigation experiment results of the robot in a dynamic physical environment under four different models. Detailed Implementation

[0046] The preferred embodiments of the invention are given below with reference to the accompanying drawings to illustrate the technical solution of the invention in detail. The corresponding drawings will be provided for detailed explanation of the invention. It should be particularly noted that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit or restrict the invention.

[0047] To overcome the shortcomings of existing mobile robot algorithms and improve the flexibility of mobile robots in autonomous learning and decision-making when facing complex and ever-changing environments, a mobile robot behavior decision-making method inspired by the brain regions related to cognitive decision-making was designed.

[0048] First, the current state and target location of the mobile robot are obtained from the environment. Second, a model-based reinforcement learning module is constructed using the spatial cognition function of the hippocampus. The current state is reduced to a two-dimensional discrete state and input into the model-based reinforcement learning module. The model-based reinforcement learning module outputs action 'a'. MB The current state is input into the model-free reinforcement learning module, which then outputs action a, based on the memory functions of the basal ganglia and hippocampus. This input is then fed into the decision controller. MF Finally, the decision controller simulates the function of the prefrontal cortex switching between model-based reinforcement learning modules and model-free reinforcement learning modules. Based on the prefrontal cortex switching mechanism, the decision controller outputs action 'a' to complete the mobile robot's behavior decision.

[0049] 1. Introduction to the principle of the solution

[0050] 1.1 Overall Description

[0051] Based on the above, and by comprehensively considering the hippocampal memory function and spatial cognition function, a design was made as follows: Figure 1 The behavioral decision-making model shown combines model-based and model-free approaches.

[0052] like Figure 1As shown, the current state information *s* obtained from the environment is input into both the model-based reinforcement learning module and the model-free reinforcement learning module. Before being input into the model-based reinforcement learning module, the state *s* is reduced in dimensionality using the GSOM (Growing Self-Organizing Map) method, mapping it to a two-dimensional discrete state space. Then, both modules send their outputs based on the current input states to the decision controller. The decision controller integrates the decision information from both the model-based and model-free reinforcement learning modules, selects the optimal action plan, and guides the mobile robot to perform actions and interact with the environment. Both the model-based and model-free reinforcement learning modules learn through the interaction between the mobile robot and the environment during their development. The experience of the mobile robot interacting with the environment is stored as memory information, and a growing cognitive map is constructed online. Figure 1 The dashed line represents the learning process of the mobile robot in offline mode. In the hybrid model, the model-free reinforcement learning module uses the basal ganglia model constructed by Wu Z, Wang D, Liu L. Integrated model of cerebellal supervised learning and basal ganglia's reinforcement learning for mobile robot behavioral decision-making[J]. Cognitive Systems Research, 2024, 88:101302. Its offline learning process is to extract a batch of experiences from the experience pool to train the Critic network. This learning method requires breaking the temporal correlation between each experience. According to the different order of memory playback, the memory playback of the hippocampus can be divided into forward playback, backward playback, unordered playback, and priority playback. There is a strong temporal correlation between the memory information of forward playback and backward playback. The discussion of the four types of memory playback in this application will be carried out in the model-based reinforcement learning module.

[0053] 1.2 Description of Model-Based Reinforcement Learning Module

[0054] The online process of model-based reinforcement learning is divided into three stages: learning, reasoning, and decision-making. The online process refers to the actual interaction with the environment.

[0055] During the learning phase, the model-based reinforcement learning module gradually constructs and optimizes the mobile robot's internal cognitive map through continuous interaction with the environment. The content of the cognitive map is as follows: Figure 2 As shown.

[0056] The states in the cognitive map are a finite number of discrete states. These discrete states are obtained through the interaction between the mobile robot and the continuous environment using the GSOM method, and the number of states continuously increases as the interaction progresses. As shown in formula (1), Euclidean distance is used to measure the input state x1=(x 11 ,x 12 ,...,x 1n ()( Figure 1 The state S in the middle) and x 2=(x 21 ,x 22 ,...,x 2n The similarity between the input state and the best matching unit is denoted by x2, where x2 represents the weight between the input state and the best matching unit. When the distance d(x1,x2) between the input state and the best matching unit is greater than the threshold D, the similarity between them is considered to be low. This condition will trigger the new state generation mechanism, generating new discrete state neurons as the best matching unit for the current input, thereby ensuring that the cognitive map can accurately reflect the dynamic changes of the environment.

[0057]

[0058] like Figure 2 As shown, the cognitive map includes a state transition function T(s,a,s') and a reward function R(s,a,s'), both of which are continuously updated as the mobile robot interacts with the environment. The reward function R(s,a,s') represents the reward obtained by the mobile robot after performing action a in state s and reaching state s'. The reward function is the reward feedback obtained by the mobile robot after interacting with the environment and is used to store the obtained reward information. The state transition function T(s,a,s') represents the probability of reaching state s' after performing action a in state s. Its value is obtained through statistical calculation, as shown in formula (2).

[0059]

[0060] In the formula, N(s,a) represents the cumulative number of times the mobile robot performs action a in state s, and N(s,a,s') represents the number of times the mobile robot reaches state s' after performing action a in state s. Considering that the environment in which the mobile robot is located may change, when the mobile robot performs N actions in state s... g After action a, if in this N g If state s' is not reached in any of the steps, then the corresponding N(s,a,s') will be reduced to half of its original value. At the same time, N(s,a) will also be adjusted accordingly, i.e., N(s,a)←N(s,a)-N(s,a,s') / 2.

[0061] The inference phase of model-based reinforcement learning is also the state-action value update phase. In the inference phase, this section employs a value iteration method for action planning and updating the state-action value function. The update formula is shown below:

[0062]

[0063] In the formula, Q(s,a) represents the state-action value function, which represents the expected cumulative reward for the mobile robot to perform action a in state s. γ represents the discount factor, a constant between 0 and 1. A represents the set of all executable actions of the mobile robot in the current state. This represents the maximum state-action value corresponding to the next state s′.

[0064] During the decision-making phase, the mobile robot calculates the corresponding state-action value based on the current state input. Then, the state-action value is converted into the selection probability of different actions through the Softmax function shown in formula (4). The action selection probability and the corresponding action are sent to the decision controller of the hybrid model, and the decision controller determines the final action to be executed.

[0065]

[0066] In the formula, P(a i ) indicates the selection of action a i The probability, It is the sum of the exponentially expressed Q values ​​of all possible actions. τ represents the temperature parameter, which is a positive real number. When τ approaches 0, the Softmax function tends to select the action with the maximum Q value; conversely, the probability of selecting all actions tends to be uniformly distributed.

[0067] The model-based reinforcement learning module described in this section not only includes an online learning process but also introduces an offline learning process inspired by the hippocampal memory replay mechanism. The offline phase is defined as the time interval between two decisions made by the mobile robot. During this period, the mobile robot virtually interacts with the cognitive map, achieving local updates to the cognitive map and the state-action value function. Depending on the order of neuron reactivation, hippocampal memory replay can be categorized into several types. The types of hippocampal memory replay discussed in this patent are as follows: Figure 3 As shown.

[0068] Figure 3The red and blue circles represent the starting and target positions of the mobile robot, respectively, and the arrows indicate different directions of movement. The order of forward playback is consistent with the order experienced by the mobile robot during online operation, but it is not exactly the same as the trajectory experienced during online operation; rather, it is a combination of multiple trajectories experienced online. In forward playback, the gray circles represent state-related neurons that will be activated. Among these neurons, the starting state for the next playback is selected according to the Softmax function shown in formula (4) (the black circle in the figure). When the playback reaches the target position or the number of playbacks exceeds the threshold N, the playback continues. r When the current forward playback cycle ends, the next replay cycle is complete. Reverse playback follows the reverse order of the mobile robot's online activity, typically starting from the target position and proceeding towards the starting position. During reverse playback, gray circles represent the preceding states of neurons in the currently active state. The Softmax function is used to select one of these preceding states as the starting state for the next playback cycle. Replay continues until the starting position is reached or the number of playback cycles exceeds a threshold N. r When the number of replays reaches a threshold N, the current round of reverse replay ends. Unordered replays are independent of the order experienced during online execution, similar to model-free reinforcement learning's method of random sampling from an experience pool. In unordered replays, gray circles represent randomly selected state neurons in each replay, which will be reactivated. The process continues until the number of replays reaches a threshold N. r The current round of unordered replay ends at this time. When performing priority replay, the online experiences of the mobile robot need to be prioritized. The priority index is the absolute value of the reward prediction error δ(t)| as shown in formula (5). The yellow circles in the figure represent state neurons with higher priority; their associated preceding state neurons will also be reactivated and updated during priority replay. When the priority index is less than the set threshold P... r Or the number of replays exceeds the threshold N. r At that time, the current round of priority replay will end.

[0069] δ(t)=R t +γV(S′ t )-V(S t )# (5)

[0070] 1.3 Decision Controller Design

[0071] In the decision-making phase of the mobile robot, the model-free reinforcement learning module and the model-based reinforcement learning module process the state input and generate action outputs. Their respective outputs are then fed into the decision controller, which selects the final action the mobile robot will execute. Considering that the cognitive map of the mobile robot in this patent model is built incrementally from scratch, the model-free reinforcement learning module and the model-based reinforcement learning module may have different decision uncertainties at different learning stages. Therefore, in the design of the decision controller, action selection entropy is used to reflect the uncertainty of the decisions made by different modules in the current state, thereby measuring the reliability of the two modules and enabling adaptive switching between them. The calculation of action selection entropy is shown in formula (6):

[0072]

[0073] In the formula, H(s) represents the action selection entropy in state s, and P(a|s) represents the probability of choosing action a in state s. The calculation of P(a|s) is carried out with reference to formula (4).

[0074] The action selection of the decision controller is shown in formula (7):

[0075]

[0076] In the formula, a MB and a MF These represent the output actions of the model-based reinforcement learning model and the model-free reinforcement learning module, respectively. `k` is a constant used to control the balance between model-free and model-based methods. In the early stages of mobile robot interaction with the environment, the cognitive map is not yet complete, and the predictive ability of the model-based reinforcement learning module is weak, resulting in high decision uncertainty and a large action selection entropy. In this situation, the decisions made by the model-free reinforcement learning module, which directly relies on environmental interaction for learning, are more reliable and will be preferred. As the cognitive map improves, model-based reinforcement learning can more accurately simulate environmental characteristics and generate relatively stable optimal policies. At this point, its action selection entropy will decrease. When it decreases below the entropy value of model-free reinforcement learning, it indicates that its decisions are more stable and suitable for planning and decision-making. If the environment changes, the predictive uncertainty of the cognitive map increases, causing the entropy value of the model-based reinforcement learning to increase again. At this time, the system will use the model-free reinforcement learning decision-making scheme to ensure that the mobile robot always adopts the most appropriate decision strategy.

[0077] To more clearly describe the overall model, the workflow of the model is given in pseudocode below.

[0078]

[0079]

[0080] The key to this invention is the proposal of a hybrid reinforcement learning model inspired by the functions of the hippocampus and prefrontal cortex. This model enhances the mobile robot's ability to learn from its environment by dynamically switching between model-based and model-free modules, thereby improving the robot's performance in optimal decision-making. Experiments show that the algorithm has good convergence and generalization, enabling the robot to quickly adapt to unfamiliar environments and complete corresponding tasks efficiently.

[0081] 2. Simulation Experiment Results and Analysis

[0082] 2.1 Experimental Setup

[0083] This section will verify the performance of the proposed model through a simulation experiment designed for a mobile robot navigation task. The positional relationships between the mobile robot, obstacles, and target point are as follows: Figure 4 As shown. The model's input is... Where θ f and θ e d represents the angle between the target point and the nearest obstacle to the mobile robot and the horizontal direction of the mobile robot, respectively. f and d e These represent the target point and the distance to the robot from the nearest obstacle, respectively. The model has a total of 36 outputs, representing 36 different directions of movement for the robot.

[0084] In the experiment, the reward the mobile robot receives for each step it takes is r. t r t It consists of three sub-rewards, r1, r2, and r3. r1 represents the goal-oriented reward, which is based on the difference Δd between the distance Δd between the mobile robot's distance from the target point before and after performing the action. f The calculation is as follows:

[0085]

[0086] In the formula, d step This represents the step size of the mobile robot. After performing an action, the robot receives a positive reward if it moves closer to the target point, and a negative reward if it moves further away. If the distance between the robot and the target point remains unchanged after performing the action, a small negative reward is received.

[0087] r2 represents the negative reward received by the mobile robot when it collides with the boundary or obstacle of the environment. The calculation formula is as follows:

[0088]

[0089] In the formula, d safe This represents the safe distance between the mobile robot and an obstacle or boundary, when d e When the distance is less than the safe distance, it is considered that the mobile robot has collided with the obstacle.

[0090] r3 represents the reward the mobile robot receives after reaching the target point, as shown in formula (10):

[0091]

[0092] In the formula, d arrive This represents the distance to the target point. When the distance between the mobile robot and the target point is less than d... arrive When the time is reached, the mobile robot is considered to have successfully reached the target point.

[0093] The instant reward r that the mobile robot receives at each time step t As shown in formula (11):

[0094] r t =r1+r2+r3#(11)

[0095] 2.2 Performance Comparison Experiment of Different Types of Memory Playback

[0096] Depending on the order of neuronal activation, hippocampal memory replay can be categorized into several modes. These include forward replay (neurons are activated in the same order as during online activity), reverse replay (neurons are activated in the opposite order), unordered replay (neurons are activated without significant correlation to the order of online activity), and prioritized replay (neurons are activated with a priority related to the importance of the remembered information). This section will integrate these four memory replay mechanisms based on the behavioral decision-making model proposed in this patent and evaluate the impact of various replay strategies on the decision-making performance of the mobile robot through mobile robot navigation experiments.

[0097] The environment used in the experiment, such as Figure 5 As shown. The environment size is 200×200, containing 25 obstacles. In the experiment in this section, when the mobile robot collides with an obstacle or the environment boundary, it does not immediately end its movement for that round. Instead, after receiving a negative reward, it returns to its state before the collision and continues to interact with the environment until the mobile robot reaches the target point, at which point its movement for that round ends. In the experiment, the mobile robot's step size d... step Set to 8, the distance d to determine the arrival of the target point arrive and the safe distance d between the mobile robot and the obstacle safe All are set to 6.

[0098] In the experiment, a model without any replay mechanism was also introduced as a control. The five models were... Figure 5 Each model was trained for 500 rounds in the environments shown. To reduce the impact of random errors in the experiments, each model in this section underwent ten repeated experiments. The reward obtained by the mobile robot in each round was calculated, and the cumulative reward curve was plotted. Figure 6 middle.

[0099] Figure 6 This study demonstrates the reward accumulation of mobile robots under different memory playback mechanisms. Overall, the reward accumulation curve shows a trend of first decreasing and then increasing. This is because the models learn completely from scratch, without any prior knowledge. In the early stages of the experiment, all models needed to explore through extensive environmental interactions. During this process, the mobile robots did not stop interacting with the environment when they collided with obstacles or environmental boundaries, resulting in a large accumulation of negative rewards in the early stages. As training progressed and the cognitive map gradually improved, the mobile robots gradually learned better strategies, making more reasonable decisions during environmental interactions, reducing the number of collisions with obstacles, and thus achieving the accumulation of positive rewards.

[0100] from Figure 6 The results also show that the priority replay method performs best during training, with the fastest reward growth rate. Rewards begin to accumulate positively around the 50th training iteration, eventually reaching the highest total reward value. This is because priority replay selects memories with higher priority for replay based on the absolute value of the reward prediction error. During the interaction between the mobile robot and its environment, collisions with obstacles or reaching the endpoint often correspond to higher absolute values ​​of reward prediction errors. Therefore, learning relevant state information in the cognitive map is further enhanced, improving the model's learning efficiency and enabling the mobile robot to make better decisions faster. Furthermore, reverse replay also performs well, second only to priority replay. From the state-action value update formula, it can be seen that the update of Q(s,a) is affected by the state-action value Q(s',a') of the next state. Reverse replay prioritizes the update of Q(s',a'), making the estimation of Q(s,a) more accurate, thus improving overall performance. In contrast, forward replay is beneficial for the model to plan based on the currently constructed cognitive map, but its estimation of Q(s,a) is not as accurate as that of backward replay, thus its performance is slightly worse. The unordered replay method randomly selects states in the cognitive map for updating, some of which are encountered accidentally by the mobile robot during exploration. Replaying these states contributes little to improving the model's performance in the current task. Therefore, the unordered replay model only outperforms the model without replay. In subsequent comparative experiments, a model with prioritized replay will be used to compare with other algorithms.

[0101] 2.3 Generalization Verification Experiment

[0102] This section will verify the generalization ability of the model proposed in this patent and compare it with the AC-DN model that combines the functions of the cerebellum and basal ganglia, the Only-BG model that only simulates the function of the basal ganglia, and the reinforcement learning algorithm SAC proposed in previous work. The experimental environment is as follows: Figure 7 As shown, the environment size is 200×200. The gray letter areas are used to randomly generate the initial and target positions of the mobile robot, and the blank areas are used to generate obstacles. There are four possible scenarios where the initial position of the mobile robot is diagonally opposite the target point. One environment is selected from each scenario to train the model. There are two training methods: the first method involves continuously alternating between the four environments for training; the second method involves the mobile robot completing training in one environment before moving on to the next. The mobile robot performs 400 training iterations in each environment, for a total of 1600 training iterations. During training, collisions with obstacles do not terminate the current round of training; training only ends when the robot reaches the target point.

[0103] During the testing phase, utilize Figure 7 The environment configuration shown was used for four rounds of experiments. In each round, 50 environments were randomly generated (each environment included the mobile robot's initial position, target position, and 20 obstacles). The generalization ability of different models was compared by counting the number of times the mobile robot successfully reached the target point.

[0104] Table 1. Comparison of generalization performance of the four models under the first training method.

[0105]

[0106] Table 2 shows the experimental results comparing the generalization performance of the four models under the second training method.

[0107]

[0108] Table 1 shows the number of times the mobile robot reached the target point during the testing phase under the first training method, as well as the average success rate of reaching the target point. Table 2 records the performance of different models in the test under the second training method.

[0109] As shown in Table 1, the model proposed in this patent, due to the presence of a reinforcement learning module, can learn more efficiently, make accurate value estimates, and thus exhibit good performance in multiple environments. Table 2 shows that the average success rate of reaching the target point decreased for all four models under the second training method. The SAC algorithm and the Only-BG model showed the most significant decrease in average success rate, nearly 40%. This is because in the second training method, the mobile robot can only enter the next environment after completing training in one environment, which places higher demands on the model's continuous learning ability. The continuous learning ability of the SAC algorithm and the Only-BG model relies on an experience pool, learning from past experiences to avoid forgetting. However, the experience pool has a size limit; anything exceeding the limit will overwrite past experiences, causing the mobile robot to forget the content learned in previous environments as the weights are updated when learning in a new environment. The average success rate of the model proposed in this patent and the AC-DN model decreased less. In the AC-DN model, the cerebellum relearns past experiences in a supervised learning manner, greatly reducing the forgetting of learned knowledge. In the model proposed in this patent, the mapping of previously learned knowledge is completed by incrementally constructing a cognitive map, which enables the mobile robot to virtually interact with the cognitive map during offline periods, thereby significantly reducing the degree of forgetting of previously learned content.

[0110] 2.4 Static Environment Comparison Experiment

[0111] This section will compare the performance of the four models in a more complex static environment, such as... Figure 8 As shown in the diagram, the environment size is 400×400. The blue circles represent the initial position of the mobile robot, the red pentagrams represent the target point, and there are 40 black squares representing 40 static obstacles. In the experiment, the mobile robot's step size was set to 10, and the safe distance d between the mobile robot and the obstacles was... safe Set to 6, the distance d to determine the arrival of the target point arrive Set to 6. The initial position coordinates of the mobile robot are (30, 25), and the coordinates of the target point are (350, 360). Different mobile robot models interact with the environment for 500 rounds each, ending the current round of interaction only when the mobile robot reaches the target point.

[0112] Each model underwent ten repeated experiments. During the interaction between the mobile robot and its environment, the cumulative rewards for different models were as follows: Figure 9 As shown.

[0113] from Figure 9As can be seen, since none of the models contain any prior knowledge, the reward accumulation curve shows a trend of first decreasing and then increasing. In the early stages of training, the AC-DN model and the model proposed in this patent have similar reward accumulations because the cerebellar model's learning and cognitive map construction are not yet complete. As training progresses, the cerebellar model's learning and the hippocampal cognitive map construction gradually improve, and the reward accumulation of both models increases rapidly. The model proposed in this patent, due to its model-based reinforcement learning planning ability, enables the mobile robot to quickly find a better strategy, thus achieving higher reward accumulation. The Only-BG model benefits from separate learning of positive and negative rewards, enabling it to quickly learn information about surrounding obstacles and reduce collisions, thus its reward accumulation value is higher than the SAC algorithm. Table 3 records some indicators of the mobile robot's interaction with the environment.

[0114] As shown in Table 3, the minimum number of steps required to reach the target point during training is the same for all four models, at 47 steps. This indicates that, with extensive interactive training, all four models have the opportunity to find a similar optimal trajectory. The start of positive reward accumulation indicates that the mobile robot encounters fewer or no obstacles during its interaction with the environment. The AC-DN model and the model proposed in this patent, due to the presence of the cerebellum model and the model-based reinforcement learning module, make more stable decisions for the mobile robot, avoiding over-exploration of the environment. Therefore, they can start accumulating positive rewards more quickly. In the third metric, the AC-DN model and the model proposed in this patent again perform best, especially the model proposed in this patent, which utilizes the model-based reinforcement learning module to combine and plan information from the cognitive map, further improving the decision-making performance of the mobile robot.

[0115] Table 3. Statistical data on mobile robots using different models in interactive learning.

[0116]

[0117] Table 3 shows that all four models have a chance to find an optimal trajectory during training. To verify the performance of the models after training, in Figure 10 The paper demonstrates the optimal decisions made by the four models based on the learned knowledge after 500 training iterations.

[0118] from Figure 10As can be seen, although the minimum number of steps required for all four models to reach the target point was 47 during training, only the model proposed in this patent made an optimal decision with 47 steps. This is because, when a mobile robot explores its environment extensively, different models have a probability of finding a path that consumes the fewest steps, but they may not necessarily have fully learned the states associated with that path. The model proposed in this patent can virtually interact with a cognitive map offline, and its reinforcement learning module can plan according to long-term goals, enabling the mobile robot to fully learn the states associated with the optimal path, thus making a better decision than other models in the verification experiment.

[0119] 2.5 Dynamic Environment Comparison Experiment

[0120] The environment of dynamic experiments, such as Figure 11 As shown, the environment size is 400×400, containing twenty obstacles. Sixteen black squares represent static obstacles, and four green triangles represent dynamic obstacles (dynamic obstacles move randomly in the environment). The parameters of the mobile robot are set the same as in the static experiment. The step size of the dynamic obstacles is the same as the step size of the mobile robot, both set to 10.

[0121] The experiment consisted of two phases: training and testing. During the training phase, no dynamic obstacles were added to the environment. Mobile robots equipped with different models interacted with the environment for 200 rounds, ending the current round only when they reached the target point. In the testing phase, dynamic obstacles were added to the environment. The mobile robots interacted with the dynamic environment for 50 rounds. If the mobile robot reached the target point without colliding with the obstacles, that round of the experiment was considered a success; otherwise, it was considered a failure.

[0122] This section involves ten rounds of repeated experiments. Figure 12 The number of successful trials for the mobile robot in the experiment is shown. Table 4 records the success rate of the mobile robot and the variance of the number of successful trials under different models in the experiment.

[0123] Table 4 Comparison of Indicators for the Four Models After Adding Dynamic Obstacles

[0124]

[0125] from Figure 12As can be observed from Table 4, the model proposed in this patent has the highest success rate in dynamic environments, and the variance of the number of successful experiments is also relatively low. This is because the model proposed in this patent can quickly update its cognitive map when the environment changes. Through virtual interaction with the cognitive map during offline processes, the mobile robot can quickly learn information about environmental changes and adjust its output actions to make more correct and stable decisions. The AC-DN model can also quickly learn information about environmental changes through the supervised learning function of the cerebellum when the environment changes, but it lacks active reasoning in the decision-making process, making its performance in dynamic environments slightly worse than that of the model proposed in this patent.

[0126] 3. Results and Analysis of Physical Experiments

[0127] To verify the model's performance in a real-world environment, this section presents a physical experiment. The experimental setup included two Rikirobot mobile robots (running Ubuntu 18.04) and two Lenovo laptops. The NoMachine software was used to remotely log into the Ubuntu desktops of the mobile robots, enabling desktop-level control, starting ROS nodes, and recording the robots' motion information during the experiment. The experiment consisted of two parts: a static environment experiment and a dynamic environment experiment, aiming to verify the AC-DN model's decision-making capabilities in real-world environments by observing the robots' performance in different conditions.

[0128] The robot's workflow is as follows: During robot movement, ROS is used to control the robot, and a model runs on the host computer. The model subscribes to the odometry topic ( / odom) to obtain the robot's pose information and subscribes to the LiDAR topic ( / scan) to obtain radar scan information of the current environment. Based on this information, the model generates the decision actions that the robot will perform, and then transmits the motion commands related to the actions to the robot through the publishing topic ( / cmd_vel), thereby achieving control over its movement.

[0129] 3.1 Static Environment Physical Experiment

[0130] Static physical experimental environments, such as Figure 13 As shown. The experimental environment measures 5m × 4.3m and contains eighteen static obstacles. The robot's initial position is taken as the origin, and the target point's coordinates are (3.8, 2.8). The robot's linear velocity is set to 0.3m / s, its angular velocity to 0.5rad / s, and the safe distance d between the robot and the obstacles is set to... safe And the distance d to reach the target point arrive All were set to 0.1m.

[0131] All models were first trained in a simulation environment, and then the trained models were loaded into the robot for physical environment experiments. Unlike interactions in the simulation environment, when the robot interacted with the physical environment, to ensure its safety, if the distance between the robot and the nearest obstacle was less than 0.1m, the current round of the experiment was immediately stopped. Robots equipped with different models interacted with the environment for thirty rounds each. Figure 14 The optimal decision paths for the four models over thirty rounds are shown through the RVIZ visualization interface.

[0132] Figure 14 The optimal path shown is the path that consumes the fewest steps while ensuring the robot reaches the target point. In this physical environment, due to the low friction between the robot and the ground, and influenced by the robot's own performance, the robot's movement varies in each round, even when operating in the same environment. Therefore, this places a significant challenge on the model's performance.

[0133] Table 5 records some statistical indicators of the four models in the thirty-round experiment.

[0134] Combination Figure 14 As shown in Table 5, the AC-DN model and the model proposed in this patent performed best in the static environment physical experiment. Robots equipped with these two models successfully reached the target point in each round of the experiment, and the variance of the number of steps required to reach the target point was also small. This indicates that these two models have good decision-making stability. Furthermore, the minimum and average number of steps required to reach the target point in each round show that the model proposed in this patent performs better. This is because in this model, the model-based reinforcement learning module can plan using a cognitive map, enabling the model to dynamically switch between long-term and short-term goals, thus performing well in optimal decision-making.

[0135] 3.2 Dynamic Environmental Physics Experiment

[0136] This section will further verify the model's performance through physical experiments under dynamic conditions. The experimental environment is as follows: Figure 15 As shown. The experimental environment measures 5m × 4.3m and contains thirteen static obstacles and one dynamic obstacle. The initial position of the robot is taken as the origin, and the target position is (3.8, 2.8). The robot's relevant parameters are set in the same way as in Section 2.1.

[0137] The training process for the model is the same as that in static physics experiments. First, the model proposed in this patent is trained in a simulation environment, and then loaded into the robot for dynamic physics experiments. At the start of the experiment, the mobile robot and the dynamic obstacles begin moving simultaneously. Figure 16The RVIZ visualization interface displays the movement trajectories of the robot and dynamic obstacles.

[0138] The motion trajectory diagram shows that the robot successfully avoided obstacles and reached the target point during the dynamic experiment. To clearly demonstrate the motion of the robot and the dynamic obstacle, the coordinates of their trajectories and the radar scan information of the environment were recorded and plotted on the same graph, as shown below. Figure 17 As shown in the image, the red dots represent environmental information scanned and recorded by the LiDAR, the blue circles represent the robot's movement trajectory, and the black asterisks represent the movement trajectories of dynamic obstacles.

[0139] Figure 18 The system labels the positions of the robot and the dynamic obstacle at corresponding times to more clearly demonstrate the decisions the robot makes when the dynamic obstacle approaches.

[0140] Figure 18 The robot and dynamic obstacles are represented by blue circles and orange stars, respectively, and the numbers in the diagram indicate the timing of the robot's and the dynamic obstacles' actions. Figure 18 As can be seen, the robot encountered the dynamic obstacle twice, at time 9 and time 21. In both encounters, the robot successfully avoided the obstacle and reached the destination. This result demonstrates that the model proposed in this patent can effectively handle obstacle avoidance problems in dynamic environments, providing reliable support for robot navigation in complex scenarios.

[0141] To compare the performance of the model proposed in this patent with other models in dynamic environments, this section conducted thirty rounds of repeated experiments. In each round, the starting position of the dynamic obstacles was randomly changed to simulate a more realistic and complex dynamic environment. During the experiment, if the robot collided during navigation, that round was considered a failure; otherwise, it was considered a success. Figure 19 The experiment visually demonstrates the number of rounds in which four different models succeeded and failed.

[0142] from Figure 19 As can be seen, the robots equipped with the proposed model and the AC-DN model performed best in the dynamic physics experiment, both successfully reaching the target point 25 times, with a success rate of approximately 83%. This result demonstrates that these two models possess strong adaptability in dynamic environments, effectively coping with changing dynamic obstacles, avoiding collisions, and successfully completing navigation tasks.

[0143] In summary, the robot's performance in both static and dynamic physics experiments demonstrates that the hybrid model combining model-free reinforcement learning and model-based reinforcement learning proposed in this patent exhibits good applicability in real-world environments. This model not only efficiently learns navigation strategies in complex environments but also demonstrates strong generalization ability and stability, enabling the mobile robot to make reasonable decisions, successfully avoid obstacles, and reach its target point smoothly when faced with dynamically changing obstacles.

[0144] The above are merely preferred embodiments of the present invention and are not intended to limit or restrict the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection declared by the present invention.

Claims

1. A mobile robot behavior decision-making method inspired by hippocampal and prefrontal cortex functions, which obtains the current state and target position of the mobile robot from the environment, characterized in that: A model-based reinforcement learning module is constructed using the spatial cognitive function of the hippocampus. The current state is reduced to a two-dimensional discrete state and input into the model-based reinforcement learning module, which then outputs action 'a'. MB The current state is input into the model-free reinforcement learning module, which then outputs action a, based on the memory functions of the basal ganglia and hippocampus. This input is then fed into the decision controller. MF The decision controller simulates the switching function of the prefrontal cortex between model-based reinforcement learning modules and model-free reinforcement learning modules. The decision controller outputs action 'a' to complete the behavior decision of the mobile robot.

2. The mobile robot behavior decision-making method inspired by hippocampal and prefrontal cortex functions as described in claim 1, characterized in that: The decision controller makes decisions by calculating action a using action selection entropy. MB and action a MF The corresponding action selection entropy H MB (s) and H MF (s), through Choose the action to perform, where k is a constant.

3. The mobile robot behavior decision-making method inspired by hippocampal and prefrontal cortex functions as described in claim 2, characterized in that: Action selection entropy is In the formula, H(s) represents the action selection entropy in state s, and P(a|s) represents the probability of choosing action a in state s.

4. The mobile robot behavior decision-making method inspired by hippocampal and prefrontal cortex functions as described in claim 1, characterized in that: The method for constructing a model-based reinforcement learning module is as follows: (1) Simulate the spatial cognitive function of the hippocampus to construct a cognitive map, which includes a state transition function and a reward function; (2) The value iteration method is used for action planning and updating of the state-action value function; Where Q(s,a) represents the state-action value function, which represents the expected cumulative reward of the mobile robot for performing action a in the current state s; γ represents the discount factor, which is a constant between 0 and 1; and A represents the set of all executable actions of the mobile robot in the next state s′. Let T(s,a,s′) represent the maximum state-action value corresponding to the next state s′, where T(s,a,s′) is the state transition function and R(s,a,s′) is the reward function. (3) Convert the state-action value into different action selection probabilities through the Softmax function, and send the action selection probabilities and corresponding actions to the decision controller; (4) The mobile robot performs action a, reaches the next state s', and obtains a reward r from the environment, which is stored in the reward function to update the cognitive map; (5) Repeat steps (2)-(4) until the preset number of training sessions is reached.

5. The mobile robot behavior decision-making method inspired by hippocampal and prefrontal cortex functions as described in claim 4, characterized in that: The state transition function is In the formula, N(s,a) represents the cumulative number of times the mobile robot performs action a in state s, and N(s,a,s') represents the number of times the mobile robot reaches state s' after performing action a in state s. When the mobile robot performs N actions in state s... g After action a, if in N g If state s' is not reached in any of the steps, then the corresponding N(s,a,s') is reduced to half of its original value, and N(s,a) is adjusted to N(s,a)←N(s,a)-N(s,a,s') / 2.

6. The mobile robot behavior decision-making method inspired by hippocampal and prefrontal cortex functions as described in claim 4, characterized in that: The Softmax function is in, It is the sum of the exponentially expressed Q values ​​of all possible actions, where τ represents the temperature parameter and is a positive real number; exp(Q(s,a) i ) / τ) represents the exponential Q value of the action.

Images