Robot brain-like behavior decision-making model and method based on multi-brain-region cooperation

Abstract

The invention discloses a robot brain-like behavior decision model and method based on multi-brain-region cooperation, and belongs to the technical field of robot intelligent control. The basal ganglion module in the invention adopts the DQN to process the action decision of the high-dimensional state space, and the Q value stability is improved through the dynamic fusion of the target network and the MDN; the cerebellum module introduces a neuron remodeling mechanism, and dynamically resets the neuron age to reverse learning ability attenuation; the hippocampus module stores task data through a memory playback mechanism, and optimizes MDN parameters offline to enhance environmental adaptability; designing a self-adaptive step length strategy, dynamically adjusting the motion step length according to the obstacle distance and the rotation angle, and balancing the navigation efficiency and the obstacle avoidance precision; and the hybrid integration module dynamically adjusts cerebellum and basal ganglion output in combination with correlation and trajectory evaluation indexes to generate an optimal decision action. The dynamic decision-making efficiency and the environmental adaptability of the robot are improved by simulating the functions of the biological brain region and combining the optimization algorithm.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control, specifically to a brain-like behavioral decision-making model and method for robots based on multi-brain region collaboration. Background Technology

[0002] Mobile robots, as a fusion of artificial intelligence and robotics, have been widely applied in industries such as manufacturing, medicine, and the military. However, most existing robots are designed for specific tasks, requiring reprogramming of their control systems when tasks or environments change, resulting in poor versatility and weak adaptability. Especially in dynamic and unknown environments, robots need to process complex scene information in real time and make autonomous decisions, where traditional control methods struggle to meet the demands for flexibility.

[0003] Current mainstream intelligent algorithms, such as deep reinforcement learning (DQN) and particle swarm optimization, can achieve a certain degree of autonomous decision-making, but they rely on large amounts of data for training, lack biological interpretability, and suffer from low computational efficiency and insufficient generalization ability in high-dimensional state spaces. For example, when dealing with complex environments, deep reinforcement learning models often face difficulties in real-time decision-making due to the explosion of state space dimensions, and their long-term learning ability is prone to decline due to neuronal aging.

[0004] Inspired by the structure of the biological brain, brain-like intelligent algorithms provide new directions for robot decision-making by simulating the collaborative mechanisms of multiple brain regions (such as the basal ganglia, cerebellum, and hippocampus). Existing research has attempted to integrate basal ganglia and cerebellum models for motor control and path planning. For example, Caligiore et al. [Caligiore D, Arbib MA, Miall RC, et al. The super-learning hypothesis: Integrating learning processes across cortex, cerebellum and basal ganglia[J]. Neuroscience & Biobehavioral Reviews, 2019, 100: 19-34.] integrated the cerebral cortex, cerebellum, and basal ganglia into a highly integrated system and proposed a series of hypotheses. Dasgupta et al. [Dasgupta S, F, Manoonpong P. Neuromodulatory adaptive combination of correlation-based learning in cerebellum and reward-based learning in basal ganglia for goal-directed behavior control[J]. Frontiers in neural circuits, 2014, 8:126.] These models simulate the learning mechanisms of the basal ganglia and cerebellum using recurrent neural network and feedforward correlation learning models, respectively, and utilize thalamic reward-modulated heterosynaptic plasticity to regulate the output of the cerebellum and basal ganglia. However, as task complexity increases, these models reveal problems such as low training efficiency, high computational resource consumption, and insufficient long-term decision-making ability.

[0005] Taking the multi-brain region model proposed by Wang et al. [Wang D, Hu Y, Ma T. Mobile robot navigation with the combination of supervised learning in cerebellum and reward-based learning in basal ganglia[J]. Cognitive Systems Research, 2020, 59: 1-14.] as an example, although it can achieve basic decision-making, it has two major problems in complex environments: (1) Insufficient real-time decision-making capability: As the dimension of task state increases, the computational delay of the model in high-dimensional space is significant, making it difficult to meet the real-time requirements in dynamic environments. (2) Long-term learning ability decline: As the activation age of neurons in the cerebellar regulatory developmental network (MDN) increases, the learning efficiency decreases, which limits the robot's adaptability in long-term tasks. Summary of the Invention

[0006] To address the shortcomings of existing robot decision-making models in terms of adaptability and long-term learning ability in complex environments, this invention provides a robot brain-like behavioral decision-making model and method based on multi-brain region collaboration. By simulating the functions of biological brain regions and combining optimization algorithms, the dynamic decision-making efficiency and environmental adaptability of robots are improved.

[0007] To achieve the above objectives, the specific solution adopted by the present invention is as follows: On one hand, this invention discloses a brain-like behavioral decision-making model for robots based on multi-brain region collaboration, comprising: The basal ganglion module employs a DQN-simulated reinforcement learning mechanism to handle action decisions in a high-dimensional state space. The DQN includes an evaluation network, a target network, and an experience replay pool. The evaluation network calculates the state-action value function in real time. The target network periodically synchronizes parameters through a delayed update mechanism to stabilize Q-value estimation. The outputs of the DQN's target network and the cerebellar regulatory developmental network (MDN) are fused with dynamically weighted parameters to generate the final Q-value. The cerebellar regulatory developmental network (MDN) simulates a supervised learning mechanism and dynamically resets neuronal age through a neuronal remodeling strategy to maintain long-term learning ability. The hippocampus module simulates the human memory replay mechanism, saving data from online tasks and retraining the cerebellar regulatory developmental network (MDN) offline to improve the model's learning ability and environmental adaptability. The adaptive step size strategy module dynamically adjusts the robot's step size based on the distance between the robot and the nearest obstacle and the rotation angle during movement. The hybrid integration module combines correlation indicators and trajectory evaluation indicators to dynamically regulate the output of the cerebellum and basal ganglia, achieving effective fusion of the output of the cerebellum and basal ganglia to generate the final behavioral decision action.

[0008] Furthermore, the dynamic ∈-greedy policy of the DQN satisfies the following formula: ∈ t =∈ min +(∈ max -∈ min )·e -λt ; Where, ∈ t Represents the exploration rate at time t, ∈ max and ∈ min Let be the maximum and minimum values ​​of the exploration rate ∈, respectively, λ be the second decay factor, and t be the time step.

[0009] Furthermore, the comprehensive reward formula for the reward value r of the reward function of the DQN is: r = r b +ω1r1+ω2r2; Where ω1 and ω2 are reward coefficients, r b The base reward function is set to a positive value when the target is reached and a negative value when a collision occurs; r1 is the obstacle distance reward value, which increases when moving away from the obstacle and decreases when moving closer; r2 is the target approach reward value, which increases when approaching the target and decreases when moving away.

[0010] Furthermore, the neuronal remodeling strategy of the cerebellar regulatory developmental network (MDN) specifically includes: When the activation times of a neuron exceed a preset threshold, its age is reset to the initial value; The cell reprogramming technology is used to simulate the regeneration process of biological neurons, and the restoration of the learning ability of neurons is achieved through the following formula: where n max represents the set neuron age threshold, n i and n′ i are the ages of the neurons before and after remodeling, respectively.

[0011] Furthermore, in the adaptive step size strategy module, the movement step size of the robot is closely related to the distance to the nearest obstacle and the angle of rotation during movement. The specific relationship is as follows: When the distance d between the robot and the nearest obstacle satisfies d > d2, the movement step size L of the robot is expressed as: When the distance d between the robot and the nearest obstacle satisfies d1 < d < d2 (medium-distance region), the movement step size L of the robot can be expressed as: When the distance d between the robot and the nearest obstacle satisfies d < d1 (close-distance region), the movement step size L of the robot can be expressed as: where L represents the finally calculated adaptive movement step size, which is jointly determined by d and Δθ. l max and l min represent the maximum movement distance and the minimum movement distance of the robot during movement, respectively. d1 and d2 represent the preset distance thresholds for dividing the proximity degree regions of the obstacles, Δθ represents the real-time rotation angle of the robot during movement, reflecting the steering amplitude. θ1 and θ2 represent the preset angle thresholds for dividing the levels of the steering amplitude. Δθ < θ1 represents small-angle steering, θ1 < Δθ < θ2 represents medium-angle steering, and Δθ > θ2 represents large-angle steering.

[0012] Furthermore, the action fusion method of the hybrid integration module is as follows: (1) Calculate the trajectory evaluation indexes G1 and G2 of the output action a1 of the cerebellum and the output action a2 of the basal ganglia; (2) Generate the final action through weighted generation by the normalized correlation coefficients λ1 and λ2 where s = λ1G1 + λ2G2.

[0013] On the other hand, the present invention discloses a robot behavior decision-making method based on multi-brain region cooperation, including the following steps: (1) Initialize the robot state, including position, target point, obstacle information, etc.; (2) Based on the current state, calculate the respective action outputs a1 and a2 through the basal ganglia module and the cerebellum module respectively; (3) Dynamically adjust the robot's step size according to the adaptive step size strategy module; (4) By integrating correlation indicators and trajectory evaluation, the final action is generated by integrating a1 and a2; (5) Perform actions and save online data in the hippocampus module to retrain the cerebellar regulatory development network and remodel neurons in an offline state.

[0014] Beneficial effects: (1) Improve decision stability and training efficiency: Optimize the stability of Q-value estimation through a weighted fusion mechanism of DQN and MDN. For example... Figure 4 As shown, the reward convergence speed of the fusion model is significantly faster than that of the standalone DQN. In the early stages of training, MDN provides reliable goal guidance, reduces invalid exploration, and enables the robot to learn the optimal policy more quickly in the high-dimensional state space. Static simulation experiments show that the behavioral decision-making model of this invention reduces the number of steps from 63 to 59 in a simple environment, and the path length also decreases from 620.9 to 592.7, verifying the improvement in training efficiency.

[0015] (2) Enhancing Long-Term Learning Capacity: The neuronal remodeling strategy effectively alleviates learning capacity decline by resetting the age of aging neurons. In long-term tasks, the model can maintain stable decision-making performance and avoid performance degradation caused by increased neuron activation. Dynamic simulation experiments show that the behavioral decision-making model of this invention has a significantly higher navigation success rate than the original model in 50 experiments, demonstrating its long-term adaptability advantage in complex environments.

[0016] (3) Optimize dynamic environment adaptability: The adaptive step size strategy achieves a balance between efficiency and safety: the maximum step size can reach l when away from obstacles. max The minimum step size when approaching is l min This makes the robot's path smoother in static environments and its obstacle avoidance more flexible in dynamic environments.

[0017] (4) Generalization advantage of multi-brain region collaboration: The hybrid integration module dynamically adjusts the output weights of brain regions through trajectory evaluation and correlation indicators, enabling the model to intelligently select the optimal decision path in different scenarios. Whether in a static complex environment (22 obstacles) or a dynamic random environment, the number of steps and path length of the behavioral decision model of this invention are better than traditional algorithms such as DQN and SAC, and it can still maintain efficient planning after exchanging start and end points, verifying its robustness and generalization ability. Attached Figure Description

[0018] Figure 1 This is a diagram of the brain region structure and pathways of a robot.

[0019] Figure 2 This is a diagram of the DQN algorithm structure.

[0020] Figure 3 This is a flowchart illustrating the regulation of developmental networks after the introduction of neuronal remodeling mechanisms.

[0021] Figure 4 This is the reward graph obtained by DQN and DQN+MDN during training.

[0022] Figure 5 This is a schematic diagram of the optimal path in a simple static simulation environment.

[0023] Figure 6 This is a schematic diagram of the optimal path in a complex static simulation environment.

[0024] Figure 7 This is a schematic diagram of the decision path of the behavioral decision-making model of the present invention after exchanging the start and end points.

[0025] Figure 8 This is a diagram showing the positional relationships in a dynamic simulation experiment.

[0026] Figure 9 This is an experimental path diagram of the original model and the behavioral decision-making model of this invention in a dynamic environment.

[0027] Figure 10 This is a performance comparison chart of the original model and the behavioral decision-making model of this invention in 50 experiments.

[0028] Figure 11 A comparison chart showing the minimum number of steps for different models under different environments.

[0029] Figure 12 This is a static physical environment diagram.

[0030] Figure 13 The motion path diagrams of the robot in a static physical environment are shown using four different models.

[0031] Figure 14 A dynamic physical environment diagram showing the positions of different dynamic obstacles.

[0032] Figure 15 The image shows the motion trajectory of the robot in two dynamic environments using the behavioral decision-making model of this invention.

[0033] Figure 16 This is a trajectory diagram of a dynamic obstacle in two dynamic environments.

[0034] Figure 17This is a motion trajectory diagram of a robot and dynamic obstacles in a dynamic physical environment.

[0035] Figure 18 This is a motion trajectory diagram of the robot and dynamic obstacles in the second dynamic physical environment. Detailed Implementation

[0036] The technical solution of the present invention will be clearly and completely described below with reference to specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0037] This invention proposes a robot behavior decision-making model and method based on multi-brain region collaboration. This behavior decision-making model combines a basal ganglia module, a cerebellar regulatory developmental network (MDN), a hippocampus module, an adaptive step size strategy module, and a hybrid integration module. The following section provides a detailed analysis of each module and the results of simulation and physical experiments using this behavior decision-making model. It should be noted that this invention is an improvement upon existing literature 1 (Wang D, Liu Q, Duan Y. Behavioral Decision-Making of Mobile Robots Simulating the Functions of Cerebellum, Basal ganglia and Hippocampus[J]. IEEE Transactions on Artificial Intelligence, 2025), and the original model mentioned below is also the behavior decision-making model from literature 1.

[0038] (I) Basal ganglion module

[0039] In existing literature 1, the basal ganglia are modeled using an Actor-Critic approach, with an RBF (Radial Basis Function) neural network as the Actor and Q-learning as the Critic. A significant problem with the Q-learning algorithm is the need for a Q-table to store state-action pairs. However, as the dimensionality of the state or action space increases, the size of the Q-table grows exponentially, leading to the curse of dimensionality. To address this issue, this invention optimizes the robot's brain-like behavioral decision-making model, whose brain region structure and pathways are as follows: Figure 1 As shown below, the basal ganglia module will be described in detail.

[0040] This invention employs a deep Q-learning network, hereinafter referred to as "DQN," in the basal ganglia module. This network uses a neural network as an approximation function for the Q-value, avoiding the limitations of tabular storage. DQN is a deep reinforcement learning algorithm that combines deep neural networks and Q-learning. It consists of three parts: an evaluation network, a target network, and an experience replay pool. The structure of the DQN algorithm is as follows: Figure 2 As shown.

[0041] In the DQN algorithm, the formula for updating the Q-value of the neural network is: Q(s t ,a t ;θ)←Q(s t ,a t ;θ)+α(y t -Q(s t ,a t ;θ)) #(1) In the formula, Q(s) t ,a t ;θ) is the state-action value function at time t, where θ is the parameter of the evaluation network and α is the learning rate.

[0042] Target Q value y t It is based on target network computation: y t =r t +γmaxQ(s t+1 ,a t+1 ;θ - #(2) In the formula, Q(s) t+1 ,a t+1 ;θ - ) is the state-action value function at time t+1, where θ - For the parameters of the target network, y t γ is the Q-value of the target network, and γ is the first decay factor (0 < γ < 1).

[0043] The gradient update in the DQN algorithm is accomplished through a loss function, typically the mean squared error loss function, as shown in the following equation: loss = E[(y t -Q(s t ,a t ;θ)) 2 #(3) In the formula, loss is the calculated loss function.

[0044] DQN uses deep neural networks to approximate the Q-value function, effectively handling high-dimensional and complex state spaces, thus alleviating the computational burden. Furthermore, by... Figure 2As can be seen, DQN introduces a target network to calculate the target Q-value and updates its parameters at fixed step intervals during training. This delayed update mechanism helps maintain the relative stability of the target Q-value, reduces the volatility of the objective function, and thus improves the convergence and stability of the model. However, DQN still faces some challenges, such as overestimation of Q-value, instability of the target value in the early stages of training, and long-term reward estimation bias, which may lead to slow convergence during training or even getting stuck in a suboptimal solution. To address this, this invention uses a fused-trained cerebellar regulatory developmental network (MDN) as an auxiliary target network, which is weighted and fused with the DQN target network to provide a more instructive Q-value estimate. Compared to traditional methods that rely solely on the DQN target network, this provides more reliable target guidance in the early stages of training, thereby improving the stability and convergence speed of DQN training. The specific process is shown in Algorithm 1.

[0045] It is important to note that in DQN networks, a balance must be maintained between exploration and exploitation during the reinforcement learning exploration process; otherwise, the robot may easily fall into suboptimal strategies in complex environments. Furthermore, the design of the reward function directly affects the robot's learning efficiency and final performance. If the reward signal is too sparse or insufficiently guided, it may lead to slow convergence or local optima. Therefore, this invention improves the DQN algorithm by optimizing the greedy strategy and reward function to enhance the algorithm's learning efficiency and adaptability. First, a dynamic ∈-greedy strategy is adopted, allowing the robot to explore the environment more extensively in the early stages of training, avoiding premature immersion in local optima, while gradually shifting to more stable strategy optimization in the later stages of training. Second, a reward mechanism based on the target and the nearest obstacle is designed, enabling the robot to make decisions not only based on immediate rewards but also to optimize overall path planning under the guidance of long-term goals. These improvements enhance the generalization ability of DQN in complex environments and improve the stability and convergence speed of the robot's autonomous decision-making.

[0046] (1) Dynamic ∈-greedy strategy

[0047] The commonly used exploration strategy in the DQN algorithm is ∈-greedy. This strategy balances exploration and utilization during training, and its formula is as follows: In the formula, a t It is the action chosen at time t, and ∈ represents the exploration rate.

[0048] In the ∈-greedy strategy, the exploration rate ∈ remains uniform across all states. This can lead to situations where some states, even those already well-learned, still exhibit a probability of ∈ for random exploration, thus impacting learning efficiency. In reality, in the early stages of training, thorough exploration helps cover more environmental states; while in the later stages, utilizing existing strategies helps accelerate convergence. Therefore, this invention designs a time-decreasing ∈ to dynamically adjust the balance between exploration and utilization. Its formula is shown below: ∈ t =∈ min +(∈ max -∈ min )·e -λt #(5) Where, ∈ t Represents the exploration rate at time t, ∈ max and ∈ min Let be the maximum and minimum values ​​of the exploration rate ∈, respectively, λ be the second decay factor, and t be the time step.

[0049] (2) Reward Function Design

[0050] In reinforcement learning, the robot needs to interact with its environment by performing actions, thereby receiving new states and rewards. The network parameters are then updated based on the reward values ​​to learn better strategies. Therefore, the reward function is crucial in reinforcement learning, and its quality directly affects the robot's behavioral decisions. In the traditional DQN algorithm, the reward function (i.e., the basic reward function r)... b The settings are as follows: C1 is usually a positive number, and C2 is usually a negative number.

[0051] The reward function in this invention has been improved and consists of three parts, namely the basic reward function r. b The reward function is defined as follows: r1 is the obstacle distance reward related to the distance to the nearest obstacle, and r2 is the target proximity reward related to the target distance. r1 and r2 represent the rewards given after the robot performs an action if it is farther from the nearest obstacle or closer to the target, and penalized otherwise. The reward function is designed as follows: r = r b +ω1r1+ω2r2 #(9) Where r represents the reward value obtained by the robot, ω1 and ω2 are reward coefficients, and d e and d′ e d represents the distance between the robot and the nearest obstacle before and after the movement. f and d′ fThese represent the distances between the robot and the target point before and after the movement, respectively.

[0052] Furthermore, in the cerebellar regulatory developmental network, the learning rate of neurons is closely related to their "age" (number of activations), and their learning ability gradually decreases with increasing activation frequency. Drawing on the neurophysiological mechanisms of cellular senescence and cell reprogramming technology, a "remodeling" process of cerebellar neurons is proposed, aiming to restore their learning ability by remodeling the age of neurons, thereby enhancing the overall learning and decision-making capabilities of the network. The cerebellar regulatory developmental network is described below.

[0053] (II) Cerebellar Regulatory Development Network

[0054] Cellular senescence is a natural process in the life cycle of an organism, particularly pronounced in the nervous system. With age, the function of the nervous system gradually declines, manifested as a reduction in neuronal connections, slower information transmission speed, decreased neural network stability, and weakened learning ability, thus affecting an individual's ability to adapt to new environments. As the aging process progresses, the plasticity and adaptability of the nervous system also decline, weakening the ability of neurons to adjust their connections and respond to environmental changes, thus slowing down learning. In contrast, in young nervous systems, neurons possess high plasticity, flexibly adjusting synaptic connections in response to external stimuli, strengthening important neural pathways, and forming long-term memories. With the accumulation of aging, this plasticity gradually decreases, significantly weakening learning efficiency and adaptability.

[0055] Inspired by physiological mechanisms, this invention introduces a neuronal remodeling mechanism into the cerebellar regulatory developmental network to simulate the neuronal regrowth process in biological systems. Specifically, when the age of neurons exceeds a certain threshold and their learning ability significantly declines, the age of the neurons is reset, which is equivalent to simulating the neuronal renewal process in the biological brain. This effectively avoids the problem of declining learning ability due to neuronal aging. Through this mechanism, the regulatory developmental network can maintain a high level of learning ability, enabling robots to have better behavioral decision-making capabilities in complex and dynamic environments. After introducing the neuronal remodeling mechanism, the renewal process of the cerebellar regulatory developmental network is as follows: Figure 3 As shown.

[0056] Depend on Figure 3 As can be seen, after identifying the winning neuron, the system first activates it and updates its weights. Subsequently, if the neuron's age exceeds a set threshold, it is considered to have entered the aging stage. At this point, the neuron needs to be remodeled to restore its learning ability. The neuron remodeling process is as follows: Where, n max This represents the set neuron age threshold, n i and n′ iThey are the ages of the neurons before and after reshaping, respectively.

[0057] (III) Hippocampus Module

[0058] The hippocampus module can simulate the human memory replay mechanism, store the data of the online execution task, and retrain the cerebellar modulation development network (MDN) in the offline state to improve the learning ability and environmental adaptability of the model. The hippocampus module in this invention belongs to the prior art and is described in detail in the existing literature 1, so it will not be elaborated in this invention.

[0059] (IV) Adaptive Step-Size Strategy Module

[0060] During the robot behavior decision-making process, a fixed step size may not be able to cope with a complex and dynamically changing environment, which limits the adaptability and flexibility of the robot to the environment. In an open area, a smaller movement step size may lead to additional computational overhead and a larger number of movement steps, reducing the navigation efficiency; while in an area with dense obstacles, a larger movement step size may make it difficult for the robot to perform precise obstacle avoidance and path adjustment, increasing the risk of collision. In addition, a fixed step size also affects the smoothness of the robot's path. Therefore, this invention adopts an adaptive movement step-size strategy to improve the flexibility, path smoothness and computational efficiency of the robot.

[0061] The movement step size of the robot is closely related to the distance to the nearest obstacle. When the distance between the robot and the nearest obstacle is close, in order to avoid collision, the movement step size of the robot is correspondingly reduced; when the distance to the nearest obstacle is far, the robot can move with a larger step size to reduce decision-making and improve the movement efficiency. In addition, the rotation angle of the robot also affects the movement step size. When the rotation angle is small, the robot can take a larger movement step size to increase the movement speed; on the contrary, when the rotation angle is large, the movement step size will be reduced to ensure the smoothness of the movement. Therefore, the movement step size of the robot is jointly affected by the distance to the nearest obstacle and the rotation angle during the movement process. The specific process is as follows: When the distance d between the robot and the nearest obstacle satisfies d > d2 (far-distance area), the movement step size L of the robot can be expressed as: When the distance d between the robot and the nearest obstacle satisfies d1 < d < d2 (mid-distance area), the movement step size L of the robot can be expressed as: When the distance d between the robot and the nearest obstacle satisfies d < d1 (near-distance area), the movement step size L of the robot can be expressed as: Among them, L represents the finally calculated adaptive movement step size, which is jointly determined by d and Δθ. lmax and l min d1 and d2 represent the maximum and minimum travel distances of the robot, respectively. d1 and d2 represent preset distance thresholds used to define the proximity zones of obstacles. Δθ represents the real-time rotation angle of the robot, reflecting the turning amplitude. θ1 and θ2 represent preset angle thresholds used to classify the levels of turning amplitude: Δθ < θ1 indicates a small-angle turn, θ1 < Δθ < θ2 indicates a medium-angle turn, and Δθ > θ2 indicates a large-angle turn.

[0062] (V) Hybrid Integration Module

[0063] During robot operation, the action outputs of the cerebellum and basal ganglia may be consistent or inconsistent. Previously, correlation indices were used to synthesize the outputs of the cerebellum and basal ganglia. However, a single correlation index is easily affected by noise and data perturbation; a small deviation may be over-amplified during calculation, thus affecting the stability and reliability of the results. Furthermore, a single correlation index only considers the similarity of the external environment and fails to adequately consider whether the action can effectively reach the expected goal. To address this issue, a trajectory evaluation index was introduced to evaluate the trajectories of the cerebellum and basal ganglia separately. A weighted average of the correlation and trajectory evaluation indices ensures environmental similarity while also taking into account the goal-oriented function, improving the accuracy of action selection and enhancing the robustness and precision of the system.

[0064] During the exploration process, the robot acquires information from the environment and transmits this information to the cerebellum and basal ganglia, respectively. Based on the acquired information, they calculate the corresponding action outputs a1 and a2, and evaluate the trajectory using these two action outputs. The evaluation formula is as follows: G = k1·head + k2·dist e +k3·dist f #(14) Where G represents trajectory evaluation, head is the evaluation function for the rotation angle during robot motion; dist e This is the obstacle distance evaluation function, representing the difference in distance between the robot and the nearest obstacle before and after movement; dist f It is the target position distance evaluation function, which represents the difference in distance between the robot and the target position before and after the movement. k1, k2, and k3 are all coefficients of the evaluation function.

[0065] When evaluating using head, use θ. 1 and θ 2 Let θ represent the robot's direction of motion before and after the motion, respectively, and let Δθ represent the angle of rotation of the robot during the motion, i.e., Δθ = |θ|. 1 -θ 2Since distance evaluation functions generally indicate better performance with larger numbers, this principle should also be followed when evaluating using the head. Based on this, the head evaluation function can be defined as follows: head=π-Δθ #(15) By integrating correlation and trajectory evaluation indicators, the motor outputs of the cerebellum and basal ganglia are fused to ensure that the actions possess both high similarity and effective goal-oriented function. The final output action 'a' can be represented as: S=λ1G1+λ2G2 #(16) Where S is a normalization coefficient, λ1 and λ2 are correlation coefficients, a1 and G1 are the cerebellar action output and trajectory evaluation, respectively, and a2 and G2 are the basal ganglia action output and trajectory evaluation, respectively.

[0066] In practice, to enhance the learning ability of MDN, the stored state-action data is used to retrain the MDN in offline mode after task execution, thereby improving its decision-making ability. During the task execution phase, the action output is jointly determined by DQN and MDN. Through this mechanism, MDN achieves the ability to execute tasks online and learn continuously offline, enabling the system to continuously improve its adaptability and learning ability over long-term operation. Therefore, the decision-making method based on the behavioral decision-making model of this invention is shown in Algorithm 2.

[0067] In summary, the behavioral decision-making model of the present invention has the following characteristics: (1) In the basal ganglia module, DQN combines the advantages of deep neural networks and Q-learning to effectively solve the computational problem in high-dimensional state space, and uses the cerebellar regulatory developmental network (MDN) as an auxiliary target network to dynamically weight and fuse with the target network of DQN to improve the stability of Q-value estimation and training effect, especially to provide more accurate guidance in the early stage of training; (2) In the cerebellar regulatory developmental network (MDN), in response to the problem of the decline in the long-term learning ability of cerebellar neurons, the neuronal remodeling mechanism is proposed by drawing on the cell reprogramming technology in biology. By remodeling the age of neurons, the learning ability of the model in long-term tasks is enhanced; (3) An adaptive step size strategy is designed so that the robot can flexibly adjust the step size, accelerate forward when away from obstacles, and accurately avoid obstacles when approaching obstacles, thereby optimizing the overall behavioral decision-making process; (4) The integration method of the hybrid model module is improved by combining correlation indicators and trajectory evaluation indicators to realize the dynamic adjustment of the output of the cerebellum and basal ganglia.

[0068] (VI) Simulation Experiment and Result Analysis

[0069] The following simulation experiment was conducted using the behavioral decision-making model of the present invention, and the results were analyzed.

[0070] To verify the performance improvement of the behavioral decision-making model of this invention compared to the original model (the behavioral decision-making model in existing literature 1), comparisons were made through static and dynamic simulation experiments. In the static simulation experiments, the task completion rate and decision accuracy of the two models were evaluated under fixed environment and task objectives to verify the performance improvement of the behavioral decision-making model of this invention under static conditions. In the dynamic simulation experiments, randomly changing dynamic obstacles were introduced to test the adaptability and robustness of the two models in complex dynamic environments, further verifying the behavioral decision-making capability of the behavioral decision-making model of this invention in dynamic scenarios.

[0071] To verify the effectiveness of integrating MDN into DQN, this invention compares and analyzes the changes in rewards obtained by standalone DQN and DQN integrated with MDN in navigation decision-making tasks, such as... Figure 4 As shown in the figure, the experimental results demonstrate that the DQN fused with MDN converges faster and yields higher reward values ​​compared to DQN alone. As an auxiliary target network, MDN provides more reliable target guidance in the early stages of training, reducing unnecessary exploration and thus improving training stability and convergence speed. This result validates the effectiveness of fusing MDN and DQN.

[0072] (1) Static Experiment

[0073] The performance of the behavioral decision-making model of the present invention was tested in a static simulation environment. The simulation environment was divided into a simple environment and a complex environment, which were respectively a 500*500 map with 10 and 22 obstacles randomly generated. The obstacles were set outside the safe range of the starting point and the target point.

[0074] Figure 5 and 6 The optimal path diagrams for the two models are shown in simple and complex simulation environments, respectively. Each model was run 50 times in each simulation environment. Under the original model, the optimal autonomous navigation path of the robot is as follows: Figure 5 As shown in (a) and 6(a); and under the behavioral decision-making model of the present invention, its optimal autonomous navigation path is as follows: Figure 5 (b) and Figure 6 As shown in (b). The small red dot represents the robot's trajectory, the large red dot represents the robot's starting position, and the green pentagram represents the robot's target position.

[0075] Combination Figure 5 , Figure 6A comprehensive analysis of Table 1 and the results shows that the behavioral decision-making model of this invention has superior performance in static simulation environments. As can be seen from the figures and tables, the behavioral decision-making model of this invention exhibits higher navigation efficiency and path planning capabilities in both simple and complex environments. Specifically, in a simple static environment, the number of steps in the behavioral decision-making model of this invention is reduced from 63 steps in the original model to 59 steps, and the path length is also reduced from 620.9 to 592.7, indicating that the behavioral decision-making model of this invention can complete the target navigation task more efficiently. Furthermore, from the perspective of path distribution, the path of the behavioral decision-making model of this invention is smoother, which not only reduces the time cost during operation but also improves the rationality and efficiency of path planning.

[0076] In complex static environments, the behavioral decision-making model of this invention still maintains superior navigation performance, reducing the number of steps from 67 in the original model to 64, and shortening the path length from 663.1 to 622.5. The path distribution is closer to the obstacles, demonstrating the accuracy and superiority of its path planning. The introduction of an adaptive step size strategy allows the robot to take larger steps in areas with fewer obstacles, reducing unnecessary decision-making and thus improving task execution efficiency. Furthermore, in areas with dense obstacles, the step size can be flexibly reduced, achieving more precise obstacle avoidance and path optimization.

[0077] Table 1. Statistical analysis of the static simulation environment results of the behavioral decision-making model and the original model of this invention. To further verify the performance of the two models, Table 2 shows the statistical results of the movement steps of the two models in 50 experiments under simple and complex environments. As can be seen from the table, the behavioral decision-making model of this invention significantly outperforms the original model. Both models incorporate a hippocampus-based memory consolidation mechanism, enabling the robot to relearn data from online task execution during offline phases, thereby continuously optimizing decision-making strategies and improving the robot's adaptability to different environments.

[0078] Table 2. Results of 50 runs (number of steps) of the behavioral decision-making model of this invention and the original model. Figure 7This diagram illustrates the navigation path of the behavioral decision-making model of this invention after exchanging start and end positions in simple and complex environments. As can be seen from the diagram, the behavioral decision-making model of this invention maintains good behavioral decision-making capabilities in both simple and complex environments after the start and end positions are exchanged. In both environments, the robot can flexibly adapt to the new start and end points, quickly plan a reasonable path, and reach the target location, with a smooth path distribution. This demonstrates that the behavioral decision-making model of this invention has high robustness and adaptability; its performance remains stable regardless of changes in start and end conditions, effectively meeting the navigation requirements in dynamic and complex environments.

[0079] In summary, the behavioral decision-making model of this invention not only achieves significant optimization in terms of navigation steps and path length, but also exhibits superior performance in path smoothness and robustness. Compared to the original model, the behavioral decision-making model of this invention enables the robot to have stronger adaptability and complete navigation tasks more efficiently. At the same time, the behavioral decision-making model of this invention demonstrates good generalization and stability in different scenarios, making it more reliable in practical applications.

[0080] (2) Dynamic Experiment

[0081] The dynamic simulation environment used in the simulation experiment of this invention is as follows: Figure 8 As shown. The experimental environment is 500*500 pixels in size and consists of a robot, a target position, static obstacles, and dynamic obstacles. Red dots represent the robot's starting position, green pentagrams represent the robot's target position, blue dots represent static obstacles, and black dots represent dynamic obstacles, of which 5 move randomly and 2 move along a straight line. At the start of the experiment, the robot and the dynamic obstacles begin moving simultaneously. The experiment ends when the robot reaches the target position or the distance between the robot and the obstacles is less than a given safe distance. Figure 8 In a dynamic simulation environment, 50 navigation experiments were conducted using both the original model and the behavioral decision-making model of this invention. Figure 9 The original model is given. Figure 9 (a) and the behavioral decision-making model of the present invention ( Figure 9 (b) The result of the 50th navigation path.

[0082] from Figure 9As can be seen, both the behavioral decision-making model of this invention and the original model successfully reached the target point. The original model's path was not smooth enough, and its obstacle avoidance actions were rather abrupt, resulting in numerous turns in the navigation path. In contrast, the behavioral decision-making model of this invention, through a flexible step size adjustment mechanism, increases the step size when moving away from obstacles to move towards the target point at a faster speed; while decreasing the step size when approaching obstacles, it achieves more refined decision-making, thereby making the path smoother and improving the flexibility and effectiveness of the obstacle avoidance strategy.

[0083] The presence of dynamic obstacles introduces significant randomness into the robot's navigation task. Furthermore, several evaluation metrics were statistically analyzed for the original model and the behavioral decision-making model of this invention during 50 experiments: the number of times the target location was reached, the number of steps taken upon reaching the target, and the distance to the nearest obstacle during the experiment. The results are as follows: Figure 10 As shown.

[0084] from Figure 10 As shown in the diagram, the blue broken line represents the number of steps the robot took to reach the target position, the red broken line represents the distance between the robot and the nearest obstacle during navigation, and the red dashed line represents the robot's collision distance. When the red broken line is lower than the dashed line, it indicates that the robot collided with an obstacle and failed to reach the target position. (Compare this to the original model...) Figure 10 (a) and the behavioral decision-making model of the present invention ( Figure 10 (b) It can be observed that the behavioral decision-making model of the present invention generally has fewer steps than the original model, and the navigation success rate of the behavioral decision-making model of the present invention is higher than that of the original model. Furthermore, the behavioral decision-making model of the present invention maintains a greater distance from obstacles during navigation, which not only enhances safety but also reflects its excellent path planning capabilities. The fluctuation in the number of steps in the behavioral decision-making model of the present invention is due to the robot reducing its step length to achieve more refined obstacle avoidance decisions when encountering obstacles, thus leading to an increase in the number of steps during obstacle avoidance. Overall, the behavioral decision-making model of the present invention demonstrates good performance in terms of navigation steps, success rate, and obstacle avoidance capability.

[0085] (3) Comparative test

[0086] To further verify the behavioral decision-making capability of the behavioral decision-making model of this invention, it was compared and analyzed with deep reinforcement learning algorithms DQN, SAC, and the cerebellar regulatory developmental network (MDN) containing dopamine and serotonin neurotransmitters. In the experiment of this invention, 10 different complex environments were randomly generated, each with a map size of 500*500, and 25 static obstacles were randomly distributed within them. The positions of the obstacles were all set outside the safe range of the starting point and the target point.

[0087] In 10 different environments, four models were used for 50 interactive learning sessions each, and the minimum number of steps required for each of the 50 interactive learning sessions in each environment was calculated. The experimental results are as follows: Figure 11 As shown in the figure, there are significant differences in the minimum number of steps among the four models. The behavioral decision-making model of this invention performs better in most environments, with a generally low minimum number of steps. In contrast, the MDN model performs the worst in most environments, with a high minimum number of steps, indicating its weak path planning ability in complex environments. The DQN and SAC models perform similarly, but are inferior to the behavioral decision-making model of this invention in most environments. To more accurately compare the performance of the four models, the various indicators shown in Table 3 were statistically analyzed.

[0088] Table 3. Statistical data (mean) of the four models in interactive learning. Model Number of successful navigation environments Minimum number of steps Shortest movement distance MDN 10 69.3 671.5 SAC 10 69.3 625.9 DQN 10 60.9 602.3 This invention model 10 57.2 585.1 As shown in Table 3, all four models successfully located the target position in 10 different environments. However, the behavior decision model of this invention performed best in terms of the fewest steps. Regarding the shortest distance, the MDN model had the longest distance, followed by the SAC model. Although the DQN model had significantly more steps than the behavior decision model of this invention, its distance was relatively close. This is because the behavior decision model of this invention employs an adaptive step size strategy. When moving away from obstacles, the model increases the step size, reducing unnecessary decisions and thus improving movement efficiency; when approaching obstacles, the step size is reduced to avoid collisions. This results in a relatively fewer decision-making steps for the behavior decision model of this invention, without significantly affecting the actual distance traveled. This dynamic adjustment strategy plays a crucial role in balancing path planning efficiency and collision avoidance, enabling the robot to improve navigation efficiency while ensuring safety.

[0089] (VII) Physical Experiments and Result Analysis

[0090] The effectiveness of the proposed model was verified through simulation experiments. To further evaluate its behavioral decision-making capabilities in real-world environments, physical experiments were designed and conducted. The equipment used in the experiments included two Rikirobot robots and two laptops. The ROS-based Rikirobot robot was used as the verification platform, and tests were conducted in both static and dynamic environments. By observing the robot's performance in different scenarios, the feasibility of the model was further verified.

[0091] (1) Static physical experiment

[0092] The experimental environment for static physical experiments, such as Figure 12As shown, the experimental environment is approximately 5m x 4m in size and contains 12 static obstacles of varying sizes and shapes. The robot's starting point is at the lower left corner, and the target point is at the upper right corner. The robot's initial position coordinates are (0, 0), and the target position coordinates are (3.7, 2.4). To ensure the robot's safety, the experiment is considered complete and the robot stops moving when the distance between the robot and the target position or the nearest obstacle is less than 0.1m.

[0093] In a static physical environment, the MDN, SAC, and DQN algorithms from the simulation experiment are compared with the behavioral decision-making model of this invention. To make the experiment observable, this experiment is illustrated using the Rviz visual interface. The red and black dots in the figure represent the robot's real-time scanning and previously recorded obstacle and boundary positions, respectively, and the black curve represents the robot's motion trajectory. Figure 13 The above four models represent the motion trajectories of the robot in the Rviz visual interface.

[0094] from Figure 13 As can be seen, among the four models mentioned above, MDN has the worst behavior decision-making performance and the longest navigation path. While DQN, SAC, and the behavior decision-making model of this invention are similar in path shape, the behavior decision-making model of this invention outperforms the DQN and SAC algorithms in terms of step length. The above figure only provides a visual observation of the results of a single experiment and cannot comprehensively demonstrate the overall performance of each model. Therefore, multiple data indicators were recorded for each of the four models after twenty trials to more comprehensively evaluate their performance. Specific data are shown in Table 4.

[0095] Table 4. Statistical analysis of the results of the four models in a static physical environment. Model Average number of steps Minimum number of steps Number of successful navigations MDN 25.67 24 15 SAC 24.07 23 15 DQN 22.94 22 18 This invention model 20.74 20 19 Table 4 records three evaluation metrics for the robot under four models: average number of steps, minimum number of steps, and number of successful navigation attempts. The data in the table shows that different models exhibit varying navigation performance in static physical environments. The behavior decision model of this invention outperforms the other models, with both its average and minimum number of steps being lower than the other three models. Furthermore, the behavior decision model of this invention has the highest navigation success rate. In contrast, the MDN model performs the worst among the four models, with a relatively long navigation path and a low navigation success rate. The SAC and DQN models outperform MDN, but there is still room for improvement. The navigation success rate of DQN is slightly better than SAC, but there is no significant difference between the two in path planning. In summary, the data in the table clearly demonstrates that the behavior decision model of this invention has the best behavior decision-making effect, providing an efficient and stable feasible solution for robot navigation.

[0096] (2) Dynamic physical experiments

[0097] To further verify the performance of the behavioral decision-making model of this invention, a dynamic physical experiment was designed, setting up ten static obstacles and one dynamic obstacle in the environment. In this experiment, the dynamic obstacle started moving from two different initial positions: in dynamic environment one, the initial position of the dynamic obstacle was (2.8, 0); in dynamic environment two, the initial position of the dynamic obstacle was (3.9, 2.2). The robot's starting and ending positions were the same as in the static physical experiment. The two different dynamic environments are as follows: Figure 14 As shown in the figure, the robot's start and end positions and the initial positions of the dynamic obstacles are marked.

[0098] The Rviz visualizations of robots and dynamic obstacles in different dynamic environments are as follows: Figure 15 and Figure 16 As shown, the trajectories of the robot and dynamic obstacles in Rviz are represented by black curves, and their operation processes are displayed independently in Rviz. Figure 15 The robot's trajectory was demonstrated in different dynamic environments, showing that the robot was able to avoid obstacles and reach the target location in dynamic environments. Figure 16 This reflects the trajectory of dynamic obstacles in two different dynamic environments.

[0099] To more clearly illustrate the movement of the robot and dynamic obstacles, radar scanning was used to save relevant map information, and the coordinates of the robot and dynamic obstacles during their movement were recorded. Finally, the obtained coordinates of the robot and dynamic obstacles, along with the map information, were plotted on the same graph, as shown below. Figure 17 and Figure 18 As shown in the diagram, blue represents the movement trajectory of the mobile robot, and black represents the movement trajectory of dynamic obstacles.

[0100] from Figure 17 and Figure 18 As can be seen, the robot and the dynamic obstacle never overlapped at any given moment, indicating that the robot could effectively avoid both static and dynamic obstacles and successfully reach the target position in both dynamic environments. Combined with... Figure 17 and Figure 15 (a) and Figure 16 (a) It can be observed that in the dynamic physical environment 1, the robot encountered a dynamic obstacle. Originally, the robot should have moved to the right, but because the dynamic obstacle was to the right of the robot, in order to avoid the dynamic obstacle, the robot turned to the upper right and eventually successfully reached the target position.

[0101] Similarly, combining Figure 18 and Figure 15 (b) and Figure 16(b) It can be observed that in dynamic environment two, the robot should move towards the upper right target direction before encountering the dynamic obstacle. However, due to the presence of the dynamic obstacle, the robot turns to the lower right, temporarily moving away from the target position to avoid the dynamic obstacle, and then adjusts its path to reach the target position. This illustrates the flexibility of the behavioral decision-making model of the present invention in complex dynamic environments, and its ability to adjust decisions in real time according to the external environment, demonstrating the superiority of the behavioral decision-making model of the present invention.

[0102] This invention significantly improves the decision-making efficiency, long-term learning ability, and dynamic adaptability of robots in complex environments by simulating multi-brain region collaboration mechanisms and combining deep neural networks, neuronal remodeling, and adaptive step size strategies. Simulation and physical experiments show that this model outperforms existing models in key indicators such as number of steps, path length, and success rate, providing an efficient and interpretable brain-like solution for autonomous navigation of intelligent robots.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention in any way. All equivalent transformations or modifications made in accordance with the essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A brain-like behavioral decision-making model for robots based on multi-brain region collaboration, characterized in that, include: The basal ganglion module employs a DQN-simulated reinforcement learning mechanism to handle action decisions in a high-dimensional state space. The DQN includes an evaluation network, a target network, and an experience replay pool. The evaluation network calculates the state-action value function in real time. The target network periodically synchronizes parameters through a delayed update mechanism to stabilize Q-value estimation. The outputs of the DQN's target network and the cerebellar regulatory development network are fused through dynamically weighted parameters to generate the final Q-value. The cerebellum regulates the developmental network, simulates the supervised learning mechanism, and dynamically resets the age of neurons through a neuronal remodeling strategy to maintain long-term learning ability. The hippocampus module simulates the human memory replay mechanism, saves data from online tasks, and retrains the cerebellar regulatory development network offline to improve the model's learning ability and environmental adaptability. The adaptive step size strategy module dynamically adjusts the robot's step size based on the distance between the robot and the nearest obstacle and the rotation angle during movement. The hybrid integration module combines correlation indicators and trajectory evaluation indicators to dynamically regulate the output of the cerebellum and basal ganglia, achieving effective fusion of the output of the cerebellum and basal ganglia to generate the final behavioral decision action.

2. The robot brain-like behavior decision-making model based on multi-brain region collaboration according to claim 1, characterized in that, The dynamic ∈-greedy policy of DQN satisfies the following formula: ∈ t ＝∈ min +(∈ max -∈ min )·e -λt ； Where, ∈ t Represents the exploration rate at time t, ∈ max and ∈ min Let be the maximum and minimum values ​​of the exploration rate ∈, respectively, λ be the second decay factor, and t be the time step.

3. The robot brain-like behavior decision-making model based on multi-brain region collaboration according to claim 1, characterized in that, The comprehensive reward formula for the reward value r of the reward function of DQN is: r=r b +ω1r1+ω2r2; Where ω1 and ω2 are reward coefficients, r b The base reward function is set to a positive value when the target is reached and a negative value when a collision occurs; r1 is the obstacle distance reward value, which increases when moving away from the obstacle and decreases when moving closer; r2 is the target approach reward value, which increases when approaching the target and decreases when moving away.

4. The robot brain-like behavior decision-making model based on multi-brain region collaboration according to claim 1, characterized in that, The neuronal remodeling strategy of the cerebellar regulatory developmental network is specifically as follows: When the number of activations of a neuron exceeds a preset threshold, its age is reset to its initial value; Cellular reprogramming technology is used to simulate the regeneration process of biological neurons, and the learning ability of neurons is restored through the following formula: Where, n max This represents the set neuron age threshold, n i and n i ′ represents the age of the neuron before and after remodeling.

5. The robot brain-like behavior decision-making model based on multi-brain region collaboration according to claim 1, characterized in that, In the adaptive step size strategy module, the robot's step size is closely related to the distance to the nearest obstacle and the rotation angle during movement, as follows: When the distance d between the robot and the nearest obstacle satisfies d > d², the robot's step size L is expressed as: When the distance d between the robot and the nearest obstacle satisfies d1 < d < d2, the robot's step size L can be expressed as: When the distance d between the robot and the nearest obstacle satisfies d < d1, the robot's step size L can be expressed as: Where L represents the final calculated adaptive motion step size, which is determined by d and Δθ. max and l min d1 and d2 represent the maximum and minimum travel distances of the robot, respectively. d1 and d2 represent preset distance thresholds used to delineate the proximity of obstacles. Δθ represents the real-time rotation angle of the robot, reflecting the turning amplitude. θ1 and θ2 represent preset angle thresholds used to classify the levels of turning amplitude: Δθ < θ1 indicates a small-angle turn, θ1 < Δθ < θ2 indicates a medium-angle turn, and Δθ > θ2 indicates a large-angle turn.

6. The robot brain-like behavior decision-making model based on multi-brain region collaboration according to claim 1, characterized in that, The action fusion method of the hybrid integration module is as follows: (1) Calculate the trajectory evaluation indices G1 and G2 for cerebellar output action a1 and basal ganglia output action a2; (2) The final action is generated by weighting the normalized correlation coefficients λ1 and λ2. Where S = λ1G1 + λ2G2.

7. A robot behavior decision-making method based on multi-brain region collaboration, characterized in that, Includes the following steps: (1) Initialize the robot state, including position, target point, obstacle information, etc.; (2) Based on the current state, calculate the respective action outputs a1 and a2 through the basal ganglia module and the cerebellum module respectively; (3) Dynamically adjust the robot's step size according to the adaptive step size strategy module; (4) By integrating correlation indicators and trajectory evaluation, the final action is generated by integrating a1 and a2; (5) Perform actions and save online data in the hippocampus module to retrain the cerebellar regulatory development network and remodel neurons in an offline state.

Images