Behavior decision-making method for simulating hippocampus-prefrontal lobe memory planning playback mechanism

Abstract

The invention relates to a behavior decision-making method for simulating a hippocampus-prefrontal lobe memory planning playback mechanism in the technical field of artificial intelligence, and the method comprises the steps: constructing the closed-loop interaction of a hippocampus planning playback network and a prefrontal lobe strategy evaluation network, and fusing the meta-reinforcement learning and a model-based strategy optimization technology; the bionic neural mechanism realizes the unification of dynamic planning and strategy optimization, and realizes the dynamic coupling of experience playback and prospective simulation in the brain-like decision process, so that the robot can learn in planning. The method comprises the following steps: acquiring environment basic data by using a model element-free reinforcement learning method, planning a certain number of steps by using a model by using a strategy optimization method based on model element reinforcement learning and taking the current basic data as a planning starting point, and training the model by using a planning result. The problems that an existing robot algorithm is limited by the complex degree of an actual environment, sample collection is difficult and the like are solved, the learning ability is effectively improved, and therefore the application requirements of the mobile robot in diversified complex scenes are met.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a behavioral decision-making method that simulates the hippocampus-prefrontal lobe memory planning and playback mechanism. Background Art

[0002] With the deep integration of robotics and artificial intelligence, mobile robots, with their autonomous mobility and intelligent operation capabilities, have been widely used in various fields, including industrial manufacturing, logistics and transportation, and healthcare. The core foundation for mobile robots to successfully complete various complex tasks lies in their accurate perception and cognition of the dynamic operating environment. However, in practical application scenarios, mobile robots often lack prior knowledge of their operating environment, which is often complex, changeable, and highly uncertain. For example, they may encounter unexpected obstacles and changes in the environment layout. Therefore, how to make efficient behavioral decisions has become a key issue that must be addressed.

[0003] For mobile robots, efficient, dynamic, and flexible behavioral decision-making is a crucial prerequisite for achieving various tasks. Researchers have long proposed numerous decision-making methods, including rule-based, probability-based, optimization-based, statistical learning, deep learning, and reinforcement learning. However, these methods all have drawbacks. For example, rule-based methods typically make decisions by setting a set of predetermined rules. While they can achieve good results for specific tasks, they lack adaptability and flexibility. Statistical learning-based methods have poor knowledge acquisition, learning, and reasoning capabilities, making them difficult to apply to the complex decision-making tasks of mobile robots in dynamic scenarios. Reinforcement learning methods use reward mechanisms to learn optimal strategies and can adapt to complex environments, but they suffer from low learning efficiency, especially when faced with high-dimensional state spaces and long-duration tasks.

[0004] In recent years, with the advancement of neuroscience, brain science, and artificial intelligence, researchers have discovered that the human brain possesses an extremely sophisticated decision-making system. Studies have shown that the hippocampus is a key brain region involved in episodic memory and spatial function. Furthermore, under uncertain conditions, the hippocampus plays a crucial role in value-based decision-making. Furthermore, various regions within the human brain collaborate to complete a range of decision-making tasks, such as the complementary task representations between the hippocampus and the prefrontal cortex. Furthermore, the collaboration or regulatory functions between various regions within the human brain are regulated by different neurotransmitters within the brain. For example, dopamine receptor activation regulates reward expectation signals in the cognitive control process of primate prefrontal neurons, and serotonin regulates the learning rate of the human brain under uncertain conditions.

[0005] Therefore, a growing number of researchers are considering how to control robots' decision-making in complex tasks by simulating the structure and function of biological brains, especially involving the collaboration of multiple important brain regions such as the hippocampus, cerebellum, basal ganglia, and thalamus. Some researchers have simulated the learning principles of the basal ganglia and cerebellum to achieve flexible behavioral decision-making in dynamic environments for mobile robots based on cerebellar supervised learning and basal ganglia reinforcement learning, improving their ability to adapt to dynamic environments. By designing a curiosity index and adjusting the exploration-exploitation weight in basal ganglia reinforcement learning, their adaptive capabilities are further enhanced. However, the learning capabilities of these brain-inspired algorithms are limited by the complexity of the actual environment or the difficulty of collecting samples.

[0006] Based on this, the field urgently needs a new behavioral decision-making method that can overcome the limitations of existing robot algorithms due to the complexity of the actual environment and the difficulty of sample collection, and effectively improve the learning ability to meet the application needs of mobile robots in diverse and complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems that existing robot algorithms are limited by the complexity of the actual environment, or by the difficulty of sample collection and limited learning ability. Inspired by the brain's hippocampus-prefrontal lobe collaborative memory planning and playback mechanism, a behavioral decision-making method that simulates the hippocampus-prefrontal lobe memory planning and playback mechanism is designed.

[0008] In order to achieve the above-mentioned object of the invention, the behavioral decision-making method proposed in the present invention that simulates the hippocampus-prefrontal lobe memory planning playback mechanism includes the following steps: S1. Design a memory planning replay model driven by experience value and environmental structure. In the memory replay phase of the memory planning replay model, a memory planning network centered on the hippocampus is constructed by simulating the neural mechanism of the human brain's forward-looking prediction based on long-term accumulated experience, which is referred to as the hippocampal planning replay network. In the planning phase of the memory planning replay model, a strategy evaluation network is constructed that includes a multi-layer attention mechanism that simulates the hierarchical structure of the prefrontal lobe, which is referred to as the prefrontal strategy evaluation network. This enables the memory planning replay model to simulate the collaborative planning mechanism of the hippocampal planning replay network and the prefrontal strategy evaluation network. S2. The memory planning replay model is integrated with the meta-reinforcement learning framework and the model-based policy optimization technology to enhance the memory replay and policy optimization process in reinforcement learning, and the unification of dynamic planning and policy optimization is achieved through the bionic neural mechanism, as well as the dynamic coupling of experience replay and forward-looking simulation in the brain-like decision-making process, so that the memory planning replay model can learn in "planning".

[0009] Preferably, in step S1, the memory planning replay model uses a model-free meta-reinforcement learning method to obtain basic environmental data. It then utilizes a policy optimization method based on model-based meta-reinforcement learning, using the current basic data as the starting point for planning and performing a certain number of planning steps. Finally, the planning results are used to train the memory planning replay model. By designing a reasonable planning depth and balancing the model's cumulative error after increasing planning depth with the long-term planning benefits, the robot can improve its learning ability while reducing the actual sample size.

[0010] Preferably, in step S1, the hippocampal planning replay network predicts future states step by step and performs multi-step simulation planning, thereby effectively breaking through the path dependence of traditional reinforcement learning when relying on immediate experience sampling.

[0011] Preferably, in step S2, a meta-reinforcement learning framework is adopted to realize closed-loop interaction between the hippocampal planning replay network and the prefrontal strategy evaluation network through an inner and outer loop double-layer working structure.

[0012] Preferably, in step S2, during the closed-loop interaction between the hippocampal planning replay network and the prefrontal policy evaluation network, the hippocampal planning replay network generates possible future scenarios by replaying historical experiences, while the prefrontal policy evaluation network is responsible for evaluating the policy effects under these scenarios, thereby enabling the intelligent agent to build and adjust the environmental model in real time in a constantly updated environment, and learn the optimal decision-making strategy based on these environmental models.

[0013] Preferably, in step S2, based on the given current state and position, the memory planning replay model constructs a corresponding environment model and learns a corresponding strategy through the outer loop structure of the meta-reinforcement learning framework, gradually predicts the future state to perform multi-step simulation planning, and enhances the effectiveness of the multi-step simulation planning through model-based policy optimization technology until the expected goal is achieved or the depth limit of the planning is reached.

[0014] Preferably, when the behavior decision-making method is applied to the behavior decision-making of a robot, the working process of the robot includes: 1) Initial robot exploration of the environment: The robot collects real-time data from the environment through various sensors to form a cognition of the current situation. The robot then encodes the perceived tasks and situations into memory units to form an initial environmental model. 2) The robot plans future behavioral decisions: The robot stores a hippocampal planning replay network based on interactive learning with an environmental model. This network predicts future states and actions based on existing memory units and the initial environmental model. The prefrontal policy evaluation network integrates the environmental model, memory units, and goal settings to assess the expected outcome of each action. The robot then makes appropriate decisions based on this assessment. 3) Robot execution decision and strategy optimization: The robot starts executing the corresponding action based on the optimal decision generated by the hippocampal planning replay network and makes necessary adjustments based on real-time feedback information. At the same time, the robot feeds back the success and failure experience of the task into the memory unit to optimize the environmental model and decision-making strategy, further improving the success rate of task completion.

[0015] Ultimately, through long-term interaction and learning with the environment, the robot gradually develops a more accurate and flexible behavioral decision-making model. Through continuous exploration, memory replay, planning, and decision-making optimization, this model enables the robot to make efficient and reasonable decisions in an ever-changing environment, autonomously complete tasks, and achieve long-term self-evolution.

[0016] The above work process includes the following in more detail: First, the robot needs to conduct preliminary exploration of the task environment in order to form the initial memory unit and environment model. In this step, the robot first needs to initialize the basic input parameters, including the experience pool capacity. N , maximum number of iterations T , maximum planning depth L , the maximum number of training rounds E , playback weight α , meta-reinforcement learning strategy and environmental models The experience pool is divided into a real experience pool that stores the experience of interacting with the real environment and a virtual experience pool that stores experience gained from model planning , each of which accounts for the experience pool capacity After that, the real experience pool, virtual experience pool, meta-reinforcement learning strategy and environment model are initialized.

[0017] Meta-reinforcement learning strategy Adopts a dual-stream architecture, including exploration strategy and utilization strategies , the formula is as follows: (1) in, are the meta-reinforcement learning policy parameters, Used to adjust the dynamic weights of the exploration strategy and the utilization strategy. This invention uses a linear attenuation mechanism to balance the exploration and utilization of the environment. This mechanism focuses on environmental exploration in the early stages of the experiment and on environmental utilization in the later stages of the experiment. The formula is as follows: (2) in, Set the initial weights for the start of training, Minimum weight, nis the current training round number.

[0018] At the beginning of each training round, the robot uses the meta-reinforcement learning policy Interact with the real environment, namely: (3) The collected trajectory data is encoded by the hippocampus module to memory units Stored in the real experience pool in the form of Among them, It is the robot that performs the action The new state observed later is is the reward the agent receives after taking this action.

[0019] At the same time, the initial environment model is trained based on the subsequent memory units ,The environment model structure adopts a two-branch neural network, which divides the environment model into two main parts, the state transfer model and the reward prediction model, and its formula is as follows: (4) (5) in, is the environment model parameter. After that, when the robot enters the planning state, it will Uniformly randomly sample a state from , and use the sampled state as the initial state of planning .by As the initial state, use the meta-reinforcement learning strategy in the constructed environment model conduct The robot needs to encode the planning results into virtual memory units , and then stored in the virtual experience pool First, in this process, it is necessary to set a suitable experience value reward function for the robot to maximize the expected reward, as shown below: (6) (7) in, is the reward discount factor. The action sequence is obtained by maximizing the experience value reward Guide the robot to perform tasks and observe the real state rewards ( , , ), and store it in Secondly, during the planning process, after each iteration, the number of iterations Add 1 until the maximum planning depth is reached LOr the robot position reaches the target position.

[0020] In actual situations, as the number of deduction steps of the environment model increases, the accumulated compound error of the model will increase rapidly, making the results obtained by the environment model very unreliable. At the same time, it is necessary to weigh the negative effects of the increased error of the model after the number of deduction steps increases and the positive effects of the training strategy being better after the number of steps increases. The present invention sets the relevant loss function To ensure the minimization of expected losses, as shown below: (8) in, and They represent the state planning loss and planning reward loss respectively, and their formulas are as follows: (9) (10) Optimize the environment model according to the loss function , to ensure the accuracy of planning results in subsequent tasks, the formula is as follows: (11) in, is the learning rate of the environment model. and virtual experience pool Perform mixed sampling and calculate strategy loss , update the meta-reinforcement learning strategy, the formula is as follows: (12) (13) (14) in, is the mixed sampling weight, is the adaptive learning rate of the meta-reinforcement learning policy network, is the advantage function, and its formula is as follows: (15) (16) in, is the action-value function, is the state value function, is the learning rate of the initially assigned meta-RL policy network.

[0021] Compared with the existing technology, the present invention has the following beneficial effects: (1) In the memory replay phase, the present invention successfully constructs a memory planning network with the hippocampus as the core by simulating the neural mechanism of the human brain to make forward-looking predictions based on long-term accumulated experience. This network effectively breaks through the path dependence of traditional reinforcement learning when it relies on immediate experience sampling by gradually predicting future states and performing multi-step simulation planning. This mechanism significantly reduces the dependence on real-world interaction data, improves the robot's adaptability and robustness in uncertain environments, and enhances the robot's learning ability in various complex scenarios; (2) The present invention adopts a meta-reinforcement learning framework, which realizes a closed-loop interaction between the hippocampal planning replay network and the prefrontal strategy evaluation network through an inner and outer loop double-layer working structure. In this process, the hippocampus generates possible future scenarios by replaying historical experiences, while the prefrontal cortex is responsible for evaluating the effectiveness of strategies in these scenarios. Through this closed-loop interaction, the intelligent agent can build and adjust the environmental model in real time in a constantly updated environment and learn the optimal decision-making strategy based on these models. This mechanism embodies the dynamic coupling of experience replay and forward-looking simulation in the brain-like decision-making process, enabling the robot to quickly adapt to the changing environment and make more efficient decisions in changing situations.

[0022] (3) The present invention combines meta-reinforcement learning with model-based policy optimization technology to achieve a seamless integration of dynamic programming and policy optimization. In this way, the present invention can not only predict the optimal decision-making plan under the current state through dynamic programming, but also combine policy optimization technology to further improve the long-term benefits of decision-making. In order to adapt to this complex decision-making process, the present invention adopts a dual-stream neural network architecture, which enables the intelligent agent to gradually shift from focusing on exploration to focusing on the effective use of acquired information during the execution of tasks. Through this gradual optimization mechanism, the intelligent agent can not only make short-term decisions quickly based on the current state and experience, but also effectively predict and plan multiple possible scenarios in the future and adopt the most suitable action strategy, thereby optimizing the decision-making effect in a variety of complex scenarios and significantly improving the efficiency of task execution.

[0023] In summary, the behavioral decision-making method proposed in the present invention overcomes the problems that existing robot algorithms are limited by the complexity of the actual environment and the difficulty of sample collection, effectively improves the learning ability, and thus meets the application needs of mobile robots in diverse and complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the principles of the present invention and the technical solutions in its implementation, the following further describes the technical solutions of the present invention using the accompanying drawings. The following drawings are only some implementation examples of the present invention. For those skilled in the art, other technical solutions can be obtained based on the following drawings without inventive work: Figure 1 2 is a schematic diagram of a behavior decision-making system framework based on a memory planning and playback mechanism in an embodiment of the present invention; Figure 2 Schematic diagram of a memory planning playback network model in an embodiment of the present invention; Figure 3 It is a simulation experiment environment of the robot in the embodiment of the present invention; Figure 4 Schematic diagram of the cumulative rewards obtained by robots with different L values ​​in the first 200 rounds in the simulation training environment according to an embodiment of the present invention; Figure 5 It is a demonstration of simulation experiment paths using robots of different models in the embodiments of the present invention; Figure 6 Schematic diagram of the rewards changes in the first 200 rounds of a simulation experiment using robots of different models in an embodiment of the present invention; Figure 7 It is a static physical experimental environment of the robot in the embodiment of the present invention; Figure 8 The figure shows the comparison of the number of movement steps of robots using different models during 30 runs in the embodiment of the present invention. Figure 9 This is a demonstration of comparative experimental paths using robots of different models in an embodiment of the present invention; Figure 10 It is a dynamic physical experimental environment of the robot in the embodiment of the present invention; Figure 11 In the embodiment of the present invention, a successful path of a robot of five models is respectively used to display; Figure 12 Schematic diagram of the obstacle avoidance process of a robot using the MPR model in an experiment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0026] Example The key to this invention is the design of a Memory Planning Replay (MPR) model driven by experience value and environmental structure. By constructing a closed-loop interaction between the hippocampal planning replay network and the prefrontal strategy evaluation network, integrating meta-reinforcement learning with model-based policy optimization techniques, and using bionic neural mechanisms to achieve the unification of dynamic planning and policy optimization, the model achieves the dynamic coupling of experience replay and forward-looking simulation in the brain-like decision-making process, enabling the robot to learn through "planning." The model uses model-free meta-reinforcement learning to obtain basic environmental data. Then, using a model-based meta-reinforcement learning policy optimization method, the model uses the current basic data as the starting point for planning, performs planning for a certain number of steps, and uses the planning results to train the model. At the same time, a reasonable planning depth is designed to balance the model error accumulation after increasing the planning depth with the long-term planning benefits, enabling the robot to improve its learning ability while reducing the actual sample size.

[0027] Specifically, this embodiment proposes a behavioral decision-making method that simulates the hippocampus-prefrontal lobe memory planning and playback mechanism proposed by the present invention, including the following steps: S1. Design a memory planning replay model driven by experience value and environmental structure. In the memory replay phase of the memory planning replay model, a memory planning network centered on the hippocampus is constructed by simulating the neural mechanism of the human brain's forward-looking prediction based on long-term accumulated experience, which is referred to as the hippocampal planning replay network. In the planning phase of the memory planning replay model, a strategy evaluation network is constructed that includes a multi-layer attention mechanism that simulates the hierarchical structure of the prefrontal lobe, which is referred to as the prefrontal strategy evaluation network. This enables the memory planning replay model to simulate the collaborative planning mechanism of the hippocampal planning replay network and the prefrontal strategy evaluation network. S2. The memory planning replay model is integrated with the meta-reinforcement learning framework and the model-based policy optimization technology to enhance the memory replay and policy optimization process in reinforcement learning, and the unification of dynamic planning and policy optimization is achieved through the bionic neural mechanism, as well as the dynamic coupling of experience replay and forward-looking simulation in the brain-like decision-making process, so that the memory planning replay model can learn in "planning".

[0028] Following the above embodiment, in step S1, the memory planning replay model obtains the basic data of the environment using the method of model-free meta-reinforcement learning, and then uses the policy optimization method based on model meta-reinforcement learning to use the current basic data as the starting point of planning and perform a certain number of steps of planning. Finally, the planning results are used to train the memory planning replay model. By designing a reasonable planning depth and balancing the model cumulative error after the planning depth increases with the long-term planning benefits, the robot can improve its learning ability while reducing the actual sample size. The hippocampal planning replay network effectively breaks through the path dependence of traditional reinforcement learning when it relies on immediate experience sampling by gradually predicting future states and performing multi-step simulation planning.

[0029] In step S2, a meta-reinforcement learning framework is employed to implement a closed-loop interaction between the hippocampal planning replay network and the prefrontal policy evaluation network through a dual-layer working structure of inner and outer loops. During this closed-loop interaction, the hippocampal planning replay network generates possible future scenarios by replaying historical experience, while the prefrontal policy evaluation network is responsible for evaluating the effectiveness of strategies in these scenarios. This enables the agent to construct and adjust environmental models in real time within a continuously updated environment and learn optimal decision-making strategies based on these environmental models.

[0030] Specifically, in step S2, based on the given current state and position, the memory planning replay model constructs the corresponding environment model and learns the corresponding strategy through the outer loop structure of the meta-reinforcement learning framework, gradually predicts the future state to perform multi-step simulation planning, and enhances the effectiveness of multi-step simulation planning through model-based policy optimization technology until the expected goal is achieved or the planning depth limit is reached. The model diagram is shown in the figure below. Figure 2 shown.

[0031] In this embodiment, based on the above hippocampus-prefrontal cortex collaborative planning mechanism, the following is designed: Figure 1 The following figure shows the framework of a mobile robot's behavior decision-making system in an unknown environment based on the memory planning replay mechanism. The workflow of this model can be described as follows: 1) Initial robot exploration of the environment: The robot collects real-time data from the environment through various sensors to form a cognition of the current situation. The robot then encodes the perceived tasks and situations into memory units to form an initial environmental model. 2) The robot plans future behavioral decisions: The robot stores a hippocampal planning replay network based on interactive learning with an environmental model. This network predicts future states and actions based on existing memory units and the initial environmental model. The prefrontal policy evaluation network integrates the environmental model, memory units, and goal settings to assess the expected outcome of each action. The robot then makes appropriate decisions based on this assessment. 3) Robot execution decision and strategy optimization: The robot starts executing the corresponding action based on the optimal decision generated by the hippocampal planning replay network and makes necessary adjustments based on real-time feedback information. At the same time, the robot feeds back the success and failure experience of the task into the memory unit to optimize the environmental model and decision-making strategy, further improving the success rate of task completion.

[0032] Ultimately, through long-term interaction and learning with the environment, the robot gradually develops a more accurate and flexible behavioral decision-making model. Through continuous exploration, memory replay, planning, and decision-making optimization, this model enables the robot to make efficient and reasonable decisions in an ever-changing environment, autonomously complete tasks, and achieve long-term self-evolution.

[0033] The above work process includes the following in more detail: First, the robot needs to conduct preliminary exploration of the task environment in order to form the initial memory unit and environment model. In this step, the robot first needs to initialize the basic input parameters, including the experience pool capacity. N , maximum number of iterations T , maximum planning depth L , the maximum number of training rounds E , playback weight α , meta-reinforcement learning strategy and environmental models The experience pool is divided into a real experience pool that stores the experience of interacting with the real environment and a virtual experience pool that stores experience gained from model planning , each of which accounts for the experience pool capacity After that, the real experience pool, virtual experience pool, meta-reinforcement learning strategy and environment model are initialized.

[0034] Meta-reinforcement learning strategy Adopts a dual-stream architecture, including exploration strategy and utilization strategies , the formula is as follows: (1) in, are the meta-reinforcement learning policy parameters, Used to adjust the dynamic weights of the exploration strategy and the utilization strategy. This invention uses a linear attenuation mechanism to balance the exploration and utilization of the environment. This mechanism focuses on environmental exploration in the early stages of the experiment and on environmental utilization in the later stages of the experiment. The formula is as follows: (2) in, Set the initial weights for the start of training, Minimum weight, nis the current training round number.

[0035] At the beginning of each training round, the robot uses the meta-reinforcement learning policy Interact with the real environment, namely: (3) The collected trajectory data is encoded by the hippocampus module to memory units Stored in the real experience pool in the form of Among them, It is the robot that performs the action The new state observed later is is the reward the agent receives after taking this action.

[0036] At the same time, the initial environment model is trained based on the subsequent memory units ,The environment model structure adopts a two-branch neural network, which divides the environment model into two main parts, the state transfer model and the reward prediction model, and its formula is as follows: (4) (5) in, is the environment model parameter. After that, when the robot enters the planning state, it will Uniformly randomly sample a state from , and use the sampled state as the initial state of planning .by As the initial state, use the meta-reinforcement learning strategy in the constructed environment model conduct The robot needs to encode the planning results into virtual memory units , and then stored in the virtual experience pool First, in this process, it is necessary to set a suitable experience value reward function for the robot to maximize the expected reward, as shown below: (6) (7) in, is the reward discount factor. The action sequence is obtained by maximizing the experience value reward Guide the robot to perform tasks and observe the real state rewards ( , , ), and store it in Secondly, during the planning process, after each iteration, the number of iterations Add 1 until the maximum planning depth is reached LOr the robot position reaches the target position.

[0037] In actual situations, as the number of deduction steps of the environment model increases, the accumulated compound error of the model will increase rapidly, making the results obtained by the environment model very unreliable. At the same time, it is necessary to weigh the negative effects of the increased error of the model after the number of deduction steps increases and the positive effects of the training strategy being better after the number of steps increases. The present invention sets the relevant loss function To ensure the minimization of expected losses, as shown below: (8) in, and They represent the state planning loss and planning reward loss respectively, and their formulas are as follows: (9) (10) Optimize the environment model according to the loss function , to ensure the accuracy of planning results in subsequent tasks, the formula is as follows: (11) in, is the learning rate of the environment model. and virtual experience pool Perform mixed sampling and calculate strategy loss , update the meta-reinforcement learning strategy, the formula is as follows: (12) (13) (14) in, is the mixed sampling weight, is the adaptive learning rate of the meta-reinforcement learning policy network, is the advantage function, and its formula is as follows: (15) (16) in, is the action-value function, is the state value function, is the learning rate of the initial meta-reinforcement learning policy network. The algorithm flow of this model is shown in Table 1.

[0038] Table 1 Algorithm of the memory planning replay model driven by experience value and environment structure (MPR algorithm)

[0039] In order to enable researchers and users in this technical field to better understand the solution of the present invention, the simulation results of the solution will be analyzed below, which is also a verification of the specific application scenario of the present invention. Based on the examples in the present invention, all other examples obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] Experimental verification of the scheme 1. Simulation experiment design and result analysis 1.1 Environment Setup Set as Figure 3 The 500 × 500 coordinate environment shown in the figure has nine irregular obstacles of varying shapes and sizes randomly placed within it. The robot's initial position (yellow solid circle) is set to (50, 50), and the target point (green solid circle) is set to (450, 450). There are no obstacles within the robot's safe range of the starting position. Because the obstacles are randomly positioned, the robot has no prior knowledge of the environment, effectively navigating an unknown environment. To validate the performance of the MPR algorithm, it is compared with the PER, HER, and MBPO algorithms, analyzing their learning and decision-making capabilities in unknown environments.

[0041] When using different models, the robot is at the starting position. When exploring the environment or performing tasks, the robot has 36 movement directions (360 / 10), and each direction moves forward with a fixed step length. When the robot reaches the target position or touches an obstacle, the movement ends and is counted as a round. In the experiment, the robots controlled by different models were placed in the Figure 3 Repeat the movement for 500 rounds in the simulation environment shown. This process is referred to as the robot training process. During this process, the robot training results and data are recorded, including the number of movement steps, success rate, reward value, and cumulative reward value.

[0042] 1.2 Parameter settings In the MPR model, action selection adopts the exploration-exploitation method, adjusting the dynamic initial weights of the exploration strategy and the utilization strategy is 0.9, the minimum weight 0.1. Total capacity of the experience pool is 100000, playback weight is 0.4, the reward discount factor γ is 0.99, and the number of hidden layer neurons is 256. For the MBPO algorithm, the weight parameter in the model loss function is 0.01, the coefficient of the target network soft update is 0.005, the initial temperature parameter is 0.2; in PER, the experience pool capacity is 50,000, which is the same as the actual experience pool capacity of the MPR model, the number of sampled experiences each time is 500, and the bias correction parameter Set to 0.3 and gradually fade to 1; the standard deviation of the exploration noise in HER In addition, each model adjusts its state to the best according to its needs.

[0043] In addition, the reward function used in the comparative experiment was set as follows: if the robot reached the finish line, it received a reward of r = 100; if the robot collided with an obstacle or wall, it received a reward of r = -50; if the robot's next position was closer to the finish line than its current position, the reward was r = 1; otherwise, r = -1. This encouraged the robot to move closer to the finish line. Furthermore, a maximum number of movement steps was set for the robot; if the number of movement steps exceeded the upper limit of 200, the robot was considered to have failed, and the reward was r = -50. This reward function was consistent across all models.

[0044] 1.3 Parameter analysis experimental results and analysis In actual situations, as the number of deduction steps of the environment model increases, the accumulated compound error of the model will increase rapidly, making the results of the environment model very unreliable. At the same time, it is necessary to weigh the negative effects of the increased error of the model after the number of deduction steps increases and the positive effects of the training strategy being better after the number of steps increases. In this case, the planning depth L The setting is very critical. In order to test different planning depths in different environments L The performance of the model, after a lot of testing and screening, L The set of values ​​is taken . Set this group L Value, verified at different planning depths L Impact on the model.

[0045] Using different L The diagram of the robot's cumulative reward under the same simulation training environment is as follows: Figure 4 As shown. Figure 4 As can be seen from the graph, MPR_4's cumulative reward increases the fastest, followed by MPR_3 and MPR_5. MPR_10 decreases slightly, but the change is not significant compared to MPR_3 and MPR_5. Furthermore, as shown by the light-colored error bands, MPR_4 has the narrowest error band, indicating that MPR_4 converges better than other models with different parameters, and its cumulative error is also superior to other models. Therefore, MPR_4 will be used in subsequent comparative experiments to compare with other algorithms.

[0046] 1.4 Simulation Experiment Results and Analysis The performance of MPR_4 is compared with three other models, PER, HER, and MBPO. Figure 5 The following table shows the optimal path diagrams of the robot in a static environment when using the four models. Table 2 shows the optimal number of navigation steps of the robot in a static simulation environment when using the four models.

[0047] Table 2 Optimal number of navigation steps in simulation training environment

[0048] Figure 6 The figure shows the reward changes in the first 200 rounds when using four models for training in a static environment.

[0049] To further validate the performance of our model, we conducted four more experiments. Each set contained 100 randomly generated, highly complex environments. The robot's starting and destination points remained identical in each environment. The robot was tested using each of the four models, and the number of times it successfully reached the destination point was recorded in each experiment. The results are shown in Table 3.

[0050] Table 3 Random environment test results

[0051] From Table 2 and Figure 3 It can be seen from Table 3 that MPR_4 has the least number of navigation steps in the static simulation environment and its path is better than the other three. It can be seen from Table 3 that MPR_4 has the most successful times in the four groups of experiments. Figure 5 It can be seen that during the training process, MPR_4 obtains rewards at a faster convergence speed and with a higher reward value. This shows that “planning” can provide more reliable target guidance for the early stages of training, thereby reducing unnecessary exploration and improving the stability and convergence speed of training. From Table 2, Table 3, Figure 5 and Figure 6The comparative experimental results show that the robot's decision path and stability are better when the MPR_4 model is used, indicating that the model has stronger environmental adaptability and strong robustness. The reasons are as follows: First, in the memory replay stage, the MPR_4 model simulates the neural mechanism of the human brain's reliance on long-term experience memory for forward-looking predictions, constructs a memory planning network centered on the hippocampus, and gradually predicts future states to perform multi-step simulation planning. This breaks through the path dependence of traditional reinforcement learning on immediate experience sampling, significantly reduces the dependence on real-world interaction data, and improves the model's adaptability and robustness in uncertain environments, as well as the robot's learning ability. Second, through the meta-reinforcement learning inner and outer loop dual-layer working structure, it realizes closed-loop interaction between the hippocampal planning replay network and the prefrontal policy evaluation network, constructs a corresponding environmental model and learns the corresponding strategy, and realizes the dynamic coupling of experience replay and forward-looking simulation in the brain-like decision-making process, enabling the robot to quickly adapt and make effective decisions. Finally, by integrating the meta-reinforcement learning framework with model-based policy optimization technology, it achieves the unification of dynamic planning and policy optimization. At the same time, adopting an adapted two-stream neural network architecture, it gradually shifts its focus from exploration to the utilization of relevant information during task execution, enabling the robot to not only make short-term decisions based on the current state and experience, but also plan for multiple possible future scenarios, thereby improving the long-term effectiveness of decision-making.

[0052] 2. Static physical experiment design and result analysis 2.1 Experimental Environment The experimental platform consists of an Autolabor Pro1 robot, a PC equipped with an AMD® Ryzen3 3200G processor and 48GB of DDR, and the program running environment is Python 3.6. Figure 7 The navigation experiment was conducted in a static physical environment, which is a 6.5m*6.5m rectangular environment. The static physical environment contains seven static obstacles of different sizes, one L-shaped obstacle, and one cross-shaped complex obstacle.

[0053] The robot's starting point was set to the upper right corner of the map, and the target was set at (5.5, 5.5), located in the lower left corner of the environment. Because the radar scanning point and the robot's collision distance are inconsistent, and the radar is located at the robot's head, the robot's volume and safety distance must be considered when determining whether it has collided with an obstacle. Therefore, all distances in this experiment refer to the distance between the robot and the obstacle or target point, not the distance measured by the radar. When the distance between the robot and the target point is less than 0.2m, the robot stops running, and the experiment ends.

[0054] 2.2 Static test results and analysis exist Figure 7 In the static physical environment shown, five algorithms, MPR-4 (the algorithm of the present invention after parameter analysis), PER, HER, MBPO, and MPR-0 (the algorithm with a planning depth of 0, i.e., the algorithm in the non-planning state in this embodiment), were tested in a real environment. Each algorithm was run 30 times in the static physical environment, and the number of movement steps taken by the robot to reach the target position and the success rate of reaching the target were recorded. The success rate, number of movement steps, and other results of each algorithm in the 30 experiments are recorded in Table 4. The results of each of the 30 experiments are recorded in Figure 8 middle.

[0055] Table 4 Comparative experiment records

[0056] As shown in Table 4, the average number of steps shows that the robot achieves the lowest minimum and average number of steps when using the MPR_4 model, indicating that the robot can find a better decision path. The average number of successful navigation steps using the MPR_4 model reaches 48.86. Although the minimum number of steps is the same as that of the PER and HER models, the step variance of the MPR_4 model is only 0.69, indicating excellent stability and superior performance to the PER, HER, MBPO, and MPR-0 models. Figure 9 The paths taken by the four algorithms when they successfully reach the target point are recorded.

[0057] The variance analysis in Table 4 shows that the MPR_4 model performs best, with similar HER and PER scores, but its performance is significantly superior to the MBPO and MPR-0 models. Although the robot's starting point and target point are the same, the MPR-0 model exhibits greater randomness compared to the other algorithms, resulting in the worst performance. The MPR_4 model has a higher success rate in terms of the number of successful navigation attempts and success rate, which is attributed to its superior stability over the other algorithms. Comparative experimental results show that the MPR_4 model has a better decision path and stability, demonstrating its stronger learning ability. This is due to the following reasons: First, during the memory replay phase, the MPR_4 model successfully constructed a memory planning network centered on the hippocampus by simulating the human brain's neural mechanisms for forward-looking predictions based on long-term accumulated experience. This network gradually predicts future states and performs multi-step simulation planning, effectively breaking through the path dependency of traditional reinforcement learning, which relies on immediate experience sampling. This mechanism significantly reduces reliance on real-world interaction data, improves the model's adaptability and robustness in uncertain environments, and enhances the robot's learning capabilities in various complex scenarios. Secondly, the model employs a meta-reinforcement learning framework, achieving a closed-loop interaction between the hippocampal planning replay network and the prefrontal policy evaluation network through a two-layer working structure: an inner and outer loop. In this process, the hippocampus replays historical experiences to generate possible future scenarios, while the prefrontal cortex is responsible for evaluating the effectiveness of strategies in these scenarios. Through this closed-loop interaction, the robot can construct and adjust environmental models in real time within a continuously updating environment, learning optimal decision-making strategies based on these models. This mechanism embodies the dynamic coupling of experience replay and forward-looking simulation in brain-inspired decision-making, enabling the robot to quickly adapt to new environmental changes and make more efficient decisions in changing situations.

[0058] Finally, MPR_4 combines meta-reinforcement learning with model-based policy optimization techniques, achieving a seamless fusion of dynamic programming and policy optimization. This approach not only allows the model to predict the optimal decision-making solution for the current state through dynamic programming, but also integrates policy optimization techniques to further improve the long-term effectiveness of decision-making. To adapt to this complex decision-making process, MPR_4 employs a two-stream neural network architecture, enabling the robot to gradually shift its focus from exploration to the effective utilization of acquired information during task execution. Through this gradual optimization mechanism, the robot not only makes rapid short-term decisions based on its current state and experience, but also effectively predicts and plans for multiple possible future scenarios and adopts the most appropriate action strategy. This optimizes decision-making in a variety of complex situations, significantly improving the overall effectiveness of task execution.

[0059] 3. Dynamic physics experiment design and result analysis 3.1 Dynamic Experimental Environment In addition to eight static obstacles of varying sizes, the dynamic environment also includes two dynamic obstacles: an experimenter and another Autolabor Pro1 robot. These two dynamic obstacles move randomly within the environment to determine the proposed model's decision-making capabilities in dynamic environments.

[0060] 3.2 Experimental results and analysis exist Figure 10 In the dynamic real-world environment shown in the figure, MPR_4, MPR_0, PER, HER, and MBPO were tested in a real-world environment. The robot ran 30 times in the dynamic real-world environment, recording the number of steps it took to reach the target position and the success rate of reaching the target. The results are shown in Table 5.

[0061] Table 5 Success times of robots using five models in dynamic environments

[0062] As shown in Table 5, in the dynamic test environment, MPR_4 has a higher success rate in terms of the number of successful navigations and the success rate. This is because it is superior to other algorithms in terms of adaptability.

[0063] Figure 11 The following shows a successful path of a robot using five models. Figure 11 As shown in the figure, the red dots in the image represent obstacle information returned by the scan / topic, and the black areas represent the obstacle RVIZ image information drawn by the map / topic. The robot's motion path and dynamic obstacle information recorded by RVIZ show that the robot exhibits clear avoidance behavior when encountering obstacles. This indicates that in an environment with dynamic obstacles, the robot can effectively avoid both static and dynamic obstacles and successfully navigate to the target point. Combined with Table 5 and Figure 11 As can be seen, the MPR_4 model has better decision paths and stability, indicating that the robot has stronger learning ability. The underlying reasons are the same as those in the simulation environment and static physical environment experiments mentioned above, so I will not elaborate on them here. Figure 12 The obstacle avoidance process of the MPR_4 model during a certain experiment is demonstrated.

[0064] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. Any technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A behavioral decision-making method simulating the hippocampal-prefrontal memory planning and playback mechanism, characterized in that: The steps include: S1. Design a memory planning replay model driven by experience value and environmental structure. During the memory replay phase of the memory planning replay model, a memory planning network centered on the hippocampus is constructed by simulating the neural mechanism of the human brain's forward-looking prediction based on long-term accumulated experience, denoted as the hippocampal planning replay network. During the planning phase of the memory planning replay model, a strategy evaluation network is constructed, denoted as the prefrontal strategy evaluation network, including a multi-layer attention mechanism that simulates the hierarchical structure of the prefrontal cortex. This enables the memory planning replay model to simulate the collaborative planning mechanism of the hippocampal planning replay network and the prefrontal strategy evaluation network. S2. The memory planning replay model is integrated with the meta-reinforcement learning framework and the model-based policy optimization technology to enhance the memory replay and policy optimization process in reinforcement learning, and the unification of dynamic planning and policy optimization is achieved through the bionic neural mechanism, as well as the dynamic coupling of experience replay and forward-looking simulation in the brain-like decision-making process, so that the memory planning replay model can learn in "planning".

2. The behavioral decision-making method simulating the hippocampus-prefrontal lobe memory planning and playback mechanism according to claim 1, characterized in that: In step S1, the memory planning replay model obtains the basic environmental data by the model-free meta-reinforcement learning method, and then uses the strategy optimization method based on model meta-reinforcement learning to use the current basic data as the starting point of planning and perform planning for a certain number of steps. Finally, the planning results are used to train the memory planning replay model.

3. The behavioral decision-making method simulating the hippocampus-prefrontal lobe memory planning and playback mechanism according to claim 1, characterized in that: In step S1, the hippocampal planning replay network gradually predicts future states and performs multi-step simulation planning, thereby effectively breaking through the path dependence of traditional reinforcement learning when relying on immediate experience sampling.

4. The behavioral decision-making method simulating the hippocampus-prefrontal lobe memory planning and playback mechanism according to claim 1, characterized in that: In step S2, a meta-reinforcement learning framework is adopted to realize the closed-loop interaction between the hippocampal planning replay network and the prefrontal strategy evaluation network through an inner and outer loop double-layer working structure.

5. The behavioral decision-making method simulating the hippocampus-prefrontal lobe memory planning and playback mechanism according to claim 4, characterized in that: In step S2, during the closed-loop interaction between the hippocampal planning replay network and the prefrontal policy evaluation network, the hippocampal planning replay network generates possible future scenarios by replaying historical experiences, while the prefrontal policy evaluation network is responsible for evaluating the policy effects in these scenarios, thereby enabling the intelligent agent to build and adjust the environmental model in real time in a constantly updated environment, and learn the optimal decision-making strategy based on these environmental models.

6. The behavioral decision-making method simulating the hippocampus-prefrontal lobe memory planning and playback mechanism according to claim 4, characterized in that: In step S2, based on the given current state and position, the memory planning replay model constructs the corresponding environment model and learns the corresponding strategy through the outer loop structure of the meta-reinforcement learning framework, gradually predicts the future state to perform multi-step simulation planning, and enhances the effectiveness of multi-step simulation planning through model-based policy optimization technology until the desired goal is achieved or the planning depth limit is reached.

7. The behavioral decision-making method simulating the hippocampus-prefrontal lobe memory planning and playback mechanism according to any one of claims 1 to 6, characterized in that: When the behavior decision-making method is applied to the behavior decision-making of a robot, the working process of the robot includes: 1) Initial robot exploration of the environment: The robot collects real-time data from the environment through various sensors to form a cognition of the current situation. The robot then encodes the perceived tasks and situations into memory units to form an initial environmental model. 2) The robot plans future behavioral decisions: The robot stores a hippocampal planning replay network based on interactive learning with an environmental model. This network predicts future states and actions based on existing memory units and the initial environmental model. The prefrontal policy evaluation network integrates the environmental model, memory units, and goal settings to assess the expected outcome of each action. The robot then makes appropriate decisions based on this assessment. 3) Robot execution decision and strategy optimization: The robot starts executing the corresponding action based on the optimal decision generated by the hippocampal planning replay network and makes necessary adjustments based on real-time feedback information. At the same time, the robot feeds back the success and failure experience of the task into the memory unit to optimize the environmental model and decision-making strategy, further improving the success rate of task completion.

Images