A brain-like multi-time scale structured memory playback method and device

Abstract

The application relates to a kind of brain-like multi-time scale structured memory playback method and equipment in the field of robot intelligent control technology, the method divides experience playback process into experience selection and value update two main stages, in experience selection stage, model constructs two structured playback modes of forward playback and backward playback, forward playback is used to keep recent experience coverage, and backward playback is used to propagate target reward or collision penalty along predecessor state in reverse direction, so as to improve the utilization efficiency of key experience.In the value update stage, the model introduces a multi-discount factor value learning mechanism, maintains independent action value functions for different time scales, and calculates multiple sets of TD errors in parallel for the same playback experience, allowing it to participate in the update of both short-term immediate returns and long-term target values simultaneously.The invention realizes the collaborative optimization of structured experience selection and multi-time scale value propagation, thereby improving the sample utilization rate and decision flexibility of the intelligent agent in complex tasks.

Description

Technical Field

[0001] This application belongs to the field of computer science, and particularly relates to the field of intelligent robot control technology. More specifically, it relates to a brain-like multi-timescale structured memory playback method and device. Background Technology

[0002] Autonomous navigation for mobile robots is a crucial research problem in the field of intelligent robotics. Its goal is to enable robots to autonomously select actions based on sensor observations in unknown environments with obstacles, safely and efficiently reaching their target location. This places high demands on navigation systems, especially in complex environments, requiring them to possess processing capabilities similar to a brain. Navigation tasks in real-world environments inherently exhibit significant multi-timescale characteristics. When approaching obstacles, robots need to focus on short-term safety and immediate obstacle avoidance; when moving away from the target, they need to consider long-term path selection and target approach. Therefore, it is necessary to design a model with reasonable multi-index priority sampling criteria that can describe valuable information at different timescales to achieve efficient learning.

[0003] Interdisciplinary research in neuroscience, brain science, and artificial intelligence has demonstrated that the human brain possesses flexible and efficient decision-making capabilities in complex environments. The hippocampus plays a crucial role in episodic memory, spatial cognition, and environmental representation. In unknown environments, the hippocampus also participates in value-based decision-making processes, influencing future behavioral choices by organizing and replaying past experiences. Further research indicates that the human brain often relies on the collaborative efforts of different brain regions when solving complex decision-making tasks. For example, the spatial cognitive system, jointly formed by the hippocampus and entorhinal cortex, provides an important representational basis for environmental structure and location information. Furthermore, various neurotransmitters also influence and coordinate the functional specialization of different brain regions in learning, memory, and decision-making processes. For instance, novel stimuli-induced dopamine release can promote the consolidation and persistence of hippocampal-related memories, and acetylcholine is believed to participate in regulating the interaction between attention and memory; its release state in the hippocampus and neocortex may affect information encoding and memory retrieval processes.

[0004] Inspired by these brain mechanisms, an increasing number of researchers are attempting to design intelligent decision-making algorithms by simulating the structure and function of the brain. They are particularly focused on the collaborative interactions of key brain regions, such as the hippocampus, basal ganglia, and neocortex. Some researchers have proposed a hierarchical self-organizing reinforcement learning framework by combining the rapid learning mechanism of the hippocampus with the slow generalization mechanism of the neocortex, improving the learning efficiency, success rate, and environmental adaptability of robot navigation. Others have proposed a hippocampus-prefrontal cortex collaborative memory planning and playback model. Through experiential memory, prospective planning, and policy evaluation, robots can simulate future behavior and optimize decisions, improving their learning efficiency and path planning capabilities in complex environments. Despite these advantages, the learning ability of these brain-like algorithms is hindered in real navigation scenarios by limitations such as uncontrollable environments or low sample utilization.

[0005] In biological learning, memory replay is considered an important mechanism for improving learning efficiency and supporting flexible decision-making. Hippocampal memory replay refers to the phenomenon where, during rest or sleep, compressed and reactivated neural activity sequences related to previous spatial experiences occur in the hippocampus. Early studies often viewed replay as the reproduction of existing trajectories, but recent research has found that replay is not simply a repetition of past experiences, but rather exhibits forward sequences, reverse sequences, shortcut sequences, and even sequences of paths not actually experienced (virtual sequences). This suggests that memory replay is not a uniform, passive process of experience reproduction, but rather an active memory access mechanism jointly regulated by empirical statistics, reward information, and environmental structure.

[0006] Existing reinforcement learning navigation methods typically improve sample utilization through experience replay. Prioritized experience replay assigns sampling priority to experiences based on Temporal-Difference (TD) error, prioritizing experiences with higher prediction errors. However, traditional prioritized experience replay methods use only a single metric to select replayable experiences, making it difficult to characterize the trajectory directionality and spatial continuity of experiences. The SFMA model has been proposed to further explain the generation mechanism of hippocampal replay from a computational perspective. This research shows that structured replay can generate diverse replay sequences without explicitly calculating the utility of each experience hypothesis and promotes spatial learning. This perspective provides important insights into experience replay mechanisms in reinforcement learning: experience replay should not be viewed merely as a process of randomly drawing samples from a cache, but can be designed as a structured and task-relevant memory access process.

[0007] Besides selecting which experiences to replay, how to update the value of those replays is also a key issue affecting the learning efficiency of an agent. Traditional reinforcement learning methods typically use a single discount factor to update the value of future rewards, compressing rewards across different time scales into a single scalar value function. However, real-world decision-making tasks often contain information across multiple time scales: short-term scales handle immediate feedback and local risks, while long-term scales evaluate delayed gains and long-term goals. A single discount factor struggles to simultaneously address both, potentially leading to an imbalance between short-sighted reactions and long-term planning on the part of the agent.

[0008] Recent research by Masset et al. provides important neuroscientific evidence for multi-temporal-scale reinforcement learning. This study indicates that different dopamine neurons may encode reward prediction error signals with different discounting time constants. That is, a population of dopamine neurons does not correspond to a single-temporal-scale TD error, but can form a multi-temporal-scale value prediction representation. Multi-discount factor value representation can decouple reward magnitude and reward timing, retaining temporal information about future rewards even when learning has not fully converged, and helping the agent flexibly adjust between short-term and long-term value estimations according to task requirements. This provides a physiological basis for constructing multi-temporal-scale reinforcement learning algorithms and suggests that maintaining value functions at multiple time scales simultaneously in navigation tasks may help improve the learning efficiency and decision-making flexibility of agents in complex tasks.

[0009] Therefore, there is an urgent need in this field to propose a brain-like multi-timescale structured memory playback method and device to improve the sample utilization and decision-making flexibility of intelligent agents in complex tasks, thereby improving the learning efficiency and navigation performance of robot navigation. Summary of the Invention

[0010] The purpose of this invention is to propose a brain-like multi-timescale structured memory playback method and device to improve the sample utilization and decision-making flexibility of intelligent agents in complex tasks, thereby improving the learning efficiency and navigation performance of robot navigation.

[0011] Based on one of the aforementioned objectives, this application proposes a brain-like multi-temporal-scale structured memory playback method, comprising the following steps: S1. Based on the memory replay mechanism of the entorhinal cortex and hippocampus in the brain, a structured memory-priority replay model is proposed. The structured memory-priority replay model divides the experience replay process into two main stages: experience selection and value update. In the experience selection stage, two structured replay modes, forward replay and backward replay, are constructed. Forward replay is used to maintain the coverage of recent experiences, while backward replay is used to propagate the target reward or collision penalty backward along the predecessor state, thereby improving the utilization efficiency of key experiences. In the value update stage, a multi-discount factor value learning mechanism is introduced to maintain independent action value functions for different time scales, and multiple sets of TD errors are calculated in parallel for the same replay experience, so that it can participate in the updating of short-term immediate benefits and long-term target values ​​at the same time. S2. Based on the multi-timescale TD error, a dynamic replay mode selection strategy is designed to adaptively adjust the replay probabilities of forward and reverse replay modes. The structured memory-first replay model enhances the reverse replay mode to accelerate the back propagation of key value information. When the value estimation tends to stabilize, the structured memory-first replay model adopts the forward replay mode to maintain experience coverage and training stability.

[0012] Preferably, the structured memory priority replay model organizes the stored experience into structured memories with temporal order and state transition relationships. The structured memory priority replay model continuously stores information experienced by the robot during the interaction process, including states, actions, rewards, and the next state. It selects an appropriate replay mode according to the current learning state, enabling the agent to utilize key experiences, including successfully reaching the target, collision failures, and near-obstacle adjustments, more efficiently with limited interaction samples, thereby improving navigation success rate and strategy adaptability.

[0013] Preferably, the workflow of the structured memory-first playback model includes: 1) In the initial exploration phase, the robot acquires real-time environmental information through sensors including lidar and odometry. Through continuous exploration, the structured memory priority playback model gradually accumulates a set of structured experience containing action states and environmental information, providing a foundation for subsequent training. 2) During the training phase, the structured memory priority replay model uses the experience stored in the experience pool to perform structured replay and multi-timescale value updates, gradually forming a navigation strategy that takes into account both local security and long-term goal convergence. 3) In the actual verification phase, the multi-timescale action value function obtained from training is used for robot navigation decision-making. The robot constructs the current navigation state based on real-time sensor data and uses the learned value function to select the optimal discrete action to complete the target navigation task.

[0014] Preferably, the structured memory priority playback model is driven by both experiential value and environmental structure, and while focusing on memory strength, the structured memory priority playback model will prioritize the playback of experiences in the vicinity of obstacles, the vicinity of targets, narrow passages, or turning areas.

[0015] Preferably, in the structured memory-priority replay model, experience replay can be analogized to the reactivation of place cells in a memory replay event. When an experience is reactivated, each stored experience is assigned a priority score based on memory strength, structural association strength with the current experience, and state inhibition level. The evaluation metrics are as follows: ; in, Indicates memory strength, influenced by frequency of experience. and reward size To decide, For memory strength and structural correlation strength The trade-off coefficients between these factors are used to evaluate the robot and select experiences applicable to the current environment for playback.

[0016] Preferably, the structural correlation strength between experiences Used to describe current experience and storage experience The structural correlation strength between them is calculated using the following formula: ; in The structural distance represents the relationship between two experiences. The smaller the structural distance, the closer the two experiences are in terms of state or goal, and the stronger the structural association between them. The larger the structural distance, the weaker the relationship between the two experiences.

[0017] Preferably, since structural distance is not equivalent to simple physical distance, in the structured memory-priority replay model, the structural representation is constructed based on the reachability relationship between states as follows: ; ; in, It is the identity matrix. As a discount factor, This is the state transition matrix under the default behavior strategy.

[0018] Preferably, in the structured memory-priority playback model, the inhibition regulation term is defined in an exponential decay form: ; in, To suppress the intensity adjustment parameter, As an inhibitor, it is used to prevent a few high-intensity experiences from being repeatedly activated in a short period of time. When it is large, A smaller value indicates that the experience has been recently replayed, and the probability of it being reactivated should be lower. After decaying over time, As the experience increases, the chances of it being used in the replay selection also increase.

[0019] Preferably, in the structured memory priority replay model, candidate experiences are normalized to sampling probabilities based on priority scoring:

[0020] in, This is to prevent extremely small constants where the denominator is zero. The structured memory priority replay model selects the experience batches to be replayed from the experience pool based on the sampling probability.

[0021] Based on another aspect of the present invention, a brain-like multi-temporal-scale structured memory playback device is also proposed, comprising: One or more processors; Computer-readable medium for storing one or more computer-readable instructions. When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the above-described neuromorphic multi-timescale structured memory playback method.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention integrates memory strength, environmental structure relevance, and suppression of feedback to calculate the priority of experience replay, so that key experiences related to the current navigation task are given priority use, while avoiding the continuous replay of a few high-priority experiences, thereby improving the utilization efficiency of limited interaction samples and the structural and task-related nature of experience replay.

[0023] 2. This invention employs multiple discount factors to update the action value at different time scales in parallel, enabling the robot to balance local obstacle avoidance safety, target approach, and path efficiency. Simultaneously, it dynamically adjusts the selection probability of forward playback (default playback) and reverse playback based on the accumulated time difference error across multiple time scales, promoting the effective dissemination of key reward and penalty information. This improves the learning efficiency and stability of the navigation strategy and enhances the robot's navigation adaptability in static and dynamic obstacle environments. Attached Figure Description

[0024] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This diagram illustrates a navigation strategy learning framework based on the MTSMR algorithm in an embodiment of the present invention. Figure 2 A framework diagram of the MTSMR algorithm in an embodiment of the present invention is shown; Figure 3 The simulation experimental environment in this embodiment of the invention is shown, wherein Figure 3 (a) is a static simulation environment. Figure 3 (b) is a dynamic simulation environment; Figure 4 The average reward change curves of each algorithm during the training process are shown in the embodiments of the present invention; Figure 5 The success rates of different algorithms in 50 verification experiments are shown in the embodiments of the present invention; Figure 6 The physical experimental environment in an embodiment of the present invention is shown; Figure 7 A robot navigation trajectory diagram from an embodiment of the present invention is shown, wherein... Figure 7 (a) shows the robot's navigation trajectory in a static environment. Figure 7 (b) is the robot's navigation trajectory in a dynamic environment; Figure 8 This invention illustrates successful robot paths using four models in a static environment, as shown in this embodiment. Figure 9 The diagram illustrates the successful robot paths using four models in a dynamic environment, as shown in this embodiment of the invention. Detailed Implementation

[0025] The following will be combined with the appendix Figures 1-9 The technical solutions of the present invention have been clearly and completely described. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This embodiment proposes a brain-like multi-temporal-scale structured memory playback method, including the following steps: S1. Based on the memory replay mechanisms of the entorhinal cortex and hippocampus in the brain, a structured memory-priority replay model is proposed to improve the utilization rate of samples and the decision-making flexibility of mobile robots in complex navigation tasks. Inspired by the hippocampal structured memory replay and the dopamine multi-timescale reward prediction error encoding mechanism, this model divides the experience replay process into two main stages: experience selection and value updating. In the experience selection stage, the model constructs two structured replay modes: forward replay and backward replay. Forward replay is used to maintain recent experience coverage, while backward replay is used to propagate the target reward or collision penalty backward along the preceding state, thereby improving the utilization efficiency of key experiences. In the value updating stage, the model introduces a multi-discount factor value learning mechanism to maintain independent action value functions for different timescales and to calculate multiple sets of TD errors in parallel for the same replay experience, allowing it to participate simultaneously in the updating of short-term immediate gains and long-term target values.

[0027] S1. A dynamic replay mode selection strategy is designed based on multi-timescale TD error to adaptively adjust the replay probabilities of forward and reverse modes. When the cumulative prediction error is large, the model enhances reverse replay to accelerate the backpropagation of key value information; when the value estimation tends to stabilize, the model uses forward replay more often to maintain experience coverage and training stability. Thus, the proposed model achieves synergistic optimization of structured experience selection and multi-timescale value propagation, thereby improving the sample utilization and decision-making flexibility of the agent in complex tasks.

[0028] The main innovative points of this invention can be summarized as follows: (1) Inspired by the memory replay mechanism of the entorhinal cortex and hippocampus in the brain, a structured memory priority replay framework is proposed. Instead of relying on a single indicator to select replay experience, a multi-indicator priority sampling mechanism is designed to improve the sample utilization rate.

[0029] (2) Construct a value update mechanism inspired by the multi-timescale reward prediction error encoding of dopamine. By maintaining independent action value functions for multiple discount factors, the model can perform TD error updates at different timescales in parallel for the same experience, so that short-term immediate benefits and long-term goal values ​​can be co-propagated in the replay stage, thereby improving the flexibility of decision-making.

[0030] (3) Design a dynamic replay control strategy driven by multi-timescale TD error. This strategy adaptively adjusts the probabilities of forward and reverse replay based on the recent cumulative prediction error, realizing dynamic coupling between experience selection and value update, thereby improving training stability and decision adaptability.

[0031] (4) The brain-like algorithm mentioned above is applied to the robot's navigation task. Its learning efficiency, navigation performance and practical application value are analyzed through comparative experiments, ablation experiments and simulation to real-world experiments.

[0032] Based on neurobiological mechanisms and drawing on the synergistic interaction between the hippocampal-entorhinal cortex memory system and the midbrain dopaminergic system, this embodiment proposes a multi-timescale structured memory replay (MTSMR) model based on multi-timescale reinforcement learning. Its overall framework is as follows: Figure 1 As shown, this model does not simply treat stored experience as randomly selected independent samples, but rather organizes it into structured memories with temporal order and state transition relationships. Furthermore, the model continuously stores information such as the robot's experienced states, actions, rewards, and next states during interactions, and selects appropriate replay modes based on the current learning state. In this way, the agent can more efficiently utilize key experiences such as successfully reaching the target, failing collisions, and adjusting near obstacles with limited interaction samples, improving navigation success rate and policy adaptability.

[0033] The workflow of this model is as follows: 1) In the initial exploration phase, the robot acquires real-time environmental information using sensors such as LiDAR and odometry. The model further combines the target point's location to calculate the target's direction relative to the robot's current orientation and the distance from the robot to the target point, thereby constructing a navigation state representation that includes the target direction, distance to local obstacles, and target distance. This state contains both short-term safety information required for local obstacle avoidance and task information required for approaching the target. Guided by the exploration strategy, the robot selects its next motion state and interacts with the environment. After each interaction, this interaction information is processed and encoded into the memory unit. Through continuous exploration, the model gradually accumulates a structured set of experience containing action states and environmental information, providing a foundation for subsequent training.

[0034] 2) During the training phase, the model utilizes the experience stored in the experience pool for structured replay and multi-timescale value updates. Once the amount of experience stored in the experience pool reaches the training condition, the system prioritizes replaying historical experiences that are more similar to the current state in terms of target direction or obstacle distribution. For the replayed experiences, the model introduces a multi-discount factor value learning mechanism to maintain independent action value functions for different timescales and calculates multiple sets of TD errors in parallel for the same experience, allowing it to participate in both short-term immediate gains and long-term goal value updates. Furthermore, the model dynamically adjusts the probability of forward replay (default replay) or reverse replay based on recent multi-timescale TD errors. Through continuous cyclical updates of "experience replay—multi-timescale update—replay mode adjustment," the robot gradually develops a navigation strategy that balances local safety and long-term goal convergence.

[0035] 3) In the practical verification phase, the multi-timescale action value function obtained from training was used for robot navigation decision-making. The robot constructs its current navigation state based on real-time sensor data and uses the learned value function to select the optimal discrete action to complete the target navigation task.

[0036] Based on the aforementioned physiological mechanisms of hippocampal memory replay, and addressing the memory replay imbalance caused by low replay sample utilization and poor decision-making flexibility, this embodiment presents an algorithmic framework for a structured memory replay model based on multi-temporal-scale reinforcement learning, which incorporates reinforcement learning methods. Figure 2 As shown, the model consists of three main parts: first, a structured memory replay mechanism is used to select key experiences for replay; second, a multi-timescale reinforcement learning mechanism is used to update the action value function of the selected replay experiences; and finally, a dynamic adjustment mechanism based on multi-timescale TD error is designed to reasonably control the probability of using forward and reverse replay. These three elements constitute a closed-loop learning process of "experience selection - value update - replay regulation," enabling the robot to make the most appropriate decisions based on the specific environment.

[0037] Specifically, the robot starts from a given starting point, selects actions based on odometry and LiDAR observations, and reaches the target location as quickly as possible while avoiding collisions. First, the robot needs to explore the task environment and interact with the real-world scene to build memory units and a basic environmental model. Structured memory playback not only focuses on memory strength but also considers experiences highly relevant to the context; in other words, it prioritizes playback of experiences from areas near obstacles, near the target, in narrow passages, or turning areas. Therefore, the model in this application is driven by both experiential value and environmental structure.

[0038] Robots interacting with their environment will form memory tuples. ,in This is the robot's current state. For the action to be performed, This is the reward obtained after the state transition. The next state after the robot has performed an action. These experience tuples Viewed as memories that can be reactivated, in the hippocampus, this replay of experiences can be analogized to the reactivation of place cells during a memory replay event. When an experience is reactivated, each stored experience... Based on memory strength, structural relevance to current experience, and state inhibition level, a priority score is assigned.

[0039] First, define memory strength. Its frequency is influenced by experience. and reward size That's for you to decide.

[0040] (1) (2) (3) in, , Indicates weight, Indicates the number of times an experience has been performed. This represents the reward function, where the reward value is determined by the distance relationship between the robot and obstacles / targets.

[0041] The structural correlation of experiences cannot simply consider the spatial distance between two states; it must also take into account walls and obstacles in the specific task environment. Therefore, it is necessary to further define the strength of the structural correlation between experiences. Used to describe current experience and storage experience Structural correlation between them: (4) in Structural distance represents the structural distance between two experiences. A smaller structural distance indicates that the two experiences are closer in terms of state or goal, and the stronger the structural association between them. A larger structural distance indicates a weaker connection between the two experiences, suggesting the possible presence of obstacles or other structural gaps. Therefore, structural distance is not simply equivalent to physical distance; it is constructed based on the reachability relationships between states. (5) (6) in, It is the identity matrix. As a discount factor, This is the state transition matrix under the default behavior strategy. Specifically, This indicates that the robot, without introducing a specific goal bias, starts from a certain state. Transition to state The base probability.

[0042] Inhibitors This is used to prevent a few high-intensity experiences from being repeatedly activated in a short period of time. After playback, its inhibition factor is set to a higher value, and the update rule is as follows: (7) (8) Among them, attenuation factor To enable the inhibitory factor to reduce the reactivation probability of recently replayed experiences, this application defines the inhibitory regulation term using an exponential decay form: (9) in, This is a parameter for suppressing intensity adjustment. When When it is large, A smaller value indicates that the experience has been recently replayed, and the probability of it being reactivated should be lower. After decaying over time, As the experience increases, the chances of it being used in the replay selection also increase.

[0043] Based on the above analysis, the evaluation index for memory-priority playback is designed as follows: (10) in This is a trade-off coefficient between memory strength and structural relevance. Using this priority evaluation criterion, the robot can select experiences applicable to the current environment for replay. Based on the priority score, candidate experiences are normalized to sampling probabilities: (11) in, To prevent extremely small constants with a denominator of zero, the model selects a batch of empirical data to be replayed from the empirical pool based on the sampling probability. .

[0044] After completing the structured experience selection, the model performs multi-timescale value updates on the experiences in the replay batch. The robot maintains multiple sets of discount factors in parallel. The corresponding value function Through parallel TD error renew: (12) (13) (14) in It's the learning rate. The number of discount factors is used to obtain a multi-timescale value function vector: (15) Among them, the small discount factor focuses on short-term immediate rewards, mainly paying attention to immediate rewards and local obstacle avoidance, while the large discount factor focuses on long-term delayed rewards, paying more attention to goal approach and path efficiency, providing a quantifiable time scale basis for replay priority. In the actual action selection stage, the robot needs to fuse value estimates from multiple time scales to form a final decision. This application defines the fused action value function as: (16) in, Indicates the state Next The fusion weights of the value functions at each time scale satisfy: (17) Ultimately, the robot selects its action based on the fusion value function: (18) After each batch update is completed, the absolute values ​​of the TD errors of all replayed experiences across all time scales are accumulated: (19) in This indicates the currently selected batch of replay experiences. The cumulative error reflects the overall value prediction bias of the selected replay experiences across multiple time horizons. Conversely, it adjusts the structural correlation in the priority settings. The specific calculation method, when using the forward playback mode, defines the structural correlation degree as follows: (20) This pattern compares the structural distance between the "current replay state" and the "candidate experience starting state," thus favoring the activation of experiences that are structurally adjacent to or similar to the current replay state. When using the reverse pattern, the structural correlation is defined as: (twenty one) This model compares the structural distance between the "current replay state" and the "next state of the candidate experience". In this way, reverse replay can preferentially activate predecessor experiences along the state transition chain, allowing key value information such as target rewards or collision penalties to propagate to the current state more quickly.

[0045] Finally, the sigmoid function is used to... It transforms into the selection probability of reverse playback: (twenty two) (twenty three) in , This is a constant and needs to be set according to the specific experimental environment. The model is based on... The probability of selecting the reverse playback mode is used to... The probability is selected to choose the positive playback mode. After the mode is selected and the playback is completed, the accumulated TD error is cleared to zero, and the next round of accumulation cycle starts again until the robot reaches the destination or encounters an obstacle, thus ending the current round of experiment.

[0046] Finally, to more clearly illustrate the working principle of this application, Algorithm 1 provides the following pseudocode of its workflow:

[0047] To enable those skilled in the art to better understand and implement this invention, the specific embodiments of the invention are further described below in conjunction with mobile robot navigation experiments. The following experimental verification process is used to illustrate the technical solution of this invention and is not intended to limit the scope of protection of this invention. Equivalent substitutions made to the robot platform, sensor type, number of discount factors, parameter settings, and control methods without departing from the core idea of ​​this invention are all within the scope of implementation of this invention.

[0048] Experimental verification of the scheme To verify the effectiveness of the proposed Multi-Timescale Structured Memory Replay (MTSMR) model in mobile robot navigation tasks, a static obstacle navigation environment was constructed in the ROS-Gazebo simulation platform, using an AutolaborPro1 mobile robot as the experimental subject. The robot acquires its own pose information via odometry and distance information to surrounding obstacles via LiDAR. The control unit issues velocity commands based on the discrete actions output by the algorithm, thereby driving the robot to complete the target navigation task. During the experiment, the robot needs to start from a given starting point and reach the target position as quickly as possible while avoiding collisions. Specific experiments include: 1) an ablation experiment to analyze the contribution of each module to the overall performance; 2) comparing the algorithm with SFMA, DDPGPER, and TD3PER algorithms to verify its performance advantages; and 3) deploying the algorithm on the AutolaborPro1 robot platform for a physical experiment to verify its practical application value.

[0049] Experimental setup 1) Gazebo training environment design: The experimental environment is as follows Figure 3 As shown, the simulation map uses a two-dimensional navigation environment, including boundary walls and five obstacles inside. The effective movement range of the map is 10m × 10m. The robot's starting position is set at (0, 0), and the target position is set at (8.5, 1.2). It should be further noted that the robot is considered successfully positioned when it reaches a circular area with a radius of 0.3m centered at (8.5, 1.2). The training uses an AutolaborPro1 robot equipped with a LiDAR system containing 901 laser beams, with a scanning range relative to the positive X-axis of the robot. arrive Radius. The training equipment was configured with an NVIDIA GTX 4070 graphics card, 32GB of RAM, and an AMD Ryzen 78845H processor. The MTSMR model training averaged 21,558 steps and took approximately 8.6 hours on average.

[0050] 2) State Space and Action Space Setup: The state space needs to describe not only the robot's current target approach relationship, but also reflect the local obstacle structure. Therefore, this application sets the state space as follows: ,in, This represents the local obstacle structure information extracted by the lidar. This represents the normalized distance from the robot to the target point. This represents the direction information of the target point relative to the robot's current orientation. To facilitate the updating of the action value function, this application further discretizes the continuous state, setting the number of discrete intervals for each dimension to be [value missing]. Therefore, the size of the discrete state space is Regarding the action space, to reduce the learning difficulty caused by continuous velocity control, this application does not simultaneously use linear velocity and angular velocity as reinforcement learning outputs, but instead sets the action as a discrete local steering decision. The action space is defined as follows: There are 21 discrete actions in total. Each action corresponds to a local turning angle of the robot's current orientation. The maximum turning range is... , No. The steering angle corresponding to each action is: When the robot performs an action, it first turns according to the selected action, and then moves forward at a fixed linear velocity. In the experiment, the linear velocity was set to... The duration of a single forward movement is When the distance to the obstacle ahead is less than At this point, the robot stops moving forward to ensure the safety of the exploration process.

[0051] 3) Reward Function Setting: To guide the robot to reach the target efficiently while ensuring safety, this application adopts a reward function consisting of termination reward and process reward. A 2000 arrival reward is given when the robot enters the target area; additional rewards are given for collisions, entering dangerous situations, or exceeding environmental boundaries. 700 500 and A penalty of 450 is applied. For unterminated states, process rewards are given based on the change in distance between the robot and the target point: positive rewards are given when moving closer to the target, and penalties are imposed when moving away from the target. This penalty is applied at each decision step. A time penalty of 0.03 is applied to reduce invalid actions and path detours. An approach penalty is applied when the clearance between the robot and an obstacle is less than 0.75m, progressively increasing as the clearance decreases to less than 0.40m and 0.16m. If the clearance is no greater than 0.065m, or if there are more than four consecutive decision steps with clearance less than 0.10m, a dangerous state is declared, and the current round is terminated. Additionally, the robot receives a small recovery reward when it actively moves away from obstacles or environmental boundaries. Both static and dynamic experiments use the above reward settings; the round reward in the training curve is the cumulative value of all single-step rewards within a round.

[0052] ablation experiment The experimental parameters are configured as shown in Table 1: Table 1 Algorithm Parameter Configuration

[0053] To verify the contribution of each key module in the proposed MTSMR model to navigation learning performance, an ablation experiment was conducted in the same Gazebo simulation environment. During the experiment, the robot's initial position and target point were randomly sampled from the free space of the environment. To ensure the effectiveness of the navigation task, the randomly sampled initial position and target point were not located in obstacle areas, and a certain minimum distance was maintained between them. The robot needs to autonomously select actions based on the current LiDAR observation information and the target's relative pose information, and reach the target point as quickly as possible while avoiding collisions. When the robot reaches the target area without collision within a preset maximum number of steps, the navigation round is considered successful; if the robot collides with an obstacle, or fails to reach the target point within the preset number of steps, the round is considered a failure.

[0054] To analyze the function of different modules, the present invention sets up the following ablation model for comparison: 1) MTSMR: Complete Model 2) MTSMR-SD: Single Discount Factor Model. This model removes the multi-timescale reinforcement learning mechanism and uses only a fixed discount factor for action value function updates, while keeping the structured playback and playback mode modulation mechanisms unchanged. This model is used to verify the impact of multi-timescale value update mechanisms on robot navigation learning performance.

[0055] 3) MTSMR-FM: Fixed Replay Mode Model. This model retains the structured priority replay and multi-timescale value update mechanism, but disables the dynamic replay mode adjustment strategy. Forward and backward replays are selected with fixed probabilities, and the replay mode is no longer adaptively adjusted based on the accumulated multi-timescale TD error. This model is used to verify the impact of the dynamic replay adjustment mechanism on value propagation efficiency and training stability.

[0056] 4) MTSMR-US: Unstructured Sampling Model. This model removes the structured memory-first sampling mechanism and replaces the experience sampling method with uniform random sampling, while keeping the other value update and playback mode settings unchanged. This model is used to verify the impact of the structured memory-first playback mechanism on the efficiency of utilizing key navigation experience.

[0057] Each algorithm is trained for 200 rounds. To more clearly reflect the learning performance during training, this invention uses a non-overlapping window averaging process for the reward curve. Specifically, the 200 training rounds are divided into 10 consecutive and non-overlapping statistical windows, each containing 20 training rounds, and the average reward is calculated. This method reduces the fluctuations caused by random starting points, random target points, and exploration actions in a single training round, making the reward trend smoother and thus more intuitively reflecting the learning performance of different models. The navigation success rate is used to measure the robot's ability to complete navigation tasks under random starting point and random target point conditions. It is defined as the proportion of rounds in which the target point is successfully reached out of the total number of verification rounds, and its calculation method is as follows: in This indicates the number of rounds in which the target point was successfully reached. This indicates the total number of verification rounds.

[0058] Figure 4 The average reward curves for different ablation models during training are presented. Overall, all models are in the exploratory phase during the early training stages, and the robot has not yet developed a stable obstacle avoidance and target approach strategy. Therefore, the average reward for the first 100 rounds is generally negative and fluctuates significantly. Because the reward function provides a large positive reward for successfully reaching the target, while imposing strong penalties for collisions, entering dangerous areas, and unsafe actions, even small changes in the number of successful or failed rounds within the window can cause significant fluctuations in the average reward. Therefore, the phased decline in the figure does not entirely represent policy degradation; it is also affected by exploratory actions and the proportion of successful, collision-, and dangerous termination rounds within the window.

[0059] The full MTSMR model exhibits significant fluctuations during the first 140 training epochs, but the average reward increases rapidly after epoch 160, reaching an average reward of 613.57 between epochs 181 and 200, turning from negative to positive. The MTSMR-SD model shows a substantial decrease around epoch 120, with the lowest average reward being [missing value]. The initial reward was 2956.56, which recovered rapidly in the later stages of training, with an average reward of 661.23 from rounds 181 to 200, slightly higher than the full model. However, its training results fluctuated more significantly compared to the full model, and its average reward remained slightly lower than the full model during the later convergence phase. This indicates that introducing multi-temporal-scale reinforcement learning can improve the stability and efficiency of learning. This improvement is mainly due to the fact that multi-temporal-scale value updates propagate both the short-term feedback from local obstacle avoidance and the long-term rewards from goal convergence, achieving a balance between rapid adaptation and long-term goals.

[0060] In the comparison between MTSMR and MTSMR-FM, the latter exhibits slower average reward growth and a significantly lower average reward during the convergence phase compared to MTSMR. This indicates that a fixed replay mode struggles to adapt to the constantly changing value propagation requirements during training, while a dynamic replay adjustment mechanism can improve value propagation efficiency and training stability. Specifically, when the accumulated TD error is large, the model needs to add backward replay to propagate the target reward or collision penalty rapidly along the predecessor state; when the value estimation gradually stabilizes, forward replay needs to be added to maintain the experience coverage and avoid over-updating a few experiences. The full MTSMR can dynamically adjust the selection probability of the two replay modes based on the accumulated multi-timescale TD error, thereby achieving a more reasonable balance between value propagation efficiency and training stability.

[0061] In the comparison between MTSMR and MTSMR-US, the average reward of MTSMR-US consistently remained at a lower level, approximately... 2391.44 to The fluctuation between 1494.06 and 1494.06 indicates that structured memory-prioritized sampling is key to improving sample utilization. Structured memory-prioritized replay can prioritize the replay of critical information such as collision failures, near-obstacle adjustments, and turning in narrow areas. After uniform random sampling in MTSMR-US, key experiences are easily diluted by a large number of ordinary state transitions. At the same time, the structural relationships utilized by forward and backward replays are difficult to play, resulting in slow propagation of reward and punishment information and the failure to form a stable and effective navigation strategy in the later stages of training.

[0062] In addition, from Figure 5 As can be seen, the success rates of each algorithm are as follows: MTSMR reaches 94% (best), MTSMR-SD is 90%, MTSMR-FM is 60%, while MTSMR-US is only 2% (worst). In summary, the ablation experiment results fully demonstrate that introducing structured memory priority playback, multi-timescale reinforcement learning, and dynamic playback control mechanisms can effectively improve the robot's learning efficiency.

[0063] Algorithm Comparison Experiment To further verify the overall performance advantages of the proposed MTSMR model in mobile robot navigation tasks, a comparative experiment was conducted with three algorithms: SFMA, DDPGPER, and TD3PER. SFMA is a structured playback model inspired by the hippocampal memory playback mechanism, capable of generating playback sequences based on experience and environmental structure. DDPGPER introduces a priority experience playback mechanism into the DDPG algorithm, while TD3PER combines a priority experience playback mechanism with the TD3 algorithm. These algorithms represent structured memory playback methods and deep reinforcement learning methods based on priority experience playback, respectively, thus comprehensively verifying the advantages of the proposed method in path planning efficiency and dynamic environment adaptability.

[0064] Algorithm comparison experiments were conducted in both static and dynamic environments. The static experimental environment is as follows: Figure 3 As shown in Table 1, the parameter settings for each comparison algorithm are as follows. To ensure the fairness of the comparison experiment, all algorithms were trained and validated under the same starting position, target position, obstacle layout, reward function, maximum number of training rounds, and success criteria. Each algorithm was trained for 200 rounds. After training, 50 independent validation rounds were performed. During the validation phase, no parameter updates were performed; the navigation task was executed using only the trained strategy. A validation round was considered successful when the robot reached the target area without collision within the preset maximum number of steps; if the robot collided with an obstacle or failed to reach the target point within the preset time, the validation round was considered a failure. The results of the comparison experiment are shown in Table 2.

[0065] Table 2 Results of Static Simulation Experiment As shown in Table 2, the average path lengths of MTSMR, DDPGPER, SFMA, and TD3PER are 850.55, 861.05, 905.12, and 842.63 cm, respectively. Among them, MTSMR has the shortest path, followed by DDPGPER, then SFMA, and finally TD3PER. In terms of success rate, MTSMR reached 94%, while SFMA, DDPGPER, and TD3PER achieved 90%, 92%, and 78%, respectively. Specifically, DDPGPER combines DDPG, which is suitable for continuous control, with priority empirical replay based on TD error, thus achieving good results; however, its single timescale value estimation and TD error-driven sample selection do not explicitly utilize environmental structural relationships, limiting the decision stability under complex conditions. SFMA organizes the replay sequence based on empirical strength, state structural similarity, and inhibition mechanisms, and can utilize environmental structure to promote spatial learning; however, its single timescale value representation cannot simultaneously take into account immediate obstacle avoidance and long-term goal approach. In this embodiment, TD3PER's average path length and verification success rate are lower than those of other comparative algorithms, indicating that under the current discrete navigation state space, sample size, and training configuration, this method is relatively less adaptable to this task.

[0066] Furthermore, to further verify the advantages of the MTSMR algorithm, this invention conducted comparative experiments in a dynamic environment, such as... Figure 3 As shown in (b), the dynamic environment adds dynamic obstacles to the static obstacle environment. The dynamic obstacles move along preset trajectories within the environment. Two TurtleBot3 robots are added as dynamic obstacles to the static environment; the first robot is fixed in place. place, with speed along Reciprocating motion, the second stage Centered on a circle with a radius of 0.35m, the object moves in a circle at an angular velocity of 0.085 rad / s, corresponding to a linear velocity of approximately 0.030 m / s. The pose of the dynamic obstacle is updated at a frequency of 15 Hz, thus creating two types of dynamic interference: lateral crossing and curvilinear motion, used to evaluate the algorithm's collision avoidance and navigation capabilities in time-varying obstacle environments. This invention also uses average path length and verification success rate as evaluation metrics in dynamic environments. The experimental results are shown in Table 3.

[0067] Table 3 Results of Dynamic Simulation Experiment As shown in Table 3, compared to the static environment, except for the SFMA algorithm, the average path length of all other algorithms increased and the verification success rate decreased in the dynamic environment. This is because the presence of dynamic obstacles increases the uncertainty of the navigation task. The robot not only needs to approach the target, but also needs to adjust the path in a timely manner according to the positional changes of dynamic obstacles to avoid collisions. Although the dynamic environment significantly increases the difficulty of the navigation task, MTSMR still achieved the highest verification success rate and the shortest average path length.

[0068] In summary, the method proposed in this invention has significant advantages and applicability in navigation tasks.

[0069] Physical experiment To verify the practical deployment capability of the proposed method, navigation experiments were conducted on the AutolaborPro1 (AP1) mobile robot. The AP1 is equipped with an AMD Ryzen 3 3200G processor and 8GB of memory, running Ubuntu 18.04. This application deploys the model obtained in the training environment onto the AP1 robot. Each algorithm is first trained in a simulation environment, and then the corresponding model is deployed to the AP1. During testing, only policy inference is performed; no online training or parameter updates are conducted. To ensure the control process is independent, each algorithm runs sequentially, and only one control node is allowed to send commands to the chassis at any given time.

[0070] During testing, AP1 acquired LiDAR data and robot pose via the ` / scan_1` and ` / odom` topics. Initial odometry readings were mapped to the training environment coordinate system, and the target position was set using ROS parameters. Each algorithm processed the sensor data according to its own state representation and decision-making mechanism, converting the selected action into linear and angular velocities, which were then sent to the chassis controller via the ` / cmd_vel` topic. This process did not invoke `move_base` or the global path planner; therefore, robot motion was entirely determined by the corresponding learning strategy. In the experiment, all algorithms were tested in the same corridor environment, initial pose, and target area. The default navigation task was to move from (0, 0) to (8.5, 1.2). To reduce execution errors caused by chassis response latency and motor dead zones, AP1 used odometry feedback to control the steering angle and single-step forward distance. The linear velocity was set to 0.22 m / s, the angular velocity was limited to 0.25-0.95 rad / s, and the underlying velocity commands were updated at a frequency of approximately 25 Hz. An emergency stop is triggered when the minimum clearance between the robot and an obstacle is no greater than 0.08m. Navigation is considered successful after entering the designated target area. The physical experimental environment is basically the same as the simulation environment, using a dynamic physical environment with another AP1 robot and a human as dynamic obstacles. See the physical experimental environment details below. Figure 6 .

[0071] Each algorithm underwent five physical experiments, primarily comparing their performance on navigation trajectories. The evaluation metrics used were distance traveled and time taken for navigation. Figure 7 This demonstrates a complete execution path of the MTSMR algorithm in both static and dynamic environments.

[0072] Figure 8 and Figure 9 The navigation trajectories of the algorithm in this application, SFMA, DDPGPER, and TD3PER, are shown in static and dynamic conditions. Tables 4 and 5 show the average path length and average motion time of the different algorithms in five experiments in static and dynamic environments, respectively.

[0073] Table 4 Experimental results under static physical environment

[0074] Table 5 Experimental results under dynamic physical environment

[0075] This demonstrates that MTSMR not only achieves the shortest navigation path among all compared algorithms but also has the shortest task completion time. This advantage primarily stems from the multi-timescale spatial memory playback mechanism: value estimation at different timescales simultaneously considers local obstacle avoidance and long-term goal approach, while structured experience playback improves the utilization efficiency and decision stability of key navigation experiences. Therefore, MTSMR can reduce ineffective turns and unnecessary detours in narrow areas, reducing energy consumption and ensuring navigation safety while shortening travel distance and time. In summary, experimental results show that AP1, equipped with the algorithm proposed in this application, exhibits superior performance in navigation tasks.

[0076] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0077] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of this application.

[0078] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A brain-like multi-temporal-scale structured memory playback method, characterized in that, Includes the following steps: S1. Based on the memory replay mechanism of the entorhinal cortex and hippocampus in the brain, a structured memory-priority replay model is proposed. The structured memory-priority replay model divides the experience replay process into two main stages: experience selection and value update. In the experience selection stage, two structured replay modes, forward replay and backward replay, are constructed. Forward replay is used to maintain the coverage of recent experiences, while backward replay is used to propagate the target reward or collision penalty backward along the predecessor state, thereby improving the utilization efficiency of key experiences. In the value update stage, a multi-discount factor value learning mechanism is introduced to maintain independent action value functions for different time scales, and multiple sets of TD errors are calculated in parallel for the same replay experience, so that it can participate in the updating of short-term immediate benefits and long-term target values ​​at the same time. S2. Based on the multi-timescale TD error, a dynamic playback mode selection strategy is designed to adaptively adjust the playback probability of forward playback mode and reverse playback mode. The structured memory priority playback model enhances the reverse playback mode to accelerate the reverse propagation of key value information. When the value estimation tends to stabilize, the structured memory-first replay model adopts a forward replay mode to maintain experience coverage and training stability.

2. The brain-like multi-temporal-scale structured memory playback method according to claim 1, characterized in that, The structured memory-first replay model organizes stored experiences into structured memories with temporal order and state transition relationships. During the interaction process, the structured memory-first replay model continuously stores information experienced by the robot, including states, actions, rewards, and the next state. It selects an appropriate replay mode based on the current learning state, enabling the agent to more efficiently utilize key experiences, including successfully reaching the target, collision failures, and near-obstacle adjustments, with limited interaction samples, thereby improving navigation success rate and strategy adaptability.

3. The brain-like multi-temporal-scale structured memory playback method according to claim 2, characterized in that, The workflow of the structured memory-first replay model includes: 1) In the initial exploration phase, the robot acquires real-time environmental information through sensors including lidar and odometry. Through continuous exploration, the structured memory priority playback model gradually accumulates a set of structured experience containing action states and environmental information, providing a foundation for subsequent training. 2) During the training phase, the structured memory priority replay model uses the experience stored in the experience pool to perform structured replay and multi-timescale value updates, gradually forming a navigation strategy that takes into account both local security and long-term goal convergence. 3) In the actual verification phase, the multi-timescale action value function obtained from training is used for robot navigation decision-making. The robot constructs the current navigation state based on real-time sensor data and uses the learned value function to select the optimal discrete action to complete the target navigation task.

4. The brain-like multi-temporal-scale structured memory playback method according to claim 1, characterized in that, The structured memory priority playback model is driven by both experiential value and environmental structure. While focusing on memory strength, the structured memory priority playback model will prioritize the playback of experiences in areas near obstacles, areas near targets, narrow passages, or turning areas.

5. The brain-like multi-temporal-scale structured memory playback method according to claim 1, characterized in that, In the structured memory-priority replay model, experience replay can be analogized to the reactivation of place cells in a memory replay event. When an experience is reactivated, each stored experience is assigned a priority score based on memory strength, structural association strength with the current experience, and state inhibition level. The evaluation metrics are as follows: ； in, Indicates memory strength, influenced by frequency of experience. and reward size To decide, For memory strength and structural correlation strength The trade-off coefficients between these factors are used to evaluate the robot and select experiences applicable to the current environment for playback.

6. The brain-like multi-temporal-scale structured memory playback method according to claim 5, characterized in that, Strength of structural correlation between experiences Used to describe current experience and storage experience The structural correlation strength between them is calculated using the following formula: ； in It represents the structural distance between two experiences. The smaller the structural distance, the closer the two experiences are in terms of state or goal, and the stronger the structural association between them. The greater the structural distance, the weaker the connection between the two experiences.

7. The brain-like multi-temporal-scale structured memory playback method according to claim 6, characterized in that, Since structural distance is not equivalent to simple physical distance, in the structured memory-first replay model, the structural representation is constructed based on the reachability relationship between states as follows: ； ； in, It is the identity matrix. As a discount factor, This is the state transition matrix under the default behavior strategy.

8. The brain-like multi-temporal-scale structured memory playback method according to claim 5, characterized in that, In the structured memory-first playback model, the inhibition regulation term is defined in an exponential decay form: ； in, To suppress the intensity adjustment parameter, As an inhibitor, it is used to prevent a few high-intensity experiences from being repeatedly activated in a short period of time. When it is large, A smaller value indicates that the experience has been recently replayed, and the probability of it being reactivated should be lower. After decaying over time, As the experience increases, the chances of it being used in the replay selection also increase.

9. The brain-like multi-temporal-scale structured memory playback method according to claim 5, characterized in that, In the structured memory-priority replay model, candidate experiences are normalized to sampling probabilities based on priority scoring: in, This is to prevent extremely small constants where the denominator is zero. The structured memory priority replay model selects the experience batches to be replayed from the experience pool based on the sampling probability.

10. A brain-like multi-temporal-scale structured memory playback device, characterized in that, include: One or more processors; A computer-readable medium for storing one or more computer-readable instructions; When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement a neuromorphic multi-timescale structured memory playback method as described in any one of claims 1 to 9.

Images