Target navigation method based on meta-learning and hippocampus reverse playback working mechanism

Abstract

The invention provides a target navigation method based on meta learning and a hippocampus reverse playback working mechanism, and aims to solve the problems of poor interpretability, low navigation efficiency and poor adaptability of a traditional target-oriented navigation method in a multi-task environment. The specific process comprises the following steps: constructing a two-dimensional spatial position cell neural network by referring to a reverse playback and rehearsal working mechanism during a rat hippocampus sharp ripple period; optimizing the initial synaptic connection weight of the neural network through meta learning; and target-oriented navigation is carried out by using the neural network weight after meta-learning optimization. The method is suitable for robot target-oriented autonomous navigation in a multi-task environment, and efficient and highly adaptive target-oriented navigation can be realized by simulating a hippocampus reverse playback and rehearsal working mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of bionic technology and robot navigation, and specifically relates to a target navigation method based on meta-learning and hippocampal reverse replay working mechanism. Background Art

[0002] Traditional robot navigation methods mainly rely on perception technologies such as vision and lidar, which perform well under certain conditions. However, in complex environments, these methods often face problems of efficiency and adaptability, especially in dynamic environments or new task scenarios. Traditional models learn slowly, making it difficult to make accurate decisions in a short period of time, and it is difficult to cope with diverse task requirements through effective knowledge transfer, resulting in a significant reduction in the autonomy and efficiency of robots in complex tasks. The role of the hippocampus in the brain is crucial, especially in spatial cognition, memory formation, and path planning. Hippocampal place cells respond to the individual's position in the environment to form an intrinsic spatial position expression, helping animals to navigate and plan paths. However, how to efficiently utilize this biological mechanism in robot navigation systems to cope with complex and changing environments remains an important challenge in the field of robot navigation.

[0003] In order to solve the above problems, some studies in recent years have tried to improve the decision-making ability of robots in complex environments by simulating the learning mechanism of the brain. As an important process of brain learning and memory consolidation, the reverse replay mechanism has been applied to path planning and target navigation learning. Reverse replay refers to the hippocampus replaying the individual's learning experience when the animal is in a sleeping or quiet state, helping to consolidate memory and improve learning efficiency. However, the existing reverse replay model is usually only applied in one-dimensional or simple environments, and its bias towards the nearest path limits its navigation efficiency in variable and complex two-dimensional environments. Preview is another important feature of the hippocampus's working mechanism, which means that when the animal realizes that there is a target at a certain location, it simulates the path to the target in advance. This mechanism enables animals to find the optimal path quickly and efficiently, and can effectively navigate even in a completely new environment. Combined with reverse replay, the preview function further enhances the individual's ability to navigate towards the target, especially in complex environments. In this context, meta-learning, as a learning method that can quickly adapt to new tasks, has received widespread attention in the field of neural networks and cognitive computing in recent years. The core idea of ​​meta-learning is to enable the algorithm to quickly adjust its learning strategy through a small amount of experience so as to quickly adapt to a changing environment. This idea is consistent with the memory updating and rapid learning ability of the hippocampus in spatial navigation.

[0004] Although working mechanisms such as reverse replay and rehearsal have shown great potential in simulating the learning process of the brain, how to effectively combine these mechanisms and efficiently apply them in robot navigation systems is still a difficult problem in current research. Therefore, designing an efficient, explainable and adaptable bionic navigation system that can integrate meta-learning with the reverse replay and rehearsal working mechanisms of hippocampal place cells to cope with complex and dynamic environments is a core problem that needs to be solved in the field of robot navigation. Summary of the invention

[0005] The present invention aims to provide a target navigation method based on meta-learning and hippocampal reverse replay mechanism, which simulates the physiological mechanism of reverse replay and rehearsal in the rat hippocampus during the sharp wave ripple period and combines meta-learning technology to improve the robot's rapid adaptability and learning efficiency in a multi-task environment, and realizes target-oriented autonomous navigation in a multi-task dynamic environment. The invention aims to solve the following three key problems:

[0006] 1. Weak interpretability of navigation decisions: Traditional goal-oriented navigation methods (such as reinforcement learning) usually focus on improving application performance, but their decision-making process is often difficult to explain. This makes the robot lack sufficient transparency and trust when performing tasks.

[0007] 2. Low efficiency of goal-oriented navigation: Traditional goal-oriented navigation methods face efficiency bottlenecks, especially in complex environments, where robots often need to go through multiple attempts to find the optimal path.

[0008] 3. Poor adaptability of multi-task navigation: When faced with multiple tasks, traditional navigation methods often require separate learning for each new task, which is inefficient.

[0009] In order to solve the above problems, the present invention proposes a target navigation method based on meta-learning and hippocampal reverse replay working mechanism. First, it is assumed here that when rats are in the period of sharp wave ripples (SWRs), hippocampal reverse replay will promote the occurrence of the preview working mechanism, and construct a place cell neural network suitable for two-dimensional space, thereby enhancing the interpretability of the robot's navigation decision. Secondly, after the robot reaches the target position, the place cells representing the target position are activated in a short period of time to form a neural sequence that propagates evenly in all directions to adjust the weights of the neurons, ensuring that the robot can "preview" the future path biased towards the target position during the next exploration like a rat, thereby quickly adapting to a single task and reducing the time required for repeated exploration. Finally, combined with meta-learning, experience sharing and knowledge transfer of multi-task navigation are realized, so that the neural network can effectively use existing knowledge when the task changes, significantly shorten the time to explore the new target position, avoid learning from scratch, and improve the adaptability of the robot's target-oriented navigation.

[0010] The specific workflow of the present invention is as follows:

[0011] Step 1: Construct a two-dimensional spatial place cell neural network.

[0012] Step 2: Optimize the initial synaptic connection weights of the neural network through meta-learning.

[0013] Step 3: Use the neural network weights optimized by meta-learning to navigate toward the target.

[0014] The specific steps are as follows:

[0015] Step 1: Construct a two-dimensional place cell neural network

[0016] The neural network is composed of position cell neurons, which evenly cover the two-dimensional space where the robot needs to navigate. The membrane potential I of neuron i i It is described by a first-order decaying differential equation, based on the excitatory input of the synapses connecting adjacent neurons and location-specific excitatory input and increases through the global suppression term I in h And reduce:

[0017]

[0018] τ = 10ms. The activation rate of the place cell can be obtained by the threshold linear function: i ′:

[0019] r i ′(t)=α(I i (t)-∈)

[0020] α=1,∈=0.02. i ′Through linear rectification, the final activation rate r is obtained i :

[0021]

[0022] Global Inhibit Item I in h It is expressed as:

[0023]

[0024] τ ih h =10ms,ω in h =0.0005. i ,F i is the short-term synaptic plasticity variable of neuron i. i is the short-term inhibition term of neuron i, expressed as:

[0025]

[0026] τ STD =300ms. i is the short-term facilitation variable of neuron i, expressed as:

[0027]

[0028] τ STF =200ms. U = 0.4. It is expressed by Gaussian distribution:

[0029]

[0030] d=2,(x i ,y i ) is the i-th position, which is the discharge field center of the cell in two-dimensional space, and (x, y) is the real-time position of the robot in two-dimensional space during target navigation. When the robot is moving, C = 5HZ; when the robot reaches the target position and stops, C = 1HZ. Position cell neuron i receives synaptic excitatory input from the 8 adjacent neurons connected to it. It is expressed as:

[0031]

[0032] w ij is the synaptic connection weight between neuron j and neuron i.

[0033] Step 2: Optimize the initial synaptic connection weights of the neural network through meta-learning

[0034] Use MAML (Model-Agnostic Meta-Learning)-meta learning algorithm for optimization.

[0035] (1) Inner layer update

[0036] Randomly initialize the synaptic connection weights w between neurons ij , select any target position (x target ,y target ), the robot’s starting position is random, each time step is defined as 1ms, and the robot is started.

[0037] Exploration phase: At each time step, update the robot's real-time position. Update the movement direction according to the neuron activity vector a

[0038]

[0039] Rotate the velocity vector V to the direction of the activity vector:

[0040]

[0041] represents normalizing the activity vector a to a unit vector, γ a is the weight along the direction of the activity vector, γ a =0.01,γ noise is the noise intensity, γ noise =0.05.

[0042] Change the robot's real-time position (x, y) by the velocity vector V:

[0043]

[0044] γ v represents the fixed movement speed of the robot, γ v =0.01.

[0045] Replay phase: When the robot reaches the target position, it stays there for 15 seconds and starts reverse replay. By stimulating the place cells encoding the target position for a long time, the network triggers a sequence that propagates evenly in all directions starting from the target position:

[0046]

[0047] Update the place cell related variables r at each time step i , D i , F i , and adjust the weight w ij

[0048]

[0049] η=1,τ w = 30s. Normalize the updated weights:

[0050]

[0051] w ij ′ is the normalized weight. When the inner layer is updated, the final weight between neurons is (2) External layer update

[0052] After the inner layer update is completed, navigate to the target position selected by the inner layer update 5 times. As the initial connection weights between neurons before each navigation, the robot's starting position is random, and the loss value Loss of each navigation is calculated:

[0053] Loss = t actual -t ideal

[0054] t actual is the actual time used, tideal is the ideal time, that is, the robot moves at a constant speed γ v The time it takes to move from the starting point to the target point in a straight line:

[0055]

[0056] P start is the random starting position of the robot, P end is the target position, D(P start ,P end ) represents the straight-line distance from the starting position to the target position.

[0057] Calculate the average loss of 5 navigations avg :

[0058]

[0059] Loss and replay time t replay The relationship between Loss(t replay )for:

[0060]

[0061] b=154.36,c=-0.1,d=1.64。 Loss is calculated by the functional relationship expressed in the above formula avg Update the outer replay time relative to the gradient of the replay time:

[0062]

[0063] t old The initial value of is 0, β is the outer layer update learning rate, β = 0.01, represents the gradient of the average loss with respect to the replay time. new is the updated outer layer replay time, which is obtained by replaying t in the reverse direction at the selected target position. new Time, at each time step, update the position cell related variables r i , D i , F i , and update the initial synaptic connection weights between place cells And normalize:

[0064]

[0065] τ w =30s, It is the parameter to be optimized by meta-learning, and its initial value is a random weight.

[0066] (3) Termination conditions

[0067] Set the maximum number of optimizations to 10,000, select a target position at random each time and repeat (1) (2). When the predetermined maximum number of optimizations is reached, the network weight optimization is considered complete. Record the synaptic connection weights of the place cells at this time.

[0068] Step 3: Use the weights of the neural network optimized by meta-learning to navigate toward the target

[0069] The above model optimized by meta-learning has realized the transfer of experience. The robot can select any target position in the two-dimensional space. The synaptic connection weights of the initial position cells are used for navigation and movement toward the selected target position.

[0070] The present invention has the following advantages:

[0071] (1) Enhance the interpretability of the navigation process:

[0072] The present invention constructs a two-dimensional spatial place cell neural network by simulating the reverse replay and preview working mechanism of rat hippocampal place cells. The neural network improves the transparency and trustworthiness of the robot in target-oriented navigation tasks.

[0073] (2) Improve the efficiency of target-oriented navigation:

[0074] The present invention applies the reverse replay working mechanism of two-dimensional space position cells to the inner layer update process of meta-learning. In the inner layer update, only a small amount of training is required to enable the robot to find the optimal path to the target position. Compared with the traditional navigation model that requires multiple trainings to find the optimal path, the present invention greatly improves the navigation efficiency and training efficiency, reduces power consumption and saves resources.

[0075] (3) Enhanced adaptability of multi-task navigation:

[0076] Through meta-learning, the present invention realizes experience sharing and knowledge transfer between tasks in multi-task navigation, allowing the robot to quickly adapt to new tasks and reduce exploration time. This mechanism avoids the process of learning from "zero" every time a new task is performed, effectively improving the adaptability of robot navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Meta-learning training flow chart

[0078] Figure 2 Schematic diagram of the synaptic connections of place cells

[0079] Figure 3 Reverse playback diagram

[0080] Figure 4 Target location diagram

[0081] Figure 5 Robot movement diagram DETAILED DESCRIPTION

[0082] In order to describe the technical solution of the invention in more detail, the present invention is further described below with reference to the accompanying drawings and examples.

[0083] like Figure 1 As shown, the specific steps of the present invention are as follows:

[0084] 1. Constructing a two-dimensional spatial place cell neural network

[0085] The present invention constructs a place cell neuron network suitable for two-dimensional space by simulating the hippocampal reverse replay and preview physiological phenomena that occur during the sharp wave ripples after rats obtain rewards.

[0086] 2. Meta-learning inner layer update process

[0087] The inner update process of meta-learning is a local training step performed for the current task, so that the model can quickly adapt to the current task. In this process, the model extracts a small number of samples from the task for iterative training. During training, when the robot reaches the target position, reverse replay is performed to adjust the synaptic weights of the place cell network. Considering that the hippocampal rehearsal working mechanism during SWRs requires selective strengthening of the synaptic connection weights of neurons on the path from the random starting position to the target position, the present invention realizes the learning function by combining Hebbian plasticity with short-term plasticity. The network can simulate the physiological mechanism of the place cell sequence triggered from the target position to spread evenly in all directions and further support the hippocampal rehearsal working mechanism.

[0088] 3. Meta-learning outer layer update process

[0089] The outer update process of meta-learning is to improve its learning ability and adaptation speed in new tasks by comprehensively optimizing model parameters on multiple tasks. In the outer update, the algorithm calculates the total loss and adjusts the initial parameters based on the inner update results of each task, so that the model can quickly adapt when facing new tasks. Since it is difficult to calculate the gradient of the loss value relative to the synaptic connection weights of the place cell network, the present invention chooses to use the replay time as an intermediate quantity, updates the replay time by calculating the gradient of the replay time relative to the loss, and then updates the synaptic connection weights of the place cell network by reverse replay to achieve the update of the initial parameters.

[0090] 4. Target navigation

[0091] Use the updated synaptic connection weights of the neural network to guide the robot to perform a goal-oriented navigation task.

[0092] The above description of the specific implementation method of the present invention is only used for those skilled in the art to understand the present invention and is not used to limit the scope of the present invention.

Claims

1. A target navigation method based on meta-learning and hippocampal reverse replay working mechanism, characterized in that: The method comprises the following steps: Step 1, construct a two-dimensional spatial place cell neural network; Step 2: Optimize the initial synaptic connection weights of the neural network through meta-learning; Step 3: Use the neural network weights optimized by meta-learning to navigate toward the target.

2. The method according to claim 1, characterized in that Step 1 of constructing a two-dimensional spatial place cell neural network is as follows: The neural network is composed of position cell neurons, which evenly cover the two-dimensional space where the robot needs to navigate. The membrane potential I of neuron i i It is described by a first-order decaying differential equation, based on the excitatory input of the synapses connecting adjacent neurons and location-specific excitatory input and increases through the global suppression term I inh And reduce: τ = 10ms; the activation rate of the place cell is obtained by the threshold linear function r i ′: r i ′(t)=α(I i (t)-v) α=1,∈=0.02;r i ′Through linear rectification, the final activation rate r is obtained i : Global Inhibit Item I inh It is expressed as: τ inh =10ms,ω inh =0.0005; D i ,F i is the short-term synaptic plasticity variable of neuron i; where D i is the short-term inhibition term of neuron i, expressed as: τ STD =300ms; F i is the short-term facilitation variable of neuron i, expressed as: τ STF =200ms; U = 0.4; It is expressed by Gaussian distribution: d=2,(x i ,y i ) is the i-th position, which is the discharge field center of the cell in two-dimensional space. (x, y) is the real-time position of the robot in two-dimensional space during target navigation. When the robot is moving, C = 5HZ. When the robot reaches the target position and stops, C = 1HZ. Position cell neuron i receives synaptic excitatory input from the eight adjacent neurons connected to it. It is expressed as: w ij is the synaptic connection weight between neuron j and neuron i.

3. A target navigation method based on meta-learning and hippocampal reverse replay working mechanism as claimed in claim 1, characterized in that: Step 2 describes the optimization of the initial synaptic connection weights of the neural network through meta-learning; specifically, as follows: Use MAML (Model-Agnostic Meta-Learning)-meta learning algorithm for optimization; (1) Inner layer update Randomly initialize the synaptic connection weights w between neurons ij , select any target position (x target ,y target ), the robot's starting position is random, each time step is defined as 1ms, and the robot is started; Exploration phase: at each time step, update the robot's real-time position; update the movement direction according to the neuron activity vector a; Rotate the velocity vector V to the direction of the activity vector: represents normalizing the activity vector a to a unit vector, γ a is the weight along the direction of the activity vector, γ a =0.01,γ noise is the noise intensity, γ noise =0.05; Change the robot's real-time position (x, y) by the velocity vector V: γ v represents the fixed movement speed of the robot, γ v =0.01; Replay phase: When the robot reaches the target position, it stays there for 15 seconds and starts reverse replay; by stimulating the position cells encoding the target position for a long time, the network triggers a sequence that propagates evenly in all directions starting from the target position: Update the place cell related variables r at each time step i , D i , F i , and adjust the weight w ij τ w = 30s; Normalize the updated weights: w ij ′ is the normalized weight; the weight between neurons at the end of the inner layer update is (2) External layer update After the inner layer update is completed, navigate to the target position selected by the inner layer update 5 times. As the initial connection weights between neurons before each navigation, the robot's starting position is random, and the loss value Loss of each navigation is calculated: Loss=t actual -t ideal t actual is the actual time used, t ideal is the ideal time, that is, the robot moves at a constant speed γ v The time it takes to move from the starting point to the target point in a straight line: P start is the random starting position of the robot, P end is the target position, D(P start ,P end ) represents the straight-line distance from the starting position to the target position; Calculate the average loss of 5 navigations avg : Loss and replay time t replay The relationship between Loss(t replay )for: b = 154.36, c = -0.1, d = 1.64; loss is calculated using the functional relationship expressed in the above formula avg Update the outer replay time relative to the gradient of the replay time: t old The initial value of is 0, β is the outer layer update learning rate, β = 0.01, represents the gradient of the average loss with respect to the replay time; t new is the updated outer layer replay time, which is obtained by replaying t in the reverse direction at the selected target position. new Time, at each time step, update the position cell related variables r i , D i , F i , and update the initial synaptic connection weights between place cells And normalize: τ w =30s, The parameters to be optimized by meta-learning, with the initial value being a random weight; (3) Termination conditions Set the maximum number of optimizations to 10,000, select a target position at random each time and repeat (1) (2). When the predetermined maximum number of optimizations is reached, the network weight optimization is considered complete; record the synaptic connection weights of the place cells at this time 4. The method according to claim 1, characterized in that The neural network weights optimized by meta-learning described in step 3 are used for target navigation; specifically, as follows: The above model optimized by meta-learning has realized the transfer of experience. The robot can select any target position in the two-dimensional space. The synaptic connection weights of the initial position cells are used for navigation and movement toward the selected target position.

Images