A mobile robot behavior decision-making method capable of incremental continuous learning

Abstract

A mobile robot behavior decision-making method capable of incremental continuous learning interacts with the environment by means of reinforcement learning, and learns the current decision online at the same time. By simulating the memory function of the human brain, the mobile robot has the ability to adapt to a changing environment. At the same time, an internal reward mechanism driven by curiosity indicators is designed to simulate the reinforcement mechanism of curiosity for human memory, realize the memory encoding of unfamiliar behavior decisions by the mobile robot, and the memory consolidation of commonly used behavior decisions, thereby improving the learning and memory ability of the mobile robot in environmental cognition. The present invention adds incremental learning to the original mobile robot decision-making method, so that the mobile robot can better adapt to navigation decision-making tasks in complex dynamic environments.

Description

Technical Field

[0001] The present invention relates to the field of mobile robot behavior decision-making methods, and in particular to a mobile robot behavior decision-making method capable of performing incremental continuous learning. Background Art

[0002] With the continuous development and updating of artificial intelligence technology, mobile robots have been widely used in the fields of industry, medicine, military, service industry, agriculture, etc. A mobile robot trained in a certain environment can perform tasks well in that environment, but when the environment changes or when performing tasks in a new environment, due to the incomplete cognition of the new environment, the decision-making effect of the mobile robot is low and the task cannot be completed well. Higher organisms, including humans, have good adaptability to complex environments, can gradually accumulate learned knowledge, and can complete current tasks based on recall of previously learned experience, while acquiring new knowledge and abilities. Therefore, studying the robot's bionic environmental cognition, learning and memory abilities, enabling robots to learn throughout their lives, and guiding future behavioral decisions through the memory and consolidation of behavioral decisions, is an important content faced in robot research and application.

[0003] With the continuous development of artificial intelligence technology, more and more intelligent algorithms are being applied to the memory behavior decision-making of mobile robots, such as deep reinforcement learning network, adaptive resonance neural network, particle swarm optimization algorithm, memory neural network, long short-term memory reinforcement learning, etc. These algorithms find the optimal solution through learning or searching, making the robot's decision more intelligent. However, these algorithms have defects such as fixed knowledge base and poor generalization ability, and the knowledge learned by the robot in the current environment cannot guide its behavior decision-making in the future environment. Summary of the invention

[0004] The purpose of the present invention is to provide a mobile robot behavior decision-making method capable of incremental continuous learning, to achieve the memory encoding of unfamiliar behavior decisions by the mobile robot, and to consolidate the memory of commonly used behavior decisions, thereby improving the learning and memory ability of the mobile robot in environmental cognition.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a mobile robot behavior decision method capable of incremental continuous learning, wherein the mobile robot continuously detects the external environment and obtains image information during operation, and the decision method divides the operation process of the mobile robot into multiple decision cycles, and sequentially gives the behavior decision of each decision cycle according to the image information, thereby controlling the mobile robot to move in accordance with the behavior decisions of the multiple decision cycles in sequence;

[0006] In each decision cycle, the hippocampus receives the input image at the current moment and obtains the pre-selected decision based on memory. a m , and obtains a probability-based pre-selection decision through the anterior cingulate cortex a p ;

[0007] The hippocampus is an incremental self-organizing neural network, including input layer X, pooling layer, intermediate layer Y and output layer Z. Each neuron in the intermediate layer Y stores its own memory information, and each memory information is obtained through perceptual information. o , Status Information s and decision-making information a The triplet composed of o , s , a >, where perception information o Pre-stored external environment image information, status information s is the spatial coordinate of the mobile robot, decision information a is the action instruction of the mobile robot; the input layer X can receive the input image and pass it to the pooling layer, which can extract features and reduce the dimension of the input image to obtain the external perception. o m And pass it to the middle layer Y, the output layer Z can make the decision information a Output to mobile robot for motion control;

[0008] The hippocampus perceives the outside world o m A triplet of multiple memory information stored < o , s , a >Perceptual information in o Compare them respectively and get the o m The most similar perceptual information o , and then use this perception information o The triplet belongs to o , s , a Decision information in > a As a memory-based pre-selection decision a m ;

[0009] The anterior cingulate gyrus is a Q-value neural network based on state-action pairs. The anterior cingulate gyrus receives the current state information. s , the current state information s With multiple decision information a They form state-action pairs respectively ( s ,a ), and get all the status information including the current time s The Q value of the state-action pair is then used to obtain the current state information through formula (1). s Multiple decision information a The probability of execution;

[0010] Formula (1);

[0011] In formula (1), m 1 is a state-action pair ( s , a )

[0012] The anterior cingulate gyrus compares the multiple execution probabilities obtained, and then takes the decision information with the highest execution probability as the a As a probability-based pre-selection decision a p ;

[0013] When pre-selection decisions are made based on memory a m and probability-based preselection decisions a p If the same, the pre-selection decision a m or a p As the behavioral decision executed by the mobile robot in the current decision cycle;

[0014] When pre-selection decisions are made based on memory a m and probability-based preselection decisions a p At the same time, according to ε The -greedy strategy makes the behavioral decision that the mobile robot performs in the current decision cycle, and then performs hippocampal update learning and anterior cingulate gyrus update learning;

[0015] The hippocampus updates learning by transmitting the current state information through the anterior cingulate gyrus. s and probability-based pre-selection decisions a p Transmitted to the hippocampus, which perceives the outside world o m , current status information s and probability-based pre-selection decisions a p Encode to obtain a short-term memory set , n 1 is the short-term memory set e Variables in uThe number of memory similarity matrix is ​​obtained by formula (2): S lt ;

[0016] Formula (2);

[0017] In formula (2), n 2 is the number of neurons in the input layer X, l is the number of existing neurons in the middle layer Y, v i，j is the first i The neurons to the middle layer Y j The weight vector of neurons, u i For short-term memory collection e The i variables;

[0018] When the memory similarity matrix S lt When it is less than the memory judgment threshold, a new neuron is added to the middle layer Y and the external perception o m , current status information s and probability-based pre-selection decisions a p The composed triples are stored as new memory information in the added neurons, realizing memory-added learning of the neural network;

[0019] When the memory similarity matrix S lt When it is not less than the memory judgment threshold, the hippocampus obtains the prediction error through formula (3): Prediction_error ;

[0020] Formula (3);

[0021] In formula (3), E t Perceived by the outside world o m , current status information s and probability-based pre-selection decisions a p The contextual memory information composed of W t Memory information stored in the middle layer Y;

[0022] The anterior cingulate gyrus obtains cognitive conflict through equations (4) and (5): Conflict ;

[0023] Formula (4);

[0024] Formula (5);

[0025] In formula (4) and (5), m 2 is the stored decision information a The number of P i Status information for the current moment s Next i Decision information a The probability of execution;

[0026] Then the hippocampus and anterior cingulate gyrus respectively Prediction_error and cognitive conflict Conflict is sent to the prefrontal cortex, which converts the prediction error Prediction_error and cognitive conflict Conflict The larger value is selected as the environmental error Error :

[0027] Formula (6);

[0028] Then the curiosity response is obtained through formula (7): Curiosity , the anxiety response value is obtained through formula (8): Anxiety , and the adjustment factor is obtained by formula (9): μ ;

[0029] Formula (7);

[0030] Formula (8);

[0031] Formula (9);

[0032] In formulas (7), (8), and (9), C max is the preset maximum value of curiosity, A max is the preset maximum anxiety value, β is the adjustment coefficient, R Rewards for the environment;

[0033] Then, the weights of the neurons in the middle layer Y are updated by formula (10), and the updated weights replace the original weights to achieve memory consolidation learning of the neural network;

[0034] Formula (10);

[0035] In formula (10), v j The middle layer Y j The weight of a neuron, ω ( n j ) is the first j The number of activations of a neuron is n The learning rate at y j is the middle layer Y j The response value of a neuron, is the input vector, μ is the regulating factor;

[0036] The anterior cingulate update learning is to update the Q value of the state-action pair through equations (11) and (12), and replace the original Q value neural network with the updated Q value;

[0037] Formula (11);

[0038] Formula (12);

[0039] In formula (11) and (12), α is the learning rate of Q value, A Decision information stored a A collection of μ is the adjustment factor.

[0040] According to the above technical solution, the beneficial effects of the present invention are:

[0041] The present invention proposes a mobile robot behavior decision-making method capable of incremental continuous learning, interacting with the environment by means of reinforcement learning, and learning the current decision online at the same time. By simulating the memory function of the human brain, the mobile robot has the ability to adapt to a changing environment. At the same time, an internal reward mechanism driven by curiosity indicators is designed to simulate the mechanism of curiosity to strengthen human memory, realize the memory encoding of unfamiliar behavior decisions by the mobile robot, and the memory consolidation of commonly used behavior decisions, thereby improving the learning and memory ability of the mobile robot in environmental cognition. The present invention adds incremental learning to the original mobile robot decision-making method, so that the mobile robot can better adapt to navigation decision-making tasks in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is the navigation experiment path for the first, fourth and seventh static environment tests;

[0043] Figure 2 This is the average recognition error change diagram for static environment tests;

[0044] Figure 3 This is the robot knowledge transformation diagram for static environment test;

[0045] Figure 4 The first, fourth, and seventh navigation experiment paths after exchanging the starting and ending points for the static environment experiment;

[0046] Figure 5 The navigation experiment paths for the first, fourth, and seventh dynamic environment tests;

[0047] Figure 6 This is the average recognition error change diagram of the dynamic environment test. DETAILED DESCRIPTION

[0048] A mobile robot behavior decision method capable of incremental continuous learning, wherein the mobile robot continuously detects the external environment and obtains image information during operation. The decision method divides the operation process of the mobile robot into multiple decision cycles, and gives the behavior decision of each decision cycle in turn according to the image information, thereby controlling the mobile robot to move in accordance with the behavior decisions of the multiple decision cycles in turn.

[0049] In each decision cycle, the hippocampus receives the input image at the current moment and obtains the pre-selected decision based on memory. a m , and obtains a probability-based pre-selection decision through the anterior cingulate cortex a p .

[0050] The hippocampus is an incremental self-organizing neural network, including input layer X, pooling layer, intermediate layer Y and output layer Z. Each neuron in the intermediate layer Y stores its own memory information, and each memory information is obtained through perceptual information. o , Status Information s and decision-making information a The triplet composed of o , s , a >, where perception information o Pre-stored external environment image information, status information s is the spatial coordinate of the mobile robot, decision information a is the action instruction of the mobile robot; the input layer X can receive the input image and pass it to the pooling layer, which can extract features and reduce the dimension of the input image to obtain the external perception. o m And pass it to the middle layer Y, the output layer Z can make the decision information a Output to the mobile robot for motion control.

[0051] The hippocampus perceives the outside world o m A triplet of multiple memory information stored < o , s ,a >Perceptual information in o Compare them respectively and get the o m The most similar perceptual information o , and then use this perception information o The triplet belongs to o , s , a Decision information in > a As a memory-based pre-selection decision a m .

[0052] The anterior cingulate gyrus is a Q-value neural network based on state-action pairs. The anterior cingulate gyrus receives the current state information. s , the current state information s With multiple decision information a They form state-action pairs respectively ( s , a ), and get all the status information including the current time s The Q value of the state-action pair is then used to obtain the current state information through formula (1). s Multiple decision information a The probability of execution;

[0053] Formula (1);

[0054] In formula (1), m 1 is a state-action pair ( s , a ) quantity.

[0055] The anterior cingulate gyrus compares the multiple execution probabilities obtained, and then takes the decision information with the highest execution probability as the a As a probability-based pre-selection decision a p .

[0056] When pre-selection decisions are made based on memory a m and probability-based preselection decisions a p If the same, the pre-selection decision a m or a p As the behavioral decision performed by the mobile robot in the current decision cycle.

[0057] When pre-selection decisions are made based on memory a m and probability-based preselection decisions ap At the same time, according to ε The -greedy strategy makes the behavioral decision that the mobile robot performs in the current decision cycle, and then performs hippocampal update learning and anterior cingulate gyrus update learning.

[0058] The hippocampus updates learning by transmitting the current state information through the anterior cingulate gyrus. s and probability-based pre-selection decisions a p Transmitted to the hippocampus, which perceives the outside world o m , current status information s and probability-based pre-selection decisions a p Encode to obtain a short-term memory set , n 1 is the short-term memory set e Variables in u The number of memory similarity matrix is ​​obtained by formula (2): S lt ;

[0059] Formula (2);

[0060] In formula (2), n 2 is the number of neurons in the input layer X, l is the number of existing neurons in the middle layer Y, v i，j is the first i The neurons to the middle layer Y j The weight vector of neurons, u i For short-term memory collection e The i variables.

[0061] When the memory similarity matrix S lt When it is less than the memory judgment threshold, a new neuron is added to the middle layer Y and the external perception o m , current status information s and probability-based pre-selection decisions a p The composed triplets are stored in the added neurons as new memory information, realizing the memory-added learning of the neural network.

[0062] When the memory similarity matrix S lt When it is not less than the memory judgment threshold, the hippocampus obtains the prediction error through formula (3): Prediction_error ;

[0063] Formula (3);

[0064] In formula (3), E t Perceived by the outside world o m , current status information s and probability-based pre-selection decisions a p The contextual memory information composed of W t Memory information stored in the middle layer Y.

[0065] The anterior cingulate gyrus obtains cognitive conflict through equations (4) and (5): Conflict ;

[0066] Formula (4);

[0067] Formula (5);

[0068] In formula (4) and (5), m 2 is the stored decision information a The number of P i Status information for the current moment s Next i Decision information a probability of execution.

[0069] Then the hippocampus and anterior cingulate gyrus respectively Prediction_error and cognitive conflict Conflict is sent to the prefrontal cortex, which converts the prediction error Prediction_error and cognitive conflict Conflict The larger value is selected as the environmental error Error :

[0070] Formula (6).

[0071] Then the curiosity response is obtained through formula (7): Curiosity , the anxiety response value is obtained through formula (8): Anxiety , and the adjustment factor is obtained by formula (9): μ ;

[0072] Formula (7);

[0073] Formula (8);

[0074] Formula (9);

[0075] In formulas (7), (8), and (9), C max is the preset maximum value of curiosity, A max is the preset maximum anxiety value, β is the adjustment coefficient, R Reward for the environment.

[0076] Then, the weights of the neurons in the middle layer Y are updated by formula (10), and the updated weights replace the original weights to achieve memory consolidation learning of the neural network;

[0077] Formula (10);

[0078] In formula (10), v j is the middle layer Y j The weight of a neuron, ω ( n j ) is the first j The number of activations of a neuron is n The learning rate at y j is the middle layer Y j The response value of a neuron, is the input vector, μ is the adjustment factor.

[0079] The anterior cingulate update learning is to update the Q value of the state-action pair through equations (11) and (12), and replace the original Q value neural network with the updated Q value;

[0080] Formula (11);

[0081] Formula (12);

[0082] In formula (11) and (12), α is the learning rate of Q value, A Decision information stored a A collection of μ is the adjustment factor.

[0083] The specific implementation method of the present invention is described by taking the path planning of a robot in static and dynamic unknown environments as an example.

[0084] 1. Static environment experiment

[0085] The physical experimental environment space of the static environment is a rectangular environment of 4m*2.4m, and 8 static obstacles are set in the environment. The initial coordinates of the robot are set to (0,0). Since the robot coordinate system adopts the Cartesian right-hand coordinate system, considering the robot's own volume and safety, the target point coordinates are set to (3.8,-2.2) in the experiment, and when the distance between the robot and the target point is less than 0.1m, the robot stops running. The robot's starting position is set to the lower left corner of the entire map, and the end point is set to the upper right corner of the map.

[0086] In the static environment experiment, the robot will conduct seven autonomous navigation experiments in the static environment. Figure 1 From left to right are the first, fourth and seventh navigation path maps. Figure 1 It can be seen that when the robot was running for the first time, it collided with obstacles due to its unfamiliarity with the environment. In the subsequent navigation experiments, the motion path of the mobile robot gradually became better, and finally the robot converged to a stable decision and navigation step number.

[0087] When the robot is at a certain location, if the similarity between the local environmental information obtained and the most similar environmental information in memory is lower than the set threshold, the local information will be considered as new memory. New memory is considered as short-term memory and needs to be converted into long-term memory and stored in the hippocampal network. If the similarity is higher than the set threshold, it means that the local environmental information currently obtained already exists in the long-term memory area, so the current memory connection will be strengthened through memory consolidation. Therefore, during the execution of the task by the mobile robot, if no new memory is generated, the decision path remains unchanged when the robot performs the next task. Figure 2 The average recognition error of the robot for the environment after each experiment in seven experiments is given. Figure 2 As can be seen, as the experiment progresses, the robot's recognition error of the environment continues to decrease, indicating that the robot is becoming more and more familiar with the environment.

[0088] Table 1 Statistics of seven experiments in static environment

[0089]

[0090] It can be seen from Table 1 that in the experiment, after the robot reaches the target point several times, the running path will gradually converge, and the number of movement steps is less than the number of movement steps when the path is initially reached. At the same time, the average recognition error of the environment gradually decreases, indicating that as the environment exploration proceeds, the robot becomes more and more familiar with the environment.

[0091] When the robot explores and learns in the environment, new neurons are inserted into the network each time a new memory is generated. Therefore, the number of neurons in the hippocampus represents the robot's memory of the environment. Because the number of neurons storing memories is increasing, the amount of knowledge remembered in the network is increasing. Figure 3 It can be seen that as the environment exploration proceeds, the amount of knowledge in the network grows less and less. Since the robot encounters less and less unlearned environmental information during the environment exploration process, the new memory of the generated environment will also decrease until the average recognition error is reduced to a reasonable range. At this time, the robot's knowledge no longer increases. Combined with Table 1, it can be seen that even if the amount of knowledge remembered by the robot no longer increases, its average recognition error for the environment is still decreasing. This is because for old memories, the robot will still consolidate their memories according to the degree of curiosity. Therefore, for old memories, the robot will narrow the difference between them and the actual environment memory, that is, consolidate the memory.

[0092] In order to further verify the incremental learning effect, the robot's starting point and end point were exchanged. At this time, the environmental information changed. Seven navigation experiments were also conducted in the static environment to test the robot's navigation effect. The robot's navigation paths in the first, fourth, and seventh experiments after exchanging the starting point and end point are shown in the figure below. Figure 4 From left to right, we can see that the first navigation path is not ideal, because after exchanging the starting point and the end point, the robot encountered unlearned environmental information during the exploration of the environment. Due to different visual directions, the views of the same environment in different directions are not similar, just like when a person walks on the same road, if the walking directions are opposite twice, the cognition of the environment will be different. In the next few experiments, the robot's movement path became better and better, because as the number of experiments increased, the unlearned information in the environment became less and less. After exchanging the starting point and the end point, the robot's knowledge also has a transformation trend of first increasing and then gradually converging. At the same time, by comparing the changes in the amount of knowledge stored by the robot in the two static environments, it can be seen that when the robot learns new environmental information in the new static environment, it does not cause the loss of the original knowledge information, but is incremental learning on the original basis.

[0093] 2. Dynamic environment experiment

[0094] On the basis of the static environment, an irregularly moving robot was added to act as a dynamic obstacle to test whether the robot can successfully navigate autonomously in a constantly changing environment to verify the experimental effect in a dynamic environment. Before the dynamic environment experiment, the robot had conducted seven navigation experiments in a static environment, and the memory already contained information about the static environment.

[0095] The robot navigation paths of the first, fourth and seventh experiments in the dynamic environment are shown in Figure 2. Figure 5From left to right, Figure 5 It can be seen that when the robot first senses a dynamic obstacle in a dynamic environment, its movement path is relatively poor. Although it can avoid dynamic obstacles, the robot's overall avoidance is relatively large because it is new environmental information that has not been learned. After recording this environmental information, in the subsequent navigation process, even if there is interference from dynamic obstacles, the robot's overall path is still gradually improved compared to the previous path. Figure 6 It can be seen that as the experiment progresses, the environmental recognition error is gradually decreasing, indicating that the robot's unfamiliarity with the dynamic environment is decreasing and the robot is becoming more and more familiar with the environment.

[0096] In a dynamic environment, due to the uncertainty caused by dynamic obstacles, the robot's knowledge has been increasing, but as the number of experiments increases, the growth rate of knowledge is also decreasing, which shows that the unknown information in the environment brought by dynamic obstacles is constantly decreasing.

[0097] Table 2 Data records of seven experiments under dynamic environment

[0098]

[0099] As can be seen from Table 2, as the amount of knowledge increases, the overall environmental recognition error shows a downward trend. In the sixth experiment, the environmental recognition error is higher than the previous one. This is because the hippocampal recognition error caused by dynamic obstacles has a large error value, which leads to an increase in the overall average recognition error. This shows that there is a memory with a high degree of unfamiliarity in the experiment, which leads to an increase in the overall number of movement steps. After the environmental recognition error decreases, the number of navigation steps decreases. The experimental results show that the robot can cope with various complex environments by learning different environments, and the knowledge acquired in the new environment will not interfere with the original knowledge.

[0100] 3. Comparative experiment

[0101] In order to further verify the performance and effect of the model, two classic path planning algorithms, the Rapid Exploration Random Tree (RRT) and the Soft Actor-Critic (SAC) algorithm in reinforcement learning, were selected for comparison. The reinforcement learning algorithm SAC was run 50 times in the simulation environment before running in the real environment to obtain a convergence path. Since RRT does not have memory ability, the RRT algorithm has no convergence path.

[0102] Table 3 Comparative experimental analysis

[0103]

[0104] As can be seen from Table 3, although the RRT method can successfully complete the navigation task, due to the large randomness of the RRT method, although it can find a path to complete the navigation task, it cannot obtain a convergence path, and does not have the ability to learn, so it has poor adaptability to the environment and low navigation efficiency. The average number of steps of the reinforcement learning algorithm SAC is similar to that of the model in this paper, but because reinforcement learning is unsupervised learning, the convergence speed is slow, so it takes a longer time to explore in the environment, and the deep reinforcement learning network structure is constant and does not have continuous learning ability, so its movement path remains consistent. However, due to the semi-supervised learning method, the model in this paper is faster than reinforcement learning in exploration efficiency, so the convergence speed is faster. At the same time, due to the existence of cognitive conflict, the model in this paper has an advantage in the ability to explore the shortest path. In summary, it can be seen that the model in this paper has better adaptability to the environment and has a strong ability to learn new knowledge.

Claims

1. A mobile robot behavior decision method capable of incremental continuous learning, wherein the mobile robot continuously detects the external environment and obtains image information during operation. The decision method divides the operation process of the mobile robot into multiple decision cycles, and sequentially gives the behavior decision of each decision cycle according to the image information, thereby controlling the mobile robot to move in accordance with the behavior decisions of the multiple decision cycles in sequence, characterized in that: In each decision cycle, the hippocampus receives the input image at the current moment and obtains the pre-selected decision based on memory. a m , and obtains a probability-based pre-selection decision through the anterior cingulate cortex a p ; The hippocampus is an incremental self-organizing neural network, including input layer X, pooling layer, intermediate layer Y and output layer Z. Each neuron in the intermediate layer Y stores its own memory information, and each memory information is obtained through perceptual information. o , Status Information s and decision-making information a The triplet composed of o , s , a >, where perception information o Pre-stored external environment image information, status information s is the spatial coordinate of the mobile robot, decision information a is the action instruction of the mobile robot; the input layer X can receive the input image and pass it to the pooling layer, which can extract features and reduce the dimension of the input image to obtain the external perception. o m And pass it to the middle layer Y, the output layer Z can make the decision information a Output to mobile robot for motion control; The hippocampus perceives the outside world o m A triplet of multiple memory information stored < o , s , a >Perceptual information in o Compare them respectively and get the o m The most similar perceptual information o , and then use this perception information o The triplet belongs to o , s , a Decision information in > a As a memory-based pre-selection decision a m ; The anterior cingulate gyrus is a Q-value neural network based on state-action pairs. The anterior cingulate gyrus receives the current state information. s , the current state information s With multiple decision information a They form state-action pairs respectively ( s , a ), and get all the status information including the current time s The Q value of the state-action pair is then used to obtain the current state information through formula (1). s Multiple decision information a The probability of execution; Formula (1); In formula (1), m 1 is a state-action pair ( s , a ) The anterior cingulate gyrus compares the multiple execution probabilities obtained, and then takes the decision information with the highest execution probability as the a As a probability-based pre-selection decision a p ; When pre-selection decisions are made based on memory a m and probability-based preselection decisions a p If the same, the pre-selection decision a m or a p As the behavioral decision executed by the mobile robot in the current decision cycle; When pre-selection decisions are made based on memory a m and probability-based preselection decisions a p At the same time, according to ε The -greedy strategy makes the behavioral decision that the mobile robot performs in the current decision cycle, and then performs hippocampal update learning and anterior cingulate gyrus update learning; The hippocampus updates learning by transmitting the current state information through the anterior cingulate gyrus. s and probability-based pre-selection decisions a p Transmitted to the hippocampus, which perceives the outside world o m , current status information s and probability-based pre-selection decisions a p Encode to obtain a short-term memory set , n 1 is the short-term memory set e Variables in u The number of memory similarity matrix is ​​obtained by formula (2): S lt ; Formula (2); In formula (2), n 2 is the number of neurons in the input layer X, l is the number of existing neurons in the middle layer Y, v i，j is the first i The neurons to the middle layer Y j The weight vector of neurons, u i For short-term memory collection e The i variables; When the memory similarity matrix S lt When it is less than the memory judgment threshold, a new neuron is added to the middle layer Y and the external perception o m , current status information s and probability-based pre-selection decisions a p The composed triples are stored as new memory information in the added neurons, realizing memory-added learning of the neural network; When the memory similarity matrix S lt When it is not less than the memory judgment threshold, the hippocampus obtains the prediction error through formula (3): Prediction_error ; Formula (3); In formula (3), E t Perceived by the outside world o m , current status information s and probability-based pre-selection decisions a p The contextual memory information composed of W t Memory information stored in the middle layer Y; The anterior cingulate gyrus obtains cognitive conflict through equations (4) and (5): Conflict ; Formula (4); Formula (5); In formula (4) and (5), m 2 is the stored decision information a The number of P i Status information for the current moment s Next i Decision information a The probability of execution; Then the hippocampus and anterior cingulate gyrus respectively Prediction_error and cognitive conflict Conflict is sent to the prefrontal cortex, which converts the prediction error Prediction_error and cognitive conflict Conflict The larger value is selected as the environmental error Error : Formula (6); Then the curiosity response is obtained through formula (7): Curiosity , the anxiety response value is obtained through formula (8): Anxiety , and the adjustment factor is obtained by formula (9): μ ; Formula (7); Formula (8); Formula (9); In formulas (7), (8), and (9), C max is the preset maximum value of curiosity, A max is the preset maximum anxiety value, β is the adjustment coefficient, R Rewards for the environment; Then, the weights of the neurons in the middle layer Y are updated by formula (10), and the updated weights replace the original weights to achieve memory consolidation learning of the neural network; Formula (10); In formula (10), v j is the middle layer Y j The weight of a neuron, ω ( n j ) is the first j The number of activations of a neuron is n The learning rate at y j is the middle layer Y j The response value of a neuron, is the input vector, μ is the regulating factor; The anterior cingulate update learning is to update the Q value of the state-action pair through equations (11) and (12), and replace the original Q value neural network with the updated Q value; Formula (11); Formula (12); In formula (11) and (12), α is the learning rate of Q value, A Decision information stored a A collection of μ is the adjustment factor.

Images