Crowd navigation method of mobile robot based on heterogeneous neural circuit strategy

Abstract

The invention discloses a crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy, which comprises the following steps: constructing a space-time diagram, and converting original local map data acquired by radar scanning into raster map data; constructing a heterogeneous neural circuit strategy network model and a reward function; inputting the time-space diagram and the raster map data into the heterogeneous neural circuit strategy network model, and outputting the action of the robot with the optimal value in the current environment by the heterogeneous neural circuit strategy network model; building a simulation environment of reinforcement learning interaction to train the heterogeneous neural circuit strategy network model, adopting a parallel PPO algorithm to accelerate the training speed of the heterogeneous neural circuit strategy network model until the training is completed, and applying the trained heterogeneous neural circuit strategy network model to an actual scene. The method has the advantages of higher navigation success rate, lower collision risk and stronger environment perception and modeling capability, and shows excellent performance in highly dynamic and crowded complex scenes.

Description

Technical Field

[0001] The invention belongs to the field of intelligent body path planning, and in particular relates to a crowd navigation method of a mobile robot. Background Art

[0002] In hospitals, airports, shopping malls and other places, mobile robots can provide guidance, consultation and other auxiliary services to improve service efficiency and user experience. However, when existing robots navigate in the crowd, they do not take into account the feelings and reactions of pedestrians. When navigating in the crowd, existing robots recklessly "rush" without regard to the psychological feelings of passers-by, or stop and wait when they meet pedestrians without regard to driving efficiency. This makes it difficult for existing mobile robots to effectively and harmoniously integrate into the crowd. The research and application of mobile robot navigation technology has promoted the development of artificial intelligence, robotics and human-computer interaction, and promoted the progress and innovation of science and technology. In addition, with the maturity of technology and the expansion of the market, the mobile robot industry will generate significant economic benefits, and the future society will enter a new era of deep integration and high collaboration between man and machine. As one of the cornerstones of the construction of an intelligent society, mobile robot navigation technology will be more widely used, thereby creating new employment opportunities and economic growth points. Overall, mobile robot navigation technology has shown extremely broad application prospects and far-reaching social significance.

[0003] The core challenges faced by mobile robot crowd navigation are mainly in the following three aspects: First, the robot needs to ensure the comfort of pedestrians when performing navigation tasks. Robot navigation cannot infringe on the pedestrians' comfortable space. Therefore, how to establish a suitable safe space for pedestrians to ensure their comfort is a challenge. This requires combining psychology, sociology, interpersonal relationships and many other disciplines to establish a suitable spatial model for pedestrians. The second is how to accurately predict pedestrian trajectories efficiently and quickly. Although there are many pedestrian trajectory prediction methods, these algorithms themselves are already very computationally intensive. If the robot is asked to use the predicted trajectory for planning, the robot's computing power is insufficient. Moreover, these pedestrian trajectory prediction algorithms rarely consider the impact of robots on pedestrian trajectories, and the prediction results are highly biased. The existing pedestrian trajectory prediction algorithms are difficult to apply to mobile robot crowd navigation applications. In addition, most of the existing learning-based methods only model pedestrians as a simple circle, only considering the physical space of pedestrians, ignoring the visual and social space of pedestrians, and the comfort of pedestrians cannot be guaranteed when the robot is navigating. Furthermore, most studies ignore the temporal factors when training neural networks and only consider the spatial structure, which will cause the trained network to be short-sighted and unable to make predictions for longer time steps.

[0004] Therefore, it is necessary to establish a new model to estimate and predict the future state of pedestrians: to enable the robot to output correct actions in the current environment, and to ensure that the robot walks flat, stable, and with a smooth trajectory, and can complete the task within a limited time without exceeding the time limit.

[0005] The invention with application number 202311521198.3 discloses a robot control method, electronic device and storage medium based on SAC (Soft Actor-Critic) and neural circuit strategy, which combines robot control with the SAC algorithm in reinforcement learning, and gets rid of the limitations of traditional control algorithms on the model. However, the above invention does not model the crowd, ignores the social attributes of the robot, and cannot guarantee the comfort of pedestrians when the robot is navigating. Summary of the invention

[0006] In view of the technical problem that the existing mobile robot navigation algorithm cannot take into account both pedestrian walking comfort and navigation performance in a crowd, the present invention proposes a crowd navigation method for a mobile robot based on a heterogeneous neural circuit strategy. By designing four NCP (Neural Circuit Policies) components in the heterogeneous neural circuit strategy network model to extract information features of the surrounding environment, the robot can see more detailed in the spatial domain and farther in the temporal domain, thereby achieving more efficient navigation in a complex crowd environment.

[0007] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] A crowd navigation method for a mobile robot based on a heterogeneous neural circuit strategy comprises the following steps:

[0009] S1. construct a space-time map through a space-time map construction module; convert the original local map data collected by radar scanning into raster map data through a radar data acquisition conversion module;

[0010] S2. Construct a heterogeneous neural circuit strategy network model, input the spatiotemporal graph and grid map data into the heterogeneous neural circuit strategy network model, and the heterogeneous neural circuit strategy network model outputs the action of the robot with the best value in the current environment state;

[0011] The heterogeneous neural circuit strategy network model includes a multilayer perceptron MLP1, a multilayer perceptron MLP2, a multilayer perceptron MLP3 and a CNN neural network, the multilayer perceptron MLP1 is connected to the neural circuit strategy NCP network NCP1, the multilayer perceptron MLP2 is connected to the neural circuit strategy NCP network NCP2, the CNN neural network is sequentially connected to the multilayer perceptron MLP4 and the neural circuit strategy NCP network NCP3, the neural circuit strategy NCP network NCP1 and the neural circuit strategy NCP network NCP2 are connected to the attention mechanism network, the attention mechanism network is connected to the multilayer perceptron MLP5, the multilayer perceptron MLP5, the neural circuit strategy NCP network NCP2, the multilayer perceptron MLP3 and the neural circuit strategy NCP network NCP3 are connected to the multilayer perceptron MLP6, and the multilayer perceptron MLP6 is sequentially connected to the neural circuit strategy NCP network NCP4 and the Actor-Critic network;

[0012] S3. Design a reward function for guiding the heterogeneous neural circuit strategy network model to optimize in the right direction, wherein the reward function includes a reward when the robot approaches or reaches the target point, a robot collision penalty, and a reward when the robot maintains a comfortable interaction distance with the pedestrian;

[0013] S4. Build a simulation environment for reinforcement learning interaction to train the heterogeneous neural circuit strategy network model. Use the parallel PPO algorithm to accelerate the training speed of the heterogeneous neural circuit strategy network model until the training is completed. Apply the trained heterogeneous neural circuit strategy network model to the actual scene. The mobile robot inputs the environmental state received at each decision moment into the trained heterogeneous neural circuit strategy network model. The trained heterogeneous neural circuit strategy network model outputs the optimal value action corresponding to each decision moment. The mobile robot executes the optimal value action corresponding to each decision moment to navigate in the real crowd.

[0014] Preferably, the structure of the space-time graph is:

[0015] G=(V,E,A)

[0016]

[0017] E=E hr ={e r-h1 ,e r-h2 ,...,e r-hm}

[0018] A=E rr ={e r-r}

[0019] G is a static graph, V is the set of robot and pedestrian nodes, E is the set of spatial edges, and A is the set of temporal edges. is the feature vector of robot node r, represents the feature vector of the mth pedestrian node, E hr represents the input encoding vector, e r-hm represents the relative position relationship between the mth pedestrian and the robot, E rr ={e r-r} represents the speed of the robot;

[0020] All spatial edge parameters in the space-time graph are shared.

[0021] Preferably, the method for converting the original local map data collected by radar scanning into grid map data is as follows: according to the trigonometric functions x=r*cos(θ) and y=r*sin(θ), the polar coordinate data (r,θ) in the original local map data are converted into (x,y) coordinate points in a Cartesian coordinate system; according to the position and orientation of the robot in the global coordinate system, the (x,y) coordinate points are unified and mapped to a preset grid map resolution; the mapped (x,y) coordinate points form a boundary contour line of the obstacle on the grid map, and a queue-based algorithm is used, starting from the boundary contour point and taking the boundary point as the starting point, and checking the adjacent grid cells in each iteration. When an idle area is encountered, it is marked as an obstacle and added to the queue, and the process is extended until a new boundary is encountered, and the marked internal grids are gradually expanded.

[0022] Preferably, the neural circuit strategy NCP network NCP1, the neural circuit strategy NCP network NCP2, the neural circuit strategy NCP network NCP3 and the neural circuit strategy NCP network NCP4 have the same network architecture and composition, and the network architecture has four layers, namely N s sensory neurons, N i interneurons, N c command neurons and N m A motor neuron.

[0023] Preferably, the reward function of the mobile robot is in, is the reward when the robot approaches or reaches the target point, For robot collision penalties, A reward for the robot to maintain a comfortable interaction distance with pedestrians.

[0024] Preferably, the reward when the robot approaches or reaches the target point for:

[0025]

[0026] Among them, p t represents the position information of the robot at the current time t, p grepresents the target position of the robot, ρ robot As the threshold, the robot can obtain positive rewards as long as it is closer to the target position at the current time t than at the time t-1, until it reaches the final target position. The robot obtains the maximum reward value when and only when it reaches the target point.

[0027] Preferably, the robot collision penalty d min < collision or(p x ,p y )∈Obstacle, where (p x ,p y ) is the current position of the robot, Obstacle is the map grid occupied by the static obstacle, and when the minimum distance d between the robot and the pedestrian min Less than the collision distance ρ collision ,When the robot contacts a static obstacle or the robot contacts a boundary wall, it is considered a collision and the robot receives the maximum penalty;

[0028] Rewards for robots to maintain a comfortable interaction distance with pedestrians I ss is the sum of the social pressures caused by the robot to the pedestrians around it.

[0029] Preferably, the method for training the heterogeneous neural circuit strategy network model is: after building a simulation environment for reinforcement learning interaction, the action of the robot with the best value in the current environment state output by the heterogeneous neural circuit strategy network model is input into the environment, the heterogeneous neural circuit strategy network interacts with the simulation environment, and the action of the best value in the current environment state is input into the simulation environment. The simulation environment will simulate the new state of the environment after the robot performs the action of the best value in the current environment state and the reward value obtained by the action of the best value in the current environment state obtained according to the reward function, and the old environment state, action, reward and new environment state are saved as a tuple in the experience pool, and data experience is collected for updating and training the Actor-Critic network, thereby training the heterogeneous neural circuit strategy network model.

[0030] Preferably, the process of the heterogeneous neural circuit strategy network outputting the action of the robot with the best value in the current environment state is as follows: the spatial edge of the space-time graph constructed in step S1 is input into the multilayer perceptron MLP1 of the heterogeneous neural circuit strategy network model, the multilayer perceptron MLP1 performs nonlinear processing on the spatial edge to obtain the robot-pedestrian spatial interaction feature at the current moment, and the neural circuit strategy NCP network NCP1 performs feature extraction on the robot-pedestrian spatial interaction feature at the current moment to obtain spatial interaction information;

[0031] Similar to the processing of spatial edges, the temporal edges of the space-time graph are input into the multilayer perceptron MLP2 of the heterogeneous neural circuit strategy network model. After nonlinear processing in the multilayer perceptron MLP2, the temporal edges are processed by the neural circuit strategy NCP network NCP2 to obtain the trajectory changes of the robot itself.

[0032] The outputs of the neural circuit strategy NCP network NCP1 and the neural circuit strategy NCP network NCP2 are input into the attention mechanism network, and the outputs of the attention mechanism network are input into the multi-layer perceptron MLP5 for dimensional transformation;

[0033] The robot node feature vector in the space-time graph is input into the multi-layer perceptron MLP3, and the multi-layer perceptron MLP3 encodes the robot node feature vector;

[0034] The raster map data is input into the CNN neural network of the heterogeneous neural circuit strategy network model. The CNN neural network outputs obstacle edge and shape feature vectors, which are input into the multilayer perceptron MLP4. The multilayer perceptron MLP4 outputs the encoding vector of the obstacle, which is input into the neural circuit strategy NCP network NCP3 for feature extraction. The output of the multilayer perceptron MLP5, the output of the neural circuit strategy NCP network NCP3, the output of the multilayer perceptron MLP3 and the output of the neural circuit strategy NCP network NCP2 are concatenated using the multilayer perceptron MLP6. The output of the multilayer perceptron MLP6 is input into the neural circuit strategy NCP4 neural network. The neural circuit strategy NCP network NCP4 is used to form a decision. Finally, the decision output by the neural circuit strategy NCP network NCP4 is input into the Actor-Critic network to calculate the value and robot action under the current environmental state.

[0035] Preferably, the output of the multi-layer perceptron MLP6 is:

[0036]

[0037] F represents the output after splicing, W mlp6 represents the weight of the multi-layer perceptron MLP6, u attn is the weighted sum of spatial edges, L o is the local grid map feature, V n is the encoding vector of the robot node feature, E t is the processed time feature information;

[0038] The decision making of the neural circuit strategy NCP network NCP4 is:

[0039]

[0040] is the decision made by the neural circuit strategy NCP network NCP4, ψ(·) represents a neural circuit strategy NCP network, W ncp4 is the network weight of the neural circuit strategy NCP network NCP4, h state is the hidden state of the neural circuit strategy NCP network before the neural circuit strategy NCP network processes data once, h ′ state It is the hidden state of the neural circuit strategy NCP network after the neural circuit strategy NCP network processes the data once.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention uses the space-time graph to construct the relationship between humans and machines, which more intuitively represents the relative relationship between humans and machines. In a relatively crowded environment, the space-time graph is easier to encode, which is beneficial to the input of the heterogeneous neural circuit policy network (LM-SNCP) model.

[0043] 2. The present invention uses a multi-layer CNN neural network to extract information features of obstacles in an environment containing heterogeneous obstacles, and integrates them with processed spatiotemporal graph data to achieve more efficient navigation in complex crowd environments.

[0044] 3. In the heterogeneous neural circuit strategy network model, the present invention designs four NCP components to extract information features of the surrounding environment. This structure can record the temporal information features of the environment and is more suitable for processing data with time relationships, thereby making predictions with longer time steps, allowing the robot to see more details in the spatial domain and farther in the temporal domain.

[0045] 4. This invention provides a robust and efficient navigation solution for highly dynamic and crowded indoor environments, laying the foundation for robots to coexist naturally with humans in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A space-time diagram.

[0048] Figure 2 This is a framework diagram of the heterogeneous neural circuit policy network (LM-SNCP) model of the present invention.

[0049] Figure 3This is the parallel PPO framework diagram of the present invention.

[0050] Figure 4 The experimental results of the present invention and other algorithms under different numbers of obstacles are compared. Figure 4 - (a) is a comparison chart of the navigation success rate of the present invention and other algorithms, Figure 4 -(b) is a comparison chart of the collision rate between the present invention and other algorithms. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] A crowd navigation method for a mobile robot based on a heterogeneous neural circuit strategy comprises the following steps:

[0053] S1. Construct a spatiotemporal graph through the spatiotemporal graph construction module to accurately capture the dynamic spatiotemporal relationship between robots and pedestrians in the crowd; convert the original local map data collected by radar scanning into raster map data through the radar data acquisition conversion module for better understanding and use.

[0054] Furthermore, if Figure 1 As shown in the figure, the positional relationship between the robot and the pedestrian is a graph structure, where the robot is regarded as a node and the pedestrians around it are regarded as nodes. Due to the spatial distance between nodes and the range of the time series, the impact of pedestrians on the robot is mainly within a small range centered on the robot and is also short-term in time. On this basis, a spatiotemporal graph is constructed to accurately capture the dynamic spatiotemporal relationship between robots and pedestrians in the crowd.

[0055] The space-time graph consists of spatial edges, temporal edges, and robot nodes. The spatial edges describe the spatial features between pedestrians and robots, aiming to capture the relative direction and distance dynamics between the robot and pedestrians. The temporal edges describe the information features of the robot at different times, aiming to capture the potential movement trends of the robot and pedestrians. The mathematical form of G = (V, E, A) is used to describe the space-time graph, where G is a static graph, V represents the set of robot and pedestrian nodes, E represents the set of spatial edges, and A represents the set of temporal edges.

[0056] At time t, the feature vector associated with pedestrian node h is and Represents the x, y coordinates of the pedestrian position and the robot node feature vector

[0057] in, represents the coordinate position of the robot at time t, represents the robot's target position, v pref represents the maximum speed of the robot, μ represents the radius of the robot, and θ represents the direction of the robot.

[0058] Eigenvectors associated with spatial edges represents the positional relationship between pedestrian node h and robot node r. Indicates the relative position of the pedestrian and the robot. The time edge is represented by the speed of the robot node r itself. represents the speed of the robot in the x direction, Represents the speed of the robot in the y direction.

[0059] This invention mainly considers the influence of people around the robot on the robot's movement, and does not consider the interaction between the intelligent agent and pedestrians. Therefore, the structure of the space-time graph is:

[0060] G=(V,E,A)

[0061]

[0062] E=E hr ={e r-h1 ,e r-h2 ,...,e r-hm}

[0063] A=E rr ={e r-r}

[0064] In the present invention, only the spatial edges are considered. Time Edge and the robot node r feature vector Representing the relationship between humans, machines and the environment through a time-space diagram is more intuitive and easier to understand, and is also conducive to coding. hr represents the input encoding vector, e r-hm represents the relative position relationship between the mth pedestrian and the robot, E rr ={e r-r} represents the time relationship between robots, that is, the speed of the robots.

[0065] Furthermore, the method for converting the original local map data collected by radar scanning into raster map data is as follows:

[0066] First, the polar coordinate data (r, θ) in the original local map data is converted to (x, y) coordinate points in the Cartesian coordinate system according to the trigonometric functions x = r*cos(θ) and y = r*sin(θ). Then, according to the position and orientation of the robot in the global coordinate system, the (x, y) coordinate points are unified and mapped to the preset grid map resolution. The mapped (x, y) coordinate points form the boundary contour line of the obstacle on the grid map. In order to completely fill the internal area of ​​the obstacle, a queue-based algorithm is used, starting from the boundary contour point, taking the boundary point as the starting point, checking the adjacent grid cells in each iteration, and marking it as an obstacle and adding it to the queue when encountering an idle area, until a new boundary is encountered, and gradually expanding the marked internal grid.

[0067] Through the above method, relying on the boundary information of radar data, the initial occupancy grid map representation is constructed quickly and efficiently.

[0068] The grid map discretizes the continuous spatial environment into uniform grid cells. Each cell represents a small area in the space and is marked as an obstacle or an idle area. It not only intuitively displays the location and shape of obstacles, but also provides structured input for subsequent path planning and obstacle avoidance operations.

[0069] S2. Construct a heterogeneous neural circuit policy network (LM-SNCP) model. The current state of the robot and pedestrian information is called the current environment state. The environment state, that is, the spatial edge of the space-time graph, the time edge of the space-time graph, the robot node feature vector containing the robot's current position, speed and other information, and the grid map data are input into the heterogeneous neural circuit policy network model. The heterogeneous neural circuit policy network model outputs the action of the robot with the best value under the current environment state.

[0070] like Figure 2 As shown, the heterogeneous neural circuit strategy network model includes a multilayer perceptron MLP1, a multilayer perceptron MLP2, a multilayer perceptron MLP3 and a CNN neural network. The multilayer perceptron MLP1 is connected to the neural circuit strategy NCP network NCP1, the multilayer perceptron MLP2 is connected to the neural circuit strategy NCP network NCP2, the CNN neural network is connected to the multilayer perceptron MLP4 and the neural circuit strategy NCP network NCP3 in sequence, the neural circuit strategy NCP network NCP1 and the neural circuit strategy NCP network NCP2 are connected to the attention mechanism network, the attention mechanism network is connected to the multilayer perceptron MLP5, the multilayer perceptron MLP5, the neural circuit strategy NCP network NCP2, the multilayer perceptron MLP3 and the neural circuit strategy NCP network NCP3 are connected to the multilayer perceptron MLP6, and the multilayer perceptron MLP6 is connected to the neural circuit strategy NCP network NCP4 and the Actor-Critic network in sequence.

[0071] Furthermore, the neural circuit strategy NCP network NCP1, neural circuit strategy NCP network NCP2, neural circuit strategy NCP network NCP3, neural circuit strategy NCP network NCP4 are used for feature extraction. The network architecture and composition of the neural circuit strategy NCP network NCP1, neural circuit strategy NCP network NCP2, neural circuit strategy NCP network NCP3, neural circuit strategy NCP network NCP4 are the same, and the network architecture has four layers, which are responsible for receiving information from the external environment or the perception of the internal state. s N sensory neurons are responsible for receiving information from sensory neurons and processing, analyzing, and integrating it to generate appropriate response signals. i Interneurons, N c command neurons are responsible for generating control instructions, and N m The motor neurons are responsible for executing the instructions issued by the command neurons. Between two consecutive layers of the neural circuit strategy NCP network, for any source neuron, n randomly selected target neurons that obey the Binomial (B, p1) distribution are selected to insert synapses with polarity obeying the Binomial (p2) distribution. The neural circuit strategy NCP network uses continuous-time ordinary differential equations to describe neuronal activity and introduces a nonlinear time-varying synaptic transmission mechanism, which enhances the expressive power of the heterogeneous neural circuit strategy network model in time series modeling tasks.

[0072] Furthermore, the multi-layer perceptron MLP1, the multi-layer perceptron MLP2, the multi-layer perceptron MLP3, the multi-layer perceptron MLP4, the multi-layer perceptron MLP5 and the multi-layer perceptron MLP6 are used as tools for data conversion.

[0073] In this embodiment, the number of layers of the multilayer perceptron MLP1, the multilayer perceptron MLP2, the multilayer perceptron MLP3, the multilayer perceptron MLP4, the multilayer perceptron MLP5 and the multilayer perceptron MLP6 are all two.

[0074] Furthermore, all spatial edge parameters in the space-time graph are shared. When nodes and edges in the space-time graph are input into the multi-layer perceptron, they are converted into feature vectors. Each feature vector needs to be processed by the heterogeneous neural circuit policy network model. Spatial edges are semantically similar, and nodes or edges can share the same learning module. By sharing all spatial edge parameters in the space-time graph, the parameters that need to be learned are reduced, and more nodes can be processed without increasing the number of parameters, which enhances the adaptability of the heterogeneous neural circuit policy network model in complex scenarios.

[0075] Specifically, when the spatiotemporal graph data is passed into the heterogeneous neural circuit policy network model, the low-dimensional semantic information is first converted into a fixed-length encoding vector through a multi-layer perceptron (MLP). Then the neural circuit policy NCP network captures the dynamic spatiotemporal information contained in the data by modeling spatiotemporal dependencies, and updates the node feature extraction layer in the multi-layer perceptron through back propagation. In actual scenarios, the number of people changes dynamically, so the set of edges also changes dynamically. Semantic-based parameter sharing makes the heterogeneous neural circuit policy network model structure more compact and can adapt to dynamic node changes.

[0076] Furthermore, multi-layer perceptrons MLP1, MLP2, and MLP3 convert low-dimensional semantic information into a fixed-length encoding vector. The spatial edge of the spatiotemporal graph constructed in step S1 is input to the multi-layer perceptron MLP1 of the heterogeneous neural circuit strategy network model, and the multi-layer perceptron MLP1 performs nonlinear processing on the spatial edge to obtain the encoding vector E ′ hr , that is, the spatial interaction characteristics of the robot and pedestrian at the current moment:

[0077]

[0078] in, represents a multi-layer perceptron MLP with nonlinear activation function, W mlp1 is the network weight of the multilayer perceptron MLP1. Since the multilayer perceptron MLP has the same operation process, the multilayer perceptron MLP at different positions is distinguished by the weight file. hr =[e r-h1 ,e r-h2 ,...,e r-hm ] is the input encoding vector, where e r-hm Represents the relative position relationship between the mth pedestrian and the robot.

[0079] Neural circuit strategy NCP network NCP1 pairs encoding vector E ′ hr Perform feature extraction to obtain spatial interaction information:

[0080] E s ,h ′ state =ψ(E ′ hr ,h state ,W ncp1 )

[0081] ψ(·) represents a neural circuit strategy NCP network. The neural circuit strategy NCP network has two inputs and outputs. The two inputs are E ′ hr and hstate , E ′ hr is the output of the multi-layer perceptron MLP1, h state is the current hidden state of the neural circuit strategy NCP network. After the neural circuit strategy NCP network has been processed once, the hidden state h of the neural circuit strategy NCP network state Will be updated to h ′ state , and output the result E after the neural circuit strategy NCP network data processing s .W ncp1 h is the network weight of the neural circuit strategy NCP network NCP1, and the neural circuit strategy NCP networks at different positions are distinguished by the weight file; state is the hidden state of the neural circuit strategy NCP network before the neural circuit strategy NCP network processes data once, h ′ state It is the hidden state of the neural circuit strategy NCP network after the neural circuit strategy NCP network processes the data once.

[0082] Similar to the processing of spatial edges, the time edges of the space-time graph are input into the multilayer perceptron MLP2 of the heterogeneous neural circuit policy network model. After nonlinear processing in the multilayer perceptron MLP2, the time edges are processed by the neural circuit policy NCP network NCP2 to obtain the trajectory changes of the robot itself:

[0083]

[0084] E t ,h ′ state =ψ(E r ′ r ,h state ,W ncp2 )

[0085] Among them, E rr ={e r-r} represents the time relationship between robots, that is, the speed of the robots, E r ′ r is the speed of the robot E rr The encoding vector, E t is the processed time feature information, W mlp2 is the network weight of the multi-layer perceptron MLP2, W ncp2 The network weights of NCP2 for the neural circuit strategy NCP network.

[0086] Next, the outputs of the neural circuit strategy NCP network NCP1 and the neural circuit strategy NCP network NCP2 are input into the attention mechanism network, and a weight is assigned to each pedestrian in the attention mechanism network. The attention calculation process is as follows:

[0087] Q t =E s W q

[0088] K t =E t W k

[0089] Q t and K t are query and key matrices, through which the correlation between elements in the input sequence can be calculated. q and W k is a trainable weight matrix used to transform the input embeddings into queries and keys.

[0090] The spatial edge attention weights are calculated using the scaled dot product attention mechanism:

[0091]

[0092] Where n represents the number of pedestrians, d k K t dimension, used to scale the similarity score, (K t ) T K t The transposed matrix of . The weighted sum of the output space edges of the attention network at time t is expressed as:

[0093] u attn =(E s ) T α T

[0094] u attn is the result of weighted summation of spatial edges, α T is α t The transposed matrix of the attention mechanism network is input into the multi-layer perceptron MLP5 for dimensional transformation.

[0095] Robot node feature vector The multilayer perceptron MLP3 input to the heterogeneous neural circuit strategy network model is used to calculate the robot node feature vector. Encoding process:

[0096]

[0097] W mlp3Represents the weight of the multi-layer perceptron MLP3, V n Represents the robot node feature vector The encoding vector of .

[0098] The grid map data is input into the CNN neural network of the heterogeneous neural circuit strategy network model, and the grid map data is convolved in the CNN neural network to capture the edge and shape features of the obstacle:

[0099] L ′ =Conv(L,W conv )

[0100] Among them, Conv represents the convolutional layer operation, W conv Represents the weight of the convolution layer. The purpose of this step is to capture the edge and shape features of obstacles. L represents the raster map data converted from the original local map data collected by radar scanning. ′ Represents the captured obstacle edge and shape feature vectors.

[0101] The output of the CNN neural network generates the encoding vector of the obstacle through the multi-layer perceptron MLP4, which is expressed as:

[0102]

[0103] W mlp4 is the network weight of the multi-layer perceptron MLP4.

[0104] The encoding vector L of the obstacle output by the multi-layer perceptron MLP4 ′ o Input into the neural circuit strategy NCP network NCP3 for feature extraction:

[0105] L o ,h ′ state =ψ(L ′ o ,h state ,W ncp3 )

[0106] Among them, L o is the local grid map feature, W mlp3 is the network weight of the multi-layer perceptron MLP3, W ncp3 The network weights of NCP3 for the neural circuit strategy NCP network.

[0107] Then, the multi-layer perceptron MLP6 is used to transform the output spatial edge weight u of the multi-layer perceptron MLP5 attn , the output local grid map feature L of the neural circuit strategy NCP network NCP3 o , the output robot node feature vector of the multi-layer perceptron MLP3 The encoding vector V n The temporal feature information E after the output of the neural circuit strategy NCP network NCP2 is processed t Stitching:

[0108]

[0109] F represents the output after splicing, W mlp6 Represents the weights of the multilayer perceptron MLP6.

[0110] Finally, the decision is made using the neural circuit strategy NCP network NCP4:

[0111]

[0112] Decision making by the neural circuit strategy NCP network NCP4, W ncp4 Network weights of NCP4 for the neural circuit strategy NCP network.

[0113] Finally, the decision-making of the neural circuit strategy NCP network NCP4 Input into the Actor-Critic network to calculate the value and robot action under the current environment state, and output the action with the best value under the current environment state.

[0114] S3. Design a reward function to guide the heterogeneous neural circuit policy network model to optimize in the right direction, wherein the reward function includes a reward when the robot approaches or reaches the target point, a robot collision penalty, and a reward when the robot maintains a comfortable interaction distance with pedestrians.

[0115] In the process of the robot approaching the target point, a larger reward is given when the robot is close to the target point, and the maximum reward is given when the robot reaches the target point.

[0116] Rewards when the robot approaches or reaches the target point for:

[0117]

[0118] Among them, p t represents the position information of the robot at the current time t, p g represents the target position of the robot, ρ robot is the threshold, which is usually the robot radius. As long as the robot is closer to the target position at the current time t than at the time t-1, it can get a positive reward until it reaches the final target position, so as to guide the robot to approach the target point. However, the robot can only get the maximum reward value when it reaches the target point, thus reducing the possibility of the robot falling into the local optimum.

[0119] When the robot collides with a person or obstacle, it will receive a robot collision penalty.

[0120]

[0121] Among them, (p x ,p y ) is the current position of the robot, Obstacle is the map grid occupied by the static obstacle, and when the minimum distance d between the robot and the pedestrian min Less than the collision distance ρ collision , when the robot contacts a static obstacle or the robot contacts a boundary wall, a collision is considered to have occurred and the robot receives the maximum penalty.

[0122] Rewards for robots to maintain a comfortable interaction distance with pedestrians for:

[0123]

[0124] I ss is the sum of the social pressures caused by the robot to the pedestrians around it.

[0125] In summary, the reward function of the mobile robot is expressed as:

[0126]

[0127] Therefore, the robot's total reward R t for:

[0128]

[0129] γ is the discount factor, which ranges from (0,1].

[0130] S4. Build a simulation environment for reinforcement learning interaction. The action of the robot with the best value in the current environment state output by the heterogeneous neural circuit strategy network model is input into the simulation environment, and then the heterogeneous neural circuit strategy network interacts with the simulation environment, and the action with the best value in the current environment state is input into the simulation environment. The simulation environment will simulate the new state of the environment after the robot makes the action with the best value in the current environment state and the reward value obtained by the action with the best value in the current environment state according to the reward function, and then save the old environment state, action, reward and new environment state as a tuple to the experience pool. This method is used to collect data experience for updating and training the Actor-Critic network, so as to train the heterogeneous neural circuit strategy network model. The parallel PPO algorithm is used to accelerate the training speed of the heterogeneous neural circuit strategy network model until the training is completed. The heterogeneous neural circuit strategy network model is applied to the actual scene. The mobile robot inputs the environmental state received at each decision moment into the trained heterogeneous neural circuit strategy network model. The trained heterogeneous neural circuit strategy network model outputs the action with the best value corresponding to each decision moment. The mobile robot performs the action with the best value corresponding to each decision moment in the real crowd for navigation.

[0131] Build an interactive environment for reinforcement learning: Install Gym into the virtual environment, use Gym to create a simulation environment according to the requirements of step S2, add robots, pedestrians, and obstacles to the environment, and set the state space, action space, and reward function.

[0132] Specifically, pedestrians in the environment use the traditional ORCA algorithm to navigate. Each pedestrian's initial position is random and has its own target point. The pedestrians themselves are set to a circle of 0.3m, with a maximum speed of 1m / s. There is a 25% chance that pedestrians will change their destination while walking. There is a certain probability that pedestrians will change their destination while walking. The robot itself is set to a circle with a radius of 0.3m.

[0133] Furthermore, when the parallel PPO algorithm is used to accelerate the model training speed, multi-process technology is introduced to speed up the process of PPO algorithm training model. The main process distributes the current strategy (LM-SNCP model parameters) to each Worker process. Each Worker process interacts with the environment according to the current strategy and collects a certain number of state-action-reward sequences as a Markov sequence. All Worker processes pass the collected rollout back to the main process, and the main process uses the PPO algorithm to update the policy network parameters based on these rollout data. The updated policy network parameters are distributed to the Worker process again.

[0134] The specific process of using parallel PPO to accelerate the training speed of the heterogeneous neural circuit strategy network model is as follows:

[0135] like Figure 3 As shown in the figure, in the parallel PPO algorithm framework, at the beginning, the weights of the value network Critic network and the policy network Actor are set to the initial value θ. At the beginning of each training cycle (episode), all simulation environments are initialized and receive the initial state of the environment. Subsequently, the PPO algorithm enters a loop that performs the following operations at each time step (step): According to the current state of the environment, the policy network is used to select an action, and then interacts with the environment to obtain a new state, corresponding rewards, and a signal whether the termination condition is reached. These interaction results are recorded in the dataset D. At the same time, the PPO algorithm calculates the generalized advantage estimate (GAE), which is a method for estimating the advantage of an action. The estimated value of GAE at time step t is It is obtained by considering the advantage value δt and the decay factor (γλ) from the current time step to the terminal time step T T-t It is calculated as the weighted sum of , where γ represents the discount rate, λ is the decay rate of GAE, and T is the maximum number of steps in the cycle.

[0136] In the following multiple training iterations, the present invention uses the loss function of the PPO algorithm, the loss function of the value function, and a regularization term to update the parameters of the heterogeneous neural circuit strategy network model. The overall loss function L Net (θ) combines the loss L of PPO ppo (θ), the loss of the value function L Value (θ) and a regularization term E(θ). The regularization term helps control the update amplitude of the heterogeneous neural circuit policy network model parameters to avoid overfitting problems. N and β are hyperparameters that control the loss function. After all iterations are completed, the updated network parameters are output to complete the training.

[0137] The performance of the model trained by the present invention and other navigation algorithms in complex scenes with different number of people was tested in the Gym simulation environment. The results are as follows: Figure 4As shown. It can be seen that in a complex environment containing dynamic pedestrians and static obstacles, the heterogeneous neural circuit policy network model (LM-SNCP) proposed in the present invention has shown excellent performance in terms of the success rate and collision rate of robot navigation tasks. Compared with the traditional ORCA algorithm and the HGAT-DRL algorithm based on graph learning, the ORCA algorithm constructs a speed obstacle set according to the position and speed of the target point and the intelligent body to avoid collisions. The HGAT-DR algorithm uses a graph neural network to construct a human-computer relationship graph and uses a reinforcement learning method to train the model to learn appropriate actions. The success rate of the present invention is significantly higher than that of the ORCA algorithm and the HGAT-DRL algorithm in scenarios with different numbers of obstacles. As Figure 4 -(a) shows that as the number of obstacles increases, the success rates of the three algorithms all decrease, but the decrease of the present invention is the smallest. When there is only one obstacle, the success rates of the three algorithms are similar, but as the number of obstacles increases, the present invention shows a stronger advantage. In the most complex scenario with five obstacles, the success rate of the present invention is as high as 0.86, which is 0.09 higher than that of the HGAT-DRL algorithm and 0.264 higher than that of the ORCA algorithm. Figure 4 -(b) shows that in terms of collision rate, the present invention is significantly lower than the other two algorithms in all cases. As the number of obstacles increases, the collision rates of the three algorithms all increase, but the present invention increases the slowest. When there is 1 obstacle, the collision rate of the present invention is as low as 0.07. When there are 2-4 obstacles, the collision rate of the present invention is basically maintained at a low level of 0.08-0.09. Even in the most complex scenario with 5 obstacles, the collision rate of the present invention is 0.13, which is significantly lower than the other two algorithms.

[0138] These experimental results show that the present invention can better plan safe and feasible paths for robots in dynamic and static complex environments, with higher navigation success rate and lower collision risk. The present invention integrates spatiotemporal graph and coarse-grained local map representation, has stronger environmental perception and modeling capabilities, and shows excellent performance in highly dynamic and crowded complex scenes.

[0139] The present invention proposes a heterogeneous neural circuit policy network model (LM-SNCP) with a coarse-grained local map, which integrates the two representation forms of spatiotemporal graph and coarse-grained local map, and can simultaneously model the interaction between robots and dynamic pedestrians, as well as handle static special-shaped obstacles. The spatiotemporal graph depicts the current crowd interaction, and the coarse-grained local map encodes the edges and shapes of obstacles. The two complement each other, giving LM-SNCP the ability to fully perceive complex environments, enabling robots to infer the future movements of the crowd and take appropriate actions to avoid pedestrians and prevent collisions. The present invention adopts multi-process technology and a mobile robot crowd navigation method based on a neural circuit strategy to study sampled environmental data, and performs end-to-end reinforcement learning training on LM-SNCP through the PPO algorithm and generalized advantage estimation, so that it can learn the optimal navigation strategy from complex scenes.

[0140] This invention provides a robust and efficient navigation solution for highly dynamic and crowded indoor environments, laying the foundation for robots to coexist naturally with humans in real scenarios.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A crowd navigation method for a mobile robot based on a heterogeneous neural circuit strategy, characterized in that: The following steps are involved: S1. construct a space-time map through a space-time map construction module; convert the original local map data collected by radar scanning into raster map data through a radar data acquisition conversion module; S2. Construct a heterogeneous neural circuit strategy network model, input the spatiotemporal graph and grid map data into the heterogeneous neural circuit strategy network model, and the heterogeneous neural circuit strategy network model outputs the action of the robot with the best value in the current environment state; The heterogeneous neural circuit strategy network model includes a multilayer perceptron MLP1, a multilayer perceptron MLP2, a multilayer perceptron MLP3 and a CNN neural network, the multilayer perceptron MLP1 is connected to the neural circuit strategy NCP network NCP1, the multilayer perceptron MLP2 is connected to the neural circuit strategy NCP network NCP2, the CNN neural network is sequentially connected to the multilayer perceptron MLP4 and the neural circuit strategy NCP network NCP3, the neural circuit strategy NCP network NCP1 and the neural circuit strategy NCP network NCP2 are connected to the attention mechanism network, the attention mechanism network is connected to the multilayer perceptron MLP5, the multilayer perceptron MLP5, the neural circuit strategy NCP network NCP2, the multilayer perceptron MLP3 and the neural circuit strategy NCP network NCP3 are connected to the multilayer perceptron MLP6, and the multilayer perceptron MLP6 is sequentially connected to the neural circuit strategy NCP network NCP4 and the Actor-Critic network; S3. Design a reward function for guiding the heterogeneous neural circuit strategy network model to optimize in the right direction, wherein the reward function includes a reward when the robot approaches or reaches the target point, a robot collision penalty, and a reward when the robot maintains a comfortable interaction distance with the pedestrian; S4. Build a simulation environment for reinforcement learning interaction to train the heterogeneous neural circuit strategy network model. Use the parallel PPO algorithm to accelerate the training speed of the heterogeneous neural circuit strategy network model until the training is completed. Apply the trained heterogeneous neural circuit strategy network model to the actual scene. The mobile robot inputs the environmental state received at each decision moment into the trained heterogeneous neural circuit strategy network model. The trained heterogeneous neural circuit strategy network model outputs the optimal value action corresponding to each decision moment. The mobile robot executes the optimal value action corresponding to each decision moment to navigate in the real crowd.

2. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 1 is characterized in that: The structure of the space-time graph is: G=(V,E,A) E=E hr ={and r-h1 ,And r-h2 ,..,And r-hm } A=E rr ={and r-r } G is a static graph, V is the set of robot and pedestrian nodes, E is the set of spatial edges, and A is the set of temporal edges. is the feature vector of robot node r, represents the feature vector of the mth pedestrian node, E hr represents the input encoding vector, e r-hm represents the relative position relationship between the mth pedestrian and the robot, E rr ={e r-r } represents the speed of the robot; All spatial edge parameters in the space-time graph are shared.

3. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 2 is characterized in that: The method for converting the original local map data collected by radar scanning into raster map data is as follows: according to the trigonometric functions x=r*cos(θ) and y=r*sin(θ), the polar coordinate data (r,θ) in the original local map data are converted into (x,y) coordinate points in the Cartesian coordinate system; according to the position and orientation of the robot in the global coordinate system, the (x,y) coordinate points are unified and mapped to a preset raster map resolution; the mapped (x,y) coordinate points form the boundary contour line of the obstacle on the raster map, and a queue-based algorithm is used, starting from the boundary contour point and taking the boundary point as the starting point, checking the adjacent grid cells in each iteration, and marking it as an obstacle and adding it to the queue when encountering an idle area, and expanding until a new boundary is encountered, and gradually expanding the marked internal grid.

4. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 1 or 3, characterized in that: The neural circuit strategy NCP network NCP1, the neural circuit strategy NCP network NCP2, the neural circuit strategy NCP network NCP3 and the neural circuit strategy NCP network NCP4 have the same network architecture and composition, and the network architecture has four layers, namely N s sensory neurons, N i interneurons, N c command neurons and N m A motor neuron.

5. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 4 is characterized in that: The reward function of the mobile robot in, is the reward when the robot approaches or reaches the target point, For robot collision penalties, A reward for the robot to maintain a comfortable interaction distance with pedestrians.

6. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 5, characterized in that: The reward when the robot approaches or reaches the target point for: Among them, p t represents the position information of the robot at the current time t, p g represents the target position of the robot, ρ robot As the threshold, the robot can obtain positive rewards as long as it is closer to the target position at the current time t than at the time t-1, until it reaches the final target position. The robot obtains the maximum reward value when and only when it reaches the target point.

7. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 6, characterized in that: The robot collision penalty d min < collision or(p x ,p y )∈Obstacle, where (p x ,p y ) is the current position of the robot, Obstacle is the map grid occupied by the static obstacle, and when the minimum distance d between the robot and the pedestrian min Less than the collision distance ρ collision ,When the robot contacts a static obstacle or the robot contacts a boundary wall, it is considered a collision and the robot receives the maximum penalty; Rewards for robots to maintain a comfortable interaction distance with pedestrians I ss is the sum of the social pressures caused by the robot to the pedestrians around it.

8. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 1 or 7, characterized in that: The method for training a heterogeneous neural circuit strategy network model is as follows: after building a simulation environment for reinforcement learning interaction, the action of the robot with the best value under the current environmental state output by the heterogeneous neural circuit strategy network model is input into the simulation environment, the heterogeneous neural circuit strategy network model interacts with the simulation environment, and the action with the best value under the current environmental state is input into the simulation environment. The simulation environment simulates the new environmental state after the robot performs the action with the best value under the current environmental state and the reward value obtained by the action with the best value under the current environmental state obtained according to the reward function, the old environmental state, action, reward and new environmental state are saved as a tuple in the experience pool, and data experience is collected for updating and training the Actor-Critic network, thereby training the heterogeneous neural circuit strategy network model.

9. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 8, characterized in that: The process of the heterogeneous neural circuit strategy network outputting the optimal value action of the robot in the current environment state is: The spatial edge of the space-time graph constructed in step S1 is input into the multilayer perceptron MLP1 of the heterogeneous neural circuit strategy network model. The multilayer perceptron MLP1 performs nonlinear processing on the spatial edge to obtain the robot-pedestrian spatial interaction features at the current moment. The neural circuit strategy NCP network NCP1 extracts the robot-pedestrian spatial interaction features at the current moment to obtain spatial interaction information. Similar to the processing of spatial edges, the temporal edges of the space-time graph are input into the multilayer perceptron MLP2 of the heterogeneous neural circuit strategy network model. After nonlinear processing in the multilayer perceptron MLP2, the temporal edges are processed by the neural circuit strategy NCP network NCP2 to obtain the trajectory changes of the robot itself. The outputs of the neural circuit strategy NCP network NCP1 and the neural circuit strategy NCP network NCP2 are input into the attention mechanism network, and the outputs of the attention mechanism network are input into the multi-layer perceptron MLP5 for dimensional transformation; The robot node feature vector in the space-time graph is input into the multi-layer perceptron MLP3, and the multi-layer perceptron MLP3 encodes the robot node feature vector; The raster map data is input into the CNN neural network of the heterogeneous neural circuit strategy network model. The CNN neural network outputs obstacle edge and shape feature vectors, which are input into the multilayer perceptron MLP4. The multilayer perceptron MLP4 outputs the encoding vector of the obstacle, which is input into the neural circuit strategy NCP network NCP3 for feature extraction. The output of the multilayer perceptron MLP5, the output of the neural circuit strategy NCP network NCP3, the output of the multilayer perceptron MLP3 and the output of the neural circuit strategy NCP network NCP2 are concatenated using the multilayer perceptron MLP6. The output of the multilayer perceptron MLP6 is input into the neural circuit strategy NCP4 neural network. The neural circuit strategy NCP network NCP4 is used to form a decision. Finally, the decision output by the neural circuit strategy NCP network NCP4 is input into the Actor-Critic network to calculate the value and robot action under the current environmental state.

10. The crowd navigation method of a mobile robot based on a heterogeneous neural circuit strategy according to claim 9, characterized in that: The output of the multi-layer perceptron MLP6 is: F represents the output after splicing, W mlp6 represents the weight of the multi-layer perceptron MLP6, u attn is the weighted sum of spatial edges, L o is the local grid map feature, V n is the encoding vector of the robot node feature, E t is the processed time feature information; The decision making of the neural circuit strategy NCP network NCP4 is: is the decision made by the neural circuit strategy NCP network NCP4, ψ(·) represents a neural circuit strategy NCP network, W ncp4 is the network weight of the neural circuit strategy NCP network NCP4, h state is the hidden state of the neural circuit strategy NCP network before the neural circuit strategy NCP network processes data once, h ′ state It is the hidden state of the neural circuit strategy NCP network after the neural circuit strategy NCP network processes the data once.

Images