Method, device and medium for safe navigation of mobile robots in complex crowd scenarios

Abstract

This invention discloses a method, device, and medium for safe navigation of a mobile robot in complex crowd scenarios. The method includes: Step 1, generating a corresponding collision-free pedestrian trajectory dataset in a simulation environment via a pedestrian trajectory generation module; Step 2, training a reinforcement learning network for a pedestrian prediction module and a hybrid decision module using the collision-free pedestrian trajectory dataset; Step 3, when the trained pedestrian prediction module and hybrid decision module jointly control the mobile robot to navigate in a complex crowd scenario in a real-world environment, the online planning unit of the hybrid decision module determines the optimal action to be performed by the mobile robot based on the probability distribution of selectable actions in the current state, combined with the future state of the pedestrians and the policy and value networks of the reinforcement learning network. This ensures that the mobile robot can safely navigate in complex crowd scenarios in a real-world environment by executing the optimal action at each decision moment. This method enables the mobile robot to safely navigate in real-world complex crowd scenarios.

Description

Technical Field

[0001] This invention relates to the field of mobile robot navigation, and more particularly to a safe navigation method for mobile robots in complex crowd scenarios. Background Technology

[0002] Crowd navigation for mobile robots has been a research topic for many years. To deploy mobile robots in real-world environments, they not only need to find a reasonable path to their destination based on a global map, but also need to dynamically adjust their path in real-time based on pedestrian behavior to achieve safe navigation. In particular, in open but high-traffic areas such as shopping malls and hospital lobbies, how to enable mobile robots to interact safely with pedestrians and efficiently reach the target point is a crucial issue. Current crowd navigation algorithms are mainly divided into speed obstacle-based methods, trajectory prediction-based methods, and deep reinforcement learning-based methods. Especially in recent years, with the development of deep reinforcement learning, more and more researchers have begun to try using this new tool to solve the crowd navigation problem for mobile robots. Deep reinforcement learning has broad research prospects and value in further expanding the crowd navigation capabilities of mobile robots and is of great significance for deploying mobile robots in the real world.

[0003] In crowd navigation methods based on deep reinforcement learning, mobile robots interact with simulated pedestrians in a pre-simulated environment, autonomously learning the patterns of pedestrian interaction to find the best response in different scenarios. Specifically, the deep reinforcement learning-based method first models the interaction between pedestrians and the mobile robot in the scene as a more generalized Markov decision process. Within this framework, the mobile robot will conduct trial and error in the constructed simulation environment. Then, based on the feedback, the mobile robot can autonomously adjust its behavior according to the optimization algorithm to achieve optimal navigation under the simulation environment conditions. In the real environment, the mobile robot can directly make efficient decisions based on the overall strategy learned in the simulation environment and the current observation. That is to say, on the one hand, the mobile robot can ensure the optimality of the obtained strategy through extensive offline training in the simulation environment; on the other hand, the strategy obtained through extensive training in the simulation environment can also ensure low computational complexity during the execution of the mobile robot. Compared with traditional methods, due to the large amount of interaction data in the learning process, mobile robots based on deep reinforcement learning often have better obstacle avoidance capabilities and can complete the predetermined navigation tasks more quickly.

[0004] Within the broad framework of deep reinforcement learning, all methods can be broadly categorized into value function-based iteration algorithms, such as DQN, and policy gradient-based algorithms, such as TRPO. In the specific problem of crowd navigation, depending on the state of the constructed Markov decision process, they can be divided into end-to-end algorithms that directly use sensor data, such as laser point clouds, as input, and algorithms that separate state processing from the policy framework, using processed state data, such as meaningful quantities like pedestrian position and velocity, as input. However, currently, regardless of the specific deep reinforcement learning-based crowd navigation algorithm, it is relatively sensitive to environmental changes. Even small differences between the actual testing environment and the training environment can significantly reduce the navigation capabilities of the mobile robot, ultimately causing actual safety hazards. Specifically, in the usual assumption, pedestrians in the training simulation environment are mainly controlled by rule-based algorithms (such as velocity-obstacle algorithms) for movement; however, in real-world scenarios, pedestrian behavior is more complex, random, and difficult to predict. Therefore, for mobile robots equipped with deep reinforcement learning-based navigation algorithms, once they encounter unexpected pedestrian behavior during operation, the mobile robots are unable to learn effective coping methods from experience in the simulation environment, which may lead to collisions and other consequences that could endanger human life. Thus, the safety of mobile robots navigating in complex crowd scenarios cannot be guaranteed.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a safe navigation method, device, and medium for mobile robots in complex crowd scenarios. During training, the mobile robot can interact with a variety of pedestrians as much as possible, and with the help of an online planning unit during runtime, it can adapt to different pedestrian strategies in the real world, ensuring safe navigation of the mobile robot in complex crowd scenarios, thereby solving the above-mentioned technical problems existing in the prior art.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A method for safe navigation of a mobile robot in complex crowd scenes, characterized in that it includes the following components for controlling the mobile robot to navigate safely in complex crowd scenes:

[0009] Step 1: Construct a simulation environment based on the actual operating environment of the mobile robot using the pedestrian trajectory generation module, and generate a corresponding collision-free pedestrian trajectory dataset in the simulation environment.

[0010] Step 2: Train the reinforcement learning networks of the pedestrian prediction module and the hybrid decision module using the collision-free pedestrian trajectory dataset obtained in Step 1.

[0011] Step 3: When the mobile robot navigates in complex crowd scenes in a real-world environment using the trained pedestrian prediction module and hybrid decision module, the mobile robot inputs the environmental state received at each decision moment into the reinforcement learning network of the trained hybrid decision module to obtain a probability distribution of possible actions in the current state. Then, the online planning unit of the hybrid decision module, based on the probability distribution of possible actions in the current state, combined with the future state of the pedestrian predicted by the pedestrian prediction module and the policy network and value network of the reinforcement learning network, determines the optimal action to be performed by the mobile robot. This enables the mobile robot to perform the optimal action corresponding to each decision moment, achieving safe navigation in complex crowd scenes in a real-world environment.

[0012] A processing apparatus, comprising:

[0013] At least one memory for storing one or more programs;

[0014] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the method of the present invention.

[0015] A readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the methods described in this invention.

[0016] Compared with existing technologies, the safe navigation method, device, and medium for mobile robots in complex crowd scenarios provided by this invention have the following advantages:

[0017] Through the method of this invention, the mobile robot can not only perform well under the pedestrian model assumed in the training simulation environment, but also respond reasonably to actual pedestrians in real environment to achieve safe and efficient movement. This facilitates the deployment of the mobile robot in real environment to achieve safe navigation in complex crowd scenarios, and provides important support for the practical application of mobile robots. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a safe navigation method for mobile robots in complex crowd scenarios, as provided in an embodiment of the present invention. Solid lines represent the execution process, while dashed lines represent the training process. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0021] First, the following explanations are provided for the terms that may be used in this article:

[0022] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0023] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0024] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0025] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0026] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0027] The following is a detailed description of the safe navigation method for mobile robots in complex crowd scenarios provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0028] like Figure 1 As shown, this embodiment of the invention provides a safe navigation method for a mobile robot in complex crowd scenes, used to control the mobile robot to navigate safely in complex crowd scenes, including:

[0029] Step 1: Construct a simulation environment based on the actual operating environment of the mobile robot using the pedestrian trajectory generation module, and generate a corresponding collision-free pedestrian trajectory dataset in the simulation environment.

[0030] Step 2: Train the reinforcement learning networks of the pedestrian prediction module and the hybrid decision module using the collision-free pedestrian trajectory dataset obtained in Step 1.

[0031] Step 3: When the mobile robot navigates in complex crowd scenes in a real-world environment using the trained pedestrian prediction module and hybrid decision module, the mobile robot inputs the environmental state received at each decision moment into the reinforcement learning network of the trained hybrid decision module to obtain a probability distribution of possible actions in the current state. Then, the online planning unit of the hybrid decision module, based on the probability distribution of possible actions in the current state, combined with the future state of the pedestrian predicted by the pedestrian prediction module and the policy network and value network of the reinforcement learning network, determines the optimal action to be performed by the mobile robot. This enables the mobile robot to perform the optimal action corresponding to each decision moment, achieving safe navigation in complex crowd scenes in a real-world environment.

[0032] Preferably, in step 1 of the above method, a simulation environment is constructed by the pedestrian trajectory generation module according to the actual operating environment required by the mobile robot, and a corresponding collision-free pedestrian trajectory dataset is generated in the simulation environment, including:

[0033] The map of the real environment in which the mobile robot needs to operate is modeled as a 2D simulation environment represented by a 2D graph. Obstacles in the environment are modeled in the 2D simulation environment, and the movement range of pedestrians is determined. The probability distribution of the starting point and ending point of all pedestrians in the 2D simulation environment is set according to the real environment in which the robot actually operates.

[0034] Based on the 2D simulation environment obtained from the modeling, when generating a set of collision-free pedestrian trajectories, sampling is first performed according to the probability distribution of pedestrian start and end points to obtain the start and end points of each pedestrian. The decision order of each pedestrian in the scene is randomly shuffled. According to the decision order of each pedestrian, the RRT algorithm is used to plan the single-source trajectory for each pedestrian based on the sampled start and end points, avoiding the trajectories already planned by other pedestrians, to obtain the combination of collision-free pedestrian trajectories with the current sampled start and end points. This process is repeated to obtain multiple combinations of collision-free pedestrian trajectories as a collision-free pedestrian trajectory dataset.

[0035] Preferably, in the above method, the map of the real environment in which the mobile robot needs to actually operate is modeled as a 2D simulation environment represented by a 2D map in the following manner: obstacles in the environment are modeled in the 2D simulation environment, the movement range of pedestrians is determined, and the probability distribution of the starting point and ending point of all pedestrians in the 2D simulation environment is set according to the real environment in which the robot actually operates.

[0036] Based on the 2D simulation environment obtained from the modeling, when generating a set of collision-free pedestrian trajectories, sampling is first performed according to the probability distribution of pedestrian start and end points to obtain the start and end points of each pedestrian. The decision-making order of each pedestrian in the scene is then randomly shuffled. Following the decision-making order of each pedestrian, the RRT algorithm is used to plan a single-source trajectory for each pedestrian based on the sampled start and end points, avoiding the planned trajectories of other pedestrians. This yields a combination of collision-free trajectories for the current sampled start and end points. This process is repeated to obtain multiple combinations of collision-free pedestrian trajectories as a collision-free pedestrian trajectory dataset, including:

[0037] Step 11: Model the map corresponding to the real-world environment where the mobile robot needs to operate as a 2D simulation environment. In the 2D simulation environment, obstacles are modeled using polygons to represent their area. Define the boundaries of the 2D simulation environment to determine the movement range of pedestrians. Simultaneously, determine the number of pedestrians in the 2D simulation environment as N, based on the complex crowd scene in the real-world environment. Number the N pedestrians as 1, 2, ..., N. Call the i-th pedestrian among the N pedestrians pedestrian i. Determine the probability distribution S of the starting point and ending point for each pedestrian. 1 ,…S N With G 1 ,…,G N , and , where the probability distributions of the starting point and ending point of pedestrian i are represented by S, respectively.i and G i express;

[0038] Step 12, Initialize the collision-free pedestrian trajectory dataset Generate a set of corresponding collision-free pedestrian trajectories according to steps 121 to 125:

[0039] Step 121, based on the probability distribution S of the starting point and ending point for each pedestrian. 1 ,…S N With G 1 ,…,G N The starting point and ending point of each pedestrian in this trajectory generation are obtained by sampling, and the starting point and ending point of each pedestrian are recorded as follows: The maximum speed of all pedestrians is denoted as

[0040] Step 122: Randomly generate a permutation (a1, a2, ..., a... N This is used to shuffle the decision-making order of all pedestrians, arranging (a1, a2, ..., a...). N The l-th term in ) represents pedestrian a l It is the l-th pedestrian performing single-source trajectory planning, where l takes the value 1, 2, ..., N;

[0041] Step 123, let the initial set of planned trajectories be...

[0042] Step 124, process the generated permutations (a1, a2, ..., a) in order. N For each pedestrian, single-source trajectory planning is performed. The single-source trajectory planning process for each pedestrian is carried out according to the following steps 1241 to 1247, including:

[0043] Step 1241, initialize the tree node set as follows The nodes in the tree node set Tree are represented as (p x ,p y ,t), where (p x ,p y Let 'a' be the coordinate point and 't' be the timestamp. Arrange the values ​​of 'a' in the table. l The combination of the starting point of pedestrian i and timestamp 0 is: Add the node to the Tree set of the RRT algorithm, with its parent node being empty, and initialize the cumulative random selection counter to 0;

[0044] Step 1242: Select the endpoint of pedestrian i with the probability of hyperparameter k in the RRT algorithm. A random point p = (p_i - p_k) is selected in the feasible region of the 2D simulation environment with a probability of 1-k. x ,p yThe cumulative random point selection counter is incremented by 1, and it is determined whether the current cumulative random point selection counter exceeds the preset limit. If it exceeds the limit, False is returned, indicating that the trajectory planning has failed and the current trajectory planning is stopped. The process returns to step 121 and is repeated. If it does not exceed the limit, step 1243 is executed.

[0045] Step 1243: Find the node NearestNode in the Tree node set whose coordinates are closest to the random point p, and denote the coordinates and timestamp of this node as nearest and t, respectively. n In the interval Given a random velocity v, and based on a set decision time interval Δt, expand the coordinates from the nearest point to the random point p in the direction min(vΔt, ||nearest-p||2) to obtain the next possible successor point.

[0046] Step 1244: Traverse all planned trajectories in the planned trajectory set Traj, and let the current trajectory be the one being traversed. Determine the movement of pedestrian i from coordinate point nearest to successor point next, and the movement of pedestrian j from... arrive The movement is checked for collisions; and at the same time, it is determined whether the trajectory from the coordinate point nearest to the successor point next collides with the obstacles modeled in the 2D simulation environment.

[0047] Step 1245: If it is determined in step 1244 that the successor point next has a collision in any case, then repeat steps 1242 to 1244 until a collision-free successor point next is found by randomly selecting a point as a valid successor point next, and continue to execute step 1246.

[0048] Step 1246: Combine the valid successor point next with the timestamp t. n +1 combination is Add it to the tree node set Tree, and its parent node is NearestNode;

[0049] Step 1247: If the valid successor point next is the endpoint Then the planned trajectory τ is obtained by following the parent node of the record. i And the trajectory τ i Add the trajectory to the planned trajectory set Traj, and return True, indicating that the trajectory planning was successful; otherwise, repeat steps 1242 to 1246 until the successor point next is the endpoint.

[0050] Step 125, in step 124, if the permutation (a1, a2, ..., a...) is... NIf the single-source trajectory planning for each pedestrian corresponding to ) is successful, a valid combination of collision-free pedestrian trajectories represented by the planned trajectory set Traj is obtained, and this valid combination of collision-free pedestrian trajectories is added to the collision-free pedestrian trajectory dataset D.

[0051] Repeat steps 121 to 125 until multiple sets of pedestrian collision-free trajectory combinations are generated as the final collision-free pedestrian trajectory dataset D.

[0052] Preferably, in the above method, the pedestrian prediction module consists of a two-layer graph attention neural network and a one-layer variational neural network;

[0053] The graph attention neural network consists of a linear transformation layer W. p It consists of an attention module and a two-layer graph attention neural network that can capture the observable state w of each pedestrian. i,t The data is aggregated to obtain the integrated state x for each pedestrian. i,t ;

[0054] The variational neural network consists of a prior network ψ prior An encoder network ψ enc A decoder network ψ dec A feature extraction network ψ for the integrated state of a pedestrian x A latent variable extraction network ψ z With a h used to remember historical information i,t The GRU network is constructed; among which, the prior network ψ prior Encoder network ψ enc Decoder network ψ dec Pedestrian Integrated State Feature Extraction Network ψ x Latent variable extraction network ψ z All are composed of fully connected layers.

[0055] Preferably, in step 2 of the above method, the pedestrian prediction module is trained using the collision-free pedestrian trajectory dataset obtained in step 1, in the following manner:

[0056] Step 211: Randomly divide the collision-free pedestrian trajectory dataset D obtained in Step 1 into a training set D. train and test set D test ;

[0057] Step 212, from training set D train A pedestrian trajectory combination is randomly sampled from the data. The pedestrian position information at time t of this pedestrian trajectory combination is represented as the pedestrian observable state (w). 1,t ,w 2,t ,…,w N,tThe pedestrian prediction module estimates the loss function required for training by predicting the future state of pedestrians based on the summary of observable pedestrian states at time t and before. The steps are as follows:

[0058] Step 2121, let the current time be t, and take the positions of N pedestrians observed from the 2D simulation environment as observable states, denoted as (w 1,t ,w 2,t ,…,w N,t );

[0059] Step 2122, the first layer of the graph attention neural network of the pedestrian prediction module passes through a common linearly varying layer W. p The observable states (w) of N pedestrians 1,t ,w 2,t ,…,w N,t Extract as feature vector

[0060] Step 2123, using the extracted feature vectors The attention weight e of pedestrian i to pedestrian j is calculated using the following vector concatenation formula. ij :

[0061]

[0062] Where a is the attention function;

[0063] Step 2124: After normalizing the attention weights of pedestrian i to pedestrian j obtained in step 2123 using the following formula (1), the first-layer graph attention neural network performs a weighted average of all pedestrian feature vectors using the following formula (2), and further performs a nonlinear transformation using the nonlinear function σ to obtain the feature vector output by the first-layer graph attention neural network.

[0064]

[0065]

[0066] Step 2125: Convert the feature vector output by the first layer graph attention neural network into a single layer. As the input to the second-layer graph attention neural network, the feature vector output by the second-layer graph attention neural network is obtained after undergoing the same processing as in steps 2122 to 2124.

[0067] Step 2126, use the following formula to convert the feature vector The feature vector output by the two-layer graph attention neural network Summation is used as the final integrated state x of pedestrian i. i,t :

[0068]

[0069] Step 2127: The final integrated state x of pedestrian i, output by the second-layer graph attention neural network. i,t With recorded historical information h i,t-1 As input to the variational neural network, the encoder network ψ of the variational neural network... enc The latent variable z at time t is obtained through reasoning. i,t Then, the latent variable z at time t is used through the GRU network. i,t Update current historical information h i,t :

[0070] [μ i,t ,σ i,t ]=ψ enc (ψ x (x i,t ),h i,t-1 );

[0071]

[0072] h i,t =GRU(h i,t-1 ,ψ x (x i,t ),z i,t );

[0073] Step 2128, using the prior network ψ of the variational neural network prior and updated current historical information h i,t For the hidden variable z at the next time step i,t+1 Estimate the value and utilize the decoder network ψ dec Based on the estimated latent variable z at the next time step i,t+1 Predict the velocity v at the next moment i,t+1 :

[0074] [μ i,t+1 ,σ i,t+1 ]=ψ prior (h i,t );

[0075]

[0076] v i,t+1 =ψ dec (z i,t+1 ,h i,t );

[0077] Step 2129, calculate the loss of this prediction through the following ELBO loss function and add it to the total loss function:

[0078] -KL(q(z i,t |x i,≤t ,z i,<t ||p(z i,t |x i,<t ,z i,<t ))+logp(v i,t+1 |z i,≤t ,x i,<t );

[0079] where KL(·||·) represents the KL divergence; log(·) represents the natural logarithm; q and p are the prior distribution and posterior distribution of the latent variables represented by the prior network ψ prior and the encoder network ψ enc of the variational neural network respectively; x i,≤t represents all the integrated states obtained by the graph attention neural network according to the original pedestrian state input at time t and previous times; z i,≤t represents all the latent variable vectors that control the pedestrian integrated state distribution at time t and previous times, and the subscript <t indicates all the corresponding vectors before time t excluding time t.

[0080] A Repeat the above step 212 to obtain the estimated result of the loss function on the training set D train ;

[0081] Step 213, use the gradient descent method to train the pedestrian prediction module according to the estimated result of the loss function on the training set D train , and use the test set D test to estimate the loss function of the entire pedestrian prediction module on the test set D test according to step 212. Repeat step 212 and step 213 until the loss function of the pedestrian prediction module on the test set D test converges, that is, complete the training of the pedestrian prediction module.

[0082] Preferably, in the above method, the hybrid decision-making module includes a reinforcement learning network and an online planning unit; wherein, the reinforcement learning network is a neural network based on the actor-critic framework that includes a policy network π θ and a value network V φ ; the policy network π θ and the value network V φ share the same single-layer graph attention neural network at the bottom layer, and this graph attention neural network can utilize different linear transformation layers W R and W pFor the input mobile robot state u t With pedestrian observable state (w 1,t ,w 2,t ,…,w N,t Perform alignment processing to obtain the integrated state x of the mobile robot. 0,t ;

[0083] The policy network π θ With Value Network V φ Finally, each layer is equipped with a fully connected layer, which can adjust the integration status x of the mobile robot. 0,t The corresponding action probability distribution and value function are obtained.

[0084] Preferably, in step 2 of the above method, the policy network π of the reinforcement learning network of the hybrid decision module is trained using the collision-free pedestrian trajectory dataset obtained in step 1 and the PPO algorithm, in the following manner. θ With the corresponding value network V φ ,include:

[0085] Step 221: Use the simulation environment constructed in Step 1 as the training environment for reinforcement learning, and represent the state of the mobile robot at time t in the training environment as u. t State u at time t t Including the current location of the mobile robot Target location Current velocity vector With the current orientation η t The probability distributions for determining the starting point, ending point, and initial orientation of the mobile robot are S, respectively. 0 G 0 ,η 0 ;

[0086] Step 222: Determine the number of training rounds as K, the number of times pedestrian trajectories are collected in each training round as K1, and the number of times network parameters are updated as K2.

[0087] Step 223: In each round of training, first collect data according to the collection count K1 of the strategy network π. θ The current strategy's execution trajectory dataset D in a 2D simulation environment R During this process, the mobile robot will interact with pedestrians and collect execution trajectory dataset D according to the following steps. R Used for training subsequent policy networks;

[0088] Step 2231: From the collision-free pedestrian trajectory dataset D obtained in Step 1, randomly sample a collision-free pedestrian trajectory combination as the pedestrian action in this data collection. In the simulation environment, each pedestrian will directly follow the sampled corresponding pedestrian trajectory to move; simultaneously, from the probability distribution S of the mobile robot's starting point, ending point, and initial orientation... 0 G 0 ,η 0 The starting point of this mobile robot's data collection was obtained through sampling. end and initial orientation

[0089] Step 2232: At time t, when the mobile robot interacts with the pedestrian, the state of the environment is s. t =(u t ,w 1,t ,w 2,t ,…,w N,t According to the mobile robot's state u t and pedestrian observable state (w 1,t ,w 2,t ,…,w N,t Policy Network π θ The single-layer graph attention neural network utilizes different linear transformation layers W R With W P To obtain eigenvectors of equal length in The feature vector of the processed mobile robot;

[0090] Step 2233, using the extracted feature vectors The attention weight e of the mobile robot for pedestrian j is calculated using the following vector connection formula. 0j :

[0091]

[0092] Where a is the attention function;

[0093] Step 2234: After normalizing the attention weights of pedestrian j obtained in step 2233 using the following formula (3), the graph attention neural network performs a weighted average of the feature vectors of the mobile robot and all pedestrians using the following formula (4), and further uses the nonlinear function σ to perform a nonlinear transformation to obtain the policy network π. θ Required integration state x 0,t :

[0094]

[0095]

[0096] Step 2235, based on the integrated state x of the mobile robot 0,t According to the policy network π θ The fully connected layer obtains the probability distribution π of the optional actions at time t. θ (a t |s t );

[0097] Step 2236: The mobile robot operates in the simulation environment according to the probability distribution π of the available actions. θ (a t |s t Perform action sampling and, according to the sampling results, a t Execute;

[0098] Step 2237, the simulation environment reacts to the action a fed back by the mobile robot. t Perform an update and transition to the new state s at time t+1. t+1 The return R at time t is obtained according to the following return function R. t and the state transition at time t (s t ,a t ,s t+1 ,R t Add to execution trajectory dataset D R :

[0099]

[0100] Where Δdist represents the decrease in distance between the mobile robot and the target point at time t and time t+1; if the distance increases, Δdist is negative; v max Δt represents the maximum moving speed of the mobile robot; Δt represents the actual time interval between two decision moments.

[0101] Repeat steps 2231 to 2237 K1 times to obtain the mobile robot according to the policy network π. θ The collected execution trajectory dataset D R ;

[0102] Step 224, Replication Strategy Network π θ The parameters θ and the value network V φ The parameters φ are θ′ and φ′, based on the execution trajectory dataset D. R Update parameters θ′ and φ′:

[0103] Step 2241: In each update, the execution trajectory dataset D is processed using the GAE algorithm according to the following formula. R A set of state transitions (s) t ,a t ,s t+1 ,Rt )Calculate state s t Use action a t Advantage function A t (s t ,a t ):

[0104]

[0105] Where T is the execution trajectory dataset D R The trajectory length in V; γ is the discount factor; λ is a manually set parameter; V φ (·) is the value function calculated by the value network based on the input state through step 2233 and its subsequent fully connected layers;

[0106] Step 2242, according to the following formula based on the dominant function A t (s t ,a t Calculate the loss function for φ′:

[0107]

[0108] Step 2243, according to the policy gradient formula of the PPO algorithm as follows, based on the advantage function A t (s t ,a t Calculate the loss function for θ′:

[0109]

[0110] Where ∈ is a hyperparameter; the clip(x,l,r) function cuts x into the interval [l,r];

[0111] Step 2244: Update φ′ and θ′ using the gradient descent algorithm and the loss functions from steps 2242 and 2243, respectively.

[0112] Repeat steps 2241 to 2244 K2 times to obtain the final updated θ′ and φ′, and assign the values ​​of θ′ and φ′ to θ and φ respectively to update the policy network and value network;

[0113] Step 225: After the update is complete, clear the execution trajectory dataset D. R ;

[0114] Repeat steps 223 to 225 for a total of K rounds, then end the training.

[0115] Preferably, in step 3 of the above method, the mobile robot inputs the environmental state received at each decision moment into the reinforcement learning network of the trained hybrid decision module to obtain a probability distribution of possible actions in the current state. Then, the online planning unit of the hybrid decision module, based on the probability distribution of possible actions in the current state, combined with the future state of the pedestrian predicted by the pedestrian prediction module and the policy network and value network of the reinforcement learning network, determines the optimal action to be performed by the mobile robot, including:

[0116] Step 31: At each decision time t, the mobile robot obtains its state u through sensors. t and pedestrian observable status Where n is the actual number of pedestrians;

[0117] Step 32, obtain the mobile robot state u t Observable status of pedestrians As the current state The policy network π is input into the reinforcement learning network of the trained hybrid decision module. θ In the above, the policy network π of the reinforcement learning network θ Based on the current state, the probability distribution of possible actions π is obtained. θ (a t |s t );

[0118] Step 33, the online planning unit of the hybrid decision-making module starts from the possible action probability distribution π θ (a t |s t Sample m optional actions in )

[0119] Step 34, based on the input pedestrian observable status Using the trained pedestrian prediction module to determine the pedestrian's velocity (v) at time t+1 1,t+1 ,v 2,t+1 ,…,v n,t+1 The prediction is performed, and the pedestrian observable state at the next moment is obtained from the prediction results. Repeat the above process to obtain the pedestrian's observable state at the next d decision times;

[0120] Step 35, for each optional action Estimation is performed using K rounds of rehearsal. In the k-th round of rehearsal, the movements are first... Using the pedestrian observable state estimate obtained in step 34 at time t+1, the state at the next time step is calculated. and benefits Then, the policy network of the reinforcement learning network is used to obtain the probability distribution π of possible actions at time t+1.θ (a t+1 |s t+1 Based on the probability distribution of the possible actions, randomly sample an action a. t+1 As the action to be performed at time t+1, the state at time t+2 is calculated based on the pedestrian observable state estimate obtained in step 34 at time t+1. With benefits This process is repeated to obtain the payoff sequence for forward simulation of d decision times. With state The scores for this round of rehearsals are calculated based on the following formula:

[0121]

[0122] After K rounds of rehearsal, the scores are summed to obtain the results for each movement. Overall score The formula for summation is:

[0123]

[0124] Step 36: Select the action with the highest score from all available actions as the optimal action output by the hybrid decision module.

[0125]

[0126] This invention also provides a processing device, comprising:

[0127] At least one memory for storing one or more programs;

[0128] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the methods described above.

[0129] The present invention further provides a readable storage medium storing a computer program that can implement the above-described method when executed by a processor.

[0130] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the safe navigation method for mobile robots in complex crowd scenarios provided by the present invention is given by specific embodiments.

[0131] Example 1

[0132] like Figure 1As shown, this invention provides a safe navigation method for mobile robots in complex crowd scenarios. In real-world deployment scenarios, the increasing number of humans and the diversity of human behavior pose significant challenges to the safe navigation of mobile robots. Therefore, this invention proposes a crowd navigation method for mobile robots based on reinforcement learning. This method enables mobile robots to possess robust navigation strategies in crowd navigation tasks and to react correctly to complex and diverse pedestrian behaviors, thus completing navigation tasks safely and efficiently.

[0133] To achieve this goal, this method mainly consists of the following parts:

[0134] (1) Pedestrian trajectory generation module: The pedestrian trajectory generation module aims to generate as many collision-free pedestrian trajectories as possible. These trajectories will be used to train the reinforcement learning network in the subsequent pedestrian prediction module and hybrid decision module, so as to reduce the difference between the training process and the pedestrian strategy in the real environment, and fundamentally enable the mobile robot to cope with different pedestrian behaviors.

[0135] (2) Pedestrian Prediction Module: The pedestrian prediction module is trained using collision-free pedestrian trajectories generated by the pedestrian trajectory generation module. During the execution phase, it can summarize the integrated status of pedestrians in real time and predict the future speed of pedestrians. This prediction result will assist the hybrid decision-making module in making targeted responses.

[0136] (3) Hybrid Decision Module: The hybrid decision module consists of two parts: a reinforcement learning network and an online planning unit. Through training in a simulation environment, the reinforcement learning network is responsible for providing a probability distribution of possible actions based on the current environmental state when running in the actual environment. The online planning unit will then solve for the approximate optimal action for possible pedestrian behaviors in the current environment based on this probability distribution of possible actions, so as to control the mobile robot. This allows it to better adapt to different possible pedestrian behaviors.

[0137] The technical solution of the present invention will be fully described below with reference to the accompanying drawings in the embodiments of the present invention: The present invention is divided into two stages: training and execution. In the training stage, the pedestrian trajectory generation module generates a collision-free pedestrian trajectory dataset based on the environment in which the mobile robot operates. This collision-free pedestrian trajectory dataset will be used for training the reinforcement learning network in the pedestrian prediction module and the hybrid decision module. In the execution stage, the environmental state received by the mobile robot at each moment is input into the trained reinforcement learning policy network to obtain a probability distribution of possible actions in the current state. Then, in the online planning unit, the predicted future state of the pedestrian by the trained pedestrian prediction module and the policy network and value network of the reinforcement learning network are combined to obtain the final output optimal action for the mobile robot to execute, thereby achieving safe navigation of the mobile robot.

[0138] The following section first describes the network structure and training process designed for each module in the training phase:

[0139] (1) Pedestrian trajectory generation module:

[0140] To enhance the adaptability of mobile robots to diverse pedestrian behaviors, this invention proposes a novel pedestrian hypothesis for training: pedestrians only consider achieving collision-free trajectory planning, without concern for other properties of their planned trajectories, such as length or time. This hypothesis, while preserving the basic logic of pedestrians, broadens the range of possible pedestrian behaviors as much as possible, enabling the mobile robot to handle a wide variety of pedestrian strategies during the learning process, thereby reducing potential performance loss when transferring navigation strategies to real-world scenarios.

[0141] Based on this collision-free pedestrian assumption, the pedestrian trajectory generation module is responsible for generating as many collision-free pedestrian trajectories as possible. These trajectories serve two purposes: firstly, they will be used as time-series data for training the pedestrian prediction module; secondly, they will be replayed in the simulated environment of reinforcement learning training to generate diverse pedestrian behaviors.

[0142] To generate as diverse pedestrian trajectories as possible, the pedestrian trajectory generation module in this invention uses a centralized multi-agent trajectory planning algorithm. Before the algorithm executes, a 2D simulation environment is created by modeling the actual environment in which the mobile robot needs to operate. Obstacles are modeled within this 2D simulation environment, and the movement range of the pedestrians is determined. The probability distribution of the starting and ending points of all pedestrians in this 2D simulation environment is set based on the actual operating environment. In this specific embodiment, the mobile robot will be deployed in a square blank area with sides of approximately 5m. In this environment, the starting points of five pedestrians will be distributed at any point on a circle with a radius of 1.8m to 2.2m, centered on the square. Their ending points will be located at the intersection of the extended line connecting the starting point and the center of this circle on the other side of the circle. Furthermore, the starting points of all pedestrians are non-overlapping.

[0143] After obtaining the 2D simulation environment through modeling, the pedestrian trajectory generation module, when generating a set of collision-free pedestrian trajectories, first samples according to the probability distribution of pedestrian start and end points to obtain the start and end points of each pedestrian. Then, it randomly shuffles the decision-making order of each pedestrian in the scene. Following the decision-making order of each pedestrian, and based on the sampled start and end points, it uses the RRT algorithm to perform single-source trajectory planning for each pedestrian, avoiding the planned trajectories of other pedestrians. This results in a combination of collision-free trajectories for the current sampled start and end points. This process is repeated to obtain multiple combinations of collision-free pedestrian trajectories as a collision-free pedestrian trajectory dataset. The complete steps are described below:

[0144] Step 11: Model the map corresponding to the real-world environment where the mobile robot needs to operate as a 2D simulation environment. In the 2D simulation environment, obstacles are modeled using polygons to represent their area. Define the boundaries of the 2D simulation environment to determine the movement range of pedestrians. Simultaneously, determine the number of pedestrians in the 2D simulation environment to be N=5, based on the complex crowd scene in the real-world environment. Number the N pedestrians as 1, 2, ..., N. Call the i-th pedestrian among the N pedestrians pedestrian i. Determine the probability distribution S of the starting point and ending point for each pedestrian. 1 ,…S N With G 1 ,…,G N , and , where the probability distributions of the starting point and ending point of pedestrian i are represented by S, respectively. i and G i express;

[0145] Step 12, Initialize the collision-free pedestrian trajectory dataset Generate a set of corresponding collision-free pedestrian trajectories according to steps 121 to 125:

[0146] Step 121, based on the probability distribution S of the starting point and ending point for each pedestrian. 1 ,…S N With G 1 ,…,G N The starting point and ending point of each pedestrian in this trajectory generation are obtained by sampling, and the starting point and ending point of each pedestrian are recorded as follows: The maximum speed of all pedestrians is denoted as

[0147] Step 122: Randomly generate a permutation (a1, a2, ..., a... N This is used to shuffle the decision-making order of all pedestrians, arranging (a1, a2, ..., a...). N The l-th term in ) represents pedestrian a l It is the l-th pedestrian performing single-source trajectory planning, where l takes the value 1, 2, ..., N;

[0148] Step 123, let the initial set of planned trajectories be...

[0149] Step 124, process the generated permutations (a1, a2, ..., a) in order. N For each pedestrian, single-source trajectory planning is performed. The single-source trajectory planning process for each pedestrian is carried out according to the following steps 1241 to 1247, including:

[0150] Step 1241, initialize the tree node set as follows The nodes in the tree node set Tree are represented as (p x ,p y,t), where (p x ,p y Let 'a' be the coordinate point and 't' be the timestamp. Arrange the values ​​of 'a' in the table. l The combination of the starting point of pedestrian i and timestamp 0 is: Add the node to the Tree set of the RRT algorithm, with its parent node being empty, and initialize the cumulative random selection counter to 0;

[0151] Step 1242: Select the endpoint of pedestrian i with the probability of hyperparameter k in the RRT algorithm. A random point p = (p_i - p_k) is selected in the feasible region of the 2D simulation environment with a probability of 1-k. x ,p y The cumulative random point selection counter is incremented by 1, and it is determined whether the current cumulative random point selection counter exceeds the preset limit. If it exceeds the limit, False is returned, indicating that the trajectory planning has failed and the current trajectory planning is stopped. The process returns to step 121 and is repeated. If it does not exceed the limit, step 1243 is executed.

[0152] Step 1243: Find the node NearestNode in the Tree node set whose coordinates are closest to the random point p, and denote the coordinates and timestamp of this node as nearest and t, respectively. n In the interval Given a random velocity v, and based on a set decision time interval Δt, expand the coordinates from the nearest point to the random point p in the direction min(vΔt, ||nearest-p||2) to obtain the next possible successor point.

[0153] Step 1244: Traverse all planned trajectories in the planned trajectory set Traj, and let the current trajectory be the one being traversed. Determine the movement of pedestrian i from coordinate point nearest to successor point next, and the movement of pedestrian j from... arrive The movement of the pedestrians is checked for collisions; and at the same time, it is determined whether the trajectory from the coordinate point nearest to the successor point next collides with the obstacles modeled in the 2D simulation environment. In this embodiment of the invention, since all pedestrians are confined to an open space with a side length of 5 meters, areas outside this range are considered as obstacles.

[0154] Step 1245: If it is determined in step 1244 that the successor point next has a collision in any case, then repeat steps 1242 to 1244 until a collision-free successor point next is found by randomly selecting a point as a valid successor point next, and continue to execute step 1246.

[0155] Step 1246: Combine the valid successor point next with the timestamp t.n +1 combination is Add it to the tree node set Tree, and its parent node is NearestNode;

[0156] Step 1247: If the valid successor point next is the endpoint Then the planned trajectory τ is obtained by following the parent node of the record. i And the trajectory τ i Add the trajectory to the planned trajectory set Traj, and return True, indicating that the trajectory planning was successful; otherwise, repeat steps 1242 to 1246 until the successor point next is the endpoint.

[0157] Step 125, in step 124, if the permutation (a1, a2, ..., a...) is... N If the single-source trajectory planning for each pedestrian corresponding to ) is successful, a valid combination of collision-free pedestrian trajectories represented by the planned trajectory set Traj is obtained, and this valid combination of collision-free pedestrian trajectories is added to the collision-free pedestrian trajectory dataset D.

[0158] Repeat steps 121 to 125 until multiple sets of pedestrian collision-free trajectory combinations are generated as the final collision-free pedestrian trajectory dataset D.

[0159] (2) Pedestrian prediction module:

[0160] Based on the generated collision-free pedestrian trajectory dataset D, the pedestrian prediction module constructs a deep learning network to learn the temporal information of pedestrian trajectories in the collision-free pedestrian trajectory dataset D, and further predicts the future behavior of pedestrians. In this invention, the pedestrian prediction module mainly consists of two networks: first, a two-layer graph attention neural network is introduced to summarize pedestrian information and obtain the integrated state of each pedestrian; second, a variational neural network is used to predict the speed of the pedestrian at the next moment based on the integrated state of a single pedestrian.

[0161] Specifically, the graph attention neural network consists of a single linear transformation layer W. p It consists of an attention module and a two-layer graph attention neural network that can capture the observable state w of each pedestrian. i,t The data is aggregated to obtain the integrated state x for each pedestrian. i,t The variational neural network consists of a prior network ψ prior An encoder network ψ enc A decoder network ψ dec A feature extraction network ψ for the integrated state of a pedestrian x A latent variable extraction network ψ z With a h used to remember historical information i,tThe GRU network is constructed; among which, the prior network ψ prior Encoder network ψ enc Decoder network ψ dec Pedestrian Integrated State Feature Extraction Network ψ x Latent variable extraction network ψ z All are composed of fully connected layers.

[0162] The following section details the training method for the pedestrian prediction module based on prediction loss, including:

[0163] Step 211: Randomly divide the collision-free pedestrian trajectory dataset D obtained in Step 1 into a training set D. train and test set D test ;

[0164] Step 212, from training set D train A pedestrian trajectory combination is randomly sampled from the data. The pedestrian position information at time t of this pedestrian trajectory combination is represented as the pedestrian observable state (w). 1,t ,w 2,t ,…,w N,t The pedestrian prediction module estimates the loss function required for training by predicting the future state of pedestrians based on the summary of observable pedestrian states at time t and before. The steps are as follows:

[0165] Step 2121, let the current time be t, and take the positions of N pedestrians observed from the 2D simulation environment as observable states, denoted as (w 1,t ,w 2,t ,…,w N,t );

[0166] Step 2122, the first layer of the graph attention neural network of the pedestrian prediction module passes through a common linearly varying layer W. p The observable states (w) of N pedestrians 1,t ,w 2,t ,…,w N,t Extract as feature vector

[0167] Step 2123, using the extracted feature vectors The attention weight e of pedestrian i to pedestrian j is calculated using the following vector concatenation formula. ij :

[0168]

[0169] Where a is the attention function;

[0170] Step 2124: After normalizing the attention weights of pedestrian i to pedestrian j obtained in step 2123 using the following formula (1), the first-layer graph attention neural network performs a weighted average of all pedestrian feature vectors using the following formula (2), and further performs a nonlinear transformation using the nonlinear function σ to obtain the feature vector output by the first-layer graph attention neural network.

[0171]

[0172]

[0173] Step 2125: Convert the feature vector output by the first layer graph attention neural network into a single layer. As the input to the second-layer graph attention neural network, the feature vector output by the second-layer graph attention neural network is obtained after undergoing the same processing as in steps 2122 to 2124.

[0174] Step 2126, use the following formula to convert the feature vector The feature vector output by the two-layer graph attention neural network Summation is the final integrated state of pedestrian i.

[0175]

[0176] Step 2127: The final integrated state x of pedestrian i, output by the second-layer graph attention neural network. i，t With recorded historical information h i，t-1 As input to the variational neural network, the encoder network ψ of the variational neural network... enc The latent variable z at time t is obtained through reasoning. i，t Then, the latent variable z at time t is used through the GRU network. i，t Update current historical information h i，t :

[0177] [μ i,t ,σ i,t ]=ψ enc (ψ x (x i,t ),h i,t-1 );

[0178]

[0179] h i,t =GRU(h i,t-1 ,ψ x (x i,t ),z i,t );

[0180] Step 2128, through the prior network ψ of the variational neural network prior and the updated current historical information h i,t , estimate the latent variable z at the next moment i,t+1 and use the decoder network ψ dec to predict the speed v at the next moment according to the estimated latent variable z at the next moment i,t+1 : i,t+1

[0181] [μ i,t+1 , σ i,t+1 = ψ<所给原文此处有误,推测应为 prior (h i,t );

[0182]

[0183] v i,t+1 = ψ dec (z i,t+1 , h i,t );

[0184] Step 2129, calculate the loss of this prediction through the following ELBO loss function and add it to the total loss function:

[0185] -KL(q(z i,t |x i,≤t , z i,<t ||p(z i,t |x i,<t , z i,<t )) + log p(v i,t+1 |z i,≤t , x i,<t );

[0186] where KL(·||·) represents the KL divergence; log(·) represents the natural logarithm; q and p are respectively the prior distribution and posterior distribution of the latent variable represented by the prior network ψ prior and ψ enc of the encoder network; x i,≤t represents all the integrated states obtained by the graph attention neural network according to the original pedestrian i state input at time t and previous times; z i,≤t represents all the latent variables that control the integrated state distribution of pedestrian i at time t and previous times, and the subscript <t indicates all the corresponding integrated states or latent variables before time t excluding time t;

[0187] Repeat the above Step 212 to obtain the estimated result of the loss function on the training set D train ; ​

[0188] Step 213: Use gradient descent according to the training set D train The estimation results of the upper loss function are used to train the pedestrian prediction module, and the test set D is used. test According to step 212, the entire pedestrian prediction module is currently estimated to be in the test set D. test Using the loss function on the test set D, repeat steps 212 and 213 until the pedestrian prediction module performs well on the test set D. test The training of the pedestrian prediction module is complete when the loss function on the graph converges.

[0189] (3) Hybrid Decision Module

[0190] In the decision-making process, to balance computational efficiency and optimality, the hybrid decision-making module of this invention consists of a reinforcement learning network and an online planning unit. In the reinforcement learning network, by modeling the interaction between the mobile robot and the pedestrian as a Markov decision process, and utilizing the PPO algorithm, the mobile robot can adaptively learn a stochastic reinforcement learning policy network π in a simulation environment. θ With the corresponding value network V φ During the execution of the mobile robot, it can further utilize the online planning unit to adjust its speed based on real-time pedestrian behavior, so as to better adapt to the diversity of pedestrian behavior in the real environment.

[0191] Specifically, in the reinforcement learning phase, this invention models the interaction between the mobile robot and the pedestrian as a Markov decision process. S is a set of states, and state s is the state at time t. t ∈S is determined by the mobile robot state u t observable state of pedestrians w t Composition, that is Mobile robot status u t Including the current location of the mobile robot Target location Current speed With the current orientation η t The observable state of pedestrian i The pedestrian's current location A is the action space, and the two-dimensional action a is... t =(v t ,ω t )∈A represent the linear velocity v of the mobile robot at time t. t With angular velocity ω tThe P transition matrix represents the rules for environmental updates. In this invention, the environmental rules mainly describe the state of the mobile robot at the next moment, based on the decision interval and the given actions and states of the mobile robot. Assuming the decision interval for the mobile robot is Δt, if the action selected by the mobile robot is (v... t ,ω t ), current location is The current orientation is η t Then the speed of the moving robot at the next moment Position coordinates With η t+1 for:

[0192] η t+1 =η t +ω t Δt;

[0193]

[0194]

[0195]

[0196]

[0197] To enable the mobile robot to reach its destination safely and efficiently, this invention uses the following revenue function R:

[0198]

[0199] Where Δdist represents the decrease in distance between the mobile robot and the target point at time t and time t+1; if the distance increases, Δdist is negative; v max Δt represents the maximum moving speed of the mobile robot; Δt represents the actual time interval between two decision moments.

[0200] The probability distribution of the initial state s0 in the Markov decision process reflects the initial state distribution of the environment. In this embodiment of the invention, the starting and ending point distributions of the simulation environment and pedestrians have been set in step one. The starting and ending point distributions of the mobile robot are similar to those of the pedestrians. The starting point will be located at any point on a circle with a radius of 1.8 meters to 2.2 meters, centered on a square. The ending point is located at the intersection of the extended line connecting the starting point and the center of this circle on the other side of the circle. The starting point of the mobile robot does not coincide with the starting point of any pedestrian, and the mobile robot initially faces the ending point.

[0201] For this abstracted Markov decision process, this invention constructs a policy network π θ With Value Network Vφ A neural network based on the actor-critic framework; policy network π θ With Value Network V φ Sharing the same single-layer graph attention neural network at the bottom layer, this graph attention neural network can utilize different linearly varying layers W. R With W p For the input mobile robot state u t With pedestrian observable state (w 1,t ,w 2,t ,…,w N,t Perform alignment processing to obtain the integrated state x of the mobile robot. 0,t The policy network π θ With Value Network V φ Finally, each layer is equipped with a fully connected layer, which can adjust the integration status x of the mobile robot. 0,t The corresponding action probability distribution and value function are obtained.

[0202] The training steps for the reinforcement learning network constructed above in a simulation environment are as follows:

[0203] Step 221: Use the simulation environment constructed in Step 1 as the training environment for reinforcement learning, and represent the state of the mobile robot at time t in the training environment as u. t State u at time t t Including the current location of the mobile robot Target location Current velocity vector With the current orientation η t The probability distributions for determining the starting point, ending point, and initial orientation of the mobile robot are S, respectively. 0 G 0 ,η 0 ;

[0204] Step 222: Determine the number of training rounds as K, the number of times pedestrian trajectories are collected in each training round as K1, and the number of times network parameters are updated as K2.

[0205] Step 223: In each round of training, first collect data according to the collection count K1 of the strategy network π. θ The current strategy's execution trajectory dataset D in a 2D simulation environment R During this process, the mobile robot will interact with pedestrians and collect execution trajectory dataset D according to the following steps. R Used for training subsequent policy networks;

[0206] Step 2231: From the collision-free pedestrian trajectory dataset D obtained in Step 1, randomly sample a collision-free pedestrian trajectory combination as the pedestrian action in this data collection. In the simulation environment, each pedestrian will directly follow the sampled corresponding pedestrian trajectory to move; simultaneously, from the probability distribution S of the mobile robot's starting point, ending point, and initial orientation... 0 G 0 ,η 0 The starting point of this mobile robot's data collection was obtained through sampling. end and initial orientation

[0207] Step 2232: At time t, when the mobile robot interacts with the pedestrian, the state of the environment is s. t =(u t ,w 1,t ,w 2,t ,…,w N,t According to the mobile robot's state u t and pedestrian observable state (w 1,t ,w 2,t ,…,w N,t Policy Network π θ The single-layer graph attention neural network utilizes different linear transformation layers W R With W P To obtain eigenvectors of equal length in The feature vector of the processed mobile robot;

[0208] Step 2233, using the extracted feature vectors The attention weight e of the mobile robot for pedestrian j is calculated using the following vector connection formula. 0j :

[0209]

[0210] Where a is the attention function;

[0211] Step 2234: After normalizing the attention weights of pedestrian j obtained in step 2233 using the following formula (3), the graph attention neural network performs a weighted average of the feature vectors of the mobile robot and all pedestrians using the following formula (4), and further uses the nonlinear function σ to perform a nonlinear transformation to obtain the policy network π. θ Required integration state x 0,t :

[0212]

[0213]

[0214] Step 2235, based on the integrated state x of the mobile robot 0,t According to the policy network π θ The fully connected layer obtains the probability distribution π of the optional actions at time t. θ (a t |s t );

[0215] Step 2236: The mobile robot operates in the simulation environment according to the probability distribution π of the available actions. θ (a t |s t Perform action sampling and, according to the sampling results, a t Execute;

[0216] Step 2237, the simulation environment reacts to the action a fed back by the mobile robot. t Perform an update and transition to the new state s at time t+1. t+1 The return R at time t is obtained according to the following return function R. t and the state transition at time t (s t ,a t ,s t+1 ,R t Add to execution trajectory dataset D R :

[0217]

[0218] Repeat steps 2231 to 2237 K1 times to obtain the mobile robot according to the policy network π. θ The collected execution trajectory dataset D R ;

[0219] Step 224, Replication Strategy Network π θ The parameters θ and the value network V φ The parameters φ are θ′ and φ′, based on the execution trajectory dataset D. R Update parameters θ′ and φ′:

[0220] Step 2241: In each update, the execution trajectory dataset D is processed using the GAE algorithm according to the following formula. R A set of state transitions (s) t ,a t ,s t+1 ,R t )Calculate state s t Use action a t Advantage function A t (s t ,a t ):

[0221]

[0222] Where T is the execution trajectory dataset D R The trajectory length in V; γ is the discount factor; λ is a manually set parameter; V φ (·) is the value function calculated by the value network based on the input state through step 2233 and its subsequent fully connected layers;

[0223] Step 2242, according to the following formula based on the dominant function A t (s t ,a t Calculate the loss function for φ′:

[0224]

[0225] Step 2243, according to the policy gradient formula of the PPO algorithm as follows, based on the advantage function A t (s t ,a t Calculate the loss function for θ′:

[0226]

[0227] Where ∈ is a hyperparameter; the clip(x,l,r) function cuts x into the interval [l,r];

[0228] Step 2244: Update φ′ and θ′ using the gradient descent algorithm and the loss functions from steps 2242 and 2243, respectively.

[0229] Repeat steps 2241 to 2244 K2 times to obtain the final updated θ′ and φ′, and assign the values ​​of θ′ and φ′ to θ and φ respectively to update the policy network and value network;

[0230] Step 225: After the update is complete, clear the execution trajectory dataset D. R ;

[0231] Repeat steps 223 to 225 for a total of K rounds, then end the training.

[0232] The following describes the process of training the above modules in a simulation environment and then deploying them to a mobile robot for execution. In this embodiment of the invention, the mobile robot turtlebot2 will run in a real-world environment similar to the simulation environment, except that the actual number of pedestrians is changed to 4; the rest of the environmental settings are consistent with the simulation environment. This demonstrates the advantages of introducing a graph attention neural network into the pedestrian prediction module and the hybrid decision-making module, and that this network structure can flexibly adapt to different numbers of pedestrians in the environment. In the real-world environment, the mobile robot turtlebot2 will use an external camera as a sensor to obtain the environmental state defined in the Markov decision-making process in real time.

[0233] In the actual execution phase, the mobile robot inputs the environmental state received at each decision moment into the reinforcement learning network of the trained hybrid decision module to obtain a probability distribution of possible actions in the current state. Then, the online planning unit of the hybrid decision module, based on the probability distribution of possible actions in the current state, combined with the future state of the pedestrian predicted by the pedestrian prediction module and the policy network and value network of the reinforcement learning network, determines the optimal action to be performed by the mobile robot.

[0234] Step 31: At each decision time t, the mobile robot obtains its state u through sensors. t and pedestrian observable status Where n = 4 is the actual number of pedestrians;

[0235] Step 32, obtain the mobile robot state u t Observable status of pedestrians As the current state The policy network π is input into the reinforcement learning network of the trained hybrid decision module. θ In the above, the policy network π of the reinforcement learning network θ Based on the current state, the probability distribution of possible actions π is obtained. θ (a t |s t );

[0236] Step 33, the online planning unit of the hybrid decision-making module starts from the possible action probability distribution π θ (a t |s t Sample m optional actions in )

[0237] Step 34, based on the input pedestrian observable status Using the trained pedestrian prediction module to determine the pedestrian's velocity (v) at time t+1 1,t+1 ,v 2,t+1 ,…,v n,t+1The prediction is performed, and the pedestrian observable state at the next moment is obtained from the prediction results. Repeat the above process to obtain the pedestrian's observable state at the next d decision times;

[0238] Step 35, for each optional action Estimation is performed using K rounds of rehearsal. In the k-th round of rehearsal, the movements are first... Using the pedestrian observable state estimate obtained in step 34 at time t+1, the state at the next time step is calculated. and benefits Then, the policy network of the reinforcement learning network is used to obtain the probability distribution π of possible actions at time t+1. θ (a t+1 |s t+1 Based on the probability distribution of the possible actions, randomly sample an action a. t+1 As the action to be performed at time t+1, the state at time t+2 is calculated based on the pedestrian observable state estimate obtained in step 34 at time t+1. With benefits This process is repeated to obtain the payoff sequence for forward simulation of d decision times. With state The scores for this round of rehearsals are calculated based on the following formula:

[0239]

[0240] After K rounds of rehearsal, the scores are summed to obtain the results for each movement. Overall score The formula for summation is:

[0241]

[0242] Step 36: Select the action with the highest score from all available actions as the optimal action output by the hybrid decision module. Safe navigation has been achieved by directly controlling turtlebot2 for movement:

[0243]

[0244] Compared to traditional algorithms based on speed obstacles and trajectory prediction, the method of this invention continues the advantages of deep reinforcement learning methods, namely, through offline training in a simulation environment, the mobile robot can achieve both optimality and timeliness in online crowd navigation.

[0245] Compared to crowd navigation algorithms also based on reinforcement learning, the method of this invention achieves breakthroughs in two main aspects: First, because this invention uses collision-free pedestrian trajectories generated by a pedestrian trajectory generation module to train the reinforcement learning network and the pedestrian prediction module, the navigation behavior generated by this invention not only achieves good performance in the training environment but also responds well to diverse and more random pedestrian strategies in the testing environment. Second, this invention uses an online planning unit to further evaluate the probability distribution of optional actions obtained from the navigation strategy, allowing the mobile robot to explicitly consider the movement of other pedestrians and select actions during execution. This further enhances the adaptability of the mobile robot to different pedestrian behaviors during navigation.

[0246] In this invention, a graph attention neural network is applied to the construction of the pedestrian prediction module and the reinforcement learning network. Compared to a typical fully connected network structure, the graph attention neural network, by using attention modules, better models the relationships between pedestrians, providing better state representations for subsequent variational neural networks and reinforcement learning networks. This further enhances the navigation capabilities of mobile robots in complex environments.

[0247] During execution, the online planning unit can perform pre-planning based on the output of the pedestrian prediction module to further select better actions. This allows the mobile robot's decisions to be based on possible pedestrian decisions. An alternative approach is to directly model the prediction of the pedestrian's future speed into the state representation of the Markov decision process, rather than performing online planning during execution. This allows the reinforcement learning policy to make decisions based on this additional information. While theoretically representing the prediction directly into the state could allow the prediction of the pedestrian's future speed to influence the mobile robot's possible behavior, this implicit influence is difficult to fully realize due to the implementation methods of neural networks and gradient descent.

[0248] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0249] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for safe navigation of a mobile robot in a complex crowd scenario, characterized in that, Used to control the safe navigation of mobile robots in complex crowd scenarios, including: Step 1: Construct a simulation environment based on the actual operating environment of the mobile robot using the pedestrian trajectory generation module, and generate a corresponding collision-free pedestrian trajectory dataset in the simulation environment. In step 1, a simulation environment is constructed by the pedestrian trajectory generation module based on the actual operating environment of the mobile robot, and a corresponding collision-free pedestrian trajectory dataset is generated in the simulation environment, including: The map of the real environment in which the mobile robot needs to operate is modeled as a 2D simulation environment represented by a 2D graph. Obstacles in the environment are modeled in the 2D simulation environment, and the movement range of pedestrians is determined. The probability distribution of the starting point and ending point of all pedestrians in the 2D simulation environment is set according to the real environment in which the robot actually operates. Based on the 2D simulation environment obtained from the modeling, when generating a set of collision-free pedestrian trajectories, the starting point and ending point of each pedestrian are first sampled according to the probability distribution of the pedestrian's starting point and ending point. The decision order of each pedestrian in the scene is randomly shuffled. According to the decision order of each pedestrian, the RRT algorithm is used to plan the single-source trajectory for each pedestrian based on the starting point and ending point of the sampled pedestrians, avoiding the trajectories already planned by other pedestrians, so as to obtain the combination of collision-free pedestrian trajectories of the current sampled starting point and ending point. This process is repeated to obtain multiple combinations of collision-free pedestrian trajectories as a collision-free pedestrian trajectory dataset. Step 2: Train the reinforcement learning networks of the pedestrian prediction module and the hybrid decision module using the collision-free pedestrian trajectory dataset obtained in Step 1. Step 3: When the mobile robot navigates in complex crowd scenes in a real-world environment using the trained pedestrian prediction module and hybrid decision module, the mobile robot inputs the environmental state received at each decision moment into the reinforcement learning network of the trained hybrid decision module to obtain a probability distribution of possible actions in the current state. Then, the online planning unit of the hybrid decision module, based on the probability distribution of possible actions in the current state, combined with the future state of the pedestrian predicted by the pedestrian prediction module and the policy network and value network of the reinforcement learning network, determines the optimal action to be performed by the mobile robot. This enables the mobile robot to perform the optimal action corresponding to each decision moment, achieving safe navigation in complex crowd scenes in a real-world environment. 2.The method for safe navigation of mobile robots in complex crowd scenarios according to claim 1, wherein, In the method, the map of the real environment in which the mobile robot needs to actually operate is modeled as a 2D simulation environment represented by a 2D map in the following manner: obstacles in the environment are modeled in the 2D simulation environment, the movement range of pedestrians is determined, and the probability distribution of the starting point and ending point of all pedestrians in the 2D simulation environment is set according to the real environment in which the robot actually operates. Based on the 2D simulation environment obtained from the modeling, when generating a set of collision-free pedestrian trajectories, sampling is first performed according to the probability distribution of pedestrian start and end points to obtain the start and end points of each pedestrian. The decision-making order of each pedestrian in the scene is then randomly shuffled. Following the decision-making order of each pedestrian, the RRT algorithm is used to plan a single-source trajectory for each pedestrian based on the sampled start and end points, avoiding the planned trajectories of other pedestrians. This yields a combination of collision-free trajectories for the current sampled start and end points. This process is repeated to obtain multiple combinations of collision-free pedestrian trajectories as a collision-free pedestrian trajectory dataset, including: Step 11: Model the map corresponding to the real-world environment where the mobile robot needs to operate as a 2D simulation environment. In the 2D simulation environment, obstacles are modeled using polygons to represent their area. The boundaries of the 2D simulation environment are defined to determine the pedestrian movement range. Simultaneously, the number of pedestrians in the 2D simulation environment is determined based on the complex crowd scene in the real-world environment. ,Will Each pedestrian is numbered as follows: ,Will The first of the pedestrians A pedestrian is called a pedestrian. And determine the probability distribution of the starting point and ending point for each pedestrian. and , and, among which, pedestrians The probability distributions of the starting and ending points are respectively used as and express; Step 12, Initialize the collision-free pedestrian trajectory dataset Generate a set of corresponding collision-free pedestrian trajectories by following steps 121 to 125: Step 121: Based on the probability distribution of each pedestrian's starting and ending points. and The starting point and ending point of each pedestrian in this trajectory generation are obtained by sampling, and the starting point and ending point of each pedestrian are recorded as follows: The maximum speed of all pedestrians is recorded as ; Step 122, randomly generate a permutation Used to disrupt the decision-making order of all pedestrians, arranging The first in Item indicates pedestrian It is the first A pedestrian performing single-source trajectory planning. Values ; Step 123, let the initial set of planned trajectories be... ; Step 124, process the generated permutations in order. For each pedestrian, single-source trajectory planning is performed. The single-source trajectory planning process for each pedestrian is carried out according to the following steps 1241 to 1247, including: Step 1241, initialize the tree node set as follows Tree node set The nodes in are represented as ,in Let be the coordinate point. For timestamps, arrange in order Corresponding pedestrians The combination of the starting point and timestamp 0 is Tree node set added to the RRT algorithm In the middle, and its parent node is empty, initialize the cumulative random selection counter to 0; Step 1242, using the hyperparameters of the RRT algorithm Probability of choosing pedestrians The End ,by The probability of selecting a random point within the feasible region of the 2D simulation environment. The cumulative random point selection counter is incremented by 1, and it is checked whether the current cumulative random point selection counter exceeds the preset limit. If it does, the process returns. If the trajectory planning fails, stop the current trajectory planning and return to step 121 to re-execute; if it does not exceed the limit, proceed to step 1243. Step 1243, find the set of tree nodes. Distance between the middle node coordinates and the random point The nearest node And record the coordinates and timestamp of this node as follows: and In the interval Random speed And based on the set decision time interval From coordinate point To random points directional expansion Find the next possible successor point ; Step 1244: Traverse the set of planned trajectories All planned trajectories, assuming the current trajectories visited are... Determine pedestrians From coordinate point To the successor point movement and pedestrians from arrive Does the movement collide? And simultaneously determine whether the movement originates from the coordinate point. To the successor point Does the trajectory collide with obstacles modeled in the 2D simulation environment? Step 1245, if the successor point was determined in step 1244 If a collision occurs in any case, repeat steps 1242 to 1244 until a collision-free successor point is found by randomly selecting a point. As a legitimate successor point And continue with step 1246; Step 1246, select the valid successor point. With timestamp Combination Add to tree node set In, and its parent node is ; Step 1247, if a valid successor point End point Then the planned trajectory is obtained by following the parent node of the record. and the trajectory Add to the set of planned trajectories In the middle, return If the trajectory planning is successful, then repeat steps 1242 to 1246 until the successor point. End point ; Step 125, in step 124, if the arrangement If the single-source trajectory planning for each pedestrian is successful, then a set of planned trajectories is obtained. The valid combination of collision-free pedestrian trajectories is represented, and this valid combination of collision-free pedestrian trajectories is added to the collision-free pedestrian trajectory dataset. middle; Repeat steps 121 to 125 until multiple sets of collision-free pedestrian trajectory combinations are generated as the final collision-free pedestrian trajectory dataset. .

3. The safe navigation method for mobile robots in complex crowd scenarios according to any one of claims 1-2, characterized in that, The pedestrian prediction module consists of a two-layer graph attention neural network and a one-layer variational neural network. The graph attention neural network consists of a linear transformation layer. It consists of an attention module and a two-layer graph attention neural network that can capture the observable state of each pedestrian. The data is aggregated to obtain the integrated status of each pedestrian. ; The variational neural network consists of a prior network. An encoder network A decoder network A feature extraction network for a pedestrian's integrated state A latent variable extraction network With a tool for remembering historical information The GRU network structure; in which, the prior network Encoder network Decoder network Pedestrian Integrated State Feature Extraction Network Latent variable extraction network All are composed of fully connected layers.

4. The safe navigation method for mobile robots in complex crowd scenarios according to claim 3, characterized in that, In step 2, the pedestrian prediction module is trained using the collision-free pedestrian trajectory dataset obtained in step 1, as follows: Step 211, take the collision-free pedestrian trajectory dataset obtained in Step 1. Randomly divided into training sets and test set ; Step 212, from the training set A pedestrian trajectory combination is randomly sampled from the data. The pedestrian location information at any given time is represented as the pedestrian observable state ( Based on the pedestrian prediction module The summary of pedestrian observable states at and before time points is used to predict future pedestrian states, thereby estimating the loss function required for training. The steps are as follows: Step 2121, let the current time be... What will be observed in a 2D simulation environment The position of each pedestrian is taken as an observable state. The observable states of a pedestrian are denoted as ( ; Step 2122: The first layer of the graph attention neural network in the pedestrian prediction module passes through a common linear variation layer. Will The observable state of an individual pedestrian ( Extracted as feature vector ; Step 2123, using the extracted feature vectors The following vector connection formula is used to calculate the pair of pedestrians i. Attention weights : ； in, For attention function; Step 2124, take the pedestrian data obtained in step 2123... For pedestrians After the attention weights are normalized using the following formula (1), the first layer graph attention neural network performs a weighted average of all pedestrian feature vectors using the following formula (2), and further utilizes a nonlinear function. After performing a nonlinear transformation, the feature vector output by the first layer of the graph attention neural network is obtained. : （1）； （2）； Step 2125: Convert the feature vector output by the first layer graph attention neural network into a single layer. As the input to the second-layer graph attention neural network, the feature vector output by the second-layer graph attention neural network is obtained after undergoing the same processing as in steps 2122 to 2124. ; Step 2126, use the following formula to convert the feature vector The feature vector output by the two-layer graph attention neural network , Summing as a pedestrian Final integration status : ； Step 2127, using the pedestrian output from the second-layer graph attention neural network. Final integration status With recorded historical information As input to the variational neural network, the encoder network of the variational neural network... Reasoning Latent variables at time Then utilize the GRU network Latent variables at time Update current historical information : ； ； ； Step 2128, using the prior network of the variational neural network And updated current historical information The latent variables for the next time step Estimation is performed, and the decoder network is utilized. Based on the estimated latent variables of the next time step Predicting the speed at the next moment : ； ； ； Step 2129: Calculate the loss for this prediction using the following ELBO loss function and add it to the total loss function: ； in, Denotes the KL divergence; log(·) represents the KL divergence. Logarithm to base 0; and These are the prior networks of variational neural networks. and encoder network The prior and posterior distributions of the latent variables are represented. Indicates all in The graph attention neural network at time points and previous times is based on the original pedestrian... All integrated states obtained from the state input; This means all of them are in Pedestrian control at and before time Integrate all latent variables of the state distribution, in the subscript This means it does not contain All of the moments All corresponding integrated states or hidden variables prior to the given time; Repeat step 212 above to obtain the training set. The estimation results of the upper loss function; Step 213: Use gradient descent according to the training set The estimation results of the upper loss function are used to train the pedestrian prediction module, and the test set is used. According to step 212, the entire pedestrian prediction module is currently estimated in the test set. Using the loss function on the test set, repeat steps 212 and 213 until the pedestrian prediction module performs well on the test set. The training of the pedestrian prediction module is complete when the loss function on the graph converges.

5. The safe navigation method for mobile robots in complex crowd scenarios according to any one of claims 1-2, characterized in that, The hybrid decision-making module includes a reinforcement learning network and an online planning unit; wherein the reinforcement learning network is a policy network. With value network Neural networks based on the actor-critic framework; policy networks With value network They share the same single-layer graph attention neural network at the bottom layer, which can utilize different linearly varying layers. and For the input mobile robot state Observable status of pedestrians Perform alignment processing to obtain the integrated state of the mobile robot. ; The policy network With value network Finally, each layer is equipped with a fully connected layer, which can adjust according to the integration status of the mobile robot. The corresponding action probability distribution and value function are obtained.

6. The safe navigation method for mobile robots in complex crowd scenarios according to claim 5, characterized in that, In step 2, the policy network of the reinforcement learning network of the hybrid decision module is trained using the collision-free pedestrian trajectory dataset obtained in step 1, employing the PPO algorithm, in the following manner: and the corresponding value network ,include: Step 221: Use the simulation environment constructed in Step 1 as the training environment for reinforcement learning, and train the mobile robot in the training environment. The state at time point is represented as , Moment State Including the current location of the mobile robot Target location Current velocity vector With current orientation The probability distributions for determining the starting point, ending point, and initial orientation of the mobile robot are as follows: ; Step 222, determine the number of training rounds. The number of times pedestrian trajectories are collected in each round of training is: And the number of times the network parameters are updated. ; Step 223, in each round of training, first, based on the number of data collections... Acquisition Strategy Network The current strategy's execution trajectory dataset in a 2D simulation environment During this process, the mobile robot will interact with pedestrians and collect execution trajectory datasets according to the following steps. Used for training subsequent policy networks; Step 2231, from the collision-free pedestrian trajectory dataset obtained in Step 1 In this simulation, a random, collision-free pedestrian trajectory combination is selected as the pedestrian's action in the data collection. In the simulation environment, each pedestrian will directly follow the sampled corresponding pedestrian trajectory. Simultaneously, the probability distribution of the mobile robot's starting point, ending point, and initial orientation is analyzed. The starting point of this mobile robot's data collection was obtained through sampling. ,end and initial orientation ; Step 2232, in the interaction between the mobile robot and the pedestrian At time 1, the state of the environment is According to the mobile robot status and pedestrian observable status Policy Network Single-layer graph attention neural networks utilize different linear transformation layers and To obtain eigenvectors of equal length ,in The feature vector of the processed mobile robot; Step 2233, using the extracted feature vector The following vector connection formula is used to calculate the mobile robot's interaction with pedestrians. Attention weights : ； in, For attention function; Step 2234, take the pedestrian data obtained in step 2233... After the attention weights are normalized by the following formula (3), the graph attention neural network performs a weighted average of the feature vectors of the mobile robot and all pedestrians by the following formula (4), and further utilizes a nonlinear function. The policy network is obtained after performing a nonlinear transformation. Required integration state : （3）； （4）； Step 2235, Integration status based on mobile robots According to the policy network The fully connected layer is obtained Probability distribution of available actions at time 1 ; Step 2236: The mobile robot follows the probability distribution of available actions in the simulation environment. Perform action sampling and follow the sampling results. Execute; Step 2237: The simulation environment reacts to the actions fed back by the mobile robot. Update and migrate Get a new state of mind all the time According to the following profit function get Benefits of each moment and will State transition at time Add to execution trajectory dataset : ； in, Indicates mobile robots in Time and The decrease in distance to the target point at any given time; if the distance increases, then... It is a negative number; This indicates the maximum moving speed of the mobile robot; This represents the actual time interval between two decision moments; Repeat steps 2231 to 2237 above. The mobile robot was obtained according to the strategy network. The collected execution trajectory dataset ; Step 224, Replication Strategy Network parameters and value network parameters for and According to the execution trajectory dataset For parameters and Update: Step 2241: In each update, the execution trajectory dataset is processed using the GAE algorithm according to the following formula. A set of state transitions Calculation state Use actions Advantage function : ； in, For the execution trajectory dataset The length of the trajectory in the middle; Discount factor; Parameters set by humans; The value function is calculated by the value network based on the input state through step 2233 and its subsequent fully connected layers; Step 2242, according to the following formula based on the dominance function right The loss function is used for calculation: ； Step 2243, according to the strategy gradient formula of the PPO algorithm, based on the advantage function... right The loss function is used for calculation: ； in, For hyperparameters; The function will Extracting the interval among; Step 2244: Using the gradient descent algorithm and the loss functions from steps 2242 and 2243, respectively... and Update; Repeat steps 2241 to 2244 above. This is the first time it has received a final update. and and will and The values ​​are assigned to respectively and To update the policy network and value network; Step 225: After the update is complete, clear the execution trajectory dataset. ; Repeat steps 223 to 225. Training ends after each round.

7. The safe navigation method for mobile robots in complex crowd scenarios according to any one of claims 1-2, characterized in that, In step 3, the mobile robot inputs the environmental state received at each decision moment into the reinforcement learning network of the trained hybrid decision module to obtain a probability distribution of possible actions in the current state. Then, the online planning unit of the hybrid decision module, based on the probability distribution of possible actions in the current state, combined with the future state of the pedestrian predicted by the pedestrian prediction module and the policy network and value network of the reinforcement learning network, determines the optimal action to be performed by the mobile robot, including: Step 31, at each decision moment The mobile robot obtains its status through sensors. and pedestrian observable status ,in This is the actual number of pedestrians; Step 32, obtain the mobile robot state Observable status of pedestrians As the current state The policy network is input into the training hybrid decision-making module of the reinforcement learning network. In the context of the reinforcement learning network, the policy network... Based on the current state, the probability distribution of possible actions is obtained. ; Step 33, the online planning unit of the hybrid decision-making module starts from the probability distribution of possible actions. Medium sampling Optional actions ; Step 34, based on the input pedestrian observable status Using a trained pedestrian prediction module to predict pedestrians Speed ​​of time Make predictions and obtain the pedestrian observable state at the next moment based on the prediction results. Repeat the above process to obtain the future. The pedestrian's observable state at each decision-making moment; Step 35, for each optional action use The preliminary simulation was estimated in the first round. In the rehearsal, first based on the movements and the result obtained in step 34 The pedestrian's observable state at time step one is estimated, and the state at the next time step is calculated. and benefits Then, the policy network of the reinforcement learning network is used to obtain... Probability distribution of possible actions at time points Randomly sample an action based on the probability distribution of the possible actions. As The expected action to be performed at the specified time, and continues according to the steps obtained in step 34. Pedestrian observable state estimation calculation at time 1 state of time With benefits Repeat this process to obtain a forward rehearsal. Payoff sequence at each decision moment With state And based on them, the score for this round of rehearsals is calculated using the following formula: ； exist After a round of rehearsals, the scores are summed to obtain the result for each movement. Overall score The summation formula is: ； Step 36: Select the action with the highest score from all available actions as the optimal action output by the hybrid decision module. : 。 8. A processing apparatus, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can perform the method according to any one of claims 1-7.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1-7.

Images