Multi-robot path planning algorithm based on priority game and A3C

Abstract

The invention discloses a multi-robot path planning algorithm based on priority game and A3C. According to the algorithm, the states and priorities of all robots are initialized firstly, and then path planning is carried out according to the priorities. In the planning process, the robot collects experience by interacting with the environment, and learning and optimization are carried out by using an A3C model. The algorithm employs a specific reward function to assess the performance of the robot to guide its learning behavior. Through continuous iteration and optimization, the robot gradually learns an optimal path planning strategy. The method has the advantage of being high in practicability and can be widely applied to scenes needing cooperative work of multiple robots, such as the fields of warehouse logistics, intelligent manufacturing and the like. Compared with a traditional path planning method, the method has the advantages that the efficiency and quality of path planning are improved, the collision rate is reduced, and better adaptability and robustness are achieved.

Description

Technical Field

[0001] The present invention relates to the fields of robotics, artificial intelligence and reinforcement learning, and in particular to a multi-robot path planning algorithm based on priority game and A3C. Background Art

[0002] With the continuous development of robotics technology, multi-robot systems have been widely used in many fields, such as warehousing logistics, intelligent manufacturing, environmental detection, etc. In these application scenarios, multiple robots usually need to work together to complete complex tasks. However, how to effectively plan the paths of multiple robots and avoid collisions and conflicts between them is a challenge facing current multi-robot systems.

[0003] Traditional path planning methods are usually based on graph search, potential field method, etc. These methods can achieve good results in simple scenarios. However, when the number of robots increases and the complexity of the environment increases, traditional methods are often unable to cope with it. In addition, these methods usually lack consideration of the interaction between robots and cannot handle the coordination problem between multiple robots well.

[0004] In recent years, with the development of artificial intelligence and reinforcement learning technology, more and more researchers have begun to explore the use of these technologies to solve the multi-robot path planning problem. Reinforcement learning allows robots to continuously try, learn and optimize in the process of interacting with the environment to find the optimal behavior strategy. As an important method in the field of reinforcement learning, the A3C algorithm has the advantages of good parallelism and high learning efficiency, and is very suitable for solving the multi-robot path planning problem.

[0005] However, the current multi-robot path planning method based on reinforcement learning still faces some challenges. For example, how to effectively deal with the priority issues between robots to avoid collisions and conflicts during path planning; how to design a suitable reward function to guide the robot's learning behavior; how to balance the relationship between exploration and utilization to improve learning efficiency, etc. Summary of the invention

[0006] In view of the limitations and challenges of traditional path planning methods when dealing with multi-robot systems, the present invention provides a multi-robot path planning algorithm based on priority game and A3C, aiming to solve the problems existing in the prior art and improve the efficiency and safety of path planning for multi-robot systems.

[0007] The present invention constructs a priority game model to assign different priorities to multiple robots, thereby effectively avoiding collisions and conflicts in path planning. At the same time, combined with the A3C reinforcement learning algorithm, the robot can autonomously learn and optimize the path planning strategy to adapt to complex and changing environments and task requirements. The present invention aims to improve the path planning efficiency and safety of multi-robot systems and provide a more intelligent and efficient solution for robot collaboration.

[0008] The present invention is achieved through the following technical solutions:

[0009] A multi-robot path planning algorithm based on priority game and A3C includes the following steps:

[0010] 1. Initialization

[0011] At the beginning of the algorithm startup, a series of initialization operations are necessary to ensure the smooth execution of the multi-robot path planning process. The specific steps are described as follows:

[0012] 1. Robot status initialization

[0013] Set the initial state of each robot, including the position P i 0 , speed V i 0 ,direction and the target position G i These state parameters will serve as the starting point for path planning and provide benchmark data for subsequent calculations. Mathematically, the robot state initialization can be expressed as:

[0014]

[0015] in, Represents the initial state of robot i.

[0016] 2. Priority setting

[0017] To avoid collisions and conflicts, an initial priority Pr needs to be set for each robot. i 0 . Priority is based on task importance W i 、Current location P i 0 、Target position G i The specific calculation formula is as follows:

[0018] Pr i 0 =σ(W i )+ρ(f(P i 0 ,G i ))

[0019] Among them, σ and ρ are weight functions, f(P i 0 ,G i ) is the evaluation function of the relationship between position and target position. This formula takes into account the importance of the task and the current state of the robot to ensure the rationality and effectiveness of the priority setting.

[0020] 3. A3C model parameter initialization

[0021] As the core algorithm, the A3C model needs to set the parameters of the actor network and the critic network in the initialization stage. The actor network parameters include weights θ a and bias b a , the critic network parameters include weights θ c and bias b c At the same time, the learning rate η needs to be set. These parameters will directly affect the learning performance of the A3C model, so they need to be carefully designed. Mathematically, the initialization of the A3C model parameters can be expressed as:

[0022]

[0023] in, Represents the initial parameter set of the A3C model.

[0024] Through the above initialization steps, the algorithm has laid a solid foundation for multi-robot path planning. In the subsequent steps, the algorithm will perform path planning based on these initialized states and parameters through priority game and A3C algorithm to achieve collaborative work and efficient path selection among robots.

[0025] 2. Priority Game

[0026] After completing the initialization step, the algorithm immediately enters the priority game stage. This stage aims to ensure that high-priority robots can get priority access to the path by setting a priority mechanism, thereby reducing the risk of collision and conflict.

[0027] In each time step, the algorithm first retrieves and updates the robot’s priority list. This list is based on the robot’s task importance W i 、Current location P i 、Target position G i And other related factors are dynamically adjusted. The priority calculation formula is as follows:

[0028] Priority i =α*W i +β*f(P i ,G i )+γ*g(Environment i)

[0029] Among them, α, β, γ are weight coefficients, f(P i ,G i ) represents the functional relationship between the robot’s current position and the target position, g(Environment i ) represents the evaluation function of the robot’s environment.

[0030] Then, the algorithm plans paths for the robots in order of priority from high to low. For each robot, the algorithm combines its current state S i (including position, speed, direction, etc.) and environmental information E i (such as obstacle positions, other robot states, etc.), and calculate the optimal path from the current position to the target position through a path planning algorithm (such as A* algorithm, RRT algorithm, etc.) i .

[0031] When planning a path, the algorithm will pay special attention to the conflict between paths. If the path of a high-priority robot has a potential conflict with the path of a low-priority robot, the algorithm will initiate an avoidance strategy. This strategy adjusts the cost function C in the path planning algorithm to avoid the conflict. i , so that low-priority robots tend to avoid high-priority robots when choosing paths. The adjustment of the cost function can be expressed as:

[0032] C i′ =C i +λ*Overlap(Path high ,Path i )

[0033] Among them, Overlap(Path high ,Path i ) represents the degree of overlap between the high priority robot path and the current robot path, and λ is the avoidance coefficient.

[0034] Through priority game, the algorithm realizes orderly path planning in multi-robot system. High-priority robots are given priority to use the path, while low-priority robots avoid conflicts by adjusting their own behaviors, thus ensuring the efficiency and safety of the overall path planning.

[0035] It should be noted that during the priority game, the algorithm needs to ensure the real-time and dynamic nature of path planning. As the robot status and environmental information are updated in real time, the priority list and path planning results need to be adjusted synchronously. Therefore, the algorithm needs to have efficient data processing capabilities and flexible decision-making mechanisms to cope with complex and changing environments and task requirements.

[0036] In summary, by introducing the priority game mechanism, the algorithm realizes orderly path planning based on robot priority, effectively reduces the risk of collision and conflict, and provides strong support for robot collaborative work.

[0037] 3. State Input and Action Selection

[0038] After clarifying the path planning order of each robot in the priority game stage, the algorithm advances to the state input and action selection stage. The core of this stage is to input the current state of the robot into the A3C model and select the corresponding action based on the model output, thereby realizing the robot's autonomous path planning.

[0039] Specifically, the algorithm first takes the current robot state information, including the position P i , speed V i , Direction D i 、Target position G i etc., as input data X i The actor network passed to the A3C model. The actor network is parameterized by the function f a (θ a ,X i ) generates the probability distribution of actions, where θ a are the parameters of the actor network.

[0040] The action probability distribution can be expressed as:

[0041] π(a|X i θ a ) = f a (θ a ,X i )

[0042] Here, a represents the actions that the robot can take.

[0043] Then, the algorithm uses the ε-greedy strategy to select the specific action to be performed by the current robot based on the output of the actor network. The ε-greedy strategy combines the ideas of exploration and utilization, selecting random actions for exploration with probability ε, and selecting the action with the highest current probability for utilization with probability 1-ε. Mathematically, the action selection process can be expressed as:

[0044]

[0045] Here, rand() is a function that generates random numbers in the range [0,1].

[0046] It is worth noting that the A3C model uses an asynchronous update mechanism, allowing multiple threads to process different robot states in parallel, thereby improving the efficiency of path planning. In the state input and action selection stage, the algorithm makes full use of this feature of the A3C model to achieve efficient parallel path planning.

[0047] Through the state input and action selection stages, the algorithm realizes the robot's autonomous path planning. Each robot selects and executes the corresponding action based on its own state information and the output of the A3C model, gradually approaching the target position. At the same time, due to the asynchronous learning and optimization characteristics of the A3C model, the robot can optimize its own path planning strategy in continuous trial and learning, and improve the quality and efficiency of path planning.

[0048] In summary, the present invention realizes robot autonomous path planning based on state input and action selection by introducing the A3C model and the ε-greedy strategy, and provides effective decision support for robot collaborative work.

[0049] 4. Environmental Interaction and Experience Collection

[0050] After completing the state input and action selection steps, the algorithm moves to the environment interaction and experience collection stage, which is the core component of the reinforcement learning process. This stage involves the robot's action execution, environment interaction, and experience collection and storage, aiming to optimize the path planning strategy using environmental feedback data.

[0051] First, the robot selects an action a i Perform actions, such as changing the direction of movement or adjusting the speed. During the execution process, the robot continuously receives feedback from the environment, including the new state P i ′ and reward value r i Environmental feedback can be formalized as state transition probability and reward function, namely P(P i ′|P i ,a i ) and R(P i ,a i ).

[0052] Next, the algorithm presents the experience of this interaction as a tuple (P i ,a i ,r i ,P i The experience replay buffer is a fixed-size data set used to store historical experience data. When the buffer is full, the algorithm uses a first-in-first-out (FIFO) or other strategy to replace the old data.

[0053] The management and maintenance of the experience replay buffer is crucial, as it ensures that the algorithm can learn from a variety of experience data. By randomly sampling samples from the buffer for training, the algorithm can break the correlation between data and improve the stability and efficiency of learning.

[0054] The learning part of the algorithm, especially the training of the A3C model, will be based on these collected empirical data. The A3C model continuously updates its parameters by optimizing the objective function, such as maximizing the expected return, to better guide the robot's action selection.

[0055] Mathematically, the expected return can be expressed as:

[0056]

[0057] Among them, θ is the parameter of the A3C model, and γ is the discount factor, which is used to weigh the importance of recent rewards and long-term rewards.

[0058] Through the environmental interaction and experience collection phase, the algorithm provides rich data support for the training of the A3C model and promotes the continuous optimization of the robot's path planning strategy. These experience data not only reflect the interaction history between the robot and the environment, but also contain important information about the quality of the action, providing strong support for the learning and optimization of the algorithm.

[0059] In summary, the environmental interaction and experience collection stage is the key link in realizing the robot's autonomous path planning. By combining the robot's action execution and environmental feedback, this stage provides the necessary data foundation for reinforcement learning and promotes the continuous improvement and enhancement of the robot's path planning strategy.

[0060] 5. Asynchronous Update

[0061] After the robot has accumulated sufficient experience data through environmental interaction, the algorithm enters the asynchronous update phase. This phase makes full use of the asynchronous learning characteristics of the A3C model and significantly improves learning efficiency and algorithm performance through multi-threaded parallel updates.

[0062] First, the algorithm randomly samples a batch of experience data from the experience playback buffer D in parallel on multiple threads, denoted as where k represents the sample index in the sampling batch.

[0063] Next, the algorithm uses the critic network Q(P,a;θ c ) to evaluate the value of this batch of experience. The critic network is based on the current state P i k and the target state P i k+1 Predict the expected reward and provide value guidance for the actor network. The predicted value of the expected reward can be expressed as:

[0064]

[0065] Then, the algorithm uses the actor network π(a|P;θ a ) to update the robot’s strategy. Based on the value guidance provided by the critic network The actor network adjusts its parameters θ by gradient descent and other methods a , to maximize the expected reward. The objective function of the strategy update can be expressed as:

[0066]

[0067] Among them, γ is a discount factor, which is used to weigh the importance of current rewards and future rewards.

[0068] Finally, the algorithm updates the network parameters θ a and θ c Asynchronously synchronize to other threads. Since the A3C model uses asynchronous updates, the learning progress of different threads is independent. Through parameter synchronization, the algorithm ensures that all threads use the latest and optimal strategy for path planning.

[0069] During the asynchronous update process, the algorithm avoids problems such as data competition and deadlock through effective thread synchronization and communication mechanisms, ensuring smooth collaboration between threads.

[0070] Through the asynchronous update phase, the algorithm efficiently utilizes experience data and continuously improves the robot's path planning performance through parallel learning and strategy optimization. The realization of this phase provides strong support for the collaborative work of robots, enabling robots to achieve efficient and accurate path planning in complex and changing environments.

[0071] In summary, the present invention realizes efficient and stable robot path planning learning by introducing the asynchronous update mechanism of the A3C model. Through multi-threaded parallel sampling, evaluation and updating, the algorithm can make full use of empirical data, improve learning efficiency and algorithm performance, and provide effective decision support for robot collaborative work.

[0072] 6. Reward Function Calculation

[0073] In the process of robot path planning, the reward function is a guide for learning behavior, and its design is crucial. The reward function in the algorithm of the present invention comprehensively considers multiple factors such as reaching the target, avoiding collisions, and reaching the target as quickly as possible. The calculation method of the reward function is described in detail below.

[0074] First, define the reward function R to reach the goal goal, when the robot successfully reaches the target position, a positive reward is given. The reward value can be adjusted according to the complexity and importance of the task, for example:

[0075] R goal =C goal

[0076] Among them, C goal is a fixed positive constant.

[0077] Next, define the reward function R for avoiding collisions collision In order to encourage the robot to avoid collisions during path planning, a certain reward value is given when the robot successfully avoids a collision. The reward value can be set according to the degree of collision risk, for example, in the form of a function inversely proportional to the collision distance:

[0078]

[0079] Among them, C collision is the constant factor of the collision reward, d collision is the current distance between the robot and the obstacle or other robots. When the distance is large, the reward value is high; when the distance is small, the reward value is low or even negative.

[0080] In addition, in order to encourage the robot to reach the target position as quickly as possible, the reward function R for reaching the target as quickly as possible is defined time The reward function can be calculated based on the time required for the robot to reach the target position, for example, in the form of a function that is inversely proportional to the arrival time:

[0081]

[0082] Among them, C time is the constant factor of the time reward, t arrival is the time required for the robot to reach the target position. The shorter the arrival time, the higher the reward value.

[0083] Combining the above three aspects of reward functions, we get the total reward function R total :

[0084] R total =R goal +R collision +R time

[0085] In practical applications, the weights of these three reward functions can be adjusted according to specific task requirements to balance the impact of different factors. The weight adjustment can be achieved by adjusting the constant factor C goal , C collision and C time to achieve.

[0086] By rationally designing the reward function, the algorithm of the present invention can guide the robot to comprehensively consider multiple factors during the path planning process, and achieve efficient and safe path selection. The calculation of the reward value will be directly used to guide the robot's learning behavior, and through continuous iteration and optimization, the robot can gradually learn better path planning strategies. This reward mechanism ensures that the robot can make reasonable decisions when facing complex environments, thereby improving the accuracy and efficiency of path planning.

[0087] 7. Iterative Optimization

[0088] In the first six steps of the algorithm, the robot has initially learned a certain path planning strategy through interaction with the environment, experience collection, asynchronous updates, and calculation of reward functions. However, in order to obtain the optimal path planning strategy, the robot needs to further conduct trial and error and learn through an iterative optimization process.

[0089] In the iterative optimization phase, the algorithm repeats steps 3 to 6 to form a closed-loop learning process. Specifically, the robot selects and executes action a based on the current state information s, then observes the environmental feedback and collects new experience data. This experience data is then used in the asynchronous update process to adjust the robot's policy parameters through the collaborative work of the critic network and the actor network. At the same time, based on the robot's performance in executing the action, the corresponding reward value r is calculated. t , used to guide subsequent learning behaviors.

[0090] As the iteration process proceeds, the robot's strategy π(a|s) is gradually optimized. The strategy optimization process can be expressed as finding a strategy π* that can maximize the expected reward. The mathematical expression of the expected reward is:

[0091]

[0092] Among them, γ is a discount factor, which is used to weigh the importance of current rewards and future rewards. Through iterative optimization, the algorithm continuously adjusts the strategy π to gradually approach the optimal strategy π*.

[0093] It is worth noting that the iterative optimization process may be affected by multiple factors such as environmental uncertainty and initial state, and it takes a certain amount of time to reach convergence. In order to accelerate the learning process, it may be necessary to combine heuristic methods or prior knowledge. At the same time, in order to avoid falling into a local optimal solution, the algorithm also needs to strike a balance between exploration and exploitation.

[0094] Through iterative optimization, the robot can continuously try and learn, and adjust its strategy based on empirical data, and eventually gradually learn the optimal path planning strategy. This process reflects the core part of the algorithm and is also the key to achieving autonomous path planning for robots.

[0095] In summary, through the iterative optimization process, the robot can continuously learn and optimize its path planning strategy, thereby achieving efficient and safe path planning. This method not only improves the robot's autonomous decision-making ability, but also provides strong support for the robot's collaborative work.

[0096] 8. Testing and Evaluation

[0097] After completing the iterative optimization process, the algorithm enters the testing and evaluation phase to fully verify the robot's path planning performance. This phase aims to ensure that the robot can demonstrate efficient and safe path planning capabilities in actual scenarios.

[0098] First, we construct a test dataset containing various scenarios, denoted as D = (s i ,g i ,O i ), where s i represents the starting state, g i represents the target state, O i Represents the obstacle distribution. The dataset should cover a wide range of situations to fully test the robot’s path planning ability.

[0099] In the test environment, the trained model is applied to the robot and key performance indicators are recorded. The main indicators include:

[0100] Path length L: The actual path length that the robot takes from the starting point to the target point. A shorter path length means higher efficiency.

[0101] Time consumption T: The total time required for the robot to complete path planning and reach the target point. A shorter time consumption means that the robot can complete the task faster.

[0102] Collision count C: The number of times the robot collides with obstacles or other robots during path planning. A lower collision count means higher safety.

[0103] During the test, the robot needs to plan its path autonomously without additional training. By performing multiple tests and calculating the average values ​​of the above indicators, the robot's path planning performance can be quantitatively evaluated.

[0104] In addition, in order to gain a deeper understanding of the model performance, comparative experiments under different algorithms or parameter settings are conducted. By adjusting parameters such as model structure, learning rate, and reward function, the impact of these changes on performance indicators can be observed. Specifically, different parameter combinations can be set and experiments can be conducted under the same test data set to compare indicators such as path length, time consumption, and number of collisions under each combination.

[0105] Finally, based on the results of testing and evaluation, the best performing model is selected for practical application. The selected model should have stable performance in a variety of scenarios and good generalization ability to cope with various changes that may occur in the actual environment.

[0106] In summary, through the testing and evaluation phase, the robot's path planning performance can be fully verified, and the optimal model can be selected for practical application. This process not only improves the robot's path planning ability, but also provides strong support for the wide application of robots in practical scenarios.

[0107] In summary, the algorithm of the present invention realizes the autonomous path planning of the robot through the steps of state input and action selection, environment interaction and experience collection, asynchronous update, reward function calculation and iterative optimization. In the testing and evaluation stage, we comprehensively evaluated the performance of the robot and selected the optimal model for practical application. This algorithm is efficient, accurate and stable, suitable for robot path planning tasks in a variety of complex scenarios, and provides strong support for robot collaborative work.

[0108] Compared with the prior art, the present invention has the following advantages and effects:

[0109] The algorithm of the present invention realizes the autonomous path planning of the robot through the steps of state input and action selection, environment interaction and experience collection, asynchronous update, reward function calculation and iterative optimization. In the testing and evaluation stage, the present invention comprehensively evaluates the performance of the robot and selects the optimal model for practical application. This algorithm is efficient, accurate and stable, and is suitable for robot path planning tasks in a variety of complex scenarios, providing strong support for robot collaborative work. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 The figure is a flow chart of a multi-robot path planning method based on priority game and A3C algorithm of the present invention. DETAILED DESCRIPTION

[0111] The present invention is further described in detail below in conjunction with specific embodiments.

[0112] The specific implementation of the multi-robot path planning method based on priority game and A3C algorithm of the present invention will be described in detail below.

[0113] 1. Initialization settings

[0114] 1. Robot status initialization

[0115] For each robot, we need to initialize its state, including its initial position x, y coordinates in the environment, initial velocity v x ,vy , initial direction θ, and target position x goal ,y goal . This state information will serve as the basis for path planning.

[0116] Mathematically, it can be expressed as: robot state Where i represents the number of the robot.

[0117] 2. Priority setting

[0118] According to the importance and urgency of the task, each robot is assigned an initial priority. The priority can be calculated based on factors such as the task type, the distance between the robot's current position and the target position, and the density of environmental obstacles.

[0119] Specifically, the priority can be calculated using the following formula: i =w1·d i +w2·o i +w3·t i , where P i is the priority of robot i, d i is the distance from robot i to the target position, o i is the impact of environmental obstacles on robot i, t i is the importance or urgency of the task, and w1, w2, w3 are the corresponding weight coefficients.

[0120] 3. A3C model initialization

[0121] Initialize the parameters of the A3C model, including the weights and biases of the actor network and the critic network. These parameters are the basis of model learning, and the optimal solution is approached through continuous trial and error and optimization.

[0122] 2. Priority Game and Path Planning

[0123] At each time step, the algorithm performs the following steps:

[0124] 1. Update the priority list

[0125] According to the current robot status and environmental information, the priority of each robot is recalculated and updated. This ensures that the priority list can reflect changes in the environment and changes in the robot status in real time.

[0126] 2. Path planning

[0127] According to the updated priority list, the paths are planned for the robots from high to low. For each robot, the optimal path from the current position to the target position is calculated using a path planning algorithm (such as the A* algorithm or the RRT algorithm) based on its current state (position, speed, direction) and environmental information (obstacle position, other robot states).

[0128] When planning a path, the algorithm will consider the positions and priorities of other robots to avoid path conflicts and collisions. For robots with higher priorities, the algorithm will give them more path selection rights to ensure that they can complete the task first.

[0129] 3. A3C model learning and optimization

[0130] During the path planning process, the A3C model continuously interacts with the environment and learns how to make the best decision based on the current state and environmental information. The model optimizes its behavior strategy through trial and error and reward mechanisms, allowing the robot to better adapt to complex and changing environments and task requirements.

[0131] Specifically, the model calculates the loss function based on the difference between the robot's actual behavior and the target behavior, and updates the network parameters through the back-propagation algorithm. At the same time, the model also gives corresponding rewards or penalties based on the robot's performance to guide its learning behavior.

[0132] As described above, the present invention can be better implemented.

[0133] The implementation methods of the present invention are not limited to the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A multi-robot path planning algorithm based on priority game and A3C, characterized by The steps include: S1: Initialization; S2: Priority game; S3: state input and action selection; S4: Environmental interaction and experience collection; S5: asynchronous update; S6: reward function calculation; S7: Iterative optimization; S8: Testing and evaluation.

2. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that The initialization in step S1 specifically includes the following sub-steps: S1-1, robot status initialization Set the initial state parameters of each robot, including position speed direction and the target position G i ; This state parameter will serve as the starting point for path planning and provide benchmark data for subsequent calculations; mathematically, the robot state initialization can be expressed as: in, represents the initial state of robot i; S1-2, priority setting To avoid collisions and conflicts, an initial priority is set for each robot. Prioritization is based on task importance. i , Current Location Target position G i The specific calculation formula is as follows: Among them, σ and ρ are weight functions, is the evaluation function of the relationship between the position and the target position; S1-3, A3C model parameter initialization As the core algorithm, the A3C model needs to set the parameters of the actor network and the critic network in the initialization stage; the actor network parameters include weights θ a and bias b a , the critic network parameters include weights θ c and bias b c ; At the same time, the learning rate η needs to be set; mathematically, the A3C model parameter initialization can be expressed as: in, Represents the initial parameter set of the A3C model.

3. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that The priority game in step S2 includes the following sub-steps: After completing the initialization step, the algorithm immediately enters the priority game phase; In this stage, a priority mechanism is set to ensure that robots with high priority can obtain the right to use the path first, thereby reducing the risk of collision and conflict; At each time step, the algorithm first retrieves and updates the robot’s priority list; this list is based on the robot’s task importance W i 、Current location P i 、Target position G i Dynamic adjustment is performed; the priority calculation formula is as follows: Priority i =α*W i +β*f(P i ,G i )+γ*g(Environment i ); Among them, α, β, γ are weight coefficients, f(P i ,G i ) represents the functional relationship between the robot’s current position and the target position, g(Environment i ) represents the evaluation function of the robot’s environment; Then, the algorithm plans paths for the robots in order of priority from high to low. For each robot, the algorithm combines its current state S i And environmental information E i , calculate the optimal path from the current location to the target location through the path planning algorithm i ; When planning a path, the algorithm will pay special attention to the conflict between paths; if the path of a high-priority robot has a potential conflict with the path of a low-priority robot, the algorithm will initiate an avoidance strategy; this strategy adjusts the cost function C in the path planning algorithm to avoid the conflict. i , so that low-priority robots tend to avoid high-priority robots when choosing paths; the adjustment of the cost function is expressed as: C i′ =C i +λ*Overlap(Path high ,Path i ); Among them, Overlap(Path high ,Path i ) represents the degree of overlap between the high priority robot path and the current robot path, and λ is the avoidance coefficient.

4. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that Step S3: state input and action selection includes the following sub-steps: After clarifying the path planning order of each robot in the priority game stage, the algorithm advances to the state input and action selection stage; this stage is to input the current state of the robot into the A3C model and select the corresponding action based on the model output, thereby realizing the robot's autonomous path planning, as follows: The algorithm first takes the current robot status information, including the position P i , speed V i , Direction D i 、Target position G i , as input data X i The actor network passed to the A3C model; the actor network is parameterized by the function f a (θ a ,X i ) generates the probability distribution of actions, where θ a are the parameters of the actor network; The action probability distribution can be expressed as: π(a|X i ;θ a )=f a (i a ,X i ); Among them, a represents the actions that the robot can take; Subsequently, the algorithm uses the ε-greedy strategy to select the specific action to be performed by the current robot based on the output of the actor network. The ε-greedy strategy combines the ideas of exploration and utilization, selecting random actions for exploration with probability ε, and selecting the action with the highest current probability for utilization with probability 1-ε. Mathematically, the action selection process can be expressed as: Here, rand() is a function that generates random numbers in the range [0,1].

5. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that Step S4: Environmental interaction and experience collection includes the following sub-steps: After completing the state input and action selection steps, the algorithm moves to the environment interaction and experience collection stage, which is the core component of the reinforcement learning process; this stage involves the robot's action execution, environment interaction, and experience collection and storage, aiming to optimize the path planning strategy using environmental feedback data; the details are as follows: First, the robot selects an action a i Perform actions, such as changing the direction of movement or adjusting the speed; during the execution process, the robot continuously receives environmental feedback, including the new state P i ′ and reward value r i ; Environmental feedback can be formalized as state transition probability and reward function, namely P(P i ′|P i ,a i ) and R(P i ,a i ); Next, the algorithm presents the experience of this interaction as a tuple (P i ,a i ,r i ,P i ′) in the form of experience replay buffer D; the experience replay buffer is a fixed-size data set used to store historical experience data; when the buffer is full, the algorithm uses first-in-first-out or other strategies to replace old data; The A3C model continuously updates its parameters by optimizing the objective function, such as maximizing the expected return, to better guide the robot's action selection; Mathematically, the expected return can be expressed as: Among them, θ is the parameter of the A3C model, and γ is the discount factor, which is used to weigh the importance of recent rewards and long-term rewards.

6. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that S5 asynchronous update includes the following sub-steps: After the robot has accumulated sufficient experience data through environmental interaction, the algorithm enters the asynchronous update phase. This phase uses the asynchronous learning characteristics of the A3C model to update in parallel through multiple threads to improve learning efficiency and algorithm performance. The details are as follows: First, the algorithm randomly samples a batch of experience data from the experience playback buffer D in parallel on multiple threads, denoted as Where k represents the sample index in the sampling batch; Next, the algorithm uses the critic network Q(P,a;θ c ) to evaluate the value of this batch of experience; the critic network evaluates the value of this batch of experience according to the current state and target state Predict the expected reward and provide value guidance for the actor network; the predicted value of the expected reward can be expressed as: The algorithm then uses the actor network π(a|P;θ a ) to update the robot’s strategy; based on the value guidance provided by the critic network The actor network adjusts its parameters θ by gradient descent and other methods a , to maximize the expected reward; the objective function of the strategy update can be expressed as: Among them, γ is a discount factor, which is used to weigh the importance of current rewards and future rewards; Finally, the algorithm updates the network parameters θ a and θ c Synchronize to other threads asynchronously.

7. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that S6 reward function calculation includes the following sub-steps: In the process of robot path planning, the reward function serves as a guide for learning behavior, as follows: First, define the reward function R to reach the goal goal , when the robot successfully reaches the target position, a positive reward is given; the reward value can be adjusted according to the complexity and importance of the task, if: R goal =C goal ; Among them, C goal is a fixed positive constant; Next, define the reward function R for avoiding collisions collision ; In order to encourage the robot to avoid collisions during path planning, a certain reward value is given when the robot successfully avoids a collision; the reward value can be set according to the degree of collision risk and can be in the form of a function inversely proportional to the collision distance: Among them, C collision is a constant factor of the collision reward, d collision is the current distance between the robot and the obstacle or other robots; when the distance is large, the reward value is high; when the distance is small, the reward value is low or even negative; In addition, in order to encourage the robot to reach the target position as quickly as possible, the reward function R for reaching the target as quickly as possible is defined time ; The reward function can be calculated based on the time required for the robot to reach the target position, assuming that it takes a function inversely proportional to the arrival time: Among them, C time is the constant factor of the time reward, t arrival is the time required for the robot to reach the target position; the shorter the arrival time, the higher the reward value; Combining the above three aspects of reward functions, we get the total reward function R total : R total =R goal +R collision +R time 。 8. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that S7 iterative optimization includes the following sub-steps: In order to obtain the optimal path planning strategy, the robot needs to further conduct trial and error and learn through an iterative optimization process. In the iterative optimization stage, the algorithm repeatedly executes steps S3 to S6 to form a closed-loop learning process. The details are as follows: The robot selects and executes action a according to the current state information s, then observes the environmental feedback and collects new experience data; these experience data are then used in the asynchronous update process to adjust the robot's policy parameters through the collaborative work of the critic network and the actor network; at the same time, according to the robot's performance in the process of executing the action, the corresponding reward value r is calculated t , used to guide subsequent learning behaviors; As the iteration process proceeds, the robot's strategy π(a|s) is gradually optimized; the strategy optimization process can be expressed as finding a strategy π* that can maximize the expected reward; the mathematical expression of the expected reward is: Among them, γ is the discount factor, which is used to weigh the importance of current rewards and future rewards. Through iterative optimization, the algorithm continuously adjusts the strategy π so that it gradually approaches the optimal strategy π*.

9. The multi-robot path planning algorithm based on priority game and A3C according to claim 1 is characterized in that S8 testing and evaluation specifically includes the following sub-steps: After completing the iterative optimization process, the algorithm enters the testing and evaluation phase to fully verify the robot's path planning performance; this phase is to ensure that the robot can demonstrate efficient and safe path planning capabilities in actual scenarios; the details are as follows: First, we construct a test dataset containing various scenarios, denoted as D = (s i ,g i ,O i ), where s i represents the starting state, g i represents the target state, O i Indicates the obstacle distribution; The dataset should cover a wide range of situations to adequately test the robot’s path planning capabilities; In a test environment, the trained model is applied to the robot and key performance indicators are recorded.

10. The multi-robot path planning algorithm based on priority game and A3C according to claim 9 is characterized in that The key performance indicators include: Path length L: the actual path length that the robot travels from the starting point to the target point; Time consumption T: the total time required for the robot to complete path planning and reach the target point; Number of collisions C: the number of times the robot collides with obstacles or other robots during path planning; During the test, the robot needs to plan its path autonomously without additional training.

Images