Multi-agent path planning method based on hierarchical reinforcement learning and goal guidance

Abstract

This application belongs to the field of artificial intelligence, specifically relating to a multi-agent path planning method based on hierarchical reinforcement learning and goal-guided approach, aiming to solve the problems of sparse rewards and low collaborative efficiency among multi-agents in long-range path planning. The method includes: constructing a hierarchical decision-making framework; constructing a low-level control network; establishing an intrinsic reward function based on a curiosity mechanism; and performing asynchronous reinforcement learning training. This application, through its hierarchical architecture and curiosity mechanism, effectively solves the exploration problem under sparse rewards, improves decision-making efficiency and collaborative convergence speed in complex environments, and ensures the safety and physical feasibility of the trajectory.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and robot autonomous decision-making, and specifically relates to a path planning method for multi-agent systems. This method is particularly suitable for long-range planning with sparse reward problems and multi-agent cooperative scenarios in complex dynamic environments. Background Technology

[0002] In recent years, artificial intelligence technologies, especially deep learning and reinforcement learning, have been profoundly empowering robotics. Multi-agent collaborative systems have demonstrated enormous application potential in scenarios such as modern warehousing and logistics, drone swarm collaborative operations, and flexible automated production lines. A core challenge in enabling the autonomous operation of such systems is how to plan collision-free, highly efficient motion trajectories for multiple agents in real time under conditions of incomplete environmental information, dynamic obstacles, and limited space. This is not only related to the efficiency of group operations but also a fundamental guarantee for the safe and stable operation of the system.

[0003] Among numerous technical approaches, deep reinforcement learning-based planning methods have become an important approach for solving path planning problems in high-dimensional state spaces due to their powerful environmental representation capabilities and end-to-end decision-making characteristics. This method directly learns the mapping strategy from environmental perception to action output through neural networks and uses reward signals to guide strategy optimization, aiming to enable the agent to autonomously seek optimization in unfamiliar and changing environments. When the task scenario expands and the number of agents increases, researchers often adopt a hierarchical decision-making approach: decomposing the complex global navigation task, with the upper layer responsible for macro-level waypoint planning and the lower layer responsible for local fine-grained obstacle avoidance, combined with a goal guidance mechanism to maintain the agent's tendency towards the final goal, thus addressing the challenges of long-distance planning.

[0004] However, existing methods still face significant bottlenecks in practical applications. In vast scenarios, due to the large distance between the starting and ending points, agents are almost unable to reach the endpoint through random exploration in the early stages of training. This results in extremely sparse effective reward signals for policy updates, leading to slow training convergence and agents easily getting stuck in ineffective wandering in the initial region. Furthermore, single-layer policy networks struggle to simultaneously achieve both "seeing far" (global optimum) and "walking steadily" (local obstacle avoidance), easily leading to decision deadlock in areas with dense obstacles or narrow passages. For multi-agent systems, existing collaborative frameworks often fail to decompose tasks finely enough and lack the intrinsic motivation to drive agents to actively explore when external rewards are scarce. This causes the system to easily get lost in local details in long-term tasks, deviating from the globally optimal path and failing to meet the real-time, robust planning requirements of highly dynamic, large-scale scenarios.

[0005] Therefore, the industry urgently needs a new method for long-range path planning that can effectively address sparse rewards and improve the efficiency of multi-agent collaboration. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multi-agent path planning method based on hierarchical reinforcement learning and goal guidance. This method aims to systematically solve the reward sparsity problem in long-range planning through hierarchical task decomposition, intrinsic curiosity-driven approaches, and asynchronous collaborative training, and significantly improve the decision-making efficiency and collaborative convergence speed of multi-agent systems in complex environments.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] Step S1: Construct a hierarchical decision framework: Digitally model the physical environment in which the multi-agent operates, rasterize the environment map to generate a global cost map, where each grid point carries the access cost information for that location, and perform an expansion operation based on the agent's physical size on obstacle edges; Utilize a high-level decision network to receive the global cost map as input, extracting spatial topological features of the environment through a predetermined number of convolutional operations, employing multiple sets of convolutional kernels with different receptive fields to capture multi-scale environmental obstacle distribution information; The high-level decision network periodically performs forward inference, mapping the extracted global feature vectors to a probability distribution map proportional to the size of the environment map, and then randomly sampling after processing with a normalized exponential function, outputting the intermediate path point sequence for each agent, and introducing a distance constraint operator during the sampling process to limit the Euclidean distance between the sub-target coordinates and the agent's current position;

[0009] Step S2, constructing the underlying control network: The underlying control network serves as a micro-control module directly facing the actuator. Its input feature vector consists of the relative vector between the agent's current position and the intermediate track point, the agent's own velocity vector, and local environmental observation data. The local environmental observation data is obtained by downsampling the raw data generated by the sensor and converting it into a predetermined number of ray features. Each ray represents the distance to the nearest obstacle within a specific angle range. The underlying control network adopts a multilayer perceptron structure. Its output layer uses an activation function to map the raw output of the neural network to a continuous action space, and converts it into actual linear velocity and angular velocity commands through a scaling transformation module. Before the commands are issued, an action smoothing filter limits the acceleration changes between adjacent time steps.

[0010] Step S3: Establish an intrinsic reward function based on the curiosity mechanism: Construct a curiosity module consisting of a forward dynamics model and a backward dynamics model. The backward dynamics model takes into account the environmental states at two consecutive time points and outputs a predicted action. It learns the feature representation of environmental state changes and extracts core features by minimizing the cross-entropy loss between the predicted action and the actual action. The forward dynamics model operates in the feature space formed by the core features, takes into account the feature vector at the current time point and the action taken, and outputs a predicted value for the feature vector at the next time point. The prediction error between the predicted value and the true feature vector is calculated, and the square of the prediction error is multiplied by a preset scaling factor and fed back to the reinforcement learning algorithm as an intrinsic reward signal to drive the agent to actively explore unknown areas with large prediction errors.

[0011] Step S4, perform asynchronous reinforcement learning training: An asynchronous reinforcement learning framework based on an actor-critic architecture is adopted. The high-level decision network and the low-level control network each have independent policy networks and value networks, and are updated asynchronously through a shared experience replay pool. The high-level reward function is composed of a weighted sum of task achievement reward, intrinsic reward, and time penalty, while the low-level reward function is composed of sub-goal tendency reward and collision penalty. Using proximal policy optimization logic, policy updates are performed by pruning probability ratios, and the parameter update frequency of the high-level decision network is set to a predetermined proportion of the update frequency of the low-level control network. Iterative optimization of multi-agent cooperative policies is achieved by exchanging network parameters through shared memory among processes.

[0012] Preferably, the process of constructing the hierarchical decision framework in step S1 includes: rasterizing the complex environment map to generate a global cost map, wherein the size of the global cost map is set to... Multiply Define the global state space as This contains the location coordinates of all agents. and the distribution characteristics of obstacles Define the high-level target space as Each element in the high-level target space represents the coordinates of a sub-target point. The high-level decision-making network executes a decision every preset number of time steps, wherein the preset number of time steps are within a preset time step interval; the high-level decision-making network adopts a convolutional neural network structure, and the input is the global cost map. Spatial features are extracted through a predetermined number of convolutional layers, each with a preset kernel size and stride. The output of the convolutional layers is processed by fully connected layers, and a probability distribution is output using a normalized exponential function. During the execution of high-level strategies, based on probability distribution Sub-target coordinate vectors are obtained by sampling on the global map. The sub-target coordinate vector Including x-axis with the vertical axis .

[0013] Preferably, the process of constructing the underlying control network in step 2 includes: defining the underlying local observation space as... Its input features include lidar scan data within the predetermined detection radius of the intelligent agent. Current velocity vector and the relative position vector with respect to the current sub-target point The relative position vector The calculation logic is for the sub-target coordinates With current position coordinates The difference; the underlying control network adopts a multilayer perceptron structure, which contains a predetermined number of hidden layers, and the number of neurons in each hidden layer is set to a preset scale; the output layer of the underlying control network uses a hyperbolic tangent activation function to map the output value to a continuous action space within a preset range. The final generated action instructions Including linear velocity commands With angular velocity command It is used to drive intelligent agents to move in physical or simulated environments.

[0014] Preferably, the process of establishing the intrinsic reward function in step 3 involves constructing a forward dynamics model. and an inverse dynamics model The inverse dynamics model Enter the current status and the state at the next moment Predict the actions taken by the intelligent agent. Define the inverse loss function. For predicting actions With real action Cross-entropy between; the forward dynamics model Feature representation of the current input state Acting on The result and action Feature representation for predicting the state at the next time step Acting on The result after prediction; define the prediction error. for Representation of true features The square of the Euclidean distance between them; defining the intrinsic reward. Scaling factor With the prediction error The product of, where Within the preset coefficient range; intrinsic reward It can encourage agents to prefer visiting areas that the prediction model is unfamiliar with, thereby forming effective movement trajectories during the exploration phase without external rewards.

[0015] Preferably, the asynchronous reinforcement learning training process in step 4 employs the actor critic algorithm framework; a high-level reward function is defined. Rewards for reaching the goal Collision penalty and intrinsic rewards The weighted sum; define the underlying reward function. Sub-goals tend towards reward With real-time obstacle avoidance penalties The sum; during high-level training, the state value function is used. Estimate the expected return of the current state and calculate the advantage function. The advantage function The value is determined by the difference between the reward after a preset number of steps and the current value estimate; this is achieved by minimizing the policy loss function. The parameters of the high-level network are updated, with the pruning factor set to a preset threshold. The low-level network performs gradient updates at each time step, and its update frequency is proportional to the update frequency of the high-level network. Through this asynchronous update mechanism, it is ensured that the sequence of guide points generated by the high-level network points to the endpoint in a macroscopic way, while the low-level network can learn a micro-control strategy to accurately reach the guide points while avoiding local obstacles.

[0016] Preferably, the state space The construction process also includes extracting local lidar features of the intelligent agent, wherein the features consist of a predetermined number of rays, and the detection range of each ray is within a preset detection interval; the global cost map Each pixel value represents the passage cost of that area, with values ​​falling within a preset cost level range. The maximum value represents an absolute obstacle, and the minimum value represents a completely open area.

[0017] Preferably, the high-level decision-making network introduces a distance constraint term when generating sub-target points. Requires sub-target coordinates With current location The distance is within a preset range to prevent the guide point from being too far away and making it difficult for the underlying network to reach.

[0018] Preferably, a decay factor is also introduced into the calculation logic of the intrinsic motivation. As the number of training rounds increases Increase, intrinsic reward The weights decay exponentially, and the base in the decay formula is set to a preset base to ensure that the agent can focus on using the learned knowledge to complete the task in the later stages of training.

[0019] Preferably, in the underlying obstacle avoidance execution logic, when the agent detects the presence of other agents or obstacles within a preset trigger radius, the collision penalty term in the reward function... It takes effect immediately, and its value is set to the preset penalty value, forcibly triggering the emergency reversal logic of the underlying strategy.

[0020] Preferably, in order to improve the stability of training, the joint training strategy introduces an experience replay mechanism, and the storage space size is set to a preset storage capacity; each time the gradient is updated, samples of a preset batch size are randomly extracted from the storage space to form a small batch of data for training.

[0021] Preferably, the mathematical logic support system involved in this invention includes the following 25 core variables and functional relationship descriptions: Variable 1 is the system global state vector. ; Variable 2 is the first Local observation vectors of each agent Variable 3 represents the probability distribution of environmental transition. Variable 4 is the high-level policy function. Variable 5 is the underlying strategy function. Variable 6 represents the high-level action, i.e., the sub-objective. Variable 7 represents the underlying action, i.e., the control command. Variable 8 is the state feature mapping function. Variable 9 is the predicted value from the forward dynamics model. Variable 10 represents the action predicted by the inverse dynamics model. Variable 11 represents the intrinsic reward value. Variable 12 represents the value of external environmental rewards. Variable 13 represents the total comprehensive reward. Variable 14 represents the discount return value. Variable 15 is the state value estimation function. Variable 16 is the dominance estimate. Variable 17 is the high-level network parameter set. Variable 18 is the set of parameters for the underlying network. Variable 19 is the strategy loss term. Variable 20 represents the value loss item. ; Variable 21 is the entropy regularization term. Variable 22 is the mean square value of the prediction error. Variable 23 is the gradient operator. Variable 24 is the learning rate parameter. Variable 25 is the total loss objective function. .

[0022] Preferably, the total loss objective function The calculation logic is as follows: policy loss term With value loss item The weighted sum, minus the coefficients The entropy regularization term of regulation ,in Set to the preset regularization coefficient to encourage exploratory strategies.

[0023] Preferably, the state feature mapping function By employing a residual network structure with a predetermined number of layers, the original high-dimensional image input is compressed into a feature vector of a predetermined dimension to improve the robustness of feature extraction.

[0024] Preferably, the high-level decision network employs a normalized exponential function with a temperature parameter during the sampling process. The temperature parameter gradually decreases linearly from a first preset temperature value at the start of training to a second preset temperature value at the end of training, so as to achieve a smooth transition from extensive exploration to precise utilization.

[0025] Preferably, the underlying control network further includes a motion smoothing module during execution, which ensures the physical feasibility of the motion trajectory by limiting the change in linear velocity between two adjacent time steps to no more than a preset linear acceleration threshold and the change in angular velocity to no more than a preset angular acceleration threshold.

[0026] Preferably, in the asynchronous reinforcement learning training, a multi-process parallel sampling technique is adopted, which simultaneously opens multiple environment instances for data collection. The processes exchange network parameters through shared memory, thereby improving the data collection efficiency.

[0027] Preferably, the high-level reward function In this context, a time penalty is introduced based on the total time taken to complete the task. Its value is proportional to the number of steps required to complete the task, and the proportionality coefficient is set to a preset proportionality coefficient, which aims to induce the agent to plan the solution with the shortest path length.

[0028] Preferably, the underlying control network receives the sub-target Then, a tendency score is calculated, which depends on the cosine of the angle between the current direction of movement and the direction of the line connecting the target point. The higher the score, the greater the immediate reward. The larger.

[0029] Preferably, during the training process, the environment transfer model employs a weight adjustment method based on importance sampling, assigning higher learning weights to samples with larger prediction errors, and setting the weight coefficient range to a preset weight interval.

[0030] Preferably, in the multi-agent collaborative logic, each agent not only considers its own observations when making decisions, but also integrates the state codes of other agents within a preset interaction radius through an attention mechanism module. The attention weights are dynamically calculated from the relative distance and relative speed between the agents.

[0031] Preferably, the global cost map The update frequency is set to a preset frequency, which can reflect the movement trajectory of dynamic obstacles in real time, thereby providing accurate global situation information for high-level decision-making networks.

[0032] Preferably, the path output by the path planning method consists of a series of continuous coordinate points, which are then smoothed by an interpolation algorithm to ensure that the agent moves smoothly and without sudden jumps in instructions when executing the path.

[0033] Preferably, the method further includes a safety monitoring module to monitor the distance between the agent and the nearest obstacle in real time. ,like If the value is below the preset safety threshold, the underlying control network will be forcibly taken over and an emergency braking action will be performed.

[0034] Preferably, when applied to a large-scale warehousing environment, the method can support at least a predetermined number of agents to perform path planning simultaneously, and the overall task completion efficiency of the system is significantly improved while ensuring no collisions.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. Effectively overcomes the reward sparsity problem in long-range path planning. By introducing intrinsic curiosity rewards based on dynamic prediction errors, the agent can receive continuous positive feedback from exploring the unknown environment even when it is far from the final goal in the early stages of training. This is like giving the agent a "desire for knowledge," driving it to actively move out of the starting area, greatly expanding the effective exploration range, thereby significantly accelerating convergence in the early stages of training and avoiding ineffective oscillations near the starting point.

[0037] 2. The hierarchical decision-making architecture significantly improves planning efficiency and quality in complex environments. This invention employs a two-tiered architecture of "high-level macro-navigation - low-level micro-obstacle avoidance," achieving clear decoupling of task logic. The high-level network focuses on global path optimization, unaffected by local, trivial obstacles; the low-level network focuses on instantaneous obstacle avoidance and smooth control, ensuring motion safety. This division of labor enables the system to plan near-globally optimal paths while also handling complex local emergencies, effectively reducing the probability of deadlock in dense obstacle areas.

[0038] 3. Significantly reduces the learning time of multi-agent cooperative strategies. The asynchronous reinforcement learning framework, combined with a goal-guiding mechanism, enables multiple agents to learn their strategies in parallel and efficiently. The sequential sub-goals provided by the high-level network offer a clear context for the underlying cooperative obstacle avoidance. Compared to traditional single-layer multi-agent reinforcement learning, this method can form effective group cooperative strategies more quickly, reducing the time and computational resource costs required for training.

[0039] 4. It possesses good environmental adaptability and system scalability. The method employs a continuous action space and flexible multimodal state input, allowing it to adapt to different platforms such as mobile robots and drones by adjusting parameters. The collaborative module integrating an attention mechanism ensures that the system maintains good decision-making stability even as the number of agents increases. The containerized deployment method also facilitates algorithm iteration, updates, and large-scale applications.

[0040] 5. Enhanced physical feasibility and overall safety of motion trajectories. The motion smoothing filter and safety envelope monitoring module in the underlying network jointly ensure that the generated control commands are smooth, continuous, and always within the safety boundaries. This allows the theoretically planned path to be faithfully reproduced by the real physical actuators, while providing reliable safety guarantees for the parallel operation of multiple agents in narrow spaces and intersections. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the overall technical architecture of the multi-agent path planning method based on hierarchical reinforcement learning and goal guidance proposed in this invention;

[0042] Figure 2 This is a schematic diagram of the core principle framework of the hierarchical decoupling architecture between high-level target guidance and low-level action control in this invention;

[0043] Figure 3 This is a flowchart of the logical flow of the curiosity mechanism based on the forward and backward dynamics model in this invention.

[0044] Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow of multi-agent local observation fusion and attention collaboration in this invention;

[0045] Figure 5 This is a flowchart illustrating the logical flow of asynchronous reinforcement learning gradient update and two-layer reward allocation in this invention.

[0046] Figure 6 This is a schematic diagram comparing the core principle of this invention with existing technologies in terms of training convergence speed and path planning success rate; Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0048] Example 1

[0049] In the implementation of the multi-agent path planning method based on hierarchical reinforcement learning and goal guidance, the entire system is constructed as a highly collaborative hierarchical decision-making architecture. The core of this architecture lies in realizing macro-level global path planning through high-level decision-making logic and achieving precise obstacle avoidance in the local environment using low-level control logic, thereby improving the operational efficiency of the multi-agent system in complex dynamic environments while ensuring the global optimality of the path.

[0050] Specifically, step S1, constructing a hierarchical decision-making framework, includes the following detailed implementation process:

[0051] During the system startup phase, the physical environment in which the multi-agent system operates is first digitally modeled. The environment map is then processed into a high-precision raster map to generate a global cost map. Each grid point in the global cost map carries the passage cost information for that location, with values ​​ranging from 0 to 255. A grid with a value of 255 represents an absolutely impassable obstacle area, while a grid with a value of 0 represents a completely open and passable area. During the rasterization process, considering the physical size of the agents, an expansion operation is performed on the edges of all obstacles, with the expansion radius set to 1.2 to 1.5 times the equivalent radius of the agent.

[0052] The high-level decision network of the hierarchical decision framework is configured as a feature extraction and target generation module based on a deep convolutional neural network. This network receives a global cost map as input and extracts the spatial topological features of the environment through a predetermined number of convolutional operations. The convolutional layers employ multiple sets of convolutional kernels with different receptive fields, such as 3x3 and 5x5 kernels, to capture environmental obstacle distribution information at different scales. Each convolutional layer is followed by a batch normalization layer and a linear rectified activation function to enhance the network's nonlinear fitting capability.

[0053] The high-level decision-making network performs forward inference every 30 to 50 time steps, outputting a sequence of intermediate waypoints for each agent. When generating intermediate waypoints, the fully connected layer at the end of the network maps the extracted global feature vectors to a probability distribution map proportional to the size of the environment map. This probability distribution is processed using a normalized exponential function, ensuring that the sum of the probabilities of each pixel on the map being selected as a sub-target is 1. The system then performs random sampling based on this probability distribution to obtain a set of sub-target coordinates. To ensure the physical reachability of the generated guide points, a distance constraint operator is introduced, requiring that the Euclidean distance between the sub-target coordinates and the agent's current position be within a preset range of 10 to 30 meters.

[0054] In the above method, step 2, constructing the underlying control network, includes the following detailed implementation process:

[0055] The underlying control network is designed as a micro-control module directly facing the actuator. Its core task is to achieve smooth motion control and avoid sudden local obstacles under the guidance of intermediate waypoints issued by higher layers. The input feature vector of the underlying network consists of three parts: the first part is the relative vector between the agent's current position and the intermediate waypoint, which includes distance and azimuth information; the second part is the agent's own velocity vector, including linear velocity and angular velocity; and the third part is the local environmental observation data, mainly from the lidar sensors deployed around the agent.

[0056] The lidar sensor scans at a frequency of 10 Hz to 20 Hz, generating 360-degree point cloud data. Before processing by the underlying network, the point cloud data is downsampled and converted into a predetermined number of ray features, each ray representing the distance to the nearest obstacle within a specific angular range. The underlying control network employs a multilayer perceptron structure, containing four hidden layers, with each layer having either 256 or 512 neurons. To prevent overfitting, random deactivation layers with a dropout rate of 0.2 are introduced between the hidden layers.

[0057] The output layer of the underlying control network employs a hyperbolic tangent activation function, mapping the original output of the neural network to a range of -1 to +1. Subsequently, a scaling module converts this value into actual physical control commands. For example, linear velocity commands are limited to 0 to 2.0 meters per second, and angular velocity commands are limited to -1.5 to +1.5 radians per second. Before the commands are sent to the motor driver, they pass through a motion smoothing filter to limit acceleration variations between adjacent time steps, ensuring the smoothness of the agent's motion.

[0058] In the above method, step 3, establishing an intrinsic reward function based on the curiosity mechanism, includes the following detailed implementation process:

[0059] To address the reward sparsity problem in large-scale environments, this embodiment constructs a curiosity module consisting of a forward dynamics model and a backward dynamics model. The backward dynamics model learns feature representations of environmental state changes, taking the environmental states at two consecutive time points as input and the predicted action as output. By minimizing the cross-entropy loss between the predicted and actual actions, the network can automatically ignore task-irrelevant background noise in the environment and extract only the core features that affect the agent's movement.

[0060] The forward dynamics model operates in the feature space extracted by the inverse model. Its inputs are the feature vector at the current time step and the action taken, and its output is the predicted value of the feature vector at the next time step. The intrinsic reward is calculated as follows: when the agent enters a previously unexplored region or encounters a complex dynamic environment, the prediction error of the forward dynamics model increases significantly. The square of this prediction error is used as the intrinsic reward signal and fed back to the reinforcement learning algorithm.

[0061] The specific formula for calculating the intrinsic reward is as follows:

[0062]

[0063] in, The scaling factor representing intrinsic reward typically ranges from 0.01 to 0.1. This represents the true state features at the next moment extracted by the inverse dynamics model; This represents the feature vector predicted by the forward dynamics model. Through this mechanism, even in the absence of external reward, the agent will actively explore unknown regions with large prediction errors due to "curiosity."

[0064] In the above method, step 4, which involves performing asynchronous reinforcement learning training, includes the following detailed implementation process:

[0065] The system employs an asynchronous reinforcement learning framework based on an actor-critic architecture. The high-level decision network and the low-level control network each have their own independent policy and value networks, but they are asynchronously updated during training through a shared experience replay pool. The experience replay pool has a capacity of 1 million sample entries, each entry containing a state, action, reward, next state, and a flag indicating whether the task has terminated.

[0066] During training, the reward function of the high-level network consists of three weighted parts: the first part is the task achievement reward, which is a large positive value given when the agent reaches the final destination; the second part is the intrinsic reward, i.e., the feedback generated by the aforementioned curiosity mechanism; and the third part is a time penalty, which deducts a small score at each time step to encourage the agent to find the shortest path. The reward function of the low-level network focuses on local tasks, including sub-goal approach rewards and collision penalties. When the distance between the agent and obstacles or other agents is less than a preset safety threshold (e.g., 0.3 meters), a large negative penalty is immediately triggered.

[0067] The training algorithm employs a near-end policy optimization logic, using a pruning probability ratio to ensure the stability of policy updates. The training process is conducted in parallel on a high-performance computing server, with 16 or 32 parallel environment instances simultaneously collecting data. The parameter update frequency of the higher-level network is set to one-fifth of that of the lower-level network. This asynchronous update mechanism ensures that the lower-level network has sufficient time to adapt to the sub-targets generated by the higher-level network, thus forming a stable hierarchical collaborative logic.

[0068] As a specific application scenario of this embodiment, consider an automated logistics warehouse with a size of 200 meters by 200 meters. The warehouse contains densely packed shelves and 50 mobile robots (agents). When performing tasks, the higher-level network first plans a series of intermediate guide points approximately 20 meters apart for each robot based on a global map of the warehouse. These guide points avoid dead ends in the densely packed shelves. After receiving the guide points, the lower-level network fuses LiDAR data in real time and dynamically avoids other moving robots while heading to the guide points. Due to the introduction of a curiosity mechanism, the robots can quickly traverse the entire warehouse in the early stages of training, establishing a complete spatial awareness, and its convergence speed is approximately three times faster than traditional methods.

[0069] Furthermore, in terms of multi-agent collaboration, this embodiment also introduces an interaction module based on an attention mechanism. When making decisions, each agent fuses its own local feature vector with the features of other agents within a 5-meter radius using a weighting calculation unit. The weights depend on the relative velocity vectors and collision risk levels between them. This mechanism enables multiple robots to automatically negotiate a yielding order when they meet in narrow passages, effectively avoiding deadlock.

[0070] At the data processing level, all sensor inputs are preprocessed. LiDAR data undergoes noise removal via one-dimensional median filtering, while the global cost map is generated using multi-scale pyramid downsampling techniques to enable feature extraction at different levels of abstraction by higher-level networks. The system's internal communication protocol employs zero-copy technology, ensuring that the latency of transmitting high-dimensional feature tensors between different network modules is less than 1 millisecond.

[0071] Example 2

[0072] Based on Example 1, this embodiment enhances the method for scenarios with stronger environmental dynamics and higher uncertainty (such as logistics sorting centers), focusing on improving the system's ability to predict dynamic obstacles and the adaptability of the reward mechanism.

[0073] Enhancement to step S1: In addition to receiving the static global cost map, the high-level decision network adds a dynamic obstacle prediction branch. This branch takes into account the local location sequence of dynamic obstacles detected by the lower-level LiDAR at the current time and the previous four time steps. This sequence is processed by an LSTM network to predict the possible location regions of these dynamic obstacles within a future period (the next high-level decision cycle), and these regions are temporarily "drawn" on a copy of the global cost map at a higher cost. Based on this cost map copy incorporating dynamic prediction information, the high-level network performs sub-target sampling, thereby proactively avoiding the predicted paths of other robots or personnel, achieving forward-looking planning.

[0074] The enhancement to step S3 introduces an adaptive environmental complexity factor. This factor is dynamically adjusted based on the grid entropy value of the area where the agent is currently located. In areas with sparsely distributed obstacles, the weight of intrinsic rewards is reduced, allowing the agent to focus on moving quickly using known paths; while in areas with extremely dense obstacles and complex topology, the weight of intrinsic rewards is automatically increased, stimulating the agent to perform more detailed micro-management exploration.

[0075] Specifically, the calculation logic of the total loss objective function in this embodiment is as follows:

[0076]

[0077] in, This represents the policy gradient loss, used to optimize the action distribution; The value function loss is represented by the mean squared error, which measures the accuracy of the value estimate. Represents policy entropy, used to encourage agents to maintain a certain degree of exploration randomness and prevent them from getting trapped in local optima; The joint prediction loss representing the forward and backward dynamics models; , and These are preset weighting coefficients, which are set to 0.5, 0.01, and 1.0 respectively in this embodiment.

[0078] Optimization of the training process: Training begins with simple scenarios (few robots, sparse obstacles, and close targets). As policy performance improves, the scenario difficulty is gradually increased (more robots, denser shelves, fast-moving dynamic obstacles, and more distant target points), allowing the agent to progressively master complex skills. A feedback channel is established from the bottom layer to the top layer. If a sub-target generated by the higher-level network triggers multiple collision alarms or cannot be approached by the lower-level network for an extended period (e.g., more than twice the expected time) during execution, that sub-target will be marked as "invalid." The higher-level network will be penalized during updates, thus learning to avoid generating such unreasonable sub-targets.

[0079] In terms of specific hardware implementation, the method in this embodiment is deployed on a heterogeneous computing platform. The high-level decision-making network runs on the graphics processor of the central server and is responsible for the periodic analysis of the global situation; the low-level control network, after model quantization and pruning, is deployed in the embedded neural network acceleration module on the vehicle of the intelligent agent. This distributed deployment scheme greatly reduces the dependence on wireless communication bandwidth. Even in the edge area of ​​a warehouse where communication is unstable, the intelligent agent can still rely on the low-level network to maintain basic obstacle avoidance and target approach capabilities.

[0080] Furthermore, this embodiment also incorporates a priority negotiation mechanism for multi-agent conflict scenarios. When the projected trajectories of two agents overlap within the next two seconds, the system calculates a priority score based on the urgency of their respective tasks and remaining battery power. The agent with the lower score receives a deceleration command generated by the underlying network, while the agent with the higher score maintains its original speed, thereby achieving traffic flow optimization at the group level.

[0081] Hardware Deployment: This embodiment employs a heterogeneous computing architecture. The high-level decision network (computationally demanding but operating at a low frequency) is deployed on the GPU of the central server in the warehouse. The low-level control network and curiosity module (requiring low latency), after model quantization, are deployed on the embedded computing unit of each robot. Both exchange sub-goals and state information infrequently via a wireless network. This edge-cloud collaborative approach reduces the stringent requirements for network real-time performance.

[0082] Example 3

[0083] This embodiment focuses on illustrating the fault tolerance and safety mechanisms of the present invention under extremely harsh or faulty conditions, as well as its extended applications in three-dimensional space (drone swarms).

[0084] Enhanced security and fault tolerance mechanisms:

[0085] Safety envelope based on obstacle control function: A safety monitoring module is added to the final output of the underlying control network. This module calculates in real time a preset safety set (e.g., the set of all possible speed commands with a distance greater than 0.2 meters from all obstacles) for the robot's current state (position, velocity). If the original command output by the neural network is outside this safety set, the module will "project" it in real time and online to the nearest point within the safety set as the final command to be executed. This provides a solid "safety baseline" for the neural network's learning process, allowing for timely correction even if the network outputs dangerous commands in the early stages of training.

[0086] Collaborative Perception Under Sensor Failure: When a robot's LiDAR malfunctions, its underlying network loses crucial local observation input. In this situation, the robot automatically switches to "collaborative following" mode. It requests LiDAR observations and its own pose from neighboring robots via a wireless network. Upon receiving these, it uses a coordinate transformation matrix to convert the neighbors' observation data to its own coordinate system, concatenating them into a "pseudo-observation" vector, which serves as input to its own underlying network. Although accuracy is compromised, guided by higher-level sub-targets and led by its neighbors, it can still roughly follow the movement, avoiding complete system failure and improving the overall robustness of the system.

[0087] Curiosity memory decay: The environment may change over time (e.g., goods are removed). To prevent the curiosity model from losing interest in previously explored areas and becoming insensitive to subsequent changes (obstacle removal), we set a "shelf life" for the samples used to train the curiosity model in the experience replay pool. After a period of time, the contribution weight of the samples to model updates decays exponentially, forcing the model to pay more attention to "recent" observations and maintain continuous curiosity about environmental changes.

[0088] Expanding into three-dimensional space (drone swarm search):

[0089] The method of this invention is applied to a collaborative search task of a UAV swarm in a forest area. In this case, the state space is expanded to three dimensions (x, y, z coordinates and yaw angle).

[0090] High-level network: The input is a 3D voxel map, and the output is a series of waypoints (sub-objectives) in 3D space.

[0091] The underlying network's inputs include its relative position to the 3D sub-target, its own 3D velocity, and point cloud features from a depth camera or 3D LiDAR (which can be simplified to multi-directional depth rays). The output is the rotational speed difference control commands for the four rotors.

[0092] Curiosity mechanism: The same task drives drones to explore uncovered forest areas.

[0093] Coordination mechanism: Attention mechanism helps drones maintain formation in dense forests and avoid collisions.

[0094] In this application, the curiosity-driven upper-layer network enables the swarm to autonomously and efficiently cover the search area, while the lower-layer network ensures that the drones can flexibly avoid tree branches. Tests show that, within the same timeframe, the drone swarm using this method achieves approximately 45% higher exploration coverage of unknown forest areas compared to the traditional method of combining global path planning with local obstacle avoidance.

[0095] Software Architecture: The system in this embodiment adopts a microservice containerized architecture. High-level planning services, low-level control services, curiosity model services, and experience replay pool services are all encapsulated as independent Docker containers, communicating asynchronously through a high-throughput message middleware. This architecture facilitates independent development, testing, upgrades, and elastic scaling of resources for different modules.

[0096] In summary, this invention decomposes long-term tasks by constructing a hierarchical decision-making framework, solves the sparse reward problem using a curiosity mechanism, and efficiently trains multi-agent collaborative strategies through asynchronous reinforcement learning. Furthermore, by integrating security monitoring, fault-tolerant logic, and a flexible software architecture, it ensures that the method is not only theoretically effective but also possesses robustness and deployability for practical engineering applications. This invention provides a systematic solution for autonomous path planning of multiple agents in large-scale, dynamic environments.

[0097] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any alternative solutions derived or readily conceived by those skilled in the art within the technical principles and concepts disclosed in the present invention should be considered to be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A multi-agent path planning method based on hierarchical reinforcement learning and goal-guided approach, characterized in that, Includes the following steps: Step S1: Construct a hierarchical decision framework: Digitally model the physical environment in which the multi-agent operates, rasterize the environment map to generate a global cost map, where each grid point carries the access cost information for that location, and perform an expansion operation based on the agent's physical size on obstacle edges; Utilize a high-level decision network to receive the global cost map as input, extracting spatial topological features of the environment through a predetermined number of convolutional operations, employing multiple sets of convolutional kernels with different receptive fields to capture multi-scale environmental obstacle distribution information; The high-level decision network periodically performs forward inference, mapping the extracted global feature vectors to a probability distribution map proportional to the size of the environment map, and then randomly sampling after processing with a normalized exponential function, outputting the intermediate path point sequence for each agent, and introducing a distance constraint operator during the sampling process to limit the Euclidean distance between the sub-target coordinates and the agent's current position; Step S2, constructing the underlying control network: The underlying control network serves as a micro-control module directly facing the actuator. Its input feature vector consists of the relative vector between the agent's current position and the intermediate track point, the agent's own velocity vector, and local environmental observation data. The local environmental observation data is obtained by downsampling the raw data generated by the sensor and converting it into a predetermined number of ray features. Each ray represents the distance to the nearest obstacle within a specific angle range. The underlying control network adopts a multilayer perceptron structure. Its output layer uses an activation function to map the raw output of the neural network to a continuous action space, and converts it into actual linear velocity and angular velocity commands through a scaling transformation module. Before the commands are issued, an action smoothing filter limits the acceleration changes between adjacent time steps. Step S3: Establish an intrinsic reward function based on the curiosity mechanism: Construct a curiosity module consisting of a forward dynamics model and a backward dynamics model. The backward dynamics model takes into account the environmental states at two consecutive time points and outputs a predicted action. It learns the feature representation of environmental state changes and extracts core features by minimizing the cross-entropy loss between the predicted action and the actual action. The forward dynamics model operates in the feature space formed by the core features, taking into account the feature vector at the current time point and the action taken, and outputs a predicted value for the feature vector at the next time point. The prediction error between the predicted value and the true feature vector is calculated, and the square of the prediction error is multiplied by a preset scaling factor and fed back as an intrinsic reward signal to the reinforcement learning algorithm to drive the agent to actively... Explore unknown regions with large prediction errors; Step S4, perform asynchronous reinforcement learning training: adopt an asynchronous reinforcement learning framework based on actor-critic architecture, where the high-level decision network and the low-level control network each have independent policy networks and value networks, and are updated asynchronously through a shared experience replay pool; the high-level reward function is composed of a weighted sum of task achievement reward, intrinsic reward, and time penalty, while the low-level reward function is composed of sub-goal tendency reward and collision penalty; utilize proximal policy optimization logic to perform policy updates by pruning probability ratios, and set the parameter update frequency of the high-level decision network to a predetermined proportion of the update frequency of the low-level control network, achieving iterative optimization of multi-agent cooperative policies by exchanging network parameters through shared memory among processes.

2. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The process of constructing the hierarchical decision framework in step one further includes: each pixel value of the global cost map represents the passage cost of the area, and the value is within a preset cost level range, where the preset maximum value represents an absolute obstacle and the preset minimum value represents a completely empty area; during the rasterization process, the expansion radius of the dilation operation is set to a preset proportional coefficient of the agent's equivalent radius; the high-level decision network is configured with a batch normalization layer and a linear rectified activation function after the convolutional layer; the high-level decision network uses a normalized exponential function with a temperature parameter during the sampling process, and the temperature parameter linearly decays from a first preset temperature value at the start of training to a second preset temperature value at the end of training; the distance constraint operator requires that the distance between the intermediate waypoint and the agent's current position is within a preset distance range to prevent the guide point from being too far away, causing the lower-level control network to fail.

3. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The process of constructing the underlying control network in step two further includes: the sensor is a lidar sensor configured around the agent, which scans at a preset scanning frequency and generates point cloud data; the underlying control network contains a predetermined number of hidden layers, and random deactivation layers with a preset deactivation rate are introduced between the hidden layers to suppress model overfitting; the motion smoothing filter ensures that the motion trajectory conforms to physical and dynamic constraints by limiting the change in linear velocity between two adjacent time steps to no more than a preset linear acceleration threshold and the change in angular velocity to no more than a preset angular acceleration threshold; after receiving a sub-target, the underlying control network calculates a tendency score, which depends on the cosine of the angle between the current motion direction and the direction of the line connecting the target point, and the tendency score is positively correlated with the sub-target tendency reward in the underlying reward function.

4. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The process of establishing the intrinsic reward function in step three further includes: the inverse dynamics model learns the feature representation of environmental state changes, automatically ignores background noise irrelevant to the task in the environment, and extracts the core features that affect the agent's movement; the core features are generated by a state feature mapping function, which adopts a residual network structure with a predetermined number of layers to compress the original high-dimensional image input into a feature vector of a predetermined dimension; an attenuation factor is introduced into the calculation logic of the intrinsic reward signal, and as the number of training rounds increases, the weight of the intrinsic reward signal in the total reward decays exponentially, and the base in the attenuation formula is set to a preset base, driving the agent to focus on using the learned knowledge to complete the path planning task in the later stage of training; the forward dynamics model adopts a weight adjustment method based on importance sampling during the training process, and assigns learning weights within a preset weight range to samples with large prediction errors.

5. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The asynchronous reinforcement learning training process in step four further includes: the storage space of the experience replay pool is set to a preset storage capacity, and each storage entry contains the current state, the executed action, the reward value, the next time step state, and the task termination flag; during each gradient update, samples of a preset batch size are randomly selected from the experience replay pool to form a mini-batch for training; the total loss objective function of the asynchronous reinforcement learning training is composed of the weighted sum of the policy loss term and the value loss term, minus the entropy regularization term adjusted by the preset regularization coefficient; the value of the time penalty term is proportional to the total number of steps required to complete the task, and the proportionality coefficient is set to a preset proportionality coefficient; the collision penalty in the underlying reward function is triggered when the agent detects an obstacle within a preset trigger radius, and its value is set to a preset penalty value.

6. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The method also includes multi-agent collaborative logic: each agent, when making a decision, integrates the state codes of other agents within a preset interaction radius through an attention mechanism module; The attention mechanism module includes a weight calculation unit, which is used to dynamically calculate attention weights based on the relative distance and relative speed between each agent. When the predicted trajectories of two agents overlap within a preset prediction time period, the system executes priority negotiation logic to calculate priority scores based on the urgency of each agent's task and remaining battery power. Agents with scores below a preset threshold execute deceleration commands generated by the underlying control network, while agents with scores above the preset threshold maintain their original speed, thereby achieving traffic flow optimization at the group level.

7. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The method also includes dynamic environment adaptation logic: the high-level decision network adds a dynamic obstacle prediction layer, which receives a local observation sequence of a continuously preset number of time steps, uses a long short-term memory network to extract the motion trajectory features of the dynamic obstacles, and projects the motion trajectory features onto the global cost map; When a sub-target generated by the high-level decision-making network triggers multiple collisions or remains unreachable for an extended period during the execution of the low-level control network, the sub-target is marked as an invalid sample, and the high-level decision-making network is prompted to adjust its strategy parameters through a feedback channel. An environmental complexity adaptive factor is introduced into the calculation logic of the intrinsic reward signal. This environmental complexity adaptive factor is dynamically adjusted based on the grid entropy value of the area where the agent is currently located. The intrinsic reward weight is reduced in areas with sparse obstacle distribution and increased in areas with dense obstacle distribution.

8. The multi-agent path planning method based on hierarchical reinforcement learning and goal guidance according to claim 1, characterized in that, The method also includes security monitoring and fault tolerance logic: a security envelope monitoring module based on the control barrier function theory is constructed to perform real-time verification before the control commands of the underlying control network enter the actuator; if the control command causes the agent to enter a state that violates security constraints, the security envelope monitoring module performs minimum disturbance correction on the original command and projects it into a preset security control set; when the agent's sensor fails and cannot obtain local observation data, the agent switches to cooperative mode, receives the observation features of neighboring agents through a wireless link, and uses a spatial transformation matrix to transform them into its own pseudo-observation data; the method adopts containerized deployment at the software architecture level, deploying high-level strategies, low-level control, curiosity modules, and experience replay pools in independent containers, and exchanging data asynchronously through a message bus.

Images