Mine robot collaborative path planning method based on multi-agent reinforcement learning

Abstract

The invention relates to a mine robot cooperative path planning method based on multi-agent reinforcement learning, and belongs to the technical field of mine robot cooperative control and intelligent decision, and the method comprises the following steps: constructing refined environment characterization based on multi-source sensor information through a terrain cost modeling module, and calculating a terrain cost value; path planning is carried out by adopting a centralized training-decentralized execution mechanism through a multi-agent reinforcement learning framework; motion conflicts between the robots are monitored and solved in real time through a dynamic conflict resolution module; and cross-robot knowledge transmission and strategy migration are realized through the knowledge sharing unit. According to the method, the path planning efficiency, the collaborative operation safety and the learning adaptability of multiple robots in complex dynamic environments such as mine catastrophe are effectively improved, and the method is suitable for collaborative detection, rescue and transportation tasks of multiple types of mine robots.

Description

Technical Field

[0001] This invention relates to the field of collaborative control and intelligent decision-making technology for mining robots, specifically to a collaborative path planning method for mining robots based on multi-agent reinforcement learning. Background Technology

[0002] In mine disaster rescue missions, multi-robot systems can significantly improve the coverage efficiency of search and rescue areas through collaborative operations. However, existing path planning methods have significant technical bottlenecks in areas such as dynamic environment adaptability, multi-robot collaborative conflict resolution, refined terrain assessment, and cross-robot knowledge sharing.

[0003] The mine disaster environment is highly dynamic, with terrain constantly changing as the disaster progresses, and unpredictable factors such as new collapses and water spread occurring frequently. Traditional path planning methods based on pre-set maps lack real-time environmental perception and replanning capabilities, causing robots to frequently get stuck in dead ends during task execution. Real-world cases show that due to the failure to detect new obstacles in time, multiple robots may be trapped in narrow passages simultaneously, resulting in a task interruption rate exceeding 30% and severely delaying rescue progress.

[0004] When multiple robots work collaboratively, motion conflicts are prone to occur in critical areas such as intersections and narrow passages, with collision rates exceeding 20%. Traditional "first-come, first-served" avoidance mechanisms lack a global optimization perspective, easily leading to decision-making deadlocks of "mutual waiting," with robot dwell times often exceeding 10 seconds. Furthermore, due to the lack of effective task allocation and area coordination mechanisms, the overlap rate of the detection areas of multiple robots exceeds 40%, resulting in a serious waste of detection resources.

[0005] Mine terrain is highly complex, containing various terrain elements such as coal slag, steep slopes, and water accumulation. Different terrains significantly affect the robot's mobility. Traditional algorithms simplify the terrain into a binary "accessible / impassable" model without quantifying the actual cost of passage. This results in planned paths that, while having the shortest geometric length, are actually difficult to traverse, increasing robot energy consumption by more than 50%.

[0006] Existing methods have significant shortcomings in learning capabilities, requiring manual pre-setting of obstacle avoidance rules (such as "turn left upon encountering an obstacle"), and exhibiting poor adaptability when facing unknown terrain (such as soft cinder areas). Furthermore, multi-robot systems lack effective knowledge-sharing mechanisms; each robot must independently explore its environment, resulting in low learning efficiency. Achieving stable performance requires more than 100 trials, making it difficult to meet the real-time requirements of disaster relief. Summary of the Invention

[0007] This invention provides a collaborative path planning method for mining robots based on multi-agent reinforcement learning, aiming to achieve efficient and safe collaborative operation of multiple robots in complex mining environments.

[0008] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:

[0009] A collaborative path planning method for mining robots based on multi-agent reinforcement learning includes the following steps:

[0010] (1) The terrain cost modeling module constructs a refined environmental characterization based on multi-source sensor information and calculates the terrain cost value;

[0011] (2) Path planning is performed using a centralized training-distributed execution mechanism through a multi-agent reinforcement learning framework;

[0012] (3) Real-time monitoring and resolution of motion conflicts between robots through a dynamic conflict resolution module;

[0013] (4) Cross-robot knowledge transfer and strategy transfer are achieved through knowledge sharing units.

[0014] Preferably, the terrain cost modeling module specifically includes: extracting slope using LiDAR and cameras. Obstacle height Road surface type and traffic width Four types of terrain feature parameters;

[0015] Set the high-cost terrain condition as slope. ≥35°, obstacle height ≥300mm or passage width ≤800mm; Establish terrain cost function It was determined through training with historical data that α+β+γ+δ=1 and α, β, γ, and δ are all greater than 0.

[0016] Priority is given by κ, which is the road cost coefficient, with κ=1.5 for the cinder area, κ=2.0 for the waterlogged area, and κ=1.0 for the rock area. α, β, γ, and δ represent the slope factor, obstacle height factor, road surface type factor, and traffic width factor, respectively.

[0017] Prior to this, the historical data comes from the mine environment simulation platform and real mine test data, including sensor readings and passage cost labels under different terrain scenarios such as steep slopes, water accumulation, coal slag accumulation and gentle rock road surfaces, and the passage cost label has a value range of [0,1].

[0018] In the raster map output by the cost function, the cost value C of each cell is used for path evaluation. The terrain is divided into four categories according to the cost value C: 0-0.3 is a safe and passable area, 0.3-0.6 is a cautious area, 0.6-0.8 is a high-risk area, and 0.8-1.0 is a prohibited area.

[0019] Preferably, the multi-agent reinforcement learning framework includes:

[0020] The state space S includes the robot's own position, velocity, surrounding terrain cost, and other robot position information;

[0021] The action space A includes forward movement, backward movement, left turn, right turn, and stopping, and its range is dynamically adjusted according to speed.

[0022] reward function , ;

[0023] in, The total reward value obtained from the current action. To explore rewards, Rewards for efficiency As a reward for the goal, As a penalty for collision, As a consequence of terrain, For region overlap penalty, As a penalty for stagnation;

[0024] The central controller is used to collect all robot experience data and optimize the global policy based on the deep deterministic policy gradient algorithm. Each robot relies on local observations to achieve distributed decision-making during the execution phase.

[0025] Priority ;

[0026] in, As an indicator variable, when the grid cell reached by the robot in this action has not yet been visited by any robot in the current task, =1, otherwise =0;

[0027] ;in, The value of the current location grid cell output by the aforementioned terrain cost modeling module; the larger the value, the more difficult it is to pass through.

[0028] A fixed penalty of 50 is deducted when the robot collides with an obstacle or another robot; this penalty is recorded as 0 when there is no collision.

[0029] Where d is the Euclidean distance from the robot's current position to the predetermined target point. As the robot gradually approaches the target point, d decreases. Increase accordingly;

[0030] :in, The terrain cost of the current location grid is used to reduce the penalty for higher costs, thus preventing robots from frequently entering high-cost areas.

[0031] When different robots occupy the same grid cell within the same time window, a fixed penalty of 3 is deducted; if no position overlap occurs, this penalty is recorded as 0.

[0032] Where t is the duration for which the robot remains stationary or moves at a speed below a preset threshold at its current position. The longer the stationary time, the greater the penalty, which is used to prevent the robot from remaining stationary for a long time.

[0033] Preferred, the specific steps of the dynamic conflict resolution module are as follows:

[0034] When the Euclidean distance between any two robots is ≤2 meters and the predicted paths intersect within a future preset time window, a conflict resolution process is triggered.

[0035] The priority weight of each robot is calculated based on task priority and remaining path length. Wherein: the task priority is represented by a pre-set discrete level, the task priority of life detection task is recorded as 3, the task priority of environmental scanning task is recorded as 2, and the task priority of material transportation task is recorded as 1.

[0036] The remaining path length is denoted as The maximum remaining path length of the robots currently participating in the conflict is denoted as . The priority weight is calculated according to the following formula: ;

[0037] in, The task priority for the nth robot is... These are the weighting coefficients;

[0038] Compare the priority weights of each robot Robots with higher priority weights gain the right of way and continue along the original path. Robots with lower priority weights temporarily deviate from the current path normal direction by a preset distance to avoid obstacles. The deviation is ≤1 meter and the avoidance time is controlled within 2 seconds. After the avoidance is completed, the robot returns to the original planned path to continue performing the task.

[0039] Preferably, the knowledge-sharing unit specifically includes:

[0040] Construct a shared experience pool, a distributed knowledge sharing mechanism, and a storage experience tuple. Where s is the state, a is the action, r is the reward, s' is the next state, C is the terrain cost, and t is the timestamp;

[0041] The shared experience pool adopts a priority experience replay mechanism, which allocates experience priorities based on time difference error. Experiences with higher priorities have a greater probability of being selected for training.

[0042] The knowledge distillation technique is used to compress the global strategy of the central controller into a lightweight model and distribute it to each robot;

[0043] The robot can directly access experience data from the shared experience pool to achieve knowledge reuse.

[0044] After adopting the above methods and steps, the present invention has the following advantages:

[0045] This invention includes a terrain cost modeling module, a multi-agent reinforcement learning framework, a dynamic conflict resolution module, and a knowledge sharing unit. The terrain cost modeling module constructs a refined environmental representation based on multi-source sensor information and calculates the terrain cost. The multi-agent reinforcement learning framework employs a centralized training-distributed execution mechanism to achieve distributed decision-making. The dynamic conflict resolution module resolves motion conflicts based on task priority and remaining path length. The knowledge sharing unit achieves cross-robot knowledge transfer through a shared experience pool and policy transfer. This invention effectively improves the path planning efficiency, collaborative operation safety, and learning adaptability of multiple robots in complex dynamic environments such as mine disasters, and is suitable for collaborative detection, rescue, and transportation tasks of various types of mine robots. Attached Figure Description

[0046] Figure 1 This is a flowchart of the process of the present invention;

[0047] Figure 2 This is a schematic diagram of terrain cost modeling according to the present invention;

[0048] Figure 3 This is a diagram of the multi-agent reinforcement learning framework of the present invention;

[0049] Figure 4 This is a schematic diagram of the dynamic conflict resolution mechanism of the present invention;

[0050] Figure 5 This is a schematic diagram of the working principle of the knowledge sharing unit of the present invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the full text.

[0052] Combined with appendix Figures 1-5 A collaborative path planning method for mining robots based on multi-agent reinforcement learning includes the following steps:

[0053] (1) The terrain cost modeling module constructs a refined environmental characterization based on multi-source sensor information and calculates the terrain cost value;

[0054] (2) Path planning is performed using a centralized training-distributed execution mechanism through a multi-agent reinforcement learning framework;

[0055] (3) Real-time monitoring and resolution of motion conflicts between robots through a dynamic conflict resolution module;

[0056] (4) Cross-robot knowledge transfer and strategy transfer are achieved through knowledge sharing units.

[0057] The terrain cost modeling module specifically includes: extracting four types of terrain feature parameters—slope θ, obstacle height h, road surface type, and passage width w—using LiDAR and cameras; setting high-cost terrain conditions: slope θ ≥ 35°, obstacle height h ≥ 300mm, or passage width w ≤ 800mm; assigning road surface cost coefficients to different road surface types: 1.5 for cinder areas, 2.0 for waterlogged areas, and 1.0 for rocky areas; and establishing a terrain cost function C = α × θ + β × h + γ × road surface coefficient + δ × (1 / w), where α, β, γ, and δ are weight coefficients, determined through training with historical data, and satisfying α + β + γ + δ = 1 and α, β, γ, and δ are all greater than 0.

[0058] The historical data comes from a mine environment simulation platform and real mine test data, including sensor readings and expert-assessed passage cost labels under no fewer than 1000 sets of different terrain scenarios (including steep slopes, water accumulation, coal slag accumulation, and gentle rock surfaces). Specifically: lidar is used to collect slope θ and obstacle height h; cameras are used to identify road surface types and provide road surface type cost coefficients κ; distance sensors are used to obtain passage width w; and expert-assessed passage cost labels are also included. The value range is [0,1].

[0059] The training method is as follows: For the terrain cost function C... Establish a multiple linear regression model to minimize the prediction cost of N samples. With toll cost label The mean square error between (1 / N)∑( - Let α² be the loss function. The weight coefficients α, β, γ, and δ are iteratively optimized using gradient descent until the loss function converges and satisfies α + β + γ + δ = 1. In the grid map output by the cost function, the cost C of each cell is used for path evaluation. When C ≥ 0.8, it is marked as a "cautious passage" area, thus providing a quantitative basis for robot path selection.

[0060] The multi-agent reinforcement learning framework includes:

[0061] The state space S includes the robot's own position, velocity, surrounding terrain cost, and other robot position information;

[0062] The action space A includes forward movement, backward movement, left turn, right turn, and stopping, and its range is dynamically adjusted according to speed.

[0063] The reward function R includes positive rewards and penalties: reward for moving to an unexplored area, reward for traveling along a low-cost path, reward for approaching the target point, penalty for collision, penalty for entering a high-cost area, penalty for overlapping areas, and penalty for stagnation.

[0064] The central controller collects all robot experience data and optimizes the global policy based on a deep deterministic policy gradient algorithm. Each robot relies on local observations to achieve distributed decision-making during the execution phase.

[0065] The dynamic conflict resolution module specifically includes: triggering a conflict resolution process when the Euclidean distance between any two robots is ≤2 meters and the predicted paths intersect within a future preset time window; and calculating the priority weight of each robot based on task priority and remaining path length. Wherein: the task priority is represented by a pre-defined discrete level, with the task priority of life detection task denoted as 3, the task priority of environmental scanning task denoted as 2, and the task priority of material transportation task denoted as 1; the remaining path length is denoted as... The maximum remaining path length of the robots currently participating in the conflict is denoted as . The priority weight is calculated according to the following formula:

[0066] ;

[0067] in, The task priority for the nth robot is... The weighting coefficient is preferably 0.7. Offline experiments were conducted using multiple sets of historical data on robot encounters and path intersections collected on a mine environment simulation platform. Values ​​were selected by traversing the [0,1] interval and minimizing the weighted sum of the robot's average waiting time and the number of conflicts. The priority weights of each robot were then compared. Robots with higher priority weights gain the right of way and continue along the original path, while robots with lower priority weights temporarily deviate from the current path normal direction by a preset distance to avoid the obstacle. The deviation is ≤1 meter and the avoidance time is controlled within 2 seconds. After the avoidance is completed, the robot returns to the original planned path to continue performing the task.

[0068] The knowledge sharing unit specifically includes: building a shared experience pool to store the robot's successful obstacle avoidance, efficient path and other experiences, with a capacity of 100,000 records; using knowledge distillation technology to compress the global strategy of the central controller into a lightweight model, reducing the parameter size by 60%, and distributing it to each robot; other robots can directly call the experience data in the shared experience pool to achieve knowledge reuse.

[0069] The shared experience pool employs a priority-based experience replay mechanism. This mechanism includes: storing data such as "state-action-reward-next state" generated by each robot during collaborative path planning as experience entries in the shared experience pool; and calculating the corresponding time difference error for each experience entry using the current value assessment network.

[0070] The larger the error, the more important the experience is for improving the current strategy; prioritize each experience based on the error magnitude.

[0071] The higher the priority of an experience, the greater the probability that it will be selected for training in the future. Therefore, when randomly selecting a small batch of experiences from the shared experience pool for network training, it is more inclined to select higher priority experiences. After each training is completed, the error of the extracted experiences is recalculated using the updated network and its priority is adjusted accordingly, so as to realize the dynamic update of the priority of each experience in the shared experience pool.

[0072] The specific implementation steps of this invention are as follows:

[0073] Step 1: System Initialization and Terrain Modeling: Target areas are assigned to each robot. The terrain cost module generates a grid map and calculates the cost value of each cell. Terrain is categorized based on cost value: areas with a cost value of 0–0.3 are safe and passable; areas with a cost value of 0.3–0.6 are areas requiring caution; areas with a cost value of 0.6–0.8 are high-risk areas; and areas with a cost value of 0.8–1.0 are prohibited areas. For typical scenarios, the cost values ​​are set as follows: flat roads with a slope less than 15° and a dry rock surface have a cost value between 0.1 and 0.2; gentle slopes with a slope between 15° and 35° have a cost value between 0.4 and 0.6; steep slopes with a slope greater than 35° have a cost value of approximately 0.8; and areas with significant water accumulation or severe coal slag buildup have a cost value above 0.9 and are marked as prohibited areas.

[0074] Step 2, Reinforcement Learning Path Planning: The robot plans a path based on the initial policy and dynamically adjusts its behavior according to the reward signal during the training process. If it encounters a high-cost area, it will replan the path. When a path conflict occurs, it will cooperate to avoid it according to the priority.

[0075] Step 3, Dynamic Obstacle Response: When a new obstacle is detected, the terrain module updates the cost value in real time, the reinforcement learning algorithm generates a detour path within 0.5 seconds, and the robots share obstacle information through communication to achieve global path optimization;

[0076] Step 4, Knowledge Sharing and System Optimization: The robot stores its successful experiences in a shared pool. Other robots can directly access the experience data in similar scenarios. After 50 training iterations, the system can reach a stable convergence state.

[0077] Example 1:

[0078] This embodiment uses a collaborative detection mission of three heterogeneous robots (parent robot A, variant robot B, and flying robot C) in a collapsed mine area as an example to illustrate the specific implementation process of the present invention. The overall system architecture and workflow diagram are as follows: Figure 1 As shown.

[0079] Step 1: System Initialization and Terrain Cost Modeling. After the system starts, target areas are first assigned to the three robots: the parent robot A is responsible for detecting the area at coordinates (100, 150) meters, the variant robot B is responsible for detecting a narrow passage with a width of 500 mm, and the flying robot C is responsible for detecting the airspace above the collapsed area.

[0080] like Figure 2 As shown, the terrain cost modeling module collects environmental data through multiple sensors mounted on the robot. Specifically: a LiDAR acquires 3D point cloud data to extract slope θ and obstacle height h; a stereo camera uses visual SLAM technology to obtain road surface type information; and an IMU sensor assists in measuring slope changes. The terrain cost function is defined as: Where α=0.3, β=0.2, γ=0.4, and δ=0.1 are weight coefficients obtained through training with historical data; κ is the road surface type coefficient, κ=1.5 for cinder areas, κ=2.0 for waterlogged areas, and κ=1.0 for rocky roads. The system generates a 1m×1m resolution raster map and calculates the value of each cell:

[0081] (1) 35° steep slope area: C=0.3×35 / 90+0.2×0+0.4×1.0+0.1×1 / 3=0.55;

[0082] (2) Water accumulation area: C = 0.3 × 0 + 0.2 × 0 + 0.4 × 2.0 + 0.1 × 1 / 2 = 0.85;

[0083] (3) Gentle rock road surface: C=0.3×0+0.2×0+0.4×1.0+0.1×1 / 2=0.45.

[0084] When C≥0.8, it is marked as a "Caution Zone" to provide a quantitative basis for subsequent route planning.

[0085] Step 2: Multi-agent reinforcement learning path planning. For example... Figure 3 As shown, a multi-agent reinforcement learning framework is constructed, adopting a centralized training-distributed execution architecture.

[0086] The state space S is represented as: Where (x, y, z) are the robot's position coordinates, v is the current velocity, and C envP is the terrain cost matrix within a 5m radius. other The relative positions of other robots (distance ≤ 10m).

[0087] Action space A includes {forward, backward, left turn θ°, right turn θ°, stay}, and the turning angle θ is dynamically adjusted according to speed: , where θ max =45°, k=0.1 is the adjustment coefficient, e is the natural base, and the allowable turning angle is dynamically adjusted according to the robot's current speed k.

[0088] The reward function is designed as follows: ,in, The total reward value obtained from the current action. To explore rewards, Rewards for efficiency As a reward for the goal, As a penalty for collision, As a consequence of terrain, For region overlap penalty, This is a penalty for stagnation. The individual reward / penalty items are defined as follows:

[0089] (1) Exploration Rewards: ;in, As an indicator variable, when the grid cell reached by the robot in this action has not yet been visited by any robot in the current task, =1, otherwise =0;

[0090] (2) Efficiency Rewards: ;in, The value of the current location grid cell output by the aforementioned terrain cost modeling module is the cost of the grid cell. The larger the value, the more difficult it is to pass through.

[0091] (3) Target Rewards: Where d is the Euclidean distance from the robot's current position to the predetermined target point. As the robot gradually approaches the target point, d decreases. Increase accordingly;

[0092] (4) Collision penalty: When the robot collides with an obstacle or another robot, a fixed penalty of 50 is deducted; when there is no collision, this penalty is recorded as 0.

[0093] (5) Cost / Penalty: Where, 𝐶 represents the terrain cost of the current location grid. The higher the cost, the greater the penalty, which is used to prevent robots from frequently entering high-cost areas.

[0094] (6) Overlapping penalty: When different robots occupy the same grid cell within the same time window, a fixed penalty of 3 is deducted; if no position overlap occurs, this penalty is recorded as 0.

[0095] (7) Stagnation penalty: t is the dwell time; where 𝑡 is the duration for which the robot remains stationary or moves at a speed below a preset threshold at its current position. The longer the dwell time, the greater the penalty, which is used to prevent the robot from remaining stationary for a long time.

[0096] The DDPG algorithm is used for policy optimization, and the loss function of the critic network is defined as follows:

[0097] ;

[0098] The target Q value 𝑦 is defined as:

[0099] ;

[0100] The meanings of each symbol are as follows:

[0101] · : The environment state vector at the current moment;

[0102] • 𝑎: In state The next action is selected by the current strategy;

[0103] •r: The immediate reward gained after performing action r in state r;

[0104] · The state transitioned to in the next moment after the action is performed;

[0105] · The current critic value assessment network estimates the reward for the state-action pair (x, y).

[0106] · The parameter vector of the current critic network;

[0107] · The target critic network in the next state and the estimated value of the target actor network output action;

[0108] · : The parameter vector of the target actor network;

[0109] · Discount factor, used to weigh current rewards against future rewards;

[0110] · Take the expected value of a small batch of samples obtained from the experience replay pool;

[0111] · The loss function of the critic network reflects the mean square error between the current estimate and the target value.

[0112] critic networks use loss functions Regarding parameters Calculate the gradient and update the parameters using gradient descent to make the estimated value Gradually approaching the target value y.

[0113] The gradient of the actor network policy is approximated as:

[0114] ;

[0115] in:

[0116] · : The action output by the current actor policy network in state 𝑠;

[0117] · : The parameter vector of the current actor network;

[0118] · The gradient of the objective function with respect to the actor parameters, used to update the policy;

[0119] · The critic network outputs the gradient of the action.

[0120] · Other symbols 𝑠, 𝑎, , Consistent with the aforementioned definition.

[0121] actor network via along Perform gradient ascent in the direction of −λ (or equivalently, gradient descent with respect to −λ), and adjust the parameters. This would increase the expected return under the current critic assessment.

[0122] Step 3: Dynamic Conflict Detection and Resolution. Establish a conflict detection mechanism based on relative position and motion trends. When the distance d between two robots is ≤ 2m and their motion paths intersect, conflict resolution is triggered, such as... Figure 4 As shown. Priority weights are calculated as follows:

[0123] ;

[0124] Where λ=0.7 is the task weight coefficient, W task For mission priorities (Life Detection = 10, Environmental Scan = 5, Supply Transport = 3), W path Path efficiency weights:

[0125] ;

[0126] in, Let this be the total length of the path the robot takes from the starting point to the target point under the current plan. This is the remaining path length for the robot from its current position to the target point; when the robot has not yet started performing the task, it has... The path efficiency weight is close to 0 when the robot is close to completing the task. Significantly smaller than The path efficiency weight is close to 10, therefore The value range is [0, 10].

[0127] For robots with low overall priority weights, their avoidance trajectory is defined in the local coordinate system as follows:

[0128] ;

[0129] ;

[0130] Where Δx,y≤1m is the maximum offset, ω=πrad / s, and the avoidance time t≤2s.

[0131] • 𝑥, 𝑦 are the current position coordinates of the robot in the local coordinate system during collision detection;

[0132] · , The instantaneous position coordinates at time t during the avoidance maneuver;

[0133] · , The maximum offset along the x-axis and y-axis is used to control the robot's lateral movement range. In this embodiment, neither of these offsets exceeds 1m. , ;

[0134] · To avoid the angular frequency of the trajectory, the swing speed of the bypass trajectory is controlled. In this embodiment, the angular frequency is selected as... ;

[0135] ·t is the duration variable of the avoidance action, which is timed from the start of the avoidance, and its value range is... The maximum avoidance time After a certain time (s), the robot will stop the avoidance maneuver and return to its original planned path.

[0136] Step 4: Knowledge Sharing and Strategy Transfer. Construct a distributed knowledge sharing mechanism, such as... Figure 5 As shown, the storage format of empirical tuples is:

[0137] ;

[0138] Where s is the state, a is the action, r is the reward, s' is the next state, C is the terrain cost, and t is the timestamp.

[0139] The shared experience pool employs a priority experience replay mechanism, and the sampling probability for the nth experience sample is defined as follows:

[0140] ;

[0141] in:

[0142] ;

[0143] The meanings of each symbol are as follows:

[0144] · The time difference error (TD error) corresponding to the nth empirical sample reflects the magnitude of the deviation of the current value network from the estimation of that empirical sample;

[0145] · The priority of the nth empirical sample is equal to the absolute value of the TD error plus a small positive constant.

[0146] · To prevent smoothing terms with a priority of zero, this embodiment takes... ;

[0147] · Priority amplification factor, used to adjust the degree of influence of TD error on sampling probability. In this embodiment, it is taken as... ,when When the value approaches 0, it degenerates into uniform random sampling. As 𝛼 increases, the probability of high TD error samples being sampled increases significantly.

[0148] · The probability of selecting the nth experience sample from the shared experience pool;

[0149] · Prioritize all experience samples in the experience pool The normalization factor ensures that the sum of the sampling probabilities of all samples is 1.

[0150] The above design allows "important experiences" with larger TD errors to be used more frequently for training, thereby accelerating network convergence.

[0151] The knowledge distillation process is achieved by minimizing the following loss function:

[0152] ;

[0153] The meanings of each symbol are as follows:

[0154] · Total loss from knowledge distillation;

[0155] · Constraining Student Network Output Based on Hard-Labeled Cross-Entropy Loss Fitting teacher network output Or, the task's true label;

[0156] · Kullback–Leibler divergence loss based on soft labels is used to measure the probability distribution of softening in student networks. Probability distribution of teacher network softening The differences between them;

[0157] · The raw output of the teacher network given an input (such as Q-score or category score);

[0158] · The original output of the student network under the same input;

[0159] · The softening probability distribution is obtained by dividing the teacher network output by the temperature parameter φ and then normalizing it using softmax.

[0160] · The softening probability distribution is obtained by dividing the student network output by the temperature parameter φ and then normalizing it using softmax.

[0161] •T: Temperature parameter, used to control the degree of softening. In this embodiment, T=3 is taken. The higher the temperature, the smoother the probability distribution, which is beneficial to the relative preference relationship between students' online learning and teachers' online learning.

[0162] · : Balance coefficient, used to adjust the relative weights of cross-entropy loss and KL divergence loss in the total loss. In this embodiment, we take 𝜆=0.5.

[0163] Through the above loss design, the student network maintains the accuracy of the task objective through the cross-entropy term, and fully inherits the teacher network's knowledge of action preferences through the KL divergence term, thereby achieving decision performance close to that of the teacher network with fewer parameters.

[0164] The present invention and its embodiments have been described above. This description is not restrictive, and the embodiments shown throughout are only one of the embodiments of the present invention. The actual structure is not limited to this. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A mine robot cooperative path planning method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: (1) The terrain cost modeling module constructs a refined environmental characterization based on multi-source sensor information and calculates the terrain cost value; (2) Path planning is performed using a centralized training-distributed execution mechanism through a multi-agent reinforcement learning framework; (3) Real-time monitoring and resolution of motion conflicts between robots through a dynamic conflict resolution module; (4) Cross-robot knowledge transfer and strategy transfer are achieved through knowledge sharing units.

2. The cooperative path planning method for mining robots based on multi-agent reinforcement learning according to claim 1, characterized in that: The terrain cost modeling module specifically includes: extracting slope using LiDAR and cameras. Obstacle height Road surface type and traffic width Four types of terrain feature parameters; Setting high cost terrain condition as slope ≥ 35°, obstacle height ≥ 300mm or passage width ≤ 800mm; Establish terrain cost function , determined by historical data training, meet α+β+γ+δ=1 and α, β, γ, δ are greater than 0.

3. The mine robot cooperative path planning method based on multi-agent reinforcement learning according to claim 2, characterized in that: κ is the road cost coefficient, with κ=1.5 for the cinder area, κ=2.0 for the waterlogged area, and κ=1.0 for the rock area. α, β, γ, and δ represent the slope factor, obstacle height factor, road surface type factor, and traffic width factor, respectively.

4. The multi-agent reinforcement learning based mine robot cooperative path planning method according to claim 2, characterized in that: The historical data comes from the mine environment simulation platform and real mine test data, including sensor readings and passage cost labels under different terrain scenarios such as steep slopes, water accumulation, coal slag accumulation and gentle rock road surfaces. The passage cost label has a value range of [0,1]. In the raster map output by the cost function, the cost value C of each cell is used for path evaluation. The terrain is divided into four categories according to the cost value C: 0-0.3 is a safe and passable area, 0.3-0.6 is a cautious area, 0.6-0.8 is a high-risk area, and 0.8-1.0 is a prohibited area.

5. The cooperative path planning method for mining robots based on multi-agent reinforcement learning according to claim 1, characterized in that: The multi-agent reinforcement learning framework includes: The state space S includes the robot's own position, velocity, surrounding terrain cost, and other robot position information; The action space A includes forward movement, backward movement, left turn, right turn, and stopping, and its range is dynamically adjusted according to speed. reward function , ； in, The total reward value obtained from the current action. To explore rewards, Rewards for efficiency As a reward for the goal, As a penalty for collision, As a consequence of terrain, For region overlap penalty, As a penalty for stagnation; The central controller is used to collect all robot experience data and optimize the global policy based on the deep deterministic policy gradient algorithm. Each robot relies on local observations to achieve distributed decision-making during the execution phase.

6. The cooperative path planning method for mining robots based on multi-agent reinforcement learning according to claim 5, characterized in that: ;in, As an indicator variable, when the grid cell reached by the robot in this action has not yet been visited by any robot in the current task, =1, otherwise =0; ;in, The value of the current location grid cell output by the aforementioned terrain cost modeling module; the larger the value, the more difficult it is to pass through. Where d is the Euclidean distance from the robot's current position to the predetermined target point. As the robot gradually approaches the target point, d decreases. Increase accordingly; A fixed penalty of 50 is deducted when the robot collides with an obstacle or another robot; this penalty is recorded as 0 when there is no collision. :in, The terrain cost of the current location grid is used to reduce the penalty for higher costs, thus preventing robots from frequently entering high-cost areas. When different robots occupy the same grid cell within the same time window, a fixed penalty of 3 is deducted; if no position overlap occurs, this penalty is recorded as 0. Where t is the duration for which the robot remains stationary or moves at a speed below a preset threshold at its current position. The longer the stationary time, the greater the penalty, which is used to prevent the robot from remaining stationary for a long time.

7. The cooperative path planning method for mining robots based on multi-agent reinforcement learning according to claim 1, characterized in that: The specific steps of the dynamic conflict resolution module are as follows: When the Euclidean distance between any two robots is ≤2 meters and the predicted paths intersect within a future preset time window, a conflict resolution process is triggered. The priority weight of each robot is calculated based on task priority and remaining path length. Wherein: the task priority is represented by a pre-set discrete level, the task priority of life detection task is recorded as 3, the task priority of environmental scanning task is recorded as 2, and the task priority of material transportation task is recorded as 1. The remaining path length is denoted as The maximum remaining path length of the robots currently participating in the conflict is denoted as . The priority weight is calculated according to the following formula: ; in, The task priority for the nth robot is... These are the weighting coefficients; Compare the priority weights of each robot Robots with higher priority weights gain the right of way and continue along the original path. Robots with lower priority weights temporarily deviate from the current path normal direction by a preset distance to avoid obstacles. The deviation is ≤1 meter and the avoidance time is controlled within 2 seconds. After the avoidance is completed, the robot returns to the original planned path to continue performing the task.

8. The cooperative path planning method for mining robots based on multi-agent reinforcement learning according to claim 1, characterized in that: The knowledge-sharing unit specifically includes: Construct a shared experience pool, a distributed knowledge sharing mechanism, and a storage experience tuple. Where s is the state, a is the action, r is the reward, s' is the next state, C is the terrain cost, and t is the timestamp; The shared experience pool adopts a priority experience replay mechanism, which allocates experience priorities based on time difference error. Experiences with higher priorities have a greater probability of being selected for training. The knowledge distillation technique is used to compress the global strategy of the central controller into a lightweight model and distribute it to each robot; The robot can directly access experience data from the shared experience pool to achieve knowledge reuse.

Images