Multi-robot path coordination system based on reinforcement learning

Abstract

The invention discloses a multi-robot path coordination system based on reinforcement learning, and the system comprises a state collection module which is used for collecting the operation state and environment perception information of each robot, and generating an original data set; the state modeling module is used for constructing local state representation for each robot in each decision period; the opponent perception module is used for executing opponent learning perception training and outputting opponent parameter estimation vectors; the improved LOLA module is used for generating a final estimation result on the basis of self-strategy updating of the traditional LOLA; the strategy updating module is used for correcting the updating direction of the reinforcement learning strategy of the local machine and generating a feasible action set; the arbitration decision module is used for executing arbitration solution in combination with the candidate path action distribution to generate a scheduling result; and the execution feedback module is used for issuing the control action to a robot executor for execution, generating a path coordination result and storing the path coordination result. According to the invention, path coordination of multiple robots is realized.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a multi-robot path coordination system based on reinforcement learning. Background Technology

[0002] Multi-robot path coordination technology has long relied on two main approaches: centralized path planning and distributed obstacle avoidance. The former, often based on a global map and unified scheduling, combines grid search, conflict basis search, and priority planning to generate paths in batches. However, it is sensitive to environmental changes and communication latency; once the scheduling cycle is prolonged, bottleneck nodes are prone to congestion and cascading waiting. The latter, often employing speed barriers, interaction force fields, and local rules to achieve on-the-spot yielding, is simple to deploy and responds quickly, but lacks global optimization and cross-node coordination capabilities, easily falling into conservative or oscillating strategies. Recent attempts at multi-agent reinforcement learning have focused on learning coordination strategies in dynamic environments, but they generally treat other robots as environmental noise or static priors, leading to convergence difficulties due to non-stationarity, policy mismatch at bottleneck nodes, and a disconnect between candidate actions and scheduling constraints.

[0003] The existing technologies mainly suffer from the following problems: First, they lack forward-looking modeling and utilization of neighborhood policy updates, making it impossible to perform targeted shaping and priority pre-allocation before congestion occurs; second, centralized solution suffers from high computational pressure, and distributed rules lack global consistency, making it difficult to balance real-time performance and optimality; third, the policy confidence of candidate actions, scheduling priority, and dynamic and safe distance constraints have not formed a computable integrated fusion link, resulting in the inability to stably implement upper-layer learning results; and fourth, the replay experience is mostly based on time series, lacking structured memory and cross-cycle reuse of topological key positions such as bottleneck nodes.

[0004] Therefore, how to provide a multi-robot path coordination system based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a multi-robot path coordination system based on reinforcement learning. By introducing reinforcement learning and opponent perception mechanisms, combined with an improved LOLA algorithm, this invention achieves efficient path coordination for multi-robot systems. At the same time, through the collaborative work of state modeling, opponent learning and perception, policy update, and arbitration decision modules, it can effectively predict and adapt to changes in the policies of neighboring robots, optimize path selection, and improve traffic efficiency.

[0006] A multi-robot path coordination system based on reinforcement learning according to an embodiment of the present invention includes the following steps:

[0007] The status acquisition module is used to collect the operating status and environmental perception information of each robot, perform timestamp alignment, interpolation and denoising on multi-channel data, and generate the original dataset.

[0008] The state modeling module is used to construct local state representations for each robot in each decision cycle based on the original dataset.

[0009] The opponent perception module is used to perform opponent learning perception training based on local state representation, predict changes in the policy parameters of the neighboring robot, and output the opponent parameter estimation vector.

[0010] An improved LOLA module is used to correct the opponent parameter estimation vector and generate the final estimation result based on the traditional LOLA self-policy update and opponent learning perception linkage mechanism.

[0011] The policy update module is used to correct the update direction of the local reinforcement learning policy based on the final estimation results, calculate the distribution of candidate path actions, and generate a set of actionable actions by combining dynamic constraints and safe distance constraints.

[0012] The arbitration decision module is used to combine the local state representations of each robot and the distribution of candidate path actions to perform arbitration solutions, generate scheduling results including priority sequences and release times, and realize multi-robot path coordination and passage scheduling.

[0013] The execution feedback module is used to send verified control actions to the robot actuators for execution, collect execution trajectories and running data, generate path coordination results and store them.

[0014] Optionally, modules can be integrated using the following methods:

[0015] Collect the operating status and environmental perception information of each robot to generate a timestamp-aligned raw dataset;

[0016] In each decision cycle, a local state representation is constructed for each robot based on the original dataset. The local state representation includes its own motion features, neighborhood interaction relationships and local topology, which are used as input for reinforcement learning policies.

[0017] The local state representations of each robot are trained to learn from the opponent's perception, predict the changes in the policy parameters of the neighboring robots, and output the opponent parameter estimation vector.

[0018] An improved LOLA is adopted, which utilizes node-based memory replay to perform hot start and weight reuse on bottleneck nodes identified by local topology, uses sparse approximation for second-order correction, and implements adversary counterfactual rolling on high-impact neighbors for directional shaping when the conflict threshold is triggered, to obtain the final estimation result.

[0019] The update direction of the local reinforcement learning policy is corrected based on the final estimation results, and the candidate path action distribution is generated;

[0020] By combining the local state representations of each robot and the distribution of candidate path actions, an arbitration decision is made and a scheduling result is generated;

[0021] The candidate path action distribution and scheduling results are fused with preset weights, and dynamic constraints and safe distance constraints are combined to generate verified control actions.

[0022] The verified control actions are sent to the robot actuators for execution, the execution trajectory and running data are recorded, and the corresponding path coordination results are generated and stored.

[0023] Optionally, the generation of the original dataset specifically includes:

[0024] Each robot is equipped with a data acquisition channel and a unique identifier to continuously record the operating status and environmental perception raw records. The timestamps of each channel are unified to the same benchmark, a unified sampling sequence is established, and missing and asynchronous segments are interpolated, denoised, and resampled to form time-aligned multi-channel records.

[0025] Based on time-aligned multi-channel records, coordinates of position, attitude, velocity, and acceleration are unified. After unification, drift correction and offset correction are performed to remove outliers and distorted segments. The output is a standardized operating status record containing position, orientation, velocity, and acceleration at each unified sampling time.

[0026] At each unified sampling time, the standardized operating status of each robot and the corresponding original environmental perception records are aggregated into a single-step observation entry. The environmental perception information includes occupancy status, proximity relationships, passage and task constraints, which are archived in chronological order and with robot identifiers, and the original dataset with timestamps aligned is output.

[0027] Optionally, the construction of the local state representation specifically includes:

[0028] Extract the time segment corresponding to the current decision cycle from the time-aligned original dataset, resample and smooth it according to the set sampling interval to eliminate noise and time drift, and obtain continuous and consistent time segments.

[0029] Dynamic features are extracted from the time sequence segments, and key indicators including motion changes, direction shifts and state stability are calculated to reflect the robot's motion trend in the current cycle. At the same time, neighboring objects that frequently interact with the robot are identified, and their relative proximity and mobility coordination are analyzed to form a neighborhood interaction vector.

[0030] Based on environmental connectivity and path conditions, the target robot and its neighbors are organized into a local topology. Connectivity, passage priority, and local constraints are quantized and encoded to form a topology description vector. Dynamic features, neighborhood interaction vectors, and topology description vectors are integrated into a local state representation in a unified format.

[0031] Optionally, the output of the adversary parameter estimation vector specifically includes:

[0032] In the current decision cycle, the local state representation and the corresponding neighborhood set are received, the historical decision records and strategy parameter sequences of each robot in the current neighborhood are retrieved, and the sequences are synchronized in time and verified in data to form a continuous state-action-parameter associated buffer.

[0033] Based on the association buffer, the state and historical parameters of each neighbor are correlated to predict the trend of the strategy parameters in the next decision step. Dynamic weights are assigned according to adjacency and spatial proximity, and the results are recorded as the opponent parameter change estimation record.

[0034] The parameter change estimation records of all neighbors are aggregated according to dynamic weights to form the opponent parameter estimation vector of the target robot. The opponent parameter estimation vector reflects the dynamic coupling relationship between the local robot and the neighborhood at the policy level, ensuring that the learning perception results are applied in real time in subsequent path coordination decisions.

[0035] Optionally, the execution process of the improved LOLA specifically includes:

[0036] The improved LOLA, based on the traditional LOLA's self-policy update and opponent learning perception linkage mechanism, adds three collaborative modules: node-based memory replay, sparse second-order correction, and event-triggered counterfactual rolling, to receive the opponent parameter estimation vector.

[0037] Using the bottleneck node identified by the local topology as the key, the hot start vector and fusion weight recorded when the current node last successfully passed through the memory replay library are read. The opponent parameter estimation vector of the current period is weighted and fused, and decay and correction are performed. If the hit fails, a cold start is performed with zero vector or default weight, and the estimation result of memory replay fusion is obtained.

[0038] A second-order correction with sparse approximation is performed on the estimation result fused by memory playback. A subset of key parameters is selected based on the historical sensitivity and current dynamic weight of each parameter component in the state-action-parameter association buffer, and a sparse mask is generated. The second-order correction amount is calculated only for the selected parameter components, and an upper limit on the magnitude and a change smoothing rule are set to avoid over-adjustment. The correction amount is superimposed on the estimation result to obtain the sparse correction result. At the same time, the parameter index, magnitude and triggering reason involved in the correction are recorded for subsequent auditing and reproduction.

[0039] When the conflict score reaches a preset threshold, the event-triggered counterfactual rolling is initiated. The neighbor with the highest impact coefficient is selected according to the impact ranking. The next parameter change direction of the candidate is generated for the neighbor. The fast evaluator is called to perform single-step rolling on each candidate combination in the one-hop time domain. The comprehensive cost including passage delay, queue growth, minimum safety distance and potential conflict is calculated. The combination with the lowest cost is selected. The sparsity correction result is adjusted in a targeted manner to form the rolling optimization result. The selected neighbors, the number of candidates and the cost details are recorded.

[0040] The rolling optimization results are associated with the target robot identifier, the current decision time, and the bottleneck node key value used, and archived. The hot start vector, fusion weight, and validity period of the key value are updated and written back to the memory replay library, and then aggregated into the final estimation result.

[0041] Optionally, the process of generating the candidate path action distribution specifically includes:

[0042] In the current decision cycle, the final estimation result and local state representation are received, and the local reinforcement learning strategy is invoked. The local reinforcement learning strategy is built on a centralized training and distributed execution architecture. The parameters are trained in a centralized manner at the edge computing node, and the optimized parameters are sent to each robot execution unit. The local policy parameters and the most recent state-action-reward record are read. The instantaneous policy gradient information is calculated according to the expectation of the reinforcement learning objective function, and the final estimation result is mapped to the policy parameter space to form the opponent perception correction vector.

[0043] The policy gradient information and the opponent perception correction vector are weighted and synthesized according to the preset mixing coefficients to obtain the update direction of this cycle. The policy parameters are updated using a fixed learning rate. Gradient clipping and smoothing constraints are applied during the update process to maintain stability. The difference vector, update magnitude and correction source before and after parameter update are recorded.

[0044] Based on the updated strategy parameters and local state representation, a preset set of path actions is scored and ranked. The set of path actions consists of discrete options including path segment selection, speed adjustment, and yielding decisions. Dynamic constraints and safety distance constraints are generated in real time during the calculation process. The dynamic constraints are calculated from the speed, acceleration, and steering angle limits of the current cycle, while the safety distance constraints are determined from the position and orientation information of neighboring robots in the neighborhood set. Based on the dynamic constraints and safety distance constraints, inoperable actions are masked and re-normalized to form a candidate path action distribution.

[0045] Optionally, the process of generating the scheduling result specifically includes:

[0046] In the current decision-making cycle, an arbitration set is defined, and robots that enter the same bottleneck node identified by the local topology are merged and processed. The local state representation and the distribution of candidate path actions are read, and the travel time, energy consumption and conflict risk are calculated for each candidate action and the comprehensive cost is synthesized.

[0047] Set arbitration constraints and selection variables to limit each robot to only one candidate action, control the number of robots passing through the same time slot according to node capacity, prohibit the simultaneous execution of mutually exclusive actions based on conflict risk, configure the order of release and minimum time interval, and verify the satisfaction of dynamic constraints and safety distance constraints;

[0048] Under the premise of satisfying all constraints, the total comprehensive cost within the arbitration set is used as the optimization objective, and the distribution of candidate path actions is superimposed to obtain the adopted actions of each robot, the priority sequence of each bottleneck node and the release time, and the results are organized into scheduling results.

[0049] Optionally, the generation of the verified control action specifically includes:

[0050] Based on the candidate path action distribution and scheduling results, the probability values ​​and priority sequences of the actions in the candidate path action distribution are weighted according to the fusion weight to generate a fusion score list, which is then stored in a one-to-one correspondence with the action number.

[0051] Based on the fusion score, the candidate path actions are sorted, and dynamic constraints and safe distance constraints are checked one by one in sequence. If any constraint is not met, the candidate is discarded, and the next candidate is checked until a passable one is selected.

[0052] The verified candidate actions are converted into control action commands, including path segment selection, speed commands, and steering commands, and time synchronization is completed by aligning with the release time to generate verified control actions.

[0053] Optionally, the generation of the corresponding path coordination result specifically includes:

[0054] The path segment selection, speed command and steering command in the verified control action sequence are synchronized and interpolated in time, aligned to the control cycle of the execution unit, and the issuance plan is arranged according to the batch number and execution window. The issuance is carried out in batches through the actuator interface and the issuance log and timestamp index are generated.

[0055] During the execution of control actions, process monitoring is initiated, and position, speed, attitude and environmental feedback are collected according to the sampling period and written to the trajectory cache in real time. Interpolation and consistency verification are performed on packet loss and abnormal readings. After execution, the trajectory cache and the issued log are aligned by timestamp, distorted segments are removed and a continuous execution trajectory dataset is formed.

[0056] Based on the continuous execution trajectory dataset and local state representation, operational metrics including path deviation, transit time, minimum safe distance, and energy consumption are calculated. The path coordination results are then compiled and archived along with robot identification, decision time, and scheduling results.

[0057] The beneficial effects of this invention are:

[0058] This invention enables robots to acquire comprehensive information about themselves and their neighborhood in real time in dynamic environments by constructing a state acquisition and local state modeling module, which significantly improves the timeliness and accuracy of state perception and provides reliable input for subsequent policy learning.

[0059] This invention adds node-based memory playback, sparse second-order correction, and event-triggered counterfactual rolling modules to the traditional opponent learning perception framework by improving the LOLA algorithm. This enables early prediction and targeted correction of neighborhood policy changes, improving the stability and adaptability of the system in non-stationary environments.

[0060] The reinforcement learning strategy of this invention adopts a centralized training and distributed execution architecture. By performing unified optimization at edge computing nodes and distributing it to each robot, the strategy update has global consistency and local flexibility, which not only ensures computational efficiency but also enhances the generalization ability of learning.

[0061] This invention introduces an arbitration mechanism and a control action generation strategy that integrates dynamic constraints and safe distance constraints in the path planning and scheduling stages, effectively avoiding path conflicts and resource contention. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0063] Figure 1 This is a flowchart of a multi-robot path coordination system based on reinforcement learning proposed in this invention;

[0064] Figure 2 This is a schematic diagram of the algorithm structure of a multi-robot path coordination system based on reinforcement learning proposed in this invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0066] refer to Figure 1-2 A multi-robot path coordination system based on reinforcement learning includes the following steps:

[0067] The status acquisition module is used to collect the operating status and environmental perception information of each robot, perform timestamp alignment, interpolation and denoising on multi-channel data, and generate the original dataset.

[0068] The state modeling module is used to construct local state representations for each robot in each decision cycle based on the original dataset.

[0069] The opponent perception module is used to perform opponent learning perception training based on local state representation, predict changes in the policy parameters of the neighboring robot, and output the opponent parameter estimation vector.

[0070] An improved LOLA module is used to correct the opponent parameter estimation vector and generate the final estimation result based on the traditional LOLA self-policy update and opponent learning perception linkage mechanism.

[0071] The policy update module is used to correct the update direction of the local reinforcement learning policy based on the final estimation results, calculate the distribution of candidate path actions, and generate a set of actionable actions by combining dynamic constraints and safe distance constraints.

[0072] The arbitration decision module is used to combine the local state representations of each robot and the distribution of candidate path actions to perform arbitration solutions, generate scheduling results including priority sequences and release times, and realize multi-robot path coordination and passage scheduling.

[0073] The execution feedback module is used to send verified control actions to the robot actuators for execution, collect execution trajectories and running data, generate path coordination results and store them.

[0074] In this embodiment, the modules are interconnected using the following method:

[0075] Collect the operating status and environmental perception information of each robot to generate a timestamp-aligned raw dataset;

[0076] In each decision cycle, a local state representation is constructed for each robot based on the original dataset. The local state representation includes its own motion features, neighborhood interaction relationships and local topology, which are used as input for reinforcement learning policies.

[0077] Each robot's local state representation is used for opponent learning and perception training to predict changes in the policy parameters of neighboring robots and output the opponent parameter estimation vector.

[0078] An improved LOLA is adopted, which utilizes node-based memory replay to perform hot start and weight reuse on bottleneck nodes identified by local topology, uses sparse approximation for second-order correction, and implements adversary counterfactual rolling on high-impact neighbors for directional shaping when the conflict threshold is triggered, to obtain the final estimation result.

[0079] The update direction of the local reinforcement learning policy is corrected based on the final estimation results, and the candidate path action distribution is generated;

[0080] By combining the local state representations of each robot and the distribution of candidate path actions, an arbitration decision is made and a scheduling result is generated;

[0081] The candidate path action distribution and scheduling results are fused with preset weights, and dynamic constraints and safe distance constraints are combined to generate verified control actions.

[0082] The verified control actions are sent to the robot actuators for execution, the execution trajectory and running data are recorded, and the corresponding path coordination results are generated and stored.

[0083] In this embodiment, the generation of the original dataset specifically includes:

[0084] Each robot is equipped with a data acquisition channel and a unique identifier to continuously record the operating status and environmental perception raw records. The timestamps of each channel are unified to the same benchmark, a unified sampling sequence is established, and missing and asynchronous segments are interpolated, denoised, and resampled to form time-aligned multi-channel records.

[0085] Based on time-aligned multi-channel records, coordinates of position, attitude, velocity, and acceleration are unified. After unification, drift correction and offset correction are performed to remove outliers and distorted segments. The output is a standardized operating status record containing position, orientation, velocity, and acceleration at each unified sampling time.

[0086] At each unified sampling time, the standardized operating status of each robot and the corresponding original environmental perception records are aggregated into a single-step observation entry. The environmental perception information includes occupancy status, proximity relationships, passage and task constraints, which are archived in chronological order and with robot identifiers, and the original dataset with timestamps aligned is output.

[0087] In this embodiment, the construction of the local state representation specifically includes:

[0088] Extract the time segment corresponding to the current decision cycle from the time-aligned original dataset, resample and smooth it according to the set sampling interval to eliminate noise and time drift, and obtain continuous and consistent time segments.

[0089] Dynamic features are extracted from the time sequence segments, and key indicators including motion changes, direction shifts and state stability are calculated to reflect the robot's motion trend in the current cycle. At the same time, neighboring objects that frequently interact with the robot are identified, and their relative proximity and mobility coordination are analyzed to form a neighborhood interaction vector.

[0090] Based on environmental connectivity and path conditions, the target robot and its neighbors are organized into a local topology. Connectivity, passage priority, and local constraints are quantized and encoded to form a topology description vector. Dynamic features, neighborhood interaction vectors, and topology description vectors are integrated into a local state representation in a unified format.

[0091] In this embodiment, the output of the opponent parameter estimation vector specifically includes:

[0092] In the current decision cycle, the local state representation and the corresponding neighborhood set are received, the historical decision records and strategy parameter sequences of each robot in the current neighborhood are retrieved, and the sequences are synchronized in time and verified in data to form a continuous state-action-parameter associated buffer.

[0093] Based on the association buffer, the state and historical parameters of each neighbor are correlated to predict the trend of the strategy parameters in the next decision step. Dynamic weights are assigned according to adjacency and spatial proximity, and the results are recorded as the opponent parameter change estimation record.

[0094] The parameter change estimation records of all neighbors are aggregated according to dynamic weights to form the opponent parameter estimation vector of the target robot. The opponent parameter estimation vector reflects the dynamic coupling relationship between the local robot and the neighborhood at the policy level, ensuring that the learning perception results are applied in real time in subsequent path coordination decisions.

[0095] This invention constructs an association buffer between state, action, and policy parameters, and introduces a dynamic weight aggregation mechanism at the neighborhood level to achieve prediction and quantitative representation of policy changes of neighboring robots. This method can maintain the consistency of policy perception in non-stationary environments where multiple robots interact, and improve the response speed and robustness to the evolution of opponent policies.

[0096] In this embodiment, the execution process of the improved LOLA specifically includes:

[0097] The improved LOLA, based on the traditional LOLA's self-policy update and opponent learning perception linkage mechanism, adds three collaborative modules: node-based memory replay, sparse second-order correction, and event-triggered counterfactual rolling, to receive the opponent parameter estimation vector.

[0098] Using the bottleneck node identified by the local topology as the key, the hot start vector and fusion weight recorded when the current node last successfully passed through the memory replay library are read. The opponent parameter estimation vector of the current period is weighted and fused, and decay and correction are performed. If the hit fails, a cold start is performed with zero vector or default weight, and the estimation result of memory replay fusion is obtained.

[0099] A second-order correction with sparse approximation is performed on the estimation result fused by memory playback. A subset of key parameters is selected based on the historical sensitivity and current dynamic weight of each parameter component in the state-action-parameter association buffer, and a sparse mask is generated. The second-order correction amount is calculated only for the selected parameter components, and an upper limit on the magnitude and a change smoothing rule are set to avoid over-adjustment. The correction amount is superimposed on the estimation result to obtain the sparse correction result. At the same time, the parameter index, magnitude and triggering reason involved in the correction are recorded for subsequent auditing and reproduction.

[0100] When the conflict score reaches a preset threshold, the event-triggered counterfactual rolling is initiated. The neighbor with the highest impact coefficient is selected according to the impact ranking. The next parameter change direction of the candidate is generated for the neighbor. The fast evaluator is called to perform single-step rolling on each candidate combination in the one-hop time domain. The comprehensive cost including passage delay, queue growth, minimum safety distance and potential conflict is calculated. The combination with the lowest cost is selected. The sparsity correction result is adjusted in a targeted manner to form the rolling optimization result. The selected neighbors, the number of candidates and the cost details are recorded.

[0101] The rolling optimization results are associated with the target robot identifier, the current decision time, and the bottleneck node key value used, and archived. The hot start vector, fusion weight, and validity period of the key value are updated and written back to the memory replay library, and then aggregated into the final estimation result.

[0102] This invention introduces node-based memory replay to achieve hot start and weight reuse in bottleneck scenarios, shortening convergence time and improving estimation consistency; it adopts sparse second-order correction to reduce computational overhead and suppress over-adjustment while ensuring decision accuracy; and it uses triggered counterfactual rolling to focus on the neighbor with the highest influence coefficient to achieve directional shaping and resolve congestion and conflict in advance. Under the combined effect, the strategy update is more stable, and the traffic efficiency and safety margin are improved simultaneously.

[0103] In this embodiment, the process of generating the candidate path action distribution specifically includes:

[0104] In the current decision cycle, the final estimation result and local state representation are received, and the local reinforcement learning strategy is invoked. The local reinforcement learning strategy is built on a centralized training and distributed execution architecture. The parameters are trained in a centralized manner at the edge computing node, and the optimized parameters are sent to each robot execution unit. The local policy parameters and the most recent state-action-reward record are read. The instantaneous policy gradient information is calculated according to the expectation of the reinforcement learning objective function, and the final estimation result is mapped to the policy parameter space to form the opponent perception correction vector.

[0105] The policy gradient information and the opponent perception correction vector are weighted and synthesized according to the preset mixing coefficients to obtain the update direction of this cycle. The policy parameters are updated using a fixed learning rate. Gradient clipping and smoothing constraints are applied during the update process to maintain stability. The difference vector, update magnitude and correction source before and after parameter update are recorded.

[0106] Based on the updated strategy parameters and local state representation, a preset set of path actions is scored and ranked. The set of path actions consists of discrete options including path segment selection, speed adjustment, and yielding decisions. Dynamic constraints and safety distance constraints are generated in real time during the calculation process. The dynamic constraints are calculated from the speed, acceleration, and steering angle limits of the current cycle, while the safety distance constraints are determined from the position and orientation information of neighboring robots in the neighborhood set. Based on the dynamic constraints and safety distance constraints, inoperable actions are masked and re-normalized to form a candidate path action distribution.

[0107] This invention achieves global consistency and local adaptive updates of the policy by introducing a reinforcement learning architecture that combines centralized training with decentralized execution. It combines the final estimation results to form an opponent-aware correction vector, enabling the policy update to take into account both its own gains and the influence of its neighborhood. It uses gradient pruning and smoothing constraints to improve training stability. In the action generation stage, it introduces dynamic constraints and safe distance constraints to ensure that the output candidate path actions are both optimal and safe, thereby improving the overall path coordination efficiency and system robustness.

[0108] In this embodiment, the process of generating the scheduling result specifically includes:

[0109] In the current decision-making cycle, an arbitration set is defined, and robots that enter the same bottleneck node identified by the local topology are merged and processed. The local state representation and the distribution of candidate path actions are read, and the travel time, energy consumption and conflict risk are calculated for each candidate action and the comprehensive cost is synthesized.

[0110] Set arbitration constraints and selection variables to limit each robot to only one candidate action, control the number of robots passing through the same time slot according to node capacity, prohibit the simultaneous execution of mutually exclusive actions based on conflict risk, configure the order of release and minimum time interval, and verify the satisfaction of dynamic constraints and safety distance constraints;

[0111] Under the premise of satisfying all constraints, the total comprehensive cost within the arbitration set is used as the optimization objective, and the distribution of candidate path actions is superimposed to obtain the adopted actions of each robot, the priority sequence of each bottleneck node and the release time, and the results are organized into scheduling results.

[0112] In this embodiment, the generation of the verified control action specifically includes:

[0113] Based on the candidate path action distribution and scheduling results, the probability values ​​and priority sequences of the actions in the candidate path action distribution are weighted according to the fusion weight to generate a fusion score list, which is then stored in a one-to-one correspondence with the action number.

[0114] Based on the fusion score, the candidate path actions are sorted, and dynamic constraints and safe distance constraints are checked one by one in sequence. If any constraint is not met, the candidate is discarded, and the next candidate is checked until a passable one is selected.

[0115] The verified candidate actions are converted into control action commands, including path segment selection, speed commands, and steering commands, and time synchronization is completed by aligning with the release time to generate verified control actions.

[0116] In this embodiment, the generation of the corresponding path coordination result specifically includes:

[0117] The path segment selection, speed command and steering command in the verified control action sequence are synchronized and interpolated in time, aligned to the control cycle of the execution unit, and the issuance plan is arranged according to the batch number and execution window. The issuance is carried out in batches through the actuator interface and the issuance log and timestamp index are generated.

[0118] During the execution of control actions, process monitoring is initiated, and position, speed, attitude and environmental feedback are collected according to the sampling period and written to the trajectory cache in real time. Interpolation and consistency verification are performed on packet loss and abnormal readings. After execution, the trajectory cache and the issued log are aligned by timestamp, distorted segments are removed and a continuous execution trajectory dataset is formed.

[0119] Based on the continuous execution trajectory dataset and local state representation, operational metrics including path deviation, transit time, minimum safe distance, and energy consumption are calculated. The path coordination results are then compiled and archived along with robot identification, decision time, and scheduling results.

[0120] Example 1:

[0121] To verify the feasibility of this invention in practice, it was applied to a warehouse picking scenario involving multi-robot collaborative handling in a factory. The work area consists of several main channels and multiple branch aisles, with bottleneck nodes such as elevator entrances, loading and unloading stations, and narrow bridge sections where passage capacity is limited. Centralized one-time path planning is sensitive to environmental disturbances and frequently fails after the plan is issued. Distributed obstacle avoidance strategies are inconsistent in yielding at bottlenecks, resulting in queue jumping and backtracking. Multi-agent learning schemes treat other robots as static noise, leading to the inability to release pressure in advance before congestion forms, resulting in repeated shaking and the accumulation of long queues. The core idea of ​​this invention is to use reinforcement learning as the decision-making backbone, introduce opponent learning perception and structured improvements of improved LOLA, and implement node-based memory playback, sparse second-order correction, and event-triggered counterfactual rolling at the local topology level. At the execution level, the distribution of candidate path actions and arbitration scheduling results are fused according to weights and verified by dynamic constraints and safe distance constraints, ultimately generating control actions that can be directly issued, thereby ensuring both efficiency and safety under high-density traffic.

[0122] The scenario deployment follows a modular system implementation. The perception side uses existing positioning and environmental sensing data as its data source, forming a temporally continuous raw dataset through timestamp alignment and resampling. The state modeling side constructs a local state representation in each decision cycle, including not only individual motion trends but also quantifying the intensity of neighborhood interactions and local topological navigability, making information about congestion points and strong coupling with other robots readable by the strategy. The adversary learning perception side establishes a state-action-parameter association buffer for each robot, continuously predicting the changing trends of neighborhood robot strategy parameters and expressing their influence coefficients with dynamic weights. When a bottleneck node is detected, the improved LOLA side reads the hot-start vector and fusion weights of the node's most recent successful passage, performs weighted fusion of the current adversary parameter estimates, applies attenuation correction, and then only performs second-order corrections on a subset of key parameters. Finally, when the conflict score exceeds a threshold... The system performs single-step counterfactual rolling evaluation and orientation shaping on the neighbor with the highest influence coefficient. The policy update side adopts a centralized training and distributed execution architecture, distributing parameters after unified optimization at edge nodes. It combines the final estimation results to form an adversary perception correction vector, which is weighted with the policy gradient to obtain the update direction, outputting the candidate path action distribution. The arbitration side aggregates candidate actions within the bottleneck area, calculates the comprehensive cost of passage time, energy consumption, and conflict risk, and introduces a confidence penalty from the candidate distribution to solve for priority and release time. The fusion and verification side weights probability weights and priority weights according to a preset ratio, performs feasibility screening of dynamic constraints and safe distance constraints one by one, and generates verified control commands. The execution and recording side issues and monitors commands according to the control cycle, generates path coordination results, and writes them back to the database, providing a basis for subsequent learning and evaluation.

[0123] To demonstrate the beneficial effects, three consecutive business batches from the same operating area were selected for a comparative experiment. The task size and cargo location popularity configuration were kept consistent. All tests were not labeled with specific times and locations, but only distinguished by relative batch numbers. Baseline 1 used centralized path planning with fixed priority release; baseline 2 used a combination of distributed obstacle avoidance and local yielding rules; and baseline 3 used a standard policy gradient multi-agent scheme without competitor learning and perception. The proposed solution operated under the same number of vehicles, the same task injection rate, and the same safety threshold.

[0124] The specific comparison data is shown in Table 1:

[0125] Table 1 Performance Comparison of the Invention System and Baseline Methods

[0126] index Baseline 1: Centralized + Fixed Priority Baseline 2: Distributed Obstacle Avoidance Baseline 3: Standard MARL The present invention solution (batch A) The present invention solution (batch B) The present invention solution (batch C) Global throughput (per unit / hour) 520 548 563 612 618 621 Median order completion time (seconds) 740 705 690 610 604 600 Bottleneck queue length (units) 9.8 8.6 7.9 4.6 4.3 4.2 Average wait time (seconds / bottleneck) 128 116 111 72 70 69 Conflict rate (times / km) 2.4 2.1 1.9 1.1 1.0 1.0 Number of violations (times / hour) 17 14 13 6 6 5 Unit energy consumption (standard value) 1.00 0.96 0.94 0.88 0.87 0.87

[0127] As shown in Table 1, this invention achieves a significant improvement in throughput efficiency while ensuring the effective execution of safety constraints. Compared to the three baselines, the median completion time is reduced by approximately 12% to 17%, the bottleneck queue and average waiting time are reduced by about half, and the conflict rate and number of violations are reduced by more than 40%. The reduction in unit energy consumption indicates a smoother decision chain at the execution level. Combining trajectory playback and log recording, it can also be observed that at key bottlenecks, node-based memory playback helps the strategy achieve a stable release rhythm with fewer control cycles, and event-triggered counterfactual rolling resolves potential conflicts in advance into acceptable yield actions. Sparse second-order correction avoids excessive oscillation of parameters in high-dimensional space, keeping updates directional and controllable. In summary, this invention achieves triple improvements in efficiency, safety, and robustness in high-density, strongly coupled warehousing scenarios, providing a verifiable engineering path for the long-term stable operation of multi-robot systems in real-world business.

[0128] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-robot path coordination system based on reinforcement learning, characterized in that, The method comprises the following steps: a state acquisition module is configured to acquire the running state and environment perception information of each robot, and to generate a timestamp-aligned original data set by time stamp alignment, interpolation and denoising of multi-channel data; a state modeling module is configured to construct a local state representation for each robot based on the original data set at each decision cycle; an opponent perception module is configured to perform opponent learning perception training based on the local state representation, predict the change of the strategy parameter of the neighbor robot, and output an opponent parameter estimation vector; an improved LOLA module is configured to correct the opponent parameter estimation vector and generate a final estimation result based on the self-strategy update and opponent learning perception linkage mechanism of the traditional LOLA; a strategy update module is configured to correct the update direction of the local reinforcement learning strategy according to the final estimation result, calculate a candidate path action distribution, and generate a set of actionable actions in combination with dynamic constraints and safety distance constraints; an arbitration decision module is configured to perform arbitration solving in combination with the local state representation and the candidate path action distribution of each robot, generate a scheduling result containing a priority sequence and a release time, and realize multi-robot path coordination and passing scheduling; an execution feedback module is configured to issue the verified control action to the robot executor for execution, acquire the execution trajectory and running data, generate a path coordination result, and store it.

2. The multi-robot path coordination system based on reinforcement learning according to claim 1, wherein, The modules are realized through the following methods: acquire the running state and environment perception information of each robot, and generate a timestamp-aligned original data set; at each decision cycle, construct a local state representation for each robot based on the original data set; perform opponent learning perception training on the local state representation of each robot, predict the change of the strategy parameter of the neighbor robot, and output an opponent parameter estimation vector; use an improved LOLA, perform hot start and weight reuse using node-based memory replay, use sparse approximation for second-order correction, and perform opponent counterfactual rolling on high-impact neighbors when the conflict threshold is triggered to obtain a final estimation result; correct the update direction of the local reinforcement learning strategy according to the final estimation result, and generate a candidate path action distribution; perform arbitration decision in combination with the local state representation and the candidate path action distribution of each robot, and generate a scheduling result; fuse the candidate path action distribution and the scheduling result according to a preset weight, and generate a verified control action in combination with dynamic constraints and safety distance constraints; issue the verified control action to the robot executor for execution, record the execution trajectory and running data, generate a corresponding path coordination result, and store it.

3. The multi-robot path coordination system based on reinforcement learning according to claim 2, wherein, The generation of the original data set specifically comprises: configure an acquisition channel and a unique identifier for each robot, continuously record running state and environment perception original records, unify the time stamps of all channels to the same reference, establish a unified sampling sequence, interpolate, denoise and resample missing segments and asynchronous segments, and form time-aligned multi-channel records; based on the time-aligned multi-channel records, unify the coordinates of position, attitude, velocity and acceleration, perform drift correction and bias correction after unification, eliminate abnormal points and distorted segments, and output standardized running state records; At each unified sampling time, the normalized running state of each robot and the corresponding environment perception raw record are converged into a single-step observation entry, which is archived in time sequence and with robot identification, and a timestamp-aligned raw data set is output.

4. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The construction of the local state representation specifically includes: A time segment corresponding to the current decision-making period is extracted from the time-aligned raw data set, resampled and smoothed according to the set sampling interval, and noise and time drift are eliminated to obtain a continuous and consistent time segment; Dynamic feature extraction is performed on the time segment, key indicators including motion change, direction transfer and state stability are calculated, and neighboring bodies frequently interacting with the robot are identified, relative proximity and movement coordination are analyzed, and a neighborhood interaction vector is formed; According to the environmental connectivity and path traffic conditions, the target robot and the neighboring bodies are combined into a local topological structure, the connectivity, traffic priority and local constraints are quantitatively coded, and a topological description vector is formed, and the dynamic features, neighborhood interaction vectors and topological description vectors are integrated into the local state representation in a unified format.

5. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The output of the opponent parameter estimation vector specifically includes: In the current decision-making period, the local state representation and the corresponding neighborhood set are received, the historical decision-making records and strategy parameter sequences of each robot in the current neighborhood are called, time synchronization and data verification are performed on each sequence, and a continuous state-action-parameter associated buffer is formed; Based on the associated buffer, the state and historical parameters of each neighbor are analyzed, the trend of the strategy parameter change in the next decision-making step is predicted, and dynamic weights are assigned according to the adjacency relationship and spatial proximity, and the result is registered as the opponent parameter change estimation record; The parameter change estimation records of all neighbors are aggregated according to the dynamic weights to form the opponent parameter estimation vector of the target robot.

6. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The execution process of the improved LOLA specifically includes: The opponent parameter estimation vector is received, and the improved LOLA adds three collaborative modules of node memory playback, sparse second-order correction and event-triggered counterfactual rolling on the basis of the self-strategy updating and opponent learning perception linkage mechanism of the traditional LOLA; Taking the bottleneck node identified by the local topological structure as the key, the hot start vector and fusion weight recorded when the current node successfully passed the last time are read from the memory playback library, the opponent parameter estimation vector of the current period is weighted and fused, and the estimation result of the memory playback fusion is obtained after attenuation and correction; The estimation result of the memory playback fusion is executed by sparse approximate second-order correction, the key parameter subset is selected according to the historical sensitivity of each parameter component in the state-action-parameter associated buffer and the current dynamic weight, the sparse mask is generated, and the second-order correction amount is calculated only for the selected parameter components, and the correction amount is added to the estimation result to obtain the sparse correction result; When the conflict score reaches the preset threshold, the event-triggered opponent counterfactual rolling is started, the neighbor with the highest impact coefficient is selected according to the impact order, the candidate next parameter change direction is generated for the neighbor, the fast evaluator is called to perform single-step rolling on each candidate combination within the one-hop time domain, the comprehensive cost including the through delay, the queue growth, the minimum safety distance and the potential conflict is calculated, the combination with the lowest cost is selected, the directional adjustment is performed on the sparse correction result, the rolling optimization result is formed, and the selected neighbor, the number of candidates and the cost details are recorded; The rolling optimization result is associated with the target robot identifier, the current decision time and the bottleneck node key value used, and is archived, and the hot start vector, the fusion weight and the validity period of the memory playback library are written back to update the key value, and are aggregated into the final estimation result.

7. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The generation process of the candidate path action distribution specifically includes: In the current decision cycle, the final estimation result and the local state representation are received, the local reinforcement learning strategy is called, the local reinforcement learning strategy is constructed based on the centralized training and decentralized execution architecture, the parameter centralized training is completed on the edge computing node, and the optimized parameters are downlink to each robot execution unit, the local policy parameters and the last state-action-reward record are read, the immediate policy gradient information is calculated according to the expected return of the reinforcement learning objective function, and the final estimation result is mapped to the policy parameter space to form the opponent perception correction vector; The policy gradient information and the opponent perception correction vector are weighted and synthesized according to the preset mixing coefficient to obtain the update direction of the current cycle, the policy parameters are updated using a fixed learning rate, gradient clipping and smoothing constraints are applied during the update process to maintain stability, and the difference vector before and after the parameter update, the update amplitude and the correction source are recorded; Based on the updated policy parameters and the local state representation, the preset path action set is scored and sorted, dynamic constraints and safety distance constraints are generated in real time during the calculation process, the dynamic constraints are calculated from the speed, acceleration and steering angle limits of the current cycle, the safety distance constraints are determined from the position and orientation information of the neighboring robots in the neighborhood set, the inexecutable actions are shielded according to the dynamic constraints and the safety distance constraints, and are normalized again to form the candidate path action distribution.

8. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The generation process of the scheduling result specifically includes: In the current decision cycle, the arbitration set is determined, the robots entering the same bottleneck node identified by the local topology structure are merged and processed, the local state representation and the candidate path action distribution are read, the travel time, energy consumption and conflict risk of each candidate action are calculated and combined to form the comprehensive cost; The arbitration constraints and the selection variables are set, it is limited that each robot can only be adopted one candidate action, the number of passing through in the same time slot is controlled according to the node capacity, and the satisfaction of the dynamic constraints and the safety distance constraints is checked; On the premise that all the constraints are met, the adopted action of each robot, the priority sequence and the release time of each bottleneck node are solved by taking the sum of the comprehensive costs in the arbitration set as the optimization target and superimposing the candidate path action distribution, and the scheduling result is arranged.

9. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The generation of the verified control action specifically includes: Based on the candidate path action distribution and the scheduling result, the probability values and priority sequences of the actions of the candidate path action distribution are weighted according to a fusion weight, and a fusion score list is generated; According to the fusion score, the candidate path actions are sorted, and the dynamic constraint and the safety distance constraint are checked one by one in sequence. If any constraint is not met, the candidate is abandoned, and the next candidate is checked until a pass is selected; The candidate action that passes the check is converted into a control action instruction, and is time-synchronized with the release time to complete time synchronization, and a checked control action is generated.

10. The multi-robot path coordination system based on reinforcement learning of claim 2, wherein, The generation of the corresponding path coordination result specifically includes: The path segment selection, speed command and steering command in the checked control action sequence are time-synchronized and interpolated, aligned to the control unit control period, arranged for batch number and execution window, and the plan is issued, and a delivery log and a time stamp index are generated; Process monitoring is started during the control action execution process, position, speed, attitude and environment feedback are collected at a sampling period, and after the execution is completed, the trajectory cache and the delivery log are aligned according to the time stamp, the distorted segment is removed, and a continuous execution trajectory data set is formed; According to the continuous execution trajectory data set and the local state representation, the running index is calculated, the path coordination result is generated, and the robot identifier, decision time and scheduling result are archived together.

Images