Multi-agent collaborative hierarchical scheduling method and system based on reinforcement learning driving
By using a reinforcement learning-driven multi-agent collaborative hierarchical scheduling method, the problems of low efficiency and high cost in shared electric bicycle battery recycling have been solved, achieving efficient and flexible battery recycling and task allocation to meet campus safety requirements.
Patent Information
- Application Number
- CN202511618971.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-06
- Publication Date
- 2026-02-13
AI Technical Summary
Existing methods for recycling shared electric bicycle batteries rely on regular manual inspections and fixed routes, which are inefficient, costly, difficult to adapt to dynamic changes in battery power, and lack the flexibility of robotic collaborative operations.
A multi-agent collaborative hierarchical scheduling method based on reinforcement learning is adopted. By constructing a task robot cost matrix and initial allocation using the Hungarian algorithm, path planning is performed by combining deep reinforcement learning and artificial potential field method, thereby realizing dynamic task assignment and path optimization. The path planning model is then integrated for online updates.
It improves the efficiency and flexibility of shared electric bicycle battery recycling, reduces labor costs, ensures efficient and collaborative recycling of low-power batteries in complex environments, and meets campus safety requirements.
Smart Images
Figure CN121525946A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of battery swapping technology for shared electric bicycles, specifically involving a multi-agent collaborative hierarchical scheduling method and system based on reinforcement learning. Background Technology
[0002] With the booming development of the sharing economy, shared electric bicycles have become an important mode of short-distance travel for students and faculty on university campuses. Battery management is a prominent issue affecting their long-term smooth operation. First, the lithium batteries used in shared electric bicycles have limited range; if low-charge batteries are not promptly recycled, a large number of vehicles will become unusable, impacting user experience. Second, traditional manual inspection / battery swapping methods are inefficient and time-consuming, making it difficult to meet the needs of large-scale operations. Furthermore, the need to hire a large number of maintenance personnel increases labor costs. Therefore, how to efficiently and intelligently recycle low-charge batteries has become an urgent problem to be solved.
[0003] Currently, the main method for recycling shared electric bicycle batteries relies on regular manual inspections and replacements. Maintenance personnel follow fixed recycling routes and schedules based on the operating environment (e.g., university campuses). This method is inefficient, labor-intensive, and slow to respond to battery swapping needs. While some applications use robots to perform inspections and battery swaps, they still employ fixed routes and schedules, lacking flexibility and adapting poorly to dynamic changes in battery charge. Furthermore, they fail to consider robot collaboration, potentially leading to uneven task distribution. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of existing technologies, such as the inability to use fixed recycling strategies and paths, lack of flexibility, and uneven task allocation, and to provide a multi-agent collaborative hierarchical scheduling method and system based on reinforcement learning that enables multi-robot collaboration and rapid response.
[0005] The technical solution adopted by this invention to solve its technical problem is:
[0006] As a first aspect, this application provides a multi-agent cooperative hierarchical scheduling method based on reinforcement learning, including the following:
[0007] S1: Construct a campus shared electric bicycle system integrating several mobile robots, and collect parameters of each parking point and basic robot data of each mobile robot in real time based on the campus shared electric bicycle system;
[0008] S2: Based on the collected parking point parameters and robot basic data, construct the task robot cost matrix and solve it using the Hungarian algorithm to obtain the initial task assignment;
[0009] S3: Call the fusion path planning model to output both task-level and path-level strategies for the mobile robot with assigned tasks;
[0010] S4: Based on the dual output results in step S3, generate hierarchical paths for each mobile robot, optimize dynamic task assignment, and execute the task.
[0011] S5: Periodically acquires the status of the campus shared electric bicycle system and mobile robot, re-triggers the online update of task allocation and global path strategy of the fusion path planning model, and synchronously updates the hierarchical path; and sends the progress information back to the central control platform;
[0012] S6: Perform a return-to-home resupply condition check on the mobile robot. When the mobile robot meets the return-to-home resupply engineering conditions, insert a return-to-home resupply subtask and resume execution after resupply; and update the allocatable mobile robot dataset.
[0013] S7: Perform cyclical scheduling of the mobile robot until the low-power batteries at each parking point are replaced and recycled.
[0014] Specifically, in step S2, obtaining the initial task assignment includes the following:
[0015] Acquire parking point parameters and robot basic data; the parking point parameters include the real-time battery level detection count and the number of low-battery cells (d) at each parking point. j The data includes: geographical location information and congestion information for each parking spot; and basic robot data such as the current location information of each mobile robot, the battery status of each mobile robot, the number of batteries carried by each mobile robot, and the maximum number of batteries that each mobile robot can carry. i ;
[0016] Calculate the path distance and travel time between each parking point by combining parking point parameters and robot basic data;
[0017] Based on the current parking point information for the battery to be replaced, the collected data, and the calculated path distance and travel time between each parking point, a task cost matrix is constructed.
[0018] Based on the constructed task cost matrix, the Hungarian algorithm is used to construct a task allocation model;
[0019] The task allocation model outputs the initial allocation result and the remaining task set T. residual ;
[0020] Each robot starts from the initial path corresponding to the initial allocation result and executes the assigned task.
[0021] Specifically, in step S5, the online update of task allocation and global path strategy includes the following:
[0022] S51: Constructing reinforcement learning states;
[0023] S52: Based on the current state of the mobile robot and the current state of the electric bicycle with the battery to be replaced, construct a set of actions, where each action represents assigning the remaining task at a certain parking point to a certain robot.
[0024] S53: Design the reward function; define the reward function by comprehensively considering task response speed, path cost, and load balancing.
[0025] S54: The scheduling strategy is trained and updated using a fusion path planning model. The fusion path planning model uses a dual-stream deep Q-network, which decomposes Q into a state value function and a dominance function. An ε-greedy strategy selects the optimal action corresponding to the maximum Q value and performs task allocation operations to allocate tasks to the mobile robot.
[0026] S55: Record the current state, action, reward, and next state, and store them in the experience replay pool; sample samples from the experience pool at a fixed frequency to train the neural network; and update the main network and the target network asynchronously.
[0027] Specifically, in step S4, the process of generating hierarchical paths for each mobile robot, optimizing dynamic task assignment, and executing the following steps includes:
[0028] S41: Use a deep reinforcement learning algorithm to output a global path strategy from the current location to the target parking point, and obtain the global path;
[0029] S42: Based on local sensing data, the artificial potential field method is applied to dynamically calculate the attraction and repulsion forces, and adjust the local direction of the global path;
[0030] S43: The dynamic window method is used to evaluate multiple velocity trajectories in real time, and the optimal trajectory is selected based on safety, path deviation and velocity stability.
[0031] S44: The robot begins to move along the planned path according to the selected trajectory, and acquires sensor data in real time to monitor dynamic obstacles in the surrounding area;
[0032] S45: Determine whether to correct the trajectory based on the monitored dynamic obstacles; if a new obstacle appears or the path deviates, return to S402 to recalculate and correct the trajectory; if the current path is not feasible, return to S401 to trigger global path planning.
[0033] S46: The mobile robot reaches the target point T = (x T ,y T When the path planning task is completed, the mobile robot executes the task.
[0034] Specifically, the global path strategy in step S41 includes the following:
[0035] S411: Initialize the starting state of the mobile robot;
[0036] S412: Set the target point T = (x T ,y T That is, the target location that the mobile robot needs to reach;
[0037] S412: Based on the assigned task, a deep reinforcement learning algorithm is used to generate a global path from the initial position to the target point;
[0038] S414: The deep reinforcement learning algorithm estimates the Q-value through a deep neural network and generates the optimal path; the optimal path is the path with the minimum cost from the starting point to the target point for the mobile robot.
[0039] Specifically, adjusting the local direction of the global path in step S42 includes the following:
[0040] S421: Based on the target point position T = (x T ,y T Calculate the target attraction potential field using the robot's current position P = (x, y) and the robot's current position P = (x, y).
[0041]
[0042] In the formula, k att It is a gain factor for the attractiveness of the target, controlling the magnitude of the target's attractiveness;
[0043] S422: For each obstacle O = (x o ,y o ), Calculate the repulsive potential field of the obstacle:
[0044]
[0045] In the formula, d(P,O) is the distance from the robot's current position to the obstacle, and k rep It is the gain factor of the repulsive force, and d0 is the range of influence of the obstacle;
[0046] S423: Combining the attractive force of the target and the repulsive force of the obstacle, the composite potential field at the current position of the mobile robot is obtained:
[0047] U total (P)=U att (P)+U rep (P);
[0048] S424: The mobile robot adjusts its motion direction based on the gradient of the synthetic potential field and local adjustment parameters, that is, adjusts the local direction of the global path. The local adjustment parameters include: k att ∈[0.5,5.0]、k rep ∈[1.0,10.0], and the radius of influence of the obstacle r o ∈[0.5,3.0].
[0049] Specifically, after adjusting the local direction of the global path, when the number of consecutive steps K ≥ 3 for the mobile robot, the angle between the potential field gradient of the mobile robot and the desired direction is greater than φ. th When ∈[60°,120°], a return to global replanning is triggered.
[0050] Specifically, selecting the optimal trajectory in step S43 includes the following:
[0051] S431: At the current moment, calculate the dynamic window of the mobile robot and simultaneously set the allowable speed range parameters and safety constraints; the speed range parameters include the maximum linear velocity v. max Maximum angular velocity w max Maximum acceleration a max angular acceleration α max The upper limit of linear acceleration, the upper limit of angular acceleration, and the minimum safe distance threshold d from obstacles. min ;
[0052] S432: Within the dynamic window, generate multiple candidate trajectories and evaluate the safety and effectiveness of each trajectory;
[0053] S433: For each candidate trajectory, calculate its evaluation function:
[0054]
[0055] In the formula, d g The closer to the target point, the better; S M Represents the smoothness of the path; d o α1, α2, and α3 represent the distance to the obstacle; α1, α2, and α3 are weighting coefficients.
[0056] S434: Select the trajectory with the highest score from the evaluation function, which is the optimal trajectory.
[0057] Specifically, the security constraints include:
[0058] α1, α2, α3∈[0,1] and α1+α2+α3=1;
[0059] And satisfy v≤v max , |ω|≤ω max ;
[0060] The arc length d from the candidate circular arc trajectory to the nearest obstacle obs Greater than vΔt+v 2 / (2a brake )+r robot In the formula, a brake r is the absolute value of the maximum deceleration. robot Let be the equivalent radius of the mobile robot.
[0061] Specifically, the updated assignable mobile robot data information includes the following:
[0062] Set the task assignment conditions that allow task allocation;
[0063] Acquire data information from the full-process monitoring and information synchronization system;
[0064] Determine whether the corresponding mobile robot meets the task allocation conditions to execute the next round of tasks;
[0065] If the task allocation conditions are met, the corresponding mobile robot is added to the allocatable mobile robot dataset and a new task is dynamically allocated from the task pool; if the task allocation conditions are not met, the corresponding mobile robot is routed back to the resupply station and enters the charging and loading state.
[0066] Specifically, the cyclical scheduling of the mobile robot includes the following:
[0067] Detect the execution status of the mobile robot;
[0068] Repeatedly update the assignable mobile robot dataset and dynamically assign tasks to the mobile robots;
[0069] Intelligent path planning and execution for mobile robots.
[0070] Secondly, this application also provides a multi-agent cooperative hierarchical scheduling system based on reinforcement learning, comprising:
[0071] Several environmental perception modules, several positioning modules, and several battery detection systems; each mobile robot is equipped with at least one environmental perception module, at least one positioning module, and at least one battery detection system.
[0072] Several execution units are mounted on the mobile robot; each execution unit includes a chassis drive system and a battery swapping device.
[0073] The central control platform is used for task perception, task allocation, path planning, data storage, dynamic optimization, and cyclic scheduling of mobile robots.
[0074] The network and communication module is used for data interaction between the mobile robot, execution unit, environmental perception module, positioning module, battery detection system and central control platform;
[0075] The campus shared electric bicycle system is used to monitor mobile robot data, parking point data, and vehicle data in real time, and synchronize data between the campus shared electric bicycle system and the central control platform through network and communication modules.
[0076] Specifically, the central control platform includes: a task awareness module, a task allocation and scheduling module, a fusion path planning module, a data storage module, and a historical data analysis and optimization module.
[0077] Specifically, it also includes an exception handling module and an execution task monitoring status module.
[0078] The beneficial effects of the multi-agent cooperative hierarchical scheduling method and system based on reinforcement learning driven by the present invention are as follows:
[0079] This invention integrates mobile robots into the existing campus shared electric bicycle system and collects parameters from each parking point and basic robot data from each mobile robot in real time. Based on the collected data, a task robot cost matrix is constructed. The Hungarian algorithm is used for initial task allocation of low-battery replacement tasks, ensuring the optimality of the initial allocation in a static scenario. A fusion path planning model is then deployed to output both task-level and path-level strategies. Internally, a reinforcement learning model generates hierarchical paths for each mobile robot, dynamically adjusting and optimizing the assignment of low-battery replacement tasks to ensure optimal allocation for each task. This guarantees that several mobile robots can efficiently and collaboratively recover low-battery batteries within a fixed time, improving the utilization efficiency of shared electric bicycles and reducing labor costs while meeting the safety requirements of campus operation. Furthermore, the system periodically acquires the status of the campus shared electric bicycle system and mobile robots, re-triggering the fusion path planning model to update task allocation and global path strategies online, as well as synchronously updating hierarchical paths. This avoids efficiency losses caused by the disconnect between initial allocation and real-time scenario, further ensuring the real-time performance, flexibility, and optimality of battery swapping tasks, and guaranteeing the long-term stable and efficient operation of the campus shared electric bicycle system.
[0080] The fusion path planning model of this invention uses a deep reinforcement learning algorithm to output the globally optimal path, applies APF to dynamically calculate attraction and repulsion, and makes local adjustments to the global path to avoid static and dynamic obstacles. Then, it uses DWA for collision detection and correction to ensure that the robot can adjust its direction and speed in a timely manner during operation to cope with complex and dynamic environments. The fusion path planning and real-time dynamic obstacle avoidance, which combines reinforcement learning model, artificial potential field method and dynamic window method, has high obstacle avoidance capability and optimizes the mutual cooperation of mobile robots based on dynamically adjusting the path of the mobile robot. Attached Figure Description
[0081] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0082] Figure 1 This is a schematic diagram of the structure of a multi-agent collaborative hierarchical scheduling system based on reinforcement learning, according to an embodiment of the present invention.
[0083] Figure 2 This is the overall flowchart of the multi-agent collaborative hierarchical scheduling system in this embodiment of the invention.
[0084] Figure 3 This is a flowchart illustrating the steps of a multi-agent collaborative hierarchical scheduling method based on reinforcement learning, according to an embodiment of the present invention.
[0085] Figure 4 This is a flowchart illustrating the specific implementation of the scheduling method in this embodiment of the invention.
[0086] Figure 5 This is a flowchart of the initial task allocation in an embodiment of the present invention.
[0087] Figure 6 This is a flowchart of dynamic task scheduling optimization in an embodiment of the present invention.
[0088] Figure 7 This is a flowchart of the path planning process for the algorithm fusion model in this embodiment of the invention.
[0089] In the diagram: 1. Environmental perception module, 2. Positioning module, 3. Battery detection system, 4. Execution unit, 5. Central control platform, 51. Task perception module, 52. Task allocation and scheduling module, 53. Fusion path planning module, 54. Data storage module, 55. Historical data analysis and optimization module, 6. Network and communication module, 7. Campus shared electric bicycle system, 8. Anomaly handling module, 9. Execution task monitoring status module. Detailed Implementation
[0090] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0091] like Figure 1 and Figure 2 The multi-agent collaborative hierarchical scheduling system based on reinforcement learning driven by the present invention includes: several environmental perception modules 1, several positioning modules 2, several battery detection systems 3, several execution units 4, a central control platform 5, a network and communication module 6, and a campus shared electric bicycle system 7; each mobile robot is equipped with at least one environmental perception module 1, at least one positioning module 2, at least one battery detection system 3, and at least one execution unit 4 for battery replacement.
[0092] Several execution units 4 are mounted on the mobile robot, and each execution unit 4 includes a chassis drive system and a battery swapping device. The central control platform 5 is used for task perception, task allocation, path planning, data storage, dynamic optimization, and cyclic scheduling of the mobile robot. The network and communication module 6 is used for data interaction between the mobile robot, execution units 4, environmental perception module 1, positioning module 2, battery detection system 3, and the central control platform 5. The campus shared electric bicycle system 7 is used to monitor mobile robot data, parking point data, and vehicle data in real time, and synchronizes data between the campus shared electric bicycle system 7 and the central control platform 5 through the network and communication module 6.
[0093] like Figure 1 As shown, the central control platform 5 includes: a task awareness module 51, a task allocation and scheduling module 52, a fusion path planning module 53, a data storage module 54, and a historical data analysis and optimization module 55. It should be further noted that the fusion path planning module 53 in this embodiment adopts a dual-stream deep Q-network.
[0094] The scheduling system in this embodiment also includes an exception handling module 8 and a task execution monitoring status module 9. The exception handling module 8 is used to detect whether the assigned mobile robot needs replacement and to detect the battery demand at the parking point. When the demand exceeds the assigned mobile robot's load capacity, it assigns another robot; when the demand is less than the assigned mobile robot's load capacity, the task continues. See [link to relevant documentation]. Figure 1 and Figure 2 As shown.
[0095] The specific process of the scheduling system in this embodiment is as follows: The battery status at the parking point is detected. If the battery status is below a set threshold, a battery swapping request is sent to the central control platform 5. The central control platform 5 collects all battery swapping requests and runs the Hungarian algorithm for initial task allocation. After receiving the task, the robot initiates the collaborative fusion of the D3QN+APF+DWA three algorithms, i.e., the fusion path planning module 53, to construct an integrated path planning system of dynamic perception and intelligent decision-making. After the mobile robot arrives at the parking point, it swaps the battery and recovers the low-power battery. After the task is completed, the robot uploads data to the central control platform 5 for status evaluation. If the battery level is below the threshold or the robot is fully loaded, it returns to the robot's central station. If not, it monitors the task status in real time to see if any new tasks are added. If not, a task ends. If so, it performs a task urgency assessment, introduces constraint optimization, and enables deep reinforcement learning for dynamic adjustment.
[0096] like Figures 3 to 7 As shown, a specific embodiment of the scheduling method for the multi-agent cooperative hierarchical scheduling system based on reinforcement learning includes the following:
[0097] S1: Construct a campus shared electric bicycle system that integrates several mobile robots, and collect parameters of each parking point and basic robot data of each mobile robot in real time based on the campus shared electric bicycle system;
[0098] S2: Based on the collected parking point parameters and robot basic data, construct the task robot cost matrix and solve it using the Hungarian algorithm to obtain the initial task assignment;
[0099] S3: Call the fusion path planning model to output both task-level and path-level strategies for the mobile robot with assigned tasks;
[0100] S4: Based on the dual output results in step S3, generate hierarchical paths for each mobile robot, optimize dynamic task assignment, and execute the task.
[0101] S5: Periodically acquires the status of the campus shared electric bicycle system and mobile robot, re-triggers the online update of task allocation and global path strategy of the fusion path planning model, and synchronously updates the hierarchical path; and sends the progress information back to the central control platform;
[0102] S6: Perform a return-to-home resupply condition check on the mobile robot. When the mobile robot meets the return-to-home resupply engineering conditions, insert a return-to-home resupply subtask and resume execution after resupply; and update the allocatable mobile robot dataset.
[0103] S7: Perform cyclical scheduling of the mobile robot until the low-power batteries at each parking point are replaced and recycled.
[0104] The parking point parameters in step S1 include the real-time battery level detection count and the number of low-battery batteries (d) at each parking point. j The data includes: geographical location information and congestion information for each parking spot; basic robot data including the current location of each mobile robot, its battery status, the number of batteries it carries, and the maximum number of batteries it can carry. i .
[0105] In step S2 of this embodiment, obtaining the initial task assignment includes the following:
[0106] S21. Acquire the collected parking point parameters and robot basic data; parking point parameters include the real-time battery level detection count and the number of low-battery cells at each parking point. j The data includes: geographical location information and congestion information for each parking spot; basic robot data including the current location of each mobile robot, its battery status, the number of batteries it carries, and the maximum number of batteries it can carry. i ;
[0107] S22. Calculate the path distance and travel time between each parking point by combining the parking point parameters and the robot's basic data;
[0108] S23. Based on the current parking point information of the battery to be replaced, the collected data information, and the path distance and travel time between each parking point calculated above, construct the task cost matrix;
[0109] S24. Based on the constructed task cost matrix, the Hungarian algorithm is used to construct a task allocation model;
[0110] S25. The task allocation model outputs the initial allocation results and the remaining task set T. residual ;
[0111] S26. Each robot starts from the initial path corresponding to the initial allocation result and executes the assigned task.
[0112] like Figure 5 As shown, in practical application, the system environment perception module 1 is activated to obtain the location information of all parking points and the number of low-battery batteries (d). j Initial positions of all robots and maximum number of batteries w i The current map information includes static obstacles and path weights. The central control platform 5 calculates the cost matrix C = [C...] between the task and the mobile robot. ij Each element represents the total task cost for robot i to reach parking point j. The cost matrix is calculated considering factors such as the distance between the task and the robot, battery power, and path obstacles.
[0113] Construct a task allocation model: for each d j Parking spot construction requirements >0; each robot can handle a maximum of w i One task; if d j ≤w i Parking point j is completely assigned to robot i; if d j >w i Assign parking spot j to robot i and record the remaining demand (d). j -w i Applying the Hungarian algorithm to solve the minimum total cost task matching problem:
[0114]
[0115] Where x ij =1 indicates that robot i is assigned to task j, otherwise it is 0.
[0116] Then, output the initial allocation results and the remaining task set T. residual This provides input for the subsequent dynamic scheduling phase.
[0117] Finally, each robot sets off according to its assigned path to execute the assigned task. The system monitors the current position, remaining load, and completed task status of each robot in real time through the environmental perception module 1, and receives new low-battery parking points or remaining demands for unfinished tasks.
[0118] In step S3, the fusion path planning model is invoked for the mobile robot with assigned tasks to output both task-level and path-level strategies. That is, the state vector representing the current state of the shared electric bicycle system is input into the dual-stream deep Q-network, and the dual-stream deep Q-network outputs both task-level and path-level actions.
[0119] Specifically, the task-level actions include: assigning decisions to select target tasks / stations for each robot; the path-level actions include: generating a global path strategy for the selected target tasks / stations, the global path strategy including a path skeleton composed of sub-targets / landmarks or its equivalent representation.
[0120] Step S4 generates hierarchical paths for each mobile robot, optimizes dynamic task assignment, and executes the following steps:
[0121] S41: Use a deep reinforcement learning algorithm to output a global path strategy from the current location to the target parking point, and obtain the global path;
[0122] S42: Based on local sensing data, the artificial potential field method is applied to dynamically calculate the attraction and repulsion forces, and adjust the local direction of the global path.
[0123] S43: The dynamic window method is used to evaluate multiple velocity trajectories in real time, and the optimal trajectory is selected based on safety, path deviation and velocity stability.
[0124] S44: The robot begins to move along the planned path according to the selected trajectory, and acquires sensor data in real time to monitor dynamic obstacles in the surrounding area;
[0125] S45: Determine whether to correct the trajectory based on the monitored dynamic obstacles; if a new obstacle appears or the path deviates, return to S42 to recalculate and correct the trajectory; if the current path is not feasible, return to S41 to trigger global path planning.
[0126] S46: The mobile robot reaches the target point T = (x T ,y T When the path planning task is completed, the mobile robot executes the task.
[0127] For details on global path strategies and the process of adjusting the local direction of global paths, please refer to [link / reference]. Figure 7 Specifically, the global path strategy in step S41 includes the following:
[0128] S411: Initialize the initial state of the mobile robot S0 = (x0, y0, v0), where x0 is the initial coordinate of the mobile robot on the X-axis of the map, y0 is the initial coordinate of the mobile robot on the Y-axis of the map, and v0 is the initial velocity of the robot.
[0129] S412: Set the target point T = (x T ,y T This refers to the target location that the mobile robot needs to reach.
[0130] S413: Based on the assigned task, a deep reinforcement learning algorithm is used to generate a global path from the initial position to the target point. At this time, state S t =(x t ,y t ,v t This indicates the robot's current position and speed.
[0131] S414: Deep reinforcement learning algorithms use deep neural networks to estimate Q-values and generate optimal paths; the optimal path is the one with the minimum cost from the starting point to the target point for the mobile robot. Its Q-value function represents the action value function, indicating the expected discounted cumulative reward of performing action a in a given state s. The specific formula is:
[0132]
[0133] In the formula, r t In state S tNext, execute action A t The immediate reward obtained afterward, τ is the discount factor, A t For the current action, A t+1 The action to be performed after the action is executed, θ is the parameter of the online Q-network, θ — Let S be the parameters of the target Q-network. Based on the output of D3QN, the mobile robot selects the optimal path that minimizes the path cost from the starting point S0 to the target point T.
[0134] The adjustment of the local direction of the global path in step S42 includes the following:
[0135] S421: Based on the target point position T = (x T ,y T Calculate the target attraction potential field using the robot's current position P = (x, y) and the robot's current position P = (x, y).
[0136]
[0137] In the formula, k att It is a gain factor for the attractiveness of the target, controlling the magnitude of the target's attractiveness;
[0138] S422: For each obstacle O = (x o ,y o ), Calculate the repulsive potential field of the obstacle:
[0139]
[0140] In the formula, d(P,O) is the distance from the robot's current position to the obstacle, and k rep It is the gain factor of the repulsive force, and d0 is the range of influence of the obstacle;
[0141] S423: Combining the attractive force of the target and the repulsive force of the obstacle, the composite potential field at the robot's current position is obtained:
[0142] U total (P)=U att (P)+U rep (P);
[0143] S424: The mobile robot adjusts its motion direction based on the gradient of the synthetic potential field and local adjustment parameters, that is, it adjusts the local direction of the global path. The formula is as follows:
[0144]
[0145] in, Let be the gradient of the potential field function, representing the direction of the robot's motion.
[0146] It should be further noted that the local adjustment parameters in this embodiment include: katt ∈[0.5,5.0]、k rep ∈[1.0,10.0], and the radius of influence of the obstacle r o ∈[0.5,3.0]. After adjusting the local direction of the global path, when the number of consecutive steps K≥3, the angle between the potential field gradient of the mobile robot and the desired direction is greater than φ. th When ∈[60°,120°], a return to global replanning is triggered.
[0147] Specifically, step S43, selecting the optimal trajectory, includes the following:
[0148] S431: At the current moment, calculate the dynamic window of the mobile robot and simultaneously set the allowable speed range parameters and safety constraints; the speed range parameters include the maximum linear velocity v. max Maximum angular velocity w max Maximum acceleration a max angular acceleration α max The upper limit of linear acceleration, the upper limit of angular acceleration, and the minimum safe distance threshold d from obstacles. min ;
[0149] S432: Within a dynamic window, generate multiple candidate trajectories and evaluate the safety and effectiveness of each trajectory;
[0150] S433: For each candidate trajectory, calculate its evaluation function:
[0151]
[0152] In the formula, d g The closer to the target point, the better; S M Represents the smoothness of the path; d o α1, α2, and α3 represent the distance to the obstacle; α1, α2, and α3 are weighting coefficients.
[0153] S434: Select the trajectory with the highest evaluation function score, which is the optimal trajectory, and use it as the robot's next motion trajectory.
[0154] After the task is completed, the D3QN policy model is updated to generate global paths more efficiently in future path planning tasks. The mobile robot returns to the starting point or other designated location to prepare for new tasks. It should be understood that the safety constraints in this embodiment include:
[0155] (1)α1, α2, α3∈[0,1] and α1+α2+α3=1;
[0156] (2) and satisfy v≤v max , |ω|≤ω max ;
[0157] (3) The arc length d from the candidate circular arc trajectory to the nearest obstacle obs Greater than vΔt+v 2 / (2a brake )+r robot In the formula, a brake r is the absolute value of the maximum deceleration. robot Let be the equivalent radius of the mobile robot. All three constraints must be satisfied simultaneously.
[0158] Specifically, such as Figure 7 As shown, this embodiment combines three algorithms for path planning of the mobile robot: global path planning, local obstacle avoidance, and dynamic trajectory correction. A three-layer structure of "strategy guidance - force field fusion - trajectory optimization" is constructed. D3QN outputs the globally optimal path, APF adjusts the robot path to avoid static and dynamic obstacles, and DWA performs collision detection and correction to ensure that the robot can adjust its direction and speed in a timely manner during operation to cope with complex and dynamic environments.
[0159] The online update task allocation and global path strategy in step S5 of this embodiment includes the following:
[0160] S51: Constructing the deep reinforcement learning state S t Among them, the deep reinforcement learning state S t Including the robot's current position coordinates P i =(x i ,y i ), can carry a low-power battery quantity w i Parking spot task requirement vector [d1, d2, ..., d N Current path cost D i,j And whether it is idle (f) i wait.
[0161] S52: Based on the current state of the mobile robot and the current state of the electric bicycle whose battery needs to be replaced, construct action set A. t Each action represents assigning the remaining tasks at a certain parking spot to a specific robot, that is:
[0162] A t =(i,j)
[0163] Where i is the idle robot number and j is the parking point number to be dispatched;
[0164] S53: Design the reward function;
[0165] Taking into account task response speed, path cost, and load balancing, the reward function R(s,a) is defined as follows:
[0166]
[0167] In the formula, T res This is the response time from when the task is generated to when it is executed; D ij It is the path distance from the selected robot to the target parking point; λ1 represents the current load balance of the robot group; the better the balance, the higher the score. λ1, λ2, and λ3 are weighting coefficients used to adjust the influence weights between task response, path efficiency, and load balance.
[0168] S54: A fusion path planning model is used to train and update the scheduling strategy. The fusion path planning model adopts a two-stream deep Q-network, and the Q-value is decomposed into a state value function V(s) and an advantage function A(s,α). The calculation formula is as follows:
[0169]
[0170] In the formula, V(s; β) represents the state value function; A(s, a; α) represents the advantage function;
[0171] A(s,a ' ; α) represents choosing action a in state s. ' The advantage function, the ε-greedy strategy selects the optimal action corresponding to the maximum Q value, and performs the task allocation operation a. * =argmaxQ(s,a), assigns the task to the mobile robot;
[0172] S55: Record the current state, action, reward, and next state (S t A t ,R t ,S t+1 The samples are stored in the experience replay pool; samples are sampled from the experience pool at a fixed frequency to train the neural network; and the main network and the target network are updated asynchronously.
[0173] The update of the allocatable mobile robot dataset in step S6 includes the following: setting task allocation conditions for task allocation; acquiring data information from the full-process monitoring and information synchronization system; determining whether the corresponding mobile robot has the task allocation conditions to execute the next round of tasks; if the task allocation conditions are met, the corresponding mobile robot is added to the allocatable mobile robot dataset and a new task is dynamically allocated from the task pool; if the task allocation conditions are not met, the corresponding mobile robot is routed back to the resupply station and enters the charging and loading state.
[0174] Step S7, which involves cyclically scheduling the mobile robot, includes the following: detecting the execution status of the mobile robot; repeatedly updating the allocable mobile robot dataset and dynamically assigning tasks to the mobile robot; and performing intelligent path planning and execution for the mobile robot. After scheduling is complete, the target robot performs the battery replacement task according to the path planning.
[0175] Throughout the process, the execution status is continuously monitored, and D3QN scheduling is retried under the following conditions: (1) the task is completed and a new task is waiting to be scheduled; (2) a new low-battery is detected at the parking point; (3) a mobile robot malfunctions or its path is interrupted; (4) the current task response times out. The system continues to repeat the above steps until all low-battery batteries at all parking points are replaced or the task window ends.
[0176] It should be understood that the specific embodiments described above are for illustrative purposes only and are not intended to limit the scope of the invention. Obvious variations or modifications derived from the spirit of the invention are still within the protection scope of the invention.
Claims
1. A multi-agent cooperative hierarchical scheduling method based on reinforcement learning, characterized in that, Includes the following: S1: Construct a campus shared electric bicycle system that integrates several mobile robots, and collect parameters of each parking point and basic robot data of each mobile robot in real time based on the campus shared electric bicycle system; S2: Based on the collected parking point parameters and robot basic data, construct the task robot cost matrix and solve it using the Hungarian algorithm to obtain the initial task assignment; S3: Call the fusion path planning model to output both task-level and path-level strategies for the mobile robot with assigned tasks; S4: Based on the dual output results in step S3, generate hierarchical paths for each mobile robot, optimize dynamic task assignment, and execute the task. S5: Periodically acquires the status of the campus shared electric bicycle system and mobile robot, re-triggers the online update of task allocation and global path strategy of the fusion path planning model, and synchronously updates the hierarchical path; and sends the progress information back to the central control platform; S6: Perform a return-to-home resupply condition check on the mobile robot. When the mobile robot meets the return-to-home resupply engineering conditions, insert a return-to-home resupply subtask and resume execution after resupply; and update the allocatable mobile robot dataset. S7: Perform cyclical scheduling of the mobile robot until the low-power batteries at each parking point are replaced and recycled.
2. The multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven according to claim 1, characterized in that: In step S2, obtaining the initial task assignment includes the following: Acquire parking point parameters and robot basic data; the parking point parameters include the real-time battery level detection count and the number of low-battery cells (d) at each parking point. j The data includes: geographical location information and congestion information for each parking spot; and basic robot data such as the current location information of each mobile robot, the battery status of each mobile robot, the number of batteries carried by each mobile robot, and the maximum number of batteries that each mobile robot can carry. i ; Calculate the path distance and travel time between each parking point by combining parking point parameters and robot basic data; Based on the current parking point information for the battery to be replaced, the collected data, and the calculated path distance and travel time between each parking point, a task cost matrix is constructed. Based on the constructed task cost matrix, the Hungarian algorithm is used to construct a task allocation model; The task allocation model outputs the initial allocation result and the remaining task set T. residual ; Each robot starts from the initial path corresponding to the initial allocation result and executes the assigned task.
3. The multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven according to claim 1, characterized in that: In step S5, the online update of task allocation and global path strategy includes the following: S51: Constructing reinforcement learning states; S52: Based on the current state of the mobile robot and the current state of the electric bicycle with the battery to be replaced, construct a set of actions, where each action represents assigning the remaining task at a certain parking point to a certain robot. S53: Design the reward function; define the reward function by comprehensively considering task response speed, path cost, and load balancing. S54: The scheduling strategy is trained and updated using a fusion path planning model. The fusion path planning model uses a dual-stream deep Q-network, which decomposes Q into a state value function and a dominance function. An ε-greedy strategy selects the optimal action corresponding to the maximum Q value and performs task allocation operations to allocate tasks to the mobile robot. S55: Record the current state, action, reward, and next state, and store them in the experience replay pool; sample samples from the experience pool at a fixed frequency to train the neural network; and update the main network and the target network asynchronously.
4. The multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven according to claim 1, characterized in that, In step S4, the process of generating hierarchical paths for each mobile robot, optimizing dynamic task assignment, and executing the following steps includes: S41: Use a deep reinforcement learning algorithm to output a global path strategy from the current location to the target parking point, and obtain the global path; S42: Based on local sensing data, the artificial potential field method is applied to dynamically calculate the attraction and repulsion forces, and adjust the local direction of the global path; S43: The dynamic window method is used to evaluate multiple velocity trajectories in real time, and the optimal trajectory is selected based on safety, path deviation and velocity stability. S44: The robot begins to move along the planned path according to the selected trajectory, and acquires sensor data in real time to monitor dynamic obstacles in the surrounding area; S45: Determine whether to correct the trajectory based on the monitored dynamic obstacles; if a new obstacle appears or the path deviates, return to S402 to recalculate and correct the trajectory; if the current path is not feasible, return to S401 to trigger global path planning. S46: The mobile robot reaches the target point T = (x T ,y T When the path planning task is completed, the mobile robot executes the task.
5. The multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven according to claim 4, characterized in that, The global path strategy in step S41 includes the following: S411: Initialize the starting state of the mobile robot; S412: Set the target point T = (x T ,y T That is, the target location that the mobile robot needs to reach; S412: Based on the assigned task, a deep reinforcement learning algorithm is used to generate a global path from the initial position to the target point; S414: The deep reinforcement learning algorithm estimates the Q-value through a deep neural network and generates the optimal path; the optimal path is the path with the minimum cost from the starting point to the target point for the mobile robot.
6. The multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven according to claim 4, characterized in that, The adjustment of the local direction of the global path in step S42 includes the following: S421: Based on the target point position T = (x T ,y T Calculate the target attraction potential field using the robot's current position P = (x, y) and the robot's current position P = (x, y). In the formula, k att It is a gain factor for the attractiveness of the target, controlling the magnitude of the target's attractiveness; S422: For each obstacle O = (x o ,y o ), Calculate the repulsive potential field of the obstacle: In the formula, d(P,O) is the distance from the robot's current position to the obstacle, and k rep It is the gain factor of the repulsive force, and d0 is the range of influence of the obstacle; S423: Combining the attractive force of the target and the repulsive force of the obstacle, the composite potential field at the current position of the mobile robot is obtained: IN total (P)=U att (P))+U rep (P)(); S424: The mobile robot adjusts its motion direction based on the gradient of the synthetic potential field and local adjustment parameters, that is, adjusts the local direction of the global path. The local adjustment parameters include: k att ∈[0.5,5.0]、k rep ∈[1.0,10.0], and the radius of influence of obstacles. ro ∈[0.5,3.0].
7. A multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven by claim 6, characterized in that, After adjusting the local orientation of the global path, when the number of consecutive steps K ≥ 3 for the mobile robot, the angle between the potential field gradient of the mobile robot and the desired direction is greater than φ. th When ∈[60°,120°], a return to global replanning is triggered.
8. The multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven according to claim 4, characterized in that, The selection of the optimal trajectory in step S43 includes the following: S431: At the current moment, calculate the dynamic window of the mobile robot and simultaneously set the allowable speed range parameters and safety constraints; the speed range parameters include the maximum linear velocity v. max Maximum angular velocity w max Maximum acceleration a max angular acceleration α max The upper limit of linear acceleration, the upper limit of angular acceleration, and the minimum safe distance threshold d from obstacles. min ; S432: Within the dynamic window, generate multiple candidate trajectories and evaluate the safety and effectiveness of each trajectory; S433: For each candidate trajectory, calculate its evaluation function: In the formula, d g The closer you are to the target point; S M Represents the smoothness of the path; d o α1, α2, and α3 represent the distance to the obstacle; α1, α2, and α3 are weighting coefficients. S434: Select the trajectory with the highest score from the evaluation function, which is the optimal trajectory.
9. A multi-agent cooperative hierarchical scheduling method based on reinforcement learning as described in claim 8, characterized in that, The security constraints include: α1, α2, α3∈[0,1] and α1+α2+α3=1; And satisfy v≤v max , |ω|≤ω max ; The arc length d from the candidate circular arc trajectory to the nearest obstacle obs Greater than vΔt+v 2 / (2a brake )+r robot In the formula, a brake r is the absolute value of the maximum deceleration. robot Let be the equivalent radius of the mobile robot.
10. A scheduling system employing a multi-agent cooperative hierarchical scheduling method based on reinforcement learning driven by any one of claims 1-9, characterized in that, include: Several environmental perception modules, several positioning modules, and several battery detection systems; each mobile robot is equipped with at least one environmental perception module, at least one positioning module, and at least one battery detection system. Several execution units are mounted on the mobile robot; each execution unit includes a chassis drive system and a battery swapping device. The central control platform is used for task perception, task allocation, path planning, data storage, dynamic optimization, and cyclic scheduling of mobile robots. The network and communication module is used for data interaction between the mobile robot, execution unit, environmental perception module, positioning module, battery detection system and central control platform; The campus shared electric bicycle system is used to monitor mobile robot data, parking point data, and vehicle data in real time, and synchronize data between the campus shared electric bicycle system and the central control platform through network and communication modules.