Unmanned environmental sanitation fleet balanced scheduling method based on near-end strategy reinforcement learning

Abstract

The invention discloses an unmanned sanitation vehicle fleet balanced scheduling method based on near-end strategy reinforcement learning, and the method comprises the steps: determining a cleaning task set of unmanned sanitation vehicles, and each task comprises the workload and the spatial position; establishing a mixed distance matrix based on the task point coordinates and the warehouse position, loading the node workload, and constructing a neighbor graph; establishing a path planning model based on the optimization target and the limiting condition; according to the number of the unmanned vehicles and the neighbor graph, generating an initial route combination by using an initialization algorithm, and finding an initial optimal route based on a route planning model; and constructing an intelligent agent based on the graph neural network, iteratively selecting a route and adjusting an operator and parameters by using an optimization algorithm, and iterating to a preset number of times to obtain an optimal path satisfying the path planning model. According to the invention, the path of the unmanned sanitation vehicle can be effectively arranged, so that the working efficiency is improved and the cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of path planning technology for unmanned sanitation vehicles, specifically involving a method for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning. Background Technology

[0002] With the rapid development of autonomous driving technology, a large number of unmanned vehicles have emerged in the market, such as unmanned sweepers and unmanned sprinkler trucks. These vehicles play an important role in improving efficiency, reducing costs, improving the working environment, and promoting environmental protection, bringing revolutionary changes to modern urban management and transportation. However, the core bottleneck of collaborative operations lies in the scientific nature of path planning—the path plan directly determines the operational efficiency, resource utilization, and operating costs. It must simultaneously meet multiple constraints such as "full task coverage without duplication," "single vehicle load matching operational capabilities," "multi-vehicle load balancing," and "minimization of maximum operation time." The essence of the problem is the multi-objective, multi-traveling salesman problem (mTSP) in complex scenarios. Traditional technical solutions are difficult to adapt to the core needs of this scenario and have many technical shortcomings that urgently need to be overcome.

[0003] The core shortcomings of existing technologies are concentrated in three aspects: target adaptation, model construction, and optimization execution. In terms of optimization targets, traditional algorithms mostly focus on "minimizing the total operation distance," ignoring the special characteristics of the workload differences between task nodes in sanitation scenarios. Measuring operation costs solely by distance can easily lead to load imbalance among multiple vehicles. Furthermore, they rarely combine "minimizing the maximum operation time" with "load balance" to construct a comprehensive target, which is out of touch with the actual operation and maintenance requirements of "balanced and efficient" operation. In terms of model construction, pure Euclidean distance models cannot adapt to the actual road characteristics between task nodes within the city, resulting in a disconnect between the planned path and the working conditions. At the same time, traditional state representation methods such as fully connected networks are difficult to capture the spatial correlation of nodes and the topological features of routes, resulting in a serious lack of perception accuracy in large-scale scenarios.

[0004] In summary, existing technologies fall short of meeting the core requirements of large-scale collaborative operations of unmanned sanitation vehicles in key dimensions such as target adaptability, model construction accuracy, and execution efficiency. Developing a balanced scheduling method for unmanned sanitation vehicle fleets that is "tailored to the characteristics of sanitation scenarios, considers multi-objective optimization, achieves high convergence efficiency, and is feasible in planning" has become crucial for overcoming the technical bottlenecks of intelligent sanitation collaborative operation and maintenance and improving the quality of urban sanitation services. This method is also the core driving force behind this invention. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method for balanced scheduling of unmanned sanitation vehicle fleets based on near-end policy reinforcement learning.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, this invention discloses a method for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning, comprising:

[0008] Step S1: Determine the number of unmanned sanitation vehicles and the set of cleaning tasks, with each task including workload and spatial location;

[0009] Step S2: Establish a hybrid distance matrix based on the task point coordinates and warehouse location, load the node workload, and construct a k-nearest neighbor graph;

[0010] The warehouse location is fixed at (0,0), and the workload is 0;

[0011] The hybrid distance matrix is ​​as follows: the distance between the warehouse and the task node is calculated using Euclidean distance, and the distance between task nodes is calculated using real-world data, i.e., the actual road distance between the corresponding task points.

[0012] The hybrid distance matrix needs to be symmetricized, and the diagonal elements should be set to 0.

[0013] Step S3: Establish a path planning objective function based on load balancing, task non-duplication, and minimizing the maximum operation time of unmanned sanitation vehicles;

[0014] Step S4: Based on the k-nearest neighbor graph, use a clustering algorithm to cluster the task nodes, generate an initial closed path, and then perform local optimization to find the initial optimal route based on the objective function.

[0015] Step S5: Construct a PPO agent based on graph neural network, use optimization algorithm to iteratively select routes and adjust operators and parameters, iterate until a preset number of times to obtain the optimal path that satisfies the objective function;

[0016] The The intelligent agent includes a graph neural network encoder and four functional heads;

[0017] The strategy head is for the set of algorithms. Make decisions;

[0018] The parameter header generates the source node, target route, and insertion position for the inter_reloc and inter_swap operators;

[0019] Value head, used for state value prediction;

[0020] Auxiliary head, used for predicting normalization ,in This represents the change in the maximum operating time of the route. This represents the change in the standard deviation of the route operation time.

[0021] Based on the above technical solution, the following improvements can be made:

[0022] As a preferred option, the objective function in step S3 is as follows:

[0023] ;

[0024] in , The number of unmanned sanitation vehicles, i.e. For each unmanned sanitation vehicle, the route sequence is defined. for The longest path length in the middle, for Standard deviation of the sequence This is the adjustment coefficient.

[0025] As a preferred option, the total load on the route is as follows:

[0026] ;

[0027] in For a single vehicle, the closed path is: the vehicle starts from the warehouse, serves several task points, and finally returns to the warehouse; It is the set of routes taken by unmanned sanitation vehicles; For path The set of all node pairs in the set; for Adjacent node pairs within; Mixed distance matrix Middle node , The distance between; For path The path after removing the repository node; For nodes The workload.

[0028] As a preferred embodiment, step S4 includes:

[0029] Step S4.1: Use a clustering algorithm to segment the task nodes into... Clusters, The number of unmanned sanitation vehicles;

[0030] Step S4.2: Construct the nearest neighbor heuristic path for each cluster;

[0031] Step S4.3: Optimize the local path using the 2-opt and Or-opt algorithms;

[0032] Step S4.4: From all initial route combinations, select the one with the smallest objective function value as the initial optimal solution. .

[0033] As a preferred embodiment, step S5 includes:

[0034] Step S5.1: Construct state features and statistical information, and calculate operator preferences by combining them with operator credit values;

[0035] Step S5.2: Select the operator according to the round and sample the corresponding parameters;

[0036] Step S5.3: Calculate the old statistics for the current route, apply the selected operator and parameters to calculate the new statistics, compare the two and decide whether to adjust the route;

[0037] Step S5.4: Calculate dictionary rewards and update operator credit based on the success rate adjustment of the operator's route;

[0038] Step S5.5: Route status update: Update the current route R to the adjusted route R'. If the objective function value corresponding to the adjusted route is better, then update the optimal route, the optimal objective function value and the corresponding number of routes.

[0039] Step S5.6: Periodically perform route reinforcement and diversity optimization operations to avoid the algorithm getting stuck in local optima;

[0040] Step S5.7: Update the parameters of the PPO agent using the update algorithm;

[0041] Step S5.8: Used to determine whether the preset maximum number of iterations has been reached;

[0042] If so, return the optimal path;

[0043] Otherwise, repeat steps S5.1-S5.7.

[0044] As a preferred embodiment, step S5.7 includes:

[0045] Step S5.7.1: Determine the input data;

[0046] Step S5.7.2: Calculate the generalized advantage estimate and set the target return;

[0047] Step S5.7.3: Update agent parameters through multiple rounds of mini-batch iteration: In each mini-batch, recalculate the log probability of actions, policy entropy, state value prediction, and auxiliary head prediction results; calculate the probability ratio, calculate the policy loss based on the probability ratio and the normalized generalized advantage, and construct the total loss by combining the value loss, auxiliary loss, and policy entropy; update agent parameters using the Adam optimizer and perform gradient pruning.

[0048] As a preferred embodiment, the sampling method for inter_reloc and inter_swap in step S5 is as follows:

[0049] Step S5.2.1: Determine the input data;

[0050] Step S5.2.2: Select the longest route in the current route set as the source route, and sample the non-warehouse nodes in the source route as operation nodes;

[0051] Step S5.2.3: Sample a route different from the source route as the target route, and perform the corresponding operation according to the target operator type to obtain the parameters.

[0052] Secondly, this invention discloses an unmanned sanitation vehicle fleet equalization scheduling device based on near-end policy reinforcement learning, comprising:

[0053] The task set determination module is used to determine the operating areas of several unmanned sanitation vehicles.

[0054] The data initialization module is used to establish a hybrid distance matrix based on task point coordinates and warehouse location, load node workload, and construct a k-nearest neighbor graph;

[0055] The warehouse location is fixed at (0,0), and the workload is 0;

[0056] The hybrid distance matrix is ​​as follows: the distance between the warehouse and the task node is calculated using Euclidean distance, and the distance between task nodes is calculated using real-world data, i.e., the actual road distance between the corresponding task points.

[0057] The hybrid distance matrix needs to be symmetricized, and the diagonal elements should be set to 0.

[0058] The objective function establishment module is used to establish the path planning objective function based on load balancing, task non-duplication, and minimizing the maximum operation time of unmanned sanitation vehicles.

[0059] The initial path generation module is used to generate initial route combinations based on the k-nearest neighbor graph using an initialization algorithm, and find the initial optimal route based on the objective function value.

[0060] The path generation module constructs a PPO agent based on a graph neural network, uses an optimization algorithm to iteratively select routes and adjust operators and parameters, and iterates to a preset number of times to obtain the optimal path that satisfies the objective function.

[0061] The The intelligent agent includes a graph neural network encoder and four functional heads;

[0062] The strategy head is for the set of algorithms. Make decisions;

[0063] The parameter header generates the source node, target route, and insertion position for the inter_reloc and inter_swap operators;

[0064] Value head, used for state value prediction;

[0065] Auxiliary head, used for predicting normalization ,in This represents the change in the maximum operating time of the route. This represents the change in the standard deviation of the route operation time.

[0066] As a preferred option, the objective function in the objective function establishment module is as follows:

[0067] ;

[0068] in , The number of unmanned sanitation vehicles, i.e. For each unmanned sanitation vehicle, the route sequence is defined. for The longest path length in the middle, for Standard deviation of the sequence This is the adjustment coefficient.

[0069] As a preferred option, the total load on the route is as follows:

[0070] ;

[0071] in For a single vehicle, the closed path is: the vehicle starts from the warehouse, serves several task points, and finally returns to the warehouse; It is the set of routes taken by unmanned sanitation vehicles; For path The set of all node pairs in the set; for Adjacent node pairs within; Mixed distance matrix Middle node , The distance between; For path The path after removing the repository node; For nodes The workload.

[0072] As a preferred embodiment, the initial path generation module includes:

[0073] Clustering initialization unit, used to divide task nodes into Clusters, The number of routes for unmanned sanitation vehicles;

[0074] The path initialization unit constructs the nearest neighbor heuristic path for each cluster;

[0075] The path optimization unit uses the 2-opt and Or-opt algorithms to perform local path optimization.

[0076] The path selection unit is used to select the route with the smallest objective function value from all initial route combinations as the initial optimal route.

[0077] As a preferred solution, the path generation module includes:

[0078] The state initialization unit constructs state features and statistical information, and calculates operator preferences by combining operator credit values;

[0079] The operator selection unit is used to select operators according to the round and sample the corresponding parameters;

[0080] The route adjustment unit is used to calculate the old statistics of the current route, apply the selected operators and parameters to calculate the new statistics, compare the two and decide whether to adjust the route.

[0081] The reward calculation unit is used to calculate dictionary rewards and update operator credits based on the success rate adjustment of the operator's route.

[0082] The route status update unit is used to update the current route to the adjusted route. If the objective function value corresponding to the adjusted route is better, the optimal route, the optimal objective function value, and the corresponding number of routes are updated.

[0083] The optimization unit is used to periodically perform route reinforcement and diversity optimization operations to prevent the algorithm from getting trapped in local optima.

[0084] The parameter update unit is used to update the parameters of the PPO agent.

[0085] The algorithm termination judgment unit is used to determine whether the preset maximum number of iterations has been reached.

[0086] If so, return the optimal path;

[0087] Otherwise, the methods within the state initialization unit, operator selection unit, route adjustment unit, reward calculation unit, route state update unit, and optimization unit are executed sequentially.

[0088] As a preferred embodiment, the parameter update unit in the path generation module includes:

[0089] A data input unit is used to determine the input data;

[0090] The objective determination unit is used to calculate the generalized advantage estimate and set the objective reward;

[0091] The parameter update execution unit is used to update the agent parameters through multiple rounds of mini-batch iterations: in each mini-batch, the log probability of the action, policy entropy, state value prediction value, and auxiliary head prediction result are recalculated; the probability ratio is calculated, and the policy loss is calculated based on the probability ratio and the normalized generalized advantage; the total loss is constructed by combining the value loss, auxiliary loss, and policy entropy; the agent parameters are updated using the Adam optimizer, and gradient pruning is performed.

[0092] As a preferred embodiment, the sampling of inter_reloc and inter_swap in the path generation module includes:

[0093] A data input unit is used to determine the input data;

[0094] The route selection unit is used to select the longest route in the current route set as the source route and sample the non-warehouse nodes in the source route as operation nodes.

[0095] The parameter sampling unit is used to sample routes different from the source route as target routes, and perform corresponding operations according to the target operator type to obtain parameters.

[0096] Furthermore, in a third aspect, the present invention discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed any of the above-described methods for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning.

[0097] This invention discloses a method for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning, which has the following beneficial effects:

[0098] First, this invention constructs a comprehensive objective function with "minimizing maximum operation time + load balancing" as the core, designs a route total load calculation model in combination with the differences in workload of task nodes, and adopts a hybrid distance matrix of "warehouse-task node Euclidean distance + actual road distance between task nodes" to fully adapt to the characteristics of sanitation operation scenarios. This solves the problems of load imbalance and path deviating from actual working conditions caused by traditional single distance models and single objective optimization, ensuring the balance and feasibility of planning results.

[0099] Second, this invention custom-constructs a "GNN encoder + four-function head PPO agent", combining operator credit mechanism, longest path priority parameter sampling and periodic diversity optimization strategy, which solves the defects of traditional reinforcement learning state representation distortion, poor operator adaptability and easy getting trapped in local optima, greatly improves the algorithm's convergence efficiency and the quality of the optimal solution, and realizes efficient path optimization in large-scale task scenarios.

[0100] Third, this invention designs an initial path generation process of "cluster initialization + NHH path construction + local optimization" to ensure the quality of the initial solution and reduce the iteration load. Combined with the route adjustment mechanism of "comparison of new and old statistical information + threshold evaluation" during iteration, it ensures the pertinence and stability of path optimization, effectively improves the overall efficiency of multi-vehicle collaborative operation, and reduces operating costs. Attached Figure Description

[0101] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 The flowchart illustrates the balanced scheduling method for unmanned sanitation vehicle fleets based on near-end policy reinforcement learning, as provided in this embodiment of the invention.

[0103] Figure 2 A flowchart of a path generation algorithm is provided for embodiments of the present invention;

[0104] Figures 3(a), 3(b), and 3(c) show the performance of four route planning algorithms (ET, UDC, OUR, and OUR Ablation) on a real dataset in terms of total task distance, path variance, and the difference between the longest and shortest paths. They also show the values ​​assigned to each vehicle by the four algorithms in terms of path length and number of task nodes. OUR is the algorithm proposed in this invention, and OURAblation is the ablation version of the algorithm proposed in this invention. Detailed Implementation

[0105] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0106] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0107] Furthermore, the expression "includes" is an "open-ended" expression, which means only that there is a corresponding component or step, and should not be interpreted as excluding additional components or steps.

[0108] To achieve the objectives of this invention, some embodiments of an unmanned sanitation vehicle fleet equilibrium scheduling method based on proximal policy reinforcement learning are provided.

[0109] like Figure 1 As shown, the method for balanced scheduling of unmanned sanitation vehicle fleets includes:

[0110] Step S1: Determine the cleaning task set for the unmanned sanitation vehicle, with each task including workload and spatial location;

[0111] Step S2: Establish a hybrid distance matrix based on the task point coordinates and warehouse location, load the node workload, and construct a k-nearest neighbor graph;

[0112] The warehouse location is fixed at (0,0), and the workload is 0;

[0113] The hybrid distance matrix is ​​as follows: the distance between the warehouse and the task node is calculated using Euclidean distance, and the distance between task nodes is calculated using real-world data, i.e., the actual road distance between the corresponding task points.

[0114] The hybrid distance matrix needs to be symmetricized, and the diagonal elements should be set to 0.

[0115] Step S3: Establish a path planning objective function based on load balancing, task non-duplication, and minimizing the maximum operation time of unmanned sanitation vehicles;

[0116] Step S4: Based on the k-nearest neighbor graph, use a clustering algorithm to cluster the task nodes, generate an initial closed path, and then perform local optimization to find the initial optimal route based on the objective function.

[0117] Step S5: Construct a proximal policy optimization agent based on graph neural networks (i.e., PPO agent), use optimization algorithms to iteratively select routes and adjust operators and parameters, iterate until a preset number of times to obtain the optimal path that satisfies the objective function.

[0118] The Intelligent agents include graph neural networks ( Encoder and four functional heads;

[0119] The policy head is a set of algorithms. Make decisions;

[0120] The parameter head generates the source node, target route, and insertion position for the inter_reloc and inter_swap operators;

[0121] Value head is used for state value prediction;

[0122] Auxiliary head, used for prediction normalization ,in This represents the change in the maximum operating time of the route. This represents the change in the standard deviation of the route operation time.

[0123] The following is a detailed explanation of each step.

[0124] Furthermore, step S3 mainly involves modeling issues.

[0125] The objective function in step S3 is as follows:

[0126] ;

[0127] in , The number of unmanned sanitation vehicles, i.e. For each unmanned sanitation vehicle, the route sequence is defined. for The longest path length in the middle, for Standard deviation of the sequence This is the adjustment coefficient.

[0128] Furthermore, the total load on the route is as follows:

[0129] ;

[0130] in For a single vehicle, the closed path is: the vehicle starts from the warehouse, serves several task points, and finally returns to the warehouse; It is the set of routes taken by unmanned sanitation vehicles; For path The set of all node pairs in the set; for Adjacent node pairs within; Mixed distance matrix Middle node , The distance between; For path The path after removing the repository node; For nodes The workload.

[0131] Furthermore, step S4 employs an angle scanning algorithm for clustering. This embodiment uses angle scanning clustering: the polar angle of each task node is calculated with the distribution center as the origin, and the nodes are divided into M contiguous sub-clusters after being sorted by polar angle, ensuring that the nodes within each cluster are geographically concentrated and reducing path travel costs.

[0132] The basic steps of the angle scanning algorithm:

[0133] 1. Initialization: Determine the baseline point for clustering, usually set its coordinates to the origin (0,0); define the set of task nodes to be clustered, and the target number of clusters M (the number of vehicles).

[0134] 2. Calculate the polar angle: For each task node, with the reference point as the origin, calculate its polar angle θ_i relative to the positive x-axis (calculated using the arctan2(y_i, x_i) function, which can distinguish the four quadrants and avoid angle ambiguity), and obtain the angle values ​​of all nodes.

[0135] 3. Angle sorting: Sort all task nodes in ascending order (clockwise or counterclockwise) according to the polar angle θ_i to form an ordered list of nodes, ensuring that the nodes are arranged in spatial order around the reference point.

[0136] 4. Clustering: Divide the sorted list of nodes into M consecutive sublists (i.e., clusters) to make the number of nodes in each cluster as balanced as possible; if the total number of task nodes is not divisible by M, the first K clusters (K = total number of nodes % M) are each allocated 1 more node, and the remaining clusters have an equal number of nodes.

[0137] 5. Output results: The algorithm does not require iteration. After the segmentation is completed, it directly outputs M clustering results. Each cluster corresponds to an initial path, thus completing the clustering partitioning.

[0138] The advantages of the angle scanning algorithm include its simple principle, extremely high computational efficiency (no iterative process, time complexity is determined only by sorting), and the clustering results have a natural geographical concentration, which can avoid the ineffective travel costs caused by cross-regional operations from the source. It also adapts to the spatial characteristics of path planning scenarios and is easily combined with 2-opt and Or-opt path optimization algorithms. The angle scanning algorithm is used for clustering task nodes (such as the operating sections of unmanned sanitation vehicles and customer points for logistics delivery) in road network matrices, decomposing the complex multi-device path planning problem into multiple single-device sub-tasks, each sub-task corresponding to a cluster.

[0139] Specifically, step S4 includes:

[0140] Step S4.1: Use a clustering algorithm to segment the task nodes into... Clusters, The number of routes for unmanned sanitation vehicles;

[0141] Step S4.2: Construct the nearest neighbor heuristic (NHH) path for each cluster;

[0142] Step S4.3: Optimize the local path using the 2-opt and Or-opt algorithms;

[0143] Step S4.4: Select from all initial route combinations The smallest one is used as the initial optimal solution. .

[0144] By following the steps above, a preliminary set of paths can be quickly generated.

[0145] Furthermore, for the generated initial path set, the algorithm of this invention is used to determine the working route of each unmanned sanitation vehicle.

[0146] Specifically, such as Figure 2 As shown, step S5 includes:

[0147] Step S5.1: Construct state features X and statistical information s, and combine them with operator credit values. Calculate operator preferences;

[0148] Step S5.2: Select the operator based on the round number. If the current loop round t is not greater than the preset round number T_RR, select the operator using a round-robin method; otherwise, select the operator based on the strategy probability. An ε-greedy strategy is used to select operators; if the selected operator is inter_reloc or inter_swap, the corresponding parameters are sampled. ;

[0149] Step S5.3: Calculate the old statistics for the current route, and then calculate the new statistics after applying the selected operator and parameters; if the new maximum operation time max_new is greater than the old maximum operation time max_old multiplied by... And the new job time standard deviation std_new is not less than the old standard deviation std_old multiplied by 1 / 2. If so, then the route adjustment corresponding to that operator is rejected;

[0150] in The solution acceptance threshold coefficient is dynamically adjusted and varies with the iteration round t.

[0151] Step S5.4: Dictionary Reward Calculation: Calculate , The exponential moving average (EMA) is used to analyze the cropped data. Smoothing is performed to obtain the reward for the current round. ;

[0152] Step S5.5: Route status update: Update the current route R to the adjusted route R'. If the objective function value corresponding to the adjusted route is better, then update the optimal route, the optimal objective function value and the corresponding number of routes.

[0153] Step S5.6: Periodically perform route reinforcement and diversity optimization operations to avoid the algorithm getting stuck in local optima;

[0154] Step S5.7: Update the parameters of the PPO agent;

[0155] Step S5.8: Determine whether the preset maximum number of iterations has been reached;

[0156] If so, return the optimal path;

[0157] Otherwise, repeat steps S5.1-S5.7.

[0158] Furthermore, step S5.7 mainly involves updating the parameters of the PPO agent, including:

[0159] Step S5.7.1: Determine the input data for parameter updates, including state features. Statistical information Selected operators Sampling parameters Logarithmic probability of actions ,award Completed Mark Auxiliary targets Graph corresponding to task nodes Mixed distance matrix ;

[0160] Step S5.7.2: Calculate features and Bootstrap value: Combine state features with statistical information to form features. And calculate the value of Bootstrap. ;

[0161] Step S5.7.3: Calculate the Generalized Advantage Estimation (GAE): Based on the reward, completion indicator, current value, and subsequent value, calculate the time-series difference error. Based on Calculate generalized advantage ;

[0162] The ;

[0163] The ;

[0164] in As a discount factor, GAE coefficient;

[0165] Step S5.7.4: Normalize Advantage and Set Target Return: For generalized advantage Normalization is performed, and the target return is calculated simultaneously. ;

[0166] The ;

[0167] in For current value;

[0168] Step S5.7.5: Perform multiple rounds of small-batch updates, including:

[0169] 1. Recalculate the probability of the new action. Entropy H, Value Assisted prediction ;

[0170] 2. Calculate the probability ratio between the old and new strategies. ;

[0171] 3. Calculate the strategy loss ,in This is the Clip coefficient (e.g., 0.2);

[0172] 4. Calculate the loss of value With auxiliary loss , ;

[0173] 5. Calculate the total loss ,in and The loss weight is (e.g., 1.0 / 0.5). This is the entropy coefficient (e.g., 0.01).

[0174] 6. Employ the Adam optimizer using total loss All parameters of the PPO agent are updated through gradient backpropagation; gradient pruning is performed at the same time to avoid gradient explosion.

[0175] 7. Calculate the KL divergence between the old and new strategies within a mini-batch. The learning rate and entropy coefficient are adaptively adjusted based on the KL divergence, where k is the KL target value, typically set to 0.01. If KL > k, the policy update magnitude is too large, so the learning rate and entropy coefficient are reduced. Suppress exploration; if KL < k / 2: the policy update magnitude is insufficient, increase the learning rate and increase the entropy coefficient. Enhance exploration;

[0176] 8. Repeat steps 1 to 7 until the preset training round E is completed, at which point the parameter update for this round ends.

[0177] Furthermore, the sampling methods for inter_reloc and inter_swap in step S5 are as follows:

[0178] 1. Determine the set of input data routes Node embedding feature H, target operator o;

[0179] 2. Traverse all routes, select the longest route as the source route, exclude warehouse nodes from the source route, and sample one node from the remaining task nodes as the node to be migrated. The similarity of the node embedding feature H will be taken into account when sampling.

[0180] 3. If o is inter_reloc, select a route from the route set that is not equal to the source route, and select an insertion position in the gap between nodes of the target route to ensure that the target route still maintains the closed structure of "repository-task node-repository" after migration;

[0181] If o is inter_swap, select a route from the route set that is not equal to the source route, and sample one node from the non-warehouse nodes of the target route as the partner node for the swap. Similarly, the node embedding feature H should be referenced to ensure the effectiveness of the swap.

[0182] 4. Generate the corresponding exchange or transition parameters and output the log probability of the action;

[0183] In summary, the advantages of the present invention are as follows:

[0184] 1. Objective Function: A comprehensive objective function was constructed that combines "minimizing maximum job time + load balancing" ( ). This approach focuses on the core requirement of collaborative sanitation operations—shortening the longest overall operation time—while ensuring balanced load across multiple vehicles through the standard deviation of route length. The calculation of total route load incorporates the workload of task nodes, breaking the limitation of traditional methods that only measure operation costs by distance. At the same time, it adopts a hybrid distance matrix (using Euclidean distance between warehouses and task nodes, and actual road distance between task nodes) to make the objective function more closely match the actual sanitation operation conditions, solving the problem of the traditional single distance model being disconnected from the actual path.

[0185] 2. Application of the optimization algorithm: In the initial path generation stage, a combination strategy of angle scanning clustering + nearest neighbor heuristic (NHH) path construction + 2-opt / Or-opt local optimization is adopted to first achieve geographically concentrated clustering of task nodes, and then optimize the single route structure. This results in fast processing speed and ensures the quality of the initial solution to reduce the load of subsequent iterations. In the iterative optimization stage, an operator credit mechanism and an ε-greedy operator selection strategy are introduced to filter out invalid adjustments. Route reinforcement and diversity optimization operations are also performed periodically to avoid the algorithm getting stuck in local optima, which greatly improves the optimization efficiency and the quality of the optimal solution.

[0186] 3. Efficient and stable PPO agent parameter update: A customized PPO architecture with a GNN encoder and four functional heads is adopted. The GNN encoder can accurately capture the spatial correlation of nodes and the topological features of the route, solving the problem of distortion in traditional state representation. The parameter update process introduces generalized advantage estimation (GAE) to balance the bias and variance of advantage estimation. The policy update magnitude is limited by the Clip policy loss, and the total loss is constructed by combining value loss, auxiliary loss and policy entropy. The Adam optimizer is used with gradient clipping to prevent gradient explosion. At the same time, the learning rate and entropy coefficient are adaptively adjusted based on KL divergence, which not only ensures training stability, but also dynamically balances the "exploration-exploitation" requirements of the policy, realizing efficient parameter iteration in large-scale scenarios.

[0187] 4. Operator Sampling: For the inter_reloc (cross-route migration) and inter_swap (cross-route exchange) operators, a source route selection strategy prioritizing the longest route is designed to directly target global job bottlenecks for optimization. During sampling, warehouse nodes are strictly excluded to ensure the legality of route closure, and source / partner nodes are sampled based on the similarity of node embedding features to improve the effectiveness of operations. Cross-route selection of target routes is forcibly limited to avoid ineffective optimization of operations on the same route. At the same time, the logarithmic probability of actions is output to provide core data for the policy loss calculation of the PPO agent, realizing the closed-loop connection of "sampling-decision-update".

[0188] 5. It is versatile and scalable, suitable for various complex road conditions and task requirements, and has high application value and market prospects.

[0189] To better illustrate the embodiments of the present invention, the following experiments were conducted in this embodiment:

[0190] The sample set used in this embodiment of the invention includes three real-world scenario tasks. The specific experimental process is as follows:

[0191] I. Performance Indicators

[0192] The embodiments of this invention use: the longest vehicle path length, the total task distance, the path variance, and the difference between the longest and shortest paths.

[0193] II. Comparative Experiment

[0194] ET: The longest vehicle path length transforms the original system of multiple agents making decisions simultaneously into a process of "generating all agent decisions in a certain order." That is, it treats multi-agent decision-making as a sequence generation problem, allowing the Transformer model (which is stronger than sequence modeling) to be used directly. In this sequence, OpenReview+1 generates a "next access point or return to the warehouse" decision at each step, progressively constructing path schemes for all agents.

[0195] UDC: Total Path Length of All Vehicles Divide: The original large instance is divided into multiple sub-instances using a Graph Neural Network (GNN). UDC proposes a training strategy that tightly couples the partitioning strategy with the subsequent conquest strategy. arXiv+1Conquer: For each sub-instance, a fixed-length sub-path solver or a neural module is used to solve and merge the sub-solutions. arXivReunion: The sub-solutions are recombined to form the overall solution for the large instance. During training, the impact of each step in the partition-conquest-reunification system on the final overall solution is considered to ensure joint optimization. The term "Unified" emphasizes that partitioning and conquest are inseparable and trained end-to-end.

[0196] OUR Ablation: The ablation version of the algorithm proposed in this invention, excluding the GNN reinforcement learning version: removing GNN state encoding and RL / PPO operator selection and learning updates; the entire process adopts a hybrid improvement of deterministic / heuristic and simulated annealing (SA); it still retains "longest path reinforcement" and "adaptive operator weights (exponential weighting based on improvement degree)".

[0197] This invention uses ET, UDC, and the algorithm provided in the embodiments of this invention to compare and evaluate the overall performance.

[0198] Figures 3(a), 3(b), and 3(c) show the performance of four route planning algorithms (ET, UDC, OUR, and OUR Ablation) on a real dataset in terms of total task distance, path variance, and the difference between the longest and shortest paths, as well as the allocation of path length and task node number to each vehicle by the four algorithms.

[0199] As can be seen from the chart, the algorithm proposed in this invention can simultaneously achieve: shorter maximum operation time, lower total driving cost, more balanced vehicle load, and maintain stable high performance in scenarios with different numbers of vehicles.

[0200] In summary, the method proposed in this invention performs best in five metrics: total task distance, path variance, difference between the longest and shortest paths, and the allocation of path length and task node number to each vehicle among the four algorithms.

[0201] This invention also discloses an unmanned sanitation vehicle fleet equalization scheduling device based on proximal policy reinforcement learning, comprising:

[0202] The task set determination module is used to determine the operating areas of several unmanned sanitation vehicles.

[0203] The data initialization module is used to establish a hybrid distance matrix based on task point coordinates and warehouse location, load node workload, and construct a k-nearest neighbor graph;

[0204] The warehouse location is fixed at (0,0), and the workload is 0;

[0205] The hybrid distance matrix is ​​as follows: the distance between the warehouse and the task node is calculated using Euclidean distance, and the distance between task nodes is calculated using real-world data, i.e., the actual road distance between the corresponding task points.

[0206] The hybrid distance matrix needs to be symmetricized, and the diagonal elements should be set to 0.

[0207] The objective function establishment module is used to establish the path planning objective function based on load balancing, task non-duplication, and minimizing the maximum operation time of unmanned sanitation vehicles.

[0208] The initial path generation module is used to generate initial route combinations based on the k-nearest neighbor graph using an initialization algorithm, and find the initial optimal route based on the objective function value.

[0209] The path generation module constructs a proximal policy optimization agent (PPO agent) based on graph neural networks. It uses optimization algorithms to iteratively select routes and adjust operators and parameters, iterating to a preset number of times to obtain the optimal path that satisfies the objective function.

[0210] The Intelligent agents include graph neural networks ( Encoder and four functional heads;

[0211] The policy head is a set of algorithms. Make decisions;

[0212] The parameter head generates the source node, target route, and insertion position for the inter_reloc and inter_swap operators;

[0213] Value head is used for state value prediction;

[0214] Auxiliary head, used for prediction normalization ,in This represents the change in the maximum operating time of the route. This represents the change in the standard deviation of the route operation time.

[0215] Furthermore, in the objective function establishment module, the objective function is as follows:

[0216] ;

[0217] in , The number of work routes, i.e. The total load for each route;

[0218] in for The longest path length in the middle, for Standard deviation of the sequence This is the adjustment coefficient.

[0219] Furthermore, the total load of the route is as follows:

[0220] ;

[0221] in For a single vehicle, the closed path is: the vehicle starts from the warehouse, serves several task points, and finally returns to the warehouse; It is the set of routes taken by unmanned sanitation vehicles; For path The set of all node pairs in the set; for Adjacent node pairs within; Mixed distance matrix Middle node , The distance between; For path The path after removing the repository node; For nodes The workload.

[0222] Furthermore, the initial path generation module includes:

[0223] Clustering initialization unit, used to divide task nodes into Clusters, The number of routes for unmanned sanitation vehicles;

[0224] The path initialization unit constructs a nearest neighbor heuristic (NHH) path for each cluster;

[0225] The path optimization unit uses the 2-opt and Or-opt algorithms to perform local path optimization.

[0226] The path selection unit is used to select the route with the smallest objective function value from all initial route combinations as the initial optimal route.

[0227] Furthermore, the path generation module includes:

[0228] The state initialization unit constructs state features and statistical information, and calculates operator preferences by combining operator credit values;

[0229] The operator selection unit is used to select operators according to the round and sample the corresponding parameters;

[0230] The route adjustment unit is used to calculate the old statistics of the current route, apply the selected operators and parameters to calculate the new statistics, compare the two and decide whether to adjust the route.

[0231] The reward calculation unit is used to calculate dictionary rewards and update operator credits based on the success rate adjustment of the operator's route.

[0232] The route status update unit is used to update the current route to the adjusted route. If the objective function value corresponding to the adjusted route is better, the optimal route, the optimal objective function value, and the corresponding number of routes are updated.

[0233] The optimization unit is used to periodically perform route reinforcement and diversity optimization operations to prevent the algorithm from getting trapped in local optima.

[0234] The parameter update unit is used to update the parameters of the PPO agent.

[0235] The algorithm termination judgment unit is used to determine whether the preset maximum number of iterations has been reached.

[0236] If so, return the optimal path;

[0237] Otherwise, the methods within the state initialization unit, operator selection unit, route adjustment unit, reward calculation unit, route state update unit, and optimization unit are executed sequentially.

[0238] Furthermore, in the path generation module, the parameter update unit includes:

[0239] A data input unit is used to determine the input data;

[0240] The objective determination unit is used to calculate the generalized advantage estimate and set the objective reward;

[0241] The parameter update execution unit is used to update the agent parameters through multiple rounds of mini-batch iterations: in each mini-batch, the log probability of the action, policy entropy, state value prediction value, and auxiliary head prediction result are recalculated; the probability ratio is calculated, and the policy loss is calculated based on the probability ratio and the normalized generalized advantage; the total loss is constructed by combining the value loss, auxiliary loss, and policy entropy; the agent parameters are updated using the Adam optimizer, and gradient pruning is performed.

[0242] Furthermore, the sampling of inter_reloc and inter_swap in the path generation module includes:

[0243] A data input unit is used to determine the input data;

[0244] The route selection unit is used to select the longest route in the current route set as the source route and sample the non-warehouse nodes in the source route as operation nodes.

[0245] The parameter sampling unit is used to sample routes different from the source route as target routes, and perform corresponding operations according to the target operator type to obtain parameters.

[0246] In this embodiment, the specific content of the unmanned sanitation vehicle fleet equalization scheduling device based on proximal policy reinforcement learning is similar to the content of the unmanned sanitation vehicle fleet equalization scheduling method based on proximal policy reinforcement learning disclosed in the above embodiments, and will not be repeated here.

[0247] Furthermore, embodiments of the present invention disclose a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed as disclosed in any of the above embodiments: a method for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning.

[0248] This invention discloses a method for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning, which has the following beneficial effects:

[0249] First, this invention constructs a comprehensive objective function with "minimizing maximum operation time + load balancing" as the core, designs a route total load calculation model in combination with the differences in workload of task nodes, and adopts a hybrid distance matrix of "warehouse-task node Euclidean distance + actual road distance between task nodes" to fully adapt to the characteristics of sanitation operation scenarios. This solves the problems of load imbalance and path deviating from actual working conditions caused by traditional single distance models and single objective optimization, ensuring the balance and feasibility of planning results.

[0250] Second, this invention custom-constructs a "GNN encoder + four-function head PPO agent", combining operator credit mechanism, longest path priority parameter sampling and periodic diversity optimization strategy, which solves the defects of traditional reinforcement learning state representation distortion, poor operator adaptability and easy getting trapped in local optima, greatly improves the algorithm's convergence efficiency and the quality of the optimal solution, and realizes efficient path optimization in large-scale task scenarios.

[0251] Third, this invention designs an initial path generation process of "cluster initialization + NHH path construction + local optimization" to ensure the quality of the initial solution and reduce the iteration load. Combined with the route adjustment mechanism of "comparison of new and old statistical information + threshold evaluation" during iteration, it ensures the pertinence and stability of path optimization, effectively improves the overall efficiency of multi-vehicle collaborative operation, and reduces operating costs.

[0252] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a floppy disk, CD-ROM, hard disk, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, that machine becomes an apparatus for practicing the present invention.

[0253] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the present invention as claimed, which is defined by the appended claims and their equivalents.

Claims

1. A method for balanced scheduling of unmanned sanitation vehicle fleets based on proximal policy reinforcement learning, characterized in that, include: Step S1: Determine the cleaning task set for the unmanned sanitation vehicle, with each task including workload and spatial location; Step S2: Establish a hybrid distance matrix based on the task point coordinates and warehouse location, load the node workload, and construct a k-nearest neighbor graph; The warehouse location is fixed at (0,0), and the workload is 0; The hybrid distance matrix is ​​as follows: the distance between the warehouse and the task node is calculated using Euclidean distance, and the distance between task nodes is calculated using real-world data, i.e., the actual road distance between the corresponding task points. The hybrid distance matrix needs to be symmetricized, and the diagonal elements should be set to 0. Step S3: Establish a path planning objective function based on load balancing, task non-duplication, and minimizing the maximum operation time of unmanned sanitation vehicles; Step S4: Based on the k-nearest neighbor graph, use a clustering algorithm to cluster the task nodes, generate an initial closed path, and then perform local optimization to find the initial optimal route based on the objective function. Step S5: Construct a PPO agent based on graph neural network, use optimization algorithm to iteratively select routes and adjust operators and parameters, iterate until a preset number of times to obtain the optimal path that satisfies the objective function; The The intelligent agent includes a graph neural network encoder and four functional heads; The strategy head is for the set of algorithms. Make decisions; The parameter header generates the source node, target route, and insertion position for the inter_reloc and inter_swap operators; Value head, used for state value prediction; Auxiliary head, used for predicting normalization ,in This represents the change in the maximum operating time of the route. This represents the change in the standard deviation of the route operation time.

2. The method for balanced scheduling of unmanned sanitation vehicle fleets according to claim 1, characterized in that, The objective function for the path planning is as follows: ； in , The number of work routes, i.e. The total load for each route; in for The longest path length in the middle, for Standard deviation of the sequence This is the adjustment coefficient.

3. The method for balanced scheduling of unmanned sanitation vehicle fleets according to claim 2, characterized in that, The total load on the route is as follows: ； in For a single vehicle, the closed path is: the vehicle starts from the warehouse, serves several task points, and finally returns to the warehouse; It is the set of routes taken by unmanned sanitation vehicles; For path The set of all node pairs in the set; for Adjacent node pairs within; Mixed distance matrix Middle node , The distance between; For path The path after removing the repository node; For nodes The workload.

4. The method for balanced scheduling of unmanned sanitation vehicle fleets according to claim 1, characterized in that, Step S4 includes: Step S4.1: Use a clustering algorithm to segment the task nodes into... Clusters, The number of routes for unmanned sanitation vehicles; Step S4.2: Construct the nearest neighbor heuristic path for each cluster; Step S4.3: Optimize the local path using the 2-opt and Or-opt algorithms; Step S4.4: Select from all initial route combinations The smallest one is used as the initial optimal solution. .

5. The method for balanced scheduling of unmanned sanitation vehicle fleets according to claim 1, characterized in that, Step S5 includes: Step S5.1: Construct state features and statistical information, and calculate operator preferences by combining them with operator credit values; Step S5.2: Select the operator according to the round and sample the corresponding parameters; Step S5.3: Calculate the old statistics for the current route, apply the selected operator and parameters to calculate the new statistics, compare the two and decide whether to adjust the route; Step S5.4: Calculate dictionary rewards and update operator credit based on the success rate adjustment of the operator's route; Step S5.5: Route status update: Update the current route R to the adjusted route R'. If the objective function value corresponding to the adjusted route is better, then update the optimal route, the optimal objective function value and the corresponding number of routes. Step S5.6: Periodically perform route reinforcement and diversity optimization operations to avoid the algorithm getting stuck in local optima; Step S5.7: Update the parameters of the PPO agent; Step S5.8: Determine whether the preset maximum number of iterations has been reached; If so, return the optimal path; Otherwise, repeat steps S5.1-S5.

7.

6. The method for balanced scheduling of unmanned sanitation vehicle fleets according to claim 5, characterized in that, Step S5.7 includes: Step S5.7.1: Determine the input data; Step S5.7.2: Calculate the generalized advantage estimate and set the target return; Step S5.7.3: Update agent parameters through multiple rounds of mini-batch iteration: In each mini-batch, recalculate the log probability of actions, policy entropy, state value prediction, and auxiliary head prediction results; calculate the probability ratio, calculate the policy loss based on the probability ratio and the normalized generalized advantage, and construct the total loss by combining the value loss, auxiliary loss, and policy entropy; update agent parameters using the Adam optimizer and perform gradient pruning.

7. The method for balanced scheduling of unmanned sanitation vehicle fleets according to claim 1, characterized in that, The sampling methods for inter_reloc and inter_swap in step S5 are as follows: Step S5.2.1: Determine the input data; Step S5.2.2: Select the longest route in the current route set as the source route, and sample the non-warehouse nodes in the source route as operation nodes; Step S5.2.3: Sample a route different from the source route as the target route, and perform the corresponding operation according to the target operator type to obtain the parameters.

8. A balanced scheduling device for unmanned sanitation vehicle fleets based on proximal policy reinforcement learning, characterized in that, include: The task set determination module is used to determine the operating areas of several unmanned sanitation vehicles. The data initialization module is used to establish a hybrid distance matrix based on task point coordinates and warehouse location, load node workload, and construct a k-nearest neighbor graph; The warehouse location is fixed at (0,0), and the workload is 0; The hybrid distance matrix is ​​as follows: the distance between the warehouse and the task node is calculated using Euclidean distance, and the distance between task nodes is calculated using real-world data, i.e., the actual road distance between the corresponding task points. The hybrid distance matrix needs to be symmetricized, and the diagonal elements should be set to 0. The objective function establishment module is used to establish the path planning objective function based on load balancing, task non-duplication, and minimizing the maximum operation time of unmanned sanitation vehicles. The initial path generation module is used to generate initial route combinations based on the k-nearest neighbor graph using an initialization algorithm, and find the initial optimal route based on the objective function value. The path generation module constructs a PPO agent based on a graph neural network, uses an optimization algorithm to iteratively select routes and adjust operators and parameters, and iterates to a preset number of times to obtain the optimal path that satisfies the objective function. The The intelligent agent includes a graph neural network encoder and four functional heads; The strategy head is for the set of algorithms. Make decisions; The parameter header generates the source node, target route, and insertion position for the inter_reloc and inter_swap operators; Value head, used for state value prediction; Auxiliary head, used for predicting normalization ,in This represents the change in the maximum operating time of the route. This represents the change in the standard deviation of the route operation time.

9. The unmanned sanitation vehicle fleet balanced dispatching equipment according to claim 8, characterized in that, The objective function is defined in the objective function establishment module, and the objective function is as follows: ； in , The number of work routes, i.e. The total load for each route; in for The longest path length in the middle, for Standard deviation of the sequence This is the adjustment coefficient.

10. The unmanned sanitation vehicle fleet balanced dispatching equipment according to claim 9, characterized in that, The total load on the route is as follows: ； in For a single vehicle, the closed path is: the vehicle starts from the warehouse, serves several task points, and finally returns to the warehouse; It is the set of routes taken by unmanned sanitation vehicles; For path The set of all node pairs in the set; for Adjacent node pairs within; Mixed distance matrix Middle node , The distance between; For path The path after removing the repository node; For nodes The workload.

11. The unmanned sanitation vehicle fleet equalization dispatching equipment according to claim 8, characterized in that, The initial path generation module includes: Clustering initialization unit, used to divide task nodes into Clusters, The number of routes for unmanned sanitation vehicles; The path initialization unit constructs the nearest neighbor heuristic path for each cluster; The path optimization unit uses the 2-opt and Or-opt algorithms to perform local path optimization. The path selection unit is used to select the route with the smallest objective function value from all initial route combinations as the initial optimal route.

12. The unmanned sanitation vehicle fleet equalization dispatching equipment according to claim 8, characterized in that, The path generation module includes: The state initialization unit constructs state features and statistical information, and calculates operator preferences by combining operator credit values; The operator selection unit is used to select operators according to the round and sample the corresponding parameters; The route adjustment unit is used to calculate the old statistics of the current route, apply the selected operators and parameters to calculate the new statistics, compare the two and decide whether to adjust the route. The reward calculation unit is used to calculate dictionary rewards and update operator credits based on the success rate adjustment of the operator's route. The route status update unit is used to update the current route to the adjusted route. If the objective function value corresponding to the adjusted route is better, then the optimal route, the optimal objective function value, and the corresponding number of routes are updated. The optimization unit is used to periodically perform route reinforcement and diversity optimization operations to prevent the algorithm from getting trapped in local optima. The parameter update unit is used to update the parameters of the PPO agent. The algorithm termination judgment unit is used to determine whether the preset maximum number of iterations has been reached. If so, return the optimal path; Otherwise, the methods within the state initialization unit, operator selection unit, route adjustment unit, reward calculation unit, route state update unit, and optimization unit are executed sequentially.

13. The unmanned sanitation vehicle fleet balanced dispatching equipment according to claim 12, characterized in that, The path generation module includes a parameter update unit: A data input unit is used to determine the input data; The objective determination unit is used to calculate the generalized advantage estimate and set the objective reward; The parameter update execution unit is used to update the agent parameters through multiple rounds of mini-batch iterations: in each mini-batch, the log probability of the action, policy entropy, state value prediction value, and auxiliary head prediction result are recalculated; the probability ratio is calculated, and the policy loss is calculated based on the probability ratio and the normalized generalized advantage; the total loss is constructed by combining the value loss, auxiliary loss, and policy entropy; the agent parameters are updated using the Adam optimizer, and gradient pruning is performed.

14. The unmanned sanitation vehicle fleet equalization dispatching equipment according to claim 8, characterized in that, The sampling of inter_reloc and inter_swap in the path generation module includes: A data input unit is used to determine the input data; The route selection unit is used to select the longest route in the current route set as the source route and sample the non-warehouse nodes in the source route as operation nodes. The parameter sampling unit is used to sample routes different from the source route as target routes, and perform corresponding operations according to the target operator type to obtain parameters.

15. A storage medium, characterized in that, The storage medium stores one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed as described in any of claims 1-7 for the balanced scheduling method of unmanned sanitation vehicle fleet based on proximal policy reinforcement learning.

Images