Garbage truck intelligent scheduling method and system based on reinforcement learning

Abstract

This invention discloses a method and system for intelligent scheduling of garbage trucks based on reinforcement learning, specifically relating to the field of intelligent scheduling technology. The method involves acquiring urban garbage collection road network topology, historical garbage production spatiotemporal distribution data, and garbage truck trajectory data; calculating the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent routes to form a negative coupling matrix; performing spectral clustering on the negative coupling matrix to divide the coupling-sensitive subgraph and extracting a set of key intersection segments; inputting the set of key intersection segments and historical garbage production spatiotemporal distribution data into a multi-agent reinforcement learning framework to construct a scheduling strategy model and output initial strategy parameters; generating a collaborative scheduling scheme based on the initial strategy parameters in a simulation environment and incrementally correcting the initial strategy parameters to obtain a converged scheduling strategy; determining the garbage truck departure time and travel path based on the converged scheduling strategy and writing it back to the route interaction graph for iterative optimization.

Description

Technical Field

[0001] This invention relates to the field of intelligent scheduling technology, and more specifically, to a method and system for intelligent scheduling of garbage trucks based on reinforcement learning. Background Technology

[0002] Existing garbage truck dispatching methods mostly rely on manual experience or static route planning, typically scheduling garbage collection vehicles based solely on fixed routes and departure times, without considering the dynamic changes in urban waste production over time and space. Furthermore, due to overlapping and repetitive routes among multiple garbage truck routes, static dispatching schemes struggle to accurately address the real-time interactions between routes, thus impacting the efficiency of garbage collection operations.

[0003] To address the aforementioned problems, a technical solution is provided. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a method and system for intelligent scheduling of garbage trucks based on reinforcement learning to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The intelligent scheduling method for garbage trucks based on reinforcement learning includes the following steps: S1: Obtain the urban waste collection road network topology, historical waste production spatiotemporal distribution data, and garbage truck operation trajectory data to construct a route interaction map that includes intersection and overlapping coverage relationships; S2: Calculate the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generate a negative coupling matrix; S3: Perform spectral clustering on the negative coupling matrix to divide the coupling-sensitive subgraph and extract the set of key intersection segments; S4: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework, construct the scheduling strategy model and output the initial strategy parameters; S5: Generate a collaborative scheduling scheme based on the initial strategy parameters in the simulation environment, monitor the cross-line negative coupling strength in real time during garbage truck scheduling, incrementally correct the initial strategy parameters, and obtain a convergent scheduling strategy. S6: Determine the departure time and driving route of garbage trucks based on the convergence scheduling strategy, and write the departure time and driving route of garbage trucks back to the route interaction graph for iterative optimization.

[0007] In a preferred embodiment, S1 specifically refers to: Obtain the topology of the urban waste collection road network and unify spatial reference; extract road segment connection relationships and intersection node identifiers to form road network association data. Acquire historical waste production spatiotemporal distribution data and aggregate it into road network associated data according to the intersection node identifier; Obtain garbage truck trajectory data and perform map matching to obtain trajectory segment sequences; Based on the trajectory segment sequence, the system identifies the intersection and overlapping coverage relationships and generates a route interaction map.

[0008] In a preferred embodiment, S2 specifically refers to: Based on the intersection node identifiers and overlapping relationships in the line interaction graph, a line association set with the intersection node identifiers as the propagation starting point is constructed. Based on the line association set, diffusion calculation is performed along the connection direction of the line interaction graph to count the degree of coverage overlap of cross node identifiers in different lines and the frequency of overlap of running trajectories. After normalizing the degree of coverage overlap and the frequency of overlap of the running trajectory, the data is mapped to negative coupling weights between lines and arranged according to the correspondence between the lines to form a negative coupling matrix.

[0009] In a preferred embodiment, S3 specifically refers to: Calculate the Laplacian matrix corresponding to the negative coupling matrix; Eigenvalue decomposition of the Laplacian matrix yields a low-dimensional representation vector for clustering; Coupled sensitive subgraphs are obtained by clustering based on low-dimensional representation vectors; Based on the coupling-sensitive subgraph, the cross nodes with the highest cumulative percentage of negative coupling weight are selected and sorted according to their negative coupling weight. Map the road segment connection relationship corresponding to the intersection node with the highest cumulative negative coupling weight to a set of key intersection segments.

[0010] In a preferred embodiment, S4 specifically refers to: Based on historical waste production spatiotemporal distribution data, the waste volume characteristics of key intersections are extracted for time windows. The time window waste volume characteristics are combined with the line identifiers associated with the key intersection segment set in the line interaction diagram to form a status feature; In the multi-agent reinforcement learning framework, a scheduling policy model is constructed using state features as input and route selection actions as output. The training objective is defined by using the negative coupling weights corresponding to the negative coupling matrix as constraints to output the initial policy parameters.

[0011] In a preferred embodiment, S5 specifically refers to: In the simulation environment, a collaborative scheduling scheme is generated by performing multi-vehicle route selection and deduction on the route interaction diagram based on the initial strategy parameters. Based on the collaborative scheduling scheme, the arrival and departure sequences of vehicles corresponding to the key intersection segments are recorded and the negative coupling strength across the lines is calculated. The initial strategy parameters are incrementally corrected and iterated cyclically based on the deviation between the cross-line negative coupling strength and the negative coupling matrix to obtain a convergent scheduling strategy.

[0012] In a preferred embodiment, S6 specifically refers to: The garbage truck departure time is determined based on the convergence scheduling strategy, and the garbage truck travel route is planned according to the route interaction diagram. Garbage trucks are dispatched to collect garbage based on their departure times and routes, and the corresponding garbage truck trajectory data is recorded. After matching the garbage truck's driving trajectory data with the map, a sequence of trajectory segments is formed and updated to the route interaction map for the next round of iteration and optimization.

[0013] On the other hand, the present invention provides a garbage truck intelligent dispatching system based on reinforcement learning, comprising: Data construction module: acquire urban waste collection road network topology, historical waste production spatiotemporal distribution data and garbage truck operation trajectory data, and construct a route interaction map including intersection and overlapping coverage relationships; Coupling Calculation Module: Calculates the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generates a negative coupling matrix; Clustering analysis module: Performs spectral clustering on the negative coupling matrix, divides the coupling-sensitive subgraph, and extracts the set of key intersection segments; Strategy modeling module: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework, construct the scheduling strategy model and output the initial strategy parameters; Simulation optimization module: Generates a collaborative scheduling scheme based on the initial strategy parameters in the simulation environment, monitors the cross-line negative coupling strength in real time during garbage truck scheduling, incrementally corrects the initial strategy parameters, and obtains a convergent scheduling strategy; Scheduling execution module: Based on the convergent scheduling strategy, the departure time and driving route of garbage trucks are determined, and the departure time and driving route of garbage trucks are written back to the route interaction graph for iterative optimization.

[0014] The technical effects and advantages of the intelligent garbage truck scheduling method and system based on reinforcement learning in this invention are as follows: Based on the urban waste collection road network topology, historical waste production spatiotemporal distribution data, and garbage truck trajectory data, a route interaction graph containing intersection and overlapping coverage relationships is constructed, enabling a structured expression of the interaction relationships between garbage truck operation routes. By calculating the negative coupling coefficient of intersection segments to adjacent route scheduling strategies and forming a negative coupling matrix, the potential interference level between routes is quantified. By performing spectral clustering on the negative coupling matrix to divide the coupling-sensitive subgraph and extract the set of key intersection segments, the scheduling focus is concentrated on key segments. The set of key intersection segments and historical waste production spatiotemporal distribution data are input into a multi-agent reinforcement learning framework to construct a scheduling strategy model and output initial strategy parameters, so that the scheduling decision takes into account both the spatiotemporal changes in waste volume and route coupling constraints. A collaborative scheduling scheme is generated in the simulation environment, and the initial strategy parameters are incrementally corrected based on the cross-route negative coupling strength to obtain a convergent scheduling strategy, so that the strategy has an iteratively optimized path in multi-vehicle collaborative scenarios. Based on the convergent scheduling strategy, the garbage truck departure time and driving route are determined and written back to the route interaction graph, realizing closed-loop iterative optimization of scheduling decisions and route interaction graphs. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the intelligent garbage truck scheduling method based on reinforcement learning according to the present invention; Figure 2 This is a schematic diagram of the intelligent garbage truck scheduling system based on reinforcement learning according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 Figure 1 The present invention provides a method for intelligent scheduling of garbage trucks based on reinforcement learning, which includes the following steps: S1: Obtain the urban waste collection road network topology, historical waste production spatiotemporal distribution data, and garbage truck operation trajectory data to construct a route interaction map that includes intersection and overlapping coverage relationships; S2: Calculate the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generate a negative coupling matrix; S3: Perform spectral clustering on the negative coupling matrix to divide the coupling-sensitive subgraph and extract the set of key intersection segments; S4: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework, construct the scheduling strategy model and output the initial strategy parameters; S5: Generate a collaborative scheduling scheme based on the initial strategy parameters in the simulation environment, monitor the cross-line negative coupling strength in real time during garbage truck scheduling, incrementally correct the initial strategy parameters, and obtain a convergent scheduling strategy. S6: Determine the departure time and driving route of garbage trucks based on the convergence scheduling strategy, and write the departure time and driving route of garbage trucks back to the route interaction graph for iterative optimization.

[0018] S1: Obtain urban waste collection road network topology, historical waste production spatiotemporal distribution data, and garbage truck operation trajectory data; construct a route interaction map including intersections and overlapping coverage relationships, including: Obtain the topology of the urban waste collection road network and unify spatial reference; extract road segment connection relationships and intersection node identifiers to form road network association data. Specifically, the process involves acquiring the urban waste collection road network topology and unifying its spatial reference. The urban waste collection road network topology includes topological data encompassing all possible road segments traversed by urban waste trucks. Unifying the spatial reference involves standardizing road network topology data from different sources to a common spatial coordinate system, such as the WGS84 geographic coordinate system. Vectorized road network data provided by urban planning departments is used for extraction. This vectorized data includes, but is not limited to, the geometric center coordinates of road segments, the connection topology of road segments, and the spatial adjacency relationships between road segments. Methods for extracting road segment connectivity include, but are not limited to, performing spatial proximity searches on each road boundary line in the vectorized road network data to determine the connectivity between any two road segments at their endpoints, thereby obtaining the road segment connectivity relationships. Simultaneously, to accurately identify the intersection nodes of waste truck routes, intersection node identifiers need to be extracted. An intersection node identifier is defined as a node where at least three road segments are directly connected in the spatial topology. This is obtained by traversing the road segment connectivity relationships and counting the number of times each road segment endpoint appears in the entire road network topology; endpoints with a connection count greater than or equal to three are identified as intersection node identifiers. After the above process is completed, road network association data including the connection relationships of all road segments and the identification of intersection nodes is obtained.

[0019] Acquire historical waste production spatiotemporal distribution data and aggregate it into road network associated data according to the intersection node identifier; Specifically, historical waste production spatiotemporal distribution data is obtained. This data represents the temporal distribution characteristics of waste generation for each road segment within a specific historical time period. The historical waste production spatiotemporal distribution data is obtained through historical records from the waste recycling management department, including but not limited to the historical waste weight at waste collection sites, the date and time of waste collection completion, and the spatial location of the corresponding road segment.

[0020] Historical waste production spatiotemporal distribution data is aggregated into road network associated data according to the cross node identifiers. Specifically, spatial matching is performed based on the spatial location of each waste collection station to determine the cross node identifiers corresponding to the road segments where the waste collection stations are located; historical waste production data is spatially aggregated based on the cross node identifiers, that is, the waste volume data of multiple waste collection stations associated with the same cross node identifier are summed; then the data within different date and time windows are accumulated or averaged to form the association between historical waste production spatiotemporal distribution data and road network associated data.

[0021] Obtain garbage truck trajectory data and perform map matching to obtain trajectory segment sequences; Specifically, the process involves acquiring garbage truck trajectory data, defined as the location information continuously recorded by urban garbage trucks during daily garbage collection operations. The acquisition methods for this data include, but are not limited to, obtaining the real-time latitude and longitude coordinates and timestamps of the garbage truck's current location through an onboard GPS system, for example, recording the vehicle's location information every minute. The acquired garbage truck trajectory data undergoes map matching processing. For each trajectory coordinate point in the data, the nearest road segment centerline is searched spatially. The vertical spatial distance between the trajectory coordinate point and the road segment centerline is then calculated, and a spatial matching distance threshold is set, for example, to 50 meters. If the spatial distance between the trajectory coordinate point and the nearest road segment centerline is less than or equal to the spatial matching distance threshold, the trajectory coordinate point is determined to belong to the corresponding road segment, and the garbage truck trajectory data is arranged in chronological order to form a trajectory-road segment sequence.

[0022] Based on the trajectory segment sequence, the system identifies intersection and overlapping coverage relationships and generates a route interaction map. Specifically, the process iterates through the trajectory segment sequence corresponding to each garbage truck, identifying cases where the same road segment is covered by the trajectory segment sequences of multiple garbage trucks as overlapping coverage relationships. Identification methods include, but are not limited to, counting the coverage frequency of all garbage truck trajectory segment sequences. If any road segment appears more than once in multiple garbage truck trajectory segment sequences, then the road segment is determined to have an overlapping coverage relationship. Methods for identifying intersection relationships include, but are not limited to, determining situations where different garbage trucks meet at intersection node markers and choose different road segments to continue driving within the garbage truck trajectory segment sequence. Then, based on the intersection node markers defined in the road network association data, the occurrence of garbage truck trajectory segment sequences at intersection node markers is marked as a route intersection relationship. Based on overlapping coverage relationships and route intersection relationships, combined with the road network association data, a route interaction map is constructed. The method for constructing the route interaction graph is as follows: the intersection node identifier is used as the node of the graph, the road segment connection relationship is used as the edge of the graph, and the repetition coverage relationship and the line intersection relationship are used as the weight attribute of the edge. The weight attribute value is calculated by statistically and normally processing the frequency of repetition coverage or the number of line intersections at the intersection node identifier in the trajectory road segment sequence. For example, the road segment with the most repetition coverage is normalized to the maximum value of the weight attribute, that is, the value is 1. The weight attribute values ​​of other road segments are calculated according to the ratio of their respective repetition coverage number to the maximum value.

[0023] S2: Calculate the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generate a negative coupling matrix, including: Based on the intersection node identifiers and overlapping relationships in the line interaction graph, a line association set with the intersection node identifiers as the propagation starting point is constructed. Specifically, the process involves reading the intersection node identifiers and overlapping coverage relationships from the route interaction graph. The route interaction graph is a network structure where intersection node identifiers are the nodes, road segment connections are the edges, and overlapping coverage relationships and road intersection relationships are the edge weights. An intersection node identifier is defined as a node where at least three road segments are directly connected in spatial topology. A overlapping coverage relationship is defined as the situation where the same road segment is covered by multiple garbage truck trajectory sequences. The process iterates through all nodes in the route interaction graph, extracting the overlapping coverage relationship data for each intersection node identifier and its corresponding road segment. This overlapping coverage relationship data includes, but is not limited to, the frequency information of how often each road segment connected to a particular intersection node is simultaneously or alternately covered by multiple garbage truck trajectory sequences.

[0024] A route association set is constructed using intersection node identifiers as the propagation starting point. This set is defined as the set of all routes that can be directly or indirectly reached by propagating along the road segment connections in the route interaction graph from each intersection node identifier. The construction of the route association set is implemented as follows: for each intersection node identifier in the route interaction graph, find all road segments directly connected to that intersection node identifier; then, based on the road segment connections, extend outwards sequentially along each directly connected road segment, expanding the route propagation path until it cannot be expanded further; finally, record all routes traversed by the expanded path to form the route association set. The route association set accurately reflects the route diffusion range starting from the intersection node identifier.

[0025] Based on the line association set, diffusion calculation is performed along the connection direction of the line interaction graph to count the degree of coverage overlap of cross node identifiers in different lines and the frequency of overlap of running trajectories. Specifically, diffusion calculation involves statistically analyzing the degree of coverage overlap and the frequency of trajectory overlap of each intersection node within the route association set identified by that node across different routes. The degree of coverage overlap is defined as the proportion of spatial coverage intersection between different routes on the same road segment, and the frequency of trajectory overlap is defined as the number of times spatial overlap occurs in the garbage truck trajectory data for the same road segment. For each pair of routes within the route association set, spatial coverage relationships are compared, and the length of the spatial coverage intersection is recorded along with the total length of each route. The degree of coverage overlap is then obtained by dividing the length of the spatial coverage intersection by the total length of each route. For the same pair of routes, trajectory segment sequences for the corresponding road segment are extracted from the garbage truck trajectory data. The overlap between these sequences is compared, and the number of overlaps is accumulated to obtain the trajectory overlap frequency data. Through diffusion calculation, the degree of coverage overlap and the frequency of trajectory overlap are obtained.

[0026] After normalizing the degree of coverage overlap and the frequency of overlap of the running trajectory, the data is mapped to negative coupling weights between lines and arranged according to the correspondence between the lines to form a negative coupling matrix. Specifically, minimum-maximum normalization transformation is applied to both the degree of coverage overlap and the frequency of overlap in operational trajectories. The specific numerical weighting fusion method for the normalized coverage overlap and frequency of overlap in operational trajectories includes, but is not limited to: assigning preset fusion coefficients to both the normalized coverage overlap and the normalized frequency of overlap in operational trajectories, and then adding them together to obtain the negative coupling weights between routes. The fusion coefficients are set by performing fitting analysis based on historical waste production spatiotemporal distribution data and garbage truck operational trajectory data to calculate the weight percentage of the impact of coverage overlap and frequency of overlap in operational trajectories on actual waste collection efficiency. For example, when coverage overlap has a significant impact on actual waste collection efficiency, the corresponding fusion coefficient is set to 0.7, and the fusion coefficient for the frequency of overlap in operational trajectories is set to 0.3. The route correspondence is determined based on the route association set, including but not limited to sequentially numbering the routes within the route association set and constructing a matrix index; then, the fused negative coupling weights are arranged according to the matrix index order to form a negative coupling matrix. In the negative coupling matrix, each element represents the negative coupling weight between the corresponding lines. The diagonal elements are the negative coupling weights of each line itself, which are fixed at 0. The off-diagonal elements represent the strength of the negative coupling relationship between different lines, which is generated by the propagation of the cross node identifier, i.e., the corresponding negative coupling weight.

[0027] S3: Perform spectral clustering on the negative coupling matrix, divide the coupling-sensitive subgraph, and extract the set of key intersection segments, including: Calculate the Laplacian matrix corresponding to the negative coupling matrix; Specifically, the negative coupling matrix is ​​a matrix composed of the negative coupling weights between lines, calculated through diffusion of the line association set and then subjected to min-max normalization transformation and weighted by fusion coefficients. The Laplace matrix is ​​defined as the difference between the degree matrix corresponding to the negative coupling matrix and the negative coupling matrix itself. The degree matrix is ​​calculated by: accumulating all negative coupling weights corresponding to each line in the negative coupling matrix row by row, using the accumulated value as the degree of each line; then arranging the degrees of each line on the diagonal, setting the remaining elements to zero, thus forming the degree matrix. If the negative coupling matrix contains n lines, the degree matrix is ​​an n-order diagonal matrix, where each element on the diagonal represents the sum of the negative coupling weights between the corresponding line and all other lines. Therefore, by calculating the degree matrix and subtracting the negative coupling matrix from it, the Laplace matrix corresponding to the negative coupling matrix can be obtained.

[0028] Eigenvalue decomposition of the Laplacian matrix yields a low-dimensional representation vector for clustering; Specifically, eigenvalues ​​and eigenvectors are calculated for the Laplace matrix to obtain all eigenvalues ​​and eigenvectors corresponding to the Laplace matrix. All eigenvalues ​​are sorted in ascending order, and the eigenvectors corresponding to several eigenvalues ​​with smaller values ​​are selected to form a low-dimensional representation vector. The number of selected eigenvalues ​​represents the number of clustering target categories. The number of clustering target categories is determined by analyzing clustering effectiveness indicators using historical waste production spatiotemporal distribution data and garbage truck trajectory data. For example, the silhouette coefficient index can be used. By comparing the silhouette coefficients of different number of categories, the number of categories that maximize or nearly maximize the silhouette coefficient is selected as the number of clustering target categories, thus determining the dimension of the low-dimensional representation vector used for clustering. After completing eigenvalue decomposition and eigenvector selection, the low-dimensional representation vector is obtained.

[0029] Coupled sensitive subgraphs are obtained by clustering based on low-dimensional representation vectors; Specifically, a coupling-sensitive subgraph is defined as a set of local lines in a line interaction graph formed due to strong negative coupling relationships between lines. The clustering process involves using low-dimensional representation vectors as input data and performing K-means clustering analysis. Initial cluster centers corresponding to the number of target clustering categories are randomly selected from the low-dimensional representation vectors. The Euclidean distance between the low-dimensional representation vector of each line and the initial cluster center is calculated. Based on the principle of minimizing Euclidean distance, the line is assigned to the category containing the nearest initial cluster center. The mean of the low-dimensional representation vectors of lines within each category is recalculated, and this mean is used as the new cluster center, and lines are reassigned. This process is repeated until the cluster center of each category no longer changes, or the change is less than a preset convergence threshold. The convergence threshold is determined experimentally, for example, by setting the convergence threshold to be less than 0.001 units of distance between two consecutive cluster center moves. After clustering analysis, each line in the line interaction graph is assigned to its corresponding category, and a corresponding coupling-sensitive subgraph is formed based on the clustering results.

[0030] Based on the coupling-sensitive subgraph, the cross nodes with the highest cumulative percentage of negative coupling weight are selected and sorted according to their negative coupling weight. Specifically, for each coupling-sensitive subgraph, all negative coupling weights corresponding to each intersection node identifier are extracted and accumulated. All intersection node identifiers within the coupling-sensitive subgraph are sorted in descending order of their accumulated values. The proportion of the accumulated negative coupling weight of each sorted intersection node identifier to the sum of all negative coupling weights is calculated. A screening threshold for the cumulative proportion of negative coupling weights is set. The screening threshold is determined by performing sensitivity analysis on historical garbage truck operation trajectory data, i.e., continuously accumulating the weights corresponding to the top-ranked intersection node identifiers from all negative coupling weights until the cumulative proportion reaches a specific percentage (e.g., 85%). This specific percentage is then set as the screening threshold. Finally, based on the screening threshold, intersection node identifiers whose cumulative proportion reaches or exceeds the screening threshold are selected, thus obtaining the intersection node identifier with the highest cumulative proportion of negative coupling weights.

[0031] Map the road segment connection relationship corresponding to the intersection node with the highest cumulative negative coupling weight to a set of key intersection segments; Specifically, the set of critical intersections is defined as the set of intersections that have the most significant impact on the negative coupling relationship between roads. The mapping method is as follows: based on the urban waste collection road network topology and road network association data, retrieve all road segment connection relationships associated with the intersection node identifier with the highest cumulative negative coupling weight. This yields the connection topology relationships of all road segments under the corresponding intersection node identifier. The obtained sets of road segment connection topologies are then merged and deduplicated to form the set of critical intersections. The set of critical intersections represents the road segments directly connected by the intersection node identifier with the most prominent negative coupling relationship in the road interaction diagram.

[0032] S4: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework to construct a scheduling policy model and output initial policy parameters, including: Based on historical waste production spatiotemporal distribution data, the waste volume characteristics of key intersections are extracted for time windows. Specifically, the set of key intersections is defined as the set of intersections that have the most significant impact on the negative coupling relationship between roads. Historical waste production spatiotemporal distribution data is defined as the distribution characteristics of waste generation for each road segment over a certain historical period. The time window waste volume characteristic is defined as the statistical characteristics of the cumulative waste volume generated on all road segments included in each set of key intersections within multiple preset time windows. The extraction method is as follows: First, determine the method for dividing the time windows, where the time window division is based on, but not limited to, the date and time of waste collection completion recorded in the historical waste production spatiotemporal distribution data. For example, selecting one hour or half hour as a time window. Then, for each road segment in each set of key intersections, calculate the waste volume within each determined time window, i.e., accumulate the waste weight recorded in the corresponding date and time window in the historical waste production spatiotemporal distribution data. Finally, for each set of key intersections, accumulate the waste volume of all corresponding road segments within the same time window again, thus forming the cumulative waste volume value for each key intersection set corresponding to each time window, i.e., obtaining the time window waste volume characteristic.

[0033] The time window waste volume characteristics are combined with the line identifiers associated with the key intersection segment set in the line interaction diagram to form a status feature; Specifically, the route interaction graph is defined as a network structure where intersection node identifiers are the nodes, road segment connections are the edges, and overlapping and intersection relationships are the weights of the edges. The route identifier is a unique marker for each route in the route interaction graph. The combination process is as follows: The route identifiers directly associated with each set of key intersection segments are extracted from the route interaction graph. The association method is to retrieve all routes passing through the corresponding intersection node identifier in the route interaction graph based on each intersection node identifier in the key intersection segment set, and record the route identifier for each route. The cumulative waste volume within each time window corresponding to each key intersection segment set is combined with the retrieved route identifiers to form a state feature including the cumulative waste volume and the route identifier. The state feature is represented using a vector representation. For each set of key intersection segments, a vector is constructed containing the cumulative waste volume for each time window and the route identifier corresponding to the cumulative waste volume, thus expressing the waste generation status and spatial distribution of each key intersection segment set under historical spatiotemporal conditions, and the corresponding route information.

[0034] In the multi-agent reinforcement learning framework, a scheduling policy model is constructed using state features as input and route selection actions as output. The training objective is defined by using the negative coupling weights corresponding to the negative coupling matrix as constraints to output the initial policy parameters. Specifically, route selection is defined as the garbage truck's choice of a garbage collection route for each set of key intersections within a specific time window. The multi-agent reinforcement learning framework is a machine learning model for simultaneous decision-making by multiple agents, implemented using a deep deterministic policy gradient algorithm. State features are input into the policy network of the deep deterministic policy gradient algorithm. The policy network is defined as a function approximator based on an artificial neural network structure. The number of neurons in the input layer of the policy network is set to the dimension of the state features. The number of hidden layers and neurons is determined through cross-validation using historical data, for example, by minimizing the validation loss. The number of neurons in the output layer of the policy network is set to the total number of selectable route identifiers. The activation function of each neuron in the output layer is normalized using the softmax function to output the probability value corresponding to each route selection action. An action network is defined to evaluate the actual garbage collection efficiency generated by each route selection action. The action network is also built based on an artificial neural network structure. Its input is the route selection action output by the policy network, and its output is the value function value corresponding to the action, representing the expected garbage collection efficiency after taking the corresponding route selection action. The parameter configuration of the action network is the same as that of the policy network, and the network size is determined through cross-validation. The policy network and action network together form the basic structure of the multi-agent reinforcement learning framework.

[0035] The training objective function is constructed as follows: the value function output by the action network is used as the positive reward, and the negative coupling weights are used as the penalty. The penalty is added to the training objective function to constrain the optimization direction of the route selection action. When the policy network outputs a route selection action, the negative coupling weights between the route and other routes are found through the negative coupling matrix. These negative coupling weights are then introduced into the objective function as a penalty, inhibiting the reinforcement learning framework from selecting route combinations with high negative coupling weights. This actively avoids negative coupling relationships between routes during the policy model training process. The objective function is expressed as the value function output by the action network minus the product of the negative coupling weights and the preset constraint coefficients. The preset constraint coefficients are determined through sensitivity analysis; for example, an initial constraint coefficient of 0.5 is set. Cross-validation is then performed based on historical garbage truck trajectory data and actual garbage collection efficiency data to select the constraint coefficient that maximizes the improvement in actual garbage collection efficiency as the final value; for example, the final constraint coefficient is set to 0.7. Once the training objective is defined, the training process can be executed within a multi-agent reinforcement learning framework. The training process uses the backpropagation algorithm to optimize the network parameters of the policy network and the action network. After training, the initial policy parameters for initial actual garbage truck scheduling can be obtained.

[0036] S5: In the simulation environment, a collaborative scheduling scheme is generated based on the initial strategy parameters. The negative coupling strength across lines during garbage truck scheduling is monitored in real time to incrementally correct the initial strategy parameters, resulting in a convergent scheduling strategy, including: In the simulation environment, a collaborative scheduling scheme is generated by performing multi-vehicle route selection and deduction on the route interaction diagram based on the initial strategy parameters. Specifically, the simulation environment is defined as a computer virtual space that simulates the entire process of garbage collection operations, based on the urban garbage collection road network topology, historical garbage production spatiotemporal distribution data, and garbage truck trajectory data. The initial policy parameters are defined as the policy network parameters obtained through joint training of the policy network and action network in a multi-agent reinforcement learning framework, used to guide the garbage truck route selection actions. These parameters include the set of connection weights between the input and hidden layers of the policy network; the set of connection weights between the hidden and output layers of the policy network; and the set of biases for neurons in each layer of the policy network. The simulation implementation is as follows: Initial policy parameters are input into the policy network. The policy network takes state features as input and outputs the probability value of each route selection action. Based on the probability value, the route selection action of each garbage truck for the set of key intersections is determined. According to the determined route selection action, the garbage truck's driving path is planned in the route interaction graph. The path planning method is a breadth-first search algorithm based on the node and edge weight attributes of the route interaction graph. Starting from the intersection node identifiers corresponding to the set of key intersections, the algorithm traverses the connection relationships of each road segment in the route interaction graph, selects the path according to the probability value output by the policy network, and records each road segment traversed. Simultaneously, in the simulation environment, each garbage truck is assigned initial position, driving speed, and dwell time parameters when collecting garbage. The initial position is determined based on historical garbage truck trajectory data. For example, the initial position of each garbage truck is set to the coordinate position of the first occurrence in the historical trajectory data. The driving speed is obtained by analyzing the historical garbage truck trajectory data to obtain the average speed of the garbage truck on each road segment, for example, set to 20 kilometers per hour. The garbage collection dwell time parameter is obtained by statistical analysis of historical garbage collection data, for example, the average dwell time for each garbage collection is 3 minutes. In the simulation environment, the garbage truck is simulated to travel along the planned path and collect garbage. The time sequence of each road segment and node passed by the garbage truck and the arrival and departure of the corresponding node are recorded, and then a collaborative scheduling scheme is generated. The collaborative scheduling scheme includes the planned path of all garbage trucks, the road segment coverage order and the time sequence information of each node.

[0037] Based on the collaborative scheduling scheme, the arrival and departure sequences of vehicles corresponding to the key intersection segments are recorded and the negative coupling strength across the lines is calculated. Specifically, the vehicle arrival and departure sequences are defined as time series consisting of the time points of each garbage truck arriving at and leaving each node in the key intersection set within the simulation environment. For each road segment within the key intersection set in the collaborative scheduling scheme, the time points of all garbage trucks passing through the corresponding road segment nodes are extracted and sorted according to the time in the actual simulation environment to form accurate vehicle arrival and departure sequences; the cross-line negative coupling strength is calculated, where the cross-line negative coupling strength is defined as the degree of negative impact on garbage collection efficiency caused by the spatial and temporal interaction of garbage trucks from different routes on the same road segment in the key intersection set; the time overlap of garbage trucks from different routes on the same road segment node time series is calculated, specifically: for any two routes corresponding to The arrival and departure sequences of vehicles are cross-referenced. If a vehicle arrives at the corresponding road segment before the previous vehicle has left, a time window of vehicle overlap is formed. The duration of all overlapping time windows is summed and divided by the total time of all garbage trucks passing through the corresponding road segment to form the time overlap ratio of the road segment. The time overlap ratio is then integrated with the total amount of garbage collected corresponding to the collaborative scheduling scheme. The total amount of garbage collected is used as the weight and multiplied by the time overlap ratio to form a cross-line negative coupling strength index with the amount of garbage collection task as the weight. Finally, the cross-line negative coupling strength of the road segments in each set of key intersections is obtained.

[0038] The initial strategy parameters are incrementally corrected and iterated cyclically based on the deviation between the cross-line negative coupling strength and the negative coupling matrix to obtain a convergent scheduling strategy. Specifically, the deviation between the cross-line negative coupling strength and the negative coupling matrix is ​​defined as the absolute value of the difference between the cross-line negative coupling strength calculated in the actual simulation environment and the corresponding elements of the pre-obtained negative coupling matrix. An incremental correction function is constructed, using the deviation as the input variable and the function output as the adjustment amount of the policy network parameters. The function is expressed as the output value of the incremental correction function equal to the deviation multiplied by a preset correction coefficient. The correction coefficient is determined through a sensitivity analysis method of historical waste collection efficiency and changes in policy network parameters. That is, multiple different correction coefficients are selected for incremental correction, and the collaborative scheduling scheme is re-deduced. Then, based on the increase in waste collection efficiency in the actual simulation environment, the correction coefficient that maximizes the increase in waste collection efficiency or converges the fastest is selected. For example, [the following is a possible interpretation of the original text]. The correction coefficient is set to 0.05. The output value of the incremental correction function is used to adjust the initial strategy parameters, that is, the current strategy network parameters are added to the output value of the incremental correction function to form the adjusted strategy parameters. Finally, the simulation environment is re-performed and the cross-line negative coupling strength is calculated again with the adjusted strategy parameters. The iteration is repeated until the deviation between the cross-line negative coupling strength and the negative coupling matrix is ​​less than the predetermined convergence threshold. The convergence threshold is determined by the stability analysis method based on the historical garbage collection efficiency. The deviation value that makes the change of garbage collection efficiency less than a certain proportion after several consecutive iterations is selected as the convergence threshold. For example, the convergence threshold is set to a deviation value less than 0.01. At this time, the scheduling strategy is considered to have reached the convergence state and a converged scheduling strategy is obtained.

[0039] S6: Determine the departure time and route of garbage trucks based on a convergent scheduling strategy, and write the departure time and route back to the route interaction graph for iterative optimization, including: The garbage truck departure time is determined based on the convergence scheduling strategy, and the garbage truck travel route is planned according to the route interaction diagram. Specifically, the convergence scheduling strategy is defined as the strategy network parameters after multiple iterative incremental corrections in the simulation environment. When the deviation between the cross-line negative coupling strength and the negative coupling matrix meets the preset convergence threshold condition, it is determined to be in a convergence state, thus obtaining the convergence scheduling strategy. The method for determining the garbage truck departure time is as follows: based on the cumulative garbage generation in each time window recorded in the historical garbage production spatiotemporal distribution data, the first time window where the cumulative garbage generation exceeds the preset garbage collection start threshold is selected as the initial departure time of the garbage truck. The method for setting the garbage collection start threshold is to determine the relationship between the historical cumulative garbage generation and the actual garbage truck start collection operation through statistical analysis of historical garbage truck operation trajectory data and garbage production data. For example, the garbage collection start threshold is set to 70% of the historical maximum cumulative garbage generation of the corresponding road segment. The selection actions of each garbage truck route corresponding to the convergence scheduling strategy are extracted sequentially. The cumulative amount of garbage is calculated, and the subsequent departure interval of the garbage trucks is determined based on the cumulative amount of garbage and the garbage truck collection efficiency. The garbage truck collection efficiency is calculated by using the ratio of the actual weight of garbage collected by each garbage truck in the historical garbage truck operation trajectory data to the corresponding time consumption as an efficiency reference. For example, if the average garbage collection efficiency of each garbage truck is determined to be 1 ton per hour through statistical analysis, and the cumulative amount of garbage generated in the next time window reaches 3 tons, then the next departure time of the garbage truck is set to 3 hours after the initial departure time, and so on. The above steps are repeated to determine the continuous departure time of all garbage trucks, thereby obtaining the garbage truck departure timetable.

[0040] The method for planning garbage truck routes based on the route interaction diagram is as follows: Based on the segment connection relationships, intersection node identifiers, and route selection actions output by the convergence scheduling strategy defined in the route interaction diagram, a route planning algorithm is used to determine the garbage truck routes. The intersection node identifiers corresponding to the set of key intersection segments in the route interaction diagram are used as the planning starting point. The probability values ​​of the route selection actions output by the convergence scheduling strategy are used as the basis for route selection. Adjacent segments are sequentially searched along the segment connection relationships in the route interaction diagram, and the cumulative probability value is calculated. The segment with the highest cumulative probability value is selected as the garbage truck route. When planning the route, the segment coverage order during the garbage truck's journey and the actual load capacity of the garbage truck are considered simultaneously. Specifically, the load capacity of the garbage truck is set to the maximum carrying capacity recorded in the historical garbage truck operation trajectory data, for example, 10 tons. When the cumulative garbage amount in the route approaches the maximum carrying capacity of the garbage truck, the route is truncated, and a route back to the garbage treatment station is planned. A new garbage truck is then started to continue the garbage collection task for the remaining segments. The above method is used to determine the routes for all garbage trucks, thus obtaining the garbage truck route planning results.

[0041] Garbage trucks are dispatched to collect garbage based on their departure times and routes, and the corresponding garbage truck trajectory data is recorded. Specifically, the determined garbage truck departure schedule and route planning results are distributed to each garbage truck in the actual garbage collection operation. Each garbage truck starts the collection operation according to the preset departure time and executes the planned route. During the journey, the garbage truck continuously records real-time garbage truck location data and corresponding timestamps through the onboard GPS system. The recording interval is set to once every minute, forming garbage truck trajectory data. The trajectory data includes the garbage truck's latitude and longitude coordinates and corresponding real-time timestamps. At the same time, when the garbage truck is actually performing garbage collection operations, the cumulative garbage collection volume and cumulative time consumed when the garbage truck arrives at each road segment node are recorded simultaneously.

[0042] After matching the garbage truck's driving trajectory data with a map, a sequence of trajectory segments is formed and updated to the route interaction map for the next round of iteration and optimization. Specifically, for each record of garbage truck trajectory data, the coordinate points are processed using a nearest neighbor distance matching algorithm. The algorithm searches the garbage collection road network topology for the road segment centerline closest to the current trajectory coordinate point. Then, it calculates the vertical spatial distance between the trajectory coordinate point and the road segment centerline, comparing this vertical spatial distance with a preset spatial matching threshold. The spatial matching threshold is determined by analyzing the positioning accuracy of historical garbage truck trajectory data. For example, a spatial matching threshold of 50 meters is set. When the vertical distance between the trajectory coordinate point and the road segment centerline is less than or equal to 50 meters, the trajectory coordinate point is considered a successful match and is marked as belonging to the corresponding road segment. All coordinate points in the garbage truck trajectory data are processed sequentially using the above matching method to form a sequence of garbage truck trajectory segments. This sequence represents a continuous record of the connection relationships between the actual road segments traversed by the garbage truck.

[0043] Based on the vehicle passing frequency and driving order of each road segment node recorded in the trajectory segment sequence, the repetition coverage relationship and intersection node identifier in the route interaction graph are updated. If multiple garbage trucks pass through a certain road segment in the trajectory segment sequence, the weight value of the corresponding road segment repetition coverage relationship in the route interaction graph is updated, that is, the garbage truck coverage frequency of the road segment is increased, and after the frequency is updated, it is normalized again to obtain a new repetition coverage relationship weight. If multiple garbage trucks are found to converge at a certain road segment node in the trajectory segment sequence and continue to travel in different directions, the weight value of the corresponding intersection node identifier in the route interaction graph is updated, that is, the frequency of the intersection at the corresponding intersection node is increased, and normalized. The updated weight attributes in the route interaction graph will become the initial input data for the next round of reinforcement learning strategy optimization training, used to optimize the intelligent scheduling strategy of garbage trucks and complete the iterative closed loop.

[0044] Example 2 The difference between Embodiment 2 and Embodiment 1 is that this embodiment introduces a garbage truck intelligent scheduling system based on reinforcement learning.

[0045] Figure 2 A schematic diagram of the intelligent garbage truck scheduling system based on reinforcement learning of the present invention is given. The intelligent garbage truck scheduling system based on reinforcement learning includes: Data construction module: acquire urban waste collection road network topology, historical waste production spatiotemporal distribution data and garbage truck operation trajectory data, and construct a route interaction map including intersection and overlapping coverage relationships; Coupling Calculation Module: Calculates the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generates a negative coupling matrix; Clustering analysis module: Performs spectral clustering on the negative coupling matrix, divides the coupling-sensitive subgraph, and extracts the set of key intersection segments; Strategy modeling module: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework, construct the scheduling strategy model and output the initial strategy parameters; Simulation optimization module: Generates a collaborative scheduling scheme based on the initial strategy parameters in the simulation environment, monitors the cross-line negative coupling strength in real time during garbage truck scheduling, incrementally corrects the initial strategy parameters, and obtains a convergent scheduling strategy; Scheduling execution module: Based on the convergent scheduling strategy, the departure time and driving route of garbage trucks are determined, and the departure time and driving route of garbage trucks are written back to the route interaction graph for iterative optimization.

[0046] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0047] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0048] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0049] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0050] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0051] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0052] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0054] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent scheduling of garbage trucks based on reinforcement learning, characterized in that, Includes the following steps: S1: Obtain the urban waste collection road network topology, historical waste production spatiotemporal distribution data, and garbage truck operation trajectory data to construct a route interaction map that includes intersection and overlapping coverage relationships; S2: Calculate the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generate a negative coupling matrix; S3: Perform spectral clustering on the negative coupling matrix to divide the coupling-sensitive subgraph and extract the set of key intersection segments; S4: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework, construct the scheduling strategy model and output the initial strategy parameters; S5: Generate a collaborative scheduling scheme based on the initial strategy parameters in the simulation environment, monitor the cross-line negative coupling strength in real time during garbage truck scheduling, incrementally correct the initial strategy parameters, and obtain a convergent scheduling strategy. S6: Determine the departure time and driving route of garbage trucks based on the convergence scheduling strategy, and write the departure time and driving route of garbage trucks back to the route interaction graph for iterative optimization.

2. The intelligent scheduling method for garbage trucks based on reinforcement learning according to claim 1, characterized in that, S1, specifically: Obtain the topology of the urban waste collection road network and unify spatial reference; extract road segment connection relationships and intersection node identifiers to form road network association data. Acquire historical waste production spatiotemporal distribution data and aggregate it into road network associated data according to the intersection node identifier; Obtain garbage truck trajectory data and perform map matching to obtain trajectory segment sequences; Based on the trajectory segment sequence, the system identifies the intersection and overlapping coverage relationships and generates a route interaction map.

3. The intelligent scheduling method for garbage trucks based on reinforcement learning according to claim 2, characterized in that, S2, specifically: Based on the intersection node identifiers and overlapping relationships in the line interaction graph, a line association set with the intersection node identifiers as the propagation starting point is constructed. Based on the line association set, diffusion calculation is performed along the connection direction of the line interaction graph to count the degree of coverage overlap of cross node identifiers in different lines and the frequency of overlap of running trajectories. After normalizing the degree of coverage overlap and the frequency of overlap of the running trajectory, the data is mapped to negative coupling weights between lines and arranged according to the correspondence between the lines to form a negative coupling matrix.

4. The intelligent scheduling method for garbage trucks based on reinforcement learning according to claim 3, characterized in that, S3, specifically: Calculate the Laplacian matrix corresponding to the negative coupling matrix; Eigenvalue decomposition of the Laplacian matrix yields a low-dimensional representation vector for clustering; Coupled sensitive subgraphs are obtained by clustering based on low-dimensional representation vectors; Based on the coupling-sensitive subgraph, the cross nodes with the highest cumulative percentage of negative coupling weight are selected and sorted according to their negative coupling weight. Map the road segment connection relationship corresponding to the intersection node with the highest cumulative negative coupling weight to a set of key intersection segments.

5. The intelligent scheduling method for garbage trucks based on reinforcement learning according to claim 4, characterized in that, S4, specifically: Based on historical waste production spatiotemporal distribution data, the waste volume characteristics of key intersections are extracted for time windows. The time window waste volume characteristics are combined with the line identifiers associated with the key intersection segment set in the line interaction diagram to form a status feature; In the multi-agent reinforcement learning framework, a scheduling policy model is constructed using state features as input and route selection actions as output. The training objective is defined by using the negative coupling weights corresponding to the negative coupling matrix as constraints to output the initial policy parameters.

6. The intelligent scheduling method for garbage trucks based on reinforcement learning according to claim 5, characterized in that, S5, specifically: In the simulation environment, a collaborative scheduling scheme is generated by performing multi-vehicle route selection and deduction on the route interaction diagram based on the initial strategy parameters. Based on the collaborative scheduling scheme, the arrival and departure sequences of vehicles corresponding to the key intersection segments are recorded and the negative coupling strength across the lines is calculated. The initial strategy parameters are incrementally corrected and iterated cyclically based on the deviation between the cross-line negative coupling strength and the negative coupling matrix to obtain a convergent scheduling strategy.

7. The intelligent scheduling method for garbage trucks based on reinforcement learning according to claim 6, characterized in that, S6, specifically: The garbage truck departure time is determined based on the convergence scheduling strategy, and the garbage truck travel route is planned according to the route interaction diagram. Garbage trucks are dispatched to collect garbage based on their departure times and routes, and the corresponding garbage truck trajectory data is recorded. After matching the garbage truck's driving trajectory data with the map, a sequence of trajectory segments is formed and updated to the route interaction map for the next round of iteration and optimization.

8. A reinforcement learning-based intelligent garbage truck scheduling system, used to implement the reinforcement learning-based intelligent garbage truck scheduling method according to any one of claims 1-7, characterized in that, include: Data construction module: acquire urban waste collection road network topology, historical waste production spatiotemporal distribution data and garbage truck operation trajectory data, and construct a route interaction map including intersection and overlapping coverage relationships; Coupling Calculation Module: Calculates the negative coupling coefficient of each intersection segment to the scheduling strategy of adjacent lines based on the line interaction diagram, and generates a negative coupling matrix; Clustering analysis module: Performs spectral clustering on the negative coupling matrix, divides the coupling-sensitive subgraph, and extracts the set of key intersection segments; Strategy modeling module: Input the set of key intersections and the spatiotemporal distribution data of historical waste production into the multi-agent reinforcement learning framework, construct the scheduling strategy model and output the initial strategy parameters; Simulation optimization module: Generates a collaborative scheduling scheme based on the initial strategy parameters in the simulation environment, monitors the cross-line negative coupling strength in real time during garbage truck scheduling, incrementally corrects the initial strategy parameters, and obtains a convergent scheduling strategy; Scheduling execution module: Based on the convergent scheduling strategy, the departure time and driving route of garbage trucks are determined, and the departure time and driving route of garbage trucks are written back to the route interaction graph for iterative optimization.

Images