Urban garbage collection and transportation method and system based on reinforcement learning
By building a driving and path model of garbage collection and transportation vehicles based on reinforcement learning, optimizing the garbage collection and transportation network, solving the problem of inefficient existing garbage collection and transportation methods, and achieving more efficient and environmentally friendly garbage disposal.
Patent Information
- Application Number
- CN202411255382.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing urban garbage collection and transportation methods rely on simple empirical data, resulting in inefficient garbage collection and transportation, lack of scientificity and systemicity, and lead to blindness and uncertainty in decision-making.
Using reinforcement learning methods, the driving model and path model of garbage collection trucks and transceiver points is constructed through Markov decision-making process, the state space, action space and reward functions are defined, and the garbage collection network is optimized by using Q learning algorithms to adjust the driving. Plans and transportation point arrangements improve the system's intelligence and independent learning capabilities.
It improves the accuracy and adaptability of garbage collection and transportation path planning, reduces the mileage and time of garbage collection and transportation vehicles, reduces fuel consumption and operating costs, reduces traffic congestion and carbon emissions, and has significant economic and environmental benefits.
Smart Images

Figure CN119026779A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of smart cities, and in particular to a method and system for collecting and transporting urban garbage based on reinforcement learning. Background Art
[0002] With the continuous acceleration of urbanization, the scale of urban domestic waste has shown a continuous growth trend, which is crucial for the maintenance of the urban environment and the guarantee of the quality of life of residents. However, the current waste disposal system faces many problems and challenges. There is inefficiency in waste collection, transportation, and treatment, which leads to waste of resources and environmental pollution, and the cost of waste disposal is high. In addition, the current waste management is often based on experience and simple statistical data, lacking scientificity and systematicity, which leads to blindness and uncertainty in decision-making. Summary of the invention
[0003] In view of this, in order to solve the technical problem that most of the existing urban garbage collection and transportation methods are based on simple empirical data, which leads to low garbage collection and transportation efficiency, the present invention proposes an urban garbage collection and transportation method based on reinforcement learning, which includes the following steps:
[0004] Based on the Markov decision process, a driving model of garbage collection and transportation vehicles and collection and delivery points is constructed;
[0005] Based on the driving model and according to the optimization objective, a first state space, a first action space and a first reward function are defined;
[0006] Based on Markov decision process, the path model is constructed;
[0007] Based on the path model, limiting the search space, dividing the second state space, the second action space and the second reward function;
[0008] Based on the driving model and the path model, the garbage collection and transportation network is optimized, and the driving plan and the layout of the collection and transportation points are adjusted.
[0009] In this embodiment, the Q learning algorithm is introduced to enable the garbage collection and transportation network to have a certain degree of intelligence and autonomous learning capabilities, and to make reasonable decisions based on historical experience and real-time data, thereby improving the intelligence level and automation level of the system.
[0010] In some embodiments, the step of constructing a driving model of the garbage collection vehicle and the collection and delivery point based on the Markov decision process specifically includes:
[0011] The connectivity of streets in the garbage collection and transportation network, the locations of garbage collection vehicles and the locations of sending and receiving points are obtained, the garbage collection and transportation network is modeled as a Markov decision process, and the driving model of garbage collection vehicles and sending and receiving points is obtained.
[0012] In some embodiments, the step of defining a first state space, a first action space, and a first reward function based on the driving model and according to an optimization objective specifically includes:
[0013] Based on the driving model, a path is optimized with the goal of reducing the driving distance of the garbage collection and transportation vehicle in the garbage collection and transportation network;
[0014] The first state space includes the road network vertices where the garbage collection and transportation vehicles are located;
[0015] The first action space includes a departure edge corresponding to each vertex;
[0016] The first reward function is defined based on the cost of the selected edges.
[0017] In some embodiments, the step of limiting the search space based on the path model, dividing the second state space, the second action space and the second reward function specifically includes:
[0018] Based on the driving model, all possible paths from the initial state to the terminal state are calculated, and a predefined subset is determined to obtain a search space;
[0019] The second state space includes an initial state, a terminal state, and a time step;
[0020] The second action space includes paths in a predefined subset;
[0021] The second reward function is defined according to path cost and traffic cost.
[0022] In some embodiments, the first state space is represented as follows:
[0023] S=[m 11 v 11 D t1 L t1 , n 12 v 12 D t1 L t1 ....m ij v ij D tn L tn ,,] i=1,2...n; j=1,2...n
[0024] Where n is the number of redundant vehicles on this road, v is the average speed of the current road, and D t L is the total distance between the collection point and the vehicle on the two-dimensional plane. tis the distance from the edge of the current road end, i is the current road, j is the car identifier on the road, and n is the total number of lanes.
[0025] The present invention also proposes a city garbage collection and transportation system based on reinforcement learning, the system comprising:
[0026] The first deep learning module constructs a driving model of the garbage collection and transportation vehicle and the collection and delivery point based on the Markov decision process; based on the driving model and according to the optimization goal, defines a first state space, a first action space and a first reward function;
[0027] A second deep learning module constructs a path model based on a Markov decision process; based on the path model, divides a second state space, a second action space, and a second reward function;
[0028] The optimization module optimizes the garbage collection and transportation network based on the driving model and the path model, and adjusts the driving plan and the layout of the collection and transportation points.
[0029] The present invention also proposes a city garbage collection and transportation device based on reinforcement learning, comprising:
[0030] at least one processor;
[0031] at least one memory for storing at least one program;
[0032] When the at least one program is executed by the at least one processor, the at least one processor implements the urban garbage collection and transportation method based on reinforcement learning as described above.
[0033] Based on the above scheme, the present invention provides a method and system for urban garbage collection and transportation based on reinforcement learning, which utilizes the Q-learning algorithm and can dynamically adjust the path according to the real-time road conditions and environmental changes, thereby improving the precision and accuracy of path planning; the driving model and path model based on Q-learning can evaluate and adjust the current road conditions in real time, so that the garbage collection and transportation vehicles can make decisions based on the latest information, thereby more effectively responding to emergencies such as traffic congestion and road closures, thereby improving the real-time and responsiveness of the system; finally, the present invention can effectively reduce the mileage and time of garbage collection and transportation vehicles, reduce fuel consumption and operating costs, and reduce traffic congestion and carbon emissions by optimizing the garbage collection and transportation path, with significant economic and environmental benefits.
[0034] In addition, traditional path planning methods often lack adaptability to environmental changes, while the Q-learning-based model can adaptively adjust the garbage collection and transportation routes through continuous learning and optimization to adapt to traffic conditions in different time periods and regions, thereby improving the adaptability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flowchart of the steps of a method for collecting and transporting urban garbage based on reinforcement learning of the present invention;
[0036] Figure 2 is a flow chart of the Q learning process of the driving edge of the present invention;
[0037] Figure 3 It is a path decision diagram of the garbage collection and transportation station of the present invention;
[0038] Figure 4 is a flow chart of the path-based Q learning algorithm of the present invention;
[0039] Figure 5 It is a structural block diagram of a city garbage collection and transportation system based on reinforcement learning of the present invention. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.
[0042] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.
[0043] Unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.
[0044] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0045] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.
[0046] Reference Figure 1 , is a flow chart of an optional example of a method for collecting and transporting urban garbage based on reinforcement learning proposed in the present invention. The method can be applied to computer equipment. The method for collecting and transporting urban garbage proposed in this embodiment may include but is not limited to the following steps:
[0047] Step S1: constructing a driving model of garbage collection and transportation vehicles and collection and delivery points based on a Markov decision process;
[0048] Step S2: Based on the driving model and according to the optimization objective, define a first state space, a first action space and a first reward function;
[0049] Step S3: constructing a path model based on the Markov decision process;
[0050] Step S4: Based on the path model, limit the search space, divide the second state space, the second action space and the second reward function;
[0051] Step S5: Based on the driving model and the path model, the garbage collection and transportation network is optimized, and the driving plan and the layout of the collection and transportation points are adjusted.
[0052] Consider that traffic assignment methods use unrealistic assumptions from the perspective of individual decision-making (algorithms based on metaheuristics). Applying MARL to the modeling of individual decision-making processes may be a reasonable approach to this problem. However, considering that such fields usually have a large number of agents and a large search space, such as the Guangzhou garbage collection and transportation network to be optimized in this paper, the curse of dimensionality may have an adverse impact on the final results. To this end, we propose two different conceptual methods for solving TAP here. Both methods model the decision-making process of the agent as an MDP (Markov decision process) and do not consider joint actions. Edge-based Q-learning does not restrict the learning search space of the agent, instead, it enables the agent to experience all possible paths. Therefore, the learning task is more complicated due to the large search space. On the other hand, path-based Q-learning restricts the search space to a subset of pre-computed paths.
[0053] The present invention regards each trip in the OD matrix as a learning agent d∈D. Each agent can learn a path from the origin to the destination. In TAP, from a learned policy π, the agent can extract a path connecting an initial state s- to a terminal state s+. For both methods proposed here, the learning process is organized in episodes (trials) and time steps (time units). An episode i is measured in t max The maximum number of time steps ends, or when all agents reach their terminal state s+ (whichever arrives first). Since the traffic assignment process here is macroscopic, the time units are discrete. The number of time steps per unit t≤t max represents a jump in the network. For example, the path p = (v0, v1, v2) connecting v0 to v2 requires t = (|p|-1) time steps.
[0054] In some feasible embodiments, the steps S1 and S2 are Q-learning based on driving edges, specifically including:
[0055] This approach is called "edge-based" because when an agent uses this approach to optimize learning, it can find a policy from the set of all policies available in the search space. The MDP model of the agent is as follows: a state s is a vertex v∈V in the road network where the agent is located. Each state s∈S has a set of available actions A(s), represented by the outgoing edges from the corresponding vertex s. The reward received by the agent is R(s, a) = -c e Given, where e is the edge corresponding to action a in state s.
[0056] If we consider that the agent can drive in cycles, this approach may have an infinite number of policies π∈Π from an initial state s- to a terminal state s+. To handle this problem, the agent’s search space is limited by the number of time steps tmax Therefore, if the agent is in t max If no path from s- to s+ is found in the number of time steps, the current episode will stop. Even if this stopping policy is part of the set of invalid policies Π, it is a possible solution experienced by the agent in the search space. In this way, the set of all possible policies, i.e., the Q-learning agent based on the driving edge, is defined as Eq.3. Here j=1,...,N i , M is the number of time steps in the current simulation, N i For i the number of policies from s- to s+ in the step; j=1,...,N′ i , M′ is the number of time steps in the current simulation, N′ i For i Step s i arrive The number of policies, driving edge Q learning process flow chart see Figure 2 In this method, Π has an important property
[0057] Π(s-, s+, t max )=Π valid ∪Π invalid
[0058] In this embodiment, based on the Markov decision process, a driving model of the garbage collection truck and the collection and delivery point is established. First, the connectivity of each street and each garbage truck are located, and the model is built in combination with the location of the garbage collection point. The decision-making of the garbage truck during driving is analyzed below. According to the optimization goal of this article (i.e., minimizing the driving distance of the garbage collection truck in the collection network and optimizing the path), the first state space, the first action space and the first reward function are defined respectively.
[0059] First state space:
[0060] In the garbage collection and transportation network, Q learning based on driving edges is mostly used in the garbage trucks and garbage collection points on the map, road topology, real-time traffic conditions and other vehicle information. In this multi-dimensional information interconnected system, garbage collection points will be matched and analyzed with transportation vehicles in real time to calculate the location information of garbage collection points, the amount of garbage generated, the frequency of cleaning, etc.; as well as the information of transportation vehicles, such as quantity, capacity, speed, etc., which is the information input of the target and the terminal. At the same time, Q learning will record the current road network information, and make corresponding judgments on the driving choices of garbage trucks through scalars such as road topology, road length, traffic flow, road status (smooth, congested), and intersection information. Let n be the redundant number of social vehicles on this road, v is the average speed of the current road, and define the state vector as<n,v> , when n is larger, the vehicle speed will be smaller. Considering the distance between the garbage collection point and the current vehicle, define D t L is the total distance between the collection point and the vehicle on the two-dimensional plane. t is the distance from the edge of the current road end. Finally, the state vector of this model is defined as:
[0061] S=[m 11 v 11 D t1 L t1 , n 12 v 12 D t1 L t1 ....m ij v ij D tn L tn ,,],i=1,2...nj=1,2...n
[0062] Where i is the current road, j is the car ID on the road, n is the total number of lanes, and the dimension of the state vector is determined by the value of n. The larger n is, the higher the dimension of the state vector is, and the more complex the corresponding model is.
[0063] First action space:
[0064] Each vertex in the road network is taken as a state, and each state represents the current position of the vehicle. Define feasible paths as actions, that is, the driving paths that the vehicle can choose, and record and adjust them according to the road edge information. According to the selected path and driving edge conditions, adjust the incentive parameters of the reward function, and use random operations appropriately when falling into a worse situation to find new solutions for planning. Use the Q learning algorithm to update the Q value according to the reward. After each step of action, update the Q value according to the reward and learning rate, and select the action with the maximum Q value as the next action strategy. Through continuous practice and feedback, optimize the Q value so that the system can learn the best driving strategy to achieve the goal of maximizing garbage collection efficiency and optimizing transportation routes. The final optimization strategy can guide garbage collection and transportation vehicles to travel efficiently in the network, reduce transportation time and cost, and improve overall transportation efficiency and service quality.
[0065] First reward function:
[0066] Real-time reward operation is an important method for supervised learning to guide the target to perform optimization operations. A reward function can be defined to evaluate the quality of each action. According to the actual situation, 1) choose a smooth road to get positive rewards, and choose a congested road to be punished; 2) After choosing a new road, the actual distance to the target point is reduced to get positive rewards, otherwise it will be punished; ultimately, it helps garbage transport vehicles to guide and reinforce learning, and ultimately the long-term benefits are smooth roads and rapid completion of tasks. At the same time, the ε-greedy strategy is adopted to select random actions with a certain probability in order to explore unknown areas and find the optimal solution for driving choices under mutual influence. The reward function is designed as:
[0067] r sum =r d +r v
[0068] where r d is the reward that the garbage truck gets for making a driving decision at each intersection, and r v is the reward for the current driving state. Assume that the garbage truck will be punished if it encounters a congested road or other situation that causes it to stop and wait, i.e., changes its driving state. If it passes normally, no reward is required. The reward examples are as follows:
[0069]
[0070] And r dAfter each driving decision control, the reward is given by comparing the current distance with the target plane distance. Considering the situation that there are many changes in the roads in the city, there is a better road after the current decision, but the current decision will make the total path length longer. In this case, a controllable space such as 50 meters can be given as a tolerance space without reward. Finally, dynamic rewards are given according to the changes in various distances. The reward function is as follows:
[0071] D t >0 means increasing distance
[0072] If after taking the selected control action, the social vehicle congestion on each lane, red light interference, etc. are all small, or the distance between the targets is continuously shortened, an additional reward will be added. This will help to better achieve congestion reduction and optimize purely for path optimization.
[0073] In a real-time randomly changing traffic environment, the system state has strong uncertainty and unpredictability. It is very difficult to directly and accurately model the impact of real-time interference between garbage trucks and social vehicles. Therefore, it is impossible to obtain the Markov decision process to directly solve the required state transfer function. Considering the actual non-idealized environment, the artificial static-to-dynamic method can reduce the accuracy loss caused by this part, which can be used for low-speed driving or r d The congestion state in the dynamic time range is defined, and the penalty is cancelled within this range. If the time exceeds continuously, the penalty is doubled, and the vehicle is expected to leave the congested section as soon as possible to find other paths. Modeling the rail-highway intersection for the garbage collection and transportation station route decision, such as Figure 3 As shown, the figure includes content such as rewards and punishments for selecting various decisions.
[0074] In some feasible embodiments, steps S3 and S4 are path-based Q learning, which specifically include:
[0075] Unlike (traveling) edge-based Q-learning, in path-based Q-learning the agent’s search space Π is limited to a predefined subset of possible solutions, as shown in Eq. 4. j is the path with the lowest cost in the jth series. This part of the paths is preprocessed before the learning process starts. Therefore, each agent receives |Π| precomputed paths from s- to s+.
[0076] Π (s-,s+) ={π i ,π2,…,π j}
[0077] The set of operations is defined in terms of the number of predefined paths used. Here we have to define a parameter K = |Π| to determine the number of path computations. These paths are the K-lowest cost paths for the OD pairs of agents without congestion. These paths are computed in the path computation graph using the shortest path finding algorithm K, which finds the shortest paths for K OD pairs without cycles.
[0078] The MDP model of the agent is as follows. The agent's state set contains only the initial state and the terminal state. In its initial state, the agent (s-) = |Π| action. Each action is a path connecting s- and s+. When the agent reaches the terminal state, it receives a reward determined by the cost of the path traveled, which is formulated as Here π a Represents the corresponding strategy for taking action a∈A(s-), and its algorithm flow chart is as follows Figure 4 shown.
[0079] Second state space:
[0080] In the garbage collection and transportation network path project, defining the state space is one of the key steps in optimizing the garbage collection path. Considering the complexity of Guangzhou's urban road network and the time factor in the garbage collection and transportation process, the state space is divided into discrete and meaningful states, and the concept of time steps is introduced.
[0081] When defining the state space, we need to consider the number of states. Assume that the state space contains N states, and the initial state can be the starting point of the garbage truck, denoted by s start , and the terminal state can be the end point of the garbage truck reaching the destination, denoted as s end In terms of state transition, the agent moves to the next state according to the chosen path. Therefore, the transition between states is usually deterministic. Suppose the agent moves from state s i After taking action a, transition to state s j , then we can use the transfer function T(s i ,a)=s j To represent the state transition process. The state is defined as the location of the garbage truck or the path segment it has passed, at a specific location in the Guangzhou urban road network. In order to convert the city map into an operational state space, it is divided into grids or regions. Each state can be represented by an integer or vector, and the location of the garbage truck is represented by two-dimensional grid coordinates. This division method enables the state space to cover all important areas and paths of the Guangzhou urban road network. Taking into account the time factor in the garbage collection and transportation process, we introduce the concept of time steps. Each time unit is regarded as a jump in the network, representing the travel time of the garbage truck on the path. In this way, the time step t can be expressed as a discrete unit to measure the time experienced by the garbage truck during the collection process.
[0082] Finally, the state space of the Guangzhou garbage collection and transportation network path project can be expressed as a set of discrete and meaningful states S = {(s start ,s end ,t)t≤T}, where each state s i It can be represented by an integer or a vector. The transition between states is represented by the transition function T(s i ,a)=s j Control. Among them, s start and end It is a node in the network, representing the starting and ending points of the garbage collection and transportation network; t i is the time step, representing the time the garbage truck travels on the path; T is the maximum number of time steps, that is, the maximum number of time steps at which the episode ends. The size of the state space depends on the division of the grid or region and the number of time steps.
[0083] Second action space:
[0084] Considering the complexity of the urban road network and the diversity of the garbage collection and transportation process, the definition of actions needs to fully consider the actual situation and have a certain degree of flexibility. Therefore, the control actions at each time step mainly include going to a specific garbage generation point to collect garbage, returning to the garbage treatment facility to unload garbage, choosing a detour path, and waiting for traffic to clear. Garbage collection vehicles need to choose to go to a specific garbage generation point to collect garbage to meet the garbage collection needs. When the garbage collection vehicle has collected a certain amount of garbage, it needs to choose to return to the garbage treatment facility to unload garbage to release vehicle capacity and continue to collect garbage. In addition, in order to cope with traffic congestion or road closures, garbage collection vehicles can choose a detour path to ensure the smooth progress of garbage collection. Finally, when encountering traffic accidents or other temporary traffic jams, garbage collection vehicles can choose to wait at the intersection for traffic to clear to ensure safe passage. The selection of these actions depends on the current state and environmental conditions. By flexibly combining and adjusting these actions, the garbage collection path can be effectively optimized and the garbage collection efficiency can be improved.
[0085] Second reward function:
[0086] As an important part of the trial-and-error-correction mechanism, the design of real-time rewards is crucial. It is used to guide the controller to learn effectively to obtain the best action selection strategy. As a scalar value, the reward must be designed as a function of the environment state to guide the learning process towards the final control goal. As mentioned above, the reward includes two parts: 1) The impact of traffic conditions on the garbage collection path, including the cost of choosing a detour to avoid traffic congestion or road closures, and the cost of waiting for traffic to clear when encountering traffic accidents or other temporary traffic jams. 2) Consideration is given to whether the garbage collection vehicle stops. The reward function is designed as:
[0087] R(s,a)=α·C path (s,a)+β·C traffic (s,a)
[0088] Where s is the current state; a is the action taken by the agent; C path (s,a) is the path cost corresponding to the agent taking action a, C traffic (s,a) is the traffic cost corresponding to the agent taking action a when facing traffic conditions; α and β are the weight coefficients of path cost and traffic cost, which are used to adjust the impact of different costs on rewards. Traffic cost part C traffic The situation of whether the garbage collection and transportation vehicle stops is taken into consideration. If the garbage collection and transportation vehicle does not stop due to traffic conditions or traffic flow problems during the driving process, the reward value of the path cost is zero; on the contrary, if the vehicle stops, the reward value of the path cost will be subject to a large negative penalty.
[0089]
[0090] Path Cost Part C path The driving path cost is introduced to take into account the impact of garbage collection and transportation vehicles on traffic flow. In cases of traffic congestion or road closures, the agent can choose a detour to avoid unnecessary delays, however, this will also incur additional costs. The reward function also considers whether the time step exceeds a preset threshold and imposes a penalty based on this.
[0091] t i >0,t i is the time step
[0092] The reward function takes into account the complexity and diversity of actual situations, and fully considers the characteristics of urban road networks and various situations in the garbage collection and transportation process. The flexibility and adjustability of the reward function enable the agent to flexibly choose actions according to different environmental conditions to maximize the optimization of the garbage collection path. In addition, the reward function balances rewards and penalties, taking into account the impact of whether the time step exceeds the preset threshold, and imposes penalties on this basis to encourage the agent to complete the task as soon as possible or avoid unnecessary waiting. path The design takes into account the complexity of the actual situation and maintains a balance between rewards and penalties, so that the agent can achieve good learning results in a variety of different situations, thereby improving the efficiency of garbage collection and transportation.
[0093] The technical solution involves the research on optimization model of garbage collection and transportation system in the integrated management system of urban domestic waste, deep learning, reinforcement learning and traffic distribution problems. In terms of the optimization of urban garbage collection and transportation system, the existing collection and transportation facilities are optimized according to their scale, quantity and location, the garbage distribution scheme between various treatment facilities is optimized, and the collection and transportation route is optimized. By using methods such as linear programming, mixed integer programming and various heuristic algorithms, the goals of reducing transportation costs and maximizing environmental benefits can be achieved. In terms of deep learning and reinforcement learning, the driving decision-making process of garbage trucks is optimized by constructing deep neural network structures and applying reinforcement learning algorithms (such as Q-learning), thereby improving the intelligence level and efficiency of the system. Considering the actual traffic conditions in the city, such as vehicle waiting time, real-time distance, etc., the reward function and action space of the deep reinforcement learning method are newly designed to meet actual needs.
[0094] Finally, two deep reinforcement learning methods, namely the improved Q-learning based on driving edge and the Q-learning based on path, are used to comprehensively improve the city's garbage collection system, adjust the driving plan and the layout of the collection points, and select and make decisions based on the actual situation. In terms of traffic allocation, methods such as all-or-nothing allocation, incremental allocation, and iterative algorithms are applied to optimize route allocation and traffic flow management.
[0095] Among them, the methods of applying the path model and the driving model are both based on the Q-learning algorithm, but they differ in the definitions of the search space, state space, action space and reward function, resulting in differences in their applicable scenarios and performance characteristics.
[0096] 1. Specific differences
[0097] Path model, path-based Q-learning:
[0098] Search space: Restricted to a predefined subset of paths, the search scope is smaller and computationally more efficient.
[0099] State space: contains only initial state and terminal state, with a small number of states and low model complexity.
[0100] Action space: contains all paths in a predefined subset of paths, with a limited but still large number of actions.
[0101] Reward function: Considers path cost and traffic cost, and pays more attention to the long-term benefits of path selection.
[0102] Applicable scenarios: Applicable to large-scale networks, especially when the search space is large and the reward signal interference is large, such as the Guangzhou garbage collection and transportation network optimization scenario.
[0103] Driving model, edge-based Q-learning:
[0104] Search space: contains all possible paths, with a larger search range and lower computational efficiency.
[0105] State space: Contains the road network vertices where garbage collection and transportation vehicles are located. It has a large number of states and a high model complexity.
[0106] Action space: contains the starting edges corresponding to each vertex, with a large number of actions, but a limited number of actions in each state.
[0107] Reward function: Considers the cost of the current driving state and the distance to the target, and focuses more on short-term benefits.
[0108] Applicable scenarios: Suitable for small-scale networks, especially when the search space is small and the reward signal is clear, such as SF scenarios.
[0109] 2.Performance characteristics:
[0110] Path model, path-based Q-learning:
[0111] Faster learning speed and higher computational efficiency. Able to find better paths and reduce the driving distance of garbage trucks. More suitable for large-scale networks and complex environments.
[0112] Driving model, edge-based Q-learning:
[0113] The learning speed is slow and the computational efficiency is low. It can adapt to dynamically changing environments, such as real-time traffic changes. It is more suitable for small-scale networks and simple environments.
[0114] 3.Selection method:
[0115] The method you choose depends on the specific application scenario and requirements. If the network is large, the search space is large, and long-term benefits need to be considered, path-based Q-learning can be selected. If the network is small, the search space is small, and it needs to adapt to a dynamically changing environment, edge-based Q-learning can be selected.
[0116] In some feasible embodiments, the step S5 specifically includes:
[0117] According to the optimization results of the model, the driving plan of the garbage collection and transportation truck is adjusted, the optimal driving route is selected, and the layout of the garbage collection and transportation points is adjusted. For example, collection and transportation points are added near congested sections of road to reduce the driving distance of the garbage collection and transportation truck. In addition, the driving plan and the layout of collection and transportation points are dynamically adjusted according to the real-time road conditions and garbage generation. For example, when congestion occurs on a certain section of road, the system can adjust the driving route in time to avoid congestion and improve efficiency.
[0118] Based on the above scheme, the present invention also provides a practical comparison example:
[0119] When evaluating and comparing the effects of path-based Q-learning and edge-based Q-learning methods in Guangzhou garbage collection and transportation optimization and SF scenarios, we used three classic TAP methods for comparison: all-or-nothing allocation (AON), incremental allocation (INA), and continuous averaging (MSA). The results show that all-or-nothing allocation is the worst in both cases, while incremental allocation, although it can better allocate network demand, is still not as good as continuous averaging. In the Guangzhou garbage collection and transportation network optimization scenario, edge-based Q-learning and path-based Q-learning obtain similar results; however, in the SF scenario, the edge-based Q-learning algorithm performs better, especially in large-scale networks with large search spaces and more reward signal interference. Overall, the edge-based Q-learning algorithm can achieve about 15% improvement in optimizing the Guangzhou garbage transfer transportation distance, while the path-based Q-learning algorithm can shorten the distance by about 20%, as shown in Table 1.
[0120] Table 1. Comparison of optimization of two algorithms
[0121]
[0122] like Figure 5 As shown, a city garbage collection and transportation system based on reinforcement learning includes:
[0123] The first deep learning module constructs a driving model of the garbage collection and transportation vehicle and the collection and delivery point based on the Markov decision process; based on the driving model and according to the optimization goal, defines a first state space, a first action space and a first reward function;
[0124] A second deep learning module constructs a path model based on a Markov decision process; based on the path model, divides a second state space, a second action space, and a second reward function;
[0125] The optimization module optimizes the garbage collection and transportation network based on the driving model and the path model, and adjusts the driving plan and the layout of the collection and transportation points.
[0126] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0127] An urban garbage collection and transportation device based on reinforcement learning:
[0128] at least one processor;
[0129] at least one memory for storing at least one program;
[0130] When the at least one program is executed by the at least one processor, the at least one processor implements the urban garbage collection and transportation method based on reinforcement learning as described above.
[0131] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0132] A storage medium stores processor-executable instructions, which are used to implement a reinforcement learning-based urban garbage collection and transportation method as described above when executed by the processor.
[0133] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0134] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for collecting and transporting urban garbage based on reinforcement learning, characterized in that: The following steps are involved: Based on the Markov decision process, a driving model of garbage collection and transportation vehicles and collection and delivery points is constructed; Based on the driving model and according to the optimization objective, a first state space, a first action space and a first reward function are defined; Based on Markov decision process, the path model is constructed; Based on the path model, limiting the search space, dividing the second state space, the second action space and the second reward function; Based on the driving model and the path model, the garbage collection and transportation network is optimized, and the driving plan and the layout of the collection and transportation points are adjusted.
2. According to claim 1, a method for collecting and transporting urban garbage based on reinforcement learning is characterized in that: The above-mentioned step of constructing a driving model of garbage collection vehicles and collection and delivery points based on the Markov decision process specifically includes: The connectivity of streets in the garbage collection and transportation network, the locations of garbage collection vehicles and the locations of sending and receiving points are obtained, the garbage collection and transportation network is modeled as a Markov decision process, and the driving model of garbage collection vehicles and sending and receiving points is obtained.
3. According to claim 1, a method for collecting and transporting urban garbage based on reinforcement learning is characterized in that: The step of defining a first state space, a first action space, and a first reward function based on the driving model and according to the optimization objective specifically includes: Based on the driving model, a path is optimized with the goal of reducing the driving distance of the garbage collection and transportation vehicle in the garbage collection and transportation network; The first state space includes the road network vertices where the garbage collection and transportation vehicles are located; The first action space includes a departure edge corresponding to each vertex; The first reward function is defined based on the cost of the selected edges.
4. The urban garbage collection and transportation method based on reinforcement learning according to claim 1 is characterized in that: The step of limiting the search space based on the path model and dividing the second state space, the second action space and the second reward function specifically includes: Based on the driving model, all possible paths from the initial state to the terminal state are calculated, and a predefined subset is determined to obtain a search space; The second state space includes an initial state, a terminal state, and a time step; The second action space includes paths in a predefined subset; The second reward function is defined according to path cost and traffic cost.
5. The urban garbage collection and transportation method based on reinforcement learning according to claim 1 is characterized in that: The first state space is expressed as follows: S=[m 11 v 11 D t1 L t1 ,n 12 v 12 D t1 L t1 ....m ij v ij D tn L tn ,,]i=1,2...n;j=1,2...n Where m is the number of redundant vehicles on this road, v is the average speed of the current road, and D t L is the total distance between the collection point and the vehicle on the two-dimensional plane. t is the distance from the edge of the current road end, i is the current road, j is the car identifier on the road, and n is the total number of lanes.
6. The urban garbage collection and transportation method based on reinforcement learning according to claim 1 is characterized in that: The first reward function is expressed as follows: r sum =r d +r v Among them, r sum represents the first reward function, r d is the reward that the garbage truck gets for making a driving decision at each intersection. v is the reward of the current driving state, D t It is the total distance between the collection point and the vehicle on the two-dimensional plane.
7. The urban garbage collection and transportation method based on reinforcement learning according to claim 1 is characterized in that: The second state space is expressed as follows: S={(s start ,s end ,t)t≤T} Among them, s start represents the starting point of the garbage collection and transportation network, s end represents the terminal location of the garbage collection and transportation network, and t represents the time step.
8. A city garbage collection and transportation system based on reinforcement learning, characterized in that: include: The first deep learning module builds a driving model of garbage collection and transportation vehicles and collection and delivery points based on the Markov decision process; Based on the driving model and according to the optimization objective, a first state space, a first action space and a first reward function are defined; The second deep learning module builds a path model based on the Markov decision process; Based on the path model, dividing a second state space, a second action space, and a second reward function; The optimization module optimizes the garbage collection and transportation network based on the driving model and the path model, and adjusts the driving plan and the layout of the collection and transportation points.
9. A city garbage collection and transportation device based on reinforcement learning, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the urban garbage collection and transportation method based on reinforcement learning as described in any one of claims 1-7.
Citation Information
Patent Citations
Robot navigation method and system based on multimode perception and reinforcement learning
CN111367282A
Park comprehensive energy system low-carbon operation method based on Q learning
CN112580867A
Sanitation vehicle real-time route planning method based on deep Q learning
CN113420942A
Classification-oriented household garbage collection and transportation path multi-objective optimization method and system
CN116127857A
Garbage collection and transportation vehicle path multi-objective optimization method based on particle swarm optimization
CN116205423A
Cited By
Multi-mode interactive garbage classification guiding and recycling system
CN119941240A
Intelligent scheduling system of environmental sanitation equipment
CN120181461A
