Path planning method and system for goods taking and delivering vehicle of heterogeneous vehicle type

Abstract

The invention relates to a heterogeneous vehicle type goods taking and delivering vehicle path planning method and system. The method comprises the steps that a heterogeneous vehicle type goods taking and delivering problem scene is constructed, and an optimization target is set to minimize the total transportation charging cost of the heterogeneous vehicle type goods taking and delivering vehicle; constructing a deep reinforcement learning model and a reinforcement learning environment adopting an iterative improved solution mode based on the heterogeneous vehicle type goods taking and delivering problem scene and the optimization target; in the training process of constructing the reinforcement learning model, a near-end strategy optimization algorithm is adopted to carry out several batches of training on the deep reinforcement learning model, and the trained deep reinforcement learning model is obtained; and the trained deep reinforcement learning model is used to solve the actual heterogeneous vehicle type goods taking and delivering problem scene, and a vehicle path planning scheme of the heterogeneous vehicle type goods taking and delivering problem scene is obtained. According to the method, the calculation time for solving the problem scene can be shortened, and a high-quality solution can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of logistics scheduling, and more specifically, to a path planning method and system for heterogeneous pickup and delivery vehicles. Background Art

[0002] As the scale of e-commerce continues to expand, the related logistics industry has also developed rapidly, and its output value has also achieved rapid growth. According to statistics from the State Post Bureau, my country's express delivery has shown a trend of rapid growth in recent years. In the past five years, the total cost of my country's logistics industry has exceeded 13 trillion yuan per year, and this cost accounts for more than 14% of GDP. The total cost of social logistics is still rising year by year. In the total cost structure of logistics, the proportion of transportation costs accounts for more than 50%, becoming the largest expenditure item for logistics companies. Therefore, the optimization strategy of logistics distribution has been widely regarded as an effective way to reduce transportation costs.

[0003] In actual scenarios, in order to effectively reduce costs, logistics pickup and delivery tasks are often combined to form a pickup and delivery problem. At the same time, considering some constraints, such as real-time, and fleets of heterogeneous vehicles with different loading capacities and different usage costs, complex application scenarios such as Heterogeneous Pickup and Delivery Problems with Time Window (HPDPTW) are formed.

[0004] At present, there are still few specialized research results in the field of HPDPTW. Although there are many methods for solving the pickup and delivery problem of heterogeneous models, there are still obvious deficiencies in the quality and efficiency of the solution when dealing with complex problems such as HPDPTW. Therefore, how to improve the solution quality and efficiency of the pickup and delivery problem of heterogeneous models is still an urgent problem to be solved. Summary of the invention

[0005] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a method and system for path planning of heterogeneous vehicle types for picking up and delivering goods, so as to improve the quality and efficiency of path planning of heterogeneous vehicle types for picking up and delivering goods.

[0006] According to a first aspect of the present application, a method for path planning of heterogeneous pickup and delivery vehicles is provided, the method comprising: Construct a problem scenario of picking up and delivering goods of heterogeneous vehicle types and set an optimization goal, wherein the optimization goal is to minimize the total transportation billing cost of the vehicle transportation of the heterogeneous vehicle types picking up and delivering goods; Based on the heterogeneous vehicle model pickup and delivery problem scenario and optimization objectives, a deep reinforcement learning model and reinforcement learning environment using an iterative improvement solution method are constructed; Constructing a training process for a reinforcement learning model, and training the deep reinforcement learning model in batches using a reinforcement learning algorithm optimized by a proximal strategy based on the training process to obtain a trained deep reinforcement learning model; The trained deep reinforcement learning model is used to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and the vehicle path planning solution for the heterogeneous vehicle model pickup and delivery problem scenario is obtained.

[0007] Since the deep reinforcement learning model has the advantage of one-time training and multiple solutions, it can obtain vehicle path planning solutions more quickly and accurately, thereby reducing the calculation time of model solving and improving the efficiency and quality of solving heterogeneous vehicle model pickup and delivery problems.

[0008] Optionally, constructing a pickup and delivery problem scenario for heterogeneous vehicle types specifically includes: Use pictures Represents the problem scenario of picking up and delivering goods of heterogeneous vehicle types, where Represents a collection of nodes, represents the travel distance between nodes; Setting customer orders and 1 warehouse. Each customer order includes a pickup point and a delivery point. Each pickup point and each delivery point are node sets. 1 node in , then a problem scenario instance includes different nodes, where node 0 represents the warehouse; For any customer order, use the node Indicates the pickup node, using node Represents the corresponding delivery node, node set Represents all delivery nodes, node collection Represents all pickup nodes, Represents a pair of pickup and delivery nodes; each pickup node The goods need to be delivered to the corresponding delivery node ; For a single node It indicates that, Representation Node Two-dimensional geographic location coordinate information ; Representation Node The transportation demand of the warehouse is set to 0; Representation Node The time window constraints include pre-defined nodes Earliest arrival time and the latest arrival time ; Representation diagram The travel distance between midpoints, Represents a slave node Go to the node All available vehicles are divided into M groups of different fleets according to their maximum load capacity. for ,in Indicates the fleet The maximum load weight of the vehicle.

[0009] By using a graph to represent the problem scenario of picking up and delivering goods of heterogeneous vehicle types, it is convenient to subsequently build a mathematical model for vehicle path planning, which helps to solve the problem scenario using a deep reinforcement learning model.

[0010] Optionally, the components of the reinforcement learning algorithm are defined as follows: State: The state includes the problem instance state and the current solution state; the problem instance state includes the coordinates, transportation demand, and time window of all nodes; the current solution state includes the front and rear adjacent nodes of each node and the distance between each node and its front and rear adjacent nodes, the vehicle type of the path where the current node is located, the remaining load of the vehicle after serving the current node, the total driving distance of the path where the current node is located, the driving distance when the vehicle arrives at the current node, the total transportation cost of the path where the current node is located, the transportation cost when the vehicle arrives at the current node, and the ratio of the total transportation cost of the path where the current node is located to the total transportation demand; Action: The action includes a pair of delivery nodes and a heuristic operator; Reward: The optimization goal of the problem is to minimize the total cost of vehicle transportation charges to complete all customer orders. The difference between the current solution and the initial solution transportation cost is recorded as , That is the reward value.

[0011] Optionally, the deep reinforcement learning model includes an actor strategy network and a critic evaluation network; The actor strategy network includes an embedding layer, a first encoder, a node decoder and an operator decoder; The problem instance state and the current solution state are respectively input into the embedding layer and embedded into the high-dimensional space to obtain instance embeddings. De-embedding ; By embedding the instance De-embedding Input to the first encoder to get the enhanced embedding ; Enhance the embedding Input node decoder, after passing through the multi-layer perceptron of the node decoder, and then normalized The node pair selection probability matrix is ​​obtained by operation, and the next pick-up and delivery node pair to be operated is selected based on the node pair selection probability matrix. ; The operator selects the decoder based on the pick-up and delivery node pairs Select the next operator to use , by pairing the pick-up and delivery nodes Sum Operator As an action Output to the reinforcement learning environment for iteration; The critic evaluation network includes a second encoder and a fully connected layer; the second encoder is used to process the input problem instance state and the current solution state, and the embedded information is obtained by inputting the problem instance state and the current solution state into the second encoder for calculation. ;Will The fully connected layer is input and the value is obtained through calculation.

[0012] By constructing an actor strategy network including an embedding layer, a first encoder, a node decoder, and an operator decoder, and a critic evaluation network consisting of a second encoder and a fully connected layer, the deep reinforcement learning model of the present application can efficiently embed the problem instance state and the current solution state into a high-dimensional space, and accurately decode the node pair selection probability matrix and the applicable heuristic operator. Through the value evaluation of the critic network, the convergence of the model training process is accelerated.

[0013] Optionally, the heuristic operator includes one or more of an intra-path exchange operator, an intra-path relocation operator, an inter-path exchange operator, an inter-path relocation operator, a path splitting operator, and a path deletion operator; Assume that the pick-up and delivery node pairs selected by the actor strategy network are , take the delivery node pair The path is , the operation of the heuristic operator is as follows: In-path exchange operator: exchange the pick-up and delivery node pairs With the path Another pair of nodes in The access order of The node pair selected in a greedy way satisfies the predefined constraints of the problem and causes the smallest cost increase after exchange; In-path relocation operator: first relocate the node pair From the path Delete it, and then add the node pair Reinsert into The position that satisfies the predefined constraints of the problem and causes the smallest cost increase after insertion; Inter-path exchange operator: exchange node pairs With another path Another pair of nodes in The access order of is a randomly selected path, The node pair selected in a greedy way satisfies the predefined constraints of the problem and causes the smallest cost increase after exchange; Inter-path relocation operator: first relocate the node pair From the path and then the node pair Reinsert to another path In the appropriate position; where the path is randomly selected, and the node pair The insertion position is the position that satisfies the predefined constraints of the problem and causes the smallest cost increase after insertion; Path splitting operator: first delete the node pair The path , and then add two vehicles, for All node pairs in are evenly inserted into the paths of the two newly added vehicles according to the original access order. The newly added vehicles are selected according to the actual problem scenario, and are selected from the same fleet as the original vehicles or from the fleet with a smaller maximum load than the original vehicles; Path deletion operator: first delete the node pair The path , then All node pairs in are reinserted into other paths. When inserting a node pair, a path is randomly selected and the location that meets the predefined constraints of the problem and causes the smallest cost increase after insertion is selected to perform the insertion operation.

[0014] By introducing heuristic operators such as intra-path exchange operator, intra-path relocation operator, inter-path exchange operator, inter-path relocation operator, path splitting operator and path deletion operator, the policy network in the reinforcement learning environment can greedily select the path with the minimum cost increase after selecting the pickup and delivery node pairs, so that each adjustment can approach a better solution, thereby improving the efficiency of the overall transportation plan and the economy of vehicle transportation billing costs.

[0015] Optionally, the reinforcement learning environment includes: State update: Based on the problem scenario of heterogeneous vehicle models picking up and delivering goods, state transition rules are constructed. The state transition rules are used to generate the next state according to the actions output by the actor policy network during model training and inference; The state transition rule is defined as: Assume that the pick-up and delivery nodes are selected Sum Operator As the next step, (1) For node pairs And the node pair The front and back adjacent nodes of the node are updated, and their respective front and back adjacent nodes and front and back adjacent distances are updated; (2) Update node pair The type of vehicle on the route; (3) Update the vehicle service completion node pair The remaining load after (4) Update node pair The total travel distance of the route; (5) Update the vehicle arrival node pair driving distance in hours; (6) Update node pair The total transportation cost of the route; (7) Update the vehicle arrival node pair Transportation costs at the time of delivery; (8) Update node pair The ratio of the total transportation cost of the route to the total transportation demand; (9) For the status update operations described in (1) to (8), update the other nodes in each path involved in the change accordingly; Dataset Generation: Generate datasets for training deep reinforcement learning models.

[0016] By constructing a state update mechanism with state transition rules, the reinforcement learning environment can accurately generate the next state based on the output actions of the actor policy network during model training and inference.

[0017] Optionally, the data set generation specifically includes: Obtain an existing dataset of pickup and delivery instances, and generate new data based on the existing dataset using coordinate transformation and random perturbation as a dataset for deep reinforcement learning model training; New data is generated by coordinate transformation, including: Invert: Convert geographic coordinates Reverse to get geographic coordinates ; Mirror: Geographic coordinates In Replace with ; Set geographic coordinates In Replace with ; Set geographic coordinates In Replace with , Replace with ; Generate new data by random perturbation, including: the geographic coordinates of each customer node , by generating random numbers Perturb it to generate new nodes .

[0018] By using random perturbations to fine-tune the geographic coordinates of customer nodes in the existing pickup and delivery instance dataset, diverse new data can be generated. These new data not only retain the characteristics of the original data, but also increase the variability and richness of the data. Random perturbations can enhance the ability of deep reinforcement learning models to adapt to different geographical distributions and changes in customer demand, thereby demonstrating greater robustness and accuracy in practical applications.

[0019] Optionally, the training process of the reinforcement learning model uses a proximal strategy optimization algorithm to perform several batches of training on the deep reinforcement learning model, specifically including: Initialize the parameters of the actor strategy network and the critic evaluation network; Divide the dataset used for deep reinforcement learning model training into several batches of data instances and train the model in several batches; The training process of each batch includes: using a greedy insertion algorithm to generate an initial solution to the problem instance; based on the initial solution, performing several steps of interaction with the reinforcement learning environment through the deep reinforcement learning model, collecting the action, reward and value of each step; calculating the advantage function value based on the collected actions, rewards and values; and updating the parameters of the actor network and the critic network through a predefined loss function based on the actions, rewards, value functions and advantage function values.

[0020] Dividing the preprocessed data set into multiple batches of data instances and iteratively training the model can significantly improve the efficiency and stability of the proximal policy optimization algorithm in deep reinforcement learning model training. Through batch processing, the model can gradually learn and optimize the current solution in each iteration, effectively reducing the risk of overfitting during training and improving the generalization ability of the model.

[0021] Optionally, the step of using a greedy insertion algorithm to generate an initial solution to the problem instance specifically includes: A1. Obtain all the pickup and delivery node pairs to be delivered, randomly select a pair of pickup and delivery nodes, and select the first vehicle from the fleet with the largest load capacity, and insert the selected pickup and delivery node pair into the delivery route of the first vehicle selected; A2. Randomly select a second pair of pick-up and delivery nodes from the remaining pick-up and delivery node pairs, and try to insert the second pair of pick-up and delivery node pairs into the delivery route of the first vehicle. If the second pair of pick-up and delivery node pairs can be inserted into the delivery route of the first vehicle without violating the predefined constraints, the insertion operation is performed; if the second pair of pick-up and delivery node pairs cannot be inserted into the delivery route of the first vehicle, select a second vehicle from the fleet with the second largest load after the first vehicle, and insert the second pair of nodes into the delivery route of the second vehicle; A3. Continue to randomly select a third pair of pick-up and delivery node pairs from the remaining pick-up and delivery node pairs, and try to insert the third pair of pick-up and delivery node pairs into the vehicle delivery routes that have been scheduled for delivery. If the third pair of pick-up and delivery node pairs cannot be inserted into all the scheduled vehicle delivery routes, continue to select a vehicle from the next fleet in descending order of load capacity, and try to perform the insertion operation; if the fleet with the smallest load capacity was selected when adding a new vehicle last time, select a new vehicle from the fleet with the largest load capacity next time.

[0022] A4. Loop through steps A2 to A3 until all pickup and delivery nodes are inserted into the vehicle's delivery route, and the initial solution of the actor policy network is obtained.

[0023] According to a second aspect of the present application, a path planning system for pickup and delivery vehicles of heterogeneous vehicle types is provided, the system comprising: A problem scenario construction module, used to construct a problem scenario of picking up and delivering goods by heterogeneous vehicle types and set an optimization goal, wherein the optimization goal is to minimize the total vehicle transportation billing cost of picking up and delivering goods by heterogeneous vehicle types; A model building module, used to build a deep reinforcement learning model and a reinforcement learning environment using an iterative improvement solution method based on the heterogeneous vehicle model pickup and delivery problem scenario and optimization objectives; A model training module is used to construct a training process of a reinforcement learning model, and based on the training process, a proximal strategy optimization algorithm is used to train the deep reinforcement learning model in several batches to obtain a trained deep reinforcement learning model; The solution generation module is used to use the trained deep reinforcement learning model to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and obtain the vehicle path planning solution for the heterogeneous vehicle model pickup and delivery problem scenario.

[0024] Based on any of the above aspects, the heterogeneous vehicle pick-up and delivery vehicle path planning method and system provided in the embodiment of the present application has the following beneficial effects compared with the prior art: 1. This application adopts a deep reinforcement learning method, which has the characteristics of one-time training and multiple solutions compared to existing precise algorithms and heuristic algorithms. By training the deep reinforcement learning model, the trained model can be used to quickly obtain solutions in multiple different solution scenarios, thereby reducing the calculation time for solving different problem scenarios.

[0025] 2. The solution process of this application is an iterative optimization process, which uses an operator selector based on deep reinforcement learning to select operation operators and performs multiple rounds of iterative updates on the initial solution. By continuously optimizing the current solution, a high-quality solution can eventually be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A schematic application scenario diagram of the route planning method for heterogeneous vehicle pick-up and delivery vehicles provided in this embodiment.

[0028] Figure 2 This is an overall flow chart of the route planning method for heterogeneous vehicle pick-up and delivery vehicles provided in this embodiment.

[0029] Figure 3 Flow chart of the method for path planning for heterogeneous vehicle pick-up and delivery vehicles provided in this embodiment.

[0030] Figure 4 Schematic diagram of the deep reinforcement learning model structure provided in this embodiment.

[0031] Figure 5 A schematic diagram of a complete solution to the problem example provided for this embodiment.

[0032] Figure 6 This embodiment provides a schematic diagram of the operation of the intra-path exchange operator.

[0033] Figure 7 A schematic diagram of the operation of the intra-path relocation operator provided in this embodiment.

[0034] Figure 8 A schematic diagram of the operation of the path exchange operator provided in this embodiment.

[0035] Fig. 9 A schematic diagram of the inter-path relocation operator operation provided in this embodiment.

[0036] Fig.10A schematic diagram of the path splitting operator operation provided in this embodiment.

[0037] Fig.11 A schematic diagram of the path deletion operator operation provided in this embodiment.

[0038] Fig.12 A schematic diagram of the process of generating the initial solution provided in this embodiment.

[0039] Fig.13 A flowchart of the algorithm application provided in this embodiment for solving the actual heterogeneous vehicle model pickup and delivery problem scenario.

[0040] Fig.14 A schematic diagram of the structure of the path planning system for heterogeneous pickup and delivery vehicles provided in this embodiment. DETAILED DESCRIPTION

[0041] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] For scenarios such as the pick-up and delivery problem with time windows and the vehicle routing problem with heterogeneous vehicle models, classic solutions include exact algorithms and heuristic algorithms. Among them, exact algorithms represented by column generation algorithms and branch-and-bound algorithms can find the optimal solution to the problem, but they are generally only applicable to small-scale problem instances. When the problem scale increases, they cannot give suitable results within a reasonable time. Heuristic algorithms represented by ant colony algorithms, taboo search algorithms, genetic algorithms, etc. have the advantage of finding ideal solutions within a reasonable time, but these algorithms also have disadvantages that cannot be ignored: the convergence speed is relatively slow, and the algorithm relies on powerful heuristic rules designed by humans.

[0045] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0046] For example, a schematic diagram of an application scenario of a method for path planning of a heterogeneous vehicle pickup and delivery vehicle provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100, the server 100 has a data processing function; the terminal device 200 has a streaming media playback function.

[0047] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0048] In one practicable manner, the server 100 and the terminal 200 may respectively execute the path planning method for picking up and delivering vehicles of heterogeneous vehicle types provided in the embodiments of the present application, or, optionally, the path planning method for picking up and delivering vehicles of heterogeneous vehicle types provided in the embodiments of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0049] like Figure 3 As shown, this embodiment provides a route planning method for picking up and delivering vehicles of heterogeneous vehicle types, which may include the following steps: S1. Construct a problem scenario of picking up and delivering goods by heterogeneous vehicle models and set an optimization goal; the optimization goal is to minimize the total vehicle transportation billing cost of picking up and delivering goods by heterogeneous vehicle models.

[0050] In this embodiment, the problem scenario of picking up and delivering goods of heterogeneous vehicle types can be illustrated in FIG. Represents the problem scenario of picking up and delivering goods of heterogeneous vehicle types, where Represents a collection of nodes, Represents the travel distance between nodes.

[0051] Setting customer orders and 1 warehouse. Each customer order includes a pickup point and a delivery point. Each pickup point and each delivery point are node sets. 1 node in , then a problem scenario instance includes different nodes, where node 0 represents the warehouse.

[0052] For any customer order, use the node Indicates the pickup node, using node Represents the corresponding delivery node, node set Represents all delivery nodes, node collection Represents all pickup nodes, Represents a pair of pickup and delivery nodes. Each pickup node The goods need to be delivered to the corresponding delivery node A single node can use It indicates that, Representation Node Two-dimensional geographic location coordinate information ; Representation Node The transportation demand of the warehouse is set to 0; Representation Node The time window constraints include pre-defined nodes Earliest arrival time and the latest arrival time In the specific implementation process, the two-dimensional geographic location coordinate information of the node can be obtained by converting the longitude and latitude of the node.

[0053] Representation diagram The travel distance between midpoints, Represents a slave node Go to the node All available vehicles are divided into M groups of different fleets according to their maximum load capacity. for ,in Indicates the fleet The maximum load weight of the vehicle.

[0054] In this embodiment, the constraints of the heterogeneous vehicle model pickup and delivery problem scenario can be defined according to actual needs. For example, the predefined constraints are: each vehicle in the distribution plan starts from the same warehouse at the beginning of the delivery, and returns to the warehouse after completing several pickup and delivery orders; the pickup point and delivery point of the same order must be delivered by the same vehicle, and the goods must be picked up from the pickup point before being delivered to the corresponding delivery point; the pickup and delivery of different orders can be carried out alternately; each customer node has a time window limit, and the vehicle must reach the node before the latest arrival time of the node. At the same time, if the vehicle arrives earlier than the earliest arrival time of the node, it needs to wait; in the process of picking up and delivering goods at the customer node, vehicle overloading is not allowed.

[0055] In this embodiment, the optimization goal of the heterogeneous vehicle model pickup and delivery problem scenario is to minimize the total vehicle transportation billing cost of completing all customer orders, which is in the form of: in, Indicates the delivery route. Indicates the starting price of the vehicle. Indicates the vehicle's additional kilometer price (that is, when the delivery distance exceeds the mileage included in the starting price, additional fees will be charged for each additional delivery distance). Indicates delivery route The driving distance, Indicates the starting distance of the vehicle.

[0056] S2. Based on the heterogeneous vehicle model pickup and delivery problem scenario and optimization objectives, a deep reinforcement learning model and reinforcement learning environment using an iterative improvement solution method are constructed.

[0057] In this embodiment, the deep reinforcement learning model includes an actor policy network and a critic evaluation network.

[0058] like Figure 4 As shown in the figure, the actor strategy network includes an embedding layer (not shown in the figure), a first encoder, a node decoder and an operator decoder. The input data of the actor strategy network includes two parts of data: the problem instance state and the current solution state. These two parts of data are embedded into a high-dimensional space through the embedding layer to obtain instance embeddings De-embedding .

[0059] In this embodiment, the first encoder is an encoder based on the attention mechanism, and the first encoder is used to fuse the instance embedding De-embedding In specific implementation, by embedding the instance De-embedding Input to the first encoder to get enhanced embedding .

[0060] The node decoder is used to select the next pick-up and delivery node pair to be operated. In the specific implementation process, the enhanced embedding of the output of the first encoder is Input node decoder, processed by the node decoder's multi-layer perceptron, and then normalized The node pair selection probability matrix is ​​obtained by operation, and the next pick-up and delivery node pair to be operated is selected based on the node pair selection probability matrix. .

[0061] The operator selection decoder is used to select the next heuristic operator to use. Represents the operators selected in the past several steps and their effects, embedding the enhanced output of the first encoder and After the connection operation, it is input into the multi-layer perceptron in the operator decoder and then normalized Operate to obtain the operator selection probability matrix and select the next operator to be used , the next node to be operated is the delivery node and the next operator used As an action Output to the reinforcement learning environment for iteration.

[0062] In this embodiment, the Critic network includes a second encoder and a fully connected layer. The second encoder is used to process the input problem instance state and the current solution state. In the specific implementation process, the problem instance state and the current solution state are input into the second encoder for calculation to obtain the embedded information. Then Input the fully connected layer and calculate the value .

[0063] The components of the reinforcement learning algorithm in this embodiment are defined below, including state, action, and reward.

[0064] State: The state is divided into problem instance state and current solution state.

[0065] The problem instance state of this embodiment is information that does not change as the solution process progresses, including the coordinates of all nodes, transportation requirements, and time windows; the current solution state is information that will change as the solution process progresses. The solution process of the reinforcement learning algorithm of this embodiment is an iterative optimization method solution process. During the solution process, a complete initial solution is first generated, and then the initial solution is updated and optimized for multiple rounds. The current solution state includes: the distance between the front and rear adjacent nodes of each node and each node and its front and rear adjacent nodes, the vehicle type of the path where the current node is located, the remaining load of the vehicle after serving the current node, the total driving distance of the path where the current node is located, the driving distance when the vehicle arrives at the current node, the total transportation cost of the path where the current node is located, the transportation cost when the vehicle arrives at the current node, and the ratio of the total transportation cost of the path where the current node is located to the total transportation demand.

[0066] In this embodiment, the path in the current solution state refers to the driving path of a vehicle in the solution of the problem instance. The path can be represented by the sequence of client nodes served by the vehicle in succession. A complete solution is composed of several paths. Figure 5 As shown, Figure 5 is a diagram of a complete solution to a problem instance.

[0067] Action: An action consists of a pick-up and delivery node pair and a heuristic operator. In each step of model training and solving, the policy network selects an operator and a pick-up and delivery node pair, and uses the selected operator to operate the selected pick-up and delivery node pair to achieve an iterative update.

[0068] Reward: The optimization goal of the problem is to minimize the total cost of vehicle transportation charges to complete all customer orders. The difference between the current solution and the initial solution is recorded as , That is the reward value.

[0069] In this embodiment, the heuristic operators available for selection by the policy network include an intra-path exchange operator, an intra-path relocation operator, an inter-path exchange operator, an inter-path relocation operator, a path splitting operator, and a path deletion operator. Figure 4-9 They are schematic diagrams of the above operator operations. Figure 6-11 Each operator is described below: For example, assume that the pick-up and delivery node pairs selected by the actor strategy network are , node pair The path is , then the operations of various operators are as follows: In-path exchange operator: exchange node pairs With the path Another pair of nodes in The access order of It is a node pair selected in a greedy way that satisfies the problem constraints and causes the smallest cost increase after exchange.

[0070] In-path relocation operator: first relocate the node pair From the path , and then reinsert it into The specific insertion position is the position that satisfies the problem constraints and causes the smallest cost increase after insertion.

[0071] Inter-path exchange operator: exchange node pairs With another path Another pair of nodes in The access order of . Among them, the path is randomly selected. It is a node pair selected in a greedy way that satisfies the problem constraints and causes the smallest cost increase after exchange.

[0072] Inter-path relocation operator: first relocate the node pair From the path and then reinsert it into another path In the appropriate position. Among them, the path is randomly selected, and the node pair The insertion position is the position that satisfies the problem constraints and causes the smallest increase in cost after insertion.

[0073] Path splitting operator: first delete the node pair The path , and then add two vehicles, for All node pairs in are evenly inserted into the paths of the two newly added vehicles according to the original access order. The newly added vehicles can be selected from the same fleet as the original vehicles or from the fleet with a smaller maximum load than the original vehicles according to the actual problem scenario. Path deletion operator: first delete the node pair The path , then All node pairs in are reinserted into other paths. When inserting a node pair, a path is randomly selected and the location that meets the problem constraints and causes the smallest cost increase after insertion is selected to perform the insertion operation.

[0074] In this embodiment, the reinforcement learning environment includes state updating and data set generation.

[0075] State update: Based on the mathematical model constructed in step S1, a state transition rule is constructed to generate the next state according to the state transition rule based on the actions output by the policy network during model training and reasoning.

[0076] The state transition rules are defined as: Assume that the pick-up and delivery nodes are selected Sum Operator As the next step, after the action selection is completed, the state changes as follows: (1) For node pairs And the node pair The front and back adjacent nodes of the node are updated, and their respective front and back adjacent nodes and front and back adjacent distances are updated; (2) Update node pair The type of vehicle on the route; (3) Update the vehicle service completion node pair The remaining load after (4) Update node pair The total travel distance of the route; (5) Update the vehicle arrival node pair driving distance in hours; (6) Update node pair The total transportation cost of the route; (7) Update the vehicle arrival node pair Transportation costs at the time of delivery; (8) Update node pair The ratio of the total transportation cost of the route to the total transportation demand; (9) For the status update operations described in (1) to (8), it is also necessary to make corresponding updates to other nodes in each path involved in the change.

[0077] Through the state update mechanism of this embodiment, the reinforcement learning model can continuously explore and optimize the solution to the problem according to the current state and the actions output by the policy network during the training and reasoning process.

[0078] Dataset generation: Generate a dataset for deep reinforcement learning model training. The specific method is: obtain the existing pick-up and delivery instance dataset, and based on the existing pick-up and delivery instance dataset, use coordinate transformation and random perturbation to generate new data as the dataset for deep reinforcement learning model training. The coordinate transformation methods include: Invert: Convert geographic coordinates Reverse to get geographic coordinates .

[0079] Mirror: Geographic coordinates In Replace with ; Set geographic coordinates In Replace with ; Set geographic coordinates In Replace with , Replace with ; The specific method of random perturbation is: for each customer node's ground coordinates , by generating random numbers .

[0080] The training and solving processes of the deep reinforcement learning model are both iterative optimization processes. During training and solving, the iterative improvement solution model first uses a greedy insertion algorithm to generate an initial solution, and then uses the deep reinforcement learning model as a heuristic operator selector to select a suitable operator from a variety of operators to optimize the solution. After multiple rounds of iterative optimization, the best solution is selected as the final solution to the problem.

[0081] like Fig.12 As shown, in this embodiment, the greedy insertion algorithm is used to generate the initial solution of the problem instance, which specifically includes the following steps: A1. Randomly select a pair of pick-up and delivery nodes, and select a vehicle from the fleet with the largest load capacity, and insert the pick-up and delivery node pair into the delivery route of the vehicle; A2. Randomly select a second pair of nodes from the remaining pick-up and delivery node pairs, and try to insert them into the delivery route of the first vehicle. If the node pair can be inserted into the delivery route of the vehicle without violating the constraints, the insertion operation is performed; if the node pair cannot be inserted into the delivery route of the first vehicle, select a second vehicle from the fleet with the second largest load after the first vehicle, and insert the second pair of nodes into the delivery route of the second vehicle; A3. Continue to randomly select the third pair of nodes from the remaining pickup and delivery node pairs, and try to insert it into the scheduled vehicle delivery routes. If this node pair cannot be inserted into all the scheduled vehicle delivery routes, then continue to select a vehicle from the next fleet in descending order of load capacity and try to perform the insertion operation; if the fleet with the smallest load capacity was selected when adding a new vehicle last time, then select a new vehicle from the fleet with the largest load capacity next time.

[0082] A4. Loop through steps A2 to A3 until all pick-up and delivery nodes are inserted into the vehicle's delivery route, thereby obtaining the initial solution.

[0083] S3. Construct a training process for a reinforcement learning model, and based on the training process, use a reinforcement learning algorithm optimized by a proximal strategy to perform several batches of training on the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0084] The proximal strategy optimization algorithm is used to perform multi-batch training on the deep reinforcement learning model. The initial solution is iteratively updated for several rounds through the deep reinforcement learning model. The interaction information between the model and the environment is recorded during the training process, and the parameters of the deep reinforcement learning model are updated according to a predefined loss function.

[0085] In this embodiment, the training steps of the proximal strategy optimization method include: S31, initialize the parameters of the actor strategy network and the critic evaluation network; S32. Divide the data set generated for deep reinforcement learning model training into several batches of data instances and train the model in several batches.

[0086] The training process of each batch includes: using a greedy insertion algorithm to generate an initial solution to the problem instance; based on the initial solution, performing several steps of interaction with the reinforcement learning environment through the deep reinforcement learning model, and collecting interaction information such as actions, rewards and values ​​at each step; calculating the advantage function value based on the collected actions, rewards and values; and updating the parameters of the actor network and the critic network through a predefined loss function based on the interaction information such as actions, rewards, value functions and the advantage function value.

[0087] Calculate the advantage function value based on interaction data , the specific calculation formula is as follows: in Indicated in The reward value obtained by the actor network at this step, is the discount factor, Indicated in Step 1: Critic network input state The value function value outputted later.

[0088] In the specific implementation process, the gradient descent method is used to update the parameters of the actor network and the critic network.

[0089] In this example, the loss function of the actor network is as follows: in represents the parameters of the actor network, represents importance sampling, represents the advantage function value, Indicates that the actor network is using parameters In case of input status Output Action The probability distribution of Indicates that the actor network is using the old parameters before being updated In the case of input status Output Action The probability distribution of is the crop factor; The loss function of the critic network is as follows: , in represents the parameters of the critic network, Indicated in Step 1: Critic network input state The value function value outputted later.

[0090] S4. Use the trained deep reinforcement learning model to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and obtain the vehicle path planning solution for the heterogeneous vehicle model pickup and delivery problem scenario.

[0091] like Fig.13 As shown, in this embodiment, the trained deep reinforcement learning model is used to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and the algorithm application process is as follows: S41, using a greedy insertion algorithm to generate an initial solution to the problem instance to be solved; S42, generating input data of the actor strategy network according to the problem instance and the information of the current solution; S43, using the trained actor strategy network to receive input data and interact with the reinforcement learning environment; the output of the actor strategy network uses greedy sampling to select heuristic operators and delivery nodes, thereby optimizing the current solution; S44. After completing a preset number of iterations, select the best solution from all iteratively generated solutions as the final solution to the problem instance.

[0092] The path planning solution obtained by adopting the vehicle path planning method of this embodiment can significantly optimize vehicle scheduling and path planning, greatly reduce unnecessary driving distance and vehicle usage times, and thus effectively reduce the total cost of vehicle transportation billing.

[0093] The technical solution of the path planning method for heterogeneous pickup and delivery vehicles provided in this application is explained below in conjunction with a specific embodiment.

[0094] like Fig.14 As shown, the embodiment of the present application further provides a path planning system 610 for picking up and delivering vehicles of different vehicle types. Optionally, the path planning system 610 for picking up and delivering vehicles of different vehicle types may include: A problem scenario construction module 611 is used to construct a problem scenario of picking up and delivering goods by heterogeneous vehicle types and set an optimization goal, wherein the optimization goal is to minimize the total transportation billing cost of the vehicles picking up and delivering goods by heterogeneous vehicle types; A model building module 612, for building a deep reinforcement learning model and a reinforcement learning environment using an iterative improvement solution method based on the heterogeneous vehicle model pickup and delivery problem scenario and optimization objectives; A model training module 613 is used to construct a training process of a reinforcement learning model, and to train the deep reinforcement learning model in several batches using a proximal strategy optimization algorithm based on the training process to obtain a trained deep reinforcement learning model; The solution generation module 614 is used to use the trained deep reinforcement learning model to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and obtain the vehicle path planning solution for the heterogeneous vehicle model pickup and delivery problem scenario.

[0095] It can be understood that the above-mentioned system embodiments and the above-mentioned method embodiments can correspond to each other, and similar descriptions of the system embodiments can refer to the method embodiments. To avoid repetition, it will not be repeated here. A heterogeneous vehicle type pick-up and delivery vehicle path planning system provided in an embodiment of the present application can execute a heterogeneous vehicle type pick-up and delivery vehicle path planning method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. The functional modules of the heterogeneous vehicle type pick-up and delivery vehicle path planning system can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0096] Specifically, each step of the method embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor. The steps of the heterogeneous vehicle model pick-up and delivery vehicle path planning method in the embodiment of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. Optionally, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0097] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for path planning of heterogeneous pickup and delivery vehicles, characterized in that: The method comprises: Construct a problem scenario of picking up and delivering goods of heterogeneous vehicle types and set an optimization goal, wherein the optimization goal is to minimize the total transportation billing cost of the vehicle transportation of the heterogeneous vehicle types picking up and delivering goods; Based on the heterogeneous vehicle model pickup and delivery problem scenario and optimization objectives, a deep reinforcement learning model and reinforcement learning environment using an iterative improvement solution method are constructed; Constructing a training process for a reinforcement learning model, and training the deep reinforcement learning model in batches using a reinforcement learning algorithm optimized by a proximal strategy based on the training process to obtain a trained deep reinforcement learning model; The trained deep reinforcement learning model is used to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and the vehicle path planning solution for the heterogeneous vehicle model pickup and delivery problem scenario is obtained.

2. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 1, characterized in that: The construction of the heterogeneous vehicle model pickup and delivery problem scenario specifically includes: Use pictures Represents the problem scenario of picking up and delivering goods of heterogeneous vehicle types, where Represents a collection of nodes, represents the travel distance between nodes; Setting customer orders and 1 warehouse. Each customer order includes a pickup point and a delivery point. Each pickup point and each delivery point are node sets. 1 node in , then a problem scenario instance includes different nodes, where node 0 represents the warehouse; For any customer order, use the node Indicates the pickup node, using node Represents the corresponding delivery node, node set Represents all delivery nodes, node collection Represents all pickup nodes, Represents a pair of pickup and delivery nodes; each pickup node The goods need to be delivered to the corresponding delivery node ; For a single node It indicates that, Representation Node Two-dimensional geographic location coordinate information ; Representation Node The transportation demand of the warehouse is set to 0; Representation Node The time window constraints include pre-defined nodes Earliest arrival time and the latest arrival time ; Representation diagram The travel distance between midpoints, Represents a slave node Go to the node All available vehicles are divided into M groups of different fleets according to their maximum load capacity. for ,in Indicates the fleet The maximum load weight of the vehicle.

3. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 1, characterized in that: The components of the reinforcement learning algorithm are defined as follows: State: The state includes the problem instance state and the current solution state; the problem instance state includes the coordinates, transportation demand, and time window of all nodes; the current solution state includes the front and rear adjacent nodes of each node and the distance between each node and its front and rear adjacent nodes, the vehicle type of the path where the current node is located, the remaining load of the vehicle after serving the current node, the total driving distance of the path where the current node is located, the driving distance when the vehicle arrives at the current node, the total transportation cost of the path where the current node is located, the transportation cost when the vehicle arrives at the current node, and the ratio of the total transportation cost of the path where the current node is located to the total transportation demand; Action: The action includes a pair of delivery nodes and a heuristic operator; Reward: The optimization goal of the problem is to minimize the total cost of vehicle transportation charges to complete all customer orders. The difference between the current solution and the initial solution transportation cost is recorded as , That is the reward value.

4. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 3, characterized in that: The deep reinforcement learning model includes an actor strategy network and a critic evaluation network; The actor strategy network includes an embedding layer, a first encoder, a node decoder and an operator decoder; The problem instance state and the current solution state are respectively input into the embedding layer and embedded into the high-dimensional space to obtain instance embeddings. De-embedding ; By embedding the instance De-embedding Input to the first encoder to get the enhanced embedding ; Enhance the embedding Input node decoder, after passing through the multi-layer perceptron of the node decoder, and then normalized The node pair selection probability matrix is ​​obtained by operation, and the next pick-up and delivery node pair to be operated is selected based on the node pair selection probability matrix. ; The operator selects the decoder based on the pick-up and delivery node pairs Select the next operator to use , by pairing the pick-up and delivery nodes Sum Operator As an action Output to the reinforcement learning environment for iteration; The critic evaluation network includes a second encoder and a fully connected layer; the second encoder is used to process the input problem instance state and the current solution state, and the embedded information is obtained by inputting the problem instance state and the current solution state into the second encoder for calculation. ;Will Input the fully connected layer to calculate the value.

5. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 4, characterized in that: The heuristic operator includes one or more of an intra-path exchange operator, an intra-path relocation operator, an inter-path exchange operator, an inter-path relocation operator, a path splitting operator, and a path deletion operator; Assume that the pick-up and delivery node pairs selected by the actor strategy network are , take the delivery node pair The path is , the operation of the heuristic operator is as follows: In-path exchange operator: exchange the pick-up and delivery node pairs With the path Another pair of nodes in The access order of The node pair selected in a greedy way satisfies the predefined constraints of the problem and causes the smallest cost increase after exchange; In-path relocation operator: first relocate the node pair From the path Delete it, and then add the node pair Reinsert into The position that satisfies the predefined constraints of the problem and causes the smallest cost increase after insertion; Inter-path exchange operator: exchange node pairs With another path Another pair of nodes in The access order of is a randomly selected path, The node pair selected in a greedy way satisfies the predefined constraints of the problem and causes the smallest cost increase after exchange; Inter-path relocation operator: first relocate the node pair From the path and then the node pair Reinsert to another path In the appropriate position; where the path is randomly selected, the node pair The insertion position is the position that satisfies the predefined constraints of the problem and causes the smallest cost increase after insertion; Path splitting operator: first delete the node pair The path , then add at least two vehicles, for All node pairs in are inserted into the paths of at least two new vehicles in the original access order. The new vehicles are selected according to the actual problem scenario, and are selected from the same fleet as the original vehicles or from the fleet with a smaller maximum load than the original vehicles. Path deletion operator: first delete the node pair The path , then All node pairs in are reinserted into other paths. When inserting a node pair, a path is randomly selected and the location that meets the predefined constraints of the problem and causes the smallest cost increase after insertion is selected to perform the insertion operation.

6. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 5, characterized in that: The reinforcement learning environment includes: State update: Based on the problem scenario of heterogeneous vehicle models picking up and delivering goods, state transition rules are constructed. The state transition rules are used to generate the next state according to the actions output by the actor policy network during model training and inference; The state transition rule is defined as: Assume that the pick-up and delivery nodes are selected Sum Operator As the next step, after the selection is completed, the status changes as follows: (1) For node pairs And the node pair The front and back adjacent nodes of the node are updated, and their respective front and back adjacent nodes and front and back adjacent distances are updated; (2) Update node pair The type of vehicle on the route; (3) Update the vehicle service completion node pair The remaining load after (4) Update node pair The total travel distance of the route; (5) Update the vehicle arrival node pair driving distance in hours; (6) Update node pair The total transportation cost of the route; (7) Update the vehicle arrival node pair Transportation costs at the time of delivery; (8) Update node pair The ratio of the total transportation cost of the route to the total transportation demand; (9) For the status update operations described in (1) to (8), update the other nodes in each path involved in the change accordingly; Dataset Generation: Generate datasets for training deep reinforcement learning models.

7. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 6, characterized in that: The data set generation specifically includes: Obtain an existing dataset of pickup and delivery instances, and generate new data based on the existing dataset using coordinate transformation and random perturbation as a dataset for deep reinforcement learning model training; New data is generated by coordinate transformation, including: Invert: Convert geographic coordinates Reverse to get geographic coordinates ; Mirror: Geographic coordinates In Replace with ; Set geographic coordinates In Replace with ; Set geographic coordinates In Replace with , Replace with ; Generate new data by random perturbation, including: the geographic coordinates of each node , by generating random numbers Perturb it to generate new nodes .

8. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 6 or 7, characterized in that: The training process of the reinforcement learning model uses a proximal strategy optimization algorithm to train the deep reinforcement learning model in several batches, specifically including: Initialize the parameters of the actor strategy network and the critic evaluation network; Divide the dataset used for deep reinforcement learning model training into several batches of data instances and train the model in several batches; The training process of each batch includes: using a greedy insertion algorithm to generate an initial solution to the problem instance; based on the initial solution, performing several steps of interaction with the reinforcement learning environment through the deep reinforcement learning model, collecting the action, reward and value of each step; calculating the advantage function value based on the collected actions, rewards and values; and updating the parameters of the actor network and the critic network through a predefined loss function based on the actions, rewards, value functions and advantage function values.

9. A method for path planning for heterogeneous pickup and delivery vehicles according to claim 8, characterized in that: The method of using the greedy insertion algorithm to generate an initial solution to the problem instance specifically includes: A1. Obtain all the pickup and delivery node pairs to be delivered, randomly select the first pair of pickup and delivery nodes, and select the first vehicle from the fleet with the largest load capacity, and insert the selected pickup and delivery node pair into the delivery route of the first selected vehicle; A2. Randomly select a second pair of pick-up and delivery nodes from the remaining pick-up and delivery node pairs, and try to insert the second pair of pick-up and delivery node pairs into the delivery route of the first vehicle. If the second pair of pick-up and delivery node pairs can be inserted into the delivery route of the first vehicle without violating the predefined constraints, the insertion operation is performed; if the second pair of pick-up and delivery node pairs cannot be inserted into the delivery route of the first vehicle, select a second vehicle from the fleet with the second largest load after the first vehicle, and insert the second pair of nodes into the delivery route of the second vehicle; A3. Continue to randomly select a third pair of pick-up and delivery node pairs from the remaining pick-up and delivery node pairs, and try to insert the third pair of pick-up and delivery node pairs into the vehicle delivery routes that have been scheduled for delivery. If the third pair of node pairs cannot be inserted into all the scheduled vehicle delivery routes, continue to select a vehicle from the next fleet in descending order of load capacity, and try to perform the insertion operation; if the fleet with the smallest load capacity was selected when adding a new vehicle last time, select a new vehicle from the fleet with the largest load capacity next time. A4. Loop through steps A2 to A3 until all pickup and delivery nodes are inserted into the vehicle's delivery route, and the initial solution of the actor policy network is obtained.

10. A path planning system for heterogeneous pickup and delivery vehicles, characterized in that: The system comprises: A problem scenario construction module, used to construct a problem scenario of picking up and delivering goods by heterogeneous vehicle types and set an optimization goal, wherein the optimization goal is to minimize the total vehicle transportation billing cost of picking up and delivering goods by heterogeneous vehicle types; A model building module, used to build a deep reinforcement learning model and a reinforcement learning environment using an iterative improvement solution method based on the heterogeneous vehicle model pickup and delivery problem scenario and optimization objectives; A model training module is used to construct a training process of a reinforcement learning model, and based on the training process, a proximal strategy optimization algorithm is used to train the deep reinforcement learning model in several batches to obtain a trained deep reinforcement learning model; The solution generation module is used to use the trained deep reinforcement learning model to solve the actual heterogeneous vehicle model pickup and delivery problem scenario, and obtain the vehicle path planning solution for the heterogeneous vehicle model pickup and delivery problem scenario.

Images