Method for solving multi-vehicle covering distribution problem by state-guided reinforcement learning driven evolutionary algorithm

Abstract

The application discloses a method for solving a multi-vehicle covering distribution problem by using a state-guided reinforcement learning driven evolution algorithm, and comprises the following steps: constructing a customer satisfaction function by using a questionnaire survey, and establishing a mathematical model of the multi-vehicle covering distribution problem; generating an individual and calculating an individual fitness value; selecting two parent individuals by using a roulette strategy, performing cross operation on the two parent individuals, and then putting the two parent individuals into a first individual pool; randomly selecting an individual from the first individual pool, calculating S1 and S2, selecting a crossover operator by using a guided reinforcement learning mechanism, performing the crossover operator to obtain a new individual, and putting the new individual into a second individual pool; calculating a reward value of each mutation operator; combining the first individual pool and the second individual pool, eliminating repeated individuals, and updating an optimal solution; when a termination condition is met, outputting the current optimal solution. The application constructs a mathematical model for simultaneously optimizing a distribution route and a PL selection, and solves the mathematical model by using an evolution algorithm driven by a reinforcement learning, so that a better result can be obtained in a shorter time, and better performance is achieved.

Description

Technical Field

[0001] This invention relates to the field of logistics and distribution, and in particular to a method for solving multi-vehicle coverage delivery problems using a state-guided reinforcement learning-driven evolutionary algorithm. Background Technology

[0002] my country's e-commerce transaction volume has continued to grow in recent years, leading to a surge in parcel delivery volume. A large number of online shopping parcels need to be delivered from merchants to customers. The delivery time by couriers and the receipt time by customers are often out of sync, easily resulting in secondary deliveries and low efficiency in last-mile delivery. To overcome this bottleneck, deploying parcel lockers (PLs) in last-mile delivery areas has become an effective solution. Courier companies temporarily store parcels in PLs, and customers can retrieve them at their own pace. Properly configured PLs can significantly improve last-mile delivery efficiency and reduce company delivery costs.

[0003] While the widespread deployment of delivery warehouses (PLs) has facilitated parcel delivery, it has also brought challenges to the last-mile delivery process. Couriers not only need to confirm the delivery mode (door-to-door delivery or PL storage) before serving customers, but also need to determine the route sequence for serving customers in order to improve customer satisfaction. This delivery scenario can be modeled as the Covering Delivery Problem (CDP): it includes three vertices: warehouses, PLs, and customers. Customers can be categorized into three types: "door-to-door delivery," "PL storage," and "both delivery modes are acceptable." The goal is to plan a Hamiltonian cycle with the minimum length and PL usage cost to meet the requirements of all customer types.

[0004] Compared to real-world scenarios, existing CDP modeling has two shortcomings: first, it only considers planning one route, which is insufficient when multiple vehicles are needed to handle large delivery volumes; second, when multiple delivery lanes (PLs) are available near a customer, it fails to consider customer satisfaction when selecting a PL. Current research on vehicle routing problems primarily focuses on the deviation between the courier's delivery time and the customer's expected delivery time for customer satisfaction. However, as the complexity of real-world delivery problems continues to increase, a single method is no longer sufficient to meet the demand for efficient problem-solving. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a simple state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem.

[0006] The technical solution of this invention to solve the above-mentioned technical problem is: a method for solving the multi-vehicle coverage delivery problem using a state-guided reinforcement learning-driven evolutionary algorithm, comprising the following steps:

[0007] Step 1: Construct a customer satisfaction function using a questionnaire survey, and then establish a mathematical model for the multi-vehicle coverage delivery problem;

[0008] Step 2: Design a hybrid heuristic method to generate individuals and calculate their fitness values;

[0009] Step 3: Use a roulette wheel strategy to select two parent individuals for crossover. Execute a greedy repair strategy on the individuals generated after the crossover and put the repaired individuals into the first pool.

[0010] Step 4: Randomly select individuals from the first pool, calculate the change in the variance of individual fitness and the change in the fitness value of the optimal solution in the population, select the crossover operator using the guided reinforcement learning mechanism and execute it to obtain new individuals, and put the new individuals into the second pool;

[0011] Step 5: Count the number of times each crossover operator is selected and the number of times it is effective, calculate the efficiency value of each crossover operator, calculate the reward value of each crossover operator, and update the Q table, which is the state-action value table.

[0012] Step 6: Merge the first pool with the second pool, eliminate duplicate individuals, and retain the individuals with the best fitness values. For each individual, update the optimal solution;

[0013] Step 7: If the optimal solution is continuous Not updated. If the value is a constant, then an initialization strategy is used to randomly generate individuals to replace 20% of the individuals in the current population;

[0014] Step 8: If the set maximum number of iterations is reached, output the current optimal solution; otherwise, go to step 3.

[0015] The method described above for solving the multi-vehicle coverage delivery problem using state-guided reinforcement learning-driven evolutionary algorithm, in step 1, involves constructing a customer satisfaction function using a questionnaire survey as follows:

[0016] A questionnaire survey was used to characterize the relationship between customer satisfaction and pickup distance. The specific process is as follows:

[0017] First, develop a questionnaire with three key questions: How far from home or work do you feel satisfied with picking up your packages from the locker? How far away do you feel dissatisfied when picking up packages? What is your average walking distance per trip?

[0018] Then, the questionnaires were distributed, collected, and processed through the Wenjuanxing platform. The statistical analysis revealed the following findings: First, the average walking distance for customers varies depending on age, gender, and physical condition; second, the ratio of the customer's satisfactory pickup distance to their average walking distance is relatively fixed; third, picking up packages beyond the customer's average walking distance leads to dissatisfaction.

[0019] Finally, use equation (1) to quantify customer satisfaction:

[0020] (1);

[0021] in, For customer satisfaction, The value range is [0,1]; For the customer's pickup distance; The average distance a customer walks on a single daily trip. The satisfaction decay coefficient, ; The ratio of the customer's satisfactory pickup distance to the average walking distance per trip. .

[0022] The above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem, in step 1, involves establishing the mathematical model of the multi-vehicle coverage delivery problem as follows:

[0023] In a last-mile delivery scenario deploying parcel lockers (PLs), delivery personnel either deliver parcels directly to customers or store them in parcel lockers near the customer. The logistics company has a fleet of delivery vehicles, each departing from the central warehouse and returning after completing its mission. The question is how to arrange these delivery routes to ensure that each customer is either visited or that a parcel locker within their pickup area is visited by a delivery vehicle. The goal of the multi-vehicle coverage delivery problem is to optimize the total vehicle travel distance and customer satisfaction while satisfying vehicle capacity constraints. Therefore, the mathematical model for the multi-vehicle coverage delivery problem is established as follows:

[0024] (2);

[0025] (3);

[0026] (4);

[0027] (5);

[0028] (6);

[0029] (7);

[0030] (8);

[0031] (9);

[0032] (10);

[0033] (11);

[0034] in, For customer satisfaction; Represents the function to be minimized; Indicates vehicle index; Indicates a collection of vehicles; Represents a set of customers; Indicates the customer index; This indicates a collection of parcel lockers; Indicates the parcel locker index; Represents a collection of central warehouses; Indicates the central repository index; Represents a set of nodes. Parcel lockers, central warehouses, and customers all serve as nodes; Indicates the node index; Indicates the node index; Represents an edge ( ) length, side ( ) is a slave node To the node The path; Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 if it is set to 1, otherwise the value is 0. This represents the fixed operating cost per vehicle. This represents the cost of punishment for maximum dissatisfaction. Indicates customer With parcel lockers Related decision variables, if customer Package lockers Service The value is 1 if it is set to 1, otherwise the value is 0. Represents the covering relation matrix, if r b Package storage lockers covering an area with a radius of [radius value]. but The value is 1 otherwise the value is 0. The average distance a customer walks on a single daily trip. Indicates customer With vehicles Related decision variables, if customer In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( ) is a slave node To the node The path; Represents a node With vehicles Related decision variables, if nodes In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( (From the central warehouse) To the node The path; Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( ) is a slave node To the central warehouse The path; Indicates central warehouse With vehicles Related decision variables, such as the central warehouse In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Represents a node Demand; Represents a node With vehicles Related decision variables, if nodes In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates parcel locker With vehicles Related decision variables, such as package lockers In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle capacity; Indicates vehicle From node To the node The load capacity afterward; Indicates vehicle From node To the node The load capacity afterward; Indicates the number of nodes in the set; This represents the total number of vehicles in the vehicle set.

[0035] The above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem employs three initial solution generation strategies in step 2 during the generation of individuals using a hybrid heuristic method. The first strategy is a random strategy, randomly selecting nodes from the parcel locker (PL) and customer set and inserting them into the current path until all customer nodes are served. The second strategy is a nearest neighbor insertion strategy, starting from the central warehouse node and selecting the node closest to the current node and satisfying the capacity constraint from the set of nodes to be visited, inserting it into the path until all customer nodes are served. The third strategy is a saving algorithm, using saving values ​​to guide the selection of insertion points until all customer nodes are served. In all three strategies, if the currently selected node is the parcel locker (PL), the coverage relation matrix is ​​used to add customer points that can be served by the parcel locker (PL) to the current path.

[0036] The above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem, in step 3, uses a roulette wheel strategy to select two parent individuals for crossover. During crossover, the coverage relationship between the customer and the parcel locker PL is not considered initially. Two-point crossover and multi-point crossover are performed based on the genetic information of the two parent individuals participating in the crossover, as detailed below:

[0037] Two-point crossover: Two crossover sites are randomly selected from the two paired paternal chromosomes, and the genes in the two position intervals are exchanged. The paternal chromosome is the paternal individual.

[0038] Multiple crossing over: Multiple crossing sites are randomly selected from the two paired parent chromosomes, and the genes at the crossing sites are exchanged.

[0039] In the above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem, step 4 uses S1 to represent the change in the variance of the population's individual fitness and S2 to represent the change in the fitness value of the optimal solution in the population. If the fitness of the current optimal individual in the population is better than that of the previous generation's optimal individual, then S2 is positive; otherwise, it is negative. Positive indicates improvement, and negative indicates no improvement. For S1, a monotonically decreasing variance function of the population's individual fitness is designed to guide the reinforcement learning's recognition of S1, with the following specific form:

[0040] (12);

[0041] in, For the first The variance of individual fitness in the population; For the first The variance of individual fitness in the population; For population generation index; The variance of the fitness of individuals in the initial population; The is the rate of decrease, with a value range of [0.8, 1); mod is the modulo function, used to calculate the modulo value every . envoy attenuation, It is a constant;

[0042] The first Actual variance of individual fitness in a generation population and In comparison, if If S1 is positive, then S1 is positive; otherwise, it is negative.

[0043] In the above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem, step 4 involves mutation operators used for local search in the evolutionary algorithm. The following nine mutation operators are designed:

[0044] 2-Swap: Select a single path and swap the two access nodes;

[0045] Fragment Reverse: Select a single path and reverse the sub-fragments within that path;

[0046] 3-Swap: Select a single path and choose three locations to reassemble the node fragment;

[0047] Reinsertion: Select a single path and move the node to a new position within the same path;

[0048] Path-to-Path Swap: Swap two nodes between different paths;

[0049] Inter-path re-insertion: Select a node at any position in one path and insert it into another path;

[0050] Service mode switching: Select any customer node and delete it, then insert the PL node corresponding to this customer node into the original position. The PL node is the package locker node.

[0051] PL Transformation: Select any PL node and replace it with another unvisited PL node;

[0052] Destroy-Repair: Remove any item from a single path After each node, it is reinserted into the optimal position.

[0053] The above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem, in step 4, the process of selecting the crossover operator using a guided reinforcement learning mechanism is as follows:

[0054] Nine mutation operators are used as the actions that the agent can take in each state. The action selection probability is calculated based on the current state and the evaluation value corresponding to the action, and the corresponding action is executed according to the action selection probability. The evaluation value is... value, The value is updated using the following formula:

[0055] (13);

[0056] in, It is the first State-action pairs during population evolution value; It is the first State-action pairs during population evolution value; For population generation index; For the agent in the first The current state of the population; For the agent in the first The actions that the next generation of the species intends to take; For the learning rate, control the pair The magnitude of the value update; For the first The reward value of the next generation; This is a discount factor, representing the degree of importance attached to future rewards; Indicates that the agent is in the first... The maximum of all possible actions in the current state of the population value, For action, It is the set of all actions.

[0057] In the above-described state-guided reinforcement learning-driven evolutionary algorithm for solving the multi-vehicle coverage delivery problem, step 5, the reward value is calculated as follows:

[0058] (14);

[0059] (15);

[0060] in, For the first During population evolution, agents respond to actions The reward value; For action In the The improvement effect of generational populations; For action In the The improvement effect of generational populations; For the first Agents choosing actions in population evolution The number of times; For the first Agents choosing actions in population evolution The number of times improvements were made.

[0061] The beneficial effects of this invention are as follows: This invention addresses the multi-vehicle coverage delivery problem by constructing a mathematical model that simultaneously optimizes delivery routes and PL (Placement Provider) selection. A customer satisfaction representation function is designed within the model, incorporating questionnaire survey results. A reinforcement learning-driven evolutionary algorithm is then used to solve the multi-vehicle coverage delivery problem. During the solution process, firstly, a hybrid heuristic method is designed to generate a high-quality initial population; then, nine mutation operators are designed based on the characteristics of the multi-vehicle coverage delivery problem for efficient searching of satisfactory feasible solutions; next, a reinforcement learning-driven search mechanism is proposed for adaptive selection of appropriate operators; finally, a monotonically decreasing variance function of the population's individual fitness is designed to guide the convergence process of the agent's learning algorithm. Experimental results show that this invention achieves better results in a shorter time compared to existing technologies when solving small, medium, and large-scale problems, demonstrating superior performance. Attached Figure Description

[0062] Figure 1 This is the overall flowchart of the present invention.

[0063] Figure 2 This is a diagram illustrating the problem of multi-vehicle delivery coverage.

[0064] Figure 3 This is an example diagram of the encoding.

[0065] Figure 4 This is an example diagram showing the intersection of two points.

[0066] Figure 5 This is an example diagram showing multiple intersections.

[0067] Figure 6This is a schematic diagram of 2-exchange between mutation operator paths.

[0068] Figure 7 This is a schematic diagram of inter-path re-insertion for the mutation operator.

[0069] Figure 8 A schematic diagram illustrating the service mode switching for the mutation operator.

[0070] Figure 9 This is a schematic diagram of the PL transformation operator.

[0071] Figure 10 A comparison chart showing the results of running small-scale examples S11_3, S14_3, S16_3, S16_4, and S22_3.

[0072] Figure 11 A comparison chart showing the results of running small-scale examples S41_4, S51_3, S75_3, S75_4, and S85_3.

[0073] Figure 12 This is a comparison chart of the running results for medium-scale examples M100_3, M100_4, M105_2, and M105_3.

[0074] Figure 13 This is a comparison chart of the running results of medium-scale examples M120_2, M120_3, M130_2, and M130_3.

[0075] Figure 14 This is a comparison chart of the running results for medium-scale examples M150_2, M150_3, M160_2, and M160_3.

[0076] Figure 15 This is a comparison chart of the running results of medium-scale examples M180_2, M180_3, M200_2, and M200_3.

[0077] Figure 16 A comparison chart showing the results of running large-scale examples L220_2, L220_3, L235_2, and L235_3.

[0078] Figure 17 A comparison chart showing the results of running large-scale examples L285_2, L285_3, L320_2, and L320_3.

[0079] Figure 18 A comparison chart showing the results of running large-scale examples L435_2, L435_3, and L530_2. Detailed Implementation

[0080] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0081] like Figure 1 As shown, the method of using state-guided reinforcement learning-driven evolutionary algorithm to solve the multi-vehicle coverage delivery problem includes the following steps:

[0082] Step 1: Construct a customer satisfaction function using a questionnaire survey, and then establish a mathematical model for the multi-vehicle coverage delivery problem.

[0083] In last-mile delivery scenarios using parcel lockers (PLs), the convenience of the handover service is a key factor influencing customer satisfaction. The distance customers travel to pick up their parcels is often a crucial indicator of their PL choice, directly determining their satisfaction level. Therefore, this invention uses the customer's travel distance to pick up their parcels to measure customer satisfaction. However, in practice, individual differences exist among customers, and different customers have varying levels of satisfaction with the same pick-up distance. Therefore, this invention further employs a questionnaire survey to accurately characterize the relationship between customer satisfaction and pick-up distance. The process of constructing a customer satisfaction function using a questionnaire survey is as follows:

[0084] A questionnaire survey was used to characterize the relationship between customer satisfaction and pickup distance. The specific process is as follows:

[0085] First, develop a questionnaire with three key questions: How far from home or work do you feel satisfied with picking up your packages from the locker? How far away do you feel dissatisfied when picking up packages? What is your average walking distance per trip?

[0086] Then, a questionnaire was completed by 350 residents and company employees in three cities who frequently use PL (Private Place) for package pickup. The questionnaires were distributed, collected, and processed through the Wenjuanxing (Questionnaire Star) platform, resulting in 332 valid questionnaires. Statistical analysis revealed the following findings: First, the average walking distance per customer varies due to age, gender, and physical condition; second, the ratio of the customer's satisfactory pickup distance to their average daily walking distance is relatively constant; third, picking up packages beyond the customer's average daily walking distance leads to dissatisfaction.

[0087] Finally, use equation (1) to quantify customer satisfaction:

[0088] (1);

[0089] in, For customer satisfaction, The value range is [0,1]; For the customer's pickup distance; The average distance a customer walks on a single daily trip. This is the satisfaction decay coefficient. ; The ratio of the customer's satisfactory pickup distance to the average walking distance per trip. .

[0090] In a last-mile delivery scenario deploying parcel lockers (PLs), delivery personnel either deliver parcels directly to customers or store them in nearby PLs. The logistics company has a fleet of delivery vehicles, each departing from a central warehouse, completing its mission, and returning. The challenge is to arrange these delivery routes to ensure that each customer is either visited or that any PL within their pickup area is visited by a delivery vehicle. The goal of this multi-vehicle coverage delivery problem is to optimize the total vehicle travel distance and customer satisfaction while meeting vehicle capacity constraints. Figure 2 This illustrates an example of a multi-vehicle delivery coverage problem, where solid lines represent vehicle delivery routes and dashed circles represent the coverage area of ​​the customer's travel radius at the center of the circle.

[0091] The process of establishing a mathematical model for the multi-vehicle coverage delivery problem is as follows:

[0092] Therefore, the mathematical model for the multi-vehicle coverage delivery problem is established as follows:

[0093] (2);

[0094] (3);

[0095] (4);

[0096] (5);

[0097] (6);

[0098] (7);

[0099] (8);

[0100] (9);

[0101] (10);

[0102] (11);

[0103] in, For customer satisfaction; Represents the function to be minimized; Indicates vehicle index; Indicates a collection of vehicles; Represents a set of customers; Indicates the customer index; This indicates a collection of parcel lockers; Indicates the parcel locker index; Represents a collection of central warehouses; Indicates the central repository index; Represents a set of nodes. Parcel lockers, central warehouses, and customers all serve as nodes; Indicates the node index; Indicates the node index; Represents an edge ( ) length, side ( ) is a slave node To the node The path; Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 if it is set to 1, otherwise the value is 0. This represents the fixed operating cost per vehicle. This represents the cost of punishment for maximum dissatisfaction. Indicates customer With parcel lockers Related decision variables, if customer Package lockers Service The value is 1 if it is set to 1, otherwise the value is 0. Represents the covering relation matrix, if r b Package storage lockers covering an area with a radius of [radius value]. but The value is 1 otherwise the value is 0. The average distance a customer walks on a single daily trip. Indicates customer With vehicles Related decision variables, if customer In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( ) is a slave node To the node The path; Represents a node With vehicles Related decision variables, if nodes In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( (From the central warehouse) To the node The path; Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( ) is a slave node To the central warehouse The path; Indicates central warehouse With vehicles Related decision variables, such as the central warehouse In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Represents a node Demand; Represents a node With vehicles Related decision variables, if nodes In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates parcel locker With vehicles Related decision variables, such as package lockers In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle capacity; Indicates vehicle From node To the node The load capacity afterward; Indicates vehicle From node To the node The load capacity afterward; Indicates the number of nodes in the set; Let represent the total number of vehicles in the vehicle set; Equation (2) is the objective function, which minimizes the total transportation cost and the cost of penalties for customer dissatisfaction; Equation (3) ensures that any customer point is either visited or served by at least one parcel locker PL; Equation (4) ensures node inflow and outflow balance; Equation (5) ensures that any vehicle must depart from the distribution center and return to the distribution center; Equation (6) is the vehicle capacity constraint; Equations (7) and (8) are both sub-loop elimination constraints; Equation (9) is the vehicle quantity constraint; Equation (10) is the PL access constraint, which states that if a customer chooses to pick up a parcel locker PL, then that parcel locker PL must be visited by at least one vehicle; Equation (11) ensures that a customer can only be served by a parcel locker PL within its coverage area.

[0104] Step 2: Design a hybrid heuristic method to generate individuals and calculate their fitness values.

[0105] The algorithm uses integer encoding, with a sequence containing the central warehouse, PL, and customer number as the individual chromosome, to represent the order in which vehicles visit. Figure 3 An example of coding is given, where code 0 represents the central warehouse, codes 1 and 2 are PLs, and the rest are customer points. This chromosome represents the delivery route arrangement of two vehicles, with driving routes 0240 and 05610 respectively.

[0106] The quality of the initial population has a significant impact on the performance of the evolutionary algorithm. To generate a high-quality initial population, three initial solution generation strategies are adopted. The first strategy is a random strategy, which randomly selects nodes from the parcel locker PL and the customer set and inserts them into the current path until all customer nodes are served. The second strategy is a nearest neighbor insertion strategy, which starts from the central warehouse node and selects the node closest to the current node and satisfying the capacity constraint from the set of nodes to be visited and inserts it into the path until all customer nodes are served. The third strategy is a saving algorithm, which uses the saving value to guide the selection of insertion points until all customer nodes are served. In all three strategies, if the currently selected node is the parcel locker PL, the covering relation matrix is ​​used to add customer points that can be served by the parcel locker PL to the current path.

[0107] Step 3: Use a roulette wheel strategy to select two parent individuals for crossover. Then, apply a greedy repair strategy to the individuals generated after the crossover and put the repaired individuals into the first pool.

[0108] A roulette wheel strategy is used to select two parent individuals for crossover. During crossover, the overlap between the customer and the parcel locker PL is not considered initially. Two-point crossover and multi-point crossover are conducted based on the genetic information of the two parent individuals participating in the crossover, as detailed below:

[0109] Two-point crossover: Two crossover sites are randomly selected from the paired paternal chromosomes, and the genes within the two site intervals are exchanged. The paternal chromosomes are the paternal individuals; for example... Figure 4 As shown, crossover occurs at positions 3 to 5 of the parent chromosome, resulting in the exchange of segments and the generation of two offspring chromosomes.

[0110] Multiple crossing over: Multiple crossing sites are randomly selected from the paired parent chromosomes, and the genes at the crossing sites are exchanged. For example... Figure 5 As shown, genes at positions 2, 6, and 7 of the father chromosome are randomly selected and exchanged to generate two offspring chromosomes.

[0111] Since the coverage relationship between customers and PLs is not considered during crossover, problems such as duplicate genes, redundant coverage (multiple PLs can serve the same customer), missing nodes (some customers or their covered PLs are not accessed), and line overload may occur after crossover. Therefore, this invention adopts greedy repair of the chromosomes after crossover. The greedy repair method is an existing technology and will not be described in detail here.

[0112] Step 4: Randomly select individuals from the first pool, calculate the change in the variance of individual fitness and the change in the fitness value of the optimal solution in the population, select the crossover operator using the guided reinforcement learning mechanism and execute it to obtain new individuals, and put the new individuals into the second pool.

[0113] Let S1 represent the change in the variance of individual fitness in the population, and S2 represent the change in the fitness value of the optimal solution in the population. Both S1 and S2 have two values: improved (denoted as +) and unimproved (denoted as -), for a total of 4 configurations, as shown in Table 1. The values ​​of S1 and S2 are selected as follows:

[0114] Table 1 State Characterization

[0115]

[0116] If the fitness of the current best individual in the population is better than that of the best individual in the previous generation, then S2 is positive; otherwise, it is negative. Positive indicates improvement, and negative indicates no improvement. For S1, the variance of the population fitness in typical evolutionary algorithms decreases with the number of iterations, but the change in variance has a certain degree of randomness. The variance between adjacent iterations may increase or decrease. Algorithm crossover, mutation operator characteristics, and environment selection strategies all affect the variance change. This poses a challenge to the state-action learning mechanism of reinforcement learning. To guide the algorithm convergence in an orderly manner, this invention designs a monotonically decreasing variance function of the population fitness to guide reinforcement learning in recognizing S1, as follows:

[0117] (12);

[0118] in, For the first The variance of individual fitness in the population; For the first The variance of individual fitness in the population; For population generation index; The variance of the fitness of individuals in the initial population; The is the rate of decrease, with a value range of [0.8, 1); mod is the modulo function, used to calculate the modulo value every . envoy attenuation, It is a constant;

[0119] The first Actual variance of individual fitness in a generation population and In comparison, if If S1 is positive, then S1 is positive; otherwise, it is negative.

[0120] Mutation operators are used for local search in evolutionary algorithms. The following nine mutation operators are designed:

[0121] 2-Swap: Select a single path and swap two access nodes; for example, in a path [0,1,4,5,2,3], swap the nodes at positions 3 and 5 to get a new path [0,1,2,5,4,3].

[0122] Fragment Reversal: Select a single path and reverse the sub-fragments in the path; for example, in a path [0,1,4,5,2,3], reverse the node fragments from position 3 to position 5 to obtain the new path [0,1,2,5,4,3].

[0123] 3-Exchange: Select a single path and select three position points to reassemble the node fragments; for example, in a path [0,1,4,5,2,3], select positions 4, 5, and 6 to divide the path into three fragments, and reconnect the fragments to obtain a new path [0,1,4,2,3,5].

[0124] Reinsertion: Select a single path and move the node to a new position within the same path; for example, in a path [0,1,4,5,2,3], after moving the node at position 6 to the node at position 3, a new path [0,1,4,3,5,2] is obtained.

[0125] Path swap 2: Swap two nodes between different paths; for example, swapping node 1 of path [0,1,4,5] with node 6 of path [0,2,3,6] results in the new paths [0,6,4,5] and [0,2,3,1]. Figure 6 As shown.

[0126] Inter-path re-insertion: Select a node at any position in one path and insert it into another path; for example, remove node 7 from the path [0,2,3,7] and insert node 4 into the path [0,1,4,5] to obtain the new paths [0,2,3] and [0,1,4,7,5]. Figure 7 As shown.

[0127] Service mode switching: Select any customer node and delete it. Then, insert the corresponding PL node (package locker node) into its original position. For example, customer 5 can be served by node 8, which is the PL node corresponding to customer 5. Delete node 5 from the path [0,1,4,5] and insert node 8 to obtain the new path [0,1,4,8]. Figure 8 As shown.

[0128] PL Transformation: Select any PL node and replace it with another unvisited PL node; for example, in the path [0,1,4,5], replace node 1, which represents a PL node, with node 8, which represents another PL node, to obtain the new path [0,8,4,5]. Figure 9 As shown.

[0129] Destroy-Repair: Remove any item from a single path After removing nodes, they are reinserted into the optimal position. For example, in the path [0,1,4,5,2,3], nodes 1 and 3 are removed and reinserted into the optimal position of the current path to obtain the new path [0,4,1,5,3,2].

[0130] Among the mutation operators mentioned above, the five mutation operators 2-swap, 3-swap, fragment inversion, re-insertion, and destruction-repair only act on a single path and do not affect the node allocation between paths. However, the four mutation operators 2-swap between paths, insertion between paths, service mode switching, and PL transformation involve node transformation between paths, which can lead to problems such as redundant coverage, missing nodes, and line overload. Greedy repair is used to implement repair.

[0131] The process of selecting crossover operators using a guided reinforcement learning mechanism is as follows:

[0132] Nine mutation operators are used as the actions that the agent can take in each state. The action selection probability is calculated based on the current state and the evaluation value corresponding to the action, and the corresponding action is executed according to the action selection probability. The evaluation value is... value, The value is updated using the following formula:

[0133] (13);

[0134] in, It is the first State-action pairs during population evolution value; It is the first State-action pairs during population evolution value; For population generation index; For the agent in the first The current state of the population; For the agent in the first The actions that the next generation of the species intends to take; For the learning rate, control the pair The magnitude of the value update; For the first The reward value of the next generation; This is a discount factor, representing the degree of importance attached to future rewards; Indicates that the agent is in the first... The maximum of all possible actions in the current state of the population value, For action, It is the set of all actions.

[0135] Step 5: Count the number of times each crossover operator is selected and the number of times it is effective, calculate the efficiency value of each crossover operator, calculate the reward value of each crossover operator, and update the Q table, which is the state-action value table.

[0136] The reward value is calculated as follows:

[0137] (14);

[0138] (15);

[0139] in, For the first During population evolution, agents respond to actions The reward value; For action In the The improvement effect of generational populations; For action In the The improvement effect of generational populations; For the first Agents choosing actions in population evolution The number of times; For the first Agents choosing actions in population evolution The number of times improvements were made.

[0140] Step 6: Merge the first pool with the second pool, eliminate duplicate individuals, and retain the individuals with the best fitness values. For each individual, update the optimal solution.

[0141] Step 7: If the optimal solution is continuous Not updated. If the value is a constant, then an initialization strategy is used to randomly generate individuals to replace 20% of the individuals in the current population.

[0142] Step 8: If the set maximum number of iterations is reached, output the current optimal solution; otherwise, go to step 3.

[0143] Experimental results

[0144] All algorithms were run in the same computing environment to ensure the comparability and repeatability of the results. The test platform hardware configuration was an Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz and a 1.80 GHz processor. All algorithms were developed using Python 3.8 and compiled and run in the PyCharm 2024.3.5 development environment. Each algorithm was run independently 20 times on each test instance, and its performance metrics were recorded. A Mann–Whitney U test with a significance level of 0.05 was used to determine whether there were significant differences in algorithm performance. Results with significantly inferior performance to the algorithm of this invention (QEA) were marked with “”. "mark.

[0145] Since there are no standard test cases for the problem under study, this invention constructs problem cases based on the TSPLIB standard case set. Specifically, the first node in the original case is the distribution center, and the remaining nodes are set as PL nodes and customer nodes in a 4:6 ratio, with the average walking distance of a customer per trip being the data source. The random number seed is generated by multiplying the average distance between nodes in the example by a random decimal between 0 and 1. For nodes with the same point set, different random number seeds are used to generate different examples. Based on the above method, this invention generates three test example sets: a small-scale example set S (v < 100), a medium-scale example set M (100 ≤ v ≤ 200), and a large-scale example set L (200 < v ≤ 600). The example names are Tv_s, where T, v, and s represent the example set number (S / M / L), the total number of nodes, and the random number seed used, respectively.

[0146] This invention employs a comparative test using the solver GUROBI, the Genetic Algorithm (GA), the Memetic Algorithm (MA), and the algorithm of this invention (QEA). The parameter settings for each algorithm are shown in Table 2. Tables 3 to 8 respectively show the performance of each algorithm on small, medium, and large-scale test sets. The computation time in Tables 3 to 8 is in seconds, and the evaluation index is deviation. The calculation method is shown in Equation (16).

[0147] (16);

[0148] in, This indicates the deviation between QEA and the comparison algorithm. This represents the indicator value of QEA. This indicates the metric value of the comparison algorithm. The metric value can be the target value or the computation time.

[0149] Table 2 Main parameter settings for each algorithm

[0150]

[0151] Small-scale test case:

[0152] Table 3 shows the test results of QEA and GUROBI on a small set of examples. Within one hour of computation, GUROBI only obtained the optimal solution for examples with fewer than 22 nodes, and the computation time of GUROBI increased exponentially with the increase of example size. For examples with more than 22 nodes, GUROBI could only obtain a feasible solution within one hour. On average, the minimum objective value of QEA and the minimum objective value of GUROBI differed by only 3.1%. In addition, QEA obtained the optimal solution for 4 examples.

[0153] Table 3. Test results on a small-scale test set (QEA vs GUROBI)

[0154]

[0155] Table 4 shows the test results of QEA, GA, and MA on a small-scale test set. Compared with GA and MA, QEA improved the target average by 7.8% and 2.4%, respectively. In terms of computation time, all three intelligent optimization algorithms can obtain a satisfactory solution within 24 seconds.

[0156] Table 4 Test results on a small set of examples (QEA vs GA, MA)

[0157]

[0158] Figure 10 , Figure 11 The results show a comparison of the objective values ​​of GA, MA, and QEA on small-scale examples. It can be seen that QEA has better solution quality than GA and MA, and QEA's advantage becomes more obvious as the problem size increases.

[0159] Medium-scale test cases:

[0160] Table 5 shows the test results of QEA and GA on a medium-scale test set, and Table 6 shows the test results of QEA and MA on the same set. Since GUROBI could not obtain a feasible solution within an acceptable time, its results are not shown in Tables 5 and 6. Compared to GA and MA, QEA improved the target minimum by 9.1% and 10.3%, respectively, and improved the target average by 10.9% and 11.4%, respectively. In terms of computation time, the average running times of GA, MA, and QEA were 31.8 seconds, 77.8 seconds, and 71.9 seconds, respectively, with QEA reducing the time by 7.6% compared to MA.

[0161] Table 5. Test results for medium-sized test cases (QEA vs GA)

[0162]

[0163] Table 6. Test results for medium-sized test sets (QEA vs MA)

[0164]

[0165] Figures 12-15 The results show a comparison of the objective values ​​of GA, MA, and QEA on medium-scale examples. It can be seen that the solution quality of QEA is still better than that of GA and MA, and it also has better stability on most examples.

[0166] Large-scale test cases:

[0167] Table 7 shows the test results of QEA and MA on a large-scale test set, and Table 8 shows the test results of QEA and GA on the same set. Since GUROBI could not obtain a feasible solution within an acceptable time, its results are not shown in Tables 7 and 8. Compared to GA and MA, QEA improved the target minimum by 15.3% and 11.0%, respectively, and improved the target average by 16.5% and 12.0%, respectively. In terms of computation time, the average running times of GA, MA, and QEA were 189.8 seconds, 234.0 seconds, and 212.7 seconds, respectively, with QEA reducing the time by 9.1% compared to MA.

[0168] Table 7. Test results on a large-scale test set (QEA vs MA)

[0169]

[0170] Table 8. Test results for large-scale test sets (QEA vs GA)

[0171]

[0172] Figures 16-18The comparison results of GA, MA and QEA on large-scale tests are shown, and it can be seen that QEA still shows good performance in scenarios with a large number of nodes.

[0173] The test results of the three types of cases show that GA, MA, and QEA outperform the solver GUROBI when solving cases with more than 100 nodes. MA outperforms GA in overall performance due to its local search mechanism and lower crossover probability. QEA, with the same main parameter settings as MA, introduces a reinforcement learning mechanism (i.e., Q-learning mechanism) to guide the selection of mutation operators, making its overall performance slightly better than MA.

[0174] ablation experiment

[0175] To further test the impact of the state guidance mechanism on the solution performance of the algorithm of this invention, the following ablation experiment was conducted: The state guidance mechanism S1 was removed, and instead, the change in the variance of the fitness of individuals in the population before and after the evolutionary process was calculated. If the variance increased, S1 was marked with "+", otherwise it was marked with "-", while keeping other settings and parameters of the algorithm unchanged. This algorithm was denoted as QEA1 and tested on small, medium, and large-scale example sets. The results in Tables 9, 10, and 11 show that the state guidance mechanism does not increase the computational cost of the algorithm. On small, medium, and large-scale example sets, the minimum target value of QEA is improved by 2.6%, 5.1%, and 5.6% respectively, and the average target value is improved by 1.7%, 5.6%, and 7.7% respectively. It can be seen that QEA uses a monotonically decreasing population variance reference function as a benchmark to guide the agent to find the best solution more efficiently, thereby more effectively leveraging the exploratory ability of reinforcement learning. Moreover, the performance improvement brought by the guidance mechanism is more significant as the problem size increases.

[0176] Table 9 Ablation Experiment Results on Small-Scale Case Sets

[0177]

[0178] Table 10 Ablation Experiment Results on Medium-Sized Case Sets

[0179]

[0180] Table 11 Ablation Experiment Results on Large-Scale Case Sets

[0181]

[0182] Experimental results based on a large number of examples show that the algorithm of this invention can achieve better results in a shorter time than GUROBI when solving small, medium and large examples, and can achieve better performance than GA and MA on all examples.

Claims

1. A method for solving the multi-vehicle coverage delivery problem using a state-guided reinforcement learning-driven evolutionary algorithm, characterized in that, Includes the following steps: Step 1: Construct a customer satisfaction function using a questionnaire survey, and then establish a mathematical model for the multi-vehicle coverage delivery problem; Step 2: Design a hybrid heuristic method to generate individuals and calculate their fitness values; Step 3: Use a roulette wheel strategy to select two parent individuals for crossover. Execute a greedy repair strategy on the individuals generated after the crossover and put the repaired individuals into the first pool. Step 4: Randomly select individuals from the first pool, calculate the change in the variance of individual fitness and the change in the fitness value of the optimal solution in the population, select the crossover operator using the guided reinforcement learning mechanism and execute it to obtain new individuals, and put the new individuals into the second pool; Step 5: Count the number of times each crossover operator is selected and the number of times it is effective, calculate the efficiency value of each crossover operator, calculate the reward value of each crossover operator, and update the Q table, which is the state-action value table. Step 6: Merge the first pool with the second pool, eliminate duplicate individuals, and retain the individuals with the best fitness values. For each individual, update the optimal solution; Step 7: If the optimal solution is continuous Not updated. If the value is a constant, then an initialization strategy is used to randomly generate individuals to replace 20% of the individuals in the current population; Step 8: If the set maximum number of iterations is reached, output the current optimal solution; otherwise, go to step 3.

2. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 1, characterized in that, In step 1, the process of constructing a customer satisfaction function using a questionnaire survey is as follows: A questionnaire survey was used to characterize the relationship between customer satisfaction and pickup distance. The specific process is as follows: First, develop a questionnaire with three key questions: How far from home or work do you feel satisfied with picking up your packages from the locker? How far away do you feel dissatisfied when picking up packages? What is your average walking distance per trip? Then, the questionnaires were distributed, collected, and processed through the Wenjuanxing platform. The statistical analysis revealed the following findings: First, the average walking distance for customers varies depending on age, gender, and physical condition; second, the ratio of the customer's satisfactory pickup distance to their average walking distance is relatively fixed; third, picking up packages beyond the customer's average walking distance leads to dissatisfaction. Finally, use equation (1) to quantify customer satisfaction: （1）； in, For customer satisfaction, The value range is [0,1]; For the customer's pickup distance; The average distance a customer walks on a single daily trip. This is the satisfaction decay coefficient. ; The ratio of the customer's satisfactory pickup distance to the average walking distance per trip. .

3. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 2, characterized in that, In step 1, the process of establishing a mathematical model for the multi-vehicle coverage delivery problem is as follows: In a last-mile delivery scenario deploying parcel lockers (PLs), delivery personnel either deliver parcels directly to customers or store them in parcel lockers near the customer. The logistics company has a fleet of delivery vehicles, each departing from the central warehouse and returning after completing its mission. The question is how to arrange these delivery routes to ensure that each customer is either visited or that a parcel locker within their pickup area is visited by a delivery vehicle. The goal of the multi-vehicle coverage delivery problem is to optimize the total vehicle travel distance and customer satisfaction while satisfying vehicle capacity constraints. Therefore, the mathematical model for the multi-vehicle coverage delivery problem is established as follows: （2）； （3）； （4）； （5）； （6）； （7）； （8）； （9）； （10）； （11）； in, For customer satisfaction; Represents the function to be minimized; Indicates vehicle index; Indicates a collection of vehicles; Represents a set of customers; Indicates the customer index; This indicates a collection of parcel lockers; Indicates the parcel locker index; Represents a collection of central warehouses; Indicates the central repository index; Represents a set of nodes. Parcel lockers, central warehouses, and customers all serve as nodes; Indicates the node index; Indicates the node index; Represents an edge ( ) length, side ( ) is a slave node To the node The path; Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 if it is set to 1, otherwise the value is 0. This represents the fixed operating cost per vehicle. This represents the cost of punishment for maximum dissatisfaction. Indicates customer With parcel lockers Related decision variables, if customer Package lockers Service The value is 1 if it is set to 1, otherwise the value is 0. Represents the covering relation matrix, if r b Package storage lockers covering an area with a radius of [radius value]. but The value is 1 otherwise the value is 0. This represents the average distance traveled in a single daily walk. Indicates customer With vehicles Related decision variables, if customer In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( ) is a slave node To the node The path; Represents a node With vehicles Related decision variables, if nodes In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( (From the central warehouse) To the node The path; Indicates vehicle With edge ( Related decision variables, vehicles Passing through the edge ( )but The value is 1 otherwise the value is 0, edge ( ) is a slave node To the central warehouse The path; Indicates central warehouse With vehicles Related decision variables, such as the central warehouse In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Represents a node Demand; Represents a node With vehicles Related decision variables, if nodes In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates parcel locker With vehicles Related decision variables, such as package lockers In the vehicle On the path The value is 1 if it is set to 1, otherwise the value is 0. Indicates vehicle capacity; Indicates vehicle From node To the node The load capacity afterward; Indicates vehicle From node To the node The load capacity afterward; Indicates the number of nodes in the set; This represents the total number of vehicles in the vehicle set.

4. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 3, characterized in that, In step 2, during the process of generating individuals using the hybrid heuristic method, three initial solution generation strategies are employed. The first strategy is a random strategy, which randomly selects a node from the parcel locker PL and the customer set and inserts it into the current path until all customer nodes are served. The second strategy is a nearest neighbor insertion strategy, which starts from the central warehouse node and selects the node closest to the current node and satisfying the capacity constraint from the set of nodes to be visited and inserts it into the path until all customer nodes are served. The third strategy is a saving algorithm, which uses saving values ​​to guide the selection of insertion points until all customer nodes are served. Among the above three generation strategies, if the currently selected node is the parcel locker PL, then the coverage relation matrix is ​​used to add customer points that can be served by the parcel locker PL into the current path.

5. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 3, characterized in that, In step 3, a roulette wheel strategy is used to select two parent individuals for crossover. During crossover, the coverage relationship between the customer and the parcel locker PL is not considered initially. Two-point crossover and multi-point crossover are conducted based on the genetic information of the two parent individuals participating in the crossover, as detailed below: Two-point crossover: Two crossover sites are randomly selected from the two paired paternal chromosomes, and the genes in the two position intervals are exchanged. The paternal chromosome is the paternal individual. Multiple crossing over: Multiple crossing sites are randomly selected from the two paired parent chromosomes, and the genes at the crossing sites are exchanged.

6. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 5, characterized in that, In step 4, S1 represents the change in the variance of individual fitness in the population, and S2 represents the change in the fitness value of the optimal solution in the population. If the fitness of the current optimal individual in the population is better than that of the previous generation, then S2 is positive; otherwise, it is negative. Positive indicates improvement, and negative indicates no improvement. For S1, a monotonically decreasing variance function of individual fitness in the population is designed to guide reinforcement learning to recognize S1, and its specific form is as follows: （12）； in, For the first The variance of individual fitness in the population; For the first The variance of individual fitness in the population; For population generation index; The variance of the fitness of individuals in the initial population; The is the rate of decrease, with a value range of [0.8, 1); mod is the modulo function, used to calculate the modulo value every . envoy attenuation, It is a constant; The first Actual variance of individual fitness in a generation population and In comparison, if If S1 is positive, then S1 is positive; otherwise, it is negative.

7. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 6, characterized in that, In step 4, mutation operators are used for local search in the evolutionary algorithm. The following nine mutation operators are designed: 2-Swap: Select a single path and swap the two access nodes; Fragment Reverse: Select a single path and reverse the sub-fragments within that path; 3-Swap: Select a single path and choose three locations to reassemble the node fragment; Reinsertion: Select a single path and move the node to a new position within the same path; Path-to-Path Swap: Swap two nodes between different paths; Inter-path re-insertion: Select a node at any position in one path and insert it into another path; Service mode switching: Select any customer node and delete it, then insert the PL node corresponding to this customer node into the original position. The PL node is the package locker node. PL Transformation: Select any PL node and replace it with another unvisited PL node; Destroy-Repair: Remove any item from a single path After each node, it is reinserted into the optimal position.

8. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 7, characterized in that, In step 4, the process of selecting the crossover operator using the guided reinforcement learning mechanism is as follows: Nine mutation operators are used as the actions that the agent can take in each state. The action selection probability is calculated based on the current state and the evaluation value corresponding to the action, and the corresponding action is executed according to the action selection probability. The evaluation value is... value, The value is updated using the following formula: （13）； in, It is the first State-action pairs during population evolution value; It is the first State-action pairs during population evolution value; For population generation index; For the agent in the first The current state of the population; For the agent in the first The actions that the next generation of the species intends to take; For the learning rate, control the pair The magnitude of the value update; For the first The reward value of the next generation; This is a discount factor, representing the degree of importance attached to future rewards; Indicates that the agent is in the first... The maximum of all possible actions in the current state of the population value, For action, It is the set of all actions.

9. The method for solving the multi-vehicle coverage delivery problem using the state-guided reinforcement learning-driven evolutionary algorithm according to claim 8, characterized in that, In step 5, the reward value is calculated as follows: （14）； （15）； in, For the first During population evolution, agents respond to actions The reward value; For action In the The improvement effect of generational populations; For action In the The improvement effect of generational populations; For the first Agents choosing actions in population evolution The number of times; For the first Agents choosing actions in population evolution The number of times improvements were made.

Images