A vehicle path optimization method based on a policy gradient super heuristic algorithm

Abstract

A vehicle path optimization method based on a super heuristic algorithm of a policy gradient comprises a research background significance and technology of a vehicle path problem, and comprises the following steps: step 1, analyzing a capacitated vehicle path problem; step 2, initializing parameters; step 3, initializing a population; step 4, selecting an operator to operate on the population through an operator selection mechanism, and calculating and storing related information in an experience pool; step 5, judging the number of iterations of the algorithm, if the preset step distance is reached, then entering step 6, learning, otherwise, entering step 4; step 6, selecting state transition data from the experience pool to train a neural network in a policy gradient algorithm; step 7, judging whether a termination running condition of the algorithm is reached, if yes, outputting an optimal solution, otherwise, returning to a main loop. The application provides a vehicle path optimization method of a super heuristic algorithm of a policy gradient with a high-level selection strategy.

Description

Technical Field

[0001] This invention relates to the fields of logistics distribution, e-commerce, intelligent optimization, and computer applications, and in particular to a vehicle route optimization method. Background Technology

[0002] Vehicle routing problems, as an important abstraction of problems in logistics and transportation, play a positive role in solving some thorny issues in the modern logistics industry. They are significant for promoting the rapid development of the logistics industry, enhancing the core competitiveness of logistics enterprises, and bringing substantial economic benefits to both enterprises and the nation. Vehicle routing problems have many derivative problems, such as vehicle routing with time windows, vehicle routing with simultaneous pickup and delivery, multi-warehouse vehicle routing, and open-ended vehicle routing. These constraints make vehicle routing problems more complex and difficult to solve. Many methods have been proposed for solving vehicle routing problems, mainly falling into two categories: exact algorithms and heuristic algorithms. Currently, there are many types of exact algorithms, including branch and bound, branch and pricing, and dynamic programming algorithms. However, exact algorithms have the disadvantage of requiring a large amount of computation when dealing with large-scale problems. Therefore, heuristic algorithms are more frequently used to solve large-scale vehicle routing problems. Heuristic algorithms are one of the best solutions for large-scale vehicle routing problems. These algorithms are broadly classified into three categories: constructive heuristic algorithms, improved heuristic algorithms, and metaheuristic algorithms. Constructive heuristics are algorithms that construct feasible solutions to instances of the optimization problem based on basic greedy ideas or other specific strategies. The quality of solutions obtained by improved heuristics largely depends on the initial solution, making them prone to getting trapped in local optima. Metaheuristic strategies are typically general heuristics that do not rely on the specific conditions of a particular problem, thus allowing them to be applied to a wider range of problems. These algorithms are increasingly being used to solve large-scale combinatorial optimization problems, including classic genetic algorithms, differential evolution, ant colony optimization, bee colony optimization, and particle swarm optimization. However, designing heuristic algorithms often requires designers to possess both a foundation in computer algorithm design and expertise in the domain of the problem being solved, resulting in a lack of versatility. Hyperheuristic algorithms, due to their strong versatility and efficiency, are widely used to solve combinatorial optimization problems. Reinforcement learning, as a powerful decision-making tool, has attracted widespread attention. Many metaheuristic algorithms have been applied to the vehicle routing problem, but developing more effective algorithms to obtain better solutions remains under investigation. Summary of the Invention

[0003] This invention aims to overcome the aforementioned shortcomings of existing technologies and provides a vehicle path optimization method based on a hyperheuristic algorithm using the policy gradient algorithm in reinforcement learning, so as to fully utilize the advantages of various metaheuristic algorithms.

[0004] The technical solution adopted by this invention to solve its technical problem is:

[0005] A vehicle path optimization method based on a policy gradient hyperheuristic algorithm includes the following steps:

[0006] Step 1 analyzes the vehicle routing problem using Augerat's instances dataset. The elements of the cost matrix for the vehicle routing problem are Euclidean distances.

[0007] Assume the distribution center is i=0, the customer points are L (i=1,2,3,…,L), the maximum number of vehicles is K (k=1,2,3,…K), each vehicle has the same carrying capacity q, and the demand at each customer point is d. i (i = 1, 2, 3, ..., L), let c be the distance from customer i to customer j. ij The goal of optimization is to minimize the travel distance, and a complete solution contains the set of all paths.

[0008] Step 2: Input the various parameters required by the system. Set the structure of the Actor and Critic neural networks and initialize the weight parameters.

[0009] Step 3: Initialize and generate an initial population Pop(p) of size Npop using the K-means clustering algorithm. i =p1,p2,p3,...,p np The fitness value f(f) of each individual in the population is calculated from the initial population according to the objective function. i =f1,f2,f3,...,f np Randomly select an individual p from the population. i and its fitness value f i Initialize P B =p i F B =f i , where P B F represents the individual with the optimal solution. B Let N represent the optimal fitness value and the number of LLHs be N. A Action takes values ​​of (1, 2, 3, ..., N) A (Integer, action selection probability P = 0.8, maximum training steps D) max= 800, the state value State = {f, diff}, where f is the degree of improvement of the optimal individual in the current population compared to the optimal individual in the previous generation, and diff is the difference degree between two generations after the population is updated by Action. Calculate State through equations (1) and (2):

[0010]

[0011]

[0012] Step 4: Randomly generate a number between [0, 1]. If rand ≤ P, then randomly select one from N A underlying operators as the action a for processing the population; otherwise, if rand > P, then input State into the Actor network to output an action a. Use the action obtained above to process the individuals in the population Pop to generate new individuals Ind and fitness values fit. Judge. If fit < fit' (the fitness value of the individuals in the previous generation), it means that the fitness value of the solution is better at this time, so save the solution and the fitness value of the solution; if fit ≥ fit', then use the adaptive acceptance criterion to judge. If the probability p > rand, then also retain the good solution and update the state at the same time. Otherwise, discard the solution. At this time, State t = State, fit' = fit'. Select the two individuals with the top two fitness values in the obtained new population, and randomly select another individual. Calculate the immediate return value Reward of the three individuals according to the fitness values of the selected three individuals. The state State at this time is the "next state". Record the information of the above three into the Experience Pool [s, a, r, s t . Among them, s is the state State of the individual Ind before processing with the action a, a is the action a taken, r represents the immediate return value Reward obtained in the calculation, which is specifically calculated by equation (3), where is the mean of the fitness values of the individuals in the current population, is the mean of the fitness values of the individuals in the previous population.

[0013] s t represents the state State of the individual Ind after processing with the action a. If the state transition data in the experience pool reaches the upper limit, then update the experience pool according to the principle of first in first out. Let State = State t (the state value of the previous population), fit' = fit, G (iteration number) = G + 1.

[0014]

[0015] Step 5: Check if mod(G, Dmax) equals 0. If not, check if the solution has not been improved for a certain number of algebras. If so, randomly select a mutation operator as action a and use action a to process the population; otherwise, proceed to step 4. If it equals 0, extract N state transition data [s, a, r, s] from the Experience Pool. t The Actor network is based on s t Provide a to the Critic network t Meanwhile, the Critic network is based on (s, a) and (s t ,a t We obtain Q(s, a) and Q(s) t a t The TD bias is calculated, and the Critic network updates its weight parameters according to the corresponding formula based on the TD bias. The Actor network updates its weight parameters based on the Q(s) provided by the Critic network. t ,a t The weight parameters are updated according to the corresponding formulas. The updates of the weight parameters of the Actor and Critic neural networks are mainly based on formulas (4) and (5):

[0016] ω t+1 =ω t +α[r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t (4)

[0017]

[0018] In the formula, ω and θ are the weight parameters of the Critic network and the Actor network, respectively; α is the learning rate of the Critic network and β is the learning rate of the Actor network; the subscript t represents the corresponding value of the previous generation, and t+1 represents the updated value of this generation.

[0019] Step 6: Determine if G is greater than the maximum number of iterations Gmax or if the solution has not improved within the set number of algebras. If neither of the above conditions is met, determine if the solution has not improved for a certain number of consecutive algebras. If so, randomly select a mutation operator as action a and use action a to process the population. Otherwise, proceed to step 4. Otherwise, output the optimal solution and complete the task.

[0020] Furthermore, in step 2, the process of generating the initial population is as follows:

[0021] 2.1) Starting from the distribution center, for the delivery route of the k-th vehicle, randomly assign customer points to the route and check the vehicle's load constraints. If the load does not exceed the standard load, continue to randomly assign customer points; otherwise, generate the (k+1)-th route. Repeat the loop until all customer points are assigned to the corresponding delivery routes, then an initial population of individuals is generated.

[0022] 2.2) Repeat the above operation multiple times to generate a population of a set number of individuals. Select the individual with the shortest delivery path in the population, i.e., the number of delivery vehicles in the delivery path is k, and set k as the number of cluster blocks.

[0023] 2.3) Take the customer point closest to the distribution center among the customer points of each vehicle's delivery as the benchmark, and divide the remaining customer points into k clusters according to their distance from the benchmark customer point;

[0024] 2.4) Randomly arrange k cluster blocks. According to the vehicle load capacity constraint, randomly select customer points from the corresponding cluster blocks and assign them to the delivery route according to the order of the cluster blocks. If the customer points in the k blocks cannot meet the vehicle load capacity of a certain delivery route, then randomly select customer points from the k+1 cluster blocks until the vehicle load capacity constraint is met. Otherwise, select a new delivery vehicle. When all customer points are assigned to the corresponding delivery routes, an initial solution is generated.

[0025] The beneficial effects of this invention are mainly reflected in the following aspects: When studying the high-capacity vehicle routing problem, it achieves good accuracy and stability in solving standard examples setA, setE, and setP. During the optimization process, the high-level strategy of the algorithm can analyze and learn the historical performance of each low-level operator in the search process, enabling the algorithm to select competitive operators to execute the current search during iteration, which helps control the convergence speed of the algorithm. The application of the solution acceptance criterion helps the algorithm escape local optima and optimizes the search process. Furthermore, for different problems, designing new low-level operators allows the high-level strategy of the algorithm to be applied to new problem domains. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0027] The present invention will now be further described with reference to the accompanying drawings.

[0028] Reference Figure 1 A vehicle path optimization method based on a policy gradient hyperheuristic algorithm includes the following steps:

[0029] Step 1 Vehicle Routing Problem Analysis: Using the Augerat’s instances dataset, the elements of the cost matrix of the vehicle routing problem are Euclidean distances;

[0030] The capacitated vehicle routing problem can be described in the following form: The graph G=(V, E) represents an undirected graph, where V={0, 1, …, n} represents a set of n + 1 vertices, and E represents the set of arcs. The vertex 0 and Vc = V\{0} represent the depot and the customer points to be served respectively. There is a corresponding weight c on each arc (i, j) ∈ E ={(i, j): i, j ∈ V, i < j}. ij c ij is symmetric, that is each customer point i ∈ Vc has a non - negative demand q i , and the demand of the depot is set to 0. The customer demands must be satisfied in subsequent delivery services. To meet the customer demands, the fleet consists of m identical vehicles, each with a maximum load capacity of Q. The demand of each customer is less than the load capacity of the vehicle, that is, q i ≤ Q, i ∈ Vc. The capacitated vehicle routing problem aims to solve a set of customer points Vc waiting for delivery services, and each customer point in Vc is served by exactly one delivery vehicle. The vehicles in the fleet start from the distribution center, and after a series of services, finally return to the distribution center.

[0031] Table 1

[0032]

[0033] Table 2

[0034]

[0035] The parameters used in the specific experiment are shown in Table 1 and Table 2.

[0036] Step 2 Initialization: In the 0 - th generation G, an initial population Pop(p i = p1, p2, p3, ..., p np ) with a population size of Npop is initialized by the K - means clustering algorithm. According to the objective function, the fitness value f(f i = f1, f2, f3, ..., f np ) of each individual in the population is calculated. Randomly select an individual p i and its fitness value f i , initialize P B = p i , F B = f i , where P B represents the optimal solution individual, FB Denote the optimal fitness value, and let the number of LLHs be N A , where Action takes values as integers in (1, 2, 3, …, N A ), and the state value State = {f, diff}.

[0037] Steps for generating the initial population group:

[0038] 2.1) Starting from the distribution center, for the delivery route of the k-th vehicle, randomly allocate customer points to this route. Judge the load constraint of the vehicle. If the standard load capacity is not exceeded, continue to randomly allocate customer points; otherwise, generate the (k + 1)-th route; repeat the loop. When all customer points are allocated to the corresponding delivery routes, an initial population individual is generated;

[0039] 2.2) Perform the above operations multiple times to generate a population of a set number of individuals. Select the individual with the shortest delivery route in the population, that is, the number of delivery vehicles in this delivery route is k, and set k as the number of clustering blocks;

[0040] 2.3) Take the customer point closest to the distribution center among the customer points delivered by each vehicle as the benchmark, and divide the remaining customer points into k clustering blocks according to the distance from the benchmark customer point;

[0041] 2.4) Randomly arrange the k clustering blocks. According to the vehicle load capacity constraint, randomly select customer points from the corresponding clustering blocks in the order of the clustering block arrangement and allocate them to the delivery route. If the customer points in the k blocks cannot meet the vehicle load of a certain delivery route, randomly select customer points from the (k + 1)-th clustering block until the vehicle load constraint is reached; otherwise, select a new delivery vehicle. When all customer points are allocated to the corresponding delivery routes, an initial solution individual is generated.

[0042] Step 4 Randomly generate a number between [0, 1]. If rand ≤ P, then randomly select one from N A underlying operators as the action a for processing the population; otherwise, if rand > P, input State into the Actor network to output an action a. Use the action obtained above to process the individuals in the population Pop to generate a new individual Ind and fitness value fit. Judge. If fit < fit' (the fitness value of the previous generation individual), it means that the fitness value of the solution is better at this time, so save the solution and the fitness value of the solution; if fit ≥ fit', then use the adaptive acceptance criterion to judge. If the probability p > rand, also retain the good solution and update the state; otherwise, discard the solution. At this time, State t=State, fit' = fit'. From the resulting new population, select the two individuals with the highest fitness values, and randomly select another individual. Calculate the immediate reward value for the three selected individuals based on their fitness values. The current state is the "next state". Record the information of the above three individuals in the Experience Pool [s, a, r, s]. t In the context of the algorithm, s represents the state of individual Ind before action a is taken, a represents the action a, and r represents the immediate reward value obtained in the calculation. t This represents the state of individual Ind after processing with action 'a'. If the experience pool reaches its limit for state transition data, it is updated using a first-in, first-out (FIFO) principle. Let State = State t (State value of the previous generation population), fit' = fit, G (number of iterations) = G+1. Action a has three main categories of operators: local optimization operators, mutation operators, and destruction and reconstruction operators. Local optimization operators can efficiently improve the quality of the current solution, enabling the algorithm to converge quickly. However, when the local optimization operator improves the solution to a certain extent, it is easy to get trapped in a local optimum and it is not easy to escape from the local optimum, affecting the convergence accuracy. Mutation operators can often generate some small perturbations to the solution during the search process, increasing the randomness and diversity of the search process to prevent getting trapped in a local optimum, but they cannot guarantee a high-quality solution. The destruction and reconstruction operators function similarly to the mutation operators. The specific types and functions of the operators are shown in Table 3:

[0043] Table 3

[0044]

[0045] Step 5: Check if mod(G, Dmax) equals 0. If not, check if the solution has not been improved for a certain number of algebras. If so, randomly select a mutation operator as action a and use action a to process the population; otherwise, proceed to step 4. If it equals 0, extract N state transition data [s, a, r, s] from the Experience Pool. t The Actor network is based on s t Provide a to the Critic network t Meanwhile, the Critic network is based on (s, a) and (s t ,a t We obtain Q(s, a) and Q(s) t a t The TD bias is calculated, and the Critic network updates its weight parameters according to the corresponding formula based on the TD bias. The Actor network updates its weight parameters based on the Q(s) provided by the Critic network.t ,a t The weight parameters are updated according to the corresponding formula.

[0046] Step 6: Determine if G is greater than the maximum number of iterations Gmax or if the solution has not improved within the set number of algebras. If neither of the above conditions is met, determine if the solution has not improved for a certain number of consecutive algebras. If so, randomly select a mutation operator as action a and use action a to process the population. Otherwise, proceed to step 4. Otherwise, output the optimal solution and complete the task.

[0047] This invention demonstrates strong search capability and stability in the vehicle routing problem with capacity. Besides solving the vehicle routing problem, it can also be applied to other combinatorial optimization problems by modifying the design of the underlying operators. It should be noted that the above embodiments are only for illustrating the technical solutions of this invention and are not intended to limit it. Those skilled in the art should understand the technical features of this invention and may modify or equivalently replace some of the technical features described above, but the essence of the technical solutions should not deviate from the spirit and scope of the embodiments of this invention.

Claims

1. A vehicle path optimization method based on a policy gradient hyperheuristic algorithm, characterized in that, The method includes the following steps: Step 1: Analyze the vehicle routing problem using Augerat's instances dataset. The elements of the cost matrix for the vehicle routing problem are Euclidean distances. Assume the distribution center is p=0, the customer point is i, the total number of customer points is L, i=1,2,3,…,L, the number of vehicles is k, the maximum number of vehicles is K, k=1,2,3,…K, each vehicle has the same carrying capacity q, and the demand at each customer point is d. i Let i = 1, 2, 3, ..., L, and let c be the distance between customer i and customer j. ij The goal of optimization is to minimize the travel distance, and a complete solution contains the set of all paths. Step 2: Input the various parameters required by the system; set the structure of the Actor and Critic neural networks and initialize the weight parameters; Step 3: Initialize and generate an initial population Pop of size np using the K-means clustering algorithm, with p individuals in the population. i p i= p1, p2, p3, ..., p np The fitness value f of each individual in the population is calculated from the initial population according to the objective function. i f i= f1, f2, f3, ..., f np Randomly select an individual p from the population i and its fitness value f i Initialize P B =p i F B =f i , where P B F represents the individual with the optimal solution. B Let N represent the optimal fitness value and the number of LLHs be N. A Action can take values ​​of (1, 2, 3, …, N). A The probability of action selection is P=0.8, and the maximum number of training steps is D. max =800, State = {f, diff}, where f refers to the degree of improvement of the best individual in the current generation compared to the best individual in the previous generation, and diff refers to the degree of difference between the two generations after the Action update; State is calculated using equations (1) and (2): Step 4: Randomly generate a number between [0, 1]. If rand ≤ P, randomly select one from N A underlying operators as the action a for processing the population; otherwise, if rand > P, input State into the Actor network to output an action a; use the obtained action to process the individuals in the population Pop to generate new individuals Ind and fitness value fit. Judge that if fit < fit’, where fit’ refers to the fitness value of the individuals in the previous generation, it means that the fitness value of the solution is better at this time, so save the solution and the fitness value of the solution; if fit ≥ fit’, use the adaptive acceptance criterion to judge. If the probability p > rand, also retain the good solution and update the state. Otherwise, discard the solution. At this time, State t = State, fit’ = fit’; Select the two individuals with the top fitness values in the obtained new population, and randomly select another individual. Calculate the immediate return value Reward of the three individuals according to the fitness values of the selected three individuals. The state State at this time is the "next state", and record the information of the above three individuals in Experience Pool [s, a, r, s t , where s is the state State of the individual Ind before processing with the action a, a is the action a taken, r represents the immediate return value Reward obtained in the above calculation, which is specifically calculated by Equation (3), where is the mean of the fitness values of the current population individuals, is the mean of the fitness values of the individuals in the previous generation population; s t Let State represent the state of individual Ind after processing with action a; if the state transition data in the experience pool reaches the upper limit, the experience pool is updated according to the first-in, first-out principle; let State = State t State t The state value of the previous generation of the population is referred to as fit'=fit, G=G+1, where G refers to the iteration generation; Step 5: Check if mod(G, Dmax) equals 0. If not, check if the solution has not been improved for a certain number of algebras. If so, randomly select a mutation operator as action a and use action a to process the population; otherwise, proceed to step 4. If it equals 0, extract N state transition data [s, a, r, s] from the Experience Pool. t The Actor network is based on s t Provide a to the Critic network t Meanwhile, the Critic network is based on (s, a) and (s t , a t We obtain Q(s, a) and Q(s) t a t The TD bias is calculated, and the Critic network updates its weight parameters according to the corresponding formula based on the TD bias. The Actor network updates its weight parameters based on the Q(s) provided by the Critic network. t , a t The weight parameters are updated according to the corresponding formulas. The updates of the weight parameters of the Actor and Critic neural networks are mainly based on formulas (4) and (5): In the formula, ω and θ are the weight parameters of the Critic network and the Actor network, respectively; α is the learning rate of the Critic network and β is the learning rate of the Actor network; the subscript t represents the corresponding value of the previous generation, and t+1 represents the updated value of this generation. Step 6: Determine if G is greater than the maximum number of iterations Gmax or if the solution has not improved within the set number of algebras. If neither of the above conditions is met, determine if the solution has not improved for a certain number of consecutive algebras. If so, randomly select a mutation operator as action a and use action a to process the population. Otherwise, proceed to step 4. Otherwise, output the optimal solution and complete the task.

2. The vehicle path optimization method based on a policy gradient hyperheuristic algorithm as described in claim 1, characterized in that, In step 2, the process of generating the initial population is as follows: 2.1) Starting from the distribution center, for the delivery route of the k-th vehicle, randomly assign customer points to the route and check the vehicle's load constraints. If the load does not exceed the standard load capacity, continue to randomly assign customer points; otherwise, generate the (k+1)-th route. Repeat the loop until all customer points are assigned to the corresponding delivery routes, then an initial population of individuals is generated. 2.2) Repeat the above operations multiple times to generate a population of a set number of individuals. Select the individual with the shortest delivery path in the population, i.e., the number of delivery vehicles in this delivery path is k, and set k as the number of cluster blocks. 2.3) Take the customer point closest to the distribution center among the customer points of each vehicle's delivery as the benchmark, and divide the remaining customer points into k clusters according to their distance from the benchmark customer point; 2.4) Randomly arrange k cluster blocks. According to the vehicle load capacity constraint, randomly select customer points from the corresponding cluster blocks and assign them to the delivery route according to the order of the cluster blocks. If the customer points in the k blocks cannot meet the vehicle load capacity of a certain delivery route, then randomly select customer points from the k+1 cluster blocks until the vehicle load capacity constraint is met. Otherwise, select a new delivery vehicle. When all customer points are assigned to the corresponding delivery routes, an initial solution is generated.

Images