Super-multi-target vehicle path planning method for deep reinforcement learning assisted evolution

Abstract

The invention discloses a super-multi-target vehicle path planning method based on deep reinforcement learning assisted evolution, and aims to dynamically guide a search direction through deep reinforcement learning for multi-target optimization and path planning problems. The core of the method comprises the following steps: 1) constructing a super-multi-target vehicle path problem mathematical model; 2) designing a deep Q network as a decision model, and adaptively selecting a high-potential search direction in iteration; 3) designing a global search strategy, and balancing global search and local optimization; 4) generating a filial generation and evaluating the fitness, and executing environment selection to update the population; and 5) calculating a reward value according to a population evolution effect, and optimizing DQN network parameters. According to the method, through a collaborative mechanism of reinforcement learning and an evolutionary algorithm, a search strategy is dynamically adjusted, the convergence speed and stability of the algorithm are effectively improved, the adaptability to a complex path planning problem is enhanced, and the method has remarkable advantages in solving efficiency, solution set quality and dynamic environment response.

Description

Technical Field

[0001] The present invention relates to the technical field of super-multi-objective optimization and vehicle path planning, and specifically to a super-multi-objective vehicle path planning method assisted by deep reinforcement learning evolution. Background Art

[0002] Since its inception in the mid-20th century, the vehicle routing problem (VRP) has gradually evolved from single-objective optimization (such as minimizing travel distance) to multi-objective collaborative optimization to meet the diverse needs of complex logistics scenarios. With the rapid development of e-commerce, on-demand delivery, and green logistics, traditional multi-objective VRP (number of objectives ≤ 3) has become inadequate. Consequently, the Many-Objective VRP (MaOVRP, number of objectives ≥ 4) has become a research focus. Its core objectives include minimizing the number of vehicles, total travel distance, time window penalties, carbon emissions, and maximizing loading and vehicle reuse rates. For example, in emergency supply distribution, multiple objectives such as transport timeliness, cost controllability, route safety, and resource utilization must be simultaneously optimized. According to a report by the International Logistics Association, over 70% of global logistics companies face the challenge of routing with conflicting objectives. Multi-objective optimization can reduce overall costs by 15% to 30%, highlighting its strategic value. In recent years, with the increasing complexity of objectives and constraints, research on MaOVRP has gradually shifted from theoretical exploration to industrial application. However, the high-dimensionality of the target space leads to an explosion in the Pareto frontier and a sparse distribution of solutions. Traditional methods struggle to balance solution quality and computational efficiency. Therefore, efficiently solving multi-target VRPs has become a key technical bottleneck in improving the intelligence of logistics systems.

[0003] In solving multi-objective vehicle routing problems, multi-objective evolutionary algorithms (MELs) have been widely studied and applied due to their capabilities for multi-objective collaborative optimization, handling high-dimensional objective spaces, flexible and scalable frameworks, and balancing solution diversity and convergence. By simulating biological evolutionary processes and formulating strategies for population initialization, fitness assessment, crossover and mutation, environmental selection, and population update, they search for high-quality solutions in the solution space while balancing the trade-offs between multiple conflicting objectives. However, when applied to solving MELs, existing MELs still suffer from drawbacks such as slow convergence and a tendency to fall into local optima. Furthermore, their problem-solving capabilities and adaptability to dynamic environments are severely limited in various practical scenarios. Among all factors influencing overall algorithm performance, the selection of the evolutionary search direction, a key step in all types of evolutionary algorithms, directly determines the evolutionary direction of the population and the quality of the solution set, thus influencing both population diversity and convergence.

[0004] The currently mainstream multi-objective evolutionary algorithms used to solve MaOVRP suffer from problems such as imbalanced selection pressure, ineffective diversity maintenance, and high computational complexity in the selection of evolutionary search directions, severely impacting solution quality and algorithmic efficiency. Furthermore, when solving real-world vehicle routing problems, different application scenarios present varying customer needs, traffic conditions, and constraints. Even carefully designed mainstream evolutionary search direction selection strategies may not be applicable to all instances. Consequently, when solving different problem instances, even when the problem instance undergoes subtle changes, the designed algorithm may not be able to effectively solve the new instance and may become unsuitable, significantly impacting the algorithm's problem-solving and dynamic adaptability. Deep reinforcement learning has demonstrated excellent performance in solving decision-making problems in various complex and stochastic environments, particularly when combined with deep neural networks. This integration significantly enhances the method's reasoning and learning capabilities for handling uncertain continuous decision-making problems. Therefore, leveraging the interactive decision-making capabilities of deep reinforcement learning, it is possible to effectively select optimal evolutionary search directions based on the characteristics of different practical application scenarios.

[0005] Based on this, this patent proposes a deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method. By designing the corresponding credit allocation strategy and deep reinforcement learning adaptive evolutionary search direction selection method, it can effectively guide large-scale decision space search, reasonably allocate computing resources, and improve the algorithm's search efficiency and problem-solving capabilities, so as to better solve various different problem instances and meet the dynamic needs of the problem. Summary of the Invention

[0006] To effectively address the problems of low search efficiency, poor dynamic adaptability, and poor problem-solving capabilities of conventional multi-objective evolutionary algorithms in the practical application of MaOVRP, this paper provides a deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method, targeting the key influencing factor of excellent evolutionary search direction selection. The method is suitable for a variety of scenarios such as large-scale logistics distribution and emergency material dispatch, and includes the following technical solutions:

[0007] Step 1: Construct a mathematical model for the multi-objective vehicle routing problem and set the corresponding constraints and three or more optimization objectives;

[0008] Step 2: Based on the input population size N, the number of targets M, the number of weight vectors V, and the maximum number of iterations MaxGen, initialize the uniformly distributed weight vector set W, the population P representing the solution to the multi-target vehicle routing problem, and the ideal point Z, and associate the weight vectors with the individuals B in the divided population;

[0009] Step 3: Initialize the deep reinforcement learning network DQN, set the appropriate environment, state s, action a, and reward function Reward, and initialize the experience replay queue and reward queue required by DQN later;

[0010] Step 4: Through deep reinforcement learning to assist the adaptive selection strategy of evolutionary search direction, DQN adaptively selects the weighted association set corresponding to the evolutionary search direction, and uses all the associated excellent individuals as part of the parent solution set Parent_DQN in the mating pool;

[0011] Step 5: Based on the fitness value of the individual, the roulette wheel method is used to select another parent solution set Parent_FE, and the two parent solution sets are merged into MatingPool;

[0012] Step 6: Perform crossover and mutation operations on the parent solutions in the mating pool to generate the offspring solution set Offspring and update the fitness value of the offspring solution;

[0013] Step 7: Environment Selection Updates the Population and Trains the DQN. First, based on the credit allocation strategy, the Tchebycheff aggregation function is used to evaluate the solution's fitness. The environment selects and updates the population P and ideal point Z, updates the reward queue R, and calculates the reward. Next, data from the experience replay queue T is sampled, the loss function L is calculated, and gradient descent is used to update the DQN network parameters, optimizing the adaptive selection strategy for the evolutionary search direction.

[0014] Step 8: Use MaxGen to control the algorithm running time, repeat steps 4 to 7 until the termination condition gen>MaxGen is met, the algorithm ends and outputs the final solution set.

[0015] Preferably, in step 1, the optimization objectives and constraints of the multi-objective vehicle path planning problem must satisfy the following mathematical definitions:

[0016] minmize F(x)=[f1(x),f2(x),…,f M (x)]

[0017]

[0018] Where M represents the number of targets and M>3, K represents the number of constraints, x=(x1,x2,…,x D )∈Ω is a D-dimensional decision vector, x k (k∈{1,…,D}) represents the decision variable. Represents the search space of decision variables, F(x) is composed of M sub-objective functions f i(x), i∈{1,…,M}determined by the target vector, G(x) is composed of K constraints g j (x), a constraint vector determined by j∈{1,…,K}, where the objective functions conflict with each other.

[0019] Preferably, in step 2, the input variables can be selected to accommodate more diverse problem instances. Simultaneously, based on the input data, a set W comprising V uniformly distributed weight vectors, a population P comprising N individuals, and an ideal point Z are initialized. The population is then divided into regions based on the weight vectors that guide the population's evolutionary direction, forming a set B.

[0020] Preferably, in step 3, the initialization process of DQN is as follows:

[0021] First, define the state s and action a: the decision variables of the solution and their weight vectors constitute the state of DQN, that is, s=(x1,…,x D ,ω1,…,ω M ); The weight vector association set corresponding to the selected evolutionary search direction is used as the action of DQN, that is, a=B i .

[0022] Then, define the reward function Reward. The specific calculation formula is as follows:

[0023] (1) The fitness value of the solution is calculated using the Tchebycheff aggregation function:

[0024]

[0025] where ω=(ω1,…,ω M ) is the weight vector corresponding to the solution x, is an ideal point consisting of the minimum target value in the population;

[0026] (2) Based on the fitness evaluation function, after the evolutionary operation generates a set of offspring solutions, the average fitness improvement of the evolutionary search direction Y replaced by the offspring solution x can be calculated as follows:

[0027] FI(x, y) = g tch (x,ω y ,z * )-g tch (y,ω y ,z * )

[0028] AFI x =∑ y∈Y FI(x,y) / Size(Y)

[0029] where ω yrepresents the weight vector of y, where Y is the set of solutions replaced by x;

[0030] (3) Based on the average fitness improvement formula, the weighted association set B corresponding to the evolutionary search direction is calculated in the following way i Reward value reward:

[0031]

[0032] In order to further improve the search efficiency of the algorithm, the algorithm defines a record containing (B i ,AFI x ) of the first-in-first-out queue R, which saves the AFI of the evolutionary direction update solution within a given time window, where AFI x is the weighted association set B corresponding to the evolutionary search direction i The fitness improvement of the generated offspring solution x, that is, the weighted association set B corresponding to the evolutionary search direction i The reward can be defined as the average value of AFI saved in queue R.

[0033] Finally, initialize the experience replay queue T and reward queue R: In order to provide the data required for DQN training, the algorithm defines an experience replay queue T. Once the child solution is generated and the reward is calculated, the tuple Saved in the experience replay queue T, where {p,ω p} is the current solution p and its weight vector ω p , B i is the weight vector of the selected evolutionary direction, {x,ω x} is the generated sub-solution x and its weight vector ω x .

[0034] Preferably, in step 4, a deep reinforcement learning-assisted adaptive evolutionary search direction selection method is used to adaptively select a portion of the parent solution set Parent_DQN in the mating pool MatingPool using DQN. The method selection process is as follows:

[0035] First, according to the input state {p,ω p}Calculate the Q value associated with the weight vector corresponding to each evolutionary direction;

[0036] Then, based on the calculated Q value, the weighted association set corresponding to the appropriate evolutionary search direction is selected through the roulette wheel method. If the weighted association set corresponding to an evolutionary search direction does not exist in the experience replay pool T, it will no longer be selected; in this case, the weighted association set will be forced to be selected instead of being selected through the roulette wheel method;

[0037] Finally, the weight association set B corresponding to the selected evolutionary search direction i All associated excellent individuals are used as part of the parent solution set Parent_DQN in the mating pool MatingPool.

[0038] Preferably, in step 5, the population is sorted based on the individual fitness value, and a roulette wheel method is used to select a parent solution set Parent_FE with the same size as the Parent_DQN set from the remaining individuals in the population P excluding Parent_DQN.

[0039] Preferably, in step 7, the fitness value of the solution is updated using the credit allocation strategy, the environment selects the population P and the ideal point Z based on the dominance relationship between individuals, and calculates the reward value reward to update the reward queue R. The method selection process is as follows:

[0040] First, the fitness value of the solution is calculated, and based on the dominance relationship between individuals, the population P and the ideal point Z are updated;

[0041] Then, based on the FIFO queue entry rule, (B i ,AFI x ) is added to the reward queue R. And according to the updated population, the average fitness improvement value AFI of the updated offspring solution to the evolutionary search direction is calculated x And the weighted association set B corresponding to the selected evolutionary search direction i The reward value reward.

[0042] Preferably, in step 7, the data in the experience replay queue T is sampled, the loss function value L is calculated, and the DQN network parameters are updated using the gradient descent method to optimize the adaptive selection strategy of the evolutionary search direction. The training process is as follows:

[0043] First, batch experience is sampled from the experience replay pool T to form a mini-batch dataset;

[0044] Then, using the formula Calculate the target Q value and get the predicted Q value q of DQN eval =Q(s t ,a t ; Q), jointly calculate the loss function value

[0045] Finally, the gradient descent method of the loss is used to backpropagate and update the parameters in the original network to obtain the updated network DQN.

[0046] Preferably, in step 8, the above steps 4 to 8 are repeated until gen>MaxGen, and the algorithm ends and outputs the final vehicle path solution set.

[0047] The deep reinforcement learning-assisted evolutionary search direction adaptive selection strategy of step 4 specifically includes the following steps:

[0048] Step 1: Input the current population P, weight vector set W, weight association set B corresponding to the evolutionary search direction, experience replay queue T, and trained network Q;

[0049] Step 2: According to the current state {p,ω p}Calculate the Q value associated with the weight vector corresponding to each evolutionary direction;

[0050] Step 3: Based on the calculated Q value, select the weighted association set corresponding to the appropriate evolutionary search direction through the roulette wheel method. If the weighted association set corresponding to an evolutionary search direction does not exist in the experience replay pool T, it will no longer be selected; in this case, the weighted association set will be forced to be selected instead of being selected through the roulette wheel method;

[0051] Step 4: Associate the weights corresponding to the selected evolutionary search direction with the set B i All associated excellent individuals are used as part of the parent solution set Parent_DQN in the mating pool MatingPool;

[0052] Step 5: Output the selected excellent parent subset Parent_DQN.

[0053] The method for applying step 7 to the environment selection, population update, and DQN training specifically includes the following steps:

[0054] Step 1: Input the current population size P, weight vector set W, weight association set B corresponding to the evolutionary search direction, generated offspring solution set Offspring, selected evolutionary search direction MatingPool, reward queue R, experience replay queue T, and current network Q;

[0055] Step 2: According to the formula Calculate the fitness value of the solution and update the population P based on the dominance relationship between individuals. At the same time, use the formula Minmize(Z,Offspring.obj) to update the ideal point Z.

[0056] Step 3: Based on the updated population P, calculate the average fitness improvement value AFI of the updated offspring solution to the evolutionary search direction x , the calculation formula is: FI(x, y) = g tch (x,ω y ,z * )-g tch (y,ω y ,z * ); AFIx =∑ y∈Y FI(x,y) / Size(Y)

[0057] Step 4: Based on the FIFO queue entry rule, (B i ,AFI x ) added to .

[0058] Step 5: According to the formula And based on the reward queue R, calculate the weight association set B corresponding to the selected evolutionary search direction i The reward value reward.

[0059] Step 6: Sample batch experience from the experience replay pool T to form a mini-batch of data sets;

[0060] Step 7: Using the formula Calculate the target Q value and get the predicted Q value q of DQN eval =Q(s t ,a t ; Q), jointly calculate the loss function value

[0061] Step 8: Use the gradient descent method of loss to backpropagate and update the parameters in the original network to obtain the updated network Q;

[0062] Step 9: Output the updated population P, reward queue R, and reward value And update the network DQN.

[0063] The advantages of the present invention are:

[0064] 1. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method described in the present invention, through the interaction of deep reinforcement learning and the environment, takes into account the characteristics of the problem instance itself and the current status of the individuals in the population, and flexibly adjusts according to the dynamic needs of different problem instances, and selects excellent evolutionary search directions with higher performance, so as to improve the low problem-solving ability and dynamic adaptability problems of the evolutionary algorithm in the design of the evolutionary search direction selection strategy. The parent solution selected according to the individual fitness value is then combined to form a mating pool. At the same time, the selection based on the roulette method reduces the possibility of the algorithm falling into the local optimum to a certain extent. The above-mentioned deep reinforcement learning assisted evolutionary search direction adaptive selection strategy enables the method to generate excellent offspring solutions with both diversity and convergence, while improving the problem-solving ability and dynamic adaptability of the algorithm, so as to be better applicable to solving various practical problem instances in the real engineering field.

[0065] 2. The deep reinforcement learning-assisted evolutionary super-multi-objective vehicle path planning method described in the present invention updates the population and trains DQN through environmental selection, designs a problem-related credit allocation strategy, and enables the overall method to clearly define the contribution of each action or decision to the final reward to guide the algorithm to adjust the strategy more accurately during the training process, which helps the algorithm to quickly find the optimal path or solution in a complex environment. At the same time, the introduction of efficient data structures such as reward queues and experience replay queues enables the method to more effectively utilize historical data, guide the algorithm to adjust the strategy more accurately during the training process, and achieve effective guidance in large-scale decision spaces, avoiding blind searches and falling into local optimal solutions, improving overall search efficiency and solution quality, and providing a more accurate and efficient solution for super-multi-objective vehicle path planning. In addition, combined with the adaptive evolutionary search direction selection method of deep reinforcement learning, this method can intelligently allocate computing resources to ensure that resources are concentrated in the most promising search directions. This not only reduces unnecessary computing overhead, but also improves the overall performance of the algorithm, making it possible to obtain better optimization results with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flowchart of the deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method.

[0067] Figure 2 This is a flow chart of a deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method in a specific embodiment.

[0068] Figure 3 It is a schematic diagram of the solution to the multi-objective vehicle path planning problem.

[0069] Figure 4 is a comparison chart of the method running times on the test problem set. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0071] The embodiment of the present invention provides a deep reinforcement learning assisted evolutionary super multi-objective vehicle path planning method flow chart as follows Figure 1 The specific pseudo code is shown in Algorithm 1: Deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method mainly includes the following steps:

[0072] Step 1: Construct a mathematical model for the multi-objective vehicle routing problem and set the corresponding constraints and three or more optimization objectives;

[0073] Step 2: Based on the input population size N, the number of targets M, the number of weight vectors V, and the maximum number of iterations MaxGen, initialize the uniformly distributed weight vector set W, the population P representing the solution to the multi-target vehicle routing problem, and the ideal point Z, and associate the weight vectors with the individuals B in the divided population;

[0074] Step 3: Initialize the deep reinforcement learning network DQN, set the appropriate environment, state s, action a, and reward function Reward, and initialize the experience replay queue and reward queue required by DQN later;

[0075] Step 4: Through deep reinforcement learning to assist the adaptive selection strategy of evolutionary search direction, DQN adaptively selects the weighted association set corresponding to the evolutionary search direction, and uses all the associated excellent individuals as part of the parent generation Parent_DQN in the mating pool;

[0076] Step 5: Based on the fitness value of the individual, use the roulette wheel method to select another parent solution set Parent_FE;

[0077] Step 6: Perform crossover and mutation operations on the parent solutions in the mating pool to generate the offspring solution set Offspring;

[0078] Step 7: Environment Selection Updates the Population and Trains the DQN. First, based on the credit allocation strategy, the Tchebycheff aggregation function is used to evaluate the fitness of the solution. The environment selects and updates the population P and the ideal point Z, calculates the reward value, and updates the reward queue R. Then, data from the experience replay queue T is sampled, the loss function value L is calculated, and gradient descent is used to update the DQN network parameters, optimizing the adaptive selection strategy for the evolutionary search direction.

[0079] Step 8: Repeat steps 4 to 7 until the termination condition gen>MaxGen is met. The algorithm ends and outputs the final solution set.

[0080]

[0081]

[0082] The deep reinforcement learning-assisted evolutionary adaptive parent solution selection strategy in the deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method is shown in the pseudo code of Algorithm 2: Deep reinforcement learning-assisted evolutionary direction selection strategy. It mainly includes the following steps:

[0083] Step 1: Input the current population P, weight vector set W, weight association set B corresponding to the evolutionary search direction, experience replay queue T, and trained network Q

[0084] Step 2: According to the current state {p,ωp}Calculate the Q value associated with the weight vector corresponding to each evolutionary direction;

[0085] Step 3: Based on the calculated Q value, select the weighted association set corresponding to the appropriate evolutionary search direction through the roulette wheel method. If the weighted association set corresponding to an evolutionary search direction does not exist in the experience replay pool T, it will no longer be selected; in this case, the weighted association set will be forced to be selected instead of being selected through the roulette wheel method;

[0086] Step 4: Associate the weights corresponding to the selected evolutionary search direction with the set B i All associated excellent individuals are used as part of the parent solution set Parent_DQN in the mating pool MatingPool;

[0087] Step 5: Output the selected excellent parent subset Parent_DQN.

[0088]

[0089] The deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method is applied to the environment selection, population update, and DQN training method. The specific pseudo code is shown in Algorithm 3: Application to Environment Selection, Population Update, and DQN Training Method. It mainly includes the following steps:

[0090] Step 1: Input the current population size P, weight vector set W, weight association set B corresponding to the evolutionary search direction, generated offspring solution set Offspring, selected evolutionary search direction MatingPool, reward queue R, experience replay queue T, and current network Q;

[0091] Step 2: According to the formula Calculate the fitness value of the solution and update the population P based on the dominance relationship between individuals. At the same time, use the formula Minmize(Z,Offspring.obj) to update the ideal point Z.

[0092] Step 3: Based on the updated population P, calculate the average fitness improvement value AFI of the updated offspring solution to the evolutionary search direction x , the calculation formula is: FI(x, y) = g tch (x,ω y ,z * )-g tch (y,ω y ,z * ); AFI x =∑ y∈Y FI(x,y) / Size(Y)

[0093] Step 4: Based on the FIFO queue entry rule, (Bi ,AFI x ) added.

[0094] Step 5: According to the formula And based on the reward queue R, calculate the weight association set B corresponding to the selected evolutionary search direction i The reward value reward.

[0095] Step 6: Sample batch experience from the experience replay pool T to form a mini-batch of data sets;

[0096] Step 7: Using the formula Calculate the target Q value and get the predicted Q value q of DQN eval =Q(s t ,a t ; Q), jointly calculate the loss function value

[0097] Step 8: Use the gradient descent method of loss to backpropagate and update the parameters in the original network to obtain the updated network Q;

[0098] Step 9: Output the updated population P, reward queue R, and reward value And update network Q.

[0099]

[0100]

[0101] The following describes a deep reinforcement learning-assisted evolutionary super-multi-objective vehicle path planning method based on a specific embodiment of a real super-multi-objective vehicle path planning problem test set, and provides a flow chart for solving the deep reinforcement learning-assisted evolutionary super-multi-objective vehicle path planning method in this specific embodiment. Figure 2 For a better understanding of the present invention.

[0102] In this specific embodiment, the deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method mainly includes the following steps:

[0103] Step 1: Collect data on actual problems, such as geographic location, customer needs, and scenario constraints, to form a test set for actual problems. At the same time, complete the modeling of multi-objective vehicle path planning problems;

[0104] The multi-objective vehicle routing problem targeted by this embodiment is a soft time window-based multi-objective vehicle routing problem that includes five optimization objectives, so as to simultaneously consider multiple requirements such as cargo loading, routing planning, and vehicle allocation. Figure 3As shown, the solution to the problem is represented by a sequence of warehouse and customer nodes, and the optimization problem and constraints are as follows:

[0105]

[0106] Where v = {0,…,N} represents the warehouse and user node numbers, node 0 is the warehouse, and the others are customer nodes. R = {r1,…,r k} represents the total path containing n vehicle paths, r1,r n is one of the paths. n =<0,…,v i ,v j ,…,0> represents the node that the vehicle path passes through, 0 represents the warehouse, v i , v j In addition, K represents the number of vehicles, C represents the vehicle capacity, and g i represents the demand of node i, s i represents the service time demand of node i, l i represents the number of loaded items at node i, d ij represents the transportation distance between nodes, [b i ,e i ] represents the time window of the client node service, corresponding to b i and e i Represent the earliest and latest service time of the node, and a i represents the time when the vehicle arrives at node i, α and β represent the time when the vehicle arrives at node b earlier than i The penalty coefficient for arriving at node i and starting service is the same as that for being later than e i The penalty coefficient for arriving at node i and starting service.

[0107] The definition of each objective in objective F = {f1, f2, f3, f4, f5} is: f1 is the number of vehicles, f2 is the total distance traveled, f3 is the total waiting time due to early arrivals, f4 is the total delay time due to late arrivals, and f5 is the sum of the unfilled capacities of all routes. Minimizing f1 aims to reduce the fixed costs of purchasing (or renting) and maintaining vehicles. The variable costs in the vehicle routing problem are estimated using f2, a function of the total travel distance. Therefore, f1 and f2 can be considered transportation costs. f3 and f4 can be considered service costs related to node satisfaction. Minimizing f5 improves the loading rate of each vehicle and avoids wasting capacity resources during vehicle allocation.

[0108] Constraints (1) and (2) indicate that vehicles are subject to load capacity and number of packages. Each vehicle's load must not exceed its maximum load capacity, and the number of packages loaded cannot exceed the maximum number of packages required. Constraints (3) and (4) indicate that all vehicles depart from the distribution center, return to the distribution center after completing all delivery tasks, and each customer is served by only one vehicle. Constraint (5) represents the attributes of the decision variables.

[0109] Step 2: Determine the input parameters population size N, target number M, weight vector size V, and maximum number of iterations MaxGen.

[0110] Step 3: Run the deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method

[0111] First, based on the input parameters population size N, target number M, weight vector size V, and maximum iteration number MaxGen, initialize a set W containing V weight vectors and a population P containing N individuals. At the same time, initialize the DQN model and its state s, action a, reward queue R, and experience replay queue T. Calculate the individual fitness value FE and ideal point Z, and obtain the association set B between the weight vector and the population individuals. Among them, the decision variables of the solution and their weight vectors constitute the state of the DQN, that is, s = (x1,…,x D ,ω1,…,ω M ); The weight vector association set corresponding to the selected evolutionary search direction is used as the action of DQN, that is, a=B i .

[0112] Then, deep reinforcement learning is used to assist the evolutionary search direction adaptive selection strategy, according to the input state {p,ω p}Calculate the Q value associated with the weight vector corresponding to each evolutionary direction. Then, according to the calculated Q value, select the weight association set corresponding to the appropriate evolutionary search direction through the roulette method. Among them, if the weight association set corresponding to an evolutionary search direction does not exist in the experience replay pool T, it will no longer be selected; in this case, the weight association set corresponding to the evolutionary search direction will be forced to be selected instead of being selected through the roulette method; finally, the weight association set B corresponding to the selected evolutionary search direction will be selected. i All associated excellent individuals are used as part of the parent solution set Parent_DQN. In addition, based on the fitness value of the individual, another parent solution set Parent_FE is selected using the roulette wheel method, and together with Parent_DQN, it forms the mating pool MatingPool. The parent solutions in the mating pool are crossover and mutation operations are performed to generate the offspring solution set Offspring;

[0113] Subsequently, the environment selects to update the population and train DQN. First, based on the credit allocation strategy, the Tchebycheff aggregation function is used Evaluate the fitness value of the solution, the environment selects and updates the population P and the ideal point Z, and uses the formula FI(x, y) = g tch (x,ω y ,z * )-g tch (y,ω y ,z * ); AFI x =∑ y∈Y FI(x,y) / Size(Y); Update the reward queue R and calculate the reward value reward. Then, sample the data in the experience replay queue T and use the formula Calculate the target Q value and get the predicted Q value q of DQN eval =Q(s t ,a t ; Q), jointly calculate the loss function value And use the gradient descent method to update the DQN network parameters and optimize the adaptive selection strategy of the evolutionary search direction;

[0114] Repeat other operations except initialization until the termination condition gen>MaxGen is met, the algorithm ends and outputs the final solution set.

[0115] Step 4: Select a suitable vehicle path planning solution based on the actual application scenario requirements and the target values ​​of the solution.

[0116] The effectiveness and convergence ability of the proposed framework are verified by experiments. The experiment selected some real logistics data and preprocessed them to form the final test problem instance. Among them, Problem 1-Problem 4 are modeled by real data of 100, 150, 200, and 245 nodes in the same logistics area. In addition, on this super-multi-objective optimization problem instance, the method proposed in the present invention is also compared with other optimization methods based on deep reinforcement learning, end-to-end network solution methods, and various super-multi-objective optimization algorithms. The maximum number of function evaluations in the experiment is set to 100,000, and each method is run 30 times on the test problem.

[0117] Table 1 summarizes the results of the method proposed in the present invention, and other most advanced optimization methods based on deep reinforcement learning, end-to-end network solving methods and other multi-objective optimization algorithms on the Problem1-Problem4 problem test set. Among them, K represents the final number of vehicles used, D represents the total driving path distance, T represents the total waiting and delay time, and V represents the total vehicle empty load. It can be observed from the data in the table that the method proposed in the present invention performs relatively balanced on multiple objectives, and has achieved optimal values ​​for most of the objective values. It can find a balanced solution set of each objective value with both diversity and convergence on different problem instances, which further proves the high problem-solving ability and dynamic demand adaptability of the method proposed in the present invention when solving different types of instances. In addition, Figure 4 The average running time of each algorithm for solving the test problem set is compared visually. The results show that the method proposed in this invention requires less solution time and has a better solution rate when solving Problem 1-Problem 4, which to a certain extent meets the real-time requirements for solving practical problems. Its optimal effect and solution rate prove the excellent algorithm efficiency, stability and ability to solve practical problems of the method proposed in this article.

[0118] Table 1

[0119]

[0120] In summary, the present invention proposes a deep reinforcement learning-assisted evolutionary method for multi-objective vehicle path planning. By designing a deep reinforcement learning adaptive evolutionary search direction selection method and an environment-selective population update and training DQN method, this method effectively guides large-scale decision space searches, rationally allocates computing resources, improves the algorithm's search efficiency and problem-solving capabilities, and better solves a variety of problem instances and meets the dynamic requirements of the problems, thereby further extending it to practical engineering applications for multi-objective vehicle path planning. Experimental results demonstrate that the proposed multi-objective vehicle path planning method, on multiple different types of multi-objective planning problem test cases, produces path planning solutions that outperform other advanced solution methods in multiple objective values, demonstrating significant performance improvements in search efficiency, problem-solving capability, adaptability to dynamic requirements, and practical applicability. It should be emphasized that those skilled in the art will readily appreciate that various optimizations and modifications, as well as additional implementations and applications beyond the provided examples, can be performed without violating the core principles of the present invention. These improvements and extensions to the present invention should also be included within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning-assisted evolutionary multi-objective vehicle path planning method, characterized by: The following steps are involved: Step 1: Construct a mathematical model for the multi-objective vehicle routing problem and set the corresponding constraints and three or more optimization objectives. Step 2: Based on the input population size N, the number of targets M, the number of weight vectors V, and the maximum number of iterations MaxGen, initialize the uniformly distributed weight vector set W and the population P representing the solution to the multi-target vehicle routing problem. Then, divide the population into regions based on the weight vectors that guide the population evolution direction to form a set B. Step 3: Initialize the deep reinforcement learning network DQN, set the appropriate environment, state s, action a, and reward function Reward, and initialize the experience replay queue T and reward queue R required by DQN later; Step 4: Through deep reinforcement learning-assisted evolutionary search direction adaptive selection strategy, DQN adaptively selects the population evolutionary search direction with high potential and associates all the excellent individuals B in the evolutionary region. i As part of the parent solution set Parent_DQN in the mating pool MatingPool; Step 5: To prevent the algorithm from falling into the local optimum, the population is sorted based on the fitness value of the individuals, and another parent solution set Parent_FE of MatingPool is selected using the roulette wheel method; Step 6: Perform crossover and mutation operations on the parent solution in the MatingPool mating pool to generate the offspring solution set Offspring and update the fitness value of the offspring solution; Step 7: The environment selects and updates the population and trains the DQN. First, based on the credit allocation strategy, the Tchebycheff aggregation function is used to evaluate the fitness value of the solution. The environment selects and updates the population P and the ideal point Z, updates the reward queue R, and calculates the reward value reward. Then, the data in the experience replay queue T is sampled, the loss function value L is calculated, and the gradient descent method is used to update the DQN network parameters to optimize the adaptive selection strategy of the evolutionary search direction. Step 8: Repeat steps 4 to 7 until the termination condition gen>MaxGen is met. The algorithm ends and outputs the final solution set.

2. The deep reinforcement learning-assisted evolutionary super-multi-objective vehicle path planning method according to claim 1 is characterized in that: In step 1, the optimization objectives and constraints of the multi-objective vehicle path planning problem must satisfy the following mathematical definitions: Where M represents the number of targets and M>3, K represents the number of constraints, x=(x1,x2,…,x D )∈Ω is a D-dimensional decision vector, x k (k∈{1,…,D}) represents the decision variable, Represents the search space of decision variables, F(x) is composed of M sub-objective functions f i (x), i∈{1,…,M}determined by the target vector, G(x) is composed of K constraints g j (x), a constraint vector determined by j∈{1,…,K}, where the objective functions conflict with each other.

3. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method according to claim 1 is characterized in that: In step 2, optional input variables can be used to adapt to more different problem instances. At the same time, based on the input data, a set W containing V uniformly distributed weight vectors, a population P containing N individuals, and an ideal point Z are initialized, and the population is divided into regions based on the weight vector that guides the evolutionary direction of the population to form a set B.

4. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method according to claim 1 is characterized in that: In step 3, the DQN state s, action a, and reward function Reward are set, and the experience replay queue T and reward queue R required by the DQN are initialized. The initialization process of the DQN is as follows: Step 3.1 Define state s and action a The decision variables of the solution and their weight vectors constitute the state of DQN, that is, s=(x1,…,x D ,ω1,…,ω M ); The weight vector association set corresponding to the selected evolutionary search direction is used as the action of DQN, that is, a=B i Step 3.2 Define the reward function Reward (1) The fitness value of the solution is calculated using the Tchebycheff aggregation function: where ω=(ω1,…,ω M ) is the weight vector corresponding to the solution x, is an ideal point consisting of the minimum target value in the population; (2) Based on the fitness evaluation function, after the evolutionary operation generates a set of offspring solutions, the average fitness improvement of the evolutionary search direction Y replaced by the offspring solution x can be calculated as follows: FI(x,y)=g tch (x,ω y ,z * )-g tch (y,ω y ,z * ) AFI x =∑ y∈Y FI(x,y) / Size(Y) where ω y represents the weight vector of y, where Y is the set of solutions replaced by x; (3) Based on the average fitness improvement formula, the weighted association set B corresponding to the evolutionary search direction is calculated in the following way i Reward value reward: In order to further improve the search efficiency of the algorithm, the algorithm defines a record containing (B i ,AFI x ) of the first-in-first-out queue R, which saves the AFI of the evolutionary direction update solution within a given time window, where AFI x is the weighted association set B corresponding to the evolutionary search direction i The fitness improvement of the generated offspring solution x, that is, the weighted association set B corresponding to the evolutionary search direction i The reward can be defined as the average value of AFI saved in queue R; Step 3.3 Initialize the experience replay queue T and reward queue R In order to provide the data required for DQN training, the algorithm defines an experience replay queue T. Once a child solution is generated and the reward is calculated, the tuple ({p,ω p },B i , {x,ω x }) is saved in the experience replay queue T, where {p,ω p } is the current solution p and its weight vector ω p , B i is the weight vector association set of the selected evolution direction, {x,ω x } is the generated sub-solution x and its weight vector ω x .

5. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method according to claim 1 is characterized in that: In step 4, the adaptive evolutionary search direction selection method assisted by deep reinforcement learning is used to adaptively select part of the parent solution set Parent_DQN in the mating pool MatingPool using DQN. The method selection process is as follows: First, according to the input state {p,ω p }Calculate the Q value associated with the weight vector corresponding to each evolutionary direction; Then, based on the calculated Q value, the weight association set corresponding to the appropriate evolutionary search direction is selected through the roulette wheel method. If the weight association set corresponding to an evolutionary search direction does not exist in the experience replay pool T, it will no longer be selected; in this case, the weight association set will be forced to be selected instead of being selected through the roulette wheel method; Finally, the weight association set B corresponding to the selected evolutionary search direction i All associated excellent individuals are used as part of the parent solution set Parent_DQN in the mating pool MatingPool.

6. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method according to claim 1, characterized in that: In step five, in order to prevent the algorithm from falling into a local optimum, the population is sorted based on the individual fitness value, and the roulette wheel method is used to select the parent solution set Parent_FE with the same size as the Parent_DQN set from the remaining individuals in the population P excluding Parent_DQN.

7. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method according to claim 1 is characterized in that: In step 7, the fitness value of the solution is updated using the credit allocation strategy. The environment selects the population P and the ideal point Z based on the dominance relationship between individuals, calculates the reward value reward, and updates the reward queue R. The method selection process is as follows: First, the fitness value of the solution is calculated, and based on the dominance relationship between individuals, the population P and the ideal point Z are updated; Then, based on the FIFO queue entry rule, (B i ,AFI x ) is added to the reward queue R, and based on the updated population, the average fitness improvement value AFI of the updated offspring solution to the evolutionary search direction is calculated x And the weighted association set B corresponding to the selected evolutionary search direction i The reward value reward.

8. The deep reinforcement learning assisted evolutionary super-multi-objective vehicle path planning method according to claim 1, characterized in that: In step 7, the data in the experience replay queue T is sampled, the loss function value L is calculated, and the DQN network parameters are updated using the gradient descent method to optimize the adaptive selection strategy of the evolutionary search direction. The training process is as follows: First, batch experience is sampled from the experience replay pool T to form a mini-batch dataset; Then, using the formula Calculate the target Q value and get the predicted Q value q of DQN eval =Q(s t ,a t ; Q), jointly calculate the loss function value Finally, the gradient descent method of the loss is used to backpropagate and update the parameters in the original network to obtain the updated network DQN.

Images