Traveling salesman problem solving method combining deep reinforcement learning and heuristic algorithm

Abstract

The invention provides a deep reinforcement learning and heuristic algorithm combined traveling salesman problem solving method. The method comprises the steps of defining a task of learning an improved heuristic algorithm to solve a traveling salesman problem as a Markov decision process; designing a new model M based on Transform, parameterizing a strategy network, and introducing an action discarding mechanism to prevent action selection from over-fitting; a deep reinforcement learning method (RL-SA) combined with a simulated annealing mechanism is provided to learn a selection strategy of a node to an operator so as to improve the performance of a 2-opt algorithm; the RL-SA uses a whale optimization algorithm to generate an initial solution to improve the sampling efficiency, and uses a Gaussian perturbation strategy to solve the sparse reward problem in reinforcement learning. The method is obviously superior to an existing learning-based method in a TSPLIB data set and a real container liner shipping data set, and the gap between the method and a highly optimized solver is further reduced.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep reinforcement learning, heuristic algorithms and combinatorial optimization, and in particular to a method for solving a traveling salesman problem by combining deep reinforcement learning and heuristic algorithms. Background Art

[0002] The combinatorial optimization problem is a fundamental problem in computer science and operations research, and is widely found in various fields such as logistics, transportation, electricity, and communications. A typical example of a combinatorial optimization problem is the Traveling Salesman Problem (TSP), which is described as a salesman who needs to visit all cities in his area and can only visit each city once and then return to the starting city. TSP has many applications in the real world, such as transportation, multi-robot systems, logistics distribution, and aviation planning. Although researchers have conducted a lot of research on TSP, it is still challenging to obtain the shortest route to complete the trip due to the NP-Hard nature of the problem. Therefore, studying and designing an effective TSP solution method is an urgent problem to be solved.

[0003] Traditional optimization methods such as exact algorithms and heuristic algorithms are the main means to solve TSP. However, as the scale of problems in practical applications continues to increase, exact algorithms may become powerless. Although heuristic algorithms lack theoretical guarantees, they can find approximately optimal solutions within a reasonable computing time. Therefore, heuristic algorithms are widely used in practice. However, heuristic algorithms require a lot of experimental processes, and the quality of the solution is highly dependent on expert domain knowledge.

[0004] In recent years, with the rapid development of artificial intelligence, the remarkable achievements of machine learning in fields such as Go and natural language processing have demonstrated its powerful learning ability. In view of this, some studies have combined deep reinforcement learning (DRL) with heuristic methods to solve TSP. However, most methods focus on learning construction heuristic algorithms, which can gradually generate complete solutions by adding nodes to partial solutions at each step. Such learning construction heuristic algorithms still rely on additional procedures (such as beam search strategies, sampling strategies, etc.) to improve solutions, and these procedures have limited capabilities because they rely on the same training construction strategy. Few studies have focused on learning improvement heuristic algorithms, which can iteratively optimize solutions through learning strategies. Traditional improvement heuristic algorithms are guided by manually designed search strategies, require a lot of expert knowledge, and may only obtain limited improvements. In contrast, DRL can automatically learn better improvement strategies for heuristic algorithms to iteratively solve TSP. Summary of the invention

[0005] In view of the shortcomings of the existing deep reinforcement learning methods and heuristic algorithms for solving TSP, the present invention proposes a method for solving the traveling salesman problem by combining deep reinforcement learning and heuristic algorithms. By designing a novel learning improvement heuristic method framework, the framework can automatically discover better heuristic improvement strategies to iteratively solve TSP.

[0006] In order to achieve the above object, the specific implementation steps of the technical solution of the present invention are as follows:

[0007] A method for solving the traveling salesman problem by combining deep reinforcement learning and heuristic algorithms includes the following steps:

[0008] Step 1: The process of learning to improve the 2-opt heuristic algorithm to solve the Traveling Salesman Problem (TSP) is defined as a reinforcement learning task, which is described using a Markov Decision Process (MDP). First, a complete initial solution is generated. Then, the neural network policy iteratively improves the quality of the solution. The policy automatically selects a node pair operator and adjusts the solution by the node pair operator at each step.

[0009] Step 2: Design a policy network model M based on the transformer model, parameterize the policy network, and introduce an action-dropout layer to prevent overfitting of action selection.

[0010] Step 3: A new deep reinforcement learning RL-SA method integrating simulated annealing mechanism is proposed. This method combines reinforcement learning and simulated annealing algorithms to train the policy network model M, that is, the learning node's selection strategy for the operation operator to achieve an improved 2-opt heuristic algorithm. Then, the training method RL-SA uses the whale optimization algorithm to generate the initial solution to improve the sampling efficiency. In addition, the Gaussian perturbation strategy is used to perturb the state to increase the diversity of solutions and solve the sparse reward problem of the reinforcement learning algorithm.

[0011] The description of the Traveling Salesman Problem (TSP) in step 1 is:

[0012] TSP can be described as given an input graph and represented as a sequence of m nodes (cities) in two-dimensional space: where c i is the coordinates and node features of the ith position. The goal of this problem is to find a node sequence called a travel solution X = (x1,…,x m ), so that each node except the starting node can only be visited once. The objective function is to minimize the Euclidean distance path, denoted as f(X), and its calculation formula is as follows.

[0013]

[0014] Among them, ‖·‖2 represents the L2 norm.

[0015] The description of the 2-opt algorithm in step 1 is:

[0016] Randomly select two non-adjacent nodes in the solution and swap them to get a new solution. Suppose there are m cities, labeled x1, x2, x3, x4, x5, x6, …, x m . Assume that the best solution at the current time step t is s t ,Right now 2-opt strategy randomly selects s t Two non-adjacent nodes in and Then the node Before and Node The node sequence remains unchanged. and nodes The sequence of nodes between is reversed to obtain a new solution, namely

[0017] The components of the Markov decision process (MDP) in step 1 are defined as:

[0018] (1) Status: Status s t It means that there are m nodes in an instance, and a solution at time step t is the node sequence

[0019] (2) Action: Action a t It means that in the current state s t The node pair operation operator under

[0020] (3) Reward value: Reward value function r t The design is as follows:

[0021]

[0022] Among them, function f is the objective function of the TSP problem; is the current optimal solution up to time t; if s t+1 Better than the current optimal solution Right now The current optimal solution will be updated; if a better solution is found, the reward r t >0, otherwise reward r t =0; therefore, the maximum cumulative reward is represented by R, which is calculated as Where T represents the maximum time step, and γ represents the discount factor, which is between [0,1].

[0023] (4) Policy function: Policy π is the state s t Select action a t Transition to next state s t+1 , starting from the initial solution s0 until the maximum time step T is reached. Specifically, the policy function is expressed as P(s T |s0), which is expressed by the probability chain rule as

[0024] (5) Transition function: state s t According to action a t , and according to the Metropolis criterion of the simulated annealing algorithm, it moves to the next state s t+1 .

[0025] The steps of the simulated annealing algorithm are:

[0026] Step 1: Randomly generate the initial solution of TSP, find the optimal solution, and initialize the temperature;

[0027] Step 2: Search near the current solution to get a new solution;

[0028] Step 3: Evaluate the new solution and calculate the objective function f of the new solution t+1 , and compare it with the objective function f of the current solution t If the new solution is better, or if the probability function P t , which is calculated as shown in the formula P t =exp(-(f t -f t+1 ) / T t ) is greater than the generated random number (0,1), the current solution will be updated; otherwise, the current solution remains unchanged, where T t is the current temperature;

[0029] Step 4: Use formula T t+1 = k × T t Update temperature, where k is the temperature attenuation coefficient, and its value is (0,1), T t is the current temperature;

[0030] Step 5: Repeat steps 2-5 until the termination condition is met and output the optimal solution.

[0031] The policy network model M in step 2 is composed of two parts, namely, an encoder and a decoder. Figure 1As shown. The encoder embeds the node features and position features of each node. The decoder generates a probability matrix for selecting each pair of nodes. The details of the encoder and decoder are described below:

[0032] (1) Encoder: The encoder takes node information and position information as input and outputs the feature (or embedding) information of each node. The output dimension is d n Specifically, firstly, the node information is converted into embedded information e by linear mapping. i The position information of each node is converted into embedded information p through sinusoidal position coding. i . Sinusoidal position encoding is represented by sine function and cosine function, and the formula is shown below.

[0033]

[0034] Among them, pos represents the position of the node; d represents the dimension of the node position; mod represents the modulus function. The node information and position information are integrated into a unified feature representation f i , that is, f i =e i +p i . Then, in order to further enhance the embedding of node information, self-attention layer and feedforward neural network layer are adopted, followed by residual connection layer and batch normalization layer. The above process is performed N times in succession with the same structure but different parameters to obtain better feature representation for policy learning. Finally, the self-attention layer enhances the feature representation of nodes through message passing and aggregation between nodes. Consider an input matrix F in , where the columns are the embedding information of the nodes and the output F of the self-attention layer out It is calculated by the following formula.

[0035]

[0036] Q=W q ·F in

[0037] K=W k ·F in

[0038] V=W v ·F in

[0039] Where Q, K and V represent the input matrix F in query, key and value; W q , W k and are trainable parameters, where Indicates a d n×d n The feed-forward layer uses parameter sharing to transform each embedding node independently and uses the ReLu activation function.

[0040] (2) Decoder: The decoder takes node embeddings as input and further processes the input information through self-attention compatibility layers, action dropout layers, mask and softmax layers. The decoder outputs a probability matrix for each selected node pair operator. The self-attention compatibility layer is used to calculate the probability value of each node pair operator being selected in the solution. Given the node embedding matrix F, the self-attention compatibility function S i,j Represents the probability value of each node being selected for the operation operator, which is calculated as follows: Where Q F The calculation method is Q F =W q ·F out , K F The calculation method of K F =W k ·F out Then, the action dropout layer uses the action-dropout mechanism to reduce the overfitting of the neural network. The role of the action dropout layer is to prevent overfitting of action selection and increase the diversity of action selection. Action dropout layer It is calculated by the following formula:

[0041]

[0042] r~Bernoulli(pr)

[0043] where the vector r represents a set of independent Bernoulli random variables, each with probability pr. Finally, the masking layer masks the meaningless diagonal elements and adds a matrix Use the Softmax function, where the node pair (i, j) is an action. First, use the formula The matrix Transform to matrix To control the matrix The entropy of u is , where u represents a constant. Then, the Softmax function is used to normalize the action distribution and obtain the probability P of the node selecting the operation operator. ij , and its calculation formula is:

[0044] The description of the deep reinforcement learning method integrating the simulated annealing mechanism in step 3 is:

[0045] When solving the TSP problem, the reinforcement learning method can only find the global optimal strategy after a large amount of exploration. However, a large amount of exploration will make the entire training process slow and unstable. The simulated annealing algorithm accepts a poor solution with a certain probability, so it can jump out of the local optimal solution. The simulated annealing algorithm is introduced in the transition function of reinforcement learning, that is, the state s t According to action a t , and according to the Metropolis criterion of the simulated annealing algorithm, it moves to the next state s t+1 The present invention integrates the simulated annealing algorithm into the deep reinforcement learning method to effectively balance the exploration phase and the utilization phase of the reinforcement learning method.

[0046] The description of the whale optimization algorithm (WOA) in step 3 is:

[0047] The Whale Optimization Algorithm is a novel swarm intelligence heuristic algorithm inspired by the bubble net feeding behavior of humpback whales. The unique bubble net feeding behavior of humpback whales includes surrounding prey (exploitation phase), spiral attack on prey (exploitation phase) and searching for prey (exploration phase).

[0048] (1) Surrounding prey: Whales are able to observe the location of prey (such as krill, fish, etc.) and completely surround them. WOA assumes that the current best whale (the optimal solution to the problem) is or is close to the location of the prey. When the current optimal solution is determined, the whales will update their positions based on the location of the current best solution. The formula for surrounding prey behavior is as follows:

[0049] s=s best -W·dis

[0050] dis=|2·rand·s best -s|

[0051] W = w·(2·rand-1)

[0052] Where dis is the vector representing the distance between the whale and the current best whale; w represents the adjustment coefficient, whose value decreases linearly from 2 to 0 as the number of iterations increases; w(∈[-w,w]) represents the parameter vector; s represents the position vector of the whale; s best represents the position vector of the current optimal whale; rand(∈[0,1]) represents a random number; |·| represents an absolute value.

[0053] (2) Bubble net attack: The bubble net attack method of humpback whales is simulated by surrounding the prey and spiral attacking the prey. The spiral attack behavior is described as the whale attacking the prey in a logarithmic spiral manner, and its formula is as follows:

[0054] s=cos(2·π·l)·e b·l·dis′+s best

[0055] dis′=|s best -s|

[0056] Where l(∈[-1,1]) represents a random number; dis′ is a vector representing the distance between the whale and the current best whale; b represents a constant value, which defines the logarithmic spiral shape. In this invention, b takes the value of 1. It is worth noting that humpback whales can use both the encirclement and spiral models to update their positions. In order to achieve this synchronous behavior, WOA executes these two position update behaviors through the parameter pw, and the formula is as follows:

[0057]

[0058] Among them, pw(∈[0,1]) represents a random number; β represents a constant.

[0059] (3) Searching for prey: During the hunting process, the prey search behavior is controlled by the change of the coefficient vector W. If |W|≤1, the whale will select the best whale for position update. If |W|>1, the whale will abandon the best whale and randomly select a whale to replace the current best whale for position update. The formula for searching for prey behavior is as follows:

[0060] s=s r -W·dis

[0061] dis=|2·rand·s r -s|

[0062] Among them, s r represents a randomly selected whale position vector.

[0063] The formula of the Gaussian perturbation strategy in step 3 is:

[0064] s=s best +ε·δ

[0065] Among them, δ represents a random vector that obeys the standard normal distribution, that is, δ~N(0,1), and its dimension is the same as s best The same; ε represents the weight of Gaussian perturbation.

[0066] Beneficial effects of the present invention: A new framework is proposed to learn improved heuristic algorithms and automatically discover better improved strategies to iteratively solve TSP. The framework of the present invention first designs a new Transformer-based model M to parameterize the policy network and introduces an action discarding mechanism to prevent overfitting of action selection. Then, a deep reinforcement learning method (called RL-SA) combined with a simulated annealing mechanism is proposed to learn the node selection strategy for the operator to achieve improved performance of the 2-opt algorithm. RL-SA uses the whale optimization algorithm to generate the initial solution to improve sampling efficiency, and uses the Gaussian perturbation strategy to solve the sparse reward problem in reinforcement learning. Experimental results show that the method of the present invention significantly outperforms existing learning-based methods and further narrows the gap with highly optimized solvers. In addition, the pre-trained model M can be used to guide the SA algorithm (called M-SA (Ours)), which performs better than existing deep model methods in small-scale, medium-scale and large-scale TSPLIB datasets, and it shows good performance on real datasets of global container liner shipping routes. For each original route, the method of the present invention can provide an optimized route, effectively reducing the total transportation distance of the original route, which helps shipping companies achieve the goals of reducing fuel costs and increasing profits. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is the strategic network model M of the present invention.

[0068] Figure 2 Schematic diagram comparing the gap between the five methods and the currently known optimal solution on 36 examples. DETAILED DESCRIPTION

[0069] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical methods.

[0070] Step 1: The present invention defines the process of learning to improve the 2-opt heuristic algorithm to solve the traveling salesman problem (TSP) as a reinforcement learning task, which is usually described by a Markov decision process (MDP). TSP can be described as given an input graph and represented as a sequence of m nodes (cities) in a two-dimensional space. where c i is the coordinates and node features of the ith position. The goal of this problem is to find a node sequence called a travel solution X = (x1,…,x m ), so that each node except the starting node can only be visited once. The present invention adopts the path that minimizes the Euclidean distance, denoted as f(X), and its calculation formula is as follows.

[0071]

[0072] Wherein, ‖·‖2 represents the L2 norm. The components of the MDP are defined according to the solution task of the present invention;

[0073] (1) Status: Status s t It means that there are m nodes in an instance, and a solution at time step t is the node sequence

[0074] (2) Action: Action a t It means that in the current state s t The node pair operation operator under

[0075] (3) Reward value: Reward value function r t The design is as follows:

[0076]

[0077] Wherein, function f is the objective function of the TSP problem, and the calculation method is as shown in step 1; is the current optimal solution up to time t; if s t+1 Better than the current optimal solution Right now The current optimal solution will be updated; if a better solution is found, the reward r t >0, otherwise reward r t =0; therefore, the maximum cumulative reward is represented by R, which is calculated as Where T represents the maximum time step, and γ represents the discount factor, which is between [0,1].

[0078] (4) Policy function: Policy π is the state s t Select action a t Transition to next state s t+1 , starting from the initial solution s0 until the maximum time step T is reached. Specifically, the policy function is expressed as P(s T |s0), which is expressed by the probability chain rule as

[0079] (5) Transition function: state s t According to action a t , and according to the Metropolis criterion of the simulated annealing algorithm, it moves to the next state s t+1 .

[0080] Step 2: If Figure 1As shown in the figure, a new model M based on the transformer model is designed to parameterize the policy network and introduce an action-dropout layer to prevent overfitting of action selection. The policy network M model consists of two parts, namely the encoder and the decoder. The encoder embeds the node features and position features of each node. The decoder generates a probability matrix for selecting each pair of node operation operators. The details of the encoder and decoder are described as follows:

[0081] (1) Encoder: The encoder takes node information and position information as input and outputs the feature (or embedding) information of each node. The output dimension is d n = 128. Specifically, it first transforms the information of each node into embedded information e through linear mapping. i The position information of each node is converted into embedded information p through sinusoidal position coding. i . Sinusoidal position encoding is represented by sine function and cosine function, and the formula is shown below.

[0082]

[0083] Among them, pos represents the position of the node; d represents the dimension of the node position; mod represents the modulus function. We fuse the node information and position information into a unified feature representation f i , that is, f i =e i +p i . Then, in order to further enhance the embedding of node information, self-attention layer and feedforward neural network layer are adopted, followed by residual connection layer and batch normalization layer. The above process is performed N times (where N=3) with the same structure but different parameters to obtain better feature representation for policy learning. Finally, the self-attention layer enhances the feature representation of nodes through message passing and aggregation between nodes. Consider an input matrix F in , where the columns are the embedding information of the nodes and the output F of the self-attention layer out It is calculated by the following formula. Both input and output are 128 dimensions.

[0084]

[0085] Q=W q ·F in (6)

[0086] K=W k ·F in (7)

[0087] V=W v ·F in (8) Where Q, K and V represent the input matrix F in query, key and value; W q ,W k and are trainable parameters, where Indicates a d n ×d n The feed-forward layer uses parameter sharing to transform each embedding node independently and uses the ReLu activation function.

[0088] (2) Decoder: The decoder takes node embeddings as input and further processes the input information through self-attention compatibility layers, action dropout layers, mask and softmax layers. The decoder outputs a probability matrix for each selected node pair operator. The self-attention compatibility layer is used to calculate the probability value of each node pair operator being selected in the solution. Considering the node embedding matrix F, the self-attention compatibility function S i,j Represents the probability value of each node pair operator (i, j) being selected, calculated as follows Where Q F The calculation method is Q F =W q ·F out , K F The calculation method of K F =W k ·F out Then, the action dropout layer uses the action-dropout mechanism to reduce the overfitting of the neural network. The role of the action dropout layer is to prevent overfitting of action selection and increase the diversity of action selection. Action dropout layer It is calculated by the following formula:

[0089]

[0090] r~Bernoulli(pr) (10)

[0091] The vector r represents a set of independent Bernoulli random variables, each with a probability of pr (pr = 0.5). Finally, the masking layer masks the meaningless diagonal elements and Use the Softmax function, where the node pair (i, j) is an action. First, use the formula The matrix Transform to matrix To control the matrix The entropy of , where u represents a constant and its value is 10. Then, the Softmax function is used to normalize the action distribution and obtain the node selection probability P for the operation operator.ij , and its calculation formula is:

[0092] Step 3: A new deep reinforcement learning (RL-SA) method integrating simulated annealing mechanism is proposed. This method combines reinforcement learning and simulated annealing algorithms to train the M model, that is, to learn the node's selection strategy for the operator to achieve an improved 2-opt heuristic algorithm. Reinforcement learning solves TSP through the interaction between the agent and the environment. The RL algorithm can only find the global optimal strategy after fully exploring the environment. However, a large amount of exploration will make the entire training process slow and unstable. The SA algorithm accepts poor solutions with a certain probability, so that it can escape from the local optimum. Therefore, we integrate the SA algorithm into the RL method to help effectively balance the exploration and development stages of the RL algorithm. Then, the training method RL-SA uses WOA to generate initial solutions to improve sampling efficiency. In addition, the Gaussian perturbation strategy is used to perturb the state to increase the diversity of solutions and solve the sparse reward problem of the RL algorithm.

[0093] In order to effectively balance the exploration phase and development phase of the RL algorithm, the Metropolis criterion in the SA algorithm is used to calculate the particle temperature at the current temperature T. t The probability of reaching equilibrium is P t , the calculation formula is as follows:

[0094] P t =exp(-(f t -f t+1 ) / T t (11)

[0095] T t+1 = k × T t (12) Where, k represents the temperature decay rate, which ranges from (0,1); f t represents the objective function at time t.

[0096] In order to improve the sampling efficiency of RL-SA, the WOA algorithm is used to generate the initial solution as follows:

[0097] (1) Surrounding prey: Whales are able to observe the location of prey (such as krill, fish, etc.) and completely surround them. WOA assumes that the current best whale (solution to the problem) is or is close to the location of the prey. When the current optimal solution is determined, the whales will update their positions based on the location of the current best solution. The formula for surrounding prey behavior is as follows:

[0098] s=s best -W·dis (13)

[0099] dis=|2·rand·s best-s| (14)

[0100] W=w·(2·rand-1) (15)

[0101] Where dis is the vector representing the distance between the whale and the current best whale; w represents the adjustment coefficient, whose value decreases linearly from 2 to 0 as the number of iterations increases; w(∈[-w,w]) represents the parameter vector; s represents the position vector of the whale; s best represents the position vector of the current optimal whale; rand(∈[0,1]) represents a random number; |·| represents an absolute value.

[0102] (2) Bubble net attack: The bubble net attack method of humpback whales is simulated by surrounding the prey and spiral attacking the prey. The spiral attack behavior is described as the whale attacking the prey in a logarithmic spiral manner, and its formula is as follows:

[0103] s=cos(2·π·l)·e b·l ·dis′+s best (16)

[0104] dis′=|s best -s| (17)

[0105] Where l(∈[-1,1]) represents a random number; dis′ is a vector representing the distance between the whale and the current best whale; b represents a constant value, which defines the logarithmic spiral shape. In this invention, b takes the value of 1. It is worth noting that humpback whales can use both the encirclement and spiral models to update their positions. In order to achieve this synchronous behavior, WOA executes these two position update behaviors through the parameter pw, and the formula is as follows:

[0106]

[0107] Wherein, pw(∈[0,1]) represents a random number; β represents a constant, and its value is 0.5 in the present invention.

[0108] (3) Searching for prey: During the hunting process, the prey search behavior is controlled by the change of the coefficient vector W. If |W|≤1, the whale will select the best whale for position update. If |W|>1, the whale will abandon the best whale and randomly select a whale to replace the current best whale for position update. The formula for searching for prey behavior is as follows:

[0109] s=s r -W·dis (19)

[0110] dis=|2·raand·s r -s| (20)

[0111] Among them, s r represents a randomly selected whale position vector.

[0112] In order to increase the diversity of solutions and solve the sparse reward problem of the RL algorithm, the Gaussian perturbation strategy is used to perturb the state. The calculation formula is:

[0113] s=s best +ε·δ (21)

[0114] Among them, δ represents a random vector that obeys the standard normal distribution, that is, δ~N(0,1), and its dimension is the same as s best The same; ε represents the weight of Gaussian disturbance, which is a constant, and the value of the present invention is 0.35.

[0115] The method of the present invention is evaluated on a typical and representative path problem, namely the TSP problem. These instances are randomly generated according to existing conventions, and the number of nodes is 50. The coordinates of the nodes in the instance are randomly sampled within the unit square (i.e., [0,1]×[0,1]) and obey a uniform distribution. 10,000 random instances of TSP are dynamically generated in each training cycle. WOA is used to generate the initial solution in the training phase, and a random method is used to generate the initial solution in the testing phase. The model M of the present invention is trained and tested on a server equipped with a GeForce RTX GPU card and a 2.10GHz Intel i7-12700 CPU. The parameter values ​​of the present invention are as follows: the discount factor γ is 0.95, the number of samples in each batch B is 512, the number of training rounds E is 200, and the initial learning rate is 10 -4 , the attenuation factor is 0.99, the initial temperature and the final temperature are 100 and 0.01 respectively, the temperature attenuation rate is 0.9, and the weight ε of the Gaussian perturbation is 0.35.

[0116] Table 1 shows the solution results of 36 small-scale and medium-scale instances selected from the TSPLIB test library using the present invention (M-SA (Ours)) and five algorithms, namely OR-Tools, SA, AM, and Wu et al. As shown in Table 1, for the 36 instances, the average gaps of OR-Tools, SA, AM, Wu et al. and M-SA (Ours) relative to the known optimal solutions are 3.46%, 9.86%, 133.53%, 17.15% and 7.34%, respectively. M-SA (Ours) outperforms SA, which indicates that our model can provide a better strategy for SA. All learning-based methods are inferior to the OR-Tools method because machine learning models usually have difficulty in achieving good generalization capabilities outside the distribution of training data. But it is worth noting that M-SA(Ours) outperforms OR-Tools in Eil51, Berlin52, Pr76, KroC100, Pr124, and KroB150 instances, and significantly outperforms AM and Wu et al.’s method in 36 instances.

[0117] In order to more vividly show the comparison of the results of several methods in Table 1, Figure 2 The percentage difference between the OR-Tools, SA, AM, Wu et al., and M-SA (Ours) methods and the currently known best methods on 36 examples is plotted on a logarithmic scale. Figure 2 It can be found that for 36 instances, the gap between our method and the known optimal solution is generally small, while the gap between SA and the other two learning-based algorithms and the known optimal solution is large. At the same time, the performance of our method is relatively stable as the size of the problem nodes increases. In addition, in order to test the method of the present invention on a large-scale dataset, we further selected 18 instances from the TSPLIB dataset, where the number of nodes is as high as 2392. As shown in Table 2, the average gaps between Wu et al.'s method and M-SA (Ours) and the currently known optimal solution are 24.84% and 20.51%, respectively. Therefore, our method has an advantage over some existing state-of-the-art methods on large-scale datasets, although this advantage is smaller than on small-scale and medium-scale datasets.

[0118] About 80% of global trade is transported by sea, so the maritime transport system is very important to the world economy. The rise and volatility of fuel prices is a serious problem for shipping companies. Fuel cost is one of the main factors affecting liner route configuration, accounting for more than 60% of ship operating costs. For any given route, the fuel cost mainly depends on the total distance of the route. Therefore, the goal is to find the route configuration with the shortest total distance, so as to recommend the best route configuration for container ships and reduce the operating costs of shipping companies. We use the method of the present invention to solve the TSP problem in liner route configuration. Compared with the original route configuration, the method of the present invention can effectively find the best route configuration that significantly reduces the total distance. Table 3 shows the original distance (kilometers) of the studied routes, and the number of nodes (i.e., the number of ports) of these routes is up to 31. All six transcontinental routes deploy large container freight capacity (calculated in 20-foot equivalent units, TEUs), indicating their special importance in global liner shipping practices. For each route, we run the method of the present invention 10 times and report the average and standard deviation of the distance optimization percentage (i.e., the percentage of original distance reduction). As shown in Table 3, the average voyage optimization percentage is the smallest at 3.52% for route 4 and the largest at 17.99% for route 1. The standard deviations of the six routes range from 0 to 0.38%. Therefore, the method of the present invention performs well on real-world liner routes.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0120] Table 1 Performance comparison of methods on small-scale and medium-scale TSPLIB datasets

[0121]

[0122]

[0123] Note: The best results among the three learning-based methods are in bold.

[0124] Table 2 Performance comparison of two learning-based methods on the large-scale TSPLIB dataset

[0125]

[0126] Note: Bold indicates the best result among the two learning-based methods

[0127] Table 3 Optimization results of six intercontinental liner routes

[0128]

Claims

1. A method for solving the traveling salesman problem combining deep reinforcement learning and heuristic algorithms, characterized in that: The steps include: Step 1: The process of learning to improve the 2-opt heuristic algorithm to solve the traveling salesman problem TSP is defined as a reinforcement learning task, which is described by a Markov decision process MDP. First, a complete initial solution is generated. Then, the neural network strategy iteratively improves the quality of the solution. The strategy automatically selects a node pair operator, and the node pair operator adjusts the solution at each step. Step 2: Design a policy network model M based on the transformer model, parameterize the policy network, and introduce an action-dropout layer to prevent overfitting of action selection; Step 3: A new deep reinforcement learning RL-SA method integrating simulated annealing mechanism is proposed. This method combines reinforcement learning and simulated annealing algorithms to train the policy network model M, that is, the learning node's selection strategy for the operation operator to achieve an improved 2-opt heuristic algorithm; then, the training method RL-SA uses the whale optimization algorithm to generate the initial solution to improve the sampling efficiency; in addition, the Gaussian perturbation strategy is used to perturb the state to increase the diversity of solutions and solve the sparse reward problem of the reinforcement learning algorithm; The description of the traveling salesman problem TSP in step 1 is: TSP is described as given an input graph, represented as a sequence of m nodes in two-dimensional space. where c i is the coordinates and node features of the ith position; the goal of this problem is to find a node sequence, called a travel solution X = (x1,…,x m ), so that each node except the starting node can only be visited once; the objective function adopts the path that minimizes the Euclidean distance, denoted as f(X), and its calculation formula is as follows; Among them, ‖·‖2 represents the L2 norm; The description of the 2-opt algorithm in step 1 is: Randomly select two non-adjacent nodes in the solution and swap them to obtain a new solution; suppose there are m nodes, labeled x1, x2, x3, x4, x5, x6, …, x m ; Assume that the best solution at the current time step t is s t ,Right now 2-opt strategy randomly selects s t Two non-adjacent nodes in and Then the node Before and Node The node sequence remains unchanged. and nodes The sequence of nodes between is reversed to obtain a new solution, namely The components of the Markov decision process MDP in step 1 are defined as: (1) State: State s t It means that there are m nodes in an instance, and a solution at time step t is the node sequence (2) Action: Action a t It means that in the current state s t The node pair operation operator under (3) Reward value: Reward value function r t The design is as follows: Among them, function f is the objective function of the TSP problem; is the current optimal solution up to time t; if s t+1 Better than the current optimal solution Right now The current optimal solution will be updated; if a better solution is found, the reward r t >0, otherwise reward r t =0; therefore, the maximum cumulative reward is represented by R, which is calculated as Where T represents the maximum time step, γ represents the discount factor whose value is between [0,1]; (4) Policy function: Policy π is the state s t Select action a t Transition to next state s t+1 , starting from the initial solution s0 until the maximum time step T is reached; specifically, the policy function is expressed as P(s T |s0), which is expressed by the probability chain rule as (5) Transition function: state s t According to action a t , and according to the Metropolis criterion of the simulated annealing algorithm, it moves to the next state s t+1 ; The policy network model M in step 2 consists of two parts, namely, an encoder and a decoder. The encoder embeds the node features and position features of each node; the decoder generates a probability matrix for selecting each pair of nodes. The details of the encoder and the decoder are as follows: (1) Encoder: The encoder takes node information and position information as input and outputs the embedding information of each node. The output dimension is d n Specifically, firstly, the node information is converted into embedded information e by linear mapping. i ; The position information of each node is converted into embedded information p through sinusoidal position coding i ; Sinusoidal position coding is represented by sine function and cosine function, and the formula is as follows; Among them, pos represents the position of the node; d represents the dimension of the node position; mod represents the modulus function; the node information and position information are fused into a unified feature representation f i , that is, f i =e i +p i ; Then, in order to further enhance the embedding of node information, self-attention layer and feedforward neural network layer are adopted, followed by residual connection layer and batch normalization layer; the above process is performed N times in succession with the same structure but different parameters to obtain better feature representation for policy learning; finally, the self-attention layer enhances the feature representation of nodes through message passing and aggregation between nodes; consider an input matrix F in , where the columns are the embedding information of the nodes and the output F of the self-attention layer out It is calculated by the following formula; Q=W q ·F in K=W k ·F in V=W v ·F in Where Q, K and V represent the input matrix F in query, key and value; W q , W k and are trainable parameters, where Indicates a d n ×d n The real matrix space of ; the feed-forward layer uses parameter sharing, independently transforms each embedding node, and uses the ReLu activation function; (2) Decoder: The decoder takes node embeddings as input and further processes the input information through self-attention compatibility layers, action dropout layers, masks, and softmax layers. The decoder outputs a probability matrix for each selected node pair operator. The self-attention compatibility layer is used to calculate the probability value of each node pair operator being selected in the solution. Given the node embedding matrix F, the self-attention compatibility function S i,j Represents the probability value of each node being selected for the operation operator, which is calculated as follows: Where Q F The calculation method is Q F =W q ·F out , K F The calculation method of K F =W k ·F out ; Then, the action dropout layer uses the action-dropout mechanism to reduce the overfitting of the neural network; the role of the action dropout layer is to prevent overfitting of action selection and increase the diversity of action selection; the action dropout layer It is calculated by the following formula: r~Bernoulli(pr) The vector r represents a set of independent Bernoulli random variables, each with probability pr; finally, the masking layer masks the meaningless diagonal elements and applies the matrix Use the Softmax function, where the node pair (i, j) is an action; first, use the formula The matrix Transform to matrix To control the matrix The entropy of u is , where u represents a constant; then, the Softmax function is used to normalize the action distribution and obtain the probability P of the node selecting the operation operator. ij , and its calculation formula is:

2. The method for solving the traveling salesman problem combining deep reinforcement learning and heuristic algorithm according to claim 1, characterized in that: The steps of the simulated annealing algorithm are: Step 1: Randomly generate the initial solution of TSP, find the optimal solution, and initialize the temperature; Step 2: Search near the current solution to get a new solution; Step 3: Evaluate the new solution and calculate the objective function f of the new solution t+1 , and compare it with the objective function f of the current solution t Compare; if the new solution is better, or if the probability function P t , which is calculated as shown in the formula P t =exp(-(f t -f t+1 ) / T t ) is greater than the generated random number (0,1), the current solution will be updated; otherwise, the current solution remains unchanged, where T t is the current temperature; Step 4: Use formula T t+1 = k × T t Update temperature, where k is the temperature attenuation coefficient, and its value is (0,1), T t is the current temperature; Step 5: Repeat steps 2-5 until the termination condition is met and output the optimal solution.

3. The method for solving the traveling salesman problem combining deep reinforcement learning and heuristic algorithm according to claim 1, characterized in that: The description of the whale optimization algorithm WOA in step 3 is: (1) Surrounding prey: Whales can observe the location of prey and completely surround them. WOA assumes the current best whale, that is, the optimal solution to the problem or the location close to the prey. When the current optimal solution is determined, the whales will update their positions based on the location of the current best solution. The formula for surrounding prey behavior is as follows: s=s best -W·dis dis=|2·rand·s best -s| W = w·(2·rand-1) Where dis is the vector representing the distance between the whale and the current best whale; w represents the adjustment coefficient, whose value decreases linearly from 2 to 0 as the number of iterations increases; w(∈[-w,w]) represents the parameter vector; s represents the position vector of the whale; s best represents the position vector of the current optimal whale; rand(∈[0,1]) represents a random number; |·| represents an absolute value; (2) Bubble net attack: The bubble net attack method of humpback whales is simulated by surrounding prey and spiral attack prey behaviors; the spiral attack behavior is described as the whale attacking prey in a logarithmic spiral manner, and its formula is as follows: s=cos(2·π·l)·e b·l ·dis′+s best dis′=|s best -s| Wherein, l(∈[-1,1]) represents a random number; dis′ is a vector representing the distance between the whale and the current best whale; b represents a constant value, which defines the logarithmic spiral shape, and in the present invention, b is 1; humpback whales can simultaneously adopt both the encirclement and spiral models to update their positions; in order to achieve this synchronous behavior, WOA executes these two position update behaviors through the parameter pw, and the formula is as follows: Where pw(∈[0,1]) represents a random number; β represents a constant; (3) Searching for prey: During the hunting process, the prey search behavior is controlled by the change of the coefficient vector W. If |W|≤1, the whale will select the best whale for position update. If |W|>1, the whale will abandon the best whale and randomly select a whale to replace the current best whale for position update. The formula for searching for prey behavior is as follows: s=s r -W·dis dis=|2·rand·s r -s| Among them, s r represents a randomly selected whale position vector.

4. The method for solving the traveling salesman problem combining deep reinforcement learning and heuristic algorithm according to claim 1, characterized in that: The formula of the Gaussian perturbation strategy in step 3 is: s=s best +e·d Among them, δ represents a random vector that obeys the standard normal distribution, that is, δ~N(0,1), and its dimension is the same as s best The same; ε represents the weight of Gaussian perturbation.

Images