Large-scale traveling salesman problem solving method based on deep reinforcement learning
By modeling the large-scale travel dealer problem as a Markov decision-making process, and adopting a fast attention mechanism and a multi-pointer network, combined with REINFORCE algorithm and parallel computing, the challenges of computing efficiency and memory utilization in large-scale travel dealer problems are solved. Achieve more efficient solution speed and memory management.
Patent Information
- Application Number
- CN202411011372.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-29
AI Technical Summary
The existing technology is difficult to solve the problem of large-scale travel dealers within a reasonable time, especially in terms of computing efficiency and memory utilization, resulting in excessive inference time and memory overflow problems.
The large-scale travel dealer problem is modeled as the Markov decision-making process, and a method based on deep reinforcement learning is adopted, and a fast attention mechanism and multi-pointer network optimization model is combined with REINFORCE algorithm and parallel computing to reduce computing time and memory consumption.
It significantly improves the training and inference speed of the model, enhances memory utilization efficiency, and can solve large-scale travel merchant problems more efficiently, and meets the solution speed requirements in practical applications.
Smart Images

Figure CN119047671A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of traveling salesman problem solving, and in particular relates to a large-scale traveling salesman problem solving method based on deep reinforcement learning. Background Art
[0002] The traveling salesman problem is a classic combinatorial optimization problem, which is defined as finding a circuit that minimizes the total travel distance or cost given a set of cities and their distance or cost matrices between each other. The circuit must pass through each city only once.
[0003] The traveling salesman problem has a wide range of applications in real life, such as distribution route planning, package delivery, store replenishment, etc. Although the traveling salesman problem itself is simple and clear, it has been proven to be an NP-hard problem, that is, it is difficult to find the optimal solution in polynomial time. With the rapid development of social economy, people have higher and higher requirements on the efficiency and cost of various logistics distribution, and the solution of the traveling salesman problem has also received increasing attention. For the traveling salesman problem, researchers have proposed a variety of solution algorithms, mainly including three categories: exact algorithms, heuristic algorithms and meta-heuristic algorithms.
[0004] Exact algorithms can achieve advantages in terms of solution quality. Currently, the most famous one is the work published by Applegate et al. in 2006, "The traveling salesman problem: a computational study. Princeton university press". This work proposed the Concorde solver, which combines the cutting plane algorithm with the branch and bound method. However, as the scale of the problem increases, its efficiency decreases significantly, and it is impossible to obtain the optimal solution or approximate solution within a reasonable time.
[0005] In small-scale problems, heuristic algorithms can give satisfactory solutions in a short time. The most famous one is the work published by Helsgaun in 2017, "An extension of the lin-kernighan-helsgaun tspsolver for constrained traveling salesman and vehicle routing problems. Technical report". This work uses an improved method to iteratively decide which edges to reconnect to reduce the path length. However, as the scale of the vehicle routing problem increases and complex constraints are added, it is often difficult to give satisfactory results.
[0006] In recent years, some groundbreaking works have combined deep learning with reinforcement learning and achieved good results in the traveling salesman problem. These works use the feature extraction capabilities of deep neural networks to design algorithms based on deep reinforcement learning. These algorithms can extract potential relationships from large-scale data that are difficult to summarize manually. However, when these works are applied to large-scale traveling salesman tasks, they will face problems such as long inference time and memory overflow. Therefore, it is necessary to design more efficient models to solve the above problems. Summary of the invention
[0007] The present invention provides a method for solving a large-scale traveling salesman problem based on deep reinforcement learning. By optimizing the memory usage of the algorithm in a targeted manner, the memory utilization efficiency is improved to support the solution of larger-scale problems. By optimizing the computational efficiency of the algorithm, the computational time is greatly reduced to meet the demand for solution speed in practical applications.
[0008] A method for solving a large-scale traveling salesman problem based on deep reinforcement learning, comprising the following steps:
[0009] (1) Model the large-scale traveling salesman problem as a Markov decision process, including states, actions, strategies, and rewards;
[0010] (2) Randomly generate a series of large-scale traveling salesman problem instances and perform data augmentation to obtain training data;
[0011] (3) Construct a large-scale traveling salesman problem solving model architecture based on end-to-end deep reinforcement learning. The model consists of an encoder and a decoder. The encoder uses a fast attention mechanism as the attention layer, and the decoder uses a special context node to represent the current decoding state. This context node is dynamically updated as the decoding step progresses, pointing to the next city node to be visited.
[0012] (4) Design a decision network and use the training data to train the constructed solution model; the decision network selects REINFORCE and uses the symmetry in the deep reinforcement learning method to improve the REINFORCE algorithm;
[0013] (5) Use the trained model to solve the traveling salesman problem and optimize the solution to improve the quality of the solution.
[0014] In the process of using reinforcement learning to solve the Great Traveling Salesman Problem, we first need to model the Great Traveling Salesman Problem as a Markov decision process. In step (1), the state is the set of cities that have been visited, the action is the next city to be visited, the strategy is the probability of selecting each city in the next step, and the reward is the path length.
[0015] In step (2), 500-2000 cities are randomly generated, and the coordinate positions of these cities are evenly distributed on a two-dimensional plane.
[0016] In step (2), data enhancement is performed through operations such as rotation, translation, flipping, and scaling.
[0017] In step (3), the encoder includes an input layer, an attention layer, and a feedforward network layer, wherein the attention layer adopts a fast attention mechanism, which transfers n 2 The matrix of size is divided into blocks.
[0018] The encoder uses an attention-based neural network architecture that can adaptively extract important features from problem instances. The attention mechanism can dynamically assign different weights to different nodes, capture the correlation between nodes, and thus better encode problem instances. By encoding the input nodes using multiple attention layers consisting of a multi-head self-attention layer and a feedforward layer, the representation of important nodes in the problem instance can be learned, providing effective information for the subsequent decision-making process.
[0019] But the ordinary self-attention mechanism requires O(n 2 ) memory consumption, and cannot solve the Great Traveling Salesman Problem. In order to save memory consumption, a fast attention mechanism is used as the self-attention layer. The memory efficiency of the fast attention mechanism is linear O(n) compared to the ordinary attention mechanism. And unlike the sparse or low-rank matrix approximation method, its output is the same as the ordinary attention mechanism.
[0020] In step (3), the decoder adopts a multi-pointer network and uses a special context node to represent the current decoding context during the decoding process. This context node will be dynamically updated as the decoding step proceeds to point to the next node to be accessed.
[0021] The decoder uses a pointer network-like architecture, using a special context node to represent the current decoding context during the decoding process. This context node is dynamically updated as the decoding steps proceed, pointing to the next node to be visited. Through this pointer-like decoding method, the decoder can directly generate the final sequence of visited nodes without going through a complex search process.
[0022] Autoregressive decoders are usually very fast, but memory consumption is high, mainly due to the attention module used in the query. To alleviate this, the present invention chooses to remove the attention module in the query. At the same time, in order to reduce the impact of the removal of this module, the decoder is improved by integrating features, and an enhanced context embedding method is used to include more comprehensive context information in the query.
[0023] In step (4), the decision network selects REINFORCE, and uses the symmetry in deep reinforcement learning methods to improve the REINFORCE algorithm and use a shared baseline:
[0024]
[0025] In the formula, π~p θ (·|s) refers to the strategy under given parameters θ and current state s, i.e., the generated path; L(π|s) represents the length of the generated path; b(s) represents the shared baseline, i.e., the average length of some strategies; p θ (π|s) represents the probability of generating path π given parameter θ and current state s.
[0026] The core idea of REINFORCE is to adjust the parameters of the policy network so that the agent can obtain higher cumulative rewards when interacting with the environment. The invention exploits the symmetry in RL methods for solving the Great Traveling Salesman Problem. This symmetry can be exploited during neural network training by running multiple Rollouts in parallel, with each trajectory having a different optimal solution as the goal of exploration.
[0027] In step (5), the original problem is split into multiple smaller shortest Hamiltonian path sub-problems, and the optimization is performed by parallel computing such as multi-core CPU or GPU, while optimizing multiple sub-problems at the same time.
[0028] Considering the complexity of the large traveling salesman problem, directly optimizing the entire problem may be inefficient. Therefore, the original problem is divided into multiple smaller-scale shortest Hamiltonian path subproblems, each of which involves fewer nodes. For the multiple shortest Hamiltonian path subproblems obtained by division, parallel computing can be used to optimize them. By using parallel computing resources such as multi-core CPUs or GPUs, multiple subproblems can be optimized at the same time, greatly improving the optimization efficiency. Through this divide-and-conquer strategy, the advantages of the initial solution can be fully utilized, while reducing the complexity of the large-scale traveling salesman problem, and further improving the optimization efficiency through parallel computing.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] 1. Based on the understanding of the large traveling salesman problem, the present invention utilizes the symmetry in the RL method to perform data enhancement and improve the robustness of the model.
[0031] 2. The present invention adopts a novel fast attention mechanism as the encoder, which reduces the model memory usage while improving the training and reasoning speed of the model.
[0032] 3. The present invention removes the attention module in the decoder, which improves the training and reasoning speed of the model. At the same time, in order to maintain the accuracy of the model, the method of enhancing context embedding is used.
[0033] 4. The present invention adopts a divide-and-conquer approach and utilizes parallel computing resources such as multi-core CPUs or GPUs to further optimize the solution results. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a flow chart of a method for solving a large-scale traveling salesman problem based on deep reinforcement learning according to an embodiment of the present invention;
[0035] Figure 2 It is the overall framework diagram of the solution model of the present invention;
[0036] Figure 3 Schematic diagram of the fast attention mechanism added in the encoding layer of the present invention. DETAILED DESCRIPTION
[0037] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.
[0038] like Figure 1 As shown, a method for solving a large-scale traveling salesman problem based on deep reinforcement learning includes the following steps:
[0039] Step 1: Establish a large-scale traveling salesman problem model based on Markov process.
[0040] State representation: The set of cities that have been visited is defined as the current state S. Each time a new city is visited, the state S is updated.
[0041] Action representation: Select the next city to be visited from the cities that have not been visited, defined as action A.
[0042] Strategy representation: Use probability distribution for unvisited cities to represent the probability of which city to choose next. This probability distribution is the strategy π.
[0043] Reward means: Every time you visit a new city, the length of the travel path increases. The ultimate goal is to find a cycle path with the shortest total length, and this path length is the reward R.
[0044] Step 2, training data generation and processing. First, randomly generate a set of two-dimensional coordinate points, which represent the locations of cities that need to be visited in sequence. The number of coordinate points can be set according to demand, and usually about 500-2000 cities will be generated. The coordinate positions of these cities can be uniformly distributed or otherwise randomly distributed on a two-dimensional plane. Normalize the coordinate values of the generated large-scale traveling salesman problem instance data, rotate, translate, flip, and scale, and enhance the data features. Among the generated city coordinate points, randomly select one of them as the starting node of the travel path.
[0045] Step 3: Build a large-scale traveling salesman problem solving model architecture based on end-to-end deep reinforcement learning. Figure 2 As shown in the figure, the overall framework of the solution model of the present invention. The model adopts an end-to-end deep reinforcement learning architecture, which mainly consists of the following two modules:
[0046] The encoder uses a neural network architecture based on the attention mechanism. The input layer encodes large-scale traveling salesman problem instances (such as city coordinates) into vector representations suitable for neural network input. The attention layer uses a multi-head self-attention mechanism to allow the network to focus on the relationship between input nodes. The feedforward layer is a feedforward neural network layer added after the attention layer to further extract and combine features. The feedforward layer can enhance the network's expressive power and learn richer feature representations. The final output of the encoder is a compact feature vector that summarizes the important information of the input large-scale traveling salesman problem. This feature vector will be passed to the subsequent decoder module to provide a basis for solving the large-scale traveling salesman problem.
[0047] The encoder uses the fast attention mechanism as the attention layer. The memory efficiency of the fast attention mechanism is linear O(n) compared to the normal attention mechanism. And unlike the sparse or low-rank matrix approximation methods, its output is the same as the normal attention mechanism.
[0048] The fast attention mechanism can basically be summarized into two main improvements. 2 The matrices of size are divided into blocks, such as Figure 3 Assume x = [x (1) , x (2) ], where x (1) The vector representing the first block, x (2) represents the vector of the second block, then:
[0049] m(x)=max(m(x) (1) ), m(x (2) ))
[0050]
[0051]
[0052] In the formula, m(x) represents the maximum value in the vector, f(x) represents an exponential vector, The maximum value m(x) in the vector is subtracted from the exponential term, and l(x) represents the sum of f(x).
[0053] Another GPU memory usage during training is back propagation. The fast attention mechanism does not need to save n during training. 2 By storing the output, m(x), and l(x), it is straightforward to inversely compute the attention matrix, keeping memory in O(n).
[0054] The decoder uses an architecture similar to the pointer network. The decoder receives feature representations from the encoder, which contain key information about the problem instance. A special context node is used to represent the current decoding state. This context node is dynamically updated as the decoding step progresses, pointing to the next city node to be visited. Based on the current context node, the decoder can directly predict the next city to be visited and add it to the generated path without the need for a complex search process.
[0055] In the decoder, a multi-pointer network is used, and the context embedding is continuously updated as the nodes are continuously selected until a feasible path is obtained. With the same first and last nodes, two paths may also lead to different distributions on the nodes to be visited, so an effective and enhanced context embedding q is used in the decoder. t , which contains the following information:
[0056] τ: Generated path
[0057] The embedding of the first node on τ, a static information, i.e. the embedding of the warehouse;
[0058] The embedding of the last node on τ is updated according to the dynamic information of the current route;
[0059] h g : Graph embedding, encoding the entire LTSP instance, i.e. summing the embeddings of all nodes in the instance: in is the embedding of the i-th node obtained by the encoder;
[0060] h τ : Embedding of τ, encoding the current partial route, that is, summing the embeddings of all nodes on τ
[0061]
[0062] The entire encoding-decoding architecture can be trained and optimized end-to-end, allowing the encoding and decoding modules to work together to solve the large-scale traveling salesman problem.
[0063] Step 4 is to design a decision network and train the model. The decision network selects REINFORCE. The REINFORCE algorithm can theoretically guarantee local optimality and relies on the Monte Carlo sampling method. The sampling gradient of REINFORCE is unbiased. However, due to Monte Carlo sampling, the variance of REINFORCE gradient estimation is large, which may reduce the learning rate. Therefore, it is necessary to subtract a reward baseline to reduce the variance, which means updating based on advantages rather than pure rewards. The simplest baseline function is the average reward of several fragments. So the formula for the decision network is as follows:
[0064]
[0065] In the formula, π~p θ (·|s) refers to the strategy given the parameters θ and the current state s, i.e., the generated path. L(π|s) represents the length of the generated path. b(s) represents the shared baseline, i.e., the average length of the partial strategies. p θ (π|s) represents the probability of generating path π given parameter θ and current state s.
[0066] The REINFORCE algorithm is improved by using the symmetry in deep reinforcement learning methods. Each trajectory has a different optimal solution as the exploration goal, forcing different trajectories to move towards all optimal solutions. Therefore, based on the original REINFORCE, it is proposed to use a shared baseline to reduce the variance of the sampled gradient.
[0067]
[0068] In the formula, R(τ j ) represents the path τ j The length of N paths, b(s) represents the average length of N paths. With a shared baseline, each trajectory now competes with other trajectories, where no two trajectories are identical. As the number of heterogeneous trajectories increases, all of this helps to set the baseline at the right level, which is very resistant to local minima.
[0069] Step 5, optimize the initial solution generated in the previous step. According to the initial solution, the original problem is divided into multiple smaller shortest Hamiltonian path sub-problems, each of which involves a smaller number of nodes. In order to effectively solve these small-scale shortest Hamiltonian path sub-problems, a special optimization model needs to be trained in advance. This model is designed to solve small-scale shortest Hamiltonian path problems quickly and efficiently. For a batch of shortest Hamiltonian path sub-problems, parallel computing resources such as multi-core CPUs or GPUs are used, and the trained model is used for optimization at the same time.
[0070] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for solving large-scale traveling salesman problem based on deep reinforcement learning, characterized in that: The following steps are involved: (1) Model the large-scale traveling salesman problem as a Markov decision process, including states, actions, strategies, and rewards; (2) Randomly generate a series of large-scale traveling salesman problem instances and perform data augmentation to obtain training data; (3) Construct a large-scale traveling salesman problem solving model architecture based on end-to-end deep reinforcement learning. The model consists of an encoder and a decoder. The encoder uses a fast attention mechanism as the attention layer, and the decoder uses a context node to represent the current decoding state. This context node is dynamically updated as the decoding step progresses, pointing to the next city node to be visited. (4) Design a decision network and use the training data to train the constructed solution model; the decision network selects REINFORCE and uses the symmetry in the deep reinforcement learning method to improve the REINFORCE algorithm; (5) Use the trained model to solve the traveling salesman problem and optimize the solution to improve the quality of the solution.
2. The method for solving the large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (1), the state is the set of cities that have been visited, the action is the next city to be visited, the strategy is the probability of selecting each city in the next step, and the reward is the path length.
3. The method for solving the large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (2), 500-2000 cities are randomly generated, and the coordinate positions of these cities are evenly distributed on a two-dimensional plane.
4. The method for solving large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (2), data enhancement is performed through operations such as rotation, translation, flipping, and scaling.
5. The method for solving large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (3), the encoder includes an input layer, an attention layer, and a feedforward network layer, wherein the attention layer adopts a fast attention mechanism, which transfers n 2 The matrix of size is divided into blocks.
6. The method for solving large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (3), the decoder adopts a multi-pointer network and uses a special context node to represent the current decoding context during the decoding process. This context node will be dynamically updated as the decoding step proceeds to point to the next node to be accessed.
7. The method for solving large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (4), the decision network selects REINFORCE, and uses the symmetry in deep reinforcement learning methods to improve the REINFORCE algorithm and use a shared baseline: In the formula, π~p θ (·|s) refers to the strategy under given parameters θ and current state s, that is, the generated path; L(π|s) represents the length of the generated path;. b(s) represents the shared baseline, i.e., the average length of some strategies; p θ (π|s) represents the probability of generating path π given parameter θ and current state s.
8. The method for solving large-scale traveling salesman problem based on deep reinforcement learning according to claim 1, characterized in that: In step (5), the original problem is split into multiple smaller shortest Hamiltonian path sub-problems, which are optimized using parallel computing.
Citation Information
Cited By
CTSP solving method and system based on convolution enhanced proxy attention mechanism
CN120354086A
A CTSP solving method and system based on convolution-enhanced proxy attention mechanism
CN120354086B
