Combined optimization algorithm based on improved POMO algorithm

Abstract

The invention discloses a combinatorial optimization algorithm based on an improved POMO algorithm, which is used for solving the problems of solution space explosion in combinatorial optimization problems such as a traveling salesman problem, large gradient estimation variance of a traditional POMO algorithm, insufficient solution space exploration and lack of local optimization capability. The method comprises the following steps: constructing a two-dimensional map with node spacing constraint; a Transform architecture encoder-decoder model containing a multi-head self-attention mechanism is adopted, and a multi-head self-attention mechanism is adopted; a multiplexing sample is decoded in parallel, a self-adaptive dominant function is introduced based on an improved plug-in of REINFORCE to reduce gradient variance, and an evolutionary algorithm plug-in containing 2-opt optimization and a dual-buffering mechanism optimization algorithm are embedded; and finally finishing the TSP path planning. According to the algorithm, O (n2logn) complexity is kept, and the large-scale TSP problem solving efficiency and precision are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of combinatorial optimization algorithm, in particular to a combinatorial optimization algorithm based on improved POMO algorithm, which is mainly used for solving the path planning problem in the combinatorial optimization problem such as traveling salesman problem (TSP). BACKGROUND

[0002] As an intersection of discrete mathematics and optimization theory, the core challenge of combinatorial optimization problem is the exponential explosion of solution space with problem size. Taking traveling salesman problem (TSP) as an example, when the number of nodes increases from 20 to 100, the theoretical solution space increases from 6.4x10 17 to about 9.3x10 157 This dimension disaster makes traditional algorithms face serious challenges in dealing with actual large-scale problems. Although the traditional exact algorithm such as dynamic programming can guarantee optimality, its O(n 2 2 n ) time complexity is only suitable for scenarios with less than 30 nodes; while the heuristic algorithm such as genetic algorithm can be solved in polynomial time, it has a significant defect - in the 100-node TSP test, the average deviation of the solution of genetic algorithm from the optimal solution is 18.7%, and the fluctuation range of each run is more than 25%, which is difficult to meet the requirements of solution stability in logistics scheduling and other scenarios.

[0003] Deep reinforcement learning provides a new data-driven paradigm for combinatorial optimization. POMO algorithm models TSP as a Markov decision process, and uses the encoder-decoder of Transformer architecture to capture node dependency, which improves the solution quality by about 12% compared with traditional heuristic algorithms on 50-node TSP. However, this algorithm has three major technical bottlenecks:

[0004] Large variance of gradient estimation: Monte Carlo sampling based on REINFORCE leads to the consumption of more than 48 hours for training a 200-node model, which is 80 times the cost of genetic algorithm;

[0005] Insufficient exploration of solution space: under the fixed baseline strategy, the probability of the model falling into local optimum when the node size exceeds 100 is as high as 63%;

[0006] Lack of local optimization ability: about 37% of the segments in the generated path have 2-opt optimization space, and traditional POMO cannot automatically trigger local improvement mechanism.

[0007] In existing improvement research, although the graph attention network enhances the modeling of node association, it increases the computational complexity by 40%; the priority experience replay only improves the sample efficiency by 2 times, which still cannot meet the real-time solving demand of industrial level. Especially in dealing with real scenes with spatial clustering characteristics (such as the dense distribution of commercial nodes in urban distribution), the traditional POMO cannot introduce prior structures, and the solution quality is about 22% lower than the theoretical optimum. The present application improves the baseline strategy and embeds a genetic algorithm plug-in, while maintaining the O(n 2 logn) computational complexity, realizing the coordinated improvement of the solving efficiency and accuracy of large-scale TSP problems. SUMMARY

[0008] In order to solve the problems existing in the traditional POMO algorithm, the present application proposes an improved strategy gradient algorithm baseline strategy combined with a genetic algorithm plug-in, a combination optimization algorithm and system based on an improved POMO algorithm. The present application improves the efficiency and accuracy of the algorithm in TSP problem path planning through the construction of an optimization map, the design of an encoder-decoder model structure, and the improvement of the optimization algorithm.

[0009] To achieve the above purpose, the present application provides a combination optimization algorithm based on an improved POMO algorithm, comprising the following steps:

[0010] 1. Constructing the path nodes of the Traveling Salesman Problem (TSP) path planning problem as an optimization map;

[0011] 2. Based on the attention mechanism and the strategy gradient algorithm (REINFORCE) based on Monte Carlo, constructing an encoder-decoder model structure;

[0012] 3. Improving the baseline strategy of the strategy gradient algorithm and combining it with a genetic algorithm plug-in to obtain a final model;

[0013] 4. Using the final model to complete the TSP problem path planning.

[0014] The optimization map constructed in the above steps adopts a two-dimensional space map, specifically a Euclidean plane coordinate system of a unit square region [0, 1] x [0, 1], and the node coordinates are generated by uniform random sampling and subjected to minimum spacing constraint (≥0.02 unit length). The system presets four node scale configurations: 20-node benchmark set, 50-node standard set, 100-node stress test set and 200-node limit verification set, and all node initial states are isolated points without any preset adjacency relationship. The map boundary adopts a periodic processing mechanism, and when the coordinates exceed the boundary, it is automatically wrapped to the opposite side. The distance matrix is calculated in real time to calculate the Euclidean distance between nodes, and any prior clustering structure or spatial partition mode is prohibited.

[0015] The encoder-decoder model structure based on attention mechanism and Monte Carlo-based policy gradient algorithm (REINFORCE) is essentially applying the Transformer architecture to combinatorial optimization problems, using multi-head self-attention mechanism to capture the dependency between nodes, mapping input nodes to high-dimensional features through the encoder, and using autoregressive generation to construct the path in the decoder. Specifically, the POMO model framework adopts the encoder-decoder structure of the Transformer architecture, the encoder contains 8 multi-head self-attention mechanisms, each with a dimension of 64, cooperates with a learnable position encoding module, maps the input nodes to a 128-dimensional feature space through 6 Transformer layers, uses residual connection and layer normalization technology to ensure training stability, and uses GeLU activation function to enhance the non-linear expression ability; the decoder uses autoregressive generation, each time step dynamically aggregates the encoder output features through the context query attention layer, with a complexity of O(n2), while maintaining a binary dynamic mask matrix to prevent node duplication.

[0016] The improved optimization POMO algorithm includes: sampling sample multiplexing, improving the baseline strategy, and adding an evolutionary algorithm optimization module.

[0017] In the above improvement strategy, the parallel decoding mechanism is supported, multiple paths are generated for the same node set according to different starting nodes to improve sample utilization, and diversified solutions are generated through different random seeds.

[0018] The REINFORCE algorithm is used to perform gradient ascent optimization on the encoder-decoder parameters, and the formula is as follows:

[0019]

[0020] Where θ represents the encoder-decoder parameter, J(θ) is the loss function, n represents the number of current batch routes, A i represents the advantage function of the i-th route, represents the gradient probability of the current route. The advantage function formula is as follows:

[0021]

[0022] Where current_repoch is the current training round, and reporch is the total number of rounds.

[0023] In addition, the shared baseline value is the average value of all routes, and the specific formula is as follows:

[0024]

[0025] Where R i is the sampling route of the same sample with different starting points.

[0026] The evolutionary algorithm plug-in optimization module initiates 2-4 rounds of lightweight evolutionary cycles after the decoder outputs N initial solutions, adopts a tournament selection strategy to retain top-k (k=N / 4) high-quality solutions as parents, and the mutation operation includes a 2-opt local optimization with a probability of 0.6, a node exchange mutation with a probability of 0.3, and a path segment inversion with a probability of 0.1, and the crossover operation adopts a sequential crossover OX operator to retain high-quality path segments;Reverse strategy search is performed on the evolved and mutated solutions, the log probability of each solution in the strategy network is calculated, the node selection gradient is obtained through automatic differentiation, the importance sampling weight with a temperature parameter T=0.1 is constructed, the double buffering mechanism of the elite pool and the evolution pool is established, and the mixing ratio of the two pools is controlled through the KL divergence threshold D_KL<0.05.

[0027] The improved POMO algorithm-based combination optimization system designed by the application comprises four modules, namely a map construction module, a model construction module, an algorithm optimization module and a path planning module.

[0028] The map construction module is used to construct path nodes of a TSP path planning problem as an optimized map;The model construction module is used to construct an encoder-decoder model structure based on an attention mechanism and a REINFORCE algorithm;The algorithm optimization module is used to improve the baseline strategy of a policy gradient algorithm and combine a genetic algorithm plug-in to optimize the POMO algorithm;The path planning module is used to use the final model to complete TSP problem path planning.

[0029] The advantages of the application are as follows:

[0030] On the basis of the traditional POMO algorithm, the application improves the convergence speed and solution quality of the algorithm by improving the baseline strategy, introducing a genetic algorithm plug-in, and adopting a parallel decoding mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the application, the following briefly introduces the drawings used:

[0032] Figure 1 Fig. 1 is a method flowchart of an embodiment of the application;

[0033] Figure 2 Fig. 2 is a schematic diagram of an optimized map;

[0034] Figure 3 : Encoder-decoder model structure schematic diagram;

[0035] Figure 4 : Improved POMO algorithm flowchart schematic diagram;

[0036] Figure 5 : Evolutionary algorithm plug-in optimization flowchart schematic diagram;

[0037] Figure 6 : Parallel decoding mechanism schematic diagram. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0040] Embodiment one

[0041] As shown in the method flowchart schematic diagram of the present embodiment, the steps include: Figure 1

[0042] Step 1. Construct an optimized map for TSP path planning

[0043] As shown in the method flowchart schematic diagram of the present embodiment, the steps include: Figure 2 A two-dimensional space map is constructed, and a Euclidean plane coordinate system of a unit square region [0, 1] x [0, 1] is adopted. Node coordinates are generated by uniform random sampling. In order to avoid too dense nodes, a minimum distance constraint is applied, and the distance is greater than or equal to 0.02 unit length. The system presets four kinds of node scale configurations: a 20-node benchmark set, a 50-node standard set, a 100-node stress test set and a 200-node limit verification set. All node initial states are isolated points, and no adjacency relationship is preset. The map boundary adopts a periodic processing mechanism, and when the node coordinates exceed the boundary, it is automatically wrapped to the opposite side. The Euclidean distance between nodes is calculated in real time to form a distance matrix, and any prior clustering structure or spatial partition mode is prohibited to be introduced to ensure that the algorithm is optimized without prior knowledge.

[0044] Step 2. Construct an encoder-decoder model structure based on attention mechanism and REINFORCE algorithm

[0045] ​An encoder-decoder model structure is constructed using an attention mechanism and a Monte Carlo-based policy gradient algorithm (REINFORCE). As shown in Figure 3 The POMO model framework adopts an encoder-decoder structure of the Transformer architecture. The encoder part contains 8 heads of multi-head self-attention mechanisms, each with a dimension of 64. The multi-head self-attention mechanism allows the model to capture the correlation between nodes from different representation subspaces. In combination with a learnable position encoding module, the position information of the nodes is integrated into the model, because the order of the nodes is very important for path planning in the TSP problem. The input nodes are mapped to a 128-dimensional feature space through 6 Transformer layers. In each Transformer layer, residual connections and layer normalization techniques are used. The residual connections can solve the gradient vanishing problem in deep network training, and the layer normalization can keep the input distribution of each layer stable and ensure the stability of the training. The GeLU activation function is used, which has better non-linear expression ability and can enhance the learning ability of the model. The decoder uses an autoregressive generation method, and at each time step, the encoder output features are dynamically aggregated through a context query attention layer. In this way, the next node to be visited can be dynamically selected based on the current constructed path information and the feature information of all nodes. At the same time, a binary dynamic mask matrix is maintained to record the nodes that have been visited, preventing repeated access to nodes and ensuring that the generated path is valid.

[0046] step3. Improved optimization POMO algorithm model to get the final path planning model

[0047] step3.1. Sample multiplexing

[0048] Supports parallel decoding mechanism, generates multiple paths for the same node set according to different starting nodes, as shown in Figure 6 Through different random seeds, diverse solutions are generated, which can increase the diversity of samples and improve sample utilization. During training, these diverse samples are used for training, which helps the model to learn a better strategy and improve the performance of the algorithm.

[0049] step3.2. Improved baseline strategy uses REINFORCE algorithm to perform gradient ascent optimization on the encoder-decoder parameters, with the formula Where θ represents the encoder-decoder parameters, J(θ) is the loss function, n represents the number of current batch routes, A i represents the advantage function reward of the i-th route, represents the gradient probability of the current route. The shared baseline value b is the average value of all routes, i.e. By using the average reward of all routes as the baseline, the variance of gradient estimation can be reduced, and the stability of training can be improved. When the reward of a certain route is higher than the average reward, it is considered that the strategy of this route is better, and a positive gradient update is given; when the reward of a certain route is lower than the average reward, it is considered that the strategy of this route needs to be improved, and a negative gradient update is given.

[0050] Step 3.3. Increase the evolutionary algorithm optimization module

[0051] As shown in Figure 5 , the evolutionary algorithm plug-in optimization module starts 2-4 rounds of lightweight evolutionary cycles after the decoder outputs N initial solutions. First, the top-k (k=N / 4) high-quality solutions are reserved as parents using the tournament selection strategy. The tournament selection strategy randomly selects a certain number of individuals from the population, and then selects the individual with the highest fitness as the parent. This method can improve the efficiency and accuracy of selection. The mutation operation includes a probability of 0.6 for 2-opt local optimization, a probability of 0.3 for node exchange mutation, and a probability of 0.1 for path segment inversion. 2-opt local optimization is a commonly used local search method in TSP problems, which optimizes the path length by exchanging two edges in the path; node exchange mutation is a random exchange of two nodes in the path, which increases the diversity of the solution; path segment inversion is a random selection of a segment in the path and inversion, which can also generate new solutions. The crossover operation uses the sequential crossover OX operator to preserve high-quality path segments, which can ensure that high-quality path segments in the parent are preserved in the offspring, improving the efficiency of the evolutionary algorithm. The reverse strategy search is performed on the evolved and mutated solutions, the log probability of each solution in the strategy network is calculated, the node selection gradient is obtained through automatic differentiation, and the importance sampling weight with a temperature parameter T=0.1 is constructed. A dual buffering mechanism of elite pool and evolution pool is established, the elite pool is used to store high-quality solutions in history, and the evolution pool is used to store solutions in the current evolution process. The mixing ratio of the two pools is controlled by the KL divergence threshold D_KL<0.05, when the KL divergence of the two pools is less than the threshold, it means that the solution distribution of the two pools is similar, and mixing can be performed, which can increase the diversity of the solution and avoid the algorithm falling into local optimum.

[0052] Step 4. Use the final model to complete the path planning of TSP problem

[0053] After the above steps, the trained final model is used to plan the path for a given TSP problem. The coordinate information of the nodes is input, the model maps the nodes to feature vectors through the encoder, and the decoder generates the optimal path according to the feature vectors and the improved POMO algorithm, and outputs the node order of the path and the path length. The model parameters of this embodiment are set as follows:

[0054] Encoder multi-head self-attention mechanism: 8 heads, each head dimension 64;

[0055] The number of transformer layers: 6; the feature space dimension: 128;

[0056] Evolution algorithm cycle rounds: 2-4 rounds;

[0057] Tournament selection top-k: k=N / 4;

[0058] Mutation operation probability: 2-opt local optimization 0.6, node exchange mutation 0.3, path segment inversion 0.1;

[0059] Temperature parameter T: 0.1;

[0060] KL divergence threshold D_KL: 0.05.

[0061] As can be seen from the above embodiments, the combination optimization algorithm based on the improved POMO algorithm can effectively solve the TSP problem and improve the efficiency and accuracy of path planning by constructing a reasonable optimization map, designing an efficient encoder-decoder model structure, and improving the optimization algorithm. The results of testing on different sizes of node sets show that the algorithm is superior to the traditional POMO algorithm in convergence speed and solution quality, and has strong practicality and promotional value.

Claims

1. A combined optimization algorithm based on an improved POMO algorithm, comprising the following steps: Construct the path nodes of the traveling salesman problem (TSP) path planning problem as an optimization map; Construct an encoder-decoder model structure based on the attention mechanism and the Monte Carlo-based policy gradient algorithm (REINFORCE); Improve the baseline strategy of the policy gradient algorithm and combine it with the genetic algorithm plug-in to obtain the final model; Use the final model to complete the TSP problem path planning.

2. The combined optimization algorithm based on the improved POMO algorithm according to claim 1, characterized in that: The two-dimensional spatial map uses a Euclidean plane coordinate system with a unit square area of ​​[0,1]×[0,1]. Node coordinates are generated by uniform random sampling and subject to a minimum spacing constraint (≥0.02 unit length). The system presets four node scale configurations: a 20-node baseline set, a 50-node standard set, a 100-node stress test set, and a 200-node extreme verification set. All nodes are initially isolated points and no adjacency relationships are preset. The map boundary uses a periodic processing mechanism, and coordinates automatically wrap around to the opposite side when they exceed the boundary. The distance matrix calculates the Euclidean distance between nodes in real time, and any a priori clustering structure or spatial partitioning pattern is prohibited.

3. The combined optimization algorithm based on the improved POMO algorithm according to claim 1 is characterized in that: The POMO model framework adopts the encoder-decoder structure of the Transformer architecture. The encoder contains an 8-head multi-head self-attention mechanism with a dimension of 64 per head, and is equipped with a learnable position encoding module. The input nodes are mapped to a 128-dimensional feature space through 6 Transformer layers. Residual connections and layer normalization techniques are used to ensure training stability, and the GeLU activation function is used to enhance nonlinear expression capabilities. The decoder adopts an autoregressive generation method. The encoder output features are dynamically aggregated through the context query attention layer at each time step. The computational complexity is O(n 2 ), while maintaining a binary dynamic mask matrix to prevent repeated node access.

4. The combined optimization algorithm based on the improved POMO algorithm according to claim 1 is characterized in that: The optimization methods of POMO include: sampling sample multiplexing, improving baseline strategy, and adding evolutionary algorithm optimization module.

5. The combined optimization algorithm based on the improved POMO algorithm according to claim 4 is characterized in that: It supports parallel decoding mechanism, generates multiple paths for the same node set based on different starting nodes to improve sample utilization; and generates diversified solutions through different random seeds.

6. The combined optimization algorithm based on the improved POMO algorithm according to claim 4 is characterized in that: The REINFORCE algorithm is used to perform gradient ascent optimization on the encoder-decoder parameters. The formula is as follows: Among them, θ represents the encoder decoding parameters, J(θ) is the loss function, N represents the number of routes in the current batch, R(τ i ) represents the reward of the i-th route, ▽ θ logp θ (τ i |s) represents the gradient probability of the current route; In addition, the shared baseline value is the average value of all routes, and the specific formula is as follows: Here, j is the sampling route of the same sample with different starting points.

7. The combined optimization algorithm based on the improved POMO algorithm according to claim 4 is characterized in that: The evolutionary algorithm plug-in optimization module starts 2-4 rounds of lightweight evolutionary cycles after the decoder outputs N initial solutions, adopts a tournament selection strategy to retain the top-k (k=N / 4) high-quality solutions as parents, and the mutation operation includes 2-opt local optimization with a probability of 0.6, node exchange mutation with a probability of 0.3, and path segment reversal with a probability of 0.

1. The crossover operation uses a sequential crossover OX operator to retain high-quality path segments; a reverse strategy search is performed on the evolved and mutated solutions, and the logarithmic probability logp of each solution in the strategy network is calculated. θ (τ i |s), obtain node selection gradient ▽ through automatic differentiation θ logp θ (τ i |s), construct the importance sampling weight w with temperature parameter T = 0.1, establish a dual buffer mechanism of elite pool and evolution pool, and control the mixing ratio of the two pools through the KL divergence threshold D_KL<0.05.

Images