Generalized traveling salesman cutting path planning method and device based on reinforcement learning

Abstract

The invention discloses a generalized traveling salesman cutting path planning method and device based on reinforcement learning. The generalized traveling salesman cutting path planning method comprises the steps that a target layout drawing containing a plurality of part graphs is obtained; extracting a node set formed by all parts and cluster information corresponding to the node set; a cutting path planning problem is converted into a generalized travel salesman problem GTSP solving operation, an environment simulation model is constructed, node information of all parts and cluster information of corresponding nodes are input, and current state information is obtained; according to the current state information, the output strategy of the trained strategy model is input into the environment simulation model to obtain new state information, all part nodes are traversed until the optimal machining sequence result of the cutting sequence of all parts is obtained, a comprehensive path planning scheme is synthesized, and the whole cutting efficiency is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial manufacturing automation, and particularly relates to a generalized traveling salesman cut path planning method and device based on reinforcement learning. BACKGROUND

[0002] With the progress of industrial automation, it is crucial to precisely plan the machining path, which is directly related to the improvement of production efficiency and the reduction of cost. In the manufacturing process, especially in the process of material cutting, how to determine an efficient and economical machining sequence has become a challenge. Usually, this problem is solved in two steps: first, determine all the perforation points that need to be processed, and then solve the classic traveling salesman problem (TSP), that is, find the shortest possible path that passes through all these points. But this method of first determining perforation points may limit the optimal choice of cutting path.

[0003] To solve this problem, we can combine the determination of perforation points and the traveling salesman problem into one problem, and regard it as a more general problem, the generalized traveling salesman problem (GTSP). The current method for solving GTSP in the processing industry scene is mainly traditional algorithms such as dynamic programming algorithm, ant colony algorithm, genetic algorithm, simulated annealing algorithm, tabu search algorithm, etc. But these methods also have significant drawbacks. First, they may encounter convergence difficulties, especially when dealing with complex and variable manufacturing environments. Second, the calculation process of these methods is very time-consuming. For batch part graph path planning, it may take hours or even a whole day to complete the task, which is unacceptable in a production environment that requires quick response. Finally, due to the limitations of these methods, they are difficult to ensure that a globally optimal solution can be obtained in all cases.

[0004] The method using deep learning technology, such as graph neural network and pointer network, etc., these are mainly to solve TSP, due to the complexity of GTSP, there is no related network architecture to solve GTSP. Moreover, these methods are a kind of supervised learning method, which needs a large amount of data and labels, which may be more difficult to achieve in practical applications. Even if there are these data, due to the reason of supervised learning, the model can only learn to the level of these known solutions, and cannot guarantee to find the real optimal solution.

[0005] However, the current attention mechanism-based reinforcement learning method has great limitations in solving path planning problems. On the one hand, in industrial scenarios, the number of cut part graphs is uncertain, and the number of points in each part graph is also variable. Therefore, we need a flexible model to handle these uncertainties. The original attention mechanism-based reinforcement learning method has a fixed number of nodes. If we need to handle different numbers of nodes, we need to retrain the model, which is obviously not suitable for industrial scenarios. On the other hand, this method can only be used to specifically design to solve the traveling salesman problem (TSP), and cannot solve the generalized traveling salesman problem (GTSP). In the planning of cutting paths, we need to select a point in different layout graphs as a cutting point to plan the path. Each layout graph is a cluster, and the points in the cluster are the points in the layout graph. This problem is not the traditional traveling salesman problem (TSP), but the generalized traveling salesman problem (GTSP), which cannot be directly used for cutting path planning. SUMMARY

[0006] The purpose of the present application is to provide a generalized traveling salesman cutting path planning method and device based on reinforcement learning, which combines the mask mechanism-based processing path planning method based on reinforcement learning, further optimizes the cutting path and improves the planning efficiency. Based on the mask mechanism in natural language processing, it can effectively handle variable-length sequences. By incorporating the mask mechanism into the model, the model can flexibly adapt to different numbers of nodes, thereby overcoming the limitations of the prior art. Such improvements not only improve the model's ability to handle problems of different scales, but also greatly enhance its applicability and efficiency in various industrial manufacturing environments.

[0007] The present application provides a generalized traveling salesman cutting path planning method based on reinforcement learning, comprising:

[0008] Obtaining a target layout graph containing a plurality of part graphs;

[0009] Extracting a node set composed of all parts and cluster information corresponding to the node set from the target layout graph, the node set being node information of all parts, and the cluster information being all nodes of the same part;

[0010] Traversing all cluster information and selecting a target node in each part, the target node being a node in each part that can form the shortest path through all node sets;

[0011] The cutting path planning problem is converted into a generalized traveling salesman problem (GTSP) solution operation, an environment simulation model is constructed, and node information of all parts and cluster information of corresponding nodes are input to obtain current state information, which at least includes part node information, part cluster information, node mask information, and current node information;

[0012] The output strategy of the trained strategy model is input into the environment simulation model according to the current state information to obtain new state information, and all part nodes are traversed until the best processing order result of the cutting order of all parts is obtained.

[0013] Preferably, the training step of the trained strategy model comprises:

[0014] Randomly generating node position coordinates and corresponding cluster information as a training data set;

[0015] The state information of the training data set is input into a preset strategy model by randomly selecting a path, and each node is randomly selected based on a probability distribution to obtain a first path order result of part cutting;

[0016] The state information of the training data set is input into a preset reference model, and the node with the highest selected probability in each node is selected by using a greedy algorithm to obtain a second path order result of part cutting;

[0017] The first path order result and the second path order result are compared, and the preset strategy model parameters are optimized and updated or replaced by the preset reference model parameters according to the comparison result.

[0018] Preferably, the preset strategy model parameters are optimized and updated or replaced by the preset reference model parameters according to the comparison result further comprises:

[0019] If The parameter θ BL is updated, otherwise it is not updated;

[0020] The current cutting path planning strategy is updated based on the gradient calculated by the loss function, and the formula is as follows:

[0021]

[0022] That is, by comparing the performance of the current preset strategy model and the performance of the preset reference model, an evaluation result is obtained, and the selection of the current cutting path planning strategy is further optimized according to the evaluation result;

[0023] Wherein, L(π i ) represents the path length output by the preset strategy model, represents the path length of the preset reference model output, θ represents the parameters in the preset strategy model, and p represents the current strategy π i The selected probability of each node in the cluster.

[0024] As preferred, the operation of converting the cutting path planning problem into a generalized traveling salesman problem GTSP and constructing an environment simulation model further comprises:

[0025] Input all node information and cluster information into the environment simulation model, automatically mask fill the data in the node position information and cluster information that do not meet the pre-defined length, and mark the node mask information;

[0026] Obtain state information from the environment simulation model and input it into the preset strategy model, obtain the probability of each node being selected in the cluster, and randomly output the path order of the target node;

[0027] Obtain state information from the environment simulation model and input it into the preset reference model, select the node with the maximum selection probability in each node in the cluster by the greedy method, and output the path order of the target node;

[0028] Update the simulation environment after each target node selection is completed, and if the current cluster has selected the target node, all nodes in the current cluster are marked as not to be accessed again;

[0029] Repeat the above steps of outputting the best path order of the target node until the complete target layout is traversed to select the target node path, and output the path order of the target node from the preset strategy model and the preset reference model respectively;

[0030] According to the path length of the two strategies calculated according to the path order of the target node output by the two models, update the parameters between the two models by comparing the length of the path, and finally update the model according to the direction of the output path length of the cutting path planning strategy selection model.

[0031] As preferred, the environment simulation model comprises:

[0032] Node position information, including the position information of all nodes, if the actual number of nodes is less than the maximum number of nodes, mask is used for filling to reach the preset number of nodes to ensure the same data dimension in the training process;

[0033] Cluster information, used to divide nodes belonging to the same part into the same cluster, and the cluster saves the coordinate index information belonging to the same part, and when the number of nodes in each part is different, mask is used for filling to keep the same dimension;

[0034] Node mask information, used to mark the state of whether each node can be accessed;

[0035] Current node information is used to mark the position of the current node, and the next target node is selected based on the current node information;

[0036] The state transition equation is used to update the environment state based on the current state information and the policy.

[0037] The policy gradient equation is used to output the path length of the current policy and update the model parameters. The current policy is the optimized and updated policy model.

[0038] Preferably, the network architecture of both the preset strategy model and the preset baseline model includes an encoder and a decoder, wherein the encoder is composed of multiple identical layers stacked together, and the encoding steps include:

[0039] The current state information is input into the encoder, i.e., input from the linear embedding layer, and the current state information is mapped to a high-dimensional space to obtain the embedded feature vector.

[0040] The embedded feature vector is input into the first attention layer, the relationship between the input features is calculated, the interaction between different perforation points is captured, and the first feature matrix is ​​output.

[0041] The output of the first attention layer is added to the input, and then the first normalization operation is performed. The feature vector after the first normalization is output to eliminate the difference in the units of different features.

[0042] The feature vector after the first normalization is input into the feedforward network for further nonlinear transformation. Then, the output of the feedforward network is added to the input, and a second normalization operation is performed to output the encoded target vector.

[0043] Preferably, the decoding steps of the decoder include:

[0044] The encoded target vector is used as the input to the decoder, i.e., input from the second attention layer, to calculate the relationship between the input features, capture the interaction between different perforation points, and output the second feature matrix;

[0045] The second feature matrix is ​​input into the linear transformation layer, and the output of the multi-head attention layer is merged into a single vector.

[0046] The single vector is input into the third attention layer, and the single vector is used as the query vector. Each node vector is used as the key vector. The query vector and the key vector are multiplied to obtain the attention score of each node.

[0047] The output of the third attention layer is used as the input of the mask layer, and the nodes that have been visited among all nodes are filtered according to the node mask information in the environment simulation model.

[0048] The attention score, filtered by the mask layer, is input into the Softmax layer and transformed into a probability distribution through the Softmax function. The final output is the mapping of the behavior policy from the current state s to action A, that is, the probability of finally selecting the punch point.

[0049] This invention provides a reinforcement learning-based generalized traveling salesman path planning device, comprising:

[0050] The data acquisition module is used to acquire a target layout diagram containing multiple part graphics;

[0051] The target selection module is used to extract the node set consisting of all parts and the cluster information corresponding to the node set according to the target layout diagram. The node set is the node information of all parts, and the cluster information is all nodes of the same part.

[0052] The GTSP conversion module is used to convert the cutting path planning problem into a generalized traveling salesman problem (GTSP) solution operation, build an environmental simulation model, and input the node information of all parts and the corresponding node cluster information to obtain the current state information, which includes the node information of each part, the part cluster information, the node mask information, and the current node information.

[0053] The strategy output module is used to output the strategy of the trained strategy model based on the current state information, input the environment simulation model to obtain new state information, traverse all part nodes until the optimal processing sequence result of cutting all parts is obtained.

[0054] The present invention also provides an electronic device, comprising:

[0055] The memory is used to store the processing program;

[0056] The processor, when executing the processing program, implements the reinforcement learning-based generalized traveling salesman cutting path planning method as described in the embodiments of the present invention.

[0057] The present invention also provides a computer-readable medium storing a computer program that, when executed by one or more processors, implements the reinforcement learning-based generalized traveling salesman path planning method as described in the embodiments of the present invention.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] This invention reduces the traditional machining path planning to a generalized travel problem solution. It eliminates the need to predetermine the perforation points; simply inputting a layout diagram containing node and cluster information into the model yields the planned machining path. This overcomes the problem in TSP (Travel Problem Solving) where predetermining perforation points restricts the optimal selection of the cutting path. This invention utilizes a GTSP problem transformation method to comprehensively consider the point set composed of all parts on the machining board, selecting an optimal cutting start point for each part and determining the cutting order. This provides a more comprehensive path planning scheme, significantly improving the efficiency of the entire cutting task.

[0060] This invention proposes a reinforcement learning-based processing path planning method that incorporates a masking mechanism. Inspired by the masking mechanism in natural language processing, this method effectively handles sequences of variable length. By integrating the masking mechanism into the model, it allows for flexible adaptation to different numbers of nodes, overcoming the limitations of existing technologies. This improvement not only enhances the model's ability to handle problems of varying scales but also significantly improves its applicability and efficiency in various industrial manufacturing environments. Furthermore, this method addresses the issues of long computation times and susceptibility to local optima in traditional algorithms. Once trained, the model solves new problems very quickly in practical applications, even for thousands of processing path planning graphs, providing results almost instantly. Simultaneously, this method also addresses the issues of data annotation and the inability to guarantee finding better solutions in traditional deep learning. The self-guided learning process of reinforcement learning in this method endows the model with the ability to transcend the limitations of initial training data, enabling it to move towards the true optimal solution. Therefore, this invention represents a significant technical optimization of existing methods, providing a more flexible and efficient solution to the processing path planning problem.

[0061] This invention integrates masking technology with attention-based reinforcement learning methods to achieve the effect of handling a variable number of processing points, enabling the model to flexibly handle processing tasks of different scales without retraining the model for each different task volume.

[0062] This invention does not rely on externally provided training datasets or pre-labeled data. Instead, it is able to generate all the data required for the training process itself.

[0063] This invention introduces a path-cutting planning strategy model, which generates strategies by randomly selecting paths, thus effectively avoiding the problem of generating only local optima. Furthermore, it collaborates with a baseline model, continuously adjusting and optimizing the strategy by comparing the paths generated by each model. Attached Figure Description

[0064] Figure 1This is a schematic diagram illustrating the steps of the reinforcement learning-based generalized traveling salesman path planning method in one embodiment of the present invention;

[0065] Figure 2 This is a target layout diagram in one embodiment of the present invention;

[0066] Figure 3 This is an example diagram illustrating the solution of the Generalized Traveling Salesman Problem (GTSP) in one embodiment of the present invention;

[0067] Figure 4 This is a flowchart of the entire model training process under the generalized traveling salesman cutting path planning based on reinforcement learning in one embodiment of the present invention;

[0068] Figure 5 This is a network architecture diagram of the encoder and decoder in one embodiment of the present invention;

[0069] Figure 6 This is a flowchart of the model inference process after training in one embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0072] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0073] It should be noted that the terms "a" and "a plurality of" used in this application disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0074] Example 1

[0075] With the advancement of industrial automation, precise planning of processing paths has become crucial, directly impacting production efficiency and cost reduction. In manufacturing processes, especially in material cutting, determining an efficient and economical processing sequence presents a challenge. Typically, this problem is solved in two steps: first, identifying all the piercing points to be processed, and then solving the classic Traveling Salesman Problem (TSP) to find the shortest possible path across all these points. However, this method of pre-determining piercing points can limit the optimal choice of cutting path.

[0076] To address this problem, we can summarize the determination of piercing points and the Traveling Salesman Problem into a single broader problem—the Generalized Traveling Salesman Problem (GTSP). Unlike the TSP, the GTSP doesn't find the shortest path through all points. Instead, it selects a point from a group of related points—the cutting points of each part—to form the overall shortest path. For example, it's not about choosing a travel route between multiple attractions in a city, but rather selecting one attraction from each of several cities to plan an overall travel route. In industrial cutting path planning, this means considering the set of points formed by all parts on the entire machining board, selecting an optimal starting point for cutting each part, and finally determining the cutting order. The GTSP provides a more comprehensive path planning solution, which is expected to significantly improve the efficiency of the entire cutting task.

[0077] like Figure 1 As shown, this invention provides a generalized traveling salesman path planning method based on reinforcement learning, comprising:

[0078] S1: Obtain the target layout drawing containing multiple part graphics;

[0079] S2: Extract the node set of all parts and the cluster information corresponding to the node set based on the target layout diagram. The node set is the node information of all parts, and the cluster information is all nodes of the same part. In this embodiment, cluster points are generated through pattern generation: all nodes in a pattern are treated as a cluster, and the optimal perforation point is obtained from the cluster using masking technology. In this scheme, if nodes are not used as perforation points, regular points can be preset and treated as cluster points in the same way.

[0080] S3: Transform the cutting path planning problem into a Traveling Salesman Problem (GTSP) solution operation, construct an environmental simulation model, and input the node information of all parts and the corresponding node cluster information to obtain the current state information, which includes the node information of each part, the part cluster information, the node mask information, the current node information, etc.

[0081] S4: Based on the current state information, input the output strategy of the trained strategy model into the environment simulation model to obtain new state information, traverse all part nodes until the optimal processing sequence result of cutting all parts is obtained.

[0082] In various embodiments of this application, the environmental simulation model includes: node location information, including the location information of all nodes; if the actual number of nodes is less than the maximum number of nodes, a mask is used to fill the gaps to achieve a preset number of nodes to ensure the same data dimension during training; cluster information, used to group nodes belonging to the same part into the same cluster, where the coordinate index information of the same part is stored; when the number of nodes in each part is different, a mask is used to fill the gaps to maintain the same dimension; node mask information, used to mark whether each node is accessible; current node information, used to mark the location information of the current node, and to select the next target node based on the current node information; and a state transition equation, used to determine the current state information. The information and policy update environment state is to mark the node mask information of all nodes in the cluster that have been visited as 0, indicating that they have been visited and will not be visited again. The policy gradient equation is used to output the path length of the current policy and update the model parameters. The policy is an optimized and updated policy model. In this embodiment, optimizing and updating the policy model means optimizing and updating the policy model parameters or replacing the policy model parameters with the baseline model. That is, a complete round of training is that the input data policy model uses a random sampling method to obtain the path order, waits for the policy model to output all the path orders, and the baseline model uses a greedy method to obtain the path order. The two results of the baseline model and the policy model are compared, and the policy model parameters are optimized and updated, or the policy model parameters are replaced with the baseline model.

[0083] See Figure 2As shown, this embodiment uses multiple different layout patterns. The cutting order of these patterns differs, requiring consideration of which pattern to cut first to determine the cutting path and minimize the cutting path. The determination of the perforation point and the Traveling Salesman Problem (TSP) can be summarized into a broader problem—the Generalized Traveling Salesman Problem (GTSP). The GTSP expands upon the traditional Traveling Salesman Problem, no longer seeking the shortest path connecting all city points without repetition. Instead, it expands the concept of city points to the concept of city point sets. That is, it seeks a city point from each city point set, ensuring that all city point sets are visited exactly once, and finding the shortest path that passes through a single point in each city point set. In the cutting process path planning problem, a part can be understood as a city point set, and the points within the part can be understood as city nodes within those city point sets. The node set formed by the node coordinates on each ring can be regarded as a cluster. We need to select a node from the node set formed by each part and plan the cutting order of each point.

[0084] The core of this embodiment lies in solving the Generalized Traveling Salesman Problem (GTSP). The specific implementation steps are as follows: Input all node information and cluster information into the environmental simulation model; automatically mask any data in the node location information and cluster information that does not meet the predefined length, and mark the node mask information; obtain state information from the environmental simulation model and input it into the preset strategy model to obtain the probability of each node in the cluster being selected and randomly output the path order of the target node; obtain state information from the environmental simulation model and input it into the preset baseline model; select the node with the highest probability of being selected among all nodes in the cluster using a greedy algorithm and output the target node. The path order of nodes; after each target node is selected, the simulation environment is updated. If the current cluster has already selected a target node, all nodes in the current cluster are marked as not to be visited again; repeat the above steps of outputting the best path order of target nodes until the entire target layout diagram is traversed and the target node path is selected, and output the path order of target nodes from the preset strategy model and the preset baseline model respectively; calculate the path length of the two strategies according to the path order of target nodes output by the two models, compare the path length between the two models and update the model parameters, and finally update the model selected by the cutting path planning strategy according to the direction of the model with the smallest output path length.

[0085] See Figure 3As shown, nodes V1-V32 and their cluster information are input into the model. The environment simulation model constructs the environment based on the input node and cluster information. This includes node position information (the location of each node), node mask information (used to mark whether each node can be accessed), and cluster information (marking which nodes belong to a cluster). Since the attention mechanism matrix multiplication requires data of the same dimension for calculation, the data needs to be padded. Node position information and cluster information that do not meet the predefined length are padded with a mask. After the environment is constructed, the state information 's' is input into the model to output the strategy, i.e., the node to be selected. The environment information is updated based on the currently selected node. If a node in a cluster has already been selected (i.e., a target node of the part has been selected), all nodes in the current cluster should be marked as inaccessible again; that is, the corresponding node in the node mask information should be marked as 0. At the end of the entire graph path planning, the total length is calculated based on the node selection strategy of the entire graph. By comparing the total length updates of the baseline model and the strategy model, the model always moves in the direction of generating the shortest path length. That is, the model takes V1-V32 nodes and the cluster information of the nodes as input and outputs the traversal sequence of nodes V3-V7-V11-V13-V18-V25-V32.

[0086] In various embodiments of this application, the training steps of the trained strategy model include: randomly generating node position coordinates and corresponding cluster information as a training dataset; the training dataset used in this embodiment is an automatically generated training data mechanism, which does not rely on externally provided training datasets or pre-labeled data. Instead, it can automatically generate all the data required for the training process. The state information of the training dataset is input into the preset strategy model, and a path is randomly selected based on the probability distribution of each node to obtain the first path order result L(π) for part cutting. i The state information of the training dataset is input into a preset benchmark model, and a greedy algorithm is used to select the node with the highest selection probability in each node to obtain the second path sequence result of part cutting. For example, the probability of selecting node 1 is 50%, node 2 is 20%, and node 3 is 30%. In the baseline model, the output would be node 1, which has the highest probability. However, the strategy model is like a large wheel, where the probability of selecting node 1 is relatively high, while the probability of selecting node 2 is relatively low. Each node has a chance of being selected.

[0087] Compare the first path order result with the second path order result, and optimize and update the preset strategy model parameters based on the comparison result, or replace the preset strategy model parameters with the preset baseline model. If Then update parameter θ BL =θ, otherwise no update; this can be understood as follows: if the policy model is better, the baseline model will replicate the parameters of the policy model; if the baseline model is optimal, the policy model will update its parameters in the direction of shorter paths. The gradient is calculated based on the loss function to update the current cutting path planning policy, as shown in the following formula:

[0088]

[0089] That is, by comparing the performance of the current preset strategy model with the performance of the preset benchmark model, the evaluation result is obtained, and the selection of the current cutting path planning strategy is further optimized based on the evaluation result;

[0090] Where L(π) i The ) indicates the path length output by the preset strategy model. The path length output by the preset baseline model is represented by θ, the parameters in the preset strategy model are represented by θ, and the current strategy π is represented by p. i The probability of.

[0091] The model training principle is explained as follows: See the flowchart of the model training process in this embodiment. Figure 4 As shown, it includes the following steps:

[0092] 1. Construct an environmental simulation model based on the point sets formed by all parts on the machining board and the corresponding cluster information. The environmental simulation model includes: Node position information: Representing the position information of all nodes. Since the data dimension must be the same during training, if different numbers of nodes are required, a mask must be added to fill the gaps to reach a specified number. For example, a maximum number of nodes can be specified, such as 512, or other appropriate values. If the actual number of nodes is less than 512, a mask is used to fill the gaps to reach the specified number. Node mask information: Marks whether each node is accessible, where 0 indicates inaccessible and 1 indicates accessible. Cluster information: Nodes belonging to the same part are grouped into the same cluster. The cluster stores the coordinate index information of the same part. Since the number of nodes in each part may not be the same, a mask is used to fill the gaps to maintain the same dimension. A maximum number of nodes in the cluster can be specified, such as 10 nodes, or other values. If the number of nodes in the cluster is insufficient, a mask is used for filling. Since the number of parts varies across different cut patterns, the number of clusters also varies. During training, a maximum cluster number can be specified, such as 50 clusters or other suitable values. If the actual number of cut patterns is less than 50, all nodes in that cluster are filled with a mask. Current node information: Marks the position of the current node, used to select the next node based on this information. State transition equation: Primarily used to update the environment state. Policy gradient equation: Primarily used to output the path length of the current policy, used to update model parameters later.

[0093] 2. Based on the state transition equations in the environmental simulation model, obtain the current state (`state`), which includes information on each component node, component cluster, node mask, and current node. Input the current state into the strategy model and the baseline model, representing the path output by the baseline model and the path output by the strategy model, respectively. The strategy model and the baseline model aim to effectively handle path planning problems and improve performance. These two models have similar structures, both containing encoders and decoders to process instances and generate paths. Their difference lies in the path selection strategy. Strategy Model: The strategy model solves the problem by randomly selecting paths based on the probability distribution of the paths. This allows the model to explore and consider multiple possible paths, rather than relying solely on a greedy strategy. Through randomness, the strategy model can better handle the uncertainty of the problem and avoid local optima. Baseline Model: The baseline model uses a greedy strategy, selecting the action or path with the highest probability at each step. This strategy simplifies the path selection process and usually leads to higher computational efficiency, although it may sacrifice some global performance. Policy models help improve global performance by exploring multiple paths, while benchmark models provide a reasonable performance baseline to help evaluate and guide the learning process of policy models, making policy models easier to train.

[0094] 3. Input the model policy into the environment simulation model, and update the environment state (state) using the state transition equation. If a node has been visited, all nodes in the cluster to which the current node belongs need to be masked. Repeat step 2 to obtain the path policy under the current state. Calculate the path length of the current policy using the policy gradient equation, which will be used to update the parameter (θ) in the baseline model and the policy model later.

[0095] 4. If Then update parameter θ BL =θ, otherwise no update. This is because when the path generated by the policy model is shorter and performs better than the path generated by the baseline model, we update the parameters of the baseline model using the parameters of the policy model, so that the better-performing policy model can be used for future path generation. If the path generated by the policy model is longer, we do not update the parameters to prevent performance degradation. This helps to gradually improve the performance of the policy model.

[0096] 5. This function represents the calculation of the gradient, i.e., the definition of the loss function, which is accomplished by comparing the performance of the current policy with that of the greedy policy. If the current policy performs better, the gradient will encourage the policy to update in that direction to improve performance. If the greedy policy performs better, the gradient will encourage the policy to update in the direction of the greedy policy to better explore the policy space.

[0097] 6. Steps 4 and 5 can be understood as follows: if the path randomly selected by the policy model is longer than the path derived by the baseline model using a greedy strategy, it indicates that the path explored by the policy model is effective. In this case, the policy responsible for generating that path in the policy model will be strengthened, meaning that the model is more likely to take similar actions when encountering similar situations in the future. Conversely, if the path generated by the policy model is inferior to the path of the baseline model, then the relevant policy will be weakened. This mechanism ensures that the model not only finds satisfactory solutions but also learns and continuously optimizes its decision-making process during exploration. Through such a feedback loop, the model can gradually learn and improve its path planning strategy, thereby achieving better path planning results in subsequent processing tasks.

[0098] In this implementation, both the preset strategy model and the preset baseline model configuration network architecture include neural networks with encoders and decoders. See [link to relevant documentation]. Figure 5 As shown in the figure, the input is the instance information constructed in the state transition equation, including information about each node, each cluster, and mask information. The output is the behavioral strategy, which is the mapping from state s to action A, i.e., the order in which a perforation point is automatically selected in each part, and this order is the optimal processing sequence.Figure 5 The area enclosed by the dashed lines shown can have N layers, for example, N = 3. This indicates that there can be N dashed-line enclosed areas for feature extraction. The specific encoder is described as follows: The current state information is input into the encoder, i.e., from the linear embedding layer. The current state information is mapped to a high-dimensional space to obtain an embedded feature vector. The embedded feature vector is input into the first attention layer to calculate the relationship between input features, capture the interaction between different perforation points, and output a first feature matrix. The output of the first attention layer is added to the input, and a first normalization operation is performed, outputting a first normalized feature vector to eliminate dimensional differences between different features. The first normalized feature vector is input into the feedforward network for further nonlinear transformation operations. The output of the feedforward network is then added to the input, and a second normalization operation is performed to output the encoded target vector. In this embodiment, the encoder input: the input state s includes: information for each node, information for each cluster, and mask information.

[0099] The specific encoder and decoder are described as follows: The encoded target vector is used as the input to the decoder, i.e., input from the second attention layer. The relationship between input features is calculated to capture the interaction between different punch points, and the second feature matrix is ​​output. The second feature matrix is ​​input to the linear transformation layer, and the outputs of the multi-head attention layer are merged into a single vector. The single vector is input to the third attention layer, where it is used as the query vector, and each node vector is used as the key vector. The query vector is multiplied by the key vector to obtain the attention score of each node. The output of the third attention layer is used as the input to the mask layer, and the nodes that have been visited are filtered according to the node mask information in the environmental simulation model. The attention score after being filtered by the mask layer is input to the Softmax layer, and it is transformed into a probability distribution through the Softmax function. The final output is the mapping of the behavior policy from the current state s to action A, i.e., the probability of finally selecting a punch point.

[0100] Those skilled in the art will understand that the encoder structure used in this embodiment is as follows:

[0101] Linear Embedding Layer: Maps the input state s to a high-dimensional space to obtain the embedding vector.

[0102] The first attention layer is a multi-head attention mechanism: it calculates the relationship between input features, enabling the model to capture the interaction between different perforation points.

[0103] Add & Normalize: The output of the attention layer is added to the input and then normalized to stabilize the learning process and prevent network degradation.

[0104] Feedforward Network: Performs further nonlinear transformations on the output of the attention layer to improve the model's expressive power.

[0105] The encoder structure used in this embodiment is as follows:

[0106] The second attention layer used in this implementation is a multi-head attention mechanism: it accepts the output of the encoder and the output of the decoder in the previous step, calculates the relationship between the input features, and enables the model to capture the interaction between different punch points.

[0107] Linear Transformation: Combines the outputs of the multi-head attention layer into a single vector.

[0108] The third attention layer used in this implementation is a single-head attention mechanism: the merged vector is used as the query (q), each node vector is used as the key (k), and the query (q) is multiplied by the key (q) to obtain the attention score of each node.

[0109] Masking layer: Based on the node masking information in the environment simulation model, the attention score of inaccessible nodes is set to negative infinity to prevent the model from selecting these nodes.

[0110] Softmax layer: The attention score is transformed into a probability distribution using the Softmax function, which helps the model decide which node to select in the next step.

[0111] The cutting planning method described in this embodiment involves the model reasoning process. (See [link]). Figure 6 As shown, this is the process of feeding a layout diagram instance containing node and cluster information into the model after training to obtain the optimal path sequence. This process is very straightforward: simply inputting a layout diagram instance containing node and cluster information allows the model to quickly calculate the optimal processing order for each part. The model can handle changes in the number of nodes or parts themselves. Furthermore, it can process batch layout diagrams simultaneously, with all computation time measured in seconds, significantly accelerating the entire path planning process. Black dots represent the tool initialization position, and dashed lines represent the tool's empty-form movement path. In this embodiment, the Generalized Traveling Salesman Problem (GTSP) is a combinatorial optimization problem that requires selecting one point from each of a set of points and finding the shortest possible path connecting these points. Masking: In machine learning, masking techniques are used to filter out unnecessary parts of a sequence, allowing the model to focus on important information.

[0112] Example 2

[0113] Based on the same concept, this invention provides a reinforcement learning-based generalized traveling salesman path cutting planning device, comprising:

[0114] The data acquisition module is used to acquire a target layout diagram containing multiple part graphics;

[0115] The target selection module is used to extract the node set of all parts and the cluster information corresponding to the node set based on the target layout diagram. The node set is the node information of all parts, and the cluster information is all nodes of the same part. The GTSP conversion module is used to convert the cutting path planning problem into a generalized traveling salesman problem (GTSP) solution operation, build an environment simulation model, and input the node information of all parts and the cluster information of the corresponding nodes to obtain the current state information. The current state information includes the node information of each part, the part cluster information, the node mask information, and the current node information.

[0116] The strategy output module is used to output the strategy of the trained strategy model based on the current state information, input the environment simulation model to obtain new state information, traverse all part nodes until the optimal processing sequence result of cutting all parts is obtained.

[0117] It should be noted that the division of the various modules in this device / system embodiment is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can also be implemented entirely in hardware; or some units can be implemented by processing element calls to software, while others can be implemented in hardware.

[0118] The implementation principles of the data acquisition module, target selection module, GTSP conversion module, and strategy output module have been described in the foregoing embodiments, and therefore will not be repeated here.

[0119] Example 3

[0120] Based on the same concept, some embodiments of this application also provide an electronic device. This electronic device includes a memory and a processor, wherein the memory stores a processing program, and the processor executes the processing program according to instructions. When the processor executes the processing program, the reinforcement learning-based generalized traveling salesman path planning method described in the foregoing embodiments is implemented. For example, a graphics card (GPU) primarily accelerates the training process during model training, and loading the model for prediction onto the GPU also accelerates the prediction process.

[0121] In some embodiments of this application, a readable storage medium is also provided, which can be a non-volatile readable storage medium or a volatile readable storage medium. The readable storage medium stores instructions that, when executed on a computer, cause an electronic device containing such a readable storage medium to perform the aforementioned reinforcement learning-based generalized traveling salesman path planning method.

[0122] It is understood that, for the aforementioned reinforcement learning-based generalized traveling salesman path planning methods, if they are all implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0123] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0124] The program code for executing the technical solutions disclosed in this application can be written in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages—such as Python and C++—and conventional procedural programming languages—such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generalized traveling salesman path planning method based on reinforcement learning, characterized in that, include: Obtain the target layout diagram containing multiple part graphics; Extract the node set consisting of all parts and the cluster information corresponding to the node set based on the target layout diagram. The node set is the node information of all parts, and the cluster information is all nodes of the same part. The cutting path planning problem is transformed into a generalized traveling salesman problem (GTSP) solution operation, and an environmental simulation model is constructed. The node information of all parts and the cluster information of the corresponding nodes are input to obtain the current state information. The current state information includes at least the node information of each part, the cluster information of the parts, the node mask information, and the current node information. Based on the current state information, the output strategy of the trained strategy model is input into the environment simulation model to obtain new state information, and all part nodes are traversed until the optimal processing sequence result of cutting all parts is obtained.

2. The reinforcement learning-based generalized traveling salesman path planning method according to claim 1, characterized in that, The training steps for the trained policy model include: Randomly generate node location coordinates and corresponding cluster information as the training dataset; The state information of the training dataset is input into the preset strategy model. The path is randomly selected based on the probability distribution of each node to obtain the first path order result of the part cutting. The state information of the training dataset is input into the preset benchmark model, and a greedy algorithm is used to select the node with the highest selection probability in each node to obtain the second path order result of part cutting. The first path order result is compared with the second path order result. Based on the comparison result, the parameters of the preset strategy model are optimized and updated, or the parameters of the preset strategy model are replaced with the preset benchmark model.

3. The reinforcement learning-based generalized traveling salesman path planning method according to claim 2, characterized in that, The step of optimizing and updating the preset strategy model parameters based on the comparison results, or replacing the preset strategy model parameters with the preset benchmark model, further includes: if Then update parameter θ BL =θ, otherwise do not update; The current cutting path planning strategy is updated based on the gradient calculated using the loss function, as shown in the following formula: That is, by comparing the performance of the current preset strategy model with the performance of the preset benchmark model, the evaluation result is obtained, and the selection of the current cutting path planning strategy is further optimized based on the evaluation result; Where L(π) i The ) indicates the path length output by the preset strategy model. The path length output by the preset baseline model is represented by θ, the parameters in the preset strategy model are represented by θ, and the current strategy π is represented by p. i The probability of selection.

4. The reinforcement learning-based generalized traveling salesman path planning method according to claim 1, characterized in that, The step of transforming the path planning problem into a generalized traveling salesman problem (GTSP) and constructing an environmental simulation model further includes: All node and cluster information is input into the environment simulation model. The system automatically fills in the mask for data in the node location and cluster information that does not meet the predefined length, and marks the node mask information. The state information is obtained from the environmental simulation model and input into the preset strategy model to obtain the probability of each node in the cluster being selected and to randomly output the path order of the target nodes. The state information is obtained from the environmental simulation model and input into the preset benchmark model. The node with the highest probability of being selected in each node of the cluster is selected in a greedy manner, and the path order of the target node is output. The simulation environment is updated after each target node is selected. If the current cluster has already selected a target node, all nodes in the current cluster are marked as not to be accessed again. Repeat the above steps to output the optimal path order for the target nodes until the entire target layout diagram is traversed and the target node paths are selected. Output the path order of the target nodes from the preset strategy model and the preset baseline model respectively. The path lengths for the two strategies are calculated based on the path order of the target nodes output by the two models. The path lengths between the two models are compared to update the model parameters. Finally, the model selected by the cutting path planning strategy is updated according to the direction of the model with the shortest output path length.

5. The reinforcement learning-based generalized traveling salesman path planning method according to claim 1, characterized in that, The environmental simulation model includes: Node location information, including the location information of all nodes. If the actual number of nodes is less than the maximum number of nodes, a mask is used to fill the gaps to reach the preset number of nodes and ensure that the data dimensions are the same during training. Cluster information is used to group nodes belonging to the same part into the same cluster. The cluster stores the coordinate index information of the same part. When the number of nodes in each part is different, a mask is used to fill the gaps to maintain the same dimensions. Node mask information is used to mark whether each node is accessible. Current node information is used to mark the position of the current node, and the next target node is selected based on the current node information; The state transition equation is used to update the environment state based on the current state information and the policy. The policy gradient equation is used to output the path length of the current policy and update the model parameters. The current policy is the optimized and updated policy model.

6. The reinforcement learning-based generalized traveling salesman path planning method according to claim 1, characterized in that, Both the preset strategy model and the preset baseline model have network architectures that include an encoder and a decoder. The encoder consists of multiple identical layers stacked together, and the encoding steps include: The current state information is input into the encoder, i.e., input from the linear embedding layer, and the current state information is mapped to a high-dimensional space to obtain the embedded feature vector. The embedded feature vector is input into the first attention layer, the relationship between the input features is calculated, the interaction between different perforation points is captured, and the first feature matrix is ​​output. The output of the first attention layer is added to the input, and then the first normalization operation is performed. The feature vector after the first normalization is output to eliminate the difference in the units of different features. The feature vector after the first normalization is input into the feedforward network for further nonlinear transformation. Then, the output of the feedforward network is added to the input, and a second normalization operation is performed to output the encoded target vector.

7. The reinforcement learning-based generalized traveling salesman path planning method according to claim 6, characterized in that, The decoding steps of the decoder include: The encoded target vector is used as the input to the decoder, i.e., input from the second attention layer, to calculate the relationship between the input features, capture the interaction between different perforation points, and output the second feature matrix; The second feature matrix is ​​input into the linear transformation layer, and the output of the multi-head attention layer is merged into a single vector. The single vector is input into the third attention layer, and the single vector is used as the query vector. Each node vector is used as the key vector. The query vector and the key vector are multiplied to obtain the attention score of each node. The output of the third attention layer is used as the input of the mask layer, and the nodes that have been visited among all nodes are filtered according to the node mask information in the environment simulation model. The attention score, filtered by the mask layer, is input into the Softmax layer and transformed into a probability distribution through the Softmax function. The final output is the mapping of the behavior policy from the current state s to action A, that is, the probability of finally selecting the punch point.

8. A generalized traveling salesman path planning device based on reinforcement learning, characterized in that, include: The data acquisition module is used to acquire a target layout diagram containing multiple part graphics; The target selection module is used to extract the node set consisting of all parts and the cluster information corresponding to the node set according to the target layout diagram. The node set is the node information of all parts, and the cluster information is all nodes of the same part. The GTSP conversion module is used to convert the cutting path planning problem into a generalized traveling salesman problem (GTSP) solution operation, build an environmental simulation model, and input the node information of all parts and the corresponding node cluster information to obtain the current state information. The current state information includes at least the node information of each part, the part cluster information, the node mask information, and the current node information. The strategy output module is used to output the strategy of the trained strategy model based on the current state information, input the environment simulation model to obtain new state information, traverse all part nodes until the optimal processing sequence result of cutting all parts is obtained.

9. An electronic device, characterized in that, include: The memory is used to store the processing program; A processor, which, when executing the processing program, implements the reinforcement learning-based generalized traveling salesman cutting path planning method as described in any one of claims 1 to 7.

10. A computer-readable medium storing a computer program, characterized in that, When the computer program is executed by one or more processors, it implements the reinforcement learning-based generalized traveling salesman cutting path planning method as described in any one of claims 1 to 7.

Images