Self-adaptive knowledge distillation method suitable for large-scale generalization of travel salesman problems

Abstract

The invention provides a self-adaptive knowledge distillation method suitable for travel salesman problem scale generalization. The method comprises the following steps: acquiring a plurality of TSP data sets of different scales and a TSP sample set; training a teacher model by using the TSP data sets of different scales to obtain a plurality of pre-trained teacher models of different knowledge levels; training a student model by using the TSP sample set, and adaptively selecting a pre-trained teacher model of a corresponding knowledge level to carry out knowledge migration auxiliary training to obtain a trained student model; iterative training is carried out continuously until the global loss function is minimized, an optimal student model is obtained, and knowledge distillation is completed; according to the invention, through an adaptive selection strategy, a student model with universality and generalization ability is trained on a small-scale instance; and meanwhile, through an adaptive soft probability distribution distillation framework, under the condition that the calculation time overhead is maintained to be approximately constant, the model performance is remarkably improved, and TSP scale generalization is realized by combining the adaptive soft probability distribution distillation framework and the model performance.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically, to an adaptive knowledge distillation method suitable for scalable generalization of a traveling salesman problem (TSP). Background Art

[0002] TSP has a wide range of applications in transportation, logistics and distribution, and the research community has conducted a lot of research on TSP. In the early days, researchers used exact algorithms to solve the problem, but as the scale of the problem increased, the time to find the exact solution increased dramatically. Since TSP is an NP-hard problem, its solution in the field of theoretical computer science is extremely challenging. Therefore, researchers began to use heuristic algorithms to solve TSP. Concorde and Gurobi are two well-known exact solvers, among which the Concorde solver can solve instances with up to 85,900 city nodes. The current robust heuristic algorithm can also achieve good results in the case of millions of city nodes. Although there are currently a variety of exact methods and heuristic algorithms that can solve large-scale TSP instances and control the error within a small range, these traditional solution methods usually require artificially designed complex constraint rules, and the calculation time is extremely large. Therefore, it is still difficult to directly extend these algorithms to practical application scenarios.

[0003] In recent years, many studies have tried to use deep learning algorithms to solve the TSP problem in response to the above problems. Unlike traditional methods, deep learning algorithms usually do not require the design of complex constraint rules and can directly solve the problem in an end-to-end manner. Therefore, this research direction has gradually received more and more attention. The Transformer model is one of the most prominent deep learning models at present. Its potential and application prospects in computer vision, natural language processing and other fields have been widely studied and confirmed. In the field of combinatorial optimization, breakthrough progress has been made in the research on solving TSP based on the Transformer model in recent years.

[0004] Knowledge distillation is an interactive learning paradigm of teacher-student models, which can effectively transfer the knowledge of large models to small models, thereby reducing the size of the model and enhancing its generalization ability. Based on this, Bi et al. (Bi J, Ma Y, Wang J, et al. Learning generalizable models for vehicle routing problems via knowledge distillation [J]. Advances in Neural Information Processing Systems, 2022, 35: 31226-31238) first introduced knowledge distillation into TSP solving, and proposed an adaptive multi-distribution knowledge distillation framework (AMDKD-AM) that combines knowledge distillation with the Transformer model. Experimental results show that the Transformer model using knowledge distillation performs well in distribution generalization ability. Although the knowledge distillation method has achieved remarkable success in improving the distribution generalization ability, as the scale of TSP instances increases, the cost of obtaining accurate labels for supervised learning models and the time overhead of instance training become unbearable in practical applications. The existing learning framework that combines knowledge distillation with Transformer, after training a small-scale instance model, deploys it to a large-scale TSP instance, and it is difficult to maintain a low error (that is, it is difficult to achieve scale generalization). Summary of the invention

[0005] In order to overcome the defect that the above-mentioned prior art cannot directly complete the scale generalization task of TSP, the present invention provides an adaptive knowledge distillation method suitable for scale generalization of the traveling salesman problem, wherein a soft probability distribution distillation method is proposed for training student models and successfully expanded to large-scale instances; at the same time, an adaptive selection strategy is proposed to optimize the training process of the student model; by combining these two strategies, the student model can gradually approach the distribution of the teacher model in the geometric space, thereby achieving scale generalization.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] An adaptive knowledge distillation method for scalable generalization of the traveling salesman problem includes the following steps:

[0008] S1: Obtain multiple TSP data sets of different sizes, and sample the data in all TSP data sets to obtain a TSP sample set;

[0009] S2: Establish a teacher model, use TSP datasets of different sizes to train the teacher model, and obtain multiple pre-trained teacher models with different knowledge levels;

[0010] S3: Establish a student model, use the TSP sample set to train the student model, and adaptively select the pre-trained teacher model of the corresponding knowledge level for knowledge transfer auxiliary training to obtain the trained student model;

[0011] S4: Calculate the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round, and perform KL divergence softening on each model;

[0012] S5: Calculate the global loss according to the preset global loss function, and determine whether the global loss is less than the preset threshold. If so, save the trained student model parameters of the current training round, obtain the optimal student model, and complete knowledge distillation; otherwise, update the trained student model parameters according to the global loss, repeat steps S3 to S5, and perform the next round of training on the student model.

[0013] Preferably, in step S1, data in all TSP data sets are sampled to obtain multiple TSP sample sets of the same size;

[0014] The size of the TSP sample set is equal to the minimum size in the TSP data set.

[0015] Preferably, in step S2, the teacher model is trained using the smallest TSP data set to obtain a pre-trained teacher model with the lowest knowledge level;

[0016] On the basis of the pre-trained teacher model with the lowest knowledge level, the scale of the TSP data set is increased step by step for training, and multiple pre-trained teacher models with knowledge levels from low to high are obtained in turn.

[0017] Preferably, in step S3, adaptively selecting a pre-trained teacher model of a corresponding knowledge level to perform knowledge transfer auxiliary training includes the following steps:

[0018] During the training process, when the number of epochs E is less than the preset value E', only the pre-trained teacher model with the lowest knowledge level is used for knowledge transfer auxiliary training;

[0019] When the epoch number E is greater than or equal to the preset value E', the Gurobi solver and softmax function are used to calculate the adaptive probabilities of pre-trained teacher models with different knowledge levels, and the pre-trained teacher model with the largest adaptive probability is selected for knowledge transfer assisted training.

[0020] Preferably, a plurality of TSP data sets of different sizes are additionally obtained as validation sets of different sizes, and the adaptive probability P is calculated according to the following formula:

[0021]

[0022] Among them, AvgGap(S,M) represents the standard solution π of the Gurobi solver under the validation set of size m. tspm (solver) and the solution of the student model tspm The error GAP between (model).

[0023] Preferably, in step S4, calculating the conditional probability distribution of the student model and the pre-trained teacher model includes the following steps:

[0024] The joint probability density function of the student model and the pre-trained teacher model in space is estimated by the cosine kernel function, which is expressed as:

[0025] p ij =p i∣j p j =δ(x i ,x j )

[0026] q ij =q i∣j q j =δ(y i ,y j )

[0027]

[0028] Among them, q ij and p ij Respectively represent the joint probability density of the pre-trained teacher model and the student model in the space; (x i ,x j ) and (y i ,y j ) represent the output vectors of the data sample pairs in the pre-trained teacher model and the student model respectively; δ represents the cosine kernel function, with a value range of [0,1], and ||·||2 represents the L2 norm; Where b is the training batch size, n represents the size of the TSP dataset, and i∈{1,…,N}, N is the number of TSP datasets; p j and q j are the marginal probability densities in the joint probability density of the pre-trained teacher model and the student model respectively;

[0029] The joint probability density function is converted into a conditional probability density function using Bayesian transformation. The conditional probability density functions of the pre-trained teacher model and the student model are:

[0030]

[0031]

[0032] Among them, pi∣j and q i∣j are the conditional probability distributions of the pre-trained teacher model and the student model, respectively, i∣j ,q i∣j ∈[0,1];

[0033] The temperature parameter τ is introduced to soften the KL divergence of the conditional probability distribution of the trained student model and the pre-trained teacher model used in the current training round.

[0034] Preferably, in step S5, the preset global loss function includes task loss and distillation loss, which is expressed as:

[0035]

[0036] in, is the global loss; Task loss for the student model; is the distillation loss between the student model and the teacher model; β is a hyperparameter, β∈[0,1];

[0037] The mission loss It is expressed as:

[0038]

[0039] in, For the conditional probability distribution p θ The expected value under (π|s), L(π) represents the solution predicted by the student model;

[0040] The distillation loss It is expressed as:

[0041]

[0042] Among them, x is the training sample, and Represent the probability distribution functions of the student model and the teacher model respectively; ψ represents the KL divergence function used to measure the consistency between the teacher model and the student model; N T is the total number of training samples used in the teacher model; is the training sample set;

[0043] By calculating the global loss Gradient Update the trained student model parameters, expressed as:

[0044]

[0045] in, is the student model of the tth round; is the student model of the t-1th round, and η is the update step size.

[0046] Preferably, the gradient of the global loss It is expressed as:

[0047]

[0048] in, The task loss gradient of the student model; is the distillation loss gradient between the student model and the teacher model;

[0049] The task loss gradient It is expressed as:

[0050]

[0051] Among them, p θ (π|s) represents the conditional probability distribution predicted by the student model, L(π) represents the solution predicted by the student model, and b(s) represents the standard solution;

[0052] The distillation loss gradient It is expressed as:

[0053]

[0054] Among them, φ is a preset function, and the calculation process is: z i is an example variable.

[0055] Preferably, in step S5, a multi-step update mechanism is introduced, and in each training round, the parameters of the trained student model are updated multiple times according to the calculation results of the global loss, so as to further enhance the adaptability of the student model and thus accelerate the convergence speed.

[0056] Preferably, both the teacher model and the student model are Transformer models, and the number of parameters of the teacher model is greater than that of the student model.

[0057] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0058] The present invention provides an adaptive knowledge distillation method suitable for the generalization of the scale of the traveling salesman problem. First, multiple TSP data sets of different scales are obtained, and data in all TSP data sets are sampled to obtain a TSP sample set; then a teacher model is established, and the teacher model is trained by using TSP data sets of different scales to obtain multiple pre-trained teacher models of different knowledge levels; then a student model is established, and the student model is trained by using the TSP sample set, and a pre-trained teacher model of a corresponding knowledge level is adaptively selected to perform knowledge transfer auxiliary training to obtain a trained student model; the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round is calculated, and KL divergence softening processing is performed respectively; finally, a global loss is calculated according to a preset global loss function, and it is judged whether the global loss is less than a preset threshold value. If so, the parameters of the trained student model of the current training round are saved to obtain the optimal student model to complete the knowledge distillation; otherwise, the parameters of the trained student model are updated according to the global loss, and the student model is re-trained for the next round;

[0059] The present invention proposes a distillation strategy for adaptively selecting a teacher model, which can solve the problem that the knowledge level gap between the student model and the teacher model is large in the early stage, resulting in low learning efficiency. A multi-level progressive training strategy is adopted in the training process of the student model, so that the knowledge level of the teacher model is gradually improved as the student model progresses; in the early stage of training, a teacher model with a lower knowledge level is used to train the student model, and as the training progresses, the knowledge level of the teacher model is gradually improved to improve the learning efficiency and training effect of the student model;

[0060] In addition, the present invention also proposes an adaptive soft probability distribution distillation (ASPDD) framework for dynamically adjusting the training plan to adapt to the individual differences of different learners; by setting the soft probability distribution distillation loss function to supervise the knowledge distillation process, the conditional probability distribution difference between the teacher model and the student model can be minimized, so that the migration samples have a high consistency in the teacher model and student model space; at the same time, softening the damage function can also enable the student model to maintain critical thinking during the training process, rather than relying entirely on the output of the teacher model. When the model is generalized from small-scale instances to larger-scale instances, the soft probability distribution distillation method can effectively improve the accuracy of the model, thereby achieving the scale generalization of the TSP problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of an adaptive knowledge distillation method suitable for scalability generalization of the traveling salesman problem provided in Example 1.

[0062] Figure 2 This is a training process diagram of an adaptive knowledge distillation method suitable for scalable generalization of the traveling salesman problem provided in Example 2.

[0063] Figure 3 This is a schematic diagram of a TSP problem search example provided in Example 2.

[0064] Figure 4 Schematic diagram of solving the TSP problem by using the scale-generalized knowledge distillation provided in Example 2. DETAILED DESCRIPTION

[0065] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0066] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0067] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0068] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment provides an adaptive knowledge distillation method suitable for generalizing the scale of the traveling salesman problem, including the following steps:

[0071] S1: Obtain multiple TSP data sets of different sizes, and sample the data in all TSP data sets to obtain a TSP sample set;

[0072] S2: Establish a teacher model, use TSP datasets of different sizes to train the teacher model, and obtain multiple pre-trained teacher models with different knowledge levels;

[0073] S3: Establish a student model, use the TSP sample set to train the student model, and adaptively select the pre-trained teacher model of the corresponding knowledge level for knowledge transfer auxiliary training to obtain the trained student model;

[0074] S4: Calculate the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round, and perform KL divergence softening on each model;

[0075] S5: Calculate the global loss according to the preset global loss function, and determine whether the global loss is less than the preset threshold. If so, save the trained student model parameters of the current training round, obtain the optimal student model, and complete knowledge distillation; otherwise, update the trained student model parameters according to the global loss, repeat steps S3 to S5, and perform the next round of training on the student model.

[0076] In the specific implementation process, firstly, multiple TSP data sets of different sizes are obtained, and the data in all TSP data sets are sampled to obtain the TSP sample set;

[0077] Then, a teacher model is established and trained using TSP datasets of different sizes to obtain multiple pre-trained teacher models with different knowledge levels;

[0078] Then, a student model is established and trained using the TSP sample set. At the same time, a pre-trained teacher model with the corresponding knowledge level is adaptively selected for knowledge transfer auxiliary training to obtain the trained student model.

[0079] Calculate the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round, and perform KL divergence softening on each.

[0080] Finally, the global loss is calculated according to the preset global loss function to determine whether the global loss is less than the preset threshold. If so, the trained student model parameters of the current training round are saved to obtain the optimal student model and complete the knowledge distillation. Otherwise, the trained student model parameters are updated according to the global loss, and the student model is retrained for the next round.

[0081] This embodiment proposes a soft probability distribution distillation loss for training student models and successfully expands it to large-scale instances. In addition, in order to solve the problem of scale generalization, this embodiment also proposes an adaptive selection strategy to optimize the training process of the student model. By combining these two methods, an adaptive soft probability distribution distillation learning framework is formed. Under this framework, the student model can gradually approach the distribution of the teacher model in the geometric space, thereby achieving scale generalization. At the same time, the knowledge of the teacher model can be transferred to larger-scale TSP instances, further enhancing the generalization ability of the Transformer model on the TSP problem, realizing knowledge transfer and model lightweighting. This method provides a new solution for solving the scale generalization of the TSP problem.

[0082] Example 2

[0083] like Figure 2 As shown, this embodiment provides an adaptive knowledge distillation method suitable for generalizing the scale of the traveling salesman problem, including the following steps:

[0084] S1: Obtain multiple TSP data sets of different sizes, and sample the data in all TSP data sets to obtain a TSP sample set;

[0085] S2: Establish a teacher model, use TSP datasets of different sizes to train the teacher model, and obtain multiple pre-trained teacher models with different knowledge levels;

[0086] S3: Establish a student model, use the TSP sample set to train the student model, and adaptively select the pre-trained teacher model of the corresponding knowledge level for knowledge transfer auxiliary training to obtain the trained student model;

[0087] S4: Calculate the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round, and perform KL divergence softening on each model;

[0088] S5: Calculate the global loss according to the preset global loss function, and determine whether the global loss is less than the preset threshold. If so, save the trained student model parameters of the current training round, obtain the optimal student model, and complete the knowledge distillation; otherwise, update the trained student model parameters according to the global loss, repeat steps S3 to S5, and train the student model for the next round;

[0089] Figure 2 The circles and triangles in the figure represent samples with different characteristics. The small, medium, and large samples correspond to the instances of 20, 50, and 100 cities in the TSP problem, namely, TSP20, TSP50, and TSP100.

[0090] In the step S1, the data in all TSP data sets are sampled to obtain multiple TSP sample sets of the same size;

[0091] The size of the TSP sample set is equal to the minimum size in the TSP data set;

[0092] In step S2, the teacher model is trained using the smallest TSP data set to obtain a pre-trained teacher model with the lowest knowledge level;

[0093] On the basis of the pre-trained teacher model with the lowest knowledge level, the scale of the TSP data set is increased step by step for training, and multiple pre-trained teacher models with knowledge levels from low to high are obtained in turn;

[0094] In step S3, adaptively selecting a pre-trained teacher model of a corresponding knowledge level to perform knowledge transfer auxiliary training includes the following steps:

[0095] During the training process, when the number of epochs E is less than the preset value E', only the pre-trained teacher model with the lowest knowledge level is used for knowledge transfer auxiliary training; in this embodiment, it is defined that the entire training process is performed for a total of 10,000 epochs, each epoch contains 128,000 training operations, and the model is saved every 50 epochs. (E') is selected through experiments to determine the best performance;

[0096] When the number of epochs E is greater than or equal to the preset value E', the Gurobi solver and softmax function are used to calculate the adaptive probabilities of pre-trained teacher models with different knowledge levels, and the pre-trained teacher model with the largest adaptive probability is selected for knowledge transfer auxiliary training;

[0097] In this embodiment, multiple TSP data sets of different sizes are obtained as validation sets of different sizes, and the adaptive probability P is calculated according to the following formula:

[0098]

[0099] Among them, AvgGap(S,M) represents the standard solution π of the Gurobi solver under the validation set of size m. tspm (solver) and the solution of the student model tspm The error GAP between (model);

[0100] In step S4, calculating the conditional probability distribution of the student model and the pre-trained teacher model includes the following steps:

[0101] The joint probability density function of the student model and the pre-trained teacher model in space is estimated by the cosine kernel function, which is expressed as:

[0102] p ij =p i∣j p j =δ(x i ,x j )

[0103] q ij =q i∣j q j =δ(y i ,y j )

[0104]

[0105] Among them, q ij and p ij Respectively represent the joint probability density of the pre-trained teacher model and the student model in the space; (x i ,x j ) and (y i ,y j ) represent the output vectors of the data sample pairs in the pre-trained teacher model and the student model respectively; δ represents the cosine kernel function, with a value range of [0,1], ||·||2 represents the L2 norm, and the L2 norm represents the “length” or “size” of the vector in space; Where b is the training batch size, n represents the size of the TSP dataset, and i∈{1,…,N}, N is the number of TSP datasets; pj and q j are the marginal probability densities in the joint probability density of the pre-trained teacher model and the student model respectively;

[0106] The joint probability density function is converted into a conditional probability density function using Bayesian transformation. The conditional probability density functions of the pre-trained teacher model and the student model are:

[0107]

[0108]

[0109] Among them, p i∣j and q i∣j are the conditional probability distributions of the pre-trained teacher model and the student model, respectively, i∣j ,q i∣j ∈[0,1];

[0110] The temperature parameter τ is introduced to soften the KL divergence of the conditional probability distribution of the trained student model and the pre-trained teacher model used in the current training round;

[0111] In step S5, the preset global loss function includes task loss and distillation loss, which is expressed as:

[0112]

[0113] in, is the global loss; Task loss for the student model; is the distillation loss between the student model and the teacher model; β is a hyperparameter, β∈[0,1];

[0114] The mission loss It is expressed as:

[0115]

[0116] in, For the conditional probability distribution p θ The expected value under (π|s), L(π) represents the solution predicted by the student model;

[0117] The distillation loss It is expressed as:

[0118]

[0119] Among them, x is the training sample, and Represent the probability distribution functions of the student model and the teacher model respectively; ψ represents the KL divergence function used to measure the consistency between the teacher model and the student model; N T is the total number of training samples used in the teacher model; is the training sample set;

[0120] By calculating the global loss Gradient Update the trained student model parameters, expressed as:

[0121]

[0122] in, is the student model of the tth round; is the student model of the t-1th round, η is the update step size;

[0123] The gradient of the global loss It is expressed as:

[0124]

[0125] in, The task loss gradient of the student model; is the distillation loss gradient between the student model and the teacher model;

[0126] The task loss gradient It is expressed as:

[0127]

[0128] Among them, p θ (π|s) represents the conditional probability distribution predicted by the student model, L(π) represents the solution predicted by the student model, and b(s) represents the standard solution;

[0129] The distillation loss gradient It is expressed as:

[0130]

[0131] Among them, φ is a preset function, and the calculation process is: z i is a sample variable;

[0132] In step S5, a multi-step update mechanism is introduced. In each training round, the parameters of the trained student model are updated multiple times according to the calculation results of the global loss, so as to further enhance the adaptability of the student model and thus accelerate the convergence speed.

[0133] The teacher model and the student model are both Transformer models, and the number of parameters of the teacher model is greater than that of the student model. In this embodiment, the teacher model is a more complex Attention Model (AM), and the student model can be a corresponding simplified version Simplified Attention Model (SAM).

[0134] In the specific implementation process, this embodiment mainly studies the transportation path planning problem, taking TSP as an example; TSP is similar to the Hamiltonian circuit problem in graph theory, which can be expressed as: given a set of node positions in a graph, it is required to visit each node and return to the starting point, each node can only be visited once, and the shortest access path (such as Figure 3 As shown in Figure 2, this is an NP-hard problem. Many scholars have conducted extensive research on it, including heuristic methods and exact methods. The TSP problem can be defined as a weighted directed graph G = {ν,ξ}, ν i ∈ν represents a node (each node represents a city), e(v i ,v j )∈ξ represents an edge connecting any two vertices, C(e(v i ,v j )) represents the Euclidean distance between any two vertices. Assume that all vertices are evenly distributed in two-dimensional space. The objective function of the TSP problem can be described as: i Start, pass through all other vertices in the graph and each vertex only once, and finally return to the starting point ν i The shortest path; that is, to find the route θ with the minimum total cost among all possible paths in the limited search space S * =argmin L(θ′∣G)=argmin∑C(e(v i ,v j ));

[0135] Knowledge distillation (KD) is a training paradigm based on interactive learning between a teacher model and a student model. Recent studies have shown that knowledge distillation can enhance the model's ability to generalize distribution and compress the model of the TSP problem. The main goal of knowledge distillation is to transfer knowledge and generate a lightweight model. The specific steps of knowledge distillation are as follows: (1) Train the teacher neural network on the dataset. During the training process, the teacher model gradually learns the knowledge hidden in the dataset, summarizes the knowledge and further explores it; (2) The student model interactively learns based on the knowledge of the teacher model, and uses the output of the teacher model as a supervision signal to pass the acquired knowledge to the student neural network; (3) The student model is also trained on the dataset and is assisted in training based on the knowledge of the teacher model. Figure 4The figure shows the solution of TSP problem by knowledge distillation with generalization scale, and the scale of the model is expanded through interactive learning;

[0136] In the teaching process, students first need to complete basic ability training; when students have mastered the basic knowledge, teachers will arrange tests to evaluate students' learning level; the tests usually include knowledge points that students have not yet fully mastered in the learning process; after the evaluation, teachers will conduct detailed analysis and correction based on students' performance in the tests, especially the wrong parts, and arrange supplementary training in a targeted manner; through this long-term, feedback-driven training, students' learning ability will gradually improve, and they will eventually be able to solve various complex problems;

[0137] The key to knowledge distillation lies in the knowledge transfer between the teacher model and the student model. The student model lacks prior knowledge in the early stage, and direct training by the senior teacher model may not achieve the best results. In this case, the knowledge level gap between the student model and the teacher model is large, resulting in low learning efficiency. To solve this problem, this embodiment proposes to adopt a multi-level progressive training strategy in the training process of the student model, so that the knowledge level of the teacher model gradually improves with the progress of the student model. In the early stage of training, the teacher model with a lower knowledge level is used to train the student model. As the training deepens, the knowledge level of the teacher model is gradually improved to improve the learning efficiency and training effect of the student model.

[0138] This embodiment first gradually trains the teacher model to ensure that it can effectively generalize on TSPs of different scales; the goal of this step is to train the teacher model in a layer-by-layer progressive manner so that it can not only solve small-scale TSP instances, but also show robust generalization capabilities in more complex problems; the training process adopts a method of gradually expanding the scale of the data set to ensure that the teacher model can obtain improved adaptability at each stage; the specific steps are as follows:

[0139] 1) First, train the primary teacher model on the TSP20 dataset (“20” means the dataset size is 20 data items, the same applies below) to acquire basic knowledge expression capabilities; select the small-scale TSP20 dataset for training, the purpose is to allow the teacher model to master the basic characteristics and solution methods of the TSP problem; through initial training on the TSP20 dataset, the teacher model can effectively learn the basic capabilities of path planning and have a certain level of knowledge expression, laying the foundation for subsequent extended training;

[0140] 2) The teacher model is then further trained on the TSP50 dataset to enhance its generalization ability to larger-scale problems. After initially mastering the basic features of TSP, the training dataset is expanded to TSP50 to improve the teacher model's understanding and ability to solve medium-scale problems. In this stage, by introducing more complex data, the teacher model learns more complex path planning patterns and features, thereby enhancing its adaptability and generalization ability to larger-scale problems.

[0141] 3) Continue to train the teacher model on the TSP100 dataset to improve its understanding and ability to solve complex problems. Through training on the TSP100 dataset, the teacher model will face more complex path planning tasks with more nodes. The goal of this stage is to enable the teacher model to handle highly complex problems and effectively learn how to identify the features of the optimal solution in large-scale data, thereby enhancing its ability to solve complex TSP problems.

[0142] 4) Generate multi-level pre-trained teacher models to adapt to TSP problems of different scales and support multi-level knowledge distillation; through the gradual training of data sets of different scales, finally generate multi-level pre-trained teacher models, each model corresponds to a TSP problem of a specific scale; these teacher models will be used to guide the training of student models, gradually transfer knowledge, and realize knowledge distillation; through this multi-level training method, the teacher model can adapt to problems of different scales, and provide targeted guidance to the student model during the knowledge distillation process, so that the student model can show good performance in solving TSP problems of different scales;

[0143] After that, a student model is established and trained using the TSP sample set. At the same time, a pre-trained teacher model with the corresponding knowledge level is adaptively selected for knowledge transfer auxiliary training to obtain the trained student model. The goal of this step is to extract information with structural characteristics by analyzing the potential patterns in the data set to guide the student model to conduct targeted training so that it can adapt to TSP problems of different scales. Through the adaptive selection strategy, the student model can gradually learn effective features related to the task scale, thereby improving its performance on problems of different scales. The specific steps are as follows:

[0144] 1) After obtaining the pre-trained teacher model, analyze the potential patterns and structural features in the dataset; generate multiple TSP20 instance samples from the dataset as the initial training data for the student model; the student model is set as a standard small Transformer model (i.e., AM model); at this stage, by training the TSP20 instance samples, the student model can initially master the basic path planning features, laying the foundation for subsequent more complex learning processes;

[0145] 2) Adopt an adaptive learning strategy to gradually optimize the performance of the student model; during training, dynamically adjust the training objectives and strategies of the student model so that it can effectively generalize in problems of different scales; specifically, adjust the learning strategy of the student model according to its current learning effect to ensure that it can gradually improve its ability to solve complex path planning tasks;

[0146] 3) During training, when the number of epochs E is less than the preset value E', use the pre-trained TSP20 teacher model to conduct basic training on the student model, and disable the adaptive selection strategy at this time;

[0147] When the error index of the student model reaches a low level, it indicates that it already has certain basic capabilities; at this stage, further train the student model by using the TSP20 teacher model to consolidate the learning effect, and disable the adaptive selection strategy at the same time to reduce unnecessary complexity during training;

[0148] 4) When the number of epochs E is greater than or equal to the preset value E', according to the performance of the student model on the TSP20, TSP50, and TSP100 validation sets, adopt an adaptive selection strategy to adjust the selection probability of each pre-trained teacher model, and calculate the error GAP between the solution of the student model and the standard solution to optimize the training effect;

[0149] Specifically, by calculating the performance of the student model on different-scale validation sets, use the softmax function to dynamically adjust the selection probability of each teacher model, so as to ensure that the student model can obtain the guidance of the most suitable teacher model for its current state, and improve the training efficiency and generalization ability;

[0150] At this stage, calculate the error GAP through the Gurobi solver, and use P to represent the adaptive selection probability. The formula is as follows:

[0151]

[0152] Among them, AvgGap(S,M) represents the error GAP between the solution π tspm (solver) of the solver and the solution π tspm (model) of the student model at scale m; π tsp,m (solver) represents the standard solution of the Gurobi solver on the TSP20, TSP50, and TSP100 validation sets, and π tsp,m (model) represents the solution of the student model on the TSP20, TSP50, and TSP100 validation sets;

[0153] When E < E′, the adaptive selection strategy is turned off (that is, select the pre-trained TSP20 as the teacher model);

[0154] At this stage, the error GAP is calculated by the Gurobi solver to quantify the difference between the student model and the standard solution, and the training strategy is adjusted according to the calculation results to ensure that the student model can effectively learn key features from the teacher model;

[0155] Afterwards, through probability density estimation and conditional probability conversion, the probability distribution difference between the teacher model and the student model is gradually reduced, thereby improving the consistency of knowledge in the transfer process; specifically, by accurately mapping the high-dimensional representation of the teacher model with the low-dimensional representation of the student model, the student model can effectively learn the feature abstraction ability and decision boundary of the teacher model to achieve higher accuracy and robustness in task solving; through this process, the consistency of knowledge is ensured during the transfer process across models, and information loss is reduced; the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round is calculated. The specific steps are as follows:

[0156] 1) When the teacher model and the student model receive the TSP20 dataset, they respectively model the interaction between the data sample pairs; this process is implemented through a multi-layer neural network, which will capture the complex feature relationship of the data sample pairs in the high-dimensional space; in order to improve the generalization performance of the model, a variety of data enhancement strategies are used to further enrich the characteristics of the input data, thereby improving the representativeness of the input data of the teacher model and the student model;

[0157] 2) The joint probability density function of the teacher model and the student model can be estimated by the kernel function. The specific estimation method is as follows:

[0158] p ij =p i∣j p j =δ(x i ,x j )

[0159] q ij =q i∣j q j =δ(y i ,y j )

[0160] Among them, q ij and p ij Represent the joint probability density function of the teacher model and the student model, (x i ,x j ) and (y i ,y j ) represent the output vectors of the teacher model and the student model data sample pairs, δ(x,y) represents the kernel function; i∈{1,…,N}, N is the size of the data set, and b is the batch size, n represents the size of the instance (for example, for TSP50, n=50); p j and q j are the marginal probability densities in the joint probability density of the pre-trained teacher model and the student model respectively;

[0161] In this step, the outputs of the teacher model and the student model are jointly estimated through the Gaussian kernel function or the cosine kernel function. This estimation process captures the geometric structure of the teacher model and the student model in the sample space, ensuring that the model is consistent in the interaction between samples. The selection of the kernel function is crucial for probability density estimation. This embodiment uses the cosine kernel function because it has superior performance in processing high-dimensional data and does not require complex hyperparameter adjustment. The specific calculation method is as follows:

[0162]

[0163] Among them, δ cosine (x,y)∈[0,1];

[0164] 3) The joint probability density function is converted into a conditional probability density function using Bayesian transformation. The conditional probability density functions of the teacher model and the student model are specifically expressed as follows:

[0165]

[0166] Among them, p i∣j ,q i∣j ∈[0,1];

[0167] This step uses Bayesian transformation to convert joint probability into conditional probability, so that the model can better understand the causal relationship and correlation between data, thereby replicating the learning ability of the teacher model in the student model;

[0168] In the Bayesian transformation implementation, the probability distribution difference between the teacher model and the student model is gradually reduced through the precise derivation of conditional probability. At the same time, the regularization term is introduced to suppress model overfitting, so that the student model can better generalize to unseen data.

[0169] The goal of training is to minimize the difference in conditional probability distribution between the teacher model and the student model to ensure that the migration samples have high consistency in the teacher model and student model space; during the training process, the model memorizes the interaction between sample pairs and maps the low-dimensional space of the student model to the geometric distribution of the teacher model in the sample space; to achieve this goal, this embodiment defines a global loss function, which comprehensively considers the KL divergence and Euclidean distance between the teacher model and the student model, thereby effectively reducing the difference in probability distribution between the two; the specific steps are as follows:

[0170] 1) Introducing the temperature parameter τ to soften the KL divergence of the conditional probability distribution between the teacher model and the student model, so as to enhance the student model's independent judgment ability during the training process and reduce its dependence on the teacher model's answers; by setting the temperature parameter, the student model can obtain more guidance in the early stage of training, and gradually improve its independence and decision-making ability in the later stage of training;

[0171] 2) Define the global loss function, taking both the task loss and the distillation loss as optimization targets. The specific global loss function is defined as follows:

[0172]

[0173] in, is the global loss; Task loss for the student model; is the distillation loss between the student model and the teacher model; β is a hyperparameter, β∈[0,1];

[0174] The mission loss It is expressed as:

[0175]

[0176] in, For the conditional probability distribution p θ The expected value under (π|s), L(π) represents the solution predicted by the student model;

[0177] The distillation loss It is expressed as:

[0178]

[0179] Among them, x is the training sample, and Represent the probability distribution functions of the student model and the teacher model respectively; ψ represents the KL divergence function used to measure the consistency between the teacher model and the student model; N T is the total number of training samples used in the teacher model; is the training sample set;

[0180] The KL divergence function is defined as follows:

[0181]

[0182] By calculating the global loss Gradient Update the trained student model parameters, expressed as:

[0183]

[0184] in, is the student model of the tth round; is the student model of the t-1th round, η is the update step size;

[0185] The gradient of the global loss It is expressed as:

[0186]

[0187] in, The task loss gradient of the student model; is the distillation loss gradient between the student model and the teacher model;

[0188] The task loss gradient The rollout method in reinforcement learning is used for estimation; the rollout method estimates the expected value of the task loss by simulating possible action sequences, thereby providing accurate gradient information for updating model parameters. The specific calculation is expressed as follows:

[0189]

[0190] Among them, p θ (π|s) represents the conditional probability distribution predicted by the student model, L(π) represents the solution predicted by the student model, and b(s) represents the standard solution;

[0191] The distillation loss gradient It is expressed as:

[0192]

[0193] Among them, φ is a preset function, and the calculation process is: z i is a sample variable;

[0194] The parameters of the student model are optimized through the global loss function to further improve its generalization ability and accuracy. During the optimization process, the student model is gradually approached to the optimal solution through the gradient descent algorithm combined with the momentum term and the learning rate decay strategy.

[0195] In order to further enhance the adaptability of the student model, this embodiment also introduces a multi-step update mechanism, which updates the model parameters multiple times in each iteration, thereby accelerating the convergence speed; the "multi-step update mechanism" can be reflected in the following ways:

[0196] 1) Nested loop structure: There are two loops in the algorithm, the outer loop is epoch = 1: E, and the inner loop is step = 1: T. Each epoch contains multiple steps. This nested loop structure is essentially a multi-step update mechanism. The outer epoch loop can be regarded as the overall training process of the model, while the inner step loop is the refinement of data processing and multiple gradient updates in each epoch.

[0197] 2) Multiple gradient calculations and updates: In each step, multiple gradient calculations are performed, including the task loss gradient and the distillation loss gradient; then the global loss is calculated using these two gradients; finally, the model parameters are updated using the step size η and the global loss; this update step is performed in each step, so the model will be updated multiple times in one epoch;

[0198] 3) Combining learning rate with global loss: After calculating the global loss, the model parameters are updated in a way that ensures that the model is updated at each step, rather than after the entire epoch.

[0199] The algorithm implements a multi-step update mechanism by performing gradient calculation and model update steps multiple times in each epoch, so that the model parameters can be more fully optimized in each epoch, thereby improving the learning effect of the model;

[0200] At the same time, this embodiment also introduces a new dynamic knowledge distillation strategy during the training process, which gradually adjusts the guidance intensity of the teacher model according to the training progress; provides more guidance in the early stage of training to help the student model quickly adapt to the task; and reduces the dependence on the teacher model in the later stage of training to enhance the independent learning ability and generalization performance of the student model; this dynamic adjustment strategy combined with the change of temperature parameters makes the process of knowledge transfer smoother and more efficient, and ultimately effectively improves the efficiency of knowledge transfer and the final performance of the model;

[0201] The algorithm pseudo code of this method is as follows:

[0202]

[0203] In addition, this embodiment also verifies the proposed method to verify the effectiveness of the adaptive selection strategy and the adaptive soft probability distribution distillation method; the numerical experiment comprehensively evaluates the various components of the adaptive knowledge distillation framework through a large amount of actual data and test scenarios to ensure that the proposed method not only has theoretical advantages but also can show good performance in practical applications; the experiment combines TSP instances of different scales to verify the stability and generalization ability of the framework when dealing with problems of different complexity;

[0204] Numerical experimental training is conducted on the TSP20 dataset, and the trained model is deployed to TSP problem instances of different scales, including TSP20, TSP50, TSP100, TSP200 and TSP1000. During the training process, the task loss function is estimated using the commonly used rollout method, and the hyperparameter setting is consistent with the basic architecture of the AMDKD-AM network. A greedy strategy is adopted during training, and the Adam optimizer is used as the optimization tool. The experimental framework is built on Pytorch to ensure the repeatability of the experimental results and the flexibility of the code implementation. In addition, in order to enhance the stability of the model, a learning rate decay strategy is used during the training process, and the experiment is repeated on multiple different random seeds to reduce the impact of random factors on the model performance, thereby ensuring the reliability of the experimental results.

[0205] The performance of the adaptive knowledge distillation framework (ASPDD) proposed in this embodiment is compared with the mainstream Transformer framework models in the current TSP problem, including the following models: the Transformer model DACT based on dynamic cyclic coding; the learning collaboration strategy model LCP; the basic Transformer model AM for the VRP problem; the Transformer framework AMDKD-AM combined with knowledge distillation;

[0206] The main goal of this method is to enhance the scale generalization ability of the Transformer model. Therefore, only the models based on the Transformer architecture are compared to verify the advantages of the proposed framework on TSP problems of different scales. To ensure the fairness of the comparison, the training and testing environments of all models are kept consistent, including the same hardware configuration and data preprocessing methods. In addition, this embodiment also tunes the hyperparameters of different models to ensure that each model is in the best performance state, making the comparison results more objective and fair.

[0207] Table 1 shows the deployment effect of the model trained on the TSP20 instance on 5 different scale instances. The evaluation indicators include computational cost, gap, and computational time.

[0208] Table 1: Comparative analysis of this method with the Transformer-based model and the AMDKD model

[0209]

[0210] Note: All tests are based on models trained on TSP20 instances, and the error GAP is solved by Concorde and Gurobi. The test set contains 10,000 instances of {20,50,100} scale, and 2,000 instances of {200,1000} scale; AMDKD-AM★ indicates a model trained using the multi-distribution teacher model described in the original paper, and AMDKD-AM◇ indicates a model trained using the basic mode;

[0211] The performance characteristics of the existing models are as follows (see Table 1 for details):

[0212] 1) DACT model: It performs poorly in terms of scale generalization, lagging behind other models in terms of both gap and time. On a single GPU, when the number of test nodes exceeds 50, the model may overflow due to insufficient memory.

[0213] 2) LCP model: The gap on the TSP20 instance is as low as 0.00%, but the time overhead is extremely high, indicating that this model needs a long time to optimize the gap; when the number of nodes exceeds 20, the performance begins to decline significantly;

[0214] 3) AM model and AMDKD-AM model: As the instance size increases, the performance of the AM(64) model decreases, while the AMDKD-AM(64) model is relatively more stable;

[0215] Compared with the AMDKD-AM model, the framework proposed in this embodiment has the following performance advantages:

[0216] 1) Balance between time consumption and performance: The ASPDD framework can significantly improve model performance while keeping time consumption relatively constant. It is also observed in the experiment that different training modes of the AMDKD-AM model show their own advantages and disadvantages on different scale instances. In order to simplify the discussion, the present invention mainly focuses on the basic training mode of AMDKD-AM.

[0217] 2) Gap optimization effect: On the TSP20 scale, the 64-dimensional ASPDD model achieved a 0.02% improvement in the gap compared to the AMDKD-AM◇ framework; on the TSP50 scale, the gap improved by 0.51%; on the TSP100 scale, the gap improved by 2.34%; and on the TSP1000 scale, the gap improved by 9.3%;

[0218] 3) Impact of embedding dimension on performance: When the embedding dimension is 64, the ASPDD framework significantly outperforms the AMDKD-AM and AM models; however, compared with other models with an embedding dimension of 128, the improvement of ASPDD (128) is relatively limited; this is mainly attributed to the strong learning ability of the large model itself, which limits the space for further performance improvement;

[0219] 4) Relationship between model capacity and performance: As the ASPDD model capacity increases, further improvement in the gap is observed only on the TSP1000 scale instance; this indicates that under the ASPDD framework, increasing the model capacity may not necessarily lead to significant performance improvements;

[0220] The above experimental results verify the effectiveness and superiority of the ASPDD framework at multiple instance scales;

[0221] In order to further verify the effectiveness of the ASPDD framework, this embodiment also randomly selected 11 benchmark data sets from TSPLIB for experiments; the model was trained on an instance with a data scale of 20, and the trained model was used to test the benchmark data set;

[0222] Table 2: Performance evaluation of ASPDD extension on TSPLIB instance

[0223]

[0224] Note: All test instances are based on models trained on TSP20 instances;

[0225] The experimental results are shown in Table 2. It can be seen that the AM model performs poorly when facing unknown data distribution and extended instances; while the AMDKD-AM model that introduces knowledge distillation and interactive learning has significantly improved performance; especially in the six benchmark data sets, the ASPDD framework proposed in this embodiment is better than the AMDKD-AM framework, showing a more superior performance;

[0226] Finally, this embodiment conducts an ablation experiment on the adaptive soft probability distribution distillation framework to comprehensively evaluate the impact of each component on the overall performance. Specifically, this embodiment selects two sets of data sets of different sizes for TSP, namely TSP20 and TSP50, to verify the effectiveness of the adaptive selection strategy and the soft probability distribution distillation method. The experimental results are shown in Table 3.

[0227] Table 3: Ablation experiment results

[0228]

[0229] In the experiment, this embodiment compared the performance of the model including the adaptive selection strategy and the soft probability distribution distillation method with the model not using these methods; the results showed that the model using the adaptive selection strategy and the soft probability distribution distillation method achieved the highest level in terms of accuracy; on the TSP20 scale data set, the model using the adaptive selection strategy had an accuracy improvement of 0.02% compared with the model not using the adaptive strategy; on the TSP50 scale, the accuracy was improved by 0.04%; and compared with the model not using the soft probability distribution distillation method, the accuracy was improved by 0.74%.

[0230] In general, this embodiment proposes an adaptive soft probability distribution distillation framework for TSP to solve the problem that the existing Transformer series models and knowledge distillation frameworks lack scalability in large-scale instances. Unlike the existing AMDKD-AM framework that mainly focuses on the distribution generalization problem, the goal of this method is to train the model on small-scale instances and be able to generalize and apply it to large-scale instances to address the challenges of the Transformer model when processing large-scale instances, such as long training time and difficulty in obtaining solution labels.

[0231] To achieve this goal, this embodiment proposes a soft probability distribution distillation loss function and an adaptive adjustment strategy. Specifically, by learning the geometric distribution characteristics of the teacher model and combining the adaptive selection strategy, a student model with universality and generalization ability is trained on a small scale instance. The proposed adaptive soft probability distribution distillation framework can significantly improve the model performance while keeping the computational time overhead roughly constant.

[0232] In addition, the deployment of lightweight models helps to improve the efficiency and cost-effectiveness of logistics and transportation, and provides effective technical support for energy conservation and emission reduction and improving the efficiency of social and economic operations. This method is in line with the trend of logistics and supply chain management towards automation and intelligence, and has important practical application value in promoting the green transformation of the logistics industry.

[0233] The same or similar reference numerals correspond to the same or similar components;

[0234] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0235] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An adaptive knowledge distillation method suitable for scalable generalization of the traveling salesman problem, characterized in that: The following steps are involved: S1: Obtain multiple TSP data sets of different sizes, and sample the data in all TSP data sets to obtain a TSP sample set; S2: Establish a teacher model, use TSP datasets of different sizes to train the teacher model, and obtain multiple pre-trained teacher models with different knowledge levels; S3: Establish a student model, use the TSP sample set to train the student model, and adaptively select the pre-trained teacher model of the corresponding knowledge level for knowledge transfer auxiliary training to obtain the trained student model; S4: Calculate the conditional probability distribution between the trained student model and the pre-trained teacher model used in the current training round, and perform KL divergence softening on each model; S5: Calculate the global loss according to the preset global loss function, and determine whether the global loss is less than the preset threshold. If so, save the trained student model parameters of the current training round, obtain the optimal student model, and complete knowledge distillation; Otherwise, update the trained student model parameters according to the global loss, repeat steps S3 to S5, and perform the next round of training on the student model.

2. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 1, characterized in that: In the step S1, the data in all TSP data sets are sampled to obtain multiple TSP sample sets of the same size; The size of the TSP sample set is equal to the minimum size in the TSP data set.

3. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 1, characterized in that: In step S2, the teacher model is trained using the smallest TSP data set to obtain a pre-trained teacher model with the lowest knowledge level; On the basis of the pre-trained teacher model with the lowest knowledge level, the scale of the TSP data set is increased step by step for training, and multiple pre-trained teacher models with knowledge levels from low to high are obtained in turn.

4. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 1, characterized in that: In step S3, adaptively selecting a pre-trained teacher model of a corresponding knowledge level to perform knowledge transfer auxiliary training includes the following steps: During the training process, when the number of epochs E is less than the preset value E', only the pre-trained teacher model with the lowest knowledge level is used for knowledge transfer auxiliary training; When the number of epochs E is greater than or equal to the preset value E', the Gurobi solver and softmax function are used to calculate the adaptive probabilities of pre-trained teacher models with different knowledge levels, and the pre-trained teacher model with the largest adaptive probability is selected for knowledge transfer assisted training.

5. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 4, characterized in that: In addition, multiple TSP data sets of different sizes are obtained as validation sets of different sizes, and the adaptive probability P is calculated according to the following formula: Among them, AvgGap(S,M) represents the standard solution π of the Gurobi solver under the validation set of size m. tspm (solver) and the solution of the student model tspm The error GAP between (model).

6. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 1, characterized in that: In step S4, calculating the conditional probability distribution of the student model and the pre-trained teacher model includes the following steps: The joint probability density function of the student model and the pre-trained teacher model in space is estimated by the cosine kernel function, which is expressed as: p ij =p i∣j p j =δ(x i ,x j ) q ij =q i∣j q j =δ(and i ,and j ) Among them, q ij and p ij Respectively represent the joint probability density of the pre-trained teacher model and the student model in the space; (x i ,x j ) and (y i ,y j ) represent the output vectors of the data sample pairs in the pre-trained teacher model and the student model respectively; δ represents the cosine kernel function, with a value range of [0,1], and ||·||2 represents the L2 norm; Where b is the training batch size, n represents the size of the TSP dataset, and i∈{1,…,N}, N is the number of TSP datasets; p j and q j are the marginal probability densities in the joint probability density of the pre-trained teacher model and the student model respectively; The joint probability density function is converted into a conditional probability density function using Bayesian transformation. The conditional probability density functions of the pre-trained teacher model and the student model are: Among them, p i∣j and q i∣j are the conditional probability distributions of the pre-trained teacher model and the student model, respectively, i∣j ,q i∣j ∈[0,1]; The temperature parameter τ is introduced to soften the KL divergence of the conditional probability distribution of the trained student model and the pre-trained teacher model used in the current training round.

7. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 6, characterized in that: In step S5, the preset global loss function includes task loss and distillation loss, which is expressed as: in, is the global loss; Task loss for the student model; is the distillation loss between the student model and the teacher model; β is a hyperparameter, β∈[0,1]; The mission loss It is expressed as: in, For the conditional probability distribution p θ The expected value under (π|s), L(π) represents the solution predicted by the student model; The distillation loss It is expressed as: Among them, x is the training sample, and Represent the probability distribution functions of the student model and the teacher model respectively; ψ represents the KL divergence function used to measure the consistency between the teacher model and the student model; N T is the total number of training samples used in the teacher model; is the training sample set; By calculating the global loss Gradient Update the trained student model parameters, expressed as: in, is the student model of the tth round; is the student model of the t-1th round, and η is the update step size.

8. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 7, characterized in that: The gradient of the global loss It is expressed as: in, The task loss gradient of the student model; is the distillation loss gradient between the student model and the teacher model; The task loss gradient It is expressed as: Among them, p θ (π|s) represents the conditional probability distribution predicted by the student model, L(π) represents the solution predicted by the student model, and b(s) represents the standard solution; The distillation loss gradient It is expressed as: Among them, φ is a preset function, and the calculation process is: z i is an example variable.

9. The adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to claim 1, characterized in that: In step S5, a multi-step update mechanism is introduced. In each training round, the parameters of the trained student model are updated multiple times according to the calculation results of the global loss, so as to further enhance the adaptability of the student model and thus accelerate the convergence speed.

10. An adaptive knowledge distillation method for scalable generalization of the traveling salesman problem according to any one of claims 1 to 9, characterized in that: Both the teacher model and the student model are Transformer models, and the number of parameters of the teacher model is greater than that of the student model.

Images