Sample classification method and system based on progressive guided distillation

Abstract

The application discloses a sample classification method and system based on progressive guided distillation, and the method comprises the following steps: in the completely independent training stage, adjusting the parameters of an efficientnet model to obtain an optimized efficientnet model; freezing the parameters of the main network in the optimized efficientnet model; in the transition stage, adjusting the parameters of the classification head in the optimized efficientnet model based on the KL divergence loss dynamic weight; in the pure distillation stage, adjusting the parameters of the classification head in the optimized efficientnet model based on the KL divergence loss value to obtain a sample classification model based on progressive guided distillation; and performing sample classification based on the sample classification model based on progressive guided distillation. The application effectively distills the teacher model into the student model, approaches the performance of the large model in the limited computing resource environment, and improves the classification prediction accuracy of samples in different fields.

Description

Technical Field

[0001] This invention relates to the field of data classification technology, and in particular to a sample classification method and system based on progressive guided distillation. Background Technology

[0002] Deep learning models face a common challenge in specialized domain classification tasks. While convolutional neural network-based image classification models perform well in natural image recognition, they exhibit significant limitations in specialized domains such as ship sonar spectrum analysis. Existing solutions suffer from the following contradictions: first, global knowledge distillation destroys the general representations learned by the backbone network; second, while traditional machine learning models trained independently can capture domain features, they cannot be optimized in synergy with deep learning architectures. Therefore, improving computational efficiency and classification prediction accuracy for samples from different domains has become an urgent problem to be solved.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a sample classification method and system based on progressive guided distillation, aiming to solve the technical problem of how to improve computational efficiency and the accuracy of classification prediction for samples in different fields.

[0005] To achieve the above objectives, the present invention provides a sample classification method based on progressive guided distillation, the method comprising: S1, Construct an initial progressive guided distillation sample classification model, which is based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model; S2, During the completely independent training phase, the efficientnet model within the initial progressive guided distillation sample classification model is trained based on all samples, and the parameters of the efficientnet model are adjusted based on the cross-entropy loss value to obtain the efficientnet model to be optimized. S3, During the transition phase, freeze the parameters of the backbone network within the EfficientNet model to be optimized; S4, Train the sample classification model of the initial progressive guided distillation based on all samples, and adjust the parameters of the classification head in the model to be optimized based on the dynamic weights of KL divergence loss; S5, during the pure distillation stage, the sample classification model of the initial progressive guided distillation is trained based on all samples, and the KL divergence loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model is calculated. S6. Based on the KL divergence loss value, adjust the parameters of the classifier head in the EfficientNet model to be optimized to obtain a sample classification model with progressive guided distillation. S7. Sample classification is performed based on the progressive guided distillation sample classification model.

[0006] Optionally, S2 includes: S2.1, During the completely independent training phase, the sample training set is input into the EfficientNet model for training; S2.2 After training is completed, the sample validation set is input into the EfficientNet model for validation to obtain the predicted class probability distribution of each sample; S2.3 Calculate the cross-entropy loss value based on the predicted class probability distribution and the corresponding true class probability distribution of each sample; S2.4, determine whether the change between the cross-entropy loss value and the previous cross-entropy loss value is less than the preset first-stage learning rate; S2.5 If so, save the model parameters of the EfficientNet model to obtain the EfficientNet model to be optimized; S2.6 If not, then based on backpropagation, adjust the model parameters of the EfficientNet model according to the cross-entropy loss value, and return to S2.1, iterating until the learning rate is less than the preset first-stage learning rate.

[0007] Optionally, S4 includes: S4.1, Train the sample classification model of the initial progressive guided distillation based on all samples, calculate the dynamic weight of KL divergence loss using a linearly increasing formula, and calculate the total loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model based on the dynamic weight of KL divergence loss. S4.2, determine whether the dynamic weights of the KL divergence loss meet the preset weight conditions; S4.3, if so, proceed to the pure distillation stage; S4.4 If not, then based on backpropagation, adjust the parameters of the classifier head in the EfficientNet model to be optimized according to the total loss value, return to S4.1, and iterate until the preset weight condition is met.

[0008] Optionally, S4.1 includes: S4.1.1 Input all samples into the EfficientNet model to be optimized to obtain the predicted class probability distribution of each sample; S4.1.2, determine boundary samples and non-boundary samples based on the predicted category probability distribution; S4.1.3, the feature vector of the boundary sample is selected from the feature vectors of all samples using the hook function; S4.1.4, Generate a boundary sample feature matrix based on the feature vector of the boundary sample; S4.1.5, Input the boundary sample feature matrix into the ELM-SVM hybrid teacher model and output the predicted class probability distribution of the boundary sample; S4.1.6 Calculate the CE loss value based on the predicted class probability distribution of the non-boundary samples; S4.1.7 Calculate the KL divergence loss value based on the predicted class probability distribution of the boundary samples within the ELF network model to be optimized and the predicted class probability distribution of the boundary samples within the ELM-SVM hybrid teacher model. S4.1.8, determine the current training round of the sample classification model of the initial progressive guided distillation during the transition phase, and calculate the dynamic weight of KL divergence loss according to the current training round using a linearly increasing formula; S4.1.9, Calculate the total loss value based on the KL divergence loss dynamic weight according to the CE loss value and the KL divergence loss value.

[0009] Optionally, the linearly increasing formula is:

[0010] In the formula, For the dynamic weights of KL divergence loss, To preset the step size, This refers to the current training round during the transition phase. This refers to the total number of training rounds within the completely independent training phase.

[0011] Optionally, S6 includes: S6.1, determine whether the change in KL divergence loss between the KL divergence loss value and the previous KL divergence loss value is less than the preset third-stage learning rate; S6.2 If so, save the parameters of the classification head in the EfficientNet model to be optimized to obtain the sample classification model of progressive guided distillation; S6.3 If not, then based on backpropagation, adjust the parameters of the classifier head in the EfficientNet model to be optimized according to the KL divergence loss value, and return to S5, iterating until it is less than the preset third-stage learning rate.

[0012] Optionally, before S1, the following steps are included: S01, Construct an initial EfficientNet model and an initial ELM-SVM hybrid teacher model. The initial EfficientNet model includes a backbone network and a classification head. The initial ELM-SVM hybrid teacher model is constructed based on a combination of multi-layer ELM and SVM, with residual connections between each layer of ELM. S02, Input the sample training set into the initial EfficientNet model for training; S03. After training is completed, the sample validation set is input into the initial EfficientNet model for validation, the predicted class probability distribution of each sample is obtained, and the boundary sample of the current training round is determined according to the predicted class probability distribution. S04, obtain the feature vector of the boundary sample through the hook function, and store the feature vector of the boundary sample into the boundary sample set; S05, calculate the cross-entropy loss and validation accuracy based on the predicted class probability distribution and the corresponding true class probability distribution of each sample; S06, determine whether the verification accuracy has reached convergence; S07, if so, save the model parameters of the initial efficientnet model to obtain the efficientnet model; S08, if not, then based on backpropagation, adjust the model parameters of the initial EfficientNet model according to the cross-entropy loss value, and return to S02, iteratively execute until the verification accuracy reaches convergence; S09, The initial ELM-SVM hybrid teacher model is trained based on the boundary sample set to obtain the ELM-SVM hybrid teacher model.

[0013] Optionally, S09 includes: S09.1, Initialize the parameters of the SVM; S09.2 Initialize the parameters of the multilayer ELM after residual connection; S09.3, input the boundary sample feature matrix of the boundary sample set into the initial ELM-SVM hybrid teacher model for training, adjust the parameters of the SVM, and obtain the boundary sample accuracy; S09.4, Determine whether the current training round is equal to the preset round; S09.5 If not, save the model parameters of the current training round and return to S09.1 until they are equal to the preset round; S09.6 Extract the model parameters corresponding to the highest boundary sample accuracy from multiple sets of model parameters to obtain the ELM-SVM hybrid teacher model.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a sample classification system based on progressive guided distillation, wherein the sample classification system based on progressive guided distillation includes: The module is used to build an initial progressive guided distillation sample classification model, which is based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model. The training module is used to train the EfficientNet model within the initial progressive guided distillation sample classification model based on all samples during the completely independent training phase, and adjust the parameters of the EfficientNet model based on the cross-entropy loss value to obtain the EfficientNet model to be optimized. The training module is also used to freeze the parameters of the backbone network within the EfficientNet model to be optimized during the transition phase. The training module is also used to train the sample classification model of the initial progressive guided distillation based on all samples, and to adjust the parameters of the classification head in the model to be optimized based on the dynamic weights of the KL divergence loss. The training module is also used to train the initial progressive guided distillation sample classification model based on all samples during the pure distillation stage, calculate the KL divergence loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model, and adjust the parameters of the classifier head in the EfficientNet model to be optimized based on the KL divergence loss value to obtain the progressive guided distillation sample classification model. The classification module is used to classify samples based on the progressive guided distillation sample classification model.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a sample classification device based on progressive guided distillation, the device comprising: a memory, a processor, and a sample classification program based on progressive guided distillation stored in the memory and executable on the processor, the sample classification program based on progressive guided distillation being configured to implement the steps of the sample classification method based on progressive guided distillation as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a sample classification program based on progressive guided distillation, wherein the sample classification program based on progressive guided distillation, when executed by a processor, implements the steps of the sample classification method based on progressive guided distillation as described above.

[0017] This invention first constructs an initial progressive guided distillation sample classification model, which is based on a combination of an EfficientNet model and an ELM-SVM hybrid teacher model. Then, during the completely independent training phase, the EfficientNet model within the initial progressive guided distillation sample classification model is trained using all samples to adjust its parameters, resulting in an optimized EfficientNet model. The parameters of the backbone network within this optimized EfficientNet model are then frozen. During the transition phase, the initial progressive guided distillation sample classification model is trained using all samples, and the parameters of the classifier head within the optimized EfficientNet model are dynamically adjusted based on the KL divergence loss. During the pure distillation phase, the initial progressive guided distillation sample classification model is trained using all samples, and the KL divergence loss between the optimized EfficientNet model and the ELM-SVM hybrid teacher model is calculated. The parameters of the classifier head within the optimized EfficientNet model are then adjusted based on the KL divergence loss value, resulting in a progressively guided distillation sample classification model. Finally, sample classification is performed based on this progressively guided distillation sample classification model. In the first stage of this invention, basic classification ability is established through pure cross-entropy loss. In the second stage, progressively weighted KL divergence loss is introduced to smoothly transition the model to knowledge distillation mode. In the third stage, the focus is on optimizing KL divergence loss. Convergence is determined when the change in loss is less than the learning rate. This allows the student model to maintain its lightweight nature while improving the classification prediction accuracy of samples from different domains. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall process of the first embodiment of the sample classification method based on progressive guided distillation of the present invention; Figure 2 This is a schematic diagram of the multi-layer ELM architecture logic of the first embodiment of the sample classification method based on progressive guided distillation of the present invention; Figure 3 This is a structural block diagram of the first embodiment of the sample classification system based on progressive guided distillation of the present invention.

[0019] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] This invention provides a sample classification method based on progressive guided distillation, referring to... Figure 1 , Figure 1This is a schematic diagram of the overall process of the first embodiment of the sample classification method based on progressive guided distillation of the present invention.

[0022] In this embodiment, the sample classification method based on progressive guided distillation includes the following steps: S1, Construct an initial progressive guided distillation sample classification model, which is based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model.

[0023] It is easy to understand that the execution subject of this embodiment can be a computer device with functions such as data processing, network communication and program execution, and this embodiment does not limit it.

[0024] In the specific implementation, it is also necessary to pre-build an EfficientNet model and an ELM-SVM hybrid teacher model, obtain multiple samples and their corresponding ground truth class labels, construct a sample set based on the multiple samples and their corresponding ground truth class labels, and then divide the multiple samples in the sample set into a sample training set and a sample validation set according to a preset ratio. This preset ratio can be customized by the user, such as 8:2. The samples can be ship sonar spectrum samples, photoelectric samples, image samples, etc.

[0025] The process is as follows: S01, construct the initial EfficientNet model and the initial ELM-SVM hybrid teacher model. The initial EfficientNet model includes a backbone network and a classification head. The initial ELM-SVM hybrid teacher model is constructed based on a combination of multi-layer ELM and SVM, with residual connections between each ELM layer. (Refer to...) Figure 2 , Figure 2 This is a schematic diagram of the multi-layer ELM architecture logic of the first embodiment of the sample classification method based on progressive guided distillation of the present invention; S02, Input the sample training set into the initial EfficientNet model for training; S03. After training is completed, the sample validation set is input into the initial EfficientNet model for validation, the predicted class probability distribution of each sample is obtained, and the boundary sample of the current training round is determined based on the predicted class probability distribution. S04, obtain the feature vector of the boundary sample through the hook function (it should be noted that the hook function can capture the feature vector of all samples. After determining the boundary sample of the current training round, the hook function can cancel the feature vector of non-boundary samples and retain the feature vector of the boundary sample), and store the feature vector of the boundary sample in the boundary sample set. S05, calculate the cross-entropy loss and validation accuracy based on the predicted class probability distribution and the corresponding true class probability distribution of each sample; S06, Determine whether the verification accuracy has converged; S07, if so, save the model parameters of the initial EfficientNet model to obtain the EfficientNet model; S08, if not, then based on backpropagation, adjust the model parameters of the initial EfficientNet model (the model parameters are the weights and biases of each layer in the model) according to the cross-entropy loss value, and return to S02, iterate until the verification accuracy reaches convergence; S09, Train the initial ELM-SVM hybrid teacher model based on the boundary sample set to obtain the ELM-SVM hybrid teacher model.

[0026] It should be understood that the backbone network within the initial EfficientNet model is responsible for extracting high-level features from the input data. Its components include: an initial convolutional layer (stem) for processing 5-channel input, and 7 MobileInverted BottleneckConv (MBConv) blocks for simultaneously expanding the channel dimension, gradually increasing the number of feature channels, enhancing feature richness and spatial compression, and gradually reducing the feature map size. (Reducing spatial resolution significantly reduces computational cost, and gradually expanding the receptive field allows lower layers to focus on pixel-level features while higher layers understand the overall structure.)

[0027] The classification head performs classification based on extracted features and consists of a global average pooling layer and a fully connected layer. The raw output of the fully connected layer (logits: unnormalized scores in the neural network) is activated by Softmax, normalized, and outputs the class probability distribution [e.g., sample 1 (0.7 0.2 0.08 0.02)].

[0028] Data stream of the classification head: Backbone network feature extraction → global average pooling → flattening → fully connected layer → logits output → softmax activation → output (class probability distribution (i.e., including probability distribution and predicted class), model training set accuracy and validation set accuracy).

[0029] It should be noted that the number of floating-point operations per second (FLOPs) is approximately equal to the size² × the number of channels. The number of floating-point operations per second is the core indicator for measuring the computational complexity of a model.

[0030] Original computational cost: 460² × 5 = 1,058,000; Final computational cost: 1² × 1280 = 1,280. Compression ratio: ≈ 1 / 800.

[0031] It should also be noted that feature extraction points are set immediately after the last convolutional layer of the backbone network of the EfficientNet model and before global average pooling (the classification head contains pooling layers and fully connected layers), and a hook mechanism is used to capture intermediate features. This provides feature information for the initial ELM-SVM hybrid teacher model, retaining more details for boundary sample analysis, and serves as the input to the initial ELM-SVM hybrid teacher model.

[0032] It should be understood that ELM+SVM is based on the OVR (One vs Rest) multi-classification strategy, which trains a classifier for each class to separate that class from other classes. Each time the classifier is trained, all boundary samples need to be called, resulting in high data utilization. In addition, combined with ELM feature extraction, it can maintain a high classification accuracy, thus forming an ELM+SVM four-classification model (i.e., the initial ELM-SVM hybrid teacher model).

[0033] It should be noted that at this point, the original features of the samples are extracted by the EfficientNet backbone network, and it is not certain which samples are boundary samples. Therefore, the hook mechanism of the feature extraction points will capture the features of all samples. Only when all samples pass through the classification head and the classification result is output can the boundary sample set be determined (samples that are misclassified or have a confidence score < a preset confidence threshold (e.g., 0.7) are judged as boundary samples). At this time, the hook will save the features of the boundary samples and cancel the features of the non-boundary samples.

[0034] Train the initial EfficientNet model, determine whether the validation accuracy has converged, and if so, save the model parameters (i.e., the weights and biases of each layer of the model) to obtain the EfficientNet model.

[0035] It should also be noted that in each training round, the classification head returns the boundary samples of the current round to the hook function. The hook saves the feature vectors of the boundary samples into the boundary sample set and cancels the features of the non-boundary samples. Since the boundary samples obtained in each round may differ, this scheme adopts a merging strategy to combine the boundary sample sets obtained from all training rounds to obtain the total boundary sample set.

[0036] In the specific implementation, the process of training the initial ELM-SVM hybrid teacher model in step S09 is as follows: S09.1, Initialize the parameters of the SVM (the parameter initialization process for the SVM is to initialize the Lagrange multipliers (i.e., the weights of each support vector) to 0, initialize the global bias term of the SVM to 0, set the default penalty factor C to 1, and the hyperparameters of the SVM are determined by the selected kernel function and are calculated automatically). S09.2 Initialize the parameters of the multilayer ELM after residual connection; S09.3, input the boundary sample feature matrix of the boundary sample set into the initial ELM-SVM hybrid teacher model for training, adjust the parameters of the SVM, and obtain the boundary sample accuracy; S09.4, Determine whether the current training round is equal to the preset round; S09.5 If not, save the model parameters of the current training round (the model parameters of ELM and the global bias term, the bias term of the support vector, and the Lagrange multiplier of the support vector of SVM), and return to S09.1, until it is equal to the preset round. S09.6 Extract the model parameters corresponding to the highest boundary sample accuracy from multiple sets of model parameters to obtain the ELM-SVM hybrid teacher model.

[0037] In this embodiment, the boundary sample feature matrix of the boundary sample set is used as input to train the initial ELM-SVM hybrid teacher model for the four-class classification performance of boundary samples.

[0038] It should also be noted that the boundary samples in the boundary sample set are divided into a boundary sample training subset and a boundary sample validation subset according to a preset ratio. Then, the boundary sample feature matrix of the boundary sample set is input into the initial ELM-SVM hybrid teacher model for training. The parameters of the SVM (i.e., the global bias term, the bias term of the support vector, and the Lagrange multipliers of the support vector) are adjusted to obtain the boundary sample accuracy. The processing method is as follows: 1. The boundary sample feature matrix of the boundary sample training subset is input into a multilayer ELM with residual connections and the feature is transformed through the constructed residual connection formula to obtain the transformed training feature matrix (i.e., sequentially processed through...). 1. After processing through each ELM layer, a preset activation function is applied after each layer performs a linear transformation, and shallow features are fused through residual connections to finally generate a high-level feature representation H_train); 2. H_train and the true class labels corresponding to each boundary sample in the boundary sample training subset are input into the SVM model for training; 3. Two boundary samples are selected from each boundary sample in the boundary sample training subset using the SOM algorithm, and the Lagrange multipliers of these two boundary samples are optimized. The optimized Lagrange multipliers are then used to update the current Lagrange multipliers of these two boundary samples; 4. Lagrange multipliers satisfying 0 < 0 are selected from the boundary sample training subset. <The transformed training feature matrix of the boundary samples of C is used as support vectors and stored in the support vector set SV, and the bias term of each support vector in the support vector set SV and the global bias term are calculated; 5. Calculate the current objective function value through the SVM objective function (i.e., the Lagrangian dual function); 6. Determine whether the difference between the previous objective function value and the current objective function value is less than the preset threshold tol; 7. When the difference between the previous objective function value and the current objective function value is less than the preset threshold tol, output the bias term and Lagrangian multiplier of each support vector, the global bias term and the support vector set, and stop training; 8. When the difference between the previous objective function value and the current objective function value is greater than or equal to the preset threshold tol, return to step 3 for iterative execution until the difference is less than the preset threshold tol; 9. After training, input the boundary sample feature matrix in the boundary sample verification subset into the multi-layer ELM with residual connection to perform feature transformation through the constructed residual connection formula, so as to obtain a transformed verification feature matrix (that is, sequentially processed by each ELM layer, each layer performs a linear transformation and then applies a preset activation function, and fuses shallow features through residual connections to finally generate the high-level feature representation H_Validation), input H_Validation together with the real category label corresponding to each boundary sample in the boundary sample verification subset into the SVM model for verification, calculate the decision function value of each boundary sample in the boundary sample verification subset through the decision function, determine the predicted category label of each boundary sample in the boundary sample verification subset according to the decision function value, and output the predicted category label of each boundary sample in the boundary sample verification subset; 10. Calculate the current batch prediction accuracy of the initial boundary sample classification model according to the predicted category label of each boundary sample in the boundary sample verification subset and the corresponding real category label, where the accuracy = the number of boundary samples successfully identified in the boundary sample verification subset / the total number of boundary samples in the boundary sample verification subset.

[0039] The residual connection formula is:

[0040]

[0041] In the formula, is the output feature matrix of the l-th layer ELM, is the activation function of the l-th layer ELM, is the output feature matrix of the (l-1)-th layer ELM, is the weight matrix of the l-th layer ELM, is the bias term of the l-th layer ELM, is the projection function, is the projection weight of the l-th layer ELM, is the feature dimension.

[0042] The decision function is:

[0043]

[0044] In the formula, For the i-th boundary sample, Let i be the decision function value for the i-th boundary sample. For the j-th support vector, For the set of support vectors, Let Lagrange multipliers be the values ​​corresponding to the j-th support vector. Let j be the true class label of the j-th support vector. For kernel function, Let J be the transformed feature matrix corresponding to the j-th support vector. Let be the transformed feature matrix of the i-th boundary sample. This is a global bias term. Let N be the bias term for the j-th support vector, and N be the number of support vectors.

[0045] It should also be noted that the preset threshold tol can be set to 1e-3 (i.e., 1). 10 -3 ).

[0046] SVM objective function:

[0047] In the formula, The current objective function value. To train the Lagrange multiplier for the i-th boundary sample within the subset of boundary samples, For the Lagrange multiplier of the j-th boundary sample in the training subset of boundary samples, The true class label of the i-th boundary sample The true class label of the j-th boundary sample For kernel function, Let be the training feature matrix after the transformation of the i-th boundary sample. Let be the training feature matrix after the transformation of the j-th boundary sample.

[0048] The model operation process is as follows: feature data (i.e., boundary sample feature matrix) → ELM layer transformation → SVM classification → probability output.

[0049] It should also be noted that for multi-class classification, the class with the largest decision function value is selected using a "one-to-many" strategy.

[0050] Since the initialization parameters of ELM (i.e., the weight matrix from the input layer to the hidden layer and the bias terms of the hidden layer neurons) are randomly determined, it will cause fluctuations in the classification accuracy of boundary samples in the ELM+SVM model. Assuming the preset number of training rounds is 10, we set the ELM initialization parameters of the group with the highest accuracy after 10 training rounds and save the model of that group. At this time, the ELM+SVM four-class classification model as the teacher model is ready, resulting in the ELM-SVM hybrid teacher model. The output result is in the form of: the probability distribution of the sample class (i.e., sample ID (probability distribution) [e.g., sample 1 (0.7 0.2 0.08 0.02)]). The probability distribution output by the ELM-SVM hybrid teacher model is the normalized result.

[0051] Furthermore, an initial progressive guided distillation sample classification model is constructed based on the EfficientNet model and the ELM-SVM hybrid teacher model (i.e., the initial progressive guided distillation sample classification model, which is to introduce the trained ELM+SVM four-class classification model as the teacher model into the EfficientNet model).

[0052] It should be understood that the initial progressive guided distillation sample classification model is subsequently trained through three training phases: a completely independent training phase, a transition phase, and a pure distillation phase, referring to... Figure 3 , Figure 3 This is a schematic diagram of the model construction process in the first embodiment of the sample classification method based on progressive guided distillation of the present invention. The three stages have different loss functions and training mechanisms to progressively guide model training. The influence of the teacher model (ELM-SVM hybrid teacher model) on the student model (EfficientNet model) is dynamically adjusted by setting dynamic weights for KL divergence loss at each stage. The optimization direction of the fully connected layer weights W and bias term b in the classification head of the EfficientNet model is guided by calculating cross-entropy loss and KL divergence loss.

[0053] S2, During the completely independent training phase, the efficientnet model within the initial progressively guided distillation sample classification model is trained based on all samples, and the parameters of the efficientnet model are adjusted based on the cross-entropy loss value to obtain the efficientnet model to be optimized.

[0054] Furthermore, S2 includes: S2.1, During the completely independent training phase, the sample training set is input into the EfficientNet model for training; S2.2 After training is completed, the sample validation set is input into the EfficientNet model for validation to obtain the predicted class probability distribution of each sample; S2.3 Calculate the cross-entropy loss value based on the predicted class probability distribution and the corresponding true class probability distribution of each sample; S2.4 Determine whether the change between the cross-entropy loss value (the cross-entropy loss value of the current round of the sample validation set) and the previous cross-entropy loss value (the cross-entropy loss value of the previous round of the sample validation set) is less than the preset first-stage learning rate; S2.5 If so, save the model parameters of the EfficientNet model to obtain the EfficientNet model to be optimized; S2.6 If not, then based on backpropagation, adjust the model parameters of the EfficientNet model according to the cross-entropy loss value, and return to S2.1, iterating until the learning rate is less than the preset first-stage learning rate.

[0055] It should also be noted that when entering a certain stage (i.e., the completely independent training stage, the transition stage, or the pure distillation stage), a new round of training is required.

[0056] In this implementation, the first stage (i.e., the completely independent training stage) involves training the backbone network and the classification head first. At this stage, the ELM-SVM hybrid teacher model is not introduced, so that the parameters of the backbone network can be fully optimized without being affected by the teacher model.

[0057] This stage uses only the cross-entropy loss function to calculate the standard classification loss (i.e., the cross-entropy loss value) between the EfficientNet model's predictions and the true labels. The learning rate for the first stage is set to 1e-4 (i.e., 1...). 10 -4 ).

[0058] It should be noted that the typical learning rate range for medium-sized CNN models like EfficientNet is 1e-4 to 1e-3 (i.e., 1... 10 -3 ), 1e-4 can balance training speed and stability.

[0059] The cross-entropy loss value of the sample validation set is monitored. When the change in cross-entropy loss of the sample validation set is less than 1e-4, the model is considered to have converged, training is stopped, the first stage is completed, and the EfficientNet model to be optimized is obtained.

[0060] In the first stage, a relatively large learning rate (1e-4) is used for rapid convergence. When the change in cross-entropy loss value compared to the previous cross-entropy loss value is less than the preset first-stage learning rate (which can be customized by the user), it indicates that the parameter update has entered the fine-tuning stage. At this point, continuing to use the same learning rate may lead to oscillations rather than improvement. Therefore, in the transition stage, the learning rate is reduced to 5e-5 (i.e., 5). 10 -5 ).

[0061] The first-stage learning rate is 1e-4, meaning the maximum change in parameters per update is approximately 1e-4. When the change in loss is less than 1e-4, the effect brought about by the parameter update is already comparable to the update magnitude, indicating that the model has reached the optimization limit at the current learning rate.

[0062] S3, During the transition phase, the parameters of the backbone network within the EfficientNet model to be optimized are frozen.

[0063] S4, train the sample classification model of the initial progressive guided distillation based on all samples, and adjust the parameters of the classification head in the model to be optimized in the EfficientNet model based on the dynamic weights of KL divergence loss.

[0064] Furthermore, S4 includes: S4.1, Train the initial progressive guided distillation sample classification model based on all samples, calculate the dynamic weight of KL divergence loss using a linearly increasing formula, and calculate the total loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model based on the dynamic weight of KL divergence loss. S4.2, Determine whether the dynamic weights of the KL divergence loss meet the preset weight conditions; S4.3, if so, proceed to the pure distillation stage; S4.4 If not, then based on backpropagation, adjust the parameters of the classification head in the EfficientNet model to be optimized according to the total loss value, return to S4.1, and iterate until the preset weight condition is met.

[0065] S4.1 includes: S4.1.1 Input all samples into the EfficientNet model to be optimized to obtain the predicted class probability distribution of each sample; S4.1.2, determine boundary samples and non-boundary samples based on the predicted category probability distribution; S4.1.3, the feature vector of the boundary sample is selected from the feature vectors of all samples by using the hook function; S4.1.4, Generate the boundary sample feature matrix based on the feature vectors of the boundary samples; S4.1.5, input the feature matrix of the boundary samples into the ELM-SVM hybrid teacher model, and output the predicted class probability distribution of the boundary samples; S4.1.6 Calculate the CE loss value based on the predicted class probability distribution of non-boundary samples; S4.1.7 Calculate the KL divergence loss value based on the predicted class probability distribution of the boundary samples in the EfficientNet model to be optimized and the predicted class probability distribution of the boundary samples in the ELM-SVM hybrid teacher model. S4.1.8, determine the current training round of the sample classification model of the initial progressive guided distillation during the transition phase, and calculate the dynamic weight of KL divergence loss according to the current training round using a linearly increasing formula; Linear increasing formula:

[0066] In the formula, For the dynamic weights of KL divergence loss, To preset the step size, This refers to the current training round during the transition phase. The total number of training rounds within a completely independent training phase; S4.1.9, Calculate the total loss value based on the KL divergence loss dynamic weight according to the CE loss value and the KL divergence loss value.

[0067] It should also be noted that in the second stage (i.e., the transition stage): the learning rate is reduced to 5e-5, the entire backbone network is frozen, and the teacher model (i.e., the ELM-SVM hybrid teacher model) is introduced. A gradual knowledge transfer strategy is adopted to avoid model oscillation caused by the sudden introduction of strong distillation signals, allowing the model to gradually adapt to the probability distribution of the teacher model, focusing on basic feature learning (CE-dominated). The dynamic weight of the KL divergence loss is gradually increased using a linearly increasing formula, from α=0.1 to 0.3.

[0068] We incorporate KL divergence loss to measure the total divergence loss between the student model's predicted distribution and the teacher model's predicted distribution.

[0069] The loss function for this stage is:

[0070]

[0071]

[0072] In the formula, L is the total loss value. This is the CE loss value. This represents the KL divergence loss value. The predicted class probability distribution for the inner boundary samples of the ELM-SVM hybrid teacher model. To optimize the probability distribution of predicted class for the boundary samples within the EfficientNet model, and The output logits of the ELM-SVM hybrid teacher model and the EfficientNet model to be optimized are respectively. For temperature coefficient, ( When =1, it is the standard Softmax).

[0073] It should be noted that CE loss ensures the model maintains its basic classification ability (hard objective), while KL loss achieves knowledge transfer (soft objective). This scheme improves the overall classification ability of the model by leveraging the teacher model's ability to classify boundary samples. Sample classification accuracy has higher priority than similarity with the ELM-SVM hybrid teacher model. Therefore, the weight of cross-entropy loss is greater than that of KL divergence loss.

[0074] Preset weight conditions need to be set, and these conditions can be customized by the user, for example... =7:3, the final balance setting of 70%:30% has been shown in multiple studies to achieve the best balance: keeping CE dominant (>50%) avoids complete reliance on teachers, and 30% KL provides sufficient regularization effect.

[0075] S5, during the pure distillation stage, the sample classification model of the initial progressive guided distillation is trained based on all samples, and the KL divergence loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model is calculated.

[0076] S6. Based on the KL divergence loss value, adjust the parameters of the classification head in the EfficientNet model to be optimized (i.e., the weights of the fully connected layers and the bias terms of the fully connected layers) to obtain the sample classification model of progressive guided distillation.

[0077] Furthermore, S6 includes: S6.1 Determine whether the change in KL divergence loss between the KL divergence loss value (i.e., the KL divergence loss value in the current training round during the pure distillation phase) and the previous KL divergence loss value (i.e., the KL divergence loss value in the previous training round during the pure distillation phase) is less than the preset third-stage learning rate. S6.2 If so, save the parameters of the classification head in the EfficientNet model to be optimized to obtain the sample classification model of progressive guided distillation; S6.3 If not, then based on backpropagation, adjust the parameters of the classifier head in the EfficientNet model to be optimized according to the KL divergence loss value, and return to S5 to iterate until it is less than the preset third-stage learning rate.

[0078] In this embodiment, the third stage (i.e., the pure distillation stage): the influence of the teacher model on the student model reaches its maximum, and the learning rate is set to 1e-5 (i.e., 1). 10 -5 This stage also freezes the backbone network parameters, ensuring that the ELM-SVM hybrid teacher model only affects the training of the fully connected layer weights W and bias term b in the classification head. Only the KL divergence loss is calculated, allowing the student model to mimic the teacher model.

[0079] Monitor the KL loss value. If the change is less than 1e-5 of the third-stage learning rate, stop training, and the third stage is complete.

[0080] In the final stage, the main reason affecting the overall classification accuracy of the model is the low classification accuracy of boundary samples. Therefore, the fully connected layer needs to leverage the ELM-SVM hybrid teacher model to improve the classification performance of boundary samples and enhance the classification ability of the fully connected layer for boundary samples.

[0081] S7. Sample classification is performed based on the progressive guided distillation sample classification model.

[0082] In the specific implementation, the sample to be classified is input into the progressive guided distillation sample classification model, and the sample features are extracted through the backbone network. The sample features are then output through the classification head to show the predicted class probability distribution of the sample.

[0083] In this embodiment, an initial progressive guided distillation sample classification model is first constructed. This model is based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model. Then, during the completely independent training phase, the EfficientNet model within the initial progressive guided distillation sample classification model is trained using all samples. The parameters of the EfficientNet model are adjusted to obtain the EfficientNet model to be optimized. The parameters of the backbone network within the EfficientNet model to be optimized are frozen. During the transition phase, the initial progressive guided distillation sample classification model is trained using all samples. The parameters of the classifier head within the EfficientNet model to be optimized are dynamically adjusted based on the KL divergence loss. During the pure distillation phase, the initial progressive guided distillation sample classification model is trained using all samples. The KL divergence loss between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model is calculated. Then, the parameters of the classifier head within the EfficientNet model to be optimized are adjusted based on the KL divergence loss value to obtain the progressive guided distillation sample classification model. Finally, sample classification is performed based on the progressive guided distillation sample classification model. In this embodiment, the model employs a design that freezes the EfficientNet backbone and only fine-tunes the classification head. This retains the powerful feature extraction capabilities of the pre-trained model while effectively transferring knowledge from the teacher model through dynamically adjusted KL divergence loss weights (i.e., dynamic KL divergence loss weights). The first stage establishes basic classification capabilities using pure cross-entropy loss. The second stage introduces progressively weighted KL divergence loss weights (linearly increasing from α=0.1 to 0.3) to smoothly transition the model to knowledge distillation mode. The third stage focuses on optimizing the KL divergence loss, determining convergence when the loss change is less than the learning rate (1e-5). This design keeps the student model lightweight. Ultimately, it demonstrates excellent generalization ability in the ship audio classification task, especially showing improved accuracy compared to traditional training methods in small sample scenarios. The entire system effectively distills the teacher model's dark knowledge into the student model, achieving near-large model performance under computationally limited environments and improving the classification prediction accuracy of samples.

[0084] Reference Figure 3 , Figure 3 This is a structural block diagram of the first embodiment of the sample classification system based on progressive guided distillation of the present invention.

[0085] Module 3001 is used to construct an initial progressive guided distillation sample classification model, which is constructed based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model. Training module 3002 is used to train the efficientnet model within the initial progressive guided distillation sample classification model based on all samples during the completely independent training phase, and adjust the parameters of the efficientnet model based on the cross-entropy loss value to obtain the efficientnet model to be optimized. The training module 3002 is also used to freeze the parameters of the backbone network in the EfficientNet model to be optimized during the transition phase. The training module 3002 is also used to train the sample classification model of the initial progressive guided distillation based on all samples, and to adjust the parameters of the classification head in the model to be optimized based on the dynamic weights of the KL divergence loss. The training module 3002 is also used to train the sample classification model of the initial progressive guided distillation based on all samples during the pure distillation stage, and to calculate the KL divergence loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model. The training module 3002 is also used to adjust the parameters of the classifier head in the EfficientNet model to be optimized based on the KL divergence loss value, so as to obtain a sample classification model with progressive guided distillation. The classification module 3003 is used to classify samples based on the progressive guided distillation sample classification model.

[0086] Other embodiments or specific implementations of the sample classification system based on progressive guided distillation of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0087] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0088] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0090] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A sample classification method based on progressive guided distillation, characterized in that, The method includes the following steps: S1, Construct an initial progressive guided distillation sample classification model, which is based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model; S2, During the completely independent training phase, the efficientnet model within the initial progressive guided distillation sample classification model is trained based on all samples, and the parameters of the efficientnet model are adjusted based on the cross-entropy loss value to obtain the efficientnet model to be optimized. S3, During the transition phase, freeze the parameters of the backbone network within the EfficientNet model to be optimized; S4, Train the sample classification model of the initial progressive guided distillation based on all samples, and adjust the parameters of the classification head in the model to be optimized based on the dynamic weights of KL divergence loss. S5, during the pure distillation stage, the sample classification model of the initial progressive guided distillation is trained based on all samples, and the KL divergence loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model is calculated. S6. Based on the KL divergence loss value, adjust the parameters of the classifier head in the EfficientNet model to be optimized to obtain a sample classification model with progressive guided distillation. S7, perform sample classification based on the progressive guided distillation sample classification model; The samples are ship sonar spectrum samples, photoelectric samples, or image samples; The S4 includes: S4.1, Train the sample classification model of the initial progressive guided distillation based on all samples, calculate the dynamic weight of KL divergence loss using a linearly increasing formula, and calculate the total loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model based on the dynamic weight of KL divergence loss. S4.2, determine whether the dynamic weights of the KL divergence loss meet the preset weight conditions; S4.3, if so, proceed to the pure distillation stage; S4.4 If not, then based on backpropagation, adjust the parameters of the classifier head in the EfficientNet model to be optimized according to the total loss value, return to S4.1, and iterate until the preset weight condition is met; S4.1 includes: S4.1.1 Input all samples into the EfficientNet model to be optimized to obtain the predicted class probability distribution of each sample; S4.1.2, determine boundary samples and non-boundary samples based on the predicted category probability distribution; S4.1.3, the feature vector of the boundary sample is selected from the feature vectors of all samples using the hook function; S4.1.3, Generate a boundary sample feature matrix based on the feature vector of the boundary sample; S4.1.4, Input the boundary sample feature matrix into the ELM-SVM hybrid teacher model, and output the predicted class probability distribution of the boundary sample; S4.1.5, Calculate the CE loss value based on the predicted class probability distribution of the non-boundary samples; S4.1.6 Calculate the KL divergence loss value based on the predicted class probability distribution of the boundary samples within the ELF network model to be optimized and the predicted class probability distribution of the boundary samples within the ELM-SVM hybrid teacher model. S4.1.7, Determine the current training round of the sample classification model of the initial progressive guided distillation during the transition phase, and calculate the dynamic weight of KL divergence loss according to the current training round using a linearly increasing formula; S4.1.8, Calculate the total loss value based on the KL divergence loss dynamic weight according to the CE loss value and the KL divergence loss value.

2. The method as described in claim 1, characterized in that, S2 includes: S2.1, During the completely independent training phase, the sample training set is input into the EfficientNet model for training; S2.2 After training is completed, the sample validation set is input into the EfficientNet model for validation to obtain the predicted class probability distribution of each sample; S2.3 Calculate the cross-entropy loss value based on the predicted class probability distribution and the corresponding true class probability distribution of each sample; S2.4, determine whether the change between the cross-entropy loss value and the previous cross-entropy loss value is less than the preset first-stage learning rate; S2.5 If so, save the model parameters of the EfficientNet model to obtain the EfficientNet model to be optimized; S2.6 If not, then based on backpropagation, adjust the model parameters of the EfficientNet model according to the cross-entropy loss value, and return to S2.1, iterating until the learning rate is less than the preset first-stage learning rate.

3. The method as described in claim 1, characterized in that, The linear increasing formula: In the formula, For the dynamic weights of KL divergence loss, To preset the step size, This refers to the current training round during the transition phase. This refers to the total number of training rounds within the completely independent training phase.

4. The method as described in claim 1, characterized in that, The S6 includes: S6.1, determine whether the change in KL divergence loss between the KL divergence loss value and the previous KL divergence loss value is less than the preset third-stage learning rate; S6.2 If so, save the parameters of the classification head in the EfficientNet model to be optimized to obtain the sample classification model of progressive guided distillation; S6.3 If not, then based on backpropagation, adjust the parameters of the classifier head in the EfficientNet model to be optimized according to the KL divergence loss value, and return to S5, iterating until it is less than the preset third-stage learning rate.

5. The method as described in claim 1, characterized in that, Before S1, it includes: S01, Construct an initial EfficientNet model and an initial ELM-SVM hybrid teacher model. The initial EfficientNet model includes a backbone network and a classification head. The initial ELM-SVM hybrid teacher model is constructed based on a combination of multi-layer ELM and SVM, with residual connections between each layer of ELM. S02, Input the sample training set into the initial EfficientNet model for training; S03. After training is completed, the sample validation set is input into the initial EfficientNet model for validation, the predicted class probability distribution of each sample is obtained, and the boundary sample of the current training round is determined according to the predicted class probability distribution. S04, obtain the feature vector of the boundary sample through the hook function, and store the feature vector of the boundary sample into the boundary sample set; S05, calculate the cross-entropy loss and validation accuracy based on the predicted class probability distribution and the corresponding true class probability distribution of each sample; S06, determine whether the verification accuracy has reached convergence; S07, if so, save the model parameters of the initial efficientnet model to obtain the efficientnet model; S08, if not, then based on backpropagation, adjust the model parameters of the initial EfficientNet model according to the cross-entropy loss value, and return to S02, iteratively execute until the verification accuracy reaches convergence; S09, Train the initial ELM-SVM hybrid teacher model based on the boundary sample set to obtain the ELM-SVM hybrid teacher model.

6. The method as described in claim 5, characterized in that, S09 includes: S09.1, Initialize the parameters of the SVM; S09.2 Initialize the parameters of the multilayer ELM after residual connection; S09.3, input the boundary sample feature matrix of the boundary sample set into the initial ELM-SVM hybrid teacher model for training, adjust the parameters of the SVM, and obtain the boundary sample accuracy; S09.4, Determine whether the current training round is equal to the preset round; S09.5 If not, save the model parameters of the current training round and return to S09.1 until they are equal to the preset round; S09.6 Extract the model parameters corresponding to the highest boundary sample accuracy from multiple sets of model parameters to obtain the ELM-SVM hybrid teacher model.

7. A sample classification system based on progressive guided distillation, characterized in that, The system includes: The module is used to build an initial progressive guided distillation sample classification model, which is based on a combination of the EfficientNet model and the ELM-SVM hybrid teacher model. The training module is used to train the EfficientNet model within the initial progressive guided distillation sample classification model based on all samples during the completely independent training phase, and adjust the parameters of the EfficientNet model based on the cross-entropy loss value to obtain the EfficientNet model to be optimized. The training module is also used to freeze the parameters of the backbone network within the EfficientNet model to be optimized during the transition phase. The training module is also used to train the sample classification model of the initial progressive guided distillation based on all samples, and to adjust the parameters of the classification head in the model to be optimized based on the dynamic weights of the KL divergence loss. The training module is also used to train the initial progressive guided distillation sample classification model based on all samples during the pure distillation stage, calculate the KL divergence loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model, and adjust the parameters of the classifier head in the EfficientNet model to be optimized based on the KL divergence loss value to obtain the progressive guided distillation sample classification model. A classification module is used to classify samples based on the sample classification model described in the progressive guided distillation. The samples are ship sonar spectrum samples, photoelectric samples, or image samples; The training module is also used for: The initial progressive guided distillation sample classification model is trained based on all samples. The dynamic weights of KL divergence loss are calculated using a linearly increasing formula. The total loss value between the EfficientNet model to be optimized and the ELM-SVM hybrid teacher model is calculated based on the dynamic weights of KL divergence loss. Determine whether the dynamic weights of the KL divergence loss meet the preset weight conditions; If so, proceed to the pure distillation stage; If not, then based on backpropagation, adjust the parameters of the classification head in the EfficientNet model to be optimized according to the total loss value, return to the step of training the sample classification model of the initial progressive guided distillation based on all samples, and iterate until the preset weight condition is met. The training module is also used for: Input all samples into the EfficientNet model to be optimized to obtain the predicted class probability distribution of each sample; Boundary samples and non-boundary samples are determined based on the predicted category probability distribution; The feature vector of the boundary sample is selected from the feature vectors of all samples by using the hook function; Generate a boundary sample feature matrix based on the feature vectors of the boundary samples; The boundary sample feature matrix is ​​input into the ELM-SVM hybrid teacher model, and the predicted class probability distribution of the boundary sample is output. The CE loss value is calculated based on the predicted class probability distribution of the non-boundary samples; The KL divergence loss value is calculated based on the predicted class probability distribution of the boundary samples within the EfficientNet model to be optimized and the predicted class probability distribution of the boundary samples within the ELM-SVM hybrid teacher model. Determine the current training round of the sample classification model of the initial progressive guided distillation during the transition phase, and calculate the dynamic weight of KL divergence loss based on the current training round using a linearly increasing formula; The total loss value is calculated based on the KL divergence loss dynamic weights, according to the CE loss value and the KL divergence loss value.

Images