An online knowledge distillation method and system based on model weight mixing

Abstract

A model weight mixing-based online knowledge distillation method and system, comprising: 1) the server collects labeled data, which is divided into a training set and a validation set; 2) select N student models with the same architecture, and construct a weight mixing model through linear weighting of student parameters; 3) select data augmentation to expand the training set into N+1 parts, which are respectively input into N student models and the weight mixing model to obtain N+1 predicted outputs and their average ensemble prediction; 4) calculate the classification loss of each student and the weight mixing model; 5) calculate the knowledge distillation loss between the output of each student and the ensemble prediction; 6) calculate the parameter optimization target of each student and update the parameters of the N students; 7) repeat steps 3-6, after each round, reconstruct the mixed weight model according to the updated student, and every fixed round Δ, fuse the parameters of the mixed weight model into the student; 8) select the model with the best performance on the validation set and deploy it to the terminal device.

Description

Technical Field

[0001] This invention belongs to the field of online knowledge distillation. It addresses the problem that existing methods rely on carefully designed architectures and complex modules, making it difficult to achieve strong performance on low-resource terminal devices. This invention proposes an online knowledge distillation method and system based on model weight hybridization. Background Technology

[0002] Deep learning has achieved breakthroughs in various tasks by building large-capacity networks pre-trained on massive datasets. To apply highly parameterized models in resource-constrained real-world scenarios, knowledge distillation aims to obtain a compact and efficient student model guided by a large-scale teacher model, achieving model compression. Online knowledge distillation liberates the distillation process from dependence on pre-trained teachers, treating all models as students and enabling mutual learning between models through peer teaching. Existing online knowledge distillation methods improve student model performance by constructing group leaders and online teachers, combining peer features, and other methods to facilitate students acquiring diverse and rich knowledge. However, these strategies focus on designing complex branching structures, requiring additional modules to be added to the backbone network, or relying on a large number of peers and their features, which consumes significant storage space, making deployment difficult on low-resource end devices.

[0003] This invention abandons the paradigm of using heterogeneous knowledge to improve distillation performance, instead focusing on explicitly constraining model generalization to achieve stronger performance with fewer parameters. The generalization ability of a deep model refers to its ability to correctly fit previously unseen data, which has been shown to be reflected by the flatness of the loss landscape basin surrounding the model. According to multi-model optimization theory, if the parameters of multiple models can be linearly added without sacrificing performance, these models tend to lie in the same flat basin of the loss landscape and are more stable to noise. This invention uses a weighted fusion model of multiple students as an ensemble coordinator, employing its classification loss to explicitly measure the generalization ability of the students, thereby obtaining a high-performance student model with low parameter count for deployment on low-capacity terminal devices. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide an online knowledge distillation method and system based on model parameter fusion.

[0005] To obtain a high-performance student model with low parameter count, this invention explores loss minimization and flattening through model weight mixing to achieve efficient online knowledge distillation. This invention constructs a hybrid weight model by linearly weighting the parameters of multiple student models and uses its classification loss to explicitly measure the sharpness around the student loss function. Each student is optimized under the supervision of this loss, reducing the loss across its entire surrounding region, resulting in an ideal, reliable, and more generalizable student model with fewer parameters.

[0006] To achieve the above objectives, the online knowledge distillation method based on model weight mixing described in this invention includes the following steps:

[0007] 1) The server collects labeled data related to the task and divides it into training set and validation set according to a certain ratio;

[0008] 2) Select N deep learning network models with the same architecture {θ1,…,θ N} is used as a student model to conduct online knowledge distillation, and through linear weighting {θ1,…,θ N To construct a weighted mixture model θ hw ;

[0009] 3) By selecting N+1 data augmentation methods based on task and data type, the training set data is expanded into N+1 parts, which are then input into N student models and the weighted mixed model respectively, resulting in N+1 predicted outputs {z}. 1 , ..., z N , z hw}, and their average ensemble prediction z en ;

[0010] 4) Calculate the classification loss for each student and the weighted mixed model.

[0011] 5) Calculate the output z for each student. i With ensemble prediction z en Knowledge distillation loss between

[0012] 6) The parameter optimization objective for each student is divided into three parts: the student's own parameter and the classification loss of the weighted mixed model. and knowledge distillation loss Calculate the gradient based on the optimization objective, and update the parameters of N students respectively;

[0013] 7) Repeat steps 3-6 for E rounds. After each round, rebuild the mixed weight model based on the updated students, and every fixed round Δ, integrate the parameters of the mixed weight model into the students.

[0014] 8) In N+1 models {θ1, ..., θ N θ hw The model that performs best on the validation set is selected and deployed to the terminal device. The terminal device inputs the acquired new data into the trained model to obtain a prediction probability vector, and then completes the relevant task.

[0015] Furthermore, the process of constructing the weighted mixed model by linearly weighting the parameters of N student models in step 2), and reconstructing the mixed weighted model based on the updated students in step 7), is as follows:

[0016]

[0017] w~Dir(1) (2) where, θ hw and θ i Let w represent the parameters of the weighted mixed model and the i-th student, respectively, and N be the total number of student models. w = [w1, ... w2] N ]∈R N It is to satisfy The weight vector. In each training epoch, it is dynamically sampled from the Dirichlet distribution Dir(1), where 1∈R N It is an N-dimensional all-one vector, and is the parameter of the Dirichlet distribution.

[0018] Furthermore, step 3) involves averaging the N+1 prediction outputs to obtain the integrated prediction z. en The process is as follows:

[0019]

[0020] Where z i ,z hw These are the prediction vectors for the i-th student and the mixed weight model, respectively.

[0021] Furthermore, step 4) involves calculating the classification loss for each student and the weighted mixed model, as described in the following formula:

[0022]

[0023] Where y∈{1,…,C} are the true labels of the images, C is the total number of labels in the dataset, and z i ∈R C It is the category prediction output vector for the i-th student.

[0024] Furthermore, step 5) involves calculating the output z for each student. i With ensemble prediction z en Knowledge distillation loss between The process is as follows:

[0025]

[0026] Where σ is the softmax function, which transforms the model output into a probability distribution for class predictions. t is a manually set temperature parameter used to soften the probability distribution for each class. · represents the dot product between vectors.

[0027] Furthermore, the calculation formula for the parameter optimization objective for each student mentioned in step 6) is as follows:

[0028]

[0029] in Let be the classification loss for the i-th student and the weighted mixed model, respectively. η is the knowledge distillation loss of the i-th student, and β are the weighting coefficients of different loss terms that can be adjusted manually.

[0030] Furthermore, in step 7), the parameters of the hybrid weight model are integrated into the students every fixed number of rounds Δ, as described in the following formula:

[0031] θ i =γ*θ hw +(1-γ)*θ i (7)

[0032] Where θ i ,θ hw These are the parameters of the i-th student and the weighted hybrid model, respectively, and γ is the manually set fusion ratio.

[0033] This invention also includes an online knowledge distillation system based on model weight mixing, comprising:

[0034] The training and validation set generation module is used by the server to collect task-related labeled data and divide it into training and validation sets according to a certain ratio.

[0035] The online knowledge distillation module is used to select N deep learning network models {θ1, ..., θ2} with the same architecture. N} is used as a student model to conduct online knowledge distillation, and through linear weighting {θ1, ..., θ N To construct a weighted mixture model θ hw ;

[0036] The prediction output module is used to select N+1 sets of data augmentation methods based on the task and data type, expanding the training set data into N+1 parts, and inputting them into N student models and the weighted mixed model respectively, to obtain N+1 prediction outputs {z}. 1 , ..., z N , z hw}, and their average ensemble prediction z en ;

[0037] The classification loss calculation module is used to calculate the classification loss for each student and the weighted mixed model.

[0038] The knowledge distillation loss calculation module is used to calculate the output z of each student. i With ensemble prediction z en Knowledge distillation loss between

[0039] The student parameter optimization module optimizes the parameters for each student. The optimization objective is divided into three parts: the student's own parameter and the classification loss of the weighted mixed model. and knowledge distillation loss Calculate the gradient based on the optimization objective, and update the parameters of N students respectively;

[0040] The repeated rounds module is used to repeat the calculations of the prediction output module, classification loss calculation module, knowledge distillation loss calculation module, and student parameter optimization module for E rounds. After each round, the mixed weight model is reconstructed based on the updated student, and the parameters of the mixed weight model are integrated into the student every fixed rounds Δ.

[0041] The optimal model deployment module is used to deploy N+1 models {θ1, ..., θ...} N θ hw The model that performs best on the validation set is selected and deployed to the terminal device; the terminal device inputs the acquired new data into the trained model to obtain the prediction probability vector, and then completes the relevant tasks.

[0042] The present invention also includes an online knowledge distillation apparatus based on model weight mixing, comprising a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement an online knowledge distillation method based on model weight mixing according to the present invention.

[0043] The present invention also includes a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements an online knowledge distillation method based on model weight mixing according to the present invention.

[0044] The beneficial effects of this invention are as follows:

[0045] This invention is an online knowledge distillation method and system based on model weight hybridization. It makes full use of multiple student models that receive different data augmentations and introduces the idea of ​​parameter integration into online distillation for the first time. By using a hybrid weight model as an integration coordinator, it integrates the knowledge learned by students under different data change scenarios, thereby expanding the scope of knowledge interaction. At the same time, the seamless integration between model parameters also serves as a form of regularization, standardizing the function and representation of each layer of the student network, and significantly improving their robustness and generalization.

[0046] Compared to traditional offline knowledge distillation, this invention does not rely on a pre-trained, high-parameter teacher model, making it suitable for scenarios with high image update frequency and large variation range. Compared to online distillation methods based on architecture design, this invention avoids decomposing the main architecture and adding extra modules by directly fusing student model parameters, achieving better distillation results simply and effectively with fewer parameters, and is easy to deploy on low-resource terminal devices. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be clearly and completely explained and described below with reference to the accompanying drawings and specific embodiments.

[0049] This invention provides a pedestrian and vehicle recognition method based on online knowledge distillation using model weight hybridization. In an image recognition application scenario, taking pedestrian and vehicle classification as an example, the online knowledge distillation task is to deploy a small-capacity model on low-resource terminal devices to effectively and cost-effectively identify pedestrians and vehicles. Specifically, it includes the following steps:

[0050] 1) Each terminal device collects image data related to pedestrians and vehicles to be identified and uploads it to the server. The server then aggregates and labels the data to generate a dataset, which is divided into a training set and a validation set according to a certain ratio.

[0051] 2) The server selects N deep learning network models {θ1, ..., θ2} with the same architecture. N} is used as a student model to conduct online knowledge distillation, and through linear weighting {θ1, ..., θ N To construct a weighted mixture model θ hw ;

[0052] 3) Select N+1 data augmentation methods related to pedestrians and vehicles, expand the training set data into N+1 parts, and input them into N student models and weighted hybrid models respectively to obtain N+1 prediction outputs {z}. 1 , ..., z N , zhw}, and their average ensemble prediction z en ;

[0053] 4) Calculate the classification loss for each student and the weighted mixed model.

[0054] 5) Calculate the output z for each student. i With ensemble prediction z en Knowledge distillation loss between

[0055] 6) The parameter optimization objective for each student is divided into three parts: the student's own parameter and the classification loss of the weighted mixed model. and knowledge distillation loss Calculate the gradient based on the optimization objective, and update the parameters of N students respectively;

[0056] 7) Repeat steps 3-6 for E rounds. After each round, rebuild the mixed weight model based on the updated students, and every fixed round Δ, integrate the parameters of the mixed weight model into the students.

[0057] 8) In N+1 models {θ1,…,θ N ,θ hw The model that performs best on the validation set is selected and deployed to various terminal devices. The terminal devices input the newly acquired images into the trained model to obtain predicted probability vectors, thereby completing the task of pedestrian or vehicle recognition.

[0058] The process of reconstructing the mixed-weight model based on the updated students in step 2) and step 7) is as follows:

[0059]

[0060] w~Dir(1) (2) where, θ en and θ i Let w represent the parameters of the weighted mixed model and the i-th student, respectively, and N be the total number of student models. w = [w1, ... w2] N ]∈R N It is to satisfy The weight vector. In each training epoch, it is dynamically sampled from the Dirichlet distribution Dir(1), where 1∈R N It is an N-dimensional all-one vector, and is the parameter of the Dirichlet distribution.

[0061] Step 3) involves averaging the N+1 prediction outputs to obtain the ensemble prediction z. en The process is as follows:

[0062]

[0063] Where z i ,z hw These are the prediction vectors for the i-th student and the mixed weight model, respectively.

[0064] Step 4) involves calculating the classification loss for each student and the weighted mixed model, as described in the following formula:

[0065]

[0066] Where y∈{1,…,C} are the true labels of the images, C is the total number of labels in the dataset, and z i ∈R C It is the category prediction output vector for the i-th student.

[0067] Step 5) describes calculating the output z for each student. i With ensemble prediction z en Knowledge distillation loss between The process is as follows:

[0068]

[0069] Where σ is the softmax function, which transforms the model output into a probability distribution for class predictions. t is a manually set temperature parameter used to soften the probability distribution for each class. · represents the dot product between vectors.

[0070] The parameter optimization objective for each student mentioned in step 6) is calculated using the following formula:

[0071]

[0072] in Let be the classification loss for the i-th student and the weighted mixed model, respectively. η is the knowledge distillation loss of the i-th student, and β are the weighting coefficients of different loss terms that can be adjusted manually.

[0073] Step 7) involves fusing the parameters of the mixed-weight model into the student model every fixed number of rounds Δ, as described in the following formula:

[0074] θ i =γ*θ hw +(1-γ)*θ i (7)

[0075] Where θ i ,θ hw These are the parameters of the i-th student and the weighted hybrid model, respectively, and γ is the fusion ratio.

[0076] Specifically, this invention is an online knowledge distillation method based on model parameter fusion. By linearly weighting the parameters of students, an ensemble model is constructed, and its classification loss is incorporated into the training of each student, guiding students to converge to the same flat basin on the loss landscape, thereby obtaining student models with higher generalization and stronger performance.

[0077] Step 1: Terminal devices such as cameras collect images of pedestrians and vehicles to be identified in real-world scenes and upload them to the server. The server aggregates and labels the images to generate a dataset of 300,000 images, containing 15,000 categories. The dataset is divided into training and validation sets in a 9:1 ratio.

[0078] Step 2: The server selects two ResNet50 residual networks as two student models {θ1, θ2} to perform online knowledge distillation. Weight vectors [w1, w2] ~ Dir([1, 1]) are sampled from the Dirichlet distribution and then processed through θ... hw =w1*θ1+w2*θ2 to construct a weighted hybrid model.

[0079] Step 3: Select three data augmentation methods: horizontal flipping, random augmentation, and automatic augmentation. Expand the training set images into three parts and input them into the two student models and the weighted hybrid model respectively, obtaining three predicted outputs {z}. 1 , z 2 , z hw}, and their average ensemble prediction z en =(z 1 +z 2 +z hw ) / 3.

[0080] Step 4: Calculate the classification loss for the two students and the weighted mixed model respectively. in

[0081] Step 5: Calculate the output z of each student. i With ensemble prediction z en Knowledge distillation loss between

[0082] Step 6: Calculate the parameter optimization objectives for both students. The parameters of the two student models are updated accordingly.

[0083] Step 7: Repeat steps 3-6 for 300 rounds. After each round, sample the weight vector [w1, w2] ~ Dir([1, 1]) from the Dirichlet distribution and pass it through θ. hw=w1*θ1+w2*θ2 Update the parameters of the weighted mixture model. Every two rounds, based on θ i =0.5*θ hw +0.5*θ i The parameters of the mixed weight model, i∈{1,2}, are respectively integrated into the two students.

[0084] Step 8, in the 3 models {θ1, θ2, θ... hw The model that performs best on the validation set is selected and deployed to terminal devices such as cameras. The terminal device inputs the newly acquired images into the trained model, resulting in a 15,000-dimensional prediction probability vector. The dimension with the largest value corresponds to the prediction category, thereby completing the task of pedestrian or vehicle recognition.

[0085] This invention also includes an online knowledge distillation system based on model weight mixing, comprising:

[0086] The training and validation set generation module is used by the server to collect task-related labeled data and divide it into training and validation sets in a 9:1 ratio.

[0087] The online knowledge distillation module selects two deep learning network models {θ1, θ2} with the same architecture as student models to perform online knowledge distillation, and constructs a weighted mixture model θ by linearly weighting {θ1, θ2}. hw ;

[0088] The prediction output module is used to select three data augmentation methods based on the task and data type, expanding the training set data into three parts, which are then input into two student models and a weighted hybrid model respectively, to obtain three prediction outputs {z}. 1 ,z 2 ,z hw}, and their average ensemble prediction z en ;

[0089] The classification loss calculation module is used to calculate the classification loss for each student and the weighted mixed model.

[0090] The knowledge distillation loss calculation module is used to calculate the output z of each student. i With ensemble prediction z en Knowledge distillation loss between

[0091] The student parameter optimization module optimizes the parameters for each student. The optimization objective is divided into three parts: the student's own parameter and the classification loss of the weighted mixed model. and knowledge distillation loss Calculate the gradient based on the optimization objective, and update the parameters of the two students respectively;

[0092] The repeated rounds module is used to repeat the calculations of the prediction output module, classification loss calculation module, knowledge distillation loss calculation module, and student parameter optimization module for 120 rounds. After each round, the mixed weight model is reconstructed based on the updated student, and every two rounds, the parameters of the mixed weight model are integrated into the student.

[0093] The optimal model deployment module is used to deploy three models {θ1, θ2, θ...} hw The model that performs best on the validation set is selected and deployed to the terminal device; the terminal device inputs the acquired new data into the trained model to obtain the prediction probability vector, and then completes the relevant tasks.

[0094] The present invention also includes an online knowledge distillation apparatus based on model weight mixing, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement an online knowledge distillation method based on model weight mixing of the present invention.

[0095] The present invention also includes a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements an online knowledge distillation method based on model weight mixing according to the present invention.

[0096] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A pedestrian and vehicle recognition method based on a model weight mixing online knowledge distillation method, comprising the following steps: 1) Each terminal device collects picture data related to pedestrians and vehicles to be identified and uploads it to the server; the server uniformly collects and labels the data set and divides it into a training set and a validation set according to a certain proportion; 2) Select N deep learning network models of the same architecture as student models to conduct online knowledge distillation and construct a weight mixed model by linear weighting ;​ 3) Select N+1 groups of data augmentation methods by task and data type, expand the training set data into N+1 parts, and input them into N student models and a weight mixed model to obtain N+1 prediction outputs , and their average ensemble prediction ; 4) Calculate the classification loss for each student and the weight mixing model ; 5) Compute each student output with integrated prediction between knowledge distillation loss ; 6) The parameter optimization target of each student is divided into three parts, the classification loss of the student itself and the weighted mixed model , and the knowledge distillation loss ; calculate the gradient according to the optimization target, and update the parameters of the N students respectively; 7) Repeat steps 3-6 for E number of rounds, after each round re-construct the mixing weight model according to the updated student and every fixed number of rounds fuse the parameters of the mixing weight model into the student; 8) In N+1 models The model with the best performance on the validation set is selected and deployed to the terminal device. The terminal device inputs the new data into the trained model to obtain a prediction probability vector, and then completes the related task. 9) The terminal device inputs the obtained new picture into the trained model to obtain a prediction probability vector, and then completes the identification task of pedestrians or vehicles; Step 2) The parameters of the linearly weighted N student models are used to construct a weight mixing model, and step 7) The mixed weight model is reconstructed according to the updated student, and the process is as follows: where, and denote the weight mixture model and the parameters of the i-th student, respectively, N is the total number of student models; is a weight vector satisfying At each round of training, is dynamically sampled from the Dirichlet distribution where is an N-dimensional all-ones vector, and is the parameter of the Dirichlet distribution.

2. The method of claim 1, wherein: Step 3) said averaging of N+1 prediction outputs to obtain an ensemble prediction The process is described by the following equation: where are the output prediction vectors of the i-th student and the mixed-weight model, respectively.

3. The method of claim 1, wherein: Step 4) The calculation of the classification loss of each student and the weight mixing model is as follows: wherein is the true label of the picture, C is the total number of classes of the dataset labels, is the class prediction output vector of the i-th student.

4. The method of claim 1, wherein: Step 5) said calculating an output for each student with integrated prediction between knowledge distillation loss whose process is described by the following formula: where is a softmax function that converts the model output into a probability distribution of class predictions; t is a temperature parameter set artificially to soften the probability distribution over each class; denotes the dot product between two vectors.

5. The method of claim 1, wherein: Step 6) The parameter optimization target of each student is calculated as follows: wherein respectively the classification loss of the i-th student and the weighted mixture model, is the knowledge distillation loss of the i-th student, are the weight coefficients of the different loss terms which can be adjusted manually.

6. The method of claim 1, wherein: Step 7) said every fixed wheel Fusing the parameters of the mixed weight model into the student is done as follows: wherein are parameters of the i-th student and the weighted blending model, respectively, is a manually set blending ratio.

7. A pedestrian and vehicle recognition system based on a model weight mixing online knowledge distillation method, characterized by comprising: A training set and validation set generation module for collecting task-related labeled data on the server and dividing it into a training set and a validation set according to a certain proportion; An online knowledge distillation module for selecting N deep learning network models of the same architecture as student models to conduct online knowledge distillation and construct a weight mixed model by linear weighting ;​ The prediction output module is configured to select N+1 groups of data enhancement manners according to the task and the data type, expand the training set data into N+1 parts, and input the N+1 parts into the N student models and the weight mixed model respectively to obtain N+1 prediction outputs. and their average ensemble prediction ; a classification loss calculation module configured to calculate a classification loss of each student and weight mixed model ; a knowledge distillation loss computation module configured to compute a knowledge distillation loss between each student output and the ensemble prediction ;​ a student parameter optimization module, configured to calculate a parameter optimization target of each student, including a classification loss of the student itself and a weighted mixed model , and a knowledge distillation loss ; calculate a gradient according to the optimization target, and update parameters of the N students respectively; The repeated round module is used for repeating the calculation of the prediction output module, the classification loss calculation module, the knowledge distillation loss calculation module and the student parameter optimization module for E rounds. After each round, the mixed weight model is reconstructed according to the updated student, and the mixed weight model is fused into the student every fixed round . The optimal model deployment module is used to deploy N+1 models. The model that performs best on the validation set is selected and deployed to the terminal device; the terminal device inputs the acquired new data into the trained model to obtain the predicted probability vector, and then completes the pedestrian or vehicle recognition related tasks. said constructing a weighted mixture model by linearly weighting said constructing a weighted mixture model by linearly weighting said constructing a weighted mixture model by linearly weighting where, and denote the weight mixture model and the parameters of the i-th student, respectively, and N is the total number of student models; is a weight vector satisfying is dynamically sampled from the Dirichlet distribution at each round of training, where is an N-dimensional all-ones vector, and is the parameter of the Dirichlet distribution.

8. An online knowledge distillation device based on model weight mixing, comprising: A memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the pedestrian and vehicle recognition method based on the model weight mixing online knowledge distillation method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A program is stored thereon, which is executed by a processor to implement the pedestrian and vehicle recognition method based on the model weight mixing online knowledge distillation method of any one of claims 1-6.

Images