A class-incremental neural network model aggregation method based on knowledge distillation

Abstract

The application provides a kind of class incremental neural network model aggregation method based on knowledge distillation, specifically comprising the following steps: step S1, obtains the incremental class information required for aggregation and the specific structure information of the model obtained by aggregation;Step S2, according to the incremental class information and the structure information, select multiple heterogeneous models as the model to be aggregated, and construct the initial aggregation model;Step S3, use unsupervised data to perform class incremental model aggregation on the model to be aggregated and the initial aggregation model, and update the initial aggregation model based on the class incremental distillation loss and feature filtering loss during aggregation until the incremental model is generated;This method supports the aggregation task between heterogeneous models to be aggregated, and only needs a single round of aggregation to make the aggregation model converge, greatly reducing the communication cost in the model aggregation process.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a method for aggregating category incremental neural network models based on knowledge distillation. Background Technology

[0002] With the development of AI technology, intelligent terminals deployed with deep neural networks (such as mobile phones, autonomous vehicles, and IoT terminals) have been widely used in various aspects of life, forming a distributed collective intelligence learning framework in conjunction with edge-cloud collaborative computing methods. These distributed intelligent terminals are deployed in different application scenarios and each possesses independent domain data. Models can learn knowledge from this independent domain data. However, for privacy protection reasons, data from different domains cannot be shared for training new models. To enable aggregated models to learn converged knowledge without accessing the original training data from each domain, a method is needed to achieve the convergence of knowledge from multiple models using only model parameters. This process is called model aggregation.

[0003] The concept of model aggregation was first proposed in the federated learning framework. Currently, mainstream model aggregation methods are mainly based on federated parameter averaging. However, parameter averaging-based model aggregation methods have many limitations. They not only require all participating models to have the same model structure, supported categories, and initialization parameters, but also require a large number of iterations to achieve final convergence, significantly increasing the communication and transmission costs of the aggregation process. Furthermore, in real-world applications, due to the differences in the application scope of models across different data domains, heterogeneous models or models supporting different categories inevitably occur between domains. Parameter-based model aggregation methods, due to their inherent limitations, are difficult to apply to model aggregation tasks in truly complex scenarios.

[0004] The knowledge distillation framework constructs a teacher-student joint learning framework, simultaneously inputting training data into both the teacher and student models. The student model is trained to closely approximate the teacher model by mimicking the soft labels output by the teacher model. An extended knowledge distillation framework incorporating model ensemble concepts can support the simultaneous distillation of knowledge from multiple teacher models into a single student model; this framework is known as multi-teacher distillation. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a model aggregation method that supports different categories and heterogeneous models while protecting privacy. The technical solution adopted is as follows:

[0006] This invention provides a method for categorical incremental neural network model aggregation based on knowledge distillation, characterized by the following steps: Step S1, obtaining the incremental category information required for aggregation and the specific structural information of the aggregated model; Step S2, selecting multiple heterogeneous models as models to be aggregated based on the incremental category information and structural information, and constructing an initial aggregation model; Step S3, using unsupervised data to aggregate the model to be aggregated and the initial aggregation model, and updating the initial aggregation model based on the cumulative loss during aggregation until an incremental model is generated, wherein the cumulative loss includes categorical incremental distillation loss and feature filtering loss.

[0007] The present invention provides a method for categorical incremental neural network model aggregation based on knowledge distillation, which may also have the following technical features: both the model to be aggregated and the initial aggregation model are composed of a feature extraction module and a classifier, and the model to be aggregated also has a feature filtering module.

[0008] The present invention provides a method for categorical incremental neural network model aggregation based on knowledge distillation, which may also have the following technical features, wherein step S3 includes the following sub-steps: Step S3-1, inputting unlabeled data into the model to be aggregated and the initial aggregation model to obtain the prediction output z0-z of the model to be aggregated. n The output z of the initial aggregation model M and intermediate layer features F j Step S3-2: Use the feature filtering module to filter the intermediate layer features F j After filtering, the features are input separately into the model to be aggregated to obtain the intermediate layer features F from the forward propagation output. j logits Step S3-3: Calculate the predicted outputs z0-z respectively. n and The cross-entropy loss on overlapping categories is accumulated to obtain the feature filtering loss; in step S3-4, the category incremental distillation loss between the output of the model to be aggregated and the initial aggregated model on the unlabeled data is calculated respectively; in step S3-5, the accumulated loss is calculated based on the feature filtering loss and the category incremental distillation loss, and the initial aggregated model is continuously updated by backpropagation until the incremental model is generated.

[0009] The present invention provides a category incremental neural network model aggregation system based on knowledge distillation, which may also have the following technical features: during the aggregation process, only the model parameters of the initial aggregation model are updated, and the model to be aggregated is only used for inference.

[0010] The present invention provides a method for aggregation of category incremental neural network models based on knowledge distillation, which also has the following technical feature: the loss function of feature filtering loss is:

[0011]

[0012] In the formula, M represents the aggregated incremental model, and the set of supported tags is L. M The output is z M C1-C n The model to be aggregated represents the model participating in the aggregation process, and supports a tag set of L. i (in The output is z i , The intermediate layer features representing the output of the j-th module of the initial aggregation model are input into the next layer of the model i to be aggregated, resulting in the output.

[0013] The present invention provides a method for aggregating category incremental neural network models based on knowledge distillation, which also has the following technical feature: the loss function of the category incremental distillation loss is:

[0014]

[0015] In the formula, p i (Y = l) is approximately p i (Y=l) in Y∈L M ∩L i The probability under the given conditions, i.e.:

[0016] This invention also provides a category incremental neural network model aggregation system based on knowledge distillation, characterized by comprising: an edge-side and a cloud-side aggregation center. The edge-side is used to transmit aggregation requests and multiple edge-side models trained based on pre-stored edge-side private data to the cloud-side aggregation center, and to update and store the incremental models sent by the cloud-side aggregation center. The aggregation request includes category information that needs to be incremented and the specific structural information of the aggregated model. The cloud-side aggregation center is used to select a model to be aggregated from the multiple received edge-side models according to the aggregation request and to construct an initial aggregation model. It uses publicly available unlabeled data to perform category incremental model aggregation on the model to be aggregated and the initial aggregation model to obtain an incremental model, and sends the aggregated incremental model to the edge-side.

[0017] Invention Function and Effect

[0018] The knowledge distillation-based class incremental neural network model aggregation method of the present invention, by introducing a knowledge distillation-based approach to implement the model aggregation process, enables the aggregated model to converge in a single round of aggregation, greatly reducing the communication cost in the model aggregation process. Simultaneously, the proposed model aggregation method improves the design of the class incremental distillation loss to reduce the impact of missupervision caused by the unsupported classes of the models to be aggregated, thereby improving the performance of the incremental model. Furthermore, it can be applied to scenarios where the models to be aggregated support different classes, i.e., class incremental scenarios. Moreover, by proposing a feature filtering loss based on a feature filtering module, the incremental model can selectively learn knowledge from multiple models to be aggregated, further improving model performance. Additionally, because the feature filtering module can adaptively adjust the size of the feature map, this method can further support aggregation tasks between heterogeneous models to be aggregated, making it more practical in real-world complex scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the framework of the category incremental neural network model aggregation method based on knowledge distillation in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of the category incremental neural network model aggregation method based on knowledge distillation in an embodiment of the present invention;

[0021] Figure 3 This is a flowchart of the polymerization process in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the feature filtering module in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the incremental distillation loss according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the unsupervised category incremental model aggregation process in an embodiment of the present invention. Detailed Implementation

[0025] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following describes the knowledge distillation-based category incremental neural network model aggregation method of this invention in detail with reference to embodiments and accompanying drawings.

[0026] <Example>

[0027] Figure 1 This is a schematic diagram of the framework of the category incremental neural network model aggregation method based on knowledge distillation in an embodiment of the present invention.

[0028] In this embodiment, the knowledge distillation-based category incremental neural network model aggregation method is deployed in the cloud aggregation center of the distributed cloud collaborative learning framework.

[0029] like Figure 1 As shown, the knowledge distillation-based category incremental neural network model aggregation system has both edge and cloud sides. The edge side consists of distributed intelligent edge devices, each considered an independent data domain with its own private data and multiple edge models trained based on that data. Because the categories of the private data differ, the categories supported by the multiple edge models also vary.

[0030] The endpoint is used to send an aggregation request to the cloud side, which includes the category information that needs to be increased, the specific structural information of the aggregated model, and multiple endpoint models.

[0031] The cloud-side aggregation center is used to select the model to be aggregated from multiple received end-side models based on the aggregation application and to build an initial aggregation model. It uses publicly available unlabeled data to perform category incremental model aggregation on the model to be aggregated and the initial aggregation model to obtain an incremental model, and then sends the aggregated incremental model to the end-side.

[0032] Figure 2 This is a flowchart of the category incremental neural network model aggregation method based on knowledge distillation in an embodiment of the present invention.

[0033] like Figure 2 As shown, the categorical incremental neural network model aggregation method uses the knowledge distillation framework to aggregate multiple models, and the process is as follows:

[0034] Step S1: Obtain the incremental category information required for aggregation and the specific structural information of the model obtained from aggregation.

[0035] In this embodiment, an aggregation request is initiated by a specific endpoint, which specifies the category of incremental data to be added in this aggregation and the specific structure of the model obtained from the aggregation. Multiple heterogeneous endpoint models are then uploaded to the cloud-side aggregation center.

[0036] Step S2: Select multiple heterogeneous models as models to be aggregated based on incremental category information and structural information, and construct an initial aggregated model.

[0037] In this embodiment, multiple edge models are selected as models to be aggregated to participate in this round of model aggregation. The cloud-side aggregation center constructs and initializes new models that support incremental categories as the initial aggregation models.

[0038] Step S3: Use unsupervised public data to perform categorical incremental model aggregation on the model to be aggregated and the initial aggregated model until the initial aggregated model converges to generate the incremental model.

[0039] Figure 3 This is a flowchart of the polymerization process in an embodiment of the present invention.

[0040] like Figure 3 As shown, the aggregation process in this embodiment includes the following sub-steps:

[0041] Step S3-1: Input the unlabeled data into the model C0-C to be aggregated. n The predicted outputs z0-z of the model to be aggregated are obtained from the initial aggregated model C′0, respectively. n The output z of the initial aggregation model M and intermediate layer features F j .

[0042] The unlabeled data can be any publicly available unlabeled natural image from any source, as long as it contains the texture of the natural image. Furthermore, the closer the unlabeled data is to the distribution of the original training data used in the aggregation endpoint model, the better the aggregation effect will be in this step.

[0043] Step S3-2: After filtering the intermediate layer features Fj using the feature filtering module, the features are input into the next module of the model to be aggregated, and then propagated forward to the end of the model to be aggregated, outputting the intermediate layer features Fj. j of

[0044] Figure 4 This is a schematic diagram of the feature filtering module in an embodiment of the present invention.

[0045] like Figure 4 As shown, the feature filtering module in this embodiment consists of a simple two-layer sub-network. The first layer is an adaptive pooling layer, used to adjust the size of the intermediate layer feature map output by the aggregation model when aggregating heterogeneous edge models, ensuring it matches the input size of the next module corresponding to the edge model. The input size of the adaptive pooling layer is the size of the feature map output by the current module's aggregation model, and the output size is the size of the feature map input by the next module of the corresponding edge model. The second layer consists of a 1x1 convolutional layer. It ensures that the number of input and output feature channels remains consistent with the number of channels in the aggregation model and the edge model. The 1x1 convolution is used to rearrange the intermediate layer features output by the aggregation model along the channel dimension; the parameters are randomly initialized and continuously updated during the aggregation process.

[0046] Step S3-3: Calculate the predicted outputs z0-z respectively. n and In overlapping categories (L) B ∪L I )∩L i The cross-entropy loss is calculated and accumulated to obtain the feature filtering loss.

[0047] In this embodiment, the feature filtering loss function between the single end-side model and the initial aggregation model is:

[0048]

[0049] In the formula, M represents the aggregated incremental model, and the set of supported tags is L. M The output is z M C1-C n The endpoint model representing the part of the aggregation process supports a tag set of L. i (in The output is z i , The intermediate layer features representing the output of the j-th module of the aggregation model are input into the next layer of the end-side model i to obtain the output.

[0050] In this embodiment, when calculating the feature filtering loss for each batch of unlabeled data, the unlabeled data is simultaneously input into each end-side model and the initial aggregation model. After calculating the feature filtering loss between each end-side model and the initial aggregation model, all feature filtering losses are accumulated and averaged to obtain the final feature filtering loss.

[0051] Steps S3-4: Calculate the aggregation model and the initial aggregation model in L respectively. M ∩L i The cross-entropy loss is calculated and accumulated to obtain the class increment distillation loss.

[0052] In this embodiment, the above-mentioned category incremental distillation loss function is:

[0053]

[0054] When calculating the incremental distillation loss by category, in order to include z i With L M The probability z M In connection, this embodiment will p i (Y = l) is approximately p i (Y=l) in Y∈L M ∩L i The probability under the given conditions, i.e.: This avoids the need for edge models to support inter-class loss calculations when calculating losses, thus reducing the impact of missupervision caused by edge models not supporting classes.

[0055] Figure 5 This is a schematic diagram of incremental distillation loss according to an embodiment of the present invention.

[0056] like Figure 5As shown, in this embodiment, for multiple end-side models supporting different categories, under the premise of substituting the above approximate estimation, the cross-entropy loss on the overlapping categories between each end-side model and the initial aggregation model is calculated in pairs and averaged to obtain the category incremental distillation loss.

[0057] Steps S3-5: For each batch of unlabeled input data, after calculating the feature filtering loss and the class incremental distillation loss respectively, calculate the cumulative loss and use backpropagation to update the parameters of the initial aggregation model and the parameters of all feature filtering modules.

[0058] During the aggregation process described above, all end-side models involved in the aggregation are used only for inference and their model parameters are not updated.

[0059] Figure 6 This is a schematic diagram of the unsupervised category incremental model aggregation process in an embodiment of the present invention.

[0060] like Figure 6 As shown, the category incremental neural network model aggregation method based on knowledge distillation supports category incremental model aggregation among multiple heterogeneous edge models. It undertakes the core task of edge-side knowledge aggregation within an edge-cloud collaborative learning framework that protects edge data privacy. Furthermore, in practical applications, the method proposed in this embodiment can support aggregation between heterogeneous edge models and can be deployed in more complex real-world scenarios.

[0061] In this embodiment, the multi-model aggregation method proposed in this embodiment and existing similar aggregation methods are compared on the public datasets ImageNet and Cifar-100. The experimental results are shown in Table 1, indicating that the multi-model aggregation method of this embodiment is superior to existing similar aggregation methods. The evaluation metric used in Table 1 is Top-1 classification accuracy (%).

[0062]

[0063] Table 1

[0064] Functions and effects of the embodiments

[0065] The category incremental neural network model aggregation method based on knowledge distillation provided in this embodiment introduces a knowledge distillation framework for model aggregation tasks. By having the aggregation model mimic the intermediate layer features and network outputs of multiple models to be aggregated, the aggregation model can simultaneously learn knowledge from multiple models to be aggregated. Furthermore, the aggregation model can achieve convergence through only a single round of aggregation. Therefore, while supporting the aggregation of heterogeneous models to be aggregated, it also significantly reduces the communication cost of transmitting a large number of models required by existing parameter aggregation methods. Moreover, by proposing and using category incremental distillation loss and feature filtering loss designed for category incremental scenarios, it can reduce the performance degradation caused by the models to be aggregated not supporting categories. At the same time, it enables selective learning of multi-data-domain model knowledge during the aggregation process. Therefore, it can significantly improve the performance of the aggregation model in category incremental scenarios and has greater practical value in real-world applications.

[0066] In this embodiment, since the aggregation center only uses publicly available natural images as unsupervised data to train the aggregation model during the aggregation process, it can ensure that the privacy of the original training data of each data domain is not leaked while realizing the aggregation of knowledge from each data domain.

[0067] In this embodiment, the aggregation method proposed in this embodiment can achieve multiple rounds of knowledge aggregation with minimal forgetting during the multi-round iterative aggregation process, making it more suitable for deployment in continuous learning frameworks in practical application scenarios.

[0068] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A category incremental neural network model aggregation system based on knowledge distillation, characterized in that, include: End-side and cloud-side aggregation centers The edge devices are distributed intelligent edge devices, each of which is considered an independent data domain, possessing independent edge-side private data and multiple edge models trained based on the private data. These multiple edge models support different categories. The endpoint is used to transmit the aggregation request and multiple endpoint models trained based on the private data to the cloud-side aggregation center, and to update and store the incremental models sent by the cloud-side aggregation center. The aggregation request includes the category information that requires incremental updates and the specific structural information of the aggregated model. The cloud-side aggregation center is used to select a model to be aggregated from multiple received edge models based on the aggregation request and construct an initial aggregation model. It then uses publicly available unlabeled data to perform incremental class model aggregation on the model to be aggregated and the initial aggregation model, updates the initial aggregation model based on the cumulative loss during aggregation until an incremental model is generated, and sends the aggregated incremental model to the edge. The cumulative loss includes class incremental distillation loss and feature filtering loss. Both the model to be aggregated and the initial aggregation model consist of a feature extraction module and a classifier. The model to be aggregated also has a feature filtering module. The cloud-side aggregation center generates the incremental model in the following manner: Unlabeled data is input into the model to be aggregated and the initial aggregation model to obtain the prediction output of the model to be aggregated. - The output of the initial aggregation model and intermediate layer features Wherein, the unlabeled data are unlabeled natural images; The intermediate layer features are processed using a feature filtering module. After filtering, the features are input into the model to be aggregated to obtain the intermediate layer features output by forward propagation. logits: - ; Calculate the predicted output respectively - and - The feature filtering loss is obtained by summing the cross-entropy losses on overlapping categories; Calculate the incremental distillation loss of categories between the output of the unlabeled data for the model to be aggregated and the initial aggregation model, respectively. The cumulative loss is calculated based on the feature filtering loss and the category incremental distillation loss, and the initial aggregation model is continuously updated using backpropagation until the incremental model is generated. The loss function for the feature filtering loss is: In the formula, Represents the incremental model after aggregation, supporting a set of tags. The output is ; - The model to be aggregated represents the model participating in the aggregation process, and supports a set of tags. The output is , Represents the initial aggregation model. The intermediate layer features output from each module are input into the model to be aggregated. The output obtained after the next layer, The loss function for the incremental distillation loss of the aforementioned category is: In the formula, .

2. The category incremental neural network model aggregation system based on knowledge distillation according to claim 1, characterized in that: in, During the aggregation process, only the model parameters of the initial aggregation model are updated, and the model to be aggregated is used only for inference.

Images