Data stream small sample learning method and system based on federated distillation neural process

Abstract

The application relates to the technical field of machine learning, and discloses a data stream small sample learning method and system based on a federated distillation neural process, which comprises the following steps: training a local attention neural process model by using local private data; transmitting model parameters to a server; the server aggregates the initial global model by using federated averaging; performing local training based on the initial model, inferring a shared common task set, and generating a logits prediction distribution; transmitting the logits distribution to the server; the server aggregates the global prediction distribution by using generalized expert product, and updates the global model parameters by using KL divergence loss distillation; receiving the updated global model, and iteratively optimizing the process of local training, generating and transmitting the logits distribution, and receiving the model. In the small sample, uneven and noisy scene of stream data, the model has good generalization, is not prone to overfitting, can be reliably judged and predicted, has high speed, saves transmission cost, and protects data privacy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, for example to a data stream small sample learning method and system based on federated distillation neural process. BACKGROUND

[0002] In recent years, large models have made continuous breakthroughs in text generation, machine translation, disease diagnosis, intelligent transportation and other fields. The model training effect depends on high-quality large-scale data, but mobile end scenarios have problems such as limited data volume on a single device and lack of labeling. Although centralized training can gather data to improve model capabilities, it faces problems such as privacy leakage and high transmission costs. Federated learning (FL) as a collaborative training method that "leaves data locally and only transmits model updates" has emerged. Through the process of server issuing a shared model, client local training transmitting increments, and server FedAvg aggregating updates, it improves the model prediction ability while protecting privacy.

[0003] Current federated learning clients generally use conventional deep neural networks (DNNs) as local models. Such models rely on static and stable distribution of large-scale data sets for training. In the context of streaming data and small samples, the distribution of data classes changes dynamically over time, and new classes contain only a few labeled samples. This leads to difficulties in updating feature representations in a timely manner and overfitting on sparse samples, limiting the generalization ability of new classes. This problem is particularly pronounced in low-latency scenarios such as online recommendations and real-time monitoring. In addition, although some research has introduced a meta-learning framework (such as MAML and Prototypical Networks) to address these challenges, by training on multiple related tasks to extract a general learning strategy for rapid adaptation to new tasks, such methods rely on deterministic parameter optimization or fixed distance metrics, both of which lack explicit modeling of prediction uncertainty, making it difficult to quantify prediction risk. In high-risk scenarios such as autonomous driving and medical diagnosis, the reliability is insufficient, and the inference phase needs to perform fine-tuning to adapt to new tasks, which increases the computational burden and latency, limiting the efficiency of real-time applications, and cannot meet the actual needs of federated learning in streaming data and small sample scenarios. Therefore, there is an urgent need for a federated learning solution that adapts to streaming data and small sample scenarios to address these issues.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. The summary is not an extensive overview of the embodiments nor is it intended to identify key / critical elements of the embodiments or to delineate the scope of the embodiments. The sole purpose of the summary is to present some concepts of the embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0006] The embodiments of the present disclosure provide a data stream small sample learning method and system based on federated distillation neural process to solve the technical problems that the DNN in the existing federated learning has poor generalization and is easy to overfit in the small sample scene of the stream data, the meta-learning framework lacks uncertainty modeling and needs to be fine-tuned in reverse, resulting in high time delay, and cannot adapt to the scene.

[0007] In some embodiments, the data stream small sample learning method based on the federated distillation neural process comprises:

[0008] training a local attention neural process model based on a local private data set;

[0009] sending parameters of the local attention neural process model to a server to enable the server to aggregate to obtain an initialized global model by using a federated averaging algorithm;

[0010] performing local training based on the received initialized global model issued by the server, performing inference on a shared public task set with the server, and generating logits prediction distribution of the public task set;

[0011] sending the logits prediction distribution to the server to enable the server to aggregate logits prediction distributions of all clients to obtain a global prediction distribution by using a generalized expert product algorithm, and updating global model parameters based on KL divergence loss between the global prediction distribution and a global model output distribution for knowledge distillation;

[0012] receiving the updated global model issued by the server, and repeating the processes of local training, generating and sending logits prediction distribution, and receiving the updated global model to perform iterative optimization.

[0013] In some embodiments, training the local attention neural process model based on the local private data set comprises:

[0014] constructing a meta-training task set, the meta-training task set containing a context set and a query set;

[0015] generating deterministic representation and latent variable uncertainty representation by processing the context set through an encoder of a convolutional neural network combined with an attention mechanism;

[0016] outputting a classification probability distribution by combining the deterministic representation, the latent variable uncertainty representation, and the query set through a decoder;

[0017] A loss function is constructed based on the classification probability distribution and the true label, and the parameters of the local attention neural process model are updated through gradient descent.

[0018] In some embodiments, a deterministic representation is generated, including:

[0019] Local features of the context set are extracted using a convolutional neural network in the encoder, and the calculation formula is as follows:

[0020]

[0021] In the formula, is the context input sample, is the label, is the local feature, is the parameter of the deterministic representation encoder of the local attention neural process model;

[0022] According to the category of the sample, the local features of the samples belonging to the same category are aggregated based on the attention mechanism to obtain the intra-class context representation of each category, and the calculation formula is as follows:

[0023]

[0024] In the formula, represents the category, is the deterministic representation, is the cross-attention weight in the same category;

[0025] The intra-class context representations are spliced to form the final deterministic representation, and the calculation formula is as follows:

[0026]

[0027] In some embodiments, the loss function is an evidence lower bound based objective function, and the calculation formula is as follows:

[0028]

[0029] In the formula, the first term is the cross-entropy loss, and the second term is the KL divergence for regularizing the implicit function representation, is the balance coefficient, is the true label of the query set, is the predicted label, is the query set input, is the deterministic representation, is the latent variable sample, represents the union of the context set and the query set, represents the context set.

[0030] In some embodiments, performing inference on a common task set shared with the server, generating logits prediction distribution of the common task set, comprising:

[0031] generating latent variable uncertain representation through an encoder of the local attention neural process model based on the common context set of the common task set, performing multiple Monte Carlo sampling on the latent variable uncertain representation to obtain multiple latent variable samples;

[0032] for each latent variable sample, performing forward propagation on a common query set of the common task set in combination with the deterministic representation generated based on the common context set of the common task set to obtain a set of logits outputs, wherein the query set of the common task set does not contain labels;

[0033] based on the multiple sets of logits outputs, statistically calculating logits mean and variance of each class to form the logits prediction distribution, the formula being as follows:

[0034]

[0035]

[0036] wherein, is the number of sampling times, is the client on the task the logits value of the class sampled for the first time.

[0037] In some embodiments, the generalized expert product algorithm adopted by the server is an entropy weight-based generalized expert product algorithm, and the server aggregates logits prediction distributions of each client to obtain a global prediction distribution by using the generalized expert product algorithm, comprising:

[0038] the server calculates corresponding entropy weights according to logits prediction distributions uploaded by each client;

[0039] the server takes entropy weights as weights to perform weighted product fusion on logits prediction distributions of each client to obtain the global prediction distribution, and the aggregation formula is as follows:

[0040] , wherein,

[0041] wherein, and are mean and variance of the global prediction distribution, is the number of clients, is the client on the task​ the entropy weight of the client, and is the client the logits mean and variance of the class on the task , is the global precision after aggregation.

[0042] In some embodiments, the entropy weight of each client is calculated according to the logits prediction distribution uploaded by the client, including:

[0043] The SoftMax function is applied to the logits mean uploaded by the client to obtain a normalized soft label probability vector, and the calculation formula is as follows:

[0044]

[0045] In the formula, denotes the prediction probability vector of the client on the task for the class ;

[0046] The Shannon entropy of the corresponding client is calculated according to the soft label probability vector, and the calculation formula is as follows:

[0047]

[0048] In the formula, is used to measure the uncertainty of the client to its prediction result on the task , is the total number of classes;

[0049] The entropy weight is calculated according to the Shannon entropy, wherein the entropy weight is negatively correlated with the Shannon entropy, and the calculation formula is as follows:

[0050]

[0051] In the formula, is a smoothing term used to avoid division by zero errors.

[0052] In some embodiments, knowledge distillation is performed based on the KL divergence loss between the global prediction distribution and the global model output distribution, and the global model parameters are updated, including:

[0053] The server uses the global model to make predictions on the public query set, and obtains the global model output distribution through Monte Carlo sampling;

[0054] The KL divergence between the global model output distribution and the global prediction distribution is calculated as the distillation loss, and the calculation formula is as follows:

[0055]

[0056] wherein, is a global model parameter, is a global model output distribution, is a global prediction distribution;

[0057] minimizing the distillation loss by a gradient descent algorithm to update the parameters of the global model.

[0058] In some embodiments, the latent variable uncertainty representation is subject to a Gaussian distribution, output by a latent variable path of the encoder, containing a mean and a variance, and the calculation formula is as follows:

[0059]

[0060] wherein, is a parameter of a latent variable encoder, is a local feature, and are the mean and variance of the latent variable distribution, respectively.

[0061] In some embodiments, the data stream small sample learning system based on the federated distillation neural process includes a server and a plurality of clients in communication with the server, wherein the client and the server are configured as the data stream small sample learning method based on the federated distillation neural process according to any one of the above.

[0062] The data stream small sample learning method and system based on the federated distillation neural process provided by the embodiments of the present disclosure can achieve the following technical effects:

[0063] The application trains a local attention neural process model based on a local private data set. The model can focus on key features of a category by means of an attention mechanism, and extract task-level prior knowledge from small sample data. The model can not only alleviate the overfitting problem in the small sample scenario of stream data from the model structure level, but also lay a foundation for subsequent uncertainty modeling through latent variable processing, making up for the weak generalization ability of the DNN model in the small sample scenario in traditional federated learning. The local model parameters are sent to the server to obtain an initialized global model by federated averaging algorithm. The initialized global model can be used as a benchmark for multiple client models, avoiding the deviation caused by the large difference between the initial models, and providing a stable starting point for the collaborative training of multiple clients. Then, the initialized global model is used as the basis for local training, and the logits prediction distribution is generated by performing inference on the public task set. In this process, the client does not need to upload the original data or complete model parameters, but only needs to transmit the logits distribution, which significantly reduces the data transmission cost and strengthens the privacy protection. The attention neural process model does not need to be fine-tuned in the reverse direction, but only needs to generate the distribution through forward inference, which greatly reduces the inference delay and adapts to the low latency requirement of the stream data scenario. After the logits prediction distribution is sent to the server, the server aggregates the logits prediction distribution of each client by using the generalized expert product algorithm, which can adaptively identify and filter reliable prediction information, suppress the interference of non-independent and identically distributed data and noise clients, and convert the distribution-level result into global model parameter updates through KL divergence distillation, so that the global model can continuously absorb the effective small sample information of multiple clients and further improve the generalization ability. Finally, the optimization process is repeated iteratively, so that the global model can continuously adapt to new tasks in the dynamic stream data scenario, and continuously consolidate the generalization performance, uncertainty modeling effect and low latency advantage, and comprehensively solve the core pain points of the prior art in this scenario.

[0064] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0065] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute a limitation on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute a proportional limitation, and wherein:

[0066] Figure 1 is a flowchart of a data stream small sample learning method based on federated distillation neural process provided by an embodiment of the disclosure;

[0067] Figure 2 is a training flowchart of a local attention neural process model provided by an embodiment of the disclosure;

[0068] Figure 3 is a generalized expert product aggregation flowchart provided by an embodiment of the disclosure;

[0069] Figure 4 is a knowledge distillation updating global model flow schematic diagram provided by an embodiment of the present disclosure;

[0070] Figure 5 is a distillation flow schematic diagram based on uncertainty distribution level provided by an embodiment of the present disclosure;

[0071] Figure 6 is a system overall architecture and training flow chart provided by an embodiment of the present disclosure;

[0072] Figure 7 is a federated neural process result on FC100 dataset provided by an embodiment of the present disclosure;

[0073] Figure 8 is a federated neural process result on miniImageNet dataset provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0074] In order to enable a person skilled in the art to more fully understand the features and technical contents of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are used only for reference and illustration, and are not intended to limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0075] The terms "first", "second", and the like in the embodiments of the present disclosure are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0076] Unless otherwise specified, the term "a plurality of" means two or more.

[0077] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0078] The term "and / or" is a description of the association relationship between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships.

[0079] The term "corresponding" can refer to an association relationship or a binding relationship, A corresponding to B means that there is an association relationship or a binding relationship between A and B.

[0080] In current federated learning, clients usually use regular deep neural networks (DNN) as local models, and the effective training of DNN highly depends on static and stable large-scale data sets with stable data distribution. However, in the case of streaming data and small samples, the data class distribution changes dynamically over time, and new classes only contain a few labeled samples, which makes it difficult for DNN to update the feature representation in time to adapt to changes, and it is easy to overfit on sparse samples, ultimately limiting its generalization ability to new classes. Although some studies introduce meta-learning frameworks (such as MAML and Prototypical Networks) to address the above challenges, by training and refining a general learning strategy on multiple related tasks to achieve rapid adaptation to new tasks, such methods either rely on deterministic parameter optimization or rely on fixed distance metrics, both lack explicit modeling of prediction uncertainty, making it difficult to quantify prediction risk, and lacking reliability in high-risk scenarios such as autonomous driving and medical diagnosis. At the same time, the inference stage needs to perform backward fine-tuning to adapt to new tasks, which will increase the computational burden and latency, limiting the efficiency of real-time applications, and cannot meet the actual needs of federated scenarios with streaming data and small samples.

[0081] Neural Process (NP) is a meta-learning method that combines deep learning and probabilistic process ideas, which can output prediction results and their uncertainty in a single forward inference. Although neural processes can output results and uncertainties through forward inference without fine-tuning, they are not yet systematically applied to federated scenarios. If we force the use of traditional FedAvg to aggregate neural processes, the posterior distribution of neural processes is a task-level probability representation (non-parametric point estimate), which will face the problems of posterior aggregation difficulty and fragmentation of client task information. In addition, client data often exhibits non-independent and identically distributed (non-IID) characteristics, and some data is of low quality and noisy, so regular FedAvg cannot distinguish the quality of updates and is easily disturbed by noisy clients, ultimately weakening the performance and uncertainty characterization ability of the global model. Therefore, how to design an aggregation and modeling mechanism that takes into account the posterior structure characteristics of neural processes and can adaptively identify and suppress the impact of noisy clients in the federated framework, so that the prediction accuracy and uncertainty reliability of the global model can still be maintained under the constraints of non-independent and identically distributed and small samples, has become a problem that needs to be solved.

[0082] To solve the above problems, the disclosure embodiment provides a data stream small sample learning method and system based on federated distillation neural process. The core idea of the present application is to solve the performance degradation problem of neural process model in the federated scenario caused by the difficulty of aligning the posterior distribution and the interference of noisy clients through a distribution-level federated aggregation and distillation mechanism. This method discards the traditional parameter averaging strategy (such as FedAvg), and instead regards each client as a probability expert, fuses its prediction distribution and distills it to the global model, thereby maintaining the prediction accuracy and uncertainty reliability of the global model under the constraints of non-independent and identically distributed and small sample. The data stream small sample learning method based on federated distillation neural process provided by the disclosure embodiment will be described below with reference to the accompanying drawings.

[0083] Figure 1 is a flowchart of a data stream small sample learning method based on federated distillation neural process provided by the disclosure embodiment. As shown in Figure 1 , the method comprises the following steps:

[0084] S101: training a local attention neural process model based on a local private data set.

[0085] In some embodiments, under the federated meta-learning framework, each of the clients has a private data set These data sets may have significant non-independent and identically distributed characteristics in terms of class distribution, and some client data may be noisy. Figure 2 is a training flowchart of a local attention neural process model provided by the disclosure embodiment. In combination with Figure 2 , when training the local attention neural process model (ANP model), each client first constructs a meta-training task set based on its private data set The meta-training task set contains a context set and a query set , wherein the context set contains small sample data under a new label, which is used to provide task-specific prior knowledge; the query set contains classification task data under a new label, which is used to evaluate the performance of the model on the task. In addition, each client and the server end also has a public task set , which also contains a public context set and a public query set , but the public query set does not contain label content, i.e. only contains .

[0086] In some embodiments, the initial input context pair This refers to the context set samples within the corresponding meta-training task set. The context set is processed by an encoder using a convolutional neural network combined with an attention mechanism to generate deterministic representations and latent variable uncertain representations. As shown in Figure 2, local features of the context set samples are first extracted by CNN features, and then these local features are divided into two branches to generate corresponding representations: one is a deterministic representation, and the other is a latent variable uncertain representation. This completes the encoder's processing of the context set.

[0087] Specifically, the encoder of the local attention neural process model is a convolutional neural network combined with an attention mechanism. In order to map the original input to a comparable representation space, a convolutional neural network is used to extract the context set. The local features of each sample are calculated using the following formula:

[0088]

[0089] In the formula, It is a context input sample. It's a tag. It is a local feature. The parameters of the encoder are used to deterministically represent the local attention neural process model.

[0090] To preserve the differences between categories, local features of samples belonging to the same category are aggregated based on an attention mechanism to obtain the intra-class context representation for each category. The calculation formula is as follows:

[0091]

[0092] In the formula, Indicates category, , It is a deterministic representation. The cross-attention weights are applied within the same category and can be expressed as the following formula:

[0093] ;

[0094] The intra-class context representations are concatenated to form the final deterministic representation, calculated using the following formula:

[0095] .

[0096] Meanwhile, the latent variable path of the local attention neural process model encodes the same contextual representation, resulting in an approximate uncertain representation for the client. This uncertain latent variable representation follows a Gaussian distribution, containing the mean and variance, and is calculated as follows:

[0097]

[0098] wherein, are parameters of the latent variable encoder, are local features, and are the mean and variance of the latent variable distribution, respectively.

[0099] In some embodiments, the latent variable uncertain representation and the deterministic representation are obtained by processing the context set through the encoder, and then the decoder obtains the context embedding in a manner of re-sampling the latent variable uncertain representation , in combination with the deterministic representation and the in the query set, outputs the classification probability distribution and implements the label prediction of the small sample image classification task according to the following formula:

[0100]

[0101] wherein, is the predicted logits value output by the decoder for the query sample after the given client deterministic representation and the latent variable uncertain representation .

[0102] In order to obtain the final predicted label, the query sample output by the decoder is first mapped to the category probability through SoftMax.

[0103] In some embodiments, a loss function is constructed based on the classification probability distribution and the true label, and the parameters of the local attention neural process model are updated through gradient descent. Wherein, the loss function is an objective function based on the Evidence Lower Bound (ELBO), and the calculation formula is as follows:

[0104]

[0105] wherein, the first term is the cross-entropy loss, which is used to optimize the classification performance, the second term is the KL divergence between the posterior distribution of the context hidden function and the prior distribution , which is used to regularize the hidden function representation, is a balance coefficient, is the true label of the query set, is the predicted label, is the query set input, is the deterministic representation, is the latent variable sample, representing the union of the context set and the query set, representing the context set. After obtaining the loss of the training task, the client learns the model parameters by gradient descent with learning rate Parameter update, the update formula is as follows:

[0106]

[0107] For the local private data set The client repeats the above process, calculates the average loss and optimizes the model parameters to adapt to the local data distribution.

[0108] S102: Send the parameters of the local attention neural process model to the server, so that the server aggregates the initialized global model by using the federated average algorithm.

[0109] In some embodiments, after completing the training of the local attention neural process model, each client extracts the encoder and decoder model parameters of the model and only sends the above model parameters to the server, instead of uploading the local private data set, which preliminarily avoids the risk of local data leakage in the initial stage of collaborative training, and meets the core privacy protection demand of federated learning.

[0110] After receiving the local attention neural process model parameters uploaded by all clients participating in the initialization of this round, the server performs parameter aggregation by using the federated average (FedAvg) algorithm. Specifically, for each parameter dimension in the model, the server calculates the mean of the corresponding parameters of all participating clients, combines the small sample training experience of the local attention neural process model of multiple clients into a unified parameter set, and finally obtains the initialized global model. The formula is as follows:

[0111]

[0112] The initialized global model integrates the task prior knowledge of multiple clients in the small sample scenario of streaming data, avoids the model bias caused by the local sparse data of a single client, and provides an initial model benchmark with multi-source information support for subsequent collaborative iterative optimization based on the public task set.

[0113] S103: Based on the initialized global model received from the server, perform local training and execute inference on the public task set shared with the server to generate logits prediction distribution of the public task set.

[0114] In some embodiments, after receiving the server-issued initialized global model, each client first performs local training based on the model, specifically, loads the parameters of the initialized global model into the local attention neural process model, performs light fine-tuning using new small-sample task data in the local private data, quickly adapts to the current client's stream data distribution, while avoiding overfitting caused by limited small-sample data volume, and ensures that the model retains global common knowledge and fits local data characteristics.

[0115] In some embodiments, after completing the local training, the client performs inference operation on the public task set shared with the server, which is composed of a labeled public context set and an unlabeled public query set, and is a unified task carrier for multiple clients and the server to align the prediction logic. During the inference process, first, the core representation is generated based on the public context set of the public task set: the encoder (the structure of convolutional neural network combined with attention mechanism) of the local attention neural process model is used to process the public context set, on one hand, to generate deterministic representation corresponding to the public context set through class grouping and intra-class attention aggregation, and on the other hand, to output latent variable uncertain representation subject to Gaussian distribution through latent variable encoder, in order to fully depict the uncertainty of prediction, the client will perform N times of Monte Carlo sampling on the latent variable uncertain representation to obtain N independent latent variable samples to cover different possible states of the latent variable distribution.

[0116] In some embodiments, for each latent variable sample, the client inputs it and the aforementioned generated deterministic representation into the decoder of the local attention neural process model together, and performs forward propagation calculation combining the public query set (unlabeled) of the public task set, and outputs a set of logits values for each sample class in the public query set. Since each latent variable sample corresponds to a different uncertain state, N times of sampling will obtain N independent sets of logits output, each of which reflects the prediction result under a specific latent variable state.

[0117] In some embodiments, based on the plurality of sets of logits outputs, i.e., the N sets of logits outputs, the client calculates the mean and variance of the logits corresponding to each category in the public query set to form a logits prediction distribution containing the mean and variance, in which the variance is used to measure the consistency of the prediction results of the same query set by the same client under different potential hidden function samplings. When the variance is large, it indicates that the prediction difference between different samplings is significant, which means that the model has not yet formed a stable judgment on this task, and therefore has a high uncertainty. On the contrary, when the variance is small, the prediction results of different samplings are more consistent, indicating that the model is more confident on this task. Based on this indicator, the server can distinguish which clients have more reliable outputs on the current public task, especially in the case of non-independent and identically distributed data distribution and noise in the data of some clients.

[0118] The logits prediction distribution not only reflects the core trend of the prediction result, but also quantifies the uncertainty of the prediction, providing more rich task information for subsequent server-side aggregation. The specific formula is as follows:

[0119]

[0120]

[0121] In the formula, is the number of samplings, is the client on the task the logits value of the category on the i-th sampling.

[0122] S104: Send the logits prediction distribution to the server, so that the server aggregates the logits prediction distributions of each client using the generalized expert product algorithm to obtain a global prediction distribution, and performs knowledge distillation based on the KL divergence loss between the global prediction distribution and the global model output distribution to update the global model parameters.

[0123] Figure 3 is a generalized expert product aggregation flowchart provided by an embodiment of the present disclosure, in combination with Figure 3 When each client uploads the logits prediction distribution containing the mean and variance to the server, the server will start the generalized expert product aggregation based on entropy weight and the KL divergence knowledge distillation process of the present application, and the specific process is as follows:

[0124] In some embodiments, the server first receives the logits prediction distribution uploaded by all clients, i.e., the client on the task for the category​ logits mean with variance To accurately distinguish the reliability of each client's prediction, especially in non-IID and noisy data scenarios, the server calculates the entropy weight according to the following steps:

[0125] Apply the SoftMax function to the logits mean uploaded by the client to obtain the normalized soft label probability vector, and the calculation formula is as follows:

[0126]

[0127] wherein, represents the prediction probability vector of the client on the task for the class , which intuitively reflects the prediction confidence of the client on the class of the task , and is the basis for subsequent uncertainty quantification.

[0128] According to the soft label probability vector , the Shannon entropy of the corresponding client is calculated, and the calculation formula is as follows:

[0129]

[0130] wherein, is used to measure the uncertainty of the client on its prediction result on the task , is the total number of classes, and the higher the entropy value , the stronger the prediction uncertainty of the client on the task, and vice versa, the higher the reliability.

[0131] To let the clients with high reliability (low entropy) contribute more weight in aggregation and ensure that the aggregation process can adaptively filter the interference of unreliable clients, the present application calculates the entropy weight according to the Shannon entropy, wherein the entropy weight is negatively correlated with the Shannon entropy, and the calculation formula is as follows:

[0132]

[0133] wherein, is a smoothing term used to avoid division by zero error when .

[0134] Product of Experts (PoE) is a model combination strategy that fuses multiple probability distributions in the form of multiplication, which will amplify the high confidence components in mechanism. If the server directly adopts the Product of Experts to aggregate the global logits distribution uploaded by the clients based on the public task set, the problem of variance collapse will occur, that is, the variance of the fused distribution is artificially compressed too small. In order to alleviate this problem, in some embodiments, after obtaining the entropy weight of each client , the server weights the logits prediction distribution of each client by the generalized Product of Experts (g-PoE) algorithm based on the entropy weight to obtain the global prediction distribution, generates the global prediction distribution, specifically, the global accuracy after aggregation is calculated , the calculation formula is as follows:

[0135]

[0136] In the formula, M is the total number of clients, Integrating the entropy weight of each client and the reciprocal of its own logits variance, the high reliability and small prediction variance of the client contribute higher aggregation accuracy.

[0137] Then the mean and global variance of the global prediction distribution are calculated, the calculation formula is as follows:

[0138]

[0139]

[0140] In the formula, And are the mean and variance of the global prediction distribution, is the number of clients, is the entropy weight of the client on the task , and are the mean and variance of the logits of the client on the task ,

[0141] The final output global prediction distribution integrates the effective knowledge of multiple clients, and through the dual constraints of entropy weight and variance, suppresses the negative effects of noise and non-independent and identically distributed data.

[0142] Figure 4 is a knowledge distillation update global model flow diagram provided by an embodiment of the present disclosure. Combined with Figure 4 ​After the aggregation of the global prediction distribution, the server needs to distill the knowledge of the distribution into the global model to realize parameter updating, and the specific process is as follows:

[0143] In some embodiments, the server calls the current global model to perform prediction on the public query set, and obtains the output distribution of the global model by consistent Monte Carlo sampling (sampling multiple times on the latent variable uncertainty representation) with the client . The distribution is consistent with the logits distribution generated by the client, and provides a unified probability space for subsequent loss calculation.

[0144] The server calculates the KL divergence between the global model output distribution and the global prediction distribution as the distillation loss, and the calculation formula is as follows:

[0145]

[0146] In the formula, is the global model parameter, is the global model output distribution, is the global prediction distribution. The loss quantifies the difference between the global model output and the aggregated knowledge of multiple clients, ensuring that the distillation process can convert the collaborative knowledge at the distribution level into improvements in model parameters.

[0147] In some embodiments, the distillation loss is minimized by a gradient descent algorithm with a learning rate to iteratively update the global model parameters , and the updated global model is distributed to each client to provide a better collaborative benchmark for the next iteration, and the update formula is as follows:

[0148]

[0149] Figure 5 is a distillation process based on uncertainty distribution level provided by an embodiment of the present disclosure, in combination with Figure 5 , the collaborative interaction process between multiple clients and the server under the federal learning framework of the present application is intuitively presented, and the client (such as client 1, client i) takes the public task set as input, and performs Monte Carlo (MC) sampling on the latent variable uncertainty representation generated by the public task set public context set by means of the local attention neural process model, and outputs the logits prediction distribution containing the mean and variance, such as , the of the client , which carries both the prediction result and the uncertainty. Then each client uploads its logits distribution to the server, which aggregates the multi-client distributions by the entropy-weighted generalized expert product algorithm, and then distills the aggregated distribution knowledge into the global model through knowledge transfer. The global model also iterates continuously with the public task set as input, and finally forms a closed loop of client-generated distribution, server-aggregated distillation, and global model update.

[0150] S105: Receive the updated global model issued by the server, and repeat the process of local training, generating and sending logits prediction distribution, and receiving the updated global model for iterative optimization.

[0151] In some embodiments, after the client receives the updated global model issued by the server, the model is loaded into the local attention neural process model, and a light fine-tuning is performed based on the new small sample task data of the local private data set. This process not only allows the global model to quickly adapt to the current client's stream data distribution characteristics, but also avoids overfitting caused by small sample data due to only light adjustment, ensuring that the model retains the collaborative knowledge of multiple clients while fitting the local data scenario. After completing the local training, the client performs inference on the public task set shared with the server again, generates deterministic representation and latent variable uncertainty representation through the public context set, and after Monte Carlo sampling and forward propagation of the public query set, the logits prediction distribution containing the mean and variance is generated and sent to the server. The server repeats the process of aggregating the logits distribution of each client based on the entropy-weighted generalized expert product algorithm, and then distills the updated global model parameters through KL divergence, and after completing one round of iteration, the updated global model is issued to the client again.

[0152] This iterative optimization process makes the global model continuously absorb the knowledge of small sample tasks from multiple clients in each round, constantly strengthen the generalization ability to dynamic stream data and the accuracy of uncertainty description, and finally achieve the goal of steadily improving model performance and continuously enhancing prediction reliability in the scenario of stream data small sample, non-independent and identically distributed client data, and noise.

[0153] Based on the same inventive concept as the above-described federated distillation neural process-based data stream small sample learning method, the present application also discloses a federated distillation neural process-based data stream small sample learning system in some embodiments. Figure 6 is a system overall architecture and training flowchart provided by an embodiment of the present disclosure. Combined with Figure 6The system comprises a server and a plurality of clients in communication with the server, the clients train an attention neural process model based on local private data and generate a prediction distribution, and the server aggregates the distributions of the clients and distills a global model to form a closed-loop iteration optimization through a generalized expert product algorithm. Specifically, the clients and the server are configured to perform the federated distillation neural process-based data stream small sample learning method described above, which will not be repeated here.

[0154] Figure 7 is a federated neural process result on an FC100 data set provided by an embodiment of the present disclosure, Figure 8 is a federated neural process result on a miniImageNet data set provided by an embodiment of the present disclosure. In combination with Figure 7 (FC100 data set) and Figure 8 (miniImageNet data set), the test accuracy comparison results of the method (G-PoE, i.e., generalized expert product based on entropy weight) of the present application, the traditional federated average method (Fedavg), the federated distribution distillation method (FedDF), and the ordinary expert product method (PoE) in the small sample task scenario are as follows: the horizontal coordinates of “I-1 / N-1 / I-5 / N-5” represent different small sample task settings, wherein I is an incremental class task, N is a new class task, I-1 and N-1 correspond to single sample (1-shot) learning, i.e., there is only one labeled sample for each new class, and I-5 and N-5 correspond to 5-shot learning, i.e., there are five labeled samples for each new class, and the vertical coordinates are test accuracy (Test Accuracy).

[0155] As can be seen from the results, in all task settings of the two data sets, the accuracy of the traditional federated average method Fedavg is the lowest; FedDF and PoE have improved, but are still inferior to G-PoE of the present application. G-PoE has the highest test accuracy in single sample (I-1 / N-1) and 5-shot (I-5 / N-5) tasks, and has a significant advantage. This result confirms the effectiveness of the present application: G-PoE quantifies the reliability of the clients through entropy weight, aggregates the distribution of the generalized expert product, effectively suppresses the interference of non-independent and identically distributed data and noise clients, and at the same time strengthens the model generalization ability in the small sample scenario, so it performs better in the data stream small sample federated learning scenario.

[0156] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method disclosed in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., and can also be a transitory storage medium.

[0157] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments represent only a few of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be changed. Parts and features of some embodiments can be included or replaced by parts and features of other embodiments. Also, the words used in this application are only used to describe the embodiments and not to limit the scope of protection. As used in the description herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or comprises" and the like mean the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device that includes the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the method, product, etc. disclosed in the embodiments, if it corresponds to the method part disclosed in the embodiments, the relevant part can be referred to the description of the method part.

[0158] Those skilled in the art can clearly understand the unit and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner can depend on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0159] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can be merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, the functional units in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

Claims

1. A dataflow few-shot learning method based on federated distillation neural processes, characterized in that, The method includes: A local attention neural process model was trained based on a local private dataset. The parameters of the local attention neural process model are sent to the server so that the server can use a federated averaging algorithm to aggregate and obtain an initial global model. Based on the initial global model received from the server, local training is performed. Inference is then performed on a common task set shared with the server to generate the logits prediction distribution of the common task set. This includes: generating an uncertain latent variable representation using the encoder of a local attention neural process model based on the common context set of the common task set; performing multiple Monte Carlo sampling on the uncertain latent variable representation to obtain multiple latent variable samples; for each latent variable sample, combining the deterministic representation generated based on the common context set of the common task set, forward propagation is performed on the common query set of the common task set to obtain a set of logits outputs, wherein the query set of the common task set does not contain labels; based on multiple sets of logits outputs, the mean and variance of logits for each category are statistically calculated to form the logits prediction distribution, as shown in the following formula: In the formula, For the number of samples, For the client In the mission The above is the first Category of secondary sampling The logits value; The predicted logits distribution is sent to the server, which then uses an entropy-weighted generalized expert product algorithm to aggregate the predicted logits distributions of each client to obtain a global predicted distribution. Knowledge distillation is then performed based on the KL divergence loss between the global predicted distribution and the global model output distribution to update the global model parameters. Specifically, the server uses the generalized expert product algorithm to aggregate the predicted logits distributions of each client to obtain the global predicted distribution. This includes: the server calculating the corresponding entropy weight based on the predicted logits distribution uploaded by each client; and the server using the entropy weight as the weight to perform a weighted product fusion of the predicted logits distributions of each client to obtain the global predicted distribution. The aggregation formula is as follows: , ,in, In the formula, and These are the mean and variance of the global prediction distribution. It is the number of clients. It is a client In the mission Entropy weight on and It is a client In the mission Top category The mean and variance of logits, It is the aggregated global precision; The system receives the updated global model from the server and repeatedly performs the processes of local training, generating and sending the logits prediction distribution, and receiving the updated global model to perform iterative optimization.

2. The data stream few-shot learning method based on federated distillation neural processes according to claim 1, characterized in that, The training of the local attention neural process model based on the local private dataset includes: Construct a meta-training task set, which includes a context set and a query set; The context set is processed by an encoder that combines a convolutional neural network with an attention mechanism to generate deterministic representations and latent variable uncertain representations; The decoder combines the deterministic representation, the latent variable uncertainty representation, and the query set to output a classification probability distribution. A loss function is constructed based on the classification probability distribution and the true labels, and the parameters of the local attention neural process model are updated by gradient descent.

3. The dataflow few-shot learning method based on federated distillation neural processes according to claim 2, characterized in that, The generation of deterministic representations includes: The local features of the context set are extracted using the convolutional neural network in the encoder, calculated as follows: In the formula, It is a context input sample. It's a tag. It is a local feature. The parameters of the encoder are deterministically represented for the local attention neural process model; Based on the sample category, local features of samples belonging to the same category are aggregated using an attention mechanism to obtain the intra-class context representation for each category. The calculation formula is as follows: In the formula, Indicates category, It is a deterministic representation. It involves cross-attention weights within the same category; The intra-class context representations are concatenated to form the final deterministic representation, calculated using the following formula: 。 4. The data stream few-shot learning method based on federated distillation neural processes according to claim 2, characterized in that, The loss function is an objective function based on a lower bound of evidence, and its calculation formula is as follows: In the formula, the first term is the cross-entropy loss, and the second term is the KL divergence used to regularize the implicit function representation. For balance coefficient, These are the actual labels of the query set. It is a predicted label. It is the query set input. It is a deterministic representation. It is a latent variable sample. This represents the union of the context set and the query set. Represents the context set.

5. The data stream few-shot learning method based on federated distillation neural processes according to claim 1, characterized in that, The calculation of entropy weights based on the predicted distribution of logits uploaded by each client includes: The SoftMax function is applied to the mean logits uploaded by the client to obtain a normalized soft-label probability vector. The calculation formula is as follows: In the formula, Indicates client In the mission Top category The predicted probability vector; The Shannon entropy of the corresponding client is calculated based on the soft tag probability vector, using the following formula: In the formula, Used for measuring clients Regarding its mission The uncertainty of the prediction results Total number of categories; The entropy weight is calculated based on the Shannon entropy, wherein the entropy weight is negatively correlated with the Shannon entropy, and the calculation formula is as follows: In the formula, This is a smoothing term used to avoid division by zero errors.

6. The dataflow few-shot learning method based on federated distillation neural processes according to claim 1, characterized in that, The knowledge distillation based on the KL divergence loss between the global prediction distribution and the global model output distribution, and the updating of the global model parameters, includes: The server uses a global model to predict the common query set, and obtains the output distribution of the global model through Monte Carlo sampling; The KL divergence between the global model output distribution and the global prediction distribution is calculated as the distillation loss, and the calculation formula is as follows: In the formula, These are global model parameters. It is the global model output distribution. It is a global prediction distribution; The distillation loss is minimized using the gradient descent algorithm to update the parameters of the global model.

7. The dataflow few-shot learning method based on federated distillation neural processes according to claim 2, characterized in that, The uncertainty of the latent variable is represented by a Gaussian distribution, and is output by the latent variable path of the encoder, including the mean and variance, calculated as follows: In the formula, These are parameters of the latent variable encoder. It is a local feature. and These are the mean and variance of the latent variable distribution, respectively.

8. A dataflow few-shot learning system based on federated distillation neural processes, characterized in that, The system includes a server and multiple clients communicating with the server, wherein the clients and the server are configured to perform the data stream few-shot learning method based on any one of claims 1 to 7.

Images