A federated category incremental learning modeling method based on stable feature prototypes
By adopting the federated class incremental learning modeling method based on stable feature prototypes in the federated model model, the catastrophic forgetting problem caused by edge-end data category increment is solved, and the stable aggregation of model parameters and the optimization of feature space are realized. .
Patent Information
- Application Number
- CN202411062873.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-05
Smart Images

Figure CN118586475B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data computing technology, specifically to the field of cloud-edge collaborative federated modeling technology for edge data category increment, and more particularly to a federated category incremental learning modeling method based on stable feature prototypes. Background Art
[0002] In the era of the Internet of Things and big data, deep learning methods have been widely used in dealing with common application tasks such as fault diagnosis component or structure prediction, image classification and segmentation, etc. Existing methods often require offline modeling based on a large amount of data accumulated in historical databases, and then apply them online. However, in a rapidly changing open world, data often appears in a streaming manner, and new categories appear from time to time. At the same time, these methods often only consider the modeling needs of a single device or factory. With the vigorous development of technologies such as industrial Internet of Things and edge computing, the modeling method under the cloud-edge collaborative framework breaks the data silos between factories. Among them, the federated learning method can integrate multi-party knowledge from different devices or factories while protecting data privacy, becoming an attractive modeling paradigm. How to perform incremental learning from non-stationary data streams under the federated learning framework is an important issue in the field of deep learning.
[0003] For a single device or factory, due to hardware limitations, only the most recent data can be stored. As the machine runs, some old data will be discarded and newly generated data will be stored. In actual industries such as manufacturing, raw materials, components, working conditions and corresponding data often increase and change gradually over a long time span. For fault diagnosis, some new categories of fault data or fault pictures will be generated. For QR code recognition, actual applications often face diverse and increasing recognition requirements, and deep learning models need to be trained based on continuously generated new data streams (i.e., tasks). When deep neural networks (DNNs) are trained on samples of new tasks or data distributions, they tend to quickly forget the previously acquired knowledge and their performance deteriorates sharply. This phenomenon is called catastrophic forgetting. Although it is possible to force the model to be iteratively retrained from scratch based on the entire historical data set, this will lead to increasing training time and computational burden and is not a feasible long-term solution.
[0004] Due to the involvement of collaborative training, the problem of incremental federated category modeling under multiple edges is more complex than that under single edges, and there is dual heterogeneity of data in time and space. In the scenario of incremental federated category modeling of data on multiple edges, catastrophic forgetting and model drift problems are superimposed, which spread mutually on multiple edges and continue to propagate with iterative optimization of cloud edges. At this time, tasks are constantly increasing due to the complexity of category combinations. If the knowledge of historical tasks is stored in the model, forgetting is inevitable. If samples are stored or the network is expanded by task, the number of tasks is large, which is not feasible for a long time when storage space is limited. Therefore, it is necessary to pursue the essence of the problem of incremental federated categories and analyze it with categories as the core. The data of the same category on different edges is distributed consistently. If knowledge is stored by category and combined with the self-update of category knowledge, that is, on the one hand, new category knowledge is supplemented, and on the other hand, old category knowledge is maintained, and collaborative optimization of edges based on category knowledge is performed, it is expected to solve the problem of incremental federated category modeling facing dual heterogeneity in time and space. Summary of the invention
[0005] The purpose of the present invention is to provide a federated category incremental learning modeling method based on stable feature prototypes to address the problem that the federated modeling method in the prior art cannot handle the incremental edge data categories, which causes local catastrophic forgetting of the model and the spread of catastrophic forgetting. The present invention establishes a category memory library on the edge to store representative samples of the category, designs a prototype network update strategy based on the playback paradigm, and designs a weighted aggregation strategy in the cloud that uses feature prototypes in a unified feature space as a reference benchmark. It stably optimizes the feature space under the federated framework, realizes the federated update of category knowledge, and solves the problem of collaborative modeling when the edge category is incremental.
[0006] The objective of the present invention is to achieve the following technical solution: a method for incremental learning modeling of federated categories based on stable feature prototypes, implemented in a cloud-edge collaborative scenario, wherein the cloud-edge collaborative scenario includes C edge terminals and a cloud, wherein each edge terminal contains a growing set of seen categories , follow the task Changing data sets , a set of category memories and a prototype network ,in Indicates The edge, Represents the parameters of the prototype network, each sample in the dataset and category memory is represented by express, Representation sample The corresponding category label; the prototype network updated by each edge after the cloud aggregates each task; the method comprises the following steps:
[0007] (1) Randomly initialize the weight parameters of the prototype network preset at each edge;
[0008] (2) Dynamically update the local category memory of each edge to obtain the updated category memory ; Based on the updated category memory The prototype network update strategy based on the playback paradigm is used to update the local prototype network of each edge. Update to obtain the updated prototype network ; The dataset of the current task t and category memory Input them into the prototype network before and after updating respectively to obtain the prototype group of old category features and new category feature prototype group ;
[0009] (3) Prototype group based on old category characteristics and new category feature prototype group , calculate the global prototype interval and local prototype interval in the cloud; calculate the aggregation weight of each edge in the cloud according to the global prototype interval and the local prototype interval, weight the prototype network of each edge according to the aggregation weight of the edge, obtain the global model after model calibration, and send it to each edge;
[0010] (4) Step (2) and step (3) are performed alternately until the preset training rounds are reached;
[0011] (5) After the collaborative optimization is completed, the global model finally received by the edge is used as the final prototype network. The edge uses the data in the local category memory library to input into the final prototype network to obtain the features of each category, and calculates its mean as the category feature prototype for classification tasks.
[0012] Furthermore, the local category memory of each edge is dynamically updated to obtain an updated category memory , including:
[0013] Category memory for each edge According to the newly added current task data of the category, calculate the category center of each category under the current prototype network, select the first P instances of each category closest to its category center as the current sample of the category memory library, and add it to the category memory library In the example above, we obtain the updated category memory .
[0014] Furthermore, the calculation of the category center of each category under the current prototype network specifically includes:
[0015] The current task data of each category is input into the current prototype network to obtain the features corresponding to all task data of the category, and the mean of the features corresponding to all task data is calculated as the category center of the category under the current prototype network.
[0016] Furthermore, the updated category memory library The prototype network update strategy based on the playback paradigm is used to update the local prototype network of each edge. Update to obtain the updated prototype network , including:
[0017] The prototype network of each edge before updating is , under the playback paradigm, a prototype network update strategy based on the playback paradigm is adopted, based on the dataset of the current task t and updated category memory Prototype Network Perform a fixed number of optimizations to obtain an updated prototype network .
[0018] Furthermore, under the playback paradigm, a prototype network update strategy based on the playback paradigm is adopted, based on the dataset of the current task t and updated category memory Prototype Network Perform a fixed number of optimizations to obtain an updated prototype network , including:
[0019] Each time the prototype network of edge i is locally optimized, it starts from the set of seen categories In Randomly select from Type, select Category Memory and the dataset of the current task t Corresponding The data constructs the support set of the current task t , category memory The remaining data in constitutes the query set of the current task t ;in represents the total number of categories seen at edge i;
[0020] Will support set Input the prototype network to obtain the features of each category, and calculate the mean of each category feature as the category feature prototype ;
[0021] Optimize the query set using stochastic gradient descent In the category feature prototype The loss function on , which is the negative log probability function ,in Representation Prototype Network According to the sample The distance from the corresponding category prototype in the feature space generates a distribution about the category, specifically: Before, the negative log probability function Initialize to 0; optimize query set When, for each category For each sample in the query set under , update its negative log probability function according to the following formula :
[0022]
[0023] Among them, P is the number of samples retained for each category in the category memory library, Represents the query set sample x input prototype network The features obtained later, Indicates non-category , express and The Euclidean distance between .
[0024] Furthermore, the dataset of the current task t and category memory Input them into the prototype network before and after updating respectively to obtain the prototype group of old category features and new category feature prototype group , including:
[0025] For edge i, select the dataset of the current task t and category memory Samples of category j in are input to the prototype network before updating In the above example, we obtain a set of features corresponding to category j, calculate the mean of the set of features, and use it as the old feature prototype of category j. We then construct the old category feature prototype group based on the old feature prototypes of all categories. ;
[0026] For edge i, select the dataset of the current task t and category memory Samples of category j in are input to the updated prototype network In the above example, we obtain a set of features corresponding to category j, calculate the mean of the set of features, and use it as the new feature prototype of category j; based on the new feature prototypes of all categories, we construct a new category feature prototype group. .
[0027] Further, the prototype group according to the old category feature and new category feature prototype group , calculate the global prototype interval and local prototype interval in the cloud, including:
[0028] The cloud aggregates the old category feature prototype groups of each edge and new category feature prototype group Then, according to the old category feature prototype group of each edge , calculate the mean of the feature prototypes of each category, and obtain the global category feature prototype group under task t ,in represents the global features of category j under task t, M represents the number of categories that the edge should eventually distinguish; the corresponding prototype interval is calculated based on the two sets of category feature prototype groups, specifically: based on the old category feature prototype group and new category feature prototype group Calculate the local prototype interval of each edge category j , according to the old category feature prototype group and global category feature prototype group Calculate the global prototype interval of each edge category j .
[0029] Furthermore, the calculating of the corresponding prototype intervals according to the two groups of category feature prototype groups specifically includes:
[0030] The two groups of category feature prototypes are and , for category j, calculate the category feature prototype group and The Euclidean distance corresponding to category j in , calculate the category feature prototype group In category j and The mean of the Euclidean distances corresponding to category j in China;
[0031] Two groups of category feature prototypes and Select the old category feature prototype group of edge i and new category feature prototype group When , the local prototype interval of the edge i category j is calculated according to the following formula ;
[0032] Two groups of category feature prototypes and Select the old category feature prototype group of edge i and global category feature prototype group When , the global prototype interval of the edge i category j is calculated according to the following formula :
[0033]
[0034] in, represents the prototype interval of category j, including the local prototype interval of category j and the global prototype interval .
[0035] Furthermore, the aggregation weight of each edge in the cloud is calculated according to the global prototype interval and the local prototype interval, and the prototype network of each edge is weighted and aggregated according to the aggregation weight of the edge to obtain the global model after model calibration, which specifically includes:
[0036] The local prototype intervals of different categories of each edge are summed and input into the Sigmoid activation function to obtain the local credibility of the edge. ;
[0037] The global prototype intervals of different categories of each edge are summed and input into the Sigmoid activation function to obtain the global credibility of the edge. ;
[0038] Get the local credibility of each edge and global credibility The mean of ;
[0039] According to the aggregation weight of each edge, the prototype network of each edge is weighted and aggregated to obtain the global model after model calibration .
[0040] Furthermore, the step (5) specifically includes:
[0041] After the collaborative optimization is completed, the global model finally received by the edge is used as the final prototype network, and the edge uses the data in the local category memory library to input into the final prototype network , get the features of each category, and calculate its mean as the prototype of the category feature;
[0042] For any sample to be classified at the edge , which is input into the final prototype network , get the features corresponding to the sample;
[0043] Calculate the Euclidean distance between the features corresponding to the sample and the feature prototypes of each category, and take the category corresponding to the category feature prototype corresponding to the minimum Euclidean distance as the classification result.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] (1) This paper proposes a federated category incremental learning framework that uses category knowledge to establish stable feature prototypes, which optimizes the local feature space and updates the feature prototypes under category increments, unifies the feature spaces of each edge, and realizes stable aggregation of cloud model parameters;
[0046] (2) The present invention designs a prototype network update strategy supported by a category memory library. By dynamically maintaining the category memory library, the catastrophic forgetting problem in the case of category increment is overcome, and the stability of model optimization is improved by training the feature space related to the category.
[0047] (3) The present invention designs a model calibration strategy based on feature prototype interval, obtains the aggregation weight of each edge model according to the prototype interval, and indirectly realizes the correction of the model aggregation stage by relying on category-related knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A flowchart of the federated category incremental learning modeling method based on stable feature prototypes of the present invention;
[0049] Figure 2 A schematic diagram of C types of incremental edges in one embodiment of the present invention;
[0050] Figure 3 Schematic diagram of the prototype network update strategy based on the playback paradigm at the edge in the present invention;
[0051] Figure 4 It is a schematic diagram of the model calibration strategy based on feature prototype interval in the cloud in the present invention;
[0052] Figure 5 It is a visualization result diagram of the category feature prototype at the edge under different methods; among them, Figure 5 (a) is a visualization result diagram of the category feature prototype of the method of the present invention at edge 1, edge 2, and edge 3. Figure 5 (b) in the figure is the visualization result diagram of the category feature prototype of PN-FedAvg-R at edge 1, edge 2, and edge 3. Figure 5 (c) in the figure is the visualization result of the category feature prototype of PN-FedProx-R at edge 1, edge 2, and edge 3. DETAILED DESCRIPTION
[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0054] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0055] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0056] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.
[0057] The federated category incremental learning modeling method based on stable feature prototypes of the present invention is implemented in a cloud-edge collaboration scenario, which includes C edge terminals and a cloud terminal, such as Figure 2 As shown, Figure 2 is a diagram containing C edges. Each edge contains a growing set of seen categories. , follow the task Changing data sets , a set of category memories and a prototype network ,in Indicates The edge, Represents the parameters of the prototype network, each sample in the dataset and category memory is represented by express, Representation sample The corresponding category labels. The prototype network updated by each edge after the cloud aggregates each task.
[0058] See also Figure 1 , the method specifically comprises the following steps:
[0059] (1) Randomly initialize the weight parameters of the prototype network preset at each edge.
[0060] It should be understood that in the cloud-edge collaboration scenario, each edge end has a preset prototype network, and the weight parameters of the prototype network need to be randomly initialized. Any neural network initialization method can be used for random initialization, such as the Xavier initialization method.
[0061] It should be noted that due to the limitation of storage capacity, the data categories of different edges increase over time, but the number of storage samples is limited, and the data of old tasks is constantly replaced by new task data. In order to achieve model collaborative training under category increment, the present invention involves a prototype network update strategy based on the playback paradigm and a model calibration strategy based on the feature prototype interval. First, the prototype network of each edge is randomly initialized. After the initialization is completed, the local update of the edge and the cloud aggregation are performed alternately, that is, step (2) and step (3) are performed alternately.
[0062] (2) Dynamically update the local category memory of each edge to obtain the updated category memory ; Based on the updated category memory The prototype network update strategy based on the playback paradigm is used to update the local prototype network of each edge. Update to obtain the updated prototype network ; The dataset of the current task t and category memory Input them into the prototype network before and after updating respectively to obtain the prototype group of old category features and new category feature prototype group .
[0063] In this embodiment, the local category memory of each edge is dynamically updated to obtain an updated category memory , specifically including: the category memory of each edge According to the newly added current task data of the category, calculate the category center of each category under the current prototype network, select the first P instances of each category closest to its category center as the current sample of the category memory library, and add it to the category memory library In the example above, we obtain the updated category memory .
[0064] Furthermore, the category center of each category under the current prototype network is calculated, specifically including: inputting the current task data of each category into the current prototype network to obtain the features corresponding to all the task data of the category, and calculating the mean of the features corresponding to all the task data as the category center of the category under the current prototype network.
[0065] In this embodiment, based on the updated category memory The prototype network update strategy based on the playback paradigm is used to update the local prototype network of each edge. Update to obtain the updated prototype network , specifically including: remember that the prototype network of each edge before updating is , under the playback paradigm, a prototype network update strategy based on the playback paradigm is adopted, based on the dataset of the current task t and updated category memory Prototype Network Perform a fixed number of optimizations to obtain an updated prototype network .
[0066] Furthermore, let the prototype network of each edge be updated as , under the playback paradigm, based on the dataset of the current task t and updated category memory Prototype Network Perform a fixed number of optimizations to obtain an updated prototype network ,like Figure 3 As shown, it specifically includes the following sub-steps:
[0067] (2.1a) Each time the prototype network of edge i is locally optimized, it starts from the set of seen categories In Randomly select from Type, select Category Memory and the dataset of the current task t Corresponding The data constructs the support set of the current task t , category memory The remaining data in constitutes the query set of the current task t ;in Represents the total number of categories seen at edge i.
[0068] (2.2a) will support the set Input the prototype network to obtain the features of each category, and calculate the mean of each category feature as the category feature prototype .
[0069] (2.3a) Optimize the query set using stochastic gradient descent (SGD) In the category feature prototype The loss function on , which is the negative log probability function ,in Representation Prototype Network According to the sample The distance from the corresponding category prototype in the feature space generates a distribution about the category, specifically: Before, the negative log probability function Initialize to 0; optimize query set When, for each category For each sample in the query set under , update its negative log probability function according to the following formula :
[0070]
[0071] Among them, P is the number of samples retained for each category in the category memory library, Represents the query set sample x input prototype network The features obtained later, Indicates non-category , express and The Euclidean distance between .
[0072] In summary, the local update of the edge requires the dynamic update of the local category memory of the edge first: in a specific embodiment, the category memory of each edge Add current task data by category, calculate the category center of each category under the current prototype network, select the first P instances of each category closest to the category center as the current sample of the memory library, and obtain the updated category memory library Then, the edge prototype network and feature prototype need to be updated based on the playback paradigm, such as Figure 3 As shown, correspondingly, the prototype network update strategy based on the playback paradigm proposed in the present invention has a specific implementation process as shown in (2.1a) to (2.3a).
[0073] In this embodiment, the data set of the current task t is and category memory Input them into the prototype network before and after updating respectively to obtain the prototype group of old category features and new category feature prototype group , specifically including the following sub-steps:
[0074] (2.1b) For edge i, select the dataset of the current task t and category memory Samples of category j in are input to the prototype network before updating In the above example, we obtain a set of features corresponding to category j, calculate the mean of the set of features, and use it as the old feature prototype of category j. We then construct the old category feature prototype group based on the old feature prototypes of all categories. .
[0075] (2.2b) For edge i, select the dataset of the current task t and category memory Samples of category j in are input to the updated prototype network In the above example, we obtain a set of features corresponding to category j, calculate the mean of the set of features, and use it as the new feature prototype of category j; based on the new feature prototypes of all categories, we construct a new category feature prototype group. .
[0076] (3) Prototype group based on old category characteristics and new category feature prototype group , calculate the global prototype interval and local prototype interval in the cloud; calculate the aggregation weight of each edge in the cloud according to the global prototype interval and the local prototype interval, and weight the prototype network of each edge according to the aggregation weight of the edge to obtain the global model after model calibration, and send it to each edge.
[0077] In this embodiment, according to the old category feature prototype group and new category feature prototype group , calculate the global prototype interval and local prototype interval in the cloud, including: the cloud aggregates the old category feature prototype groups of each edge and new category feature prototype group Then, according to the old category feature prototype group of each edge , calculate the mean of the feature prototypes of each category, and obtain the global category feature prototype group under task t ,in represents the global features of category j under task t, M represents the number of categories that the edge should eventually distinguish; the corresponding prototype interval is calculated based on the two sets of category feature prototype groups, specifically: based on the old category feature prototype group and new category feature prototype group Calculate the local prototype interval of each edge category j , according to the old category feature prototype group and global category feature prototype group Calculate the global prototype interval of each edge category j .
[0078] Further, the corresponding prototype interval is calculated according to the two sets of category feature prototype groups, specifically including: the two sets of category feature prototype groups are and , for category j, calculate the category feature prototype group and The Euclidean distance corresponding to category j in , calculate the category feature prototype group In category j and The mean of the Euclidean distances corresponding to the non-class j; the prototype groups of the two class features and Select the old category feature prototype group of edge i and new category feature prototype group When , the local prototype interval of the edge i category j is calculated according to the following formula ; Two groups of category feature prototype groups and Select the old category feature prototype group of edge i and global category feature prototype group When , the global prototype interval of the edge i category j is calculated according to the following formula :
[0079]
[0080] in, represents the prototype interval of category j, including the local prototype interval of category j and the global prototype interval .
[0081] In this embodiment, the aggregation weight of each edge in the cloud is calculated according to the global prototype interval and the local prototype interval, and the prototype network of each edge is weighted and aggregated according to the aggregation weight of the edge to obtain the global model after model calibration, such as Figure 4 As shown, it specifically includes the following sub-steps:
[0082] (3.1) The local prototype intervals of different categories of each edge are summed and input into the Sigmoid activation function to obtain the local credibility of the edge .
[0083] Furthermore, the purpose of the Sigmoid activation function is to convert the input value Mapped to an output value between 0 and 1, expressed as:
[0084]
[0085] in, Represents the Sigmoid activation function.
[0086] (3.2) The global prototype intervals of different categories of each edge are summed and input into the Sigmoid activation function to obtain the global credibility of the edge .
[0087] It should be understood that the model credibility of the prototype network update can be calculated based on the prototype intervals under different categories of each edge. The prototype intervals are divided into local prototype intervals and global prototype intervals. The corresponding credibility needs to be calculated, that is, the local credibility and global credibility corresponding to the above steps (3.1) and (3.2).
[0088] (3.3) Take the local credibility of each edge and global credibility The mean of .
[0089] (3.4) According to the aggregation weight of each edge, the prototype network of each edge is weighted and aggregated to obtain the global model after model calibration .
[0090] It should be understood that after the calculation of the global prototype interval and the local prototype interval is completed, the model calibration strategy based on the feature prototype interval is executed in the cloud, and its specific implementation process is as follows Figure 4 As shown, that is, the above steps (3.1) to (3.4).
[0091] (4) Steps (2) and (3) are performed alternately until the preset training rounds are reached.
[0092] (5) After the collaborative optimization is completed, the global model finally received by the edge is used as the final prototype network. The edge uses the data in the local category memory library to input into the final prototype network to obtain the features of each category, and calculates its mean as the category feature prototype for classification tasks.
[0093] (5.1) After the collaborative optimization is completed, the global model finally received by the edge is used as the final prototype network, and the edge uses the data in the local category memory library to input into the final prototype network , get the features of each category, and calculate its mean as the category feature prototype.
[0094] (5.2) For any sample to be classified at the edge , which is input into the final prototype network , and obtain the features corresponding to the sample.
[0095] (5.3) Calculate the Euclidean distance between the features corresponding to the sample and the feature prototypes of each category, and take the category corresponding to the category feature prototype corresponding to the minimum Euclidean distance as the classification result.
[0096] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.
[0097] In this embodiment, the classification of the color image dataset CIFAR10 is taken as an example, which contains 10 categories, namely airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks. Each category has 6,000 pictures, and the size of each picture is In this data set, 50,000 samples are used as training sets and 10,000 samples are used as test sets. In this cloud-edge collaboration scenario, there are three edge terminals, such as Figure 2As shown, at this time . A single task per edge Three random images of different categories are collected, with 100 samples in each category. Assume that the edge tasks change in time synchronously, and the model is updated synchronously after the cloud model is corrected. It should be noted that the edge model training is parallel, the cloud model is updated and sent to the edge, and the edge model update can be performed asynchronously.
[0098] In the cloud-edge collaboration scenario, the federated category incremental learning modeling method based on stable feature prototypes specifically includes the following steps:
[0099] (1) Randomly initialize the weight parameters of the prototype network at each edge.
[0100] (2) Dynamically update the local category memory of each edge: The category memory of each edge According to the category, the current task data is added, the category center of each category under the current prototype network is calculated, and the first P instances of each category closest to the category center are selected as the current samples of the category memory library to obtain the updated category memory library. .
[0101] (3) Local update of prototype network based on playback paradigm: The prototype network model of each edge before update is , under the playback paradigm, the prototype network update strategy based on the playback paradigm is adopted, and the prototype network is updated to . The dataset of the current task t and category memory Input into the prototype network model before and after updating to obtain the prototype group of old category features and new category feature prototype group .
[0102] (4) Calculate the global prototype and prototype interval in the cloud: After the cloud aggregates the category feature prototypes of each edge, it calculates the global prototype and prototype interval based on the old category feature prototypes of each edge. ( , calculate the mean of the feature prototypes of each category and get the task The global category feature prototype group under ; According to the two sets of category feature prototype groups, the corresponding prototype interval can be calculated. Specifically, the category of each edge is calculated according to the new and old sets of category feature prototype groups The local prototype interval , the edge categories are calculated based on the old category feature prototype group and the global category feature prototype group The aggregate prototype interval .
[0103] (5) Perform prototype interval-based model calibration in the cloud: According to the prototype intervals of different categories at each edge, the model credibility of the prototype network update is calculated. Specifically, the local credibility is calculated based on the local prototype interval. , the global credibility is calculated based on the aggregate prototype interval ; Take the average of the local credibility and global credibility of each edge as the weight for the edge model aggregation , according to the aggregation weight of the edge Weighted aggregation of the prototype network of each edge to obtain the global model And send it to each edge.
[0104] (6) Step (2)-step (3) and step (4)-step (5) are performed alternately and iterated 500 times to complete the collaborative optimization.
[0105] (7) After the collaborative optimization is completed, the edge uses the data in the local category memory library to input the final prototype network model distributed in the cloud to obtain the features of each category and calculate its mean as the category feature prototype for the classification task.
[0106] Based on the above-mentioned color image dataset CIFAR10, an image classification model constructed using the method described in the present invention can be obtained. The image classification model can be used online for image classification tasks, and the classification effect can be verified on a test set.
[0107] In the case of multi-edge data category increments, the accuracy of model classification is shown in Table 1. FedAvg and FedProx are used as comparison methods for federated aggregation. "PN-FedAvg-R" represents the FedAvg algorithm with the designed edge replay paradigm added, and "PN-FedProx-R" represents the FedProx algorithm with the designed edge replay paradigm added. Compared with the former method, the model parameters are corrected for the client drift problem during cloud aggregation. "R w / o" means that the edge replay paradigm in the designed method is ablated, that is, only the designed weighted aggregation strategy based on prototype interval is retained. Table 1 specifically shows the classification effect of the model on the current new classes and old classes at different edges. It can be found that the method of the present invention is superior to the comparison method and the method in the ablation experiment.
[0108] Table 1. Comparison of classification accuracy (%) between the present invention and other federated modeling methods when multi-edge category increments
[0109]
[0110] Figure 5The visualization results of the feature prototypes of each edge category in the method proposed in this invention and PN-FedAvg-R and PN-FedProx-R with playback paradigm after principal component analysis (PCA) dimensionality reduction are shown. The origins of different shades in the figure represent prototypes of different categories. Figure 5 It can be seen that compared with other methods, the relative positions of the feature prototypes of each edge category obtained by the method of the present invention are more consistent, indicating that the weighted aggregation strategy based on the prototype interval makes the feature spaces of different edges tend to be consistent during federated optimization. Figure 5 The visualization results of PN-FedAvg-R at edge 1, edge 2, and edge 3 shown in (b) and Figure 5 From the visualization results of PN-FedProx-R on edge 1, edge 2, and edge 3 shown in (c), it can be seen that PN-FedAvg-R and PN-FedProx-R have category feature prototypes that are too close in edge 2 and edge 3, respectively, while the feature prototypes of the method proposed in the present invention are more separated, as shown in Figure 1. Figure 5 As shown in (a) in , the classification accuracy is higher in Table 1.
[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A federated category incremental learning modeling method based on stable feature prototypes, implemented in a cloud-edge collaboration scenario, characterized by: The cloud-edge collaboration scenario includes C edge devices and a cloud, where each edge device contains a growing set of seen categories. Datasets that vary with task t A set of category memories and a prototype network f φ,i (·), where i represents the i-th edge, φ represents the parameters of the prototype network, each sample in the data set and the category memory is represented by (x, y), y represents the category label corresponding to the sample x, and the data set is a color image data set; the prototype network after each edge is updated after each task is aggregated in the cloud; the method includes the following steps: (1) Randomly initialize the weight parameters of the prototype network preset at each edge; (2) Dynamically update the local category memory of each edge to obtain the updated category memory Based on the updated category memory The prototype network update strategy based on the playback paradigm is used to update the local prototype network of each edge. Update to get the updated prototype network The dataset of the current task t and category memory are input into the prototype network before and after the update to obtain the prototype group of the old category features and new category feature prototype group (3) Based on the prototype group of old category characteristics and new category feature prototype group Calculate the global prototype interval and local prototype interval in the cloud; calculate the aggregation weight of each edge in the cloud based on the global prototype interval and the local prototype interval, weight the prototype network of each edge according to the aggregation weight of the edge, obtain the global model after model calibration, and send it to each edge; (4) Step (2) and step (3) are performed alternately until a preset number of training rounds is reached; (5) After the collaborative optimization is completed, the global model finally received by the edge is used as the final prototype network. The edge uses the data in the local category memory library to input into the final prototype network to obtain the features of each category, and calculates its mean as the category feature prototype for classification tasks.
2. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 1 is characterized in that: The local category memory of each edge is dynamically updated to obtain an updated category memory Specifically include: Category memory for each edge According to the newly added current task data of the category, calculate the category center of each category under the current prototype network, select the first P instances of each category closest to its category center as the current sample of the category memory library, and add it to the category memory library In the example above, we obtain the updated category memory 3. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 2 is characterized in that: The calculation of the category center of each category under the current prototype network specifically includes: The current task data of each category is input into the current prototype network to obtain the features corresponding to all task data of the category, and the mean of the features corresponding to all task data is calculated as the category center of the category under the current prototype network.
4. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 1 is characterized in that: The updated category memory The prototype network update strategy based on the playback paradigm is used to update the local prototype network of each edge. Update to get the updated prototype network Specifically include: The prototype network of each edge before updating is Under the playback paradigm, a prototype network update strategy based on the playback paradigm is adopted, based on the dataset of the current task t and updated category memory Prototype Network Perform a fixed number of optimizations to obtain an updated prototype network 5. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 4 is characterized in that: In the playback paradigm, the prototype network update strategy based on the playback paradigm is adopted, based on the dataset of the current task t and updated category memory Prototype Network Perform a fixed number of optimizations to obtain an updated prototype network Specifically include: Each time the prototype network of edge i is locally optimized, it starts from the set of seen categories In Randomly select K c Type, select Category Memory and the dataset of the current task t Corresponding K c The data constructs the support set of the current task t Category Memory The remaining data in constitutes the query set of the current task t in represents the total number of categories seen at edge i; Will support set Input the prototype network, obtain the features of each category, and calculate the mean of each category feature as the category feature prototype p[j]; Optimize the query set using stochastic gradient descent The loss function on the category feature prototype p[j] is the negative logarithmic probability function Represents the prototype network f φ A distribution about the category is generated based on the distance between the sample (x, y) and the corresponding category prototype in the feature space, specifically: in the optimized query set Before, the negative log probability function Initialize to 0; optimize query set When , for each sample in the query set under each category j, its negative logarithmic probability function is updated according to the following formula Among them, P is the number of samples retained for each category in the category memory library, f φ (x) represents the query set sample x input into the prototype network f φ The features obtained after that, j′ represents non-category j, d(f φ (x),p[j]) represents f φ The Euclidean distance between (x) and p[j].
6. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 1 is characterized in that: The dataset of the current task t and category memory are input into the prototype network before and after the update to obtain the prototype group of the old category features and new category feature prototype group Specifically include: For edge i, select the dataset of the current task t and category memory Samples of category j in are input to the prototype network before updating In the above example, we obtain a set of features corresponding to category j, calculate the mean of the set of features, and use it as the old feature prototype of category j. We then construct the old category feature prototype group based on the old feature prototypes of all categories. For edge i, select the dataset of the current task t and category memory Samples of category j in are input to the updated prototype network In the above example, we obtain a set of features corresponding to category j, calculate the mean of the set of features, and use it as the new feature prototype of category j; construct a new category feature prototype group based on the new feature prototypes of all categories.
7. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 1 is characterized in that: The prototype group according to the old category characteristics and new category feature prototype group The global prototype interval and local prototype interval are calculated in the cloud, including: The cloud aggregates the old category feature prototype groups of each edge and new category feature prototype group Then, according to the old category feature prototype group of each edge Calculate the mean of the feature prototypes of each category to obtain the global category feature prototype group under task t where gp t [j] represents the global feature of category j under task t, and M represents the number of categories that the edge should eventually distinguish. The corresponding prototype interval is calculated based on the two sets of category feature prototype groups, specifically: and new category feature prototype group Calculate the local prototype interval of each edge category j According to the old category characteristic prototype group and the global category feature prototype group GP t Calculate the global prototype interval of each edge category j 8. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 7 is characterized in that: The step of calculating the corresponding prototype intervals according to the two sets of category feature prototype groups specifically includes: The two groups of category feature prototypes are respectively a and p b , for category j, calculate the category feature prototype group p a and p b The Euclidean distance d corresponding to category j in + [j](p a ,p b ), calculate the category feature prototype group p a In categories j and p b The mean of the Euclidean distances corresponding to category j in China; Two groups of category feature prototype groups p a and p b Select the old category feature prototype group FP of edge i i t and new category feature prototype group UP i t When , the local prototype interval of the edge i category j is calculated according to the following formula Two groups of category feature prototype groups p a and p b Select the old category feature prototype group FP of edge i i t and the global category feature prototype group GP t When , the global prototype interval μ of the edge i category j is calculated according to the following formula i agg,t [j]: Among them, μ[j] represents the prototype interval of category j, including the local prototype interval of category j and the global prototype interval 9. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 1 is characterized in that: The calculation of the aggregation weight of each edge in the cloud according to the global prototype interval and the local prototype interval, weighted aggregation of the prototype network of each edge according to the aggregation weight of the edge, and obtaining the global model after model calibration specifically includes: The local prototype intervals of different categories of each edge are summed and input into the Sigmoid activation function to obtain the local credibility of the edge. The global prototype intervals of different categories of each edge are summed and input into the Sigmoid activation function to obtain the global credibility of the edge. Get the local credibility of each edge and global credibility The mean of According to the aggregation weight of each edge, the prototype network of each edge is weighted and aggregated to obtain the global model after model calibration 10. The method for incremental learning modeling of federated categories based on stable feature prototypes according to claim 1 is characterized in that: The step (5) specifically comprises: After the collaborative optimization is completed, the global model finally received by the edge is used as the final prototype network, and the edge uses the data in the local category memory library to input into the final prototype network Get the features of each category and calculate their mean as the prototype of the category feature; For any sample to be classified (x, y) at the edge, input it into the final prototype network Get the features corresponding to the sample; Calculate the Euclidean distance between the features corresponding to the sample and the feature prototypes of each category, and take the category corresponding to the category feature prototype corresponding to the minimum Euclidean distance as the classification result.
Citation Information
Patent Citations
Federal incremental learning method based on feature prototype
CN114861936A
Training method for improving new and old category distinction degree of existing category incremental learning
CN116089883A
