A federated semi-supervised learning system and method based on adaptive weights and applications thereof

Abstract

The present application relates to a kind of federal semi-supervised learning system based on adaptive weight, method and its application, train the labeled data and unlabeled data in any client in federal learning, utilize the loss of unlabeled data to update full-connection module in reverse and generate new weight, obtain new loss and repeat updating full-connection module in reverse, repeat training, carry out semi-supervised learning;System is managed globally with server and is configured iteration updated main model, with client side obtains local data, downloads main model from server and utilizes local data to train main model, update, upload updated main model to server;It is applied to semi-supervised learning between heterogeneous client and its heterogeneous data.The present application effectively improves the flexibility of model in the case of different data domains in multiple clients in semi-supervised training process, increases the sufficiency of unlabeled data utilization in model training process, and is robust;It is suitable for multiple data distribution scenarios, and privacy problems are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of transmission of digital information, such as telegraph communication, and in particular to a federated semi-supervised learning system based on adaptive weights, a method thereof and applications thereof. BACKGROUND

[0002] In recent years, due to the rapid development of the Internet and its large-scale popularity in society, a large amount of multimedia data has emerged, such as Internet short videos, life photos, etc. Most intelligent device production companies also benefit from this large amount of data, and use the data to iterate intelligent detection models and classification models.

[0003] However, taking a mobile phone as an example, most mobile phone photos and videos belong to the personal privacy of the user, and these data cannot be collected by intelligent algorithms. Therefore, how to protect the privacy of the user while using these data that belong to the user to iterate a personalized intelligent algorithm for the user, or how to update and iterate the model by integrating the advantages of other companies' models without sharing data for an artificial intelligence company that exists in multiple countries has become a problem to be solved. These problems can be attributed to a machine learning problem that starts with data privacy considerations, and federated learning technology can well solve this problem.

[0004] Federated learning mainly uses multiple clients to jointly train a model under the organization of a server, and at the same time, the client does not need to transmit any private data of its own, but only needs to transmit the model parameters. For example, a mobile phone company acts as a server, and all the mobile phones held by users of this brand act as a large number of clients. The server distributes a training algorithm model to the users, and the users update the model using local computing power and local data and then upload it to the server. This distributed training method can very well protect the privacy and security of the user. In the model training process, there is no transmission of sensitive private data between users and between users and the server. This training scheme has been widely used in the technical fields of user product updates and cross-regional company security.

[0005] However, most of the past federated learning techniques mainly focus on supervised learning, i.e., assuming that all user-side data is labeled, but in actual scenarios, labeling a large amount of local data will consume more human resources and increase costs, therefore, how to use a small amount of labeled data and a large amount of unlabeled data to train and achieve higher model performance becomes the key to saving human costs. In particular, in federated learning, most user-side data is redundant, and it is impossible to label all picture or information data, and when federated learning is performed, the local data of all clients is heterogeneous, and as in the foregoing example, all user mobile phones are respectively and independently used as clients, because of the differences in personal photographing preferences and life scenes, the data of each user is different, and thus the training data domain gap between clients is large, and the data quantity also differs.

[0006] In the face of these challenges, Jeong et al. proposed a practical federated semi-supervised method in "Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning", i.e., only a small amount of labeled data is needed in each client to improve the model performance, in order to solve the problem of data heterogeneity between different user sides, the quality of data pseudo-labels is determined by transmitting model consensus information between different clients to improve the performance of the main model on different domains; in addition, in order to alleviate the communication resource consumption caused by transmitting model interaction information, Jeong also adopts a sparse model parameter transmission method to reduce the bandwidth consumption caused by the model in communication. However, early work often uses some traditional semi-supervised learning schemes such as Fixmatch, Fedmatch, etc. in each client, without fully considering the problems of these methods in federated learning applications. For example, these semi-supervised methods often use a fixed threshold to screen high-quality pseudo-labels of unlabeled data, but the training process of any client may be different, and the distribution of its training data may also be different from other clients, so the training state will also be greatly different, and using a fixed threshold to screen pseudo-labels for all clients will make the model lack flexibility in training; on the other hand, for a client, maintaining a fixed threshold during model training is also insufficient for data utilization, the main reason being that the model's judgment ability is low at the beginning of training and the model's judgment accuracy is high at the end of training, how to carefully design the threshold of pseudo-labels becomes the key to improving semi-supervised models. SUMMARY

[0007] The present application solves the problems existing in the prior art and provides a federated semi-supervised learning system and method based on adaptive weights and application thereof.

[0008] The technical solution adopted by the present application is a federated semi-supervised learning method based on adaptive weights, which trains the labeled data and unlabeled data in any client in federated learning, uses the loss of the unlabeled data to update the full connection module in reverse, generates new weights based on the updated full connection module, thereby obtaining new loss and repeatedly updating the full connection module in reverse, repeating the training steps, and performing semi-supervised learning. The present application adjusts the main model in fine granularity by measuring the weight of the unlabeled data in the optimization process, and the main model is not updated when the unlabeled data updates the full connection module.

[0009] Preferably, the method comprises the following steps:

[0010] Step 1: any client in federated learning downloads a main model from a server; the client stores local data, including labeled data and unlabeled data; and a maximum number of iterations is defined;

[0011] Step 2: a batch of quantitative labeled data and unlabeled data are randomly sampled from the client, input into the main model, the loss of the labeled data and the logits of the unlabeled data are obtained, and the logits of the unlabeled data are input into the full connection module; wherein the ratio of the labeled data and the unlabeled data is fixed, and the ratio can be adjusted according to requirements;

[0012] Step 3: the full connection module outputs corresponding weights of all unlabeled data, and the updated loss of the unlabeled data is obtained after weighting the logits of the unlabeled data corresponding to the weights;

[0013] Step 4: the full connection module is updated in reverse based on the loss of the unlabeled data; if the maximum number of iterations has been reached, step 6 is performed, otherwise, the next step is performed;

[0014] Step 5: based on the updated full connection module, new weights of all unlabeled data are output, new loss of the unlabeled data is obtained after weighting the logits of the unlabeled data corresponding to the new weights, the number of iterations is increased by 1, and step 4 is repeated;

[0015] Step 6: the parameters of the main model are updated based on the loss of the labeled data and the current loss of the unlabeled data, and the training is completed.

[0016] Preferably, in step 1, the main model is a replaceable classification model ResNet, including but not limited to ResNet9, ResNet18, ResNet34, ResNet50 or other available network structures.

[0017] Preferably, in step 2, the unlabeled data is more than the labeled data, and in practical applications, the labeled data should be more than 15 times the unlabeled data.

[0018] Preferably, in step 6, during training, the labeled data is input into the main model for training, and cross-entropy is used for supervision; the unlabeled data calculates the loss through the full-connection module to learn the weight of each data, and the parameters of the full-connection module are updated according to the loss, and after repeating a preset number of times, the main model is updated based on the unlabeled data loss and the labeled data loss.

[0019] Preferably, the client updates the main model and iterates for a preset number of rounds, and then uploads the latest main model to the server.

[0020] Preferably, all the models uploaded by the clients are fused based on a fusion strategy.

[0021] Preferably, the fusion strategy is to fuse according to the parameter files submitted by the clients every preset time, or to select a number of parameter files uploaded by the clients for fusion; the interval time of the former is 5h, 12h, 24h, etc., and the parameter files submitted by the clients are fused, and the latter selects the first n parameter files according to the parameter files uploaded by the clients for fusion, and n is less than the total number of clients. In practical applications, the parameter fusion strategy is to calculate the parameter file weight proportion of each client according to the data volume of the client, and then update to the server side.

[0022] A system using the adaptive weight-based federated semi-supervised learning method, the system comprising:

[0023] a server for global management and configured with an iteratively updated main model;

[0024] a plurality of clients for obtaining local data, processing the local data through the main model, updating the main model and uploading it to the server;

[0025] Any of the clients downloads the main model from the server and trains and updates the main model using local data, and uploads the updated main model to the server.

[0026] An application of the adaptive weight-based federated semi-supervised learning method, applied to semi-supervised learning between heterogeneous clients and their heterogeneous data.

[0027] The application relates to a federated semi-supervised learning system based on adaptive weights, a method thereof and application thereof, which trains labeled data and unlabeled data in any client in federated learning, reversely updates a full connection module by using the loss of the unlabeled data, generates new weights based on the updated full connection module, thereby obtaining new loss and repeatedly reversely updating the full connection module, repeatedly training steps, and performing semi-supervised learning; the system globally manages and configures a main model with iterative updates by using a server, and a client obtains local data, downloads the main model from the server and trains and updates the main model by using the local data, and uploads the updated main model to the server; and the application is applied to semi-supervised learning between heterogeneous clients and heterogeneous data thereof.

[0028] The application has the following beneficial effects:

[0029] (1) In view of the problem that past federated semi-supervised learning excessively depends on fixed thresholds on local algorithms, thereby lacking flexibility on all clients with different data domains, an adaptive weight updating mechanism is proposed, the weight is used to measure the contribution size of each unlabeled data to parameter updating in this iterative update, and the flexibility of the model in the case of different data domains of multiple clients in the semi-supervised training process is effectively improved.

[0030] (2) In view of the need for dynamic decision of the quality of unlabeled data in the training process, the sufficiency of the use of unlabeled data in the model training process is increased; an algorithm mechanism of double optimization is defined, the weight parameter of each unlabeled data is adaptively adjusted according to the performance feedback of the module on the labeled data, the weight adjustment is defined according to the accuracy of the labeled data, the model adaptively updates the weight of each unlabeled data according to its own training process, the model has higher flexibility and stronger robustness;

[0031] (3) The application is suitable for multiple data distribution scenes, and better model precision is obtained compared with past methods;

[0032] (4) Information interaction between clients is not needed, and privacy problems are better avoided;

[0033] (5) The overall structure of training is designed, so that the unlabeled data can influence model optimization according to the importance of different data during training. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The method flowchart of the application is shown in the figure;

[0035] Figure 2 The semi-supervised learning schematic diagram in the application is shown in the figure, wherein the dashed box shows a repeated module, and the final output of the module is used for updating the main model parameters;

[0036] Figure 3 Figure 1 is a structural schematic diagram of a system in the present application. DETAILED DESCRIPTION

[0037] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] The present application relates to a kind of federal semi-supervised learning method based on adaptive weight, the method is trained in the annotation data and unlabelled data in any client in federal learning, utilize the loss of unlabelled data reverse update full connection module, generate new weight based on updated full connection module, to obtain new loss and repeat reverse update full connection module, repeat training step, carry out semi-supervised learning.

[0039] In the present application, for a certain client (client) in federal learning, the local data it has contains a small part of labeled data, and more is a large amount of unlabeled data;The client downloads a main model from the server and trains it using local data;Taking image classification task as an example, the main model is a replaceable classification model ResNet, all data are picture data, wherein the labeled data represents the category of the picture is known, and the unlabeled data comes from the sampling distribution based on category Dirichlet, and there is a large difference and difference in class distribution and data distribution of each category with labeled data, close to actual heterogeneous data problem.

[0040] The concept of the present application is that in the training iteration, quantitative labeled data and unlabeled data are randomly selected for training, for the unlabeled data, the features of the logits layer are also extracted by the main model, and a full connection module is used to output the weights of all unlabeled data; when optimizing the main model, unlike the past scheme of using a fixed threshold to screen out high-quality unlabeled data, all unlabeled data are used, and the continuous weights on the module are used to measure the contribution of the unlabeled data in this optimization process; it is worth noting that the full connection module is independent of the main model and can be optimized; when using the corresponding weight to measure the contribution of the unlabeled data in this training iteration, the purpose is that this full connection module participates in the iteration of the main model (using the loss loss function of the main model), and itself also optimizes its parameters based on this loss (since in the loss, the loss function is dependent on the parameters of the full connection module); specifically, in the training process, first, the full connection module is used to loss weight the predicted labels of all unlabeled data, the full connection module is optimized, and based on the optimized full connection module, the main model is further optimized.

[0041] The optimization scheme of the present application enables the full connection module to optimize itself according to the feedback of the labeled data on the main model, and then further generates the weight of the unlabeled data, thereby producing more accurate weights of the unlabeled data, and this process is synchronized with the training process and is customized for each client, which has better flexibility in federated learning and better robustness for heterogeneous clients and heterogeneous data.

[0042] As shown in Figure 1 , the method comprises the following steps:

[0043] Step 1: any client in the federated learning downloads the main model from the server; the client stores local data, and the local data includes labeled data and unlabeled data; define the maximum number of iterations;

[0044] In the step 1, the main model is a replaceable classification model ResNet.

[0045] As shown in Figure 2 , the following is the process of semi-supervised learning.

[0046] Step 2: randomly sample a batch of quantitative labeled data and unlabeled data from the client, input the main model, get the loss of the labeled data and the logits of the unlabeled data, and input the logits of the unlabeled data to the full connection module;

[0047] In the step 2, the unlabeled data is more than the labeled data.

[0048] In this invention, to be precise, the amount of labeled data is much smaller than the amount of unlabeled data. Generally, the amount of unlabeled data is 15 times that of labeled data, and the ratio between the two remains unchanged during application.

[0049] Step 3: The fully connected module outputs the corresponding weights of all unlabeled data, and after weighting them with the logits of the unlabeled data, the updated loss of the unlabeled data is obtained;

[0050] Step 4: Update the fully connected module in reverse based on the loss of the unlabeled data; if the maximum number of iterations has been reached, proceed to step 6; otherwise, proceed to the next step.

[0051] Step 5: Based on the updated fully connected module, output the new weights of all unlabeled data, and obtain the new loss of the unlabeled data by weighting it with the corresponding logits of the unlabeled data. Increment the iteration count by 1, and repeat step 4.

[0052] Step 6: Update the parameters of the main model based on the loss of the labeled data and the current loss of the unlabeled data, and training ends.

[0053] In step 6, during training, labeled data is input into the main model for training, and cross-entropy is used for supervision; unlabeled data learns the weights of each data point through the fully connected module to calculate the loss, and the parameters of the fully connected module are updated according to the loss. After repeating this process a preset number of times, the main model is updated based on the loss of unlabeled data and the loss of labeled data.

[0054] After the client updates the main model and iterates through the preset number of rounds, it uploads the latest main model to the server.

[0055] All models uploaded by clients are fused based on the fusion strategy.

[0056] The fusion strategy involves fusing parameters based on client-submitted parameter files at preset intervals, or selecting several client-uploaded parameter files for fusion.

[0057] To achieve the above, the present invention also relates to a computer-readable storage medium storing a program for adaptive weight-based federated semi-supervised learning, which, when executed by a processor, implements the aforementioned method for adaptive weight-based federated semi-supervised learning.

[0058] To achieve the above, the present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method of federated semi-supervised learning based on adaptive weights.

[0059] like Figure 3As shown, the present invention also relates to a system employing the aforementioned federated semi-supervised learning method based on adaptive weights, the system comprising:

[0060] A server-side component is used for global management and is configured with an iteratively updated main model.

[0061] Several clients are used to acquire local data, process the local data through the main model, update the main model, and upload it to the server.

[0062] Each of the aforementioned clients downloads the main model from the server and trains and updates the main model using local data, then uploads the updated main model to the server.

[0063] In this invention, the client trains a local model using local data, while the server coordinates and plans the distribution of the model to the client and updates the model collected from all clients.

[0064] This invention also relates to an application of the aforementioned federated semi-supervised learning method based on adaptive weights, applied to semi-supervised learning between heterogeneous clients and their heterogeneous data.

[0065] This invention presents an application scenario: for example, a mobile phone company wants to update the intelligent image editing model on its users' phones. In this scenario, users are very sensitive to their privacy data, so federated learning is needed for distributed learning. In this scenario, due to the large amount of data stored on each user's phone, annotation is very time-consuming and resource-intensive. A semi-supervised approach can better meet the needs of practical applications. In addition, most users cannot determine whether the unlabeled data is in the same data distribution as the labeled data after annotating the data, and this is difficult to achieve in real-world scenarios. At the same time, the unlabeled data that has not been manually filtered also differs greatly from the labeled data in terms of scenario and domain. In this scenario, adaptive weights can be used to handle unlabeled data more flexibly, providing the main model with correct feedback related to the unlabeled data under dual optimization, while eliminating the need for fixed thresholds to restrict all clients.

[0066] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0071] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A federated semi-supervised learning method based on adaptive weights, characterized in that: The method is used for image classification tasks, and all data are image data. The method trains on labeled and unlabeled data in any client in federated learning, uses the loss of unlabeled data to back-update the fully connected module, generates new weights based on the updated fully connected module, thereby obtaining a new loss, and repeats the back-update of the fully connected module, repeating the training steps to perform semi-supervised learning. The method includes the following steps: Step 1: Any client in federated learning downloads the main model from the server; the client stores local data, including labeled and unlabeled data; define the maximum number of iterations; Step 2: Randomly sample a batch of labeled and unlabeled data from the client, input them into the main model, obtain the loss of the labeled data and the logits of the unlabeled data, and input the logits of the unlabeled data into the fully connected module; Step 3: The fully connected module outputs the corresponding weights of all unlabeled data, and after weighting them with the logits of the unlabeled data, the updated loss of the unlabeled data is obtained; Step 4: Update the fully connected module in reverse based on the loss of the unlabeled data; if the maximum number of iterations has been reached, proceed to step 6; otherwise, proceed to the next step. Step 5: Based on the updated fully connected module, output the new weights of all unlabeled data, and obtain the new loss of the unlabeled data by weighting it with the corresponding logits of the unlabeled data. Increment the iteration count by 1, and repeat step 4. Step 6: Update the parameters of the main model based on the loss of the labeled data and the current loss of the unlabeled data, and training ends.

2. The federated semi-supervised learning method based on adaptive weights according to claim 1, characterized in that: In step 1, the main model is the replaceable classification model ResNet.

3. The federated semi-supervised learning method based on adaptive weights according to claim 1, characterized in that: In step 2, there is more unlabeled data than labeled data.

4. The federated semi-supervised learning method based on adaptive weights according to claim 1, characterized in that: In step 6, during training, labeled data is input into the main model for training, and cross-entropy is used for supervision; unlabeled data learns the weights of each data point through the fully connected module to calculate the loss, and the parameters of the fully connected module are updated according to the loss. After repeating this process a preset number of times, the main model is updated based on the loss of unlabeled data and the loss of labeled data.

5. The federated semi-supervised learning method based on adaptive weights according to claim 1, characterized in that: After the client updates the main model and iterates through the preset number of rounds, it uploads the latest main model to the server.

6. The federated semi-supervised learning method based on adaptive weights according to claim 1, characterized in that: All models uploaded by clients are fused based on the fusion strategy.

7. A federated semi-supervised learning method based on adaptive weights according to claim 6, characterized in that: The fusion strategy involves fusing parameters based on client-submitted parameter files at preset intervals, or selecting several client-uploaded parameter files for fusion.

8. A system employing the federated semi-supervised learning method based on adaptive weights as described in any one of claims 1 to 7, characterized in that: The system includes: A server-side component is used for global management and is configured with an iteratively updated main model. Several clients are used to acquire local data, process the local data through the main model, update the main model, and upload it to the server. Each of the aforementioned clients downloads the main model from the server and trains and updates the main model using local data, then uploads the updated main model to the server.

9. An application of the federated semi-supervised learning method based on adaptive weights as described in any one of claims 1 to 7, characterized in that: It is applied to semi-supervised learning between heterogeneous clients and their heterogeneous data.

Images