Federated learning methods, systems, devices, and media

Abstract

The present application relates to the field of computer technology and provides a federated learning method, system, device and medium. The method includes: using multiple first clients to train a first local model based on unlabeled training samples, sending the trained first local model to a server, the server integrating the first local model to obtain a pre-trained model, multiple second clients downloading the pre-trained model from the server, using the pre-trained model as the second local model, fine-tuning the second local model based on labeled observation samples, sending the fine-tuned second local model to the server, the server integrating the pre-trained model and the second local model to obtain a target model. The embodiment of the present application adds a client that uses unlabeled sample training to federated learning, combined with a joint training method of pre-training and fine-tuning, which can not only improve the generalization of the target model, but also efficiently utilize unlabeled data and save labeling costs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a federated learning method, system, device, and medium. Background Art

[0002] Federated learning is a distributed machine learning technology that enables the joint training of a machine learning model across multiple centers without exchanging local data. However, in real-life applications such as the Internet of Vehicles and drones, many clients have large amounts of unlabeled data. Manually labeling this image data requires significant time and labor costs. Traditional federated learning frameworks only process labeled data, resulting in low model generalization and an inability to efficiently utilize unlabeled data from some clients.

[0003] Therefore, how to provide a federated learning method that can not only ensure the generalization performance of the target model but also be suitable for scenarios where some clients have unlabeled data has become an urgent problem to be solved. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a federated learning method, system, electronic device and computer-readable storage medium, which can improve the generalization of the model, efficiently utilize unlabeled data, and save labeling costs.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a federated learning method, which is applied to a federated learning system. The system includes a server, multiple first clients, and multiple second clients. The server is communicatively connected to the multiple first clients and the multiple second clients, respectively. The method includes:

[0006] Training the first local model based on unlabeled training samples respectively through the multiple first clients, and sending the trained first local model to the server;

[0007] Receiving, by the server, the first local models sent by the plurality of first clients, and integrating the plurality of first local models to obtain a pre-trained model;

[0008] downloading the pre-trained model from the server through a plurality of second clients, using the pre-trained model as a second local model, fine-tuning the second local model based on the labeled observation samples, and sending the fine-tuned second local model to the server;

[0009] The server receives the second local models sent by multiple second clients, and integrates the pre-trained model and multiple second local models to obtain a target model.

[0010] According to some embodiments of the present invention, the federated learning method provided by the server includes receiving the first local models sent by the plurality of first clients and integrating the plurality of first local models to obtain a pre-trained model, including:

[0011] receiving, by the server, the first local models sent by the plurality of first clients, and integrating the plurality of first local models to obtain a first intermediate model;

[0012] The first intermediate model is tested through the server. If the test result of the first intermediate model does not meet the first training end condition, the step of training the first local model based on the unlabeled training samples through multiple first clients is returned to, until the test result of the first intermediate model meets the first training end condition, and the first intermediate model is used as the pre-training model.

[0013] According to the federated learning method provided by some embodiments of the present invention, after the server receives the first local models sent by the multiple first clients and integrates the multiple first local models to obtain a pre-trained model, and before the multiple second clients respectively download the pre-trained model from the server, the method further includes:

[0014] downloading the pre-trained model from the server respectively through the first clients, training the pre-trained model based on unlabeled training samples, and sending the trained pre-trained model to the server;

[0015] Receiving, by the server, the pre-trained models sent by the plurality of first clients, and integrating the plurality of pre-trained models to obtain a second intermediate model;

[0016] The second intermediate model is tested by the server. If the test result of the second intermediate model does not meet the preset second training end condition, the process returns to the step of downloading the pre-trained model from the server through multiple first clients respectively until the test result of the second intermediate model meets the second training end condition, and the second intermediate model is used as the final pre-trained model.

[0017] According to some embodiments of the present invention, the federated learning method provided by the server includes receiving the second local models sent by the plurality of second clients, and integrating the pre-trained model and the plurality of second local models to obtain a target model, including:

[0018] Receiving, by the server, the second local models sent by the plurality of second clients, and integrating the pre-trained model and the plurality of second local models to obtain a third intermediate model;

[0019] The third intermediate model is tested by the server. If the test result of the third intermediate model does not meet the preset third training end condition, the process returns to the step of downloading the pre-trained model from the server through multiple second clients respectively until the test result of the third intermediate model meets the third training end condition, and the third intermediate model is used as the target model.

[0020] According to some embodiments of the present invention, the federated learning method provided by the first local model is an image classification model, and the training of the first local model based on unlabeled training samples by the plurality of first clients includes:

[0021] Performing image amplification on each image sample in the unlabeled training samples to obtain an amplified image corresponding to each image sample;

[0022] The first local model is self-supervisedly trained based on the augmented image corresponding to each of the image samples.

[0023] According to some embodiments of the present invention, the federated learning method provides self-supervised training of the first local model based on the augmented image corresponding to each image sample, including:

[0024] Inputting the augmented image corresponding to each of the image samples into the first local model, so as to output the image features corresponding to each of the augmented images through the first local model;

[0025] Mapping the image features corresponding to each of the amplified images to the same vector space to obtain a vector representation corresponding to each of the amplified images;

[0026] determining a first similarity value between the vector representations obtained based on the same image sample and a second similarity value between the vector representations obtained based on different image samples;

[0027] With the goal of minimizing the first similarity value and maximizing the second similarity value, a first loss function is constructed, and self-supervised training is performed on the first local model based on the first loss function.

[0028] According to the federated learning method provided by some embodiments of the present invention, fine-tuning the second local model based on the labeled observation samples includes:

[0029] Inputting each observed sample in the labeled observed samples into the second local model to obtain a class prediction probability value corresponding to each of the observed samples through the second local model;

[0030] Determining the true category corresponding to each of the observed samples, and constructing a second loss function based on the category prediction probability value and the true category corresponding to each of the observed samples;

[0031] Fine-tune the second local model based on the second loss function.

[0032] To achieve the above objectives, a second aspect of an embodiment of the present application proposes a federated learning system, comprising:

[0033] The system includes a server, a plurality of first clients and a plurality of second clients, wherein the server is in communication connection with the plurality of first clients and the plurality of second clients respectively; wherein,

[0034] The first client includes:

[0035] A model training module, configured to train a first local model based on unlabeled training samples;

[0036] A first sending module, configured to send the trained first local model to the server;

[0037] The server includes:

[0038] A first receiving module, configured to receive the first local models sent by a plurality of the first clients;

[0039] A first integration module, configured to integrate the plurality of the first local models to obtain a pre-trained model;

[0040] A second receiving module, configured to receive the second local models sent by a plurality of second clients;

[0041] A second integration module is used to integrate the pre-trained model and the plurality of second local models to obtain a target model;

[0042] The second client includes:

[0043] A first downloading module is used to download the pre-trained model from the server and use the pre-trained model as a second local model;

[0044] A model fine-tuning module, configured to fine-tune the second local model based on labeled observation samples;

[0045] A second sending module is used to send the fine-tuned second local model to the server.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program implements the method described in the first aspect above when executed by the processor.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the method described in the first aspect above.

[0048] The present application proposes a federated learning method, system, electronic device and computer-readable storage medium. The federated learning method is applied to a federated learning system. The system includes a server, multiple first clients and multiple second clients. The server is communicated with the multiple first clients and the multiple second clients respectively. The method includes: training a first local model based on unlabeled training samples by the multiple first clients, and sending the trained first local model to the server, receiving the first local models sent by the multiple first clients by the server, and integrating the multiple first local models to obtain a pre-trained model, downloading the pre-trained model from the server by the multiple second clients, using the pre-trained model as the second local model, fine-tuning the second local model based on labeled observation samples, and sending the fine-tuned second local model to the server, receiving the second local models sent by the multiple second clients by the server, and integrating the pre-trained model and the multiple second local models to obtain a target model. In an embodiment of the present application, a client that uses unlabeled data for training is added to federated learning, and a joint training method of pre-training and fine-tuning is combined. The first client pre-trains the model based on unlabeled training samples, while the second client fine-tunes the pre-trained model based on labeled observation samples. This can not only improve the generalization of the target model, but also efficiently utilize unlabeled data and save labeling costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flowchart of a federated learning method provided in an embodiment of the present application;

[0050] Figure 2 This is a flowchart of a federated learning method provided in another embodiment of the present application;

[0051] Figure 3 yes Figure 2 Schematic diagram of the sub-step flow of step S220;

[0052] Figure 4 This is a flowchart of a federated learning method provided in another embodiment of the present application;

[0053] Figure 5 This is a flowchart of a federated learning method provided in another embodiment of the present application;

[0054] Figure 6 This is a flowchart of a federated learning method provided in another embodiment of the present application;

[0055] Figure 7 This is a flowchart of a federated learning method provided in another embodiment of the present application;

[0056] Figure 8 This is an implementation environment diagram of a federated learning method provided in an embodiment of the present application;

[0057] Figure 9 This is a schematic diagram of the structure of a federated learning system provided in an embodiment of the present application;

[0058] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Federated learning is a distributed machine learning technology that enables the joint training of a machine learning model across multiple centers without exchanging local data. However, in real-life applications such as the Internet of Vehicles and drones, many clients have large amounts of unlabeled data. Manually labeling this image data requires significant time and labor costs. Traditional federated learning frameworks only process labeled data, resulting in low model generalization and an inability to efficiently utilize unlabeled data from some clients.

[0062] Therefore, how to provide a federated learning method that can not only ensure the generalization performance of the target model but also be suitable for scenarios where some clients have unlabeled data has become an urgent problem to be solved.

[0063] Based on this, the embodiments of the present application provide a federated learning method, system, electronic device and computer-readable storage medium, which can improve the generalization of the model, efficiently utilize unlabeled data, and save labeling costs.

[0064] The embodiments of the present application provide a federated learning method, system, electronic device, and computer-readable storage medium, which are specifically described through the following embodiments.

[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0066] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0068] First, the federated learning method in the embodiment of the present application is described. The method is applied to the federated learning system. Figure 8 , Figure 8 FIG1 shows an implementation environment diagram of a federated learning method provided in an embodiment of the present application, such as Figure 8 As shown, the federated learning system includes a server, multiple first clients, and multiple second clients. The server is connected to the multiple first clients and the multiple second clients respectively. Figure 1 , Figure 1 FIG1 shows a flow chart of a federated learning method provided in an embodiment of the present application. Figure 1 As shown, the federated learning method includes but is not limited to steps S110 to S140:

[0069] Step S110: Train the first local model based on unlabeled training samples respectively through multiple first clients, and send the trained first local model to the server.

[0070] Step S120: receiving, through the server, the first local models sent by the multiple first clients, and integrating the multiple first local models to obtain a pre-trained model.

[0071] In step S130, the pre-trained model is downloaded from the server through multiple second clients respectively, the pre-trained model is used as the second local model, the second local model is fine-tuned based on the labeled observation samples, and the fine-tuned second local model is sent to the server.

[0072] Step S140: receiving, through the server, the second local models sent by the plurality of second clients, and integrating the pre-trained model and the plurality of second local models to obtain a target model.

[0073] It is understandable that the first client has unlabeled training samples. The first client is added to the federated learning, whereby multiple first clients train the first local model based on the unlabeled training samples. The multiple first local models are then integrated and processed by the server to obtain a pre-trained model. Finally, the second client fine-tunes the pre-trained model based on the labeled training samples. Based on the joint training method of pre-training and fine-tuning, the model parameters will not change significantly due to fine-tuning, so the model parameters of all clients will remain within a controllable range. This can solve the problem of large differences between different client models, resulting in performance degradation, when the training sample data is non-independent and identically distributed (Non-IID).

[0074] It should be noted that the first local model can be provided by the server, and multiple first clients can download the first local model from the server. The multiple first clients train the first local model based on local unlabeled training samples, update the model parameters of the first local model, and upload the updated first local model to the server.

[0075] In step S120 of some embodiments, see Figure 5 , Figure 5This is a flow chart of a federated learning method provided in an embodiment of the present application. Figure 5 As shown, the step S120 further includes:

[0076] receiving, by the server, the first local models sent by the plurality of first clients, and integrating the plurality of first local models to obtain a first intermediate model;

[0077] The first intermediate model is tested through the server. If the test result of the first intermediate model does not meet the first training end condition, the step of training the first local model based on the unlabeled training samples through multiple first clients is returned to, until the test result of the first intermediate model meets the first training end condition, and the first intermediate model is used as the pre-training model.

[0078] It should be noted that the first intermediate model obtained after the integration processing is tested by the server. If the test result of the first intermediate model does not meet the first training end condition, multiple first clients continue to train the first local model based on the unlabeled training samples, and the cycle continues until the first intermediate model obtained by integrating the first local model through the server meets the first training end condition, and the first intermediate model is used as the pre-training model.

[0079] In a specific embodiment, the test results of the first intermediate model of this cycle are recorded by the server. When the test results of the first intermediate model are better than the test results of the previous cycle, the cycle is ended, and the first intermediate model is used as a pre-trained model to execute subsequent steps; if the test results of the first intermediate model are not better than the test results of the previous cycle, the cycle continues.

[0080] It can be understood that by setting the above-mentioned loop steps, federated learning can improve the model performance of the pre-trained model as much as possible with limited training samples.

[0081] In some embodiments, see Figure 6 , Figure 6 This is a flow chart of a federated learning method provided in an embodiment of the present application. Figure 6 As shown, after step S120 and before step S130, the method further includes:

[0082] downloading the pre-trained model from the server respectively through the first clients, training the pre-trained model based on unlabeled training samples, and sending the trained pre-trained model to the server;

[0083] Receiving, by the server, the pre-trained models sent by the plurality of first clients, and integrating the plurality of pre-trained models to obtain a second intermediate model;

[0084] The second intermediate model is tested by the server. If the test result of the second intermediate model does not meet the preset second training end condition, the process returns to the step of downloading the pre-trained model from the server through multiple first clients respectively until the test result of the second intermediate model meets the second training end condition, and the second intermediate model is used as the final pre-trained model.

[0085] It should be noted that after the pre-trained model is obtained by integrating the first local model through the server, the pre-trained model is downloaded from the server through the first client, and the pre-trained model is trained based on the unlabeled training samples. Then, the second intermediate model obtained by integrating the pre-trained model is tested by the server. If the test result of the second intermediate model does not meet the second training end condition, multiple first clients continue to train the pre-trained model based on the unlabeled training samples, and the cycle continues until the second intermediate model obtained by integrating the pre-trained model through the server meets the second training end condition, and the second intermediate model is used as the final pre-trained model.

[0086] In a specific embodiment, the test results of the second intermediate model in this cycle are recorded by the server. When the test results of the second intermediate model are better than the test results of the previous cycle, the cycle continues; when the test results of the second intermediate model are not better than the test results of the previous cycle, the cycle ends, and the second intermediate model is used as the final pre-training model to execute subsequent steps, that is, the above-mentioned cycle steps are executed until the model performance of the second intermediate model no longer improves.

[0087] It can be understood that by setting the above-mentioned loop steps, federated learning can further improve the model performance of the final pre-trained model with limited training samples.

[0088] In step S140 of some embodiments, see Figure 7 , Figure 7 A schematic diagram of a federated learning method provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, step S140 includes:

[0089] Receiving, by the server, the second local models sent by the plurality of second clients, and integrating the pre-trained model and the plurality of second local models to obtain a third intermediate model;

[0090] The third intermediate model is tested by the server. If the test result of the third intermediate model does not meet the preset third training end condition, the process returns to the step of downloading the pre-trained model from the server through multiple second clients respectively until the test result of the third intermediate model meets the third training end condition, and the third intermediate model is used as the target model.

[0091] It should be noted that the pre-trained model and the second local model are integrated by the server to obtain a third intermediate model. If the test result of the third intermediate model meets the third training end condition, the second local model is further fine-tuned based on labeled observation samples through multiple second clients, and the cycle is repeated until the third intermediate model obtained by integrating the pre-trained model and the second local model by the server meets the third training end condition, and the third intermediate model is used as the target model.

[0092] In a specific embodiment, the test results of the third intermediate model in this cycle are recorded by the server. When the test results of the third intermediate model are better than the test results of the previous cycle, the cycle continues; when the test results of the third intermediate model are not better than the test results of the previous cycle, the cycle ends and the third intermediate model is used as the target model, that is, the above-mentioned cycle steps are executed until the model performance of the third intermediate model no longer improves.

[0093] It can be understood that by setting the above-mentioned loop steps, federated learning can improve the model performance of the target model as much as possible with limited training samples.

[0094] In some embodiments, the first local model is an image classification model. Figure 2 , Figure 2 FIG1 shows a flow chart of a federated learning method provided in an embodiment of the present application. Figure 2 As shown, the first local model is trained based on unlabeled training samples by the multiple first clients, including but not limited to steps S210 and S220:

[0095] Step S210 , performing image amplification on each image sample in the unlabeled training samples to obtain an amplified image corresponding to each image sample.

[0096] Step S220: performing self-supervisory training on the first local model based on the augmented image corresponding to each of the image samples.

[0097] It can be understood that when the first local model is an image classification model, the corresponding unlabeled training samples on the first client are image samples. The first client performs image amplification on each image sample in the unlabeled training samples to obtain the amplified image corresponding to each image sample.

[0098] It should be understood that the image amplification performed on each image sample may be performed by random cropping and then adjusting back to the original size, random color distortion, or random Gaussian blurring.

[0099] In some embodiments, see Figure 3 , Figure 3 Shown Figure 2 Schematic diagram of the sub-step flow of step S220. Figure 3 As shown, step S220 includes but is not limited to steps S310 to S340:

[0100] In step S310 , the augmented image corresponding to each of the image samples is input into a first local model, so as to output image features corresponding to each of the augmented images through the first local model.

[0101] Step S320 : Mapping the image features corresponding to each of the augmented images to the same vector space to obtain a vector representation corresponding to each of the augmented images.

[0102] Step S330 : determining a first similarity value between the vector representations obtained based on the same image sample and a second similarity value between the vector representations obtained based on different image samples.

[0103] Step S340 : constructing a first loss function with the goal of minimizing the first similarity value and maximizing the second similarity value, and performing self-supervised training on the first local model based on the first loss function.

[0104] For example, the image sample p corresponds to the augmented image x i and x j , the image x will be enlarged i and x j Input to the encoder of the first local model to extract the augmented image x i The corresponding image feature h i And the augmented image x j The corresponding image feature h j , and then the image feature h i and h j Input to the projection head G(x) to transform the image feature h i and h j Mapped to the same vector space, we get the augmented image x i The corresponding vector representation Z i And the augmented image x j The corresponding vector representation z j .

[0105] It should be noted that the training objectives are to minimize the first similarity between vector representations obtained based on the same image sample and to maximize the second similarity between vector representations obtained based on different image samples, to construct a first loss function, and to perform self-supervised training on the first local model based on the first loss function. For example, for the vector representation z obtained from the image sample p, i and z j , and the vector representation z obtained from the image sample q n and z m , that is, minimizing the vector representation z i and z j and vector representation z n and z m The first similarity value between them, and maximize the vector representation z i and z n 、z i and z m Between, and vector representation z j and z n 、z j and z m The second similarity value between them.

[0106] It can be understood that training the first local model with the goal of minimizing the first similarity value between vector representations obtained based on the same image sample and maximizing the second similarity value between vector representations obtained based on different image samples can improve the image classification performance of the first local model.

[0107] In some embodiments, determining a first similarity value between the vector representations obtained based on the same image sample and a second similarity value between the vector representations obtained based on different image samples includes:

[0108] Determining a first cosine similarity value between the vector representations obtained based on the same image sample and a second cosine similarity value between the vector representations obtained based on different image samples;

[0109] The step of constructing a first loss function with the goal of minimizing the first similarity value and maximizing the second similarity value includes:

[0110] A first loss function is constructed with the goal of minimizing the first cosine similarity value and maximizing the second cosine similarity value.

[0111] It can be understood that the cosine similarity value is used to measure the cosine angle between two vector representations. The smaller the cosine similarity value between the two vector representations, the more similar the corresponding image samples are; the larger the cosine similarity value between the two vector representations, the more different the corresponding image samples are.

[0112] It should be understood that the similarity values between vector representations may also be calculated using other similarity metrics such as Euclidean distance and Mahalanobis distance.

[0113] In a specific embodiment, the first loss function is determined by the following formula:

[0114]

[0115] Wherein, l1 is the first loss function, N is the sample size, sim(·) represents the cosine similarity, and z i and z j is a vector representation, exp(·) is an exponential function, and τ is a temperature control parameter used to adjust the gradient of the first loss function.

[0116] In some embodiments, the server integrates the models based on the Federated Averaging Algorithm (FedAvg) method.

[0117] It is understandable that, taking the server integrating the first local model as an example, the server provides model parameters w to m first clients (set V). t The first local model is trained by m first clients based on unlabeled training samples. The server integrates the trained first local models. The integration formula is:

[0118]

[0119] Among them, n k is the number of unlabeled training samples on the kth first client, n is the total number of samples on the m first clients, are the model parameters of the first local model trained by the k-th first client.

[0120] In some embodiments, participating Figure 4 , Figure 4 FIG1 shows a flow chart of a federated learning method provided in an embodiment of the present application. Figure 4 As shown, the fine-tuning of the second local model based on the labeled observation samples includes:

[0121] In step S410 , each observation sample in the labeled observation samples is input into the second local model to obtain a category prediction probability value corresponding to each of the observation samples through the second local model.

[0122] Step S420: Determine the true category corresponding to each of the observed samples, and construct a second loss function based on the category prediction probability value and the true category corresponding to each of the observed samples.

[0123] Step S430: Fine-tune the second local model based on the second loss function.

[0124] It can be understood that each observation sample i in the labeled observation samples is input into the second local model to obtain the category prediction probability value P corresponding to each observation sample i through the second local model. ic , P ic The predicted probability value representing the observation sample i belonging to category C. Based on the annotation information of the observation sample i, the true category corresponding to each observation sample i is determined. Then, based on the category prediction probability value and the true category corresponding to each observation sample i, a second loss function is constructed, and the second local model is fine-tuned based on the second loss function.

[0125] In a specific embodiment, the second loss function is determined by the following formula:

[0126]

[0127] Wherein, l2 is the second loss function, N is the sample size, M is the number of categories of the observed samples, and P is ic is the predicted probability value of the observed sample i belonging to category c, and the y ic is a sign function that takes the value 1 if the true category of the observation sample i is equal to c, and takes the value 0 otherwise.

[0128] In some embodiments, the server tests the integrated model using an accuracy evaluation index.

[0129] It is understandable that the classification prediction results output by an image classification model may be correct or incorrect. There are four situations corresponding to the training samples and the prediction results:

[0130] (1) The training sample is positive and the prediction result is positive, that is, the positive sample is correctly classified, which is represented by True Positive (TP);

[0131] (2) The training sample is positive, but the prediction result is negative, that is, the positive sample is misclassified, which is represented by False Negative (FN);

[0132] (3) The training sample is negative and the prediction result is also negative, that is, the negative sample is correctly classified and represented by True Negative (TN);

[0133] (4) The training sample is negative, but the prediction result is positive, which means the negative sample is misclassified and represented by False Positive (FP).

[0134] For example, for an animal image classification task, there are pictures A and B, where picture A is a dog and picture B is a cat. TP means that the model classifies picture A as a picture containing a dog, FN means that the model classifies picture A as a picture not containing a dog, TN means that the model classifies picture B as a picture not containing a dog, and FP means that the model classifies picture B as a picture containing a dog.

[0135] It can be understood that TP and TN are correct classification results, while FN and FP are incorrect classification results. Therefore, the model performance of the test model can be evaluated by the ratio of the number of correctly predicted samples to the total number of samples. Therefore, the calculation method of Accuracy can be determined by the following formula:

[0136]

[0137] For example, Figure 5 As shown, based on the accuracy evaluation index, the first intermediate model is tested by the server to obtain the accuracy of the first intermediate model in the previous cycle. And the accuracy in this cycle The first training end condition is that the accuracy of the first intermediate model tested in this cycle is greater than the accuracy in the previous cycle, that is, when the accuracy When , the loop ends and the first intermediate model is used as the pre-training model; when the accuracy , the process returns to the step of training the first local model based on the unlabeled training samples through multiple first clients, until the accuracy of the first intermediate model tested in this cycle is greater than the accuracy in the previous cycle.

[0138] For example, Figure 6 As shown, based on the accuracy evaluation index, the second intermediate model is tested by the server to obtain the accuracy of the second intermediate model in the previous cycle. And the accuracy in this cycle The second training end condition is that the accuracy of the second intermediate model tested in this cycle is less than the accuracy in the previous cycle. That is, the cycle continues until the accuracy of the second intermediate model no longer improves. Therefore, when the accuracy When , the cycle ends and the second intermediate model is used as the final pre-training model; when the accuracy , return to the step of downloading the pre-trained model from the server through multiple first clients respectively until the accuracy of the second intermediate model tested in this round of loop is less than the accuracy in the previous round of loop.

[0139] For example, Figure 7 As shown, based on the accuracy evaluation index, the third intermediate model is tested by the server to obtain the accuracy of the third intermediate model in the previous cycle. And the accuracy in this cycle The third training end condition is that the accuracy of the third intermediate model tested in this cycle is less than the accuracy in the previous cycle. In other words, the cycle continues until the accuracy of the third intermediate model no longer improves. Therefore, when the accuracy When the accuracy is , the cycle ends and the third intermediate model is used as the target model; , return to the step of downloading the pre-trained model from the server through multiple second clients respectively, until the accuracy of the third intermediate model tested in this round of loop is less than the accuracy in the previous round of loop.

[0140] It should be noted that in addition to the accuracy evaluation index provided in the above embodiment, the first intermediate model, the second intermediate model and the third intermediate model can also be tested using computer vision evaluation indicators such as precision and recall. It should also be noted that the first intermediate model, the second intermediate model and the third intermediate model can be tested using different evaluation indicators, and the embodiments of the present application do not impose too many restrictions on this.

[0141] The following describes the federated learning method provided by this application through a specific embodiment:

[0142] See also Figure 8 The federated learning method is applied to a federated learning system. The federated learning method is applied to a federated learning system, wherein the system includes a server, a plurality of first clients, and a plurality of second clients. The server is respectively in communication with the plurality of first clients and the plurality of second clients. The method includes:

[0143] Step 1: Train a first local model based on unlabeled training samples through a first client, and send the trained first local model to a server;

[0144] Step 2. Receive, through the server, first local models sent by multiple first clients, and integrate the first local models to obtain a pre-trained model;

[0145] Step 2.1. Receive, through the server, first local models sent by multiple first clients, and integrate the first local models to obtain a first intermediate model;

[0146] Step 2.2. Test the first intermediate model through the server. If the test result of the first intermediate model does not meet the first training end condition, return to step 1 and continue until the test result of the first intermediate model meets the first training end condition. The first intermediate model is used as the pre-trained model.

[0147] Step 3. Downloading pre-trained models from the server through multiple first clients, training the pre-trained models based on unlabeled training samples, and sending the trained pre-trained models to the server;

[0148] Step 4. Receive, through the server, the pre-trained models sent by the multiple first clients, and integrate the pre-trained models to obtain a second intermediate model;

[0149] Step 5. Test the second intermediate model through the server. If the test result of the second intermediate model does not meet the preset second training end condition, return to step 3 until the test result of the second intermediate model meets the second training end condition, and use the second intermediate model as the final pre-trained model;

[0150] Step 6. Download the pre-trained model from the server through multiple second clients respectively, use the pre-trained model as the second local model, fine-tune the second local model based on the labeled observation samples, and send the fine-tuned second local model to the server;

[0151] Step 7. Receive, through the server, the second local models sent by the plurality of second clients, and integrate the pre-trained model and the second local model to obtain a target model;

[0152] Step 7.1. Receive, through the server, the second local models sent by the plurality of second clients, and integrate the pre-trained model and the second local model to obtain a third intermediate model;

[0153] Step 7.2. Test the third intermediate model through the server. If the test result of the third intermediate model does not meet the preset third training end condition, return to step 6 until the test result of the third intermediate model meets the third training end condition, and use the third intermediate model as the target model.

[0154] The present application proposes a federated learning method, which is applied to a federated learning system. The system includes a server, multiple first clients and multiple second clients. The server is communicated with the multiple first clients and the multiple second clients respectively. The method includes: training a first local model based on unlabeled training samples by the multiple first clients, and sending the trained first local model to the server, receiving the first local models sent by the multiple first clients by the server, and integrating the first local models to obtain a pre-trained model, downloading the pre-trained model from the server by the multiple second clients, using the pre-trained model as the second local model, fine-tuning the second local model based on labeled observation samples, and sending the fine-tuned second local model to the server, receiving the second local models sent by the multiple second clients by the server, and integrating the pre-trained model and the second local model to obtain a target model. In an embodiment of the present application, a client that uses unlabeled data for training is added to federated learning, and a joint training method of pre-training and fine-tuning is combined. The first client pre-trains the model based on unlabeled training samples, while the second client fine-tunes the pre-trained model based on labeled observation samples. This can not only improve the generalization of the target model, but also efficiently utilize unlabeled data and save labeling costs.

[0155] See Figure 9 The embodiment of the present application further provides a federated learning system 100, which includes a server 110, a plurality of first clients 120, and a plurality of second clients 130. The server 110 is respectively in communication with the plurality of first clients 120 and the plurality of second clients 130; wherein,

[0156] The first client 120 includes:

[0157] A model training module 121 is configured to train a first local model based on unlabeled training samples;

[0158] A first sending module 122, configured to send the trained first local model to the server;

[0159] The server 110 includes:

[0160] A first receiving module 111 is configured to receive the first local models sent by a plurality of the first clients;

[0161] A first integration module 112 is configured to integrate the plurality of first local models to obtain a pre-trained model;

[0162] A second receiving module 113 is configured to receive the second local models sent by a plurality of second clients;

[0163] A second integration module 114 is configured to integrate the pre-trained model and the plurality of second local models to obtain a target model;

[0164] The second client 130 includes:

[0165] A first downloading module 131 is configured to download the pre-trained model from the server and use the pre-trained model as a second local model;

[0166] A model fine-tuning module 132, configured to fine-tune the second local model based on labeled observation samples;

[0167] The second sending module 133 is configured to send the fine-tuned second local model to the server.

[0168] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0169] See Figure 10 , Figure 10 The hardware structure of an electronic device provided in an embodiment of the present application is shown. The electronic device includes:

[0170] The processor 210 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant computer programs to implement the technical solutions provided in the embodiments of the present application.

[0171] The memory 220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 220 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called by the processor 210 to execute the federated learning method of the embodiments of this application.

[0172] Input / output interface 230, used to implement information input and output;

[0173] Communication interface 240, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and bus 250, which transmits information between each component of the device (such as processor 210, memory 220, input / output interface 230 and communication interface 240);

[0174] The processor 210 , the memory 220 , the input / output interface 230 and the communication interface 240 are connected to each other in communication within the device via the bus 250 .

[0175] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the above-mentioned federated learning method.

[0176] The memory is a computer-readable storage medium that can be used to store software programs and computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0179] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0180] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0181] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0182] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or can be integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, the functional units in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0186] If the assembled units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0187] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A federated learning method, characterized in that: The method is applied to a federated learning system, the system including a server, a plurality of first clients, and a plurality of second clients, the server being communicatively connected with the plurality of first clients and the plurality of second clients, respectively, and the method including: Performing image amplification on each image sample in the unlabeled training samples to obtain an amplified image corresponding to each image sample; Inputting the augmented image corresponding to each of the image samples into a first local model, so as to output image features corresponding to each of the augmented images through the first local model; wherein the first local model is an image classification model; Mapping the image features corresponding to each of the amplified images to the same vector space to obtain a vector representation corresponding to each of the amplified images; determining a first similarity value between the vector representations obtained based on the same image sample and a second similarity value between the vector representations obtained based on different image samples; With the goal of minimizing the first similarity value and maximizing the second similarity value, a first loss function is constructed, and self-supervised training is performed on the first local model based on the first loss function; wherein the first loss function is as follows: l1 is the first loss function, N is the number of image samples, sim() is the cosine similarity, z i ,z j is a vector representation corresponding to the image sample; Sending the trained first local model to the server; Receiving, by the server, the first local models sent by the plurality of first clients, and integrating the plurality of first local models to obtain a pre-trained model; downloading the pre-trained model from the server through a plurality of second clients, using the pre-trained model as a second local model, fine-tuning the second local model based on the labeled observation samples, and sending the fine-tuned second local model to the server; The server receives the second local models sent by multiple second clients, and integrates the pre-trained model and multiple second local models to obtain a target model.

2. The federated learning method according to claim 1, characterized in that: The receiving, by the server, the first local models sent by the plurality of first clients, and integrating the plurality of first local models to obtain a pre-trained model includes: receiving, by the server, the first local models sent by the plurality of first clients, and integrating the plurality of first local models to obtain a first intermediate model; The first intermediate model is tested through the server. If the test result of the first intermediate model does not meet the first training end condition, the step of training the first local model based on the unlabeled training samples through multiple first clients is returned to, until the test result of the first intermediate model meets the first training end condition, and the first intermediate model is used as the pre-training model.

3. The federated learning method according to claim 1, wherein: After receiving, by the server, the first local models sent by the plurality of first clients and integrating the plurality of first local models to obtain a pre-trained model, and before downloading, by the plurality of second clients, the pre-trained model from the server respectively, the method further includes: downloading the pre-trained model from the server respectively through the first clients, training the pre-trained model based on unlabeled training samples, and sending the trained pre-trained model to the server; Receiving, by the server, the pre-trained models sent by the plurality of first clients, and integrating the plurality of pre-trained models to obtain a second intermediate model; The second intermediate model is tested by the server. If the test result of the second intermediate model does not meet the preset second training end condition, the process returns to the step of downloading the pre-trained model from the server through multiple first clients respectively until the test result of the second intermediate model meets the second training end condition, and the second intermediate model is used as the final pre-trained model.

4. The federated learning method according to claim 1, wherein: The receiving, by the server, the second local models sent by the plurality of second clients, and integrating the pre-trained model and the plurality of second local models to obtain a target model includes: Receiving, by the server, the second local models sent by the plurality of second clients, and integrating the pre-trained model and the plurality of second local models to obtain a third intermediate model; The third intermediate model is tested by the server. If the test result of the third intermediate model does not meet the preset third training end condition, the process returns to the step of downloading the pre-trained model from the server through multiple second clients respectively until the test result of the third intermediate model meets the third training end condition, and the third intermediate model is used as the target model.

5. The federated learning method according to claim 1, wherein: The fine-tuning of the second local model based on the labeled observation samples includes: Inputting each observed sample in the labeled observed samples into the second local model to obtain a class prediction probability value corresponding to each of the observed samples through the second local model; Determining the true category corresponding to each of the observed samples, and constructing a second loss function based on the category prediction probability value and the true category corresponding to each of the observed samples; Fine-tune the second local model based on the second loss function.

6. A federated learning system, characterized in that: The system includes a server, a plurality of first clients and a plurality of second clients, wherein the server is in communication connection with the plurality of first clients and the plurality of second clients respectively; wherein, The first client includes: Model training module for Performing image amplification on each image sample in the unlabeled training samples to obtain an amplified image corresponding to each image sample; Inputting the augmented image corresponding to each of the image samples into a first local model, so as to output image features corresponding to each of the augmented images through the first local model; wherein the first local model is an image classification model; Mapping the image features corresponding to each of the amplified images to the same vector space to obtain a vector representation corresponding to each of the amplified images; determining a first similarity value between the vector representations obtained based on the same image sample and a second similarity value between the vector representations obtained based on different image samples; With the goal of minimizing the first similarity value and maximizing the second similarity value, a first loss function is constructed, and self-supervised training is performed on the first local model based on the first loss function; wherein the first loss function is as follows: l1 is the first loss function, N is the number of image samples, sim() is the cosine similarity, z i ,z j is a vector representation corresponding to the image sample; A first sending module, configured to send the trained first local model to the server; The server includes: A first receiving module, configured to receive the first local models sent by a plurality of the first clients; A first integration module, configured to integrate the plurality of the first local models to obtain a pre-trained model; A second receiving module, configured to receive the second local models sent by a plurality of second clients; A second integration module is used to integrate the pre-trained model and the plurality of second local models to obtain a target model; The second client includes: A first downloading module is used to download the pre-trained model from the server and use the pre-trained model as a second local model; A model fine-tuning module, configured to fine-tune the second local model based on labeled observation samples; A second sending module is used to send the fine-tuned second local model to the server.

7. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program, which is executed by the at least one processor to enable the at least one processor to perform the federated learning method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the federated learning method according to any one of claims 1 to 5 is implemented.

Images