Federal semi-supervised learning method, federal semi-supervised learning device and electronic equipment

Abstract

The invention provides a federal semi-supervised learning method and device and electronic equipment, and the method comprises the steps: a client carries out the first round of training based on an initial regular teacher model, an initial student model and an initial teacher model, obtains a student model and a teacher model of the current round, and uploads the student model and the teacher model to a server; the server side aggregates the student models of the current round to obtain a global model of the current round; and carrying out weighted averaging on the teacher model of the current round to obtain a regular teacher model of the current round, and sending the regular teacher model to the client to circularly carry out the next round of training until a global model meeting a preset condition is generated. According to the federal semi-supervised learning method and device and the electronic equipment provided by the embodiment of the invention, the robustness and generalization ability of the models facing non-independent identically distributed data can be improved by combining the prediction of the local teacher model and the regular teacher model.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a federated semi-supervised learning method, device and electronic equipment. Background Art

[0002] Traditional federated learning methods use methods such as introducing aggregation weights and adding penalty items to curb the divergence problem when the model is aggregated. Although this alleviates the problem of data heterogeneity to a certain extent, the performance of most methods is still low under highly heterogeneous data distribution. At the same time, due to this data heterogeneity, there may be a risk of overfitting. The model may overfit on some clients, resulting in client deviation. Summary of the invention

[0003] To solve the existing technical problems, the embodiments of the present invention provide a federated semi-supervised learning method, device, electronic device and computer-readable storage medium.

[0004] In the first aspect, an embodiment of the present invention provides a federated semi-supervised learning method, comprising: a server sends an initial regular teacher model obtained by initialization to multiple clients, each of the clients performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model obtained by local initialization to obtain a student model and a teacher model of the current round; each of the clients uploads the student model and the teacher model of the current round to the server, and the server aggregates the student model of the current round to obtain a global model of the current round; the server performs a weighted average of the teacher models of the current round to obtain a regular teacher model of the current round; the server sends the global model of the current round and the regular teacher model of the current round to multiple clients for the next round of local training in a loop until a global model that meets preset conditions is generated.

[0005] Optionally, the server aggregates the student models of the current round to obtain the global model of the current round, including: the server aggregates the student models of the current round based on an aggregation strategy weighted by model parameter distance to obtain the global model of the current round.

[0006] Optionally, each of the clients performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model obtained by local initialization to obtain the student model and teacher model of the current round, including: the client uses labeled data to train the initial student model to obtain the first prediction result; the client uses data-enhanced unlabeled data to train the initial student model to obtain the second prediction result; the client uses unlabeled data to train the initial teacher model to obtain the third prediction result; the client uses unlabeled data to train the initial regular teacher model to obtain the fourth prediction result; based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the student model and teacher model of the current round are obtained.

[0007] Optionally, based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the student model and the teacher model of the current round are obtained, including: the initial student model is learned by minimizing the loss between the first prediction result and the true label, the initial teacher model is learned by minimizing the loss between the third prediction result and the second prediction result, and the initial regular teacher model is learned by minimizing the loss between the fourth prediction result and the second prediction result, so as to obtain the total loss of the local training of the client in the current round, and obtain the student model and the teacher model of the current round based on the total loss.

[0008] Optionally, the total loss of the current round of local training of the client is obtained, including: introducing hyperparameters into the loss between the third prediction result and the second prediction result and the loss between the fourth prediction result and the second prediction result, and the hyperparameters are used to balance the contribution between the third prediction result generated by the initial teacher model and the fourth prediction result generated by the initial regular teacher model.

[0009] Optionally, the loss between the first prediction result and the true label is the supervision loss Lsup; the loss between the third prediction result and the second prediction result is the consistency loss Lcon; and the loss between the fourth prediction result and the second prediction result is the consistency regularization loss Lreg.

[0010] In the second aspect, an embodiment of the present invention also provides a federated semi-supervised learning device, comprising: a client training module, a server aggregation module and a loop training module; the client training module is used for the server to send the initial regular teacher model initialized to multiple clients, and each client performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model initialized locally to obtain the student model and teacher model of the current round; the server aggregation module is used for each client to upload the student model and teacher model of the current round to the server, and the server aggregates the student model of the current round to obtain the global model of the current round; the server performs weighted average of the teacher models of the current round to obtain the regular teacher model of the current round; the loop training module is used for the server to send the global model of the current round and the regular teacher model of the current round to multiple clients for the next round of local training in a loop until a global model that meets preset conditions is generated.

[0011] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory, and when the computer program is executed by the processor, the federated semi-supervised learning method described in the first aspect is implemented.

[0012] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the federated semi-supervised learning method described in the first aspect above is implemented.

[0013] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed, implements the federated semi-supervised learning method described in the first aspect or any possible design method of the first aspect.

[0014] The federated semi-supervised learning method, device, electronic device and computer-readable storage medium provided by the embodiments of the present invention are different from the traditional federated semi-supervised algorithm. It is proposed to add a regular teacher model to prevent the phenomenon of client deviation, and aggregate the teacher model of each client to obtain a regular teacher model. Similar to the global model, the regular teacher model is also distributed to each client by the server before the client training. During the local training process of the client, the regular teacher model and the local teacher model of the client jointly guide the training of the local student model to deal with the problem of client deviation caused by the non-independent and identically distributed (Non-IID) problem. At the same time, the effect of joint guidance of the regular teacher model and the local teacher model of the client is achieved. By combining the predictions of the local teacher model and the regular teacher model, the embodiments of the present invention can improve the robustness and generalization ability of the model when facing non-independent and identically distributed data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0016] Figure 1 A flowchart of a federated semi-supervised learning method provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of a framework of a federated semi-supervised learning method provided in an embodiment of the present invention is shown; Figure 3 A schematic diagram of the structure of a federated semi-supervised learning device provided by an embodiment of the present invention is shown; Figure 4 A schematic diagram of the structure of an electronic device for executing a federated semi-supervised learning method provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0017] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0018] Figure 1 FIG. 4 is a flow chart of a federated semi-supervised learning method provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps 101-103.

[0019] Step 101: The server sends the initialized initial regular teacher model to multiple clients. Each client performs the first round of local training based on the initial regular teacher model and the initial student model and initial teacher model initialized locally to obtain the student model and teacher model of the current round.

[0020] In an embodiment of the present invention, the server needs to initialize a regular teacher model, that is, an initial regular teacher model, and send the initial regular teacher model to each client to provide guidance for the client's training. Correspondingly, each client also needs to initialize and generate a student model and a teacher model locally, that is, each client generates an initial student model and an initial teacher model to start the first round of local training; it can be understood that the training process of the model is a cyclic training process of continuous adjustment and optimization. Among them, the initialization process is: the server randomly initializes the parameters of the learnable lightweight module, and these parameters are then distributed to each client for communication and local optimization. The initialization process of the regular teacher model is the same as the initialization process of the student model and the local teacher model, and can be directly initialized.

[0021] After each client receives the initial regular teacher model sent by the server, it can use the initial regular teacher model, the initial student model and the initial teacher model to perform the first round of local training, that is, each client performs local training in parallel, and then obtains the trained student model and teacher model, that is, the student model and teacher model of the current round. It should be noted that if this is the first round of local training, the student model and teacher model of the current round can be understood as the student model and teacher model of the first round.

[0022] Step 102: Each client uploads the student model and teacher model of the current round to the server, and the server aggregates the student model of the current round to obtain the global model of the current round; the server performs weighted average of the teacher model of the current round to obtain the regular teacher model of the current round.

[0023] After the model training process of the above step 101, each client can upload the student model and teacher model of the current round obtained by training to the server, and the server will process the student model of the current round uploaded by multiple clients and the teacher model of the current round uploaded by multiple clients respectively. Specifically, the server can aggregate the student models of the current round to obtain the global model of the current round. The global model can be understood as the optimized student model. The aggregation process can be aggregated using the traditional federated weighted average aggregation strategy, which is not specifically limited in the embodiment of the present invention. Correspondingly, the server can use the weighted average method to obtain the regular teacher model of the current round for the teacher model of the current round. For example, the server can respectively aggregate or weighted average the student models of multiple first rounds and the teacher models of multiple first rounds using the traditional federated weighted average aggregation strategy to obtain the global model of the first round and the regular teacher model of the first round.

[0024] Step 103: The server sends the global model of the current round and the regular teacher model of the current round to multiple clients for the next round of local training in a loop until a global model that meets preset conditions is generated.

[0025] After the global model of the current round is obtained by aggregation and the regular teacher model of the current round is obtained by weighted average, the server continues to send the two to multiple clients, so that the client can continue the next round of local training based on the global model and regular teacher model optimized by the server. For example, after the global model of the first round is obtained by aggregation and the regular teacher model of the first round is obtained by weighted average, the server continues to send the two to multiple clients, and each client performs the second round of local training based on the global model of the first round (regarded as the student model of the current round) and the regular teacher model of the first round, that is, the current round is the second round at this time. It should be noted that the local training process of each round in the embodiment of the present invention is performed by the same technical means until the preset conditions for jumping out of the loop are met, and the global model of the current round that meets the preset conditions can be used as the final output result. The preset condition can be a preset number of cycles, or it can also be a preset accuracy that the global model needs to meet, etc., which is not limited in the embodiment of the present invention.

[0026] The federated semi-supervised learning method provided by the embodiment of the present invention is different from the traditional federated semi-supervised algorithm. It proposes to add a regular teacher model to prevent the phenomenon of client deviation, and aggregate the teacher model of each client to obtain a regular teacher model. Similar to the global model, the regular teacher model is also distributed to each client by the server before the client training. During the local training process of the client, the regular teacher model and the local teacher model of the client jointly guide the training of the local student model to deal with the problem of client deviation caused by the non-independent and identically distributed (Non-IID) problem. At the same time, the effect of joint guidance of the regular teacher model and the local teacher model of the client is achieved. By combining the predictions of the local teacher model and the regular teacher model, the embodiment of the present invention can improve the robustness and generalization ability of the model when facing non-independent and identically distributed data.

[0027] The inventors further discovered that in the application of federated learning, when faced with non-IID distributed data, the traditional average weighted aggregation method may not be sufficient to achieve the best global model performance. This is because the weighted average aggregation ignores the differences in data distribution between clients, which may cause the global model to be biased towards clients with large data volumes or feature distributions that deviate from the global average. In order to solve this problem, the embodiments of the present invention propose further improvements.

[0028] Optionally, the above step 103 "the server aggregates the student models of the current round to obtain the global model of the current round" may include: the server aggregates the student models of the current round based on the model parameter distance weighted aggregation strategy to obtain the global model of the current round. That is, the aggregation strategy adopted in the embodiment of the present invention may be a model parameter distance weighted aggregation strategy (Model Parameter Distance-Weighted Aggregation Strategy, DWA), which optimizes the aggregation process by considering the difference between each client model and the global model.

[0029] The core idea of ​​the model parameter distance-weighted aggregation strategy (DWA) is that the difference between the client model parameters and the weighted average of the global model parameters can indirectly reflect the degree of difference between the client data characteristics and the global data characteristics. A smaller distance means that the client's data distribution may be closer to the global average distribution, so its model parameters should be given a larger weight during aggregation. On the contrary, a client with a larger distance may mean that its data distribution deviates greatly from the global distribution, and its contribution to the global model should be relatively reduced to prevent the global model from being overly biased towards these special data distributions. The calculation of the weight takes two factors into consideration: one is the amount of data on the client, and the other is the distance between its model parameters and the global average model parameters. This embodiment of the present invention introduces a hyperparameter To adjust the impact of distance, we can balance the differences in data volume and data distribution during the aggregation process. The weighted average form allows the global model to better capture the commonalities of all client data while mitigating the negative impact of non-IID data distribution on model performance. For example, the hyperparameter To adjust the effect of distance, first, , used to represent each client Global model with weighted average calculation The difference between represents the distance weight between the client model parameters and the global average model parameters, Indicates the amount of data on the client side; it will be normalized later , where K represents the number of clients; finally, the normalized difference aggregation is used to obtain the global model ; Hyperparameters The value range is [0.9, 0.99].

[0030] This aggregation method based on weighted model parameter distance not only improves the robustness of the model in the face of complex data distribution, but also helps to accelerate the convergence of the global model, thereby achieving better performance in federated learning scenarios with non-independent and identically distributed (Non-IID) data distribution.

[0031] Optionally, the above step 101 "each client performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model obtained by local initialization to obtain the student model and teacher model of the current round" may include the following step A.

[0032] Step A: The client uses labeled data to train the initial student model to obtain a first prediction result; the client uses unlabeled data to train the initial student model to obtain a second prediction result; the client uses data-enhanced unlabeled data to train the initial teacher model to obtain a third prediction result; the client uses unlabeled data to train the initial regular teacher model to obtain a fourth prediction result; based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the student model and teacher model of the current round are obtained.

[0033] Among them, Figure 2 As shown, Figure 2 The framework diagram of the federated semi-supervised learning method is shown in FIG. Each client uses labeled data D l To train the local initial student model to get the first prediction result y l Each client can also perform unlabeled data D u Perform data enhancement to obtain enhanced unlabeled data (D u ), using unlabeled data with data augmentation (D u ) to train the local initial student model to obtain the prediction result y u Each client uses unlabeled data D u Train the local initial teacher model and get the third prediction result Each client uses unlabeled data D u Train the received initial regularized teacher model and get the fourth prediction result Among them, the regular teacher model can provide additional guidance for the student model, and the local teacher model can also participate in guiding the training of the student model. That is to say, in a round of local training, the prediction results of the regular teacher model (i.e., the fourth prediction result) and the prediction results of the local teacher model (i.e., the third prediction result) can both guide the student model. For example, The third prediction result and the second prediction result y u By comparison, the student model and teacher model of the current round are obtained comprehensively.

[0034] Optionally, the above step A of "obtaining the student model and teacher model of the current round based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result" may include the following step B.

[0035] Step B: The initial student model minimizes the first prediction result y l The initial teacher model learns by minimizing the loss between the third prediction result and the second prediction result y u The initial regular teacher model learns by minimizing the fourth prediction result and the second prediction result y u The total loss of the client's local training in the current round is obtained by learning the loss between , based on the total loss The student model and the teacher model of the current round are obtained. It should be noted that the loss function of the unlabeled data in the embodiment of the present invention can also use the cross entropy loss function.

[0036] Optionally, the above step B of "obtaining the total loss of the client's current round of local training" may include the following step C.

[0037] Step C: Introducing hyperparameters in the loss between the third prediction result and the second prediction result and the loss between the fourth prediction result and the second prediction result, the hyperparameters are used to balance the contribution between the third prediction result generated by the initial teacher model and the fourth prediction result generated by the initial regular teacher model.

[0038] Specifically, by the third prediction result corresponding to the teacher model and the second prediction result y u Introduce a hyperparameter in the loss between , the fourth prediction result corresponding to the regular teacher model and the second prediction result y u Introduce another hyperparameter in the loss hyperparameter between This method allows the algorithm to adjust the intensity of the influence of the regular teacher model on the student model training to adapt to different data distributions and training requirements. Specifically, The choice of the value reflects the importance of guidance from the regular teacher model relative to guidance from the local teacher model when pursuing model consistency. This hyperparameter The range is [0,0.1] and increases smoothly with training rounds. The value means that the prediction of the regular teacher model has a higher weight when calculating the loss, which is very important when dealing with client data distributions that are highly non-IID. At the same time, the use of soft labels captures information about the global data distribution, which helps guide the student model to learn a more generalized feature representation, thereby alleviating model bias caused by differences in data distribution between clients.

[0039] Optionally, the loss between the first prediction result and the true label is the supervision loss L sup , supervision loss L sup The second prediction result y used to measure the student model u The difference between the third prediction result and the second prediction result is the consistency loss L con , by calculating the third prediction result and the second prediction result y of the student model u For comparison, we introduce the consistency loss L con . Consistency loss L con Used for training unlabeled data to measure the second prediction result y of the student model u The third prediction result with the local teacher model The difference between the fourth prediction result and the second prediction result is the consistency regularization loss L reg , by calculating the fourth prediction result And the second prediction result y of the student model u For comparison, we introduce the consistency regularization loss L reg . Consistency regularization loss L reg The second prediction result y used to measure the student model u The fourth prediction result with the regular teacher model The difference between.

[0040] The pseudo code of this algorithm is provided below to illustrate the entire algorithm process.

[0041] This embodiment is an iterative process. K clients are marked as k, B represents the batch size of local training, E represents the number of local training rounds on the client, I represents the number of iterations of local training, N represents the amount of local data on the client, and T represents the total number of rounds of federated learning. represents the learning rate. The global model, student model, teacher model, and regular teacher model are respectively represented as , , , and The input is the data and model of N clients, and the output is the trained global model. .

[0042] procedure Server executes: For each client Initialize the student model with the same parameters

[0043] For all clients , initialize the local teacher model For the same parameters Initialize the regular teacher model as

[0044] for each round do for each client in parallel do

[0045] end for

[0046]

[0047]

[0048]

[0049]

[0050] end for return

[0051] end procedure procedure

[0052] Initializing the loss

[0053] for each local epoch do for each iteration do

[0054]

[0055]

[0056]

[0057] Update the client's model parameters using the gradient descent algorithm

[0058]

[0059] end for end for end procedure Based on the above, the total loss of client local training is calculated as:

[0060] The total loss consists of three parts, namely, the supervised learning loss for labeled data sets is a combination of the cross entropy function and the Dice loss function:

[0061] The semi-supervised learning method for unlabeled data uses the consistency loss between the local teacher model and the student model to improve the model's learning ability for unlabeled data. The local teacher model generates soft labels for unlabeled data to guide the learning of the student model. The consistency loss is expressed as:

[0062] In order to use the regular teacher model and the local teacher model to jointly guide the training of the local model to deal with the client offset problem caused by the Non-IID problem, there is also a regular loss of the regular teacher model and the local student model:

[0063] The total loss of the final local training consists of three parts, which are respectively determined by two hyperparameters and They are used to balance the importance of supervision loss and consistency loss and to balance the contribution between the predicted labels generated by the local teacher model and the predicted labels generated by the regular teacher model, respectively.

[0064] in, represents the exponential moving average, Indicates the total number of data on the client. Indicates the use of parameters For unlabeled data The predicted output of the student model is represents the two-norm, represents the cross entropy loss function, represents the Dice loss function, Indicates that the parameters used are The model has labeled data The predicted output of the input, represents the true label.

[0065] The above describes in detail the federated semi-supervised learning method provided by an embodiment of the present invention. The method can also be implemented by a corresponding device. The following describes in detail the federated semi-supervised learning device provided by an embodiment of the present invention.

[0066] Figure 3 FIG. 2 shows a schematic diagram of the structure of a federated semi-supervised learning device provided by an embodiment of the present invention. Figure 3 As shown, the federated semi-supervised learning device includes a processor. The processor includes: a client training module 31, a server aggregation module 32 and a loop training module 33.

[0067] The client training module 31 is used for the server to send the initialized initial regular teacher model to multiple clients. Each client performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model initialized locally to obtain the student model and teacher model of the current round.

[0068] The server aggregation module 32 is used for each of the clients to upload the student model and teacher model of the current round to the server, and the server aggregates the student model of the current round to obtain the global model of the current round; the server performs weighted average of the teacher model of the current round to obtain the regular teacher model of the current round.

[0069] The cyclic training module 33 is used for the server to send the global model of the current round and the regular teacher model of the current round to multiple clients for the next round of local training until a global model that meets preset conditions is generated.

[0070] Optionally, the cyclic training module 33 includes: a DWA aggregation unit.

[0071] The DWA aggregation unit is used for the server to aggregate the student model of the current round based on the model parameter distance weighted aggregation strategy to obtain the global model of the current round.

[0072] Optionally, the client training module 31 includes: a training submodule.

[0073] The training submodule is used for the client to train the initial student model using labeled data to obtain a first prediction result; the client to train the initial student model using data-enhanced unlabeled data to obtain a second prediction result; the client to train the initial teacher model using unlabeled data to obtain a third prediction result; the client to train the initial regular teacher model using unlabeled data to obtain a fourth prediction result; based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the student model and teacher model of the current round are obtained.

[0074] Optionally, the training submodule includes: a training unit.

[0075] The training unit is used to learn the initial student model by minimizing the loss between the first prediction result and the true label, the initial teacher model by minimizing the loss between the third prediction result and the second prediction result, and the initial regular teacher model by minimizing the loss between the fourth prediction result and the second prediction result, to obtain the total loss of the current round of local training of the client, and to obtain the student model and teacher model of the current round based on the total loss.

[0076] Optionally, the training unit includes: a training subunit.

[0077] The training subunit introduces hyperparameters in the loss between the third prediction result and the second prediction result and the loss between the fourth prediction result and the second prediction result, and the hyperparameters are used to balance the contribution between the third prediction result generated by the initial teacher model and the fourth prediction result generated by the initial regular teacher model.

[0078] Optionally, the loss between the first prediction result and the true label is the supervision loss Lsup; the loss between the third prediction result and the second prediction result is the consistency loss Lcon; and the loss between the fourth prediction result and the second prediction result is the consistency regularization loss Lreg.

[0079] The device provided by the embodiment of the present invention is different from the traditional federated semi-supervised algorithm. It proposes to add a regular teacher model to prevent the phenomenon of client deviation, and aggregate the teacher model of each client to obtain a regular teacher model. Similar to the global model, the regular teacher model is also distributed to each client by the server before the client training. During the local training process of the client, the regular teacher model and the local teacher model of the client jointly guide the training of the local student model to deal with the problem of client deviation caused by the non-independent and identically distributed (Non-IID) problem. At the same time, the effect of joint guidance of the regular teacher model and the local teacher model of the client is achieved. By combining the predictions of the local teacher model and the regular teacher model, the embodiment of the present invention can improve the robustness and generalization ability of the model when facing non-independent and identically distributed data.

[0080] It should be noted that the federated semi-supervised learning device provided in the above embodiment only uses the division of the above functional modules as an example to implement the corresponding functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the federated semi-supervised learning device provided in the above embodiment and the federated semi-supervised learning method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0081] According to one aspect of the present application, an embodiment of the present invention further provides a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part. When the computer program is executed by the processor, the federated semi-supervised learning method provided by the embodiment of the present application is executed.

[0082] In addition, an embodiment of the present invention further provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are respectively connected via a bus. When the computer program is executed by the processor, each process of the above-mentioned federated semi-supervised learning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0083] For details, see Figure 4 As shown, the electronic device includes a bus 1110 , a processor 1120 , a transceiver 1130 , a bus interface 1140 , a memory 1150 , and a user interface 1160 .

[0084] In an embodiment of the present invention, the electronic device further includes: a computer program stored in the memory 1150 and executable on the processor 1120, and when the computer program is executed by the processor 1120, each process of the above-mentioned federated semi-supervised learning method embodiment is implemented.

[0085] The transceiver 1130 is configured to receive and send data under the control of the processor 1120 .

[0086] In an embodiment of the present invention, a bus architecture (represented by bus 1110 ) may include any number of interconnected buses and bridges, and bus 1110 connects various circuits including one or more processors represented by processor 1120 and a memory represented by memory 1150 .

[0087] Bus 1110 represents one or more of any of several types of bus structures, including a memory bus and memory controller, a peripheral bus, an Accelerate Graphical Port (AGP), a processor, or a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include: Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA), Peripheral Component Interconnect (PCI) bus.

[0088] The processor 1120 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment may be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processors include: a general processor, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a programmable logic array (PLA), a microcontroller unit (MCU) or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. For example, the processor may be a single-core processor or a multi-core processor, and the processor may be integrated into a single chip or located on multiple different chips.

[0089] Processor 1120 can be a microprocessor or any conventional processor. The method steps disclosed in the embodiment of the present invention can be directly executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a readable storage medium known in the art, such as a random access memory (RAM), a flash memory (FlashMemory), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a register, etc. The readable storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0090] The bus 1110 may also connect various other circuits such as peripheral devices, voltage regulators or power management circuits, and the bus interface 1140 provides an interface between the bus 1110 and the transceiver 1130, which are well known in the art. Therefore, the embodiments of the present invention will not be further described.

[0091] The transceiver 1130 may be one element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on a transmission medium. For example, the transceiver 1130 receives external data from other devices, and the transceiver 1130 is used to send data processed by the processor 1120 to other devices. Depending on the nature of the computer system, a user interface 1160 may also be provided, such as a touch screen, a physical keyboard, a display, a mouse, a speaker, a microphone, a trackball, a joystick, and a stylus.

[0092] It should be understood that in an embodiment of the present invention, the memory 1150 may further include a memory remotely arranged relative to the processor 1120, and these remotely arranged memories may be connected to the server through a network. One or more parts of the above-mentioned network may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a public switched telephone network (PSTN), a plain old telephone service network (POTS), a cellular telephone network, a wireless network, a wireless fidelity (Wi-Fi) network, and a combination of two or more of the above-mentioned networks. For example, the cellular telephone network and the wireless network can be a Global System for Mobile Communications (GSM) system, a Code Division Multiple Access (CDMA) system, a Worldwide Interoperability for Microwave Access (WiMAX) system, a General Packet Radio Service (GPRS) system, a Wideband Code Division Multiple Access (WCDMA) system, a Long Term Evolution (LTE) system, a LTE Frequency Division Duplex (FDD) system, a LTE Time Division Duplex (TDD) system, an Advanced Long Term Evolution (LTE-A) system, a Universal Mobile Telecommunications (UMTS) system, an Enhanced Mobile Broadband (eMBB) system, a Massive Machine Type of Communication (mMTC) system, an Ultra Reliable Low Latency Communications (uRLLC) system, etc.

[0093] It should be understood that the memory 1150 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory includes: a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0094] Volatile memory includes: Random Access Memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as: Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM) and Direct Rambus RAM (DRRAM). The memory 1150 of the electronic device described in the embodiment of the present invention includes but is not limited to the above and any other suitable types of memory.

[0095] In the embodiment of the present invention, the memory 1150 stores the following elements of the operating system 1151 and the application program 1152: executable modules, data structures, or subsets thereof, or extended sets thereof.

[0096] Specifically, the operating system 1151 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 1152 includes various applications, such as a media player (Media Player) and a browser (Browser), which are used to implement various application services. The program for implementing the method of the embodiment of the present invention may be included in the application 1152. The application 1152 includes applets, objects, components, logic, data structures, and other computer system executable instructions that perform specific tasks or implement specific abstract data types.

[0097] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned federated semi-supervised learning method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0098] Computer readable storage media include: permanent and non-permanent, removable and non-removable media, which are tangible devices that can retain and store instructions for use by instruction execution devices. Computer readable storage media include: electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, and any suitable combination of the above. Computer readable storage media include: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassette storage, magnetic tape disk storage or other magnetic storage devices, memory sticks, mechanical encoding devices (such as punched cards or raised structures in grooves with instructions recorded thereon) or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined in the embodiments of the present invention, computer-readable storage media do not include temporary signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one location or distributed on multiple network units. Some or all of the units may be selected according to actual needs to solve the problem to be solved by the embodiments of the present invention.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (including: a personal computer, a server, a data center or other network equipment) to perform all or part of the steps of the method described in each embodiment of the present invention. The above-mentioned storage medium includes various media that can store program codes as listed above.

[0103] In the description of the embodiments of the present invention, those skilled in the art should know that the embodiments of the present invention can be implemented as methods, devices, electronic devices and computer-readable storage media. Therefore, the embodiments of the present invention can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), a combination of hardware and software. In addition, in some embodiments, the embodiments of the present invention can also be implemented in the form of a computer program product in one or more computer-readable storage media, and the computer-readable storage medium contains computer program code.

[0104] The above-mentioned computer-readable storage medium may adopt any combination of one or more computer-readable storage media. Computer-readable storage media include: electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices or any combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device, or device.

[0105] The computer program code contained in the above-mentioned computer-readable storage medium can be transmitted by any appropriate medium, including: wireless, wire, optical cable, radio frequency (RF) or any suitable combination thereof.

[0106] The computer program code for performing the operation of the embodiment of the present invention can be written in assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or in one or more programming languages ​​or a combination thereof, wherein the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as C language or a similar programming language. The computer program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, and completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer or to an external computer through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0107] The embodiments of the present invention describe the provided methods, devices, and electronic devices through flowcharts and / or block diagrams.

[0108] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer-readable program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified by the boxes in the flowchart and / or block diagram.

[0109] These computer-readable program instructions may also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to work in a specific manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction device product including functions / operations specified in the blocks in the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable data processing apparatus to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0111] The above is only a specific implementation of the embodiment of the present invention, but the protection scope of the embodiment of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiment of the present invention, which should be included in the protection scope of the embodiment of the present invention. Therefore, the protection scope of the embodiment of the present invention should be based on the protection scope of the claims.

Claims

1. A federated semi-supervised learning method, characterized in that: include: The server sends the initialized initial regular teacher model to multiple clients, and each client performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model initialized locally to obtain the student model and teacher model of the current round; Each of the clients uploads the student model and the teacher model of the current round to the server, and the server aggregates the student models of the current round to obtain a global model of the current round; The server performs weighted averaging of the teacher models of the current round to obtain a regular teacher model of the current round; The server sends the global model of the current round and the regular teacher model of the current round to multiple clients for the next round of local training until a global model that meets preset conditions is generated.

2. The method according to claim 1, characterized in that The server aggregates the student models of the current round to obtain the global model of the current round, including: the server aggregates the student models of the current round based on an aggregation strategy weighted by model parameter distance to obtain the global model of the current round.

3. The method according to claim 1, characterized in that Each of the clients performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model obtained by local initialization to obtain the student model and teacher model of the current round, including: The client uses labeled data to train the initial student model to obtain a first prediction result; the client uses data-enhanced unlabeled data to train the initial student model to obtain a second prediction result; the client uses unlabeled data to train the initial teacher model to obtain a third prediction result; the client uses unlabeled data to train the initial regular teacher model to obtain a fourth prediction result; based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the student model and teacher model of the current round are obtained.

4. The method according to claim 3, characterized in that The obtaining of the student model and the teacher model of the current round based on the first prediction result, the second prediction result, the third prediction result and the fourth prediction result comprises: The initial student model is learned by minimizing the loss between the first prediction result and the true label, the initial teacher model is learned by minimizing the loss between the third prediction result and the second prediction result, and the initial regular teacher model is learned by minimizing the loss between the fourth prediction result and the second prediction result, and the total loss of the current round of local training of the client is obtained, and the student model and teacher model of the current round are obtained based on the total loss.

5. The method according to claim 4, characterized in that The obtaining of the total loss of the current round of local training of the client includes: introducing hyperparameters into the loss between the third prediction result and the second prediction result and the loss between the fourth prediction result and the second prediction result, wherein the hyperparameters are used to balance the contribution between the third prediction result generated by the initial teacher model and the fourth prediction result generated by the initial regular teacher model.

6. The method according to claim 4, characterized in that The loss between the first prediction result and the true label is the supervision loss L sup The loss between the third prediction result and the second prediction result is the consistency loss L con ; The loss between the fourth prediction result and the second prediction result is the consistency regularization loss L reg .

7. A federated semi-supervised learning device, characterized in that: include: Client training module, server aggregation module and loop training module; The client training module is used for the server to send the initial regular teacher model initialized to multiple clients, and each client performs a first round of local training based on the initial regular teacher model and the initial student model and initial teacher model initialized locally to obtain the student model and teacher model of the current round; The server aggregation module is used for each client to upload the student model and teacher model of the current round to the server, and the server aggregates the student model of the current round to obtain the global model of the current round; the server performs weighted average of the teacher model of the current round to obtain the regular teacher model of the current round; The cyclic training module is used for the server to send the global model of the current round and the regular teacher model of the current round to multiple clients for the next round of local training until a global model that meets preset conditions is generated.

8. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: The processor executes the computer program stored in the memory to implement the steps in the federated semi-supervised learning method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the federated semi-supervised learning method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises a computer program, which, when executed, implements the steps in the federated semi-supervised learning method as claimed in any one of claims 1 to 6.

Images