Federated learning method, device, computer equipment, storage medium and program product

Abstract

Embodiments of the present application disclose a federated learning method and device, computer equipment, a storage medium and a program product, and relate to the field of artificial intelligence. The method comprises: obtaining a local model of a client; extracting local data features of the client from local training samples of a local training sample set by using the local model of the client; mapping the local data features of the client to local latent space features of the client, the local latent space features of the client being feature representations of the local training samples in a latent space, and the similarity between the local latent space features of different clients being higher than the similarity between the local data features of different clients; and sending the local latent space features of the client to a server in a federated learning, so that the server aggregates local models of different clients based on the local latent space features of the different clients to obtain an aggregated model. The present application can enhance the consistency of feature expression of the local models of each client.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a federated learning method, apparatus, computer device, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence technology, federated learning can enable multiple clients (such as terminal devices in different hospitals) to collaboratively train a deep learning model (such as a classification or segmentation model for medical images).

[0003] In related technologies, federated learning is typically divided into a local training phase and a global aggregation phase. During the local training phase, each client trains its own model using its own training samples. In the global aggregation phase, the aggregator combines the local models from each client to obtain an aggregated model, which is then returned to the respective clients. Because the training samples of each client are not known to other clients, federated learning can guarantee data security among the parties involved.

[0004] However, the device models and parameters of different clients may differ (for example, medical images of the same patient and the same body part collected by terminal devices in different hospitals may differ), which may affect the consistency of feature representation in the local models of different clients. This causes each client to optimize towards the local optimal solution instead of focusing on the global objective. This will cause the aggregate model to be interfered with by mismatched local optimal solutions, resulting in optimization deviation of the aggregate model and reducing the predictive ability of the aggregate model. Summary of the Invention

[0005] This application provides a federated learning method, apparatus, computer device, storage medium, and program product. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a federated learning method, which is executed by a client in federated learning, and the method includes:

[0007] Obtain the client's local model, which is trained based on the local training sample set;

[0008] Using the client's local model, local data features are extracted from the local training samples in the local training sample set.

[0009] The local data features of the client are mapped to the local latent space features of the client. The local latent space features of the client are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0010] The local latent space features of the client are sent to the server in the federated learning process, so that the server can aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model.

[0011] On the other hand, embodiments of this application provide a federated learning apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire the local model of the client, which is trained based on the local training sample set.

[0013] The local data feature extraction module is used to extract the local data features of the client from the local training samples of the local training sample set through the client's local model;

[0014] The local latent space feature extraction module is used to map the local data features of the client to the local latent space features of the client. The local latent space features of the client are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0015] A sending module is used to send the local latent space features of the client to the server in the federated learning process, so that the server can aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model. On the other hand, embodiments of this application provide a federated learning method, which is executed by the server in the federated learning process, and the method includes:

[0016] The local latent space features of different clients in federated learning are received. The local latent space features of a client are obtained by mapping the local data features of the client. The local data features of the client are extracted from the local training samples in the local training sample set through the local model of the client. The local latent space features of the client are the feature representation of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0017] Based on the local latent space features of the different clients, the local models of the different clients are aggregated to obtain an aggregated model; the local models of the clients are trained based on the local training sample set.

[0018] On the other hand, embodiments of this application provide a federated learning apparatus, the apparatus comprising:

[0019] The receiving module is used to receive local latent space features from different clients in federated learning. The local latent space features of a client are obtained by mapping the local data features of the client. The local data features of the client are extracted from the local training samples in the local training sample set through the local model of the client. The local latent space features of the client are the feature representation of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0020] The aggregation module is used to aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model; the local models of the clients are trained based on the local training sample set.

[0021] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the federated learning method as described above.

[0022] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the federated learning method as described above.

[0023] On the other hand, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the federated learning method provided in various optional implementations of the above aspects.

[0024] In this embodiment, the client trains a local model using a local training sample set and extracts local data features from the local training samples in the local training sample set. These local data features reflect the data features of the client's local training sample set. The local data features are then mapped to local latent space features, allowing the latent space features to reflect the feature representation of the local training sample set in the latent space (e.g., reflecting the brightness of medical images in the local training sample set). Since the similarity between local latent space features of different clients is higher than the similarity between local data features of different clients, the consistency of local latent space feature expressions across clients is improved compared to the consistency between local data features across clients. By improving the consistency of local latent space feature expressions across clients, different clients can learn more common features.

[0025] Each client sends its local latent space features to the server, which then aggregates the local models based on these features. This reduces the impact of data distribution differences between clients. Since each client learns more common features, it focuses more on the global objective rather than the local optimization objective during training, avoiding the problem of local update bias. This results in better aggregation performance and improved accuracy of the aggregated model.

[0026] Meanwhile, since the aggregation model is obtained by each client learning more common features, when the aggregation model is applied to different clients, even if the data distribution between the clients is very different, the aggregation model can achieve relatively stable performance in each client, thus improving the fairness of the aggregation model to each client. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram illustrating how frequency-domain-based standardization methods improve client consistency in related technologies.

[0029] Figure 2 This is a schematic diagram of a computer system in a federated learning scenario provided by an exemplary embodiment of this application;

[0030] Figure 3 This is a flowchart of a federated learning method provided in an exemplary embodiment of this application;

[0031] Figure 4 This is a schematic diagram of consistency training provided in an exemplary embodiment of this application;

[0032] Figure 5 This is a schematic diagram of a local model provided in an exemplary embodiment of this application, which includes network layers of different levels.

[0033] Figure 6 This is a schematic diagram illustrating feature redundancy in the context of improving client consistency, provided by an exemplary embodiment of this application.

[0034] Figure 7 This is a schematic diagram of feature whitening within a client provided in an exemplary embodiment of this application;

[0035] Figure 8This is a schematic diagram illustrating the grouping of training sample data features according to feature channels, provided in an exemplary embodiment of this application.

[0036] Figure 9 This is a flowchart of a federated learning method provided in another embodiment of this application;

[0037] Figure 10 This is a schematic diagram illustrating how a server aggregates the local models of each client based on aggregation weights to obtain an aggregated model, as provided in an exemplary embodiment of this application.

[0038] Figure 11 This is a structural block diagram of a federated learning device provided in an exemplary embodiment of this application;

[0039] Figure 12 This is a structural block diagram of a federated learning device provided in another exemplary embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] First, the terms used in the embodiments of this application will be introduced.

[0043] Federated Machine Learning (FLM) is a distributed artificial intelligence model learning method that enables multiple organizations to use data and perform machine learning modeling while adhering to user privacy protection, data security, and legal regulations. FLM effectively solves the data silo problem, allowing multiple organizations to jointly model data without sharing their local training data, resulting in an aggregated model. In FLM, the organizations participating in modeling are called clients (or participants), and the party that aggregates the local models from multiple clients is called the server (or collaborator).

[0044] In federated learning, the client is the terminal device that possesses local training data. In related technologies, the client downloads the initial model from the server, trains the initial model using its own local training data, and uploads the model update parameters to the server. The server then securely aggregates the model update parameters sent by each client and distributes the aggregated model to each client. Because the client uploads model update parameters to the server, rather than its own local training data, there is no risk of local training data leakage, thus protecting the privacy and security of the local training data.

[0045] In federated learning, the server receives model update parameters sent by each client, performs secure aggregation of these parameters to obtain an aggregated model, and then distributes the aggregated model to each client.

[0046] Client consistency refers to the consistency of feature representations across local models on different clients in federated learning. In federated learning, differences in data distribution may exist between different clients. These differences arise because variations in the models and parameters of terminal equipment used by different hospitals can lead to variations in the acquired medical images, resulting in inconsistencies in feature representations across different clients. For example, using terminal equipment from different hospitals to photograph the same area of ​​the same patient can produce images with varying brightness. During the training of local models, these inconsistencies in feature representations across clients will lead to different optimization directions, rather than focusing on the global objective, thus reducing the predictive power of the aggregated model. Therefore, to improve the predictive power of the aggregated model, it is necessary to enhance the consistency of feature representations across different clients, guiding the local models on different clients to learn more shared features.

[0047] Feature redundancy: While guiding local models across different clients to learn more shared features, the increasing number of learned shared features can lead to feature redundancy. Feature redundancy refers to different feature channels capturing the same representation for the output of network layers in the local model. Therefore, there is a trade-off between improving the consistency of feature representations across clients and avoiding feature redundancy.

[0048] Latent space: A multidimensional vector space used to represent the latent features or attributes of data. Vectors in the latent space represent the underlying features or attributes of the data, but these features are often difficult to interpret directly. The core idea of ​​latent space is to map high-dimensional raw data to a low-dimensional latent space for better understanding and processing of the data. This mapping process can be achieved through linear layers, which compress the input data of the first dimension into data of the second dimension (i.e., latent space features). The first dimension is greater than the second dimension; that is, the linear layer compresses high-dimensional data into low-dimensional data.

[0049] In federated learning scenarios, differences in device models and parameters across different clients can lead to low consistency in feature representation among the local models on different clients. For example, different X-ray machines in different hospitals may have different adjustment parameters and contrast agent levels, resulting in variations in detail highlighting, brightness, lighting characteristics, artifact size, or other medical image differences in the images acquired from different hospitals. Since the local models on different clients are trained on local training data with these medical image differences, the consistency in feature representation among the different local models is low.

[0050] To improve client consistency, related technologies may employ frequency-domain-based standardization methods to decompose medical images in the frequency domain, obtain amplitude and spectrum, and then average the amplitude spectrum to reduce the impact of image differences.

[0051] See Figure 1 , Figure 1 This is a schematic diagram illustrating how frequency-domain-based standardization methods improve client consistency in related technologies.

[0052] like Figure 1 As shown, medical images acquired by different clients in related technologies have image differences, such as the image brightness of medical image 110-1 being lower than that of medical image 120-1.

[0053] In related technologies, the medical image is first decomposed in the frequency domain to obtain the amplitude spectrum and the frequency spectrum. For example, the medical image 110-1 is decomposed in the frequency domain to obtain the frequency spectrum 111 and the amplitude spectrum 112; and the medical image 120-1 is decomposed in the frequency domain to obtain the frequency spectrum 121 and the amplitude spectrum 122.

[0054] Since image differences are more pronounced in the amplitude spectrum, the relevant technique uses the average amplitude spectrum of all medical images instead of the amplitude spectrum of each sample. This involves standardizing each amplitude spectrum to obtain an average amplitude spectrum 130, and then using an inverse Fourier transform to map the average amplitude spectrum and the spectrum of each image back to the original feature space. For example, the average amplitude spectrum 130 and spectrum 111 are mapped to a new medical image 110-2, and the average amplitude spectrum 130 and spectrum 121 are mapped to a new medical image 120-2. Comparing the image differences between the new medical images 110-2 and 120-2, and between the original medical images 110-1 and 120-1, it can be seen that the relevant technique can reduce the impact of image differences, thereby improving client consistency.

[0055] While the above approach, through frequency domain decomposition and the introduction of average amplitude spectrum, can reduce the differences between local data from different clients, it may also remove representative features from each client during standardization, thus reducing the diversity of training data. This method performs well when standardization effectively adjusts image differences, but for images that still exhibit representational differences from the training data after standardization, it can actually reduce the model's generalization performance.

[0056] Therefore, this application provides a federated learning method that can reduce the impact of differences in data distribution among different clients, guide local models of different clients to learn more common features, improve the consistency of feature representations among different clients, and also enhance the predictive ability of the aggregation model.

[0057] It should be noted that this specification uses a medical imaging scenario as an example to introduce the federated learning method, but this does not constitute any limitation on the application scenarios of the federated learning method. The federated learning method proposed in this application can be used in medical image classification or segmentation tasks such as clinical staging of diabetic retinopathy, prostate location segmentation in MRI images, and breast cancer classification in pathological images. In addition, the federated learning method proposed in this application can also be applied to other medical imaging tasks or any other possible scenario types, such as in scenarios where different media or advertisers jointly learn advertising placement, or in scenarios where different application platforms jointly perform image detection. This application does not impose any limitations on the application scenarios of federated learning.

[0058] See Figure 2 , Figure 2 This is a schematic diagram of a computer system in a federated learning scenario provided by an exemplary embodiment of this application. The computer system includes a client 110 and a server 120. The client 110 includes clients 110-1, 110-2, and 110-3 from different hospitals. The three clients are only illustrative; in practice, the number of clients in federated learning can be any number, including two or more, without limitation.

[0059] Client 110 is a terminal device with a local training sample set. Client 110 can be a desktop computer, tablet computer, in-vehicle terminal, projection computer, smartphone, or other terminal device. In some embodiments, client 110 is a terminal device connected to medical imaging equipment, including but not limited to terminal devices connected to medical imaging equipment such as computed tomography (CT) equipment, magnetic resonance imaging (MR) equipment, X-ray imaging equipment, nuclear medicine diagnostic equipment, and ultrasound equipment (US). Client 110 can be used to acquire the locally acquired training sample set from the medical imaging equipment. In different federated learning scenarios, the client can be of different types of terminal devices; there is no limitation on this.

[0060] In some embodiments, medical images acquired by different clients may have differences in image quality. For example... Figure 1 In the process, the medical image 111 obtained by client 110-1 differs from the medical image 112 obtained by client 110-2 and the medical image 113 obtained by client 110-3. For example, the brightness of the diabetic retinopathy image in medical image 112 is higher than that in medical images 111 and 113.

[0061] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0062] In some embodiments, client 110 and server 120 can communicate via a wired network and / or a wireless network. In some embodiments, the clients do not communicate with each other.

[0063] In some embodiments, during the local training phase of federated learning, client 110 can obtain an initial model from server 120 and train the initial model locally based on the local training sample set to obtain a local model.

[0064] In some embodiments, during the consistency optimization phase proposed in this application, client 110 can map local data features to initial local latent space features through an initial linear layer and send the initial local latent space features to server 120. In some embodiments, server 120 can determine the consistency loss based on the initial local latent space features of each client and send a model update instruction to the client based on the consistency loss, so that the client updates the parameters of the local model and the initial linear layer based on the model update instruction, or updates the parameters of the initial linear layer based on the model update instruction. Further details regarding the consistency optimization phase are provided in the following embodiments and will not be elaborated upon here.

[0065] In some embodiments, during the model aggregation phase of federated learning, client 110 can upload its local model to server 120, and server 120 can aggregate the local models of each client to obtain an aggregated model.

[0066] In some embodiments, during the model aggregation phase of federated learning, client 110 may also map local data features to local latent space features through a linear layer and send the local latent space features to server 120. Server 120 determines the aggregation weights corresponding to each client based on the local latent space features of each client, and performs weighted aggregation on the local models of different clients based on the aggregation weights to obtain an aggregated model. Further details regarding weighted aggregation can be found in the embodiments described later, and will not be elaborated upon here.

[0067] In some embodiments, the server 120 can distribute the aggregated model to each client, and each client can make predictions for local medical imaging tasks based on the aggregated model.

[0068] It should be noted that the above embodiments only illustrate the general architecture of a computer system in a federated learning scenario. The system may also include more or fewer components, or combine certain components. This embodiment does not constitute a limitation in this regard.

[0069] See Figure 3 , Figure 3 This is a flowchart of a federated learning method provided in an exemplary embodiment of this application. In some embodiments, the method is executed by a client in the federated learning process. The method includes the following steps.

[0070] Step 301: Obtain the client's local model, which is trained based on the local training sample set.

[0071] In some embodiments, the local training sample set is a training sample set collected locally by the client. In some embodiments, the local training sample set is obtained by the client from a local medical imaging device. For example, the local training sample set of client 110-1 corresponding to hospital A is MRI medical images collected by the MRI device in hospital A, while the local training sample set of client 110-2 corresponding to hospital B is MRI medical images collected by the MRI device in hospital B.

[0072] Optionally, the local training sample set includes local training samples and corresponding sample result labels. In some embodiments, the local training samples are medical images, and the sample result labels are the task labels corresponding to the medical images (e.g., labels determined based on actual doctor diagnoses). For example only, the local training samples in the local training sample set are 1000 breast tissue images, and the sample result label for each breast tissue image can be 0 (indicating negative breast cancer) or 1 (indicating positive breast cancer). For example only, the local training samples are 5000 prostate MRI images, and the sample result label for each prostate MRI image can be a bounding box of the prostate in the MRI image. It should be noted that the specific form of the local training samples and sample result labels in the local training sample set can be determined by those skilled in the art according to the actual needs of the model prediction task, and there are no limitations on this.

[0073] In some embodiments, the client can train its local model based on a local training sample set in various possible ways. For example, the parameters of the local model can be updated using backpropagation and stochastic gradient descent.

[0074] A local model is a model trained by the client using a local training sample set.

[0075] Optionally, the local model includes a feature extraction network and a task prediction network. The feature extraction network extracts training sample data features from the local training sample set, while the task prediction network determines the prediction result based on these features. The client can determine the task loss based on the difference between the predicted result and the label, and update the parameters of the local model accordingly.

[0076] Optionally, the local model may include network layers of different levels. Optionally, the features extracted from the training samples by different network layers may have different feature sizes.

[0077] Step 302: Extract local data features from the local training samples in the local training sample set using the client's local model.

[0078] Local data features are used to reflect the data characteristics of local training samples in the client's local training sample set.

[0079] Regarding the method for determining local data features, in some embodiments, the client extracts the training sample data features of each local training sample in the local training sample set through the trained local model, and determines the local data features based on the training sample data features of each local training sample.

[0080] For illustrative purposes only, client c's local training sample set contains 1000 breast tissue images. The client uses the trained local model to extract the features of the training sample data corresponding to each of the 1000 breast tissue images, denoted as f. c1 f c2 ...f c1000 Next, the client will f c1 f c2 ...f c1000 The average value is determined as the local data feature of the local training samples in the local training sample set.

[0081] In some embodiments, since the local data features are integrated with the training sample data features of each local training sample in the local training sample set, the local data features can reflect the data features of the local training sample set corresponding to the client to a certain extent.

[0082] Step 303: Map the local data features of the client to the local latent space features of the client. The local latent space features of the client are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0083] In one possible implementation, the client can map local data features to its local latent space features using a recurrent neural network (RNN) or a deep neural network (DNN).

[0084] In another possible implementation, the client maps local data features to local latent space features through a linear layer, where the parameters of the linear layer are obtained through consistent training based on the local latent space features of different clients.

[0085] A linear layer, also known as a fully connected layer, is a type of neural network layer. Linear layers process the input vector using matrix multiplication and the addition of bias vectors to obtain the output vector. The matrix and bias vector are the learnable parameters of the linear layer.

[0086] In some embodiments, the client will use local data characteristics Inputting a linear layer yields the local latent space feature Z. c The formula is as follows.

[0087]

[0088] Among them, W c Let b be the matrix of the linear layer in client c. c Let be the bias vector of the linear layer in client c. Optionally, the matrix and bias vector of the linear layer in different clients can be the same or different.

[0089] The client's local latent space features are the feature representations of the local training samples in the latent space.

[0090] Optionally, the client's local latent space features are the latent features of the local training samples in the latent space.

[0091] Latent space features refer to the latent features of data extracted through data compression and pattern recognition techniques in the latent space. The latent space is a hypothetical, typically not directly observable, multi-dimensional space containing the latent features or attributes of the data. In machine learning and deep learning, latent space features are the latent patterns or features of a dataset learned by the model through training.

[0092] In some embodiments, the local latent space feature Z c It is a linear layer for local data features The mapping is due to the characteristics of local data. To a certain extent, it reflects the data characteristics of the local training sample set; therefore, the local latent space feature Z c It can, to some extent, reflect the feature representation of the local training sample set in the latent space (e.g., the brightness of medical images in the local training sample set).

[0093] For illustrative purposes only, the local latent space feature Z of client c1 c1 The medical images in the local training sample set representing client c1 have low brightness, and the local latent space feature Z of client c2 is... c2 The medical images in the local training sample set representing client c2 have higher brightness.

[0094] In some embodiments, clients can update the parameters of the linear layer through consistency training to improve the similarity between the local latent space features of different clients, making the similarity between the local latent space features of different clients higher than the similarity between the local data features of different clients. That is, the purpose of consistency training is to improve the consistency of the local latent space features of different clients.

[0095] In some embodiments, prior to consistency training, due to differences in data distribution among different clients (e.g., differences in medical image brightness caused by differences in equipment models and parameters of medical imaging devices), the consistency of feature representations across clients is low, resulting in different local latent space features Z from different clients. c The differences are significant. For example, the local latent space feature of client c1 is Z. c1 The local latent space feature of client c2 is Z. c2 Before consistency training, Z c1 With Z c2 The vector distance between them is relatively large.

[0096] In some embodiments, during the consistency training phase, each client performs consistency training and updates the parameters of the linear layer to improve the consistency of local latent space features among different clients.

[0097] In some embodiments, after consistency training, the differences between the local latent space features of different clients become smaller compared to before consistency training because consistency training improves the consistency of the local latent space features of different clients. For example, Z c1 With Z c2 The vector distance between them becomes smaller compared to before consistency training, meaning that the similarity between local latent space features of different clients is improved, and is higher than the similarity between local data features of different clients.

[0098] Regarding the specific method of updating the parameters of the linear layer through consistency training, in some embodiments, each client can send the local latent space features output by its respective linear layer to the server. The server determines the consistency loss based on the local latent space features of each client and sends a model update instruction to each client based on the consistency loss, instructing each client to update the parameters of the linear layer.

[0099] Regarding the specific method of updating the parameters of the linear layer through consistency training, in some embodiments, the server can determine the consistency loss based on the local latent space features of each client, calculate the model gradient of each client based on the consistency loss, and then return the model gradient to the client so that the client can update the parameters of the linear layer based on the model gradient.

[0100] Regarding the specific method of updating the parameters of the linear layer through consistency training, in some other embodiments, the server can determine the consistency loss based on the local latent space features of each client and return the consistency loss to the client, which then calculates the model gradient based on the consistency loss and updates the parameters of the linear layer based on the model gradient.

[0101] For more information on consistency loss, see [link to relevant documentation]. Figure 4 The details and related descriptions will not be repeated here.

[0102] Step 304: Send the client's local latent space features to the server in the federated learning process, so that the server can aggregate the local models of different clients based on the local latent space features of different clients to obtain an aggregated model.

[0103] In some embodiments, the client extracts local latent space features from local data features based on the linear layer obtained by consistency training, and sends the client's local latent space features and the client's local model to the server.

[0104] In one possible implementation, the server can determine the aggregation weight for each client based on the local latent space features sent by each client, and then aggregate the local models of each client using these aggregation weights to obtain an aggregated model. For example, the model parameters of the local models can be weighted and aggregated according to the aggregation weights corresponding to each client to obtain the aggregated model.

[0105] For more information on aggregate weights, see [link to relevant documentation]. Figure 10 The details and related descriptions will not be repeated here.

[0106] In another possible implementation, the server can also determine whether the aggregation requirement is met based on the local latent space features sent by different clients. If the aggregation requirement is met, the local models of each client are aggregated to obtain an aggregated model. For example, if the differences between the local latent space features of different clients are small (e.g., less than a difference threshold), it is determined that the aggregation requirement is met.

[0107] In some embodiments, the server distributes the aggregated model to each client. Each client can perform multiple iterations based on the aggregated model, that is, update the aggregated model multiple times through steps 301 to 304, until the iteration stops when the iteration stopping condition is met (e.g., the number of iterations reaches a threshold, or the accuracy of the aggregated model meets the accuracy condition), and the final aggregated model is obtained.

[0108] In some embodiments, each client can process its local model prediction task through the final aggregated model. Taking the model prediction task as a classification task of breast tissue images as an example, the client inputs the breast tissue image to be classified into the feature extraction network in the aggregated model to extract the data features of the breast tissue image; then, the data features are input into the task prediction network in the aggregated model to obtain the classification result corresponding to the breast tissue image. For example, the classification result is 0 (indicating breast cancer negative) or the classification result is 1 (indicating breast cancer positive).

[0109] In summary, the client-side model, trained on a local training sample set, extracts local data features from the local training samples, ensuring these features reflect the data characteristics of the client's local training sample set. These local data features are then mapped to local latent space features, allowing them to reflect the feature representation of the local training sample set in the latent space (e.g., the brightness of medical images within the local training sample set). Since the similarity between local latent space features across different clients is higher than the similarity between local data features across different clients, the consistency of local latent space feature representations across clients is improved compared to the consistency between local data features across clients. By improving the consistency of local latent space feature representations across clients, different clients can learn more shared features.

[0110] Each client sends its local latent space features to the server, which then aggregates the local models based on these features. This reduces the impact of data distribution differences between clients. Since each client learns more common features, it focuses more on the global objective rather than the local optimization objective during training, avoiding the problem of local update bias. This results in better aggregation performance and improved accuracy of the aggregated model.

[0111] Meanwhile, since the aggregation model is obtained by each client learning more common features, when the aggregation model is applied to different clients, even if the data distribution between the clients is very different, the aggregation model can achieve relatively stable performance in each client, thus improving the fairness of the aggregation model to each client.

[0112] Regarding the specific methods of consistency training, in one possible implementation, the client maps its local data features to its local latent space features using a linear layer. The parameters of the linear layer are obtained based on consistency training. In some embodiments, the client can update the parameters of the linear layer based on the consistency loss.

[0113] In some embodiments, the client obtains an initial linear layer.

[0114] Optionally, the initial parameters of the initial linear layer can be configured and determined by the client, or the client can download the initial linear layer from the server. Optionally, the initial parameters of different initial linear layers can be the same or different.

[0115] In some embodiments, the client maps local data features to the client's initial local latent space features through an initial linear layer.

[0116] Among them, the local data features are extracted by the local model from the local training samples in the local training sample set, and the initial local latent space features are obtained by mapping the local data features to the initial linear layer.

[0117] See Figure 4 , Figure 4 This is a schematic diagram of consistency training provided by an exemplary embodiment of this application.

[0118] like Figure 4 As shown, during the consistency training process, the client 401 extracts the local data features of the local training samples in the local training sample set through the trained local model 410.

[0119] In some embodiments, the client extracts the training sample data features of each local training sample in the local training sample set through the client's local model, and determines the average value of the element values ​​in the training sample data features of each local training sample as the element value in the local data features.

[0120] For illustrative purposes only, the local training sample set of client 401 (client c) consists of 1000 breast tissue images. Client 401 trains a local model using this local training sample set. After each epoch (i.e., after all local training samples in the training sample set have been used), client 401 uses the trained local model to extract the features of the training sample data corresponding to each of the 1000 breast tissue images, denoted as f. c1 f c2 ...f c1000 Next, the client will use 401 to send f c1 f c2 ...f c1000 The average value is determined as the local data feature of the local training samples in the local training sample set.

[0121] In some embodiments, client 401 uses the initial linear layer 420 to feature local data. Mapped to the client's initial local latent space feature Z c0 .

[0122]

[0123] Among them, W c0 b is the initial matrix of the initial linear layer in client 401. c0 This is the initial bias vector for the initial linear layer in client 401.

[0124] In some embodiments, the client sends its initial local latent space characteristics to the server.

[0125] For example, client 401 sends its initial latent space feature Z to server 402. c0 Meanwhile, other clients in the federated learning process also send their initial latent space features to server 402.

[0126] In some embodiments, the server receives initial local latent space characteristics from different clients.

[0127] The initial local latent space features of the client are obtained by mapping the initial local data features to the initial linear layer, and the initial local data features are extracted from the local training samples in the local training sample set by the client's local model.

[0128] In some embodiments, the server determines a consistency loss based on the initial local latent space features of different clients. The consistency loss characterizes the differences between the initial local latent space features of different clients.

[0129] Optionally, those skilled in the art can determine the specific form of consistency loss according to actual needs, for example, by using the Frobenius norm (F-norm), L1 distance, L2 distance or any other possible form of loss to determine the consistency loss.

[0130] As an example only, to improve Cross-Client Consistency (CCC), the consistency loss L determined by the F-norm is used. ccc It can be represented by the following formula.

[0131]

[0132] Where C represents the number of clients in federated learning, p and q represent any two clients, and z p z represents the initial local latent space features of client p. q H represents the initial local latent space feature of client q, H represents the height dimension of the initial local latent space feature, and W represents the width dimension of the initial local latent space feature.

[0133] In some embodiments, the server sends model update instructions to different clients based on consistency loss. The clients are used to update their local model and the parameters of the initial linear layer based on the model update instructions, or to update the parameters of the initial linear layer.

[0134] In some embodiments, the client receives a model update instruction sent by the server based on consistency loss, and updates the client's local model and the parameters of the initial linear layer based on the model update instruction, or updates the parameters of the initial linear layer based on the model update instruction.

[0135] Optionally, the model update instruction may include the model gradient calculated by the server based on the consistency loss. The client updates its local model and the parameters of the initial linear layer according to the model gradient included in the model update instruction, or updates the parameters of the initial linear layer.

[0136] Optionally, the model update instruction may include a consistency loss. The client calculates the model gradient based on the consistency loss included in the model update instruction, and updates the client's local model and the parameters of the initial linear layer based on the model gradient, or updates the parameters of the initial linear layer.

[0137] In one possible implementation, each client can update only the parameters of the initial linear layer according to the model update instruction. For example, client 401 updates only the parameters of the initial linear layer 420 according to the model update instruction to obtain the trained linear layer.

[0138] In another possible implementation, each client can simultaneously update the parameters of the initial linear layer and the local model according to the model update instruction. For example, client 401 updates the parameters of the initial linear layer 420 and the local model 410 simultaneously according to the model update instruction, thus obtaining the trained linear layer and the updated local model.

[0139] In this embodiment, the client maps local data features to initial local latent space features through an initial linear layer and sends these initial local latent space features to the server. The server determines a consistency loss based on the initial local latent space features sent by each client and sends a model update instruction to the client based on the consistency loss. This allows the client to update its local model and the parameters of the initial linear layer, or, alternatively, update the parameters of the initial linear layer based on the model update instruction. Since the consistency loss characterizes the differences between the initial local latent space features of each client, after consistency training, the differences between the local latent space features of each client are smaller than before consistency training. This improves the consistency of the local latent space feature representations of each client, enabling each client to learn more common features and achieving consistency training.

[0140] In some embodiments, the local model includes network layers of different levels.

[0141] Optionally, different levels of network layers include network layers at different stages, where the feature dimensions output by network layers at different stages are different (e.g., H and W are different).

[0142] For example, the local model contains a network layer with 4 stages.

[0143] See Figure 5 , Figure 5 This is a schematic diagram of a local model provided in an exemplary embodiment of this application, which includes network layers of different levels.

[0144] like Figure 5 As shown, the local models of each client (including client A, client B, and client C) in federated learning each contain four levels of network layers.

[0145] In some embodiments, the client extracts local data features corresponding to different levels of local training samples in the local training sample set through different levels of network layers in the local model trained by the client.

[0146] In some embodiments, the client maps the local data features of different levels of the client to the initial local latent space features of the client at different levels through an initial linear layer. The local data features of different levels of the client are output by different levels of network layers in the client's local model.

[0147] For example, client A maps the local data features of the first, second, third, and fourth levels in the local model to the initial local latent space features of the first level through the initial linear layer in client A. The initial local latent space features of the second level The initial local latent space features of the third level and the initial local latent space features of the fourth level

[0148] Similarly, client B maps the local data features of the first, second, third, and fourth levels in the local model to the initial local latent space features of the first level through its initial linear layer. The initial local latent space features of the second level The initial local latent space features of the third level and the initial local latent space features of the fourth level

[0149] Similarly, client C maps the local data features of the first, second, third, and fourth levels in the local model to the initial local latent space features of the first level through the initial linear layer of client C. The initial local latent space features of the second level The initial local latent space features of the third level and the initial local latent space features of the fourth level

[0150] In some embodiments, the client sends the initial local latent space features of the client at different levels to the server, so that the server determines the consistency sub-loss corresponding to the level based on the initial local latent space features of different clients at the same level, and uses the sum of the consistency sub-losses corresponding to each level as the consistency loss.

[0151] For example, client A will use the initial local latent space features from the first to the fourth level. as well as Client B sends the initial local latent space features from the first to the fourth level to the server. as well as The client C sends the initial local latent space features from the first to the fourth level to the server. as well as The data is sent to the server. The server determines the consistency loss based on the initial local latent space features at the same level across different clients.

[0152] For example, the server is based on as well as Based on the differences, determine the consistency sub-loss corresponding to the first level; based on as well as Based on the differences, determine the consistency sub-loss corresponding to the second level; based on as well as Based on the differences, determine the consistency sub-loss corresponding to the third level; based on as well as The differences are used to determine the consistency sub-loss corresponding to the fourth level. Finally, the server adds up the consistency sub-losses of each level to obtain the final consistency loss.

[0153] In this embodiment, the client sends initial local latent space features at different levels to the server. The server determines the consistency loss based on the initial local latent space features at the same level from different clients. This allows the consistency loss to include information on the feature representations of different network layers in the local models of each client, thereby improving the consistency of client feature representations more precisely.

[0154] In the above embodiments, the federated learning method proposed in this application improves the consistency of feature representations across different clients through consistency training, enabling each client to learn more common features. However, this explicit approach to improving client consistency often increases the redundancy of feature representations. Feature redundancy refers to the situation where different feature channels capture the same representation for the output of the network layers in the local model.

[0155] See Figure 6 , Figure 6 This is a schematic diagram illustrating feature redundancy in the context of improving client consistency, provided by an exemplary embodiment of this application.

[0156] Figure 6 The covariance matrix corresponding to the features of the training sample data extracted by the local model from the local training samples is visualized, and the correlation between different feature channels in the training sample data features is expressed by grayscale. White represents the highest correlation, and black represents the lowest correlation. For example, points on the main diagonal of the covariance matrix represent a correlation of 1 between the same feature channels, so the main diagonal is represented by white in the visualization. Points outside the main diagonal represent the correlation between different feature channels; the closer a point is to white, the higher the correlation between feature channels, and the closer a point is to black, the lower the correlation between feature channels.

[0157] like Figure 6 As shown, comparing the baseline with the consistency training method, consistency training leads to increased correlation between different feature channels in the training sample data; that is, the relevance of the representations captured by different feature channels increases. For example, after 10 epochs of training, the covariance matrix corresponding to the consistency training method has more white points compared to the covariance matrix corresponding to the baseline method. Therefore, while consistency training improves client-side consistency, it also introduces feature redundancy, causing different feature channels to capture more identical or similar features.

[0158] Figure 6 The study also showed that as training progresses and the number of epochs increases, the correlation between the representations captured by different feature channels further increases. For example, for the covariance matrix corresponding to the consistency training method, the number of white points in the image after 100 epochs of training is greater than the number of white points in the image after 10 epochs of training.

[0159] This demonstrates a trade-off between client consistency and feature redundancy. If we focus solely on client consistency, increased feature redundancy may hinder overall performance from reaching expectations. Conversely, if we prioritize feature diversity, maintaining consistency across clients becomes more challenging.

[0160] The federated learning method proposed in the above embodiments improves the consistency of feature representations among different clients through consistency training. This guides different clients to learn more common features, causing each client to learn towards a global objective rather than local optimization, thereby improving the aggregation effect of the aggregation model. However, as the consistency of feature representations among different clients increases, feature redundancy also increases, leading to different channels capturing the same or similar features, thus reducing the diversity of feature representations and consequently reducing the model's predictive ability.

[0161] Based on this, this application further proposes a training method that combines consistency training with feature whitening. Intra-Client Whitening (ICW) is a channel decoupling method used to reduce the correlation between different feature channels. It should be noted that in this application, the purpose of intra-client whitening is not to completely eliminate the correlation between different feature channels, but rather to enable the local model to learn more about the stronger correlations between different feature channels while ignoring the weaker correlations. Therefore, the intra-client whitening proposed in this application can also be called "relaxed whitening".

[0162] like Figure 6 As shown, combining consistency training with feature whitening significantly reduces feature redundancy and improves the expressive diversity of different feature channels compared to using consistency training alone. Figure 6 The number of white points in the image corresponding to "consistent training + feature whitening" is lower than the number of white points in the image corresponding to "consistent training".

[0163] The following describes feature whitening within the client proposed in other embodiments of this application.

[0164] See Figure 7 , Figure 7 This is a schematic diagram of feature whitening within a client provided in an exemplary embodiment of this application.

[0165] In some embodiments, the client determines the sample prediction result corresponding to the local training sample based on the training sample data features extracted from the local training sample by the client's local model.

[0166] For example, client 701 inputs local training samples into local model 710, the feature extraction network in local model 710 extracts the features of training sample data, and inputs the features of training sample data into task prediction network in local model 710 to obtain sample prediction results.

[0167] In some embodiments, the client determines the task loss 721 based on the sample prediction results and the sample result labels.

[0168] Optionally, the task loss can be determined based on the sample prediction results and sample result labels, using cross-entropy loss, mean squared error loss, KL divergence loss, L1 norm loss, or any other possible form, without any restrictions.

[0169] For example, the task loss of client c can be characterized as

[0170] In some embodiments, the client may also determine the correlation loss 722 based on the feature correlation between different feature channels in the training sample data features.

[0171] Among them, correlation loss is used to characterize the difference between the correlation of the original features and the correlation of the expected features among different feature channels.

[0172] In one possible implementation, the correlation between the original features of different feature channels can be characterized by the original cross-correlation matrix, and the correlation between the expected features of different feature channels can be characterized by the expected cross-correlation matrix.

[0173] The cross-correlation matrix can be determined by calculating the cross-correlation coefficient between the i-th and j-th feature channels in different feature channels. For the specific calculation method of the cross-correlation matrix, please refer to the definition in related technologies; it will not be elaborated here.

[0174] In some embodiments, the original cross-correlation matrix is ​​calculated based on the cross-correlation coefficients between different feature channels in the training sample data features extracted by the local model.

[0175] In some embodiments, the desired cross-correlation matrix can be predetermined by the client. For example only, the desired cross-correlation matrix may have elements on the main diagonal with values ​​of 1 and elements in the remaining positions with values ​​of 0.3.

[0176] In this case, the element values ​​in the expected cross-correlation matrix are less than or equal to the element values ​​in the original cross-correlation matrix. Therefore, the correlation loss constructed by the original cross-correlation matrix and the expected cross-correlation matrix can reduce the feature correlation between different feature channels in the training sample data.

[0177] Optionally, the correlation loss can be determined based on the difference between the original cross-correlation matrix and the expected cross-correlation matrix between different feature channels.

[0178] In some embodiments, the client trains its local model 710 based on the total loss 723 of relevance loss 722 and task loss 721. The total loss L for client c... c It can be expressed as the following formula.

[0179]

[0180] in, Indicates mission loss. Let λ represent the correlation loss, and λ be the weight between the two.

[0181] In this embodiment, in addition to the task loss, a correlation loss is added, which can make the correlation between the original features of different feature channels in the training sample data features extracted by the local model gradually approach the correlation of the desired features, thereby reducing the correlation between different feature channels.

[0182] In some embodiments, it is assumed that the total number of feature channels in the training sample data features is C. i Then the client can target all C i Alternatively, the original cross-correlation matrix can be calculated for each feature channel. Or, the training sample data can be grouped according to the feature channels to obtain at least two feature groups, and the original cross-correlation matrix corresponding to each feature group can be calculated separately.

[0183] By grouping the features of the training sample data, the correlation between different feature channels can be reduced only for a single feature group, rather than reducing the correlation between different feature channels in the entire training sample data. This reduces redundant features while maintaining consistency in feature representation for the client. In other words, grouping the features of the training sample data achieves a "relaxed whitening" effect.

[0184] In some embodiments, the client can group the features of the training sample data according to the feature channels to obtain at least two feature groups.

[0185] For example only, assume that the training sample data features F c The total number of feature channels in is C i If the number of groups is G, then there are G feature groups, and each feature group contains C feature channels. i / G.

[0186] See Figure 8 , Figure 8This is a schematic diagram illustrating the grouping of training sample data features according to feature channels, provided in an exemplary embodiment of this application.

[0187] like Figure 8 As shown, the total number of feature channels contained in the training sample data feature 810 extracted by the local model from the local training samples is C. i The client groups the training sample data features 810 according to the group size G, and then groups the training sample data features F. c Divided into G feature groups

[0188] For example, feature group 811 contains C feature channels. i / G, the height dimension of feature group 811 is H i The width dimension of feature group 811 is W. i .

[0189] In some embodiments, the client determines the original cross-correlation matrix corresponding to the feature group based on the correlation between the feature channels in the feature group. The original cross-correlation matrix is ​​used to characterize the correlation between the original features of different feature channels in the feature group.

[0190] In some embodiments, the original cross-correlation matrix corresponding to the feature group can be calculated based on the cross-correlation coefficient between the i-th feature channel and the j-th feature channel in different feature channels of the feature group. For the specific calculation method of the cross-correlation matrix, please refer to the definition in related technologies, which will not be elaborated here.

[0191] For example, the G original cross-correlation matrices corresponding to the G feature groups can be represented as follows:

[0192] In some embodiments, the client determines the expected cross-correlation matrix based on the original cross-correlation matrix corresponding to each feature group.

[0193] The expected cross-correlation matrix is ​​used to characterize the correlation between expected features among different feature channels in a feature group.

[0194] Regarding the specific method for determining the expected cross-correlation matrix, in some embodiments, the client determines the element values ​​in the average cross-correlation matrix by averaging the element values ​​in the original cross-correlation matrices corresponding to each feature group. The average cross-correlation matrix is ​​expressed by the following formula.

[0195]

[0196] Where, Σ c,k This represents the original cross-correlation matrix corresponding to the k-th feature group.

[0197] In some embodiments, the client determines the expected cross-correlation matrix corresponding to the feature group based on the difference between the element values ​​at the same position in the original cross-correlation matrix and the average cross-correlation matrix corresponding to the feature group.

[0198] The difference between the element values ​​at the same position in the original cross-correlation matrix and the average cross-correlation matrix corresponding to the feature group can be obtained through the difference matrix v. c,g The formula is as follows.

[0199]

[0200] Where, Σ c,g This represents the original cross-correlation matrix corresponding to the g-th feature group. This is the average cross-correlation matrix.

[0201] In some embodiments, when the difference is greater than the difference threshold, or when the element value is located on the main diagonal, it is expected that the element value at the corresponding position in the cross-correlation matrix is ​​1.

[0202] In some embodiments, when the difference is less than or equal to the difference threshold and the element value is not located on the main diagonal, the element value at the corresponding position in the cross-correlation matrix is ​​expected to be 0.

[0203] Optionally, the difference threshold is 0.

[0204] That is, the expected cross-correlation matrix It can be expressed by the following formula.

[0205]

[0206] in, Represents the difference matrix v c,g The value of the element in the i-th row and j-th column.

[0207] Figure 8 The original cross-correlation matrix ∑ of the feature groups is displayed visually. c,g Cross-correlation matrix with expectation The relationship between them.

[0208] like Figure 8 It can be seen that the expected cross-correlation matrix The original cross-correlation matrix Σ was extracted. c,g The data represents the typical correlations, while ignoring the original cross-correlation matrix ∑. c,g The correlation is not representative.

[0209] In some embodiments, the client determines the correlation loss of a feature group based on the original cross-correlation matrix and the expected cross-correlation matrix corresponding to the feature group.

[0210] In some embodiments, the client determines the correlation loss based on the sum of the correlation losses of each feature group.

[0211] For example, the correlation loss of client c It can be expressed as the following formula.

[0212]

[0213] Where, Σ c,g Let ψ represent the original cross-correlation matrix corresponding to the g-th feature group. c,g Let represent the expected cross-correlation matrix corresponding to the g-th feature group.

[0214] In some embodiments, the local model includes network layers of different levels.

[0215] Regarding the determination of correlation loss, in some embodiments, the client can determine the correlation loss at different levels based on the feature correlation between different feature channels in the training sample data features at different levels.

[0216] The features of the training sample data at different levels are output by the network layers at different levels in the local model.

[0217] For example, in a local model containing four network layers, the client can calculate the original cross-correlation matrix and the expected cross-correlation matrix between different feature channels of the training sample data for the first to fourth layers, respectively. Then, based on the difference between the original cross-correlation matrix and the expected cross-correlation matrix for the first to fourth layers, the client can determine the correlation loss corresponding to each of the first to fourth layers. as well as

[0218] In some embodiments, the client trains its local model based on the total loss of relevance loss and task loss at different levels.

[0219] As an example only, the client can use the correlation loss corresponding to the fourth level. and mission losses Update the parameters of the fourth layer of the local model and the correlation loss based on the third layer. and mission losses Update the parameters of the third-level network layer in the local model... and so on.

[0220] By training the local model with the total loss of correlation loss and task loss at different levels, the local model can be optimized more precisely for different levels, thereby improving the training effect of the local model.

[0221] In this embodiment, the training sample data features are grouped according to feature channels to obtain at least two feature groups. The difference between the original cross-correlation matrix and the expected cross-correlation matrix is ​​calculated for a single feature group. This can reduce the correlation between different feature channels in a single feature group, rather than reducing the correlation between all feature channels in the entire training sample data. Therefore, the degree of reduction of redundant features can be weakened, thereby ensuring the improvement of client consistency to a certain extent. This plays the role of "relaxed whitening" and achieves a balance between client consistency and feature whitening.

[0222] Regarding the specific method of grouping training sample data features according to feature channels, in some embodiments, the client can also determine the number of groups based on the number of different clients in federated learning.

[0223] The number of groups G is positively correlated with the number of clients C.

[0224] In some embodiments, the client groups the features of the training sample data according to the feature channels and the number of groups, thus obtaining the feature groups of the specified number of groups.

[0225] For illustrative purposes only, the number of groups G equals the number of clients C. For example, in federated learning with 5 clients, within each client, the training sample data features are divided into 5 groups according to feature channels.

[0226] The purpose of setting the number of groups G to be positively correlated with the number of clients C is that when the number of clients is large, it becomes more difficult to maintain the consistency of feature representations among different clients. Therefore, increasing the number of groups accordingly can further weaken the reduction of redundant features, thereby better ensuring that client consistency is improved.

[0227] See Figure 9 , Figure 9 This is a flowchart of a federated learning method provided in another embodiment of this application. In some embodiments, the method is executed by a server in the federated learning process. The method includes the following steps.

[0228] Step 901: Receive the local latent space features of different clients in federated learning. The local latent space features of the clients are obtained by mapping the local data features of the clients. The local data features of the clients are extracted from the local training samples in the local training sample set through the local model of the clients. The local latent space features of the clients are the feature representation of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0229] Step 901 is executed by the server, and Figure 3Steps 301 to 303, where the client acts as the execution subject, are essentially identical. Therefore, for details regarding step 901, please refer to [link / reference needed]. Figure 3 The corresponding content will not be repeated here.

[0230] Step 902: Based on the local latent space features of different clients, aggregate the local models of different clients to obtain an aggregated model; the local models of the clients are trained based on the local training sample set.

[0231] Regarding the specific method by which the server aggregates local models from different clients based on their local latent space features, one possible implementation is that the server can determine whether the aggregation requirements are met based on the local latent space features sent by each client. If the aggregation requirements are met, the server aggregates the local models from each client to obtain an aggregated model. For example, if the differences between the local latent space features of different clients are small (e.g., less than a difference threshold), the server determines that the aggregation requirements are met.

[0232] Regarding the specific method by which the server aggregates the local models of different clients based on the local latent space features of different clients, in another possible implementation, the server can determine the aggregation weights corresponding to the local models of different clients based on the local latent space features of different clients, and then aggregate the local models of different clients by weighting based on the aggregation weights to obtain the aggregated model.

[0233] Regarding the specific method by which the server determines the aggregate weights corresponding to the local models of different clients, in some embodiments, the server determines the average value of the local latent space features of different clients as the global latent space features.

[0234] In some embodiments, the local model includes network layers of different levels, and the server determines the global latent space feature corresponding to that level by averaging the local latent space features of different clients at the same level.

[0235] This can be expressed by the following formula.

[0236]

[0237] in, This represents the global latent space features corresponding to the i-th level. This represents the local latent space feature of the i-th level of the c-th client.

[0238] In some embodiments, the server determines the aggregation weights corresponding to the local models of different clients based on the similarity between the local latent space features and the global latent space features of different clients.

[0239] Among them, the aggregation weight is positively correlated with the similarity.

[0240] In some embodiments, the server determines the aggregate weights corresponding to the network layer at the same level in the local models of different clients based on the local latent space features of different clients at the same level.

[0241] Among them, the local latent space features of different clients at the same level are output by the network layer and linear layer at the same level in the local model of different clients.

[0242] Optionally, cosine similarity can be used to characterize the similarity between the local latent space features and the global latent space features of each client. The formula is as follows.

[0243]

[0244] in, Let represent the cosine similarity between the local latent space feature of the i-th level of the c-th client and the corresponding global latent space feature of the i-th level.

[0245] Optionally, the server can normalize the cosine similarity for each client and use the normalized result as the aggregate weight for the client's local model. The formula for the normalized aggregate weight is as follows.

[0246]

[0247] in, This represents the aggregate weight corresponding to the local model of the c-th client.

[0248] In some embodiments, the server weights and aggregates the local models of different clients based on aggregation weights to obtain an aggregated model.

[0249] In some embodiments, the server weights and aggregates the network layers of the same level in the local models of different clients according to the aggregation weights corresponding to the same level of network layers in the local models of different clients, thereby obtaining an aggregated model containing network layers of different levels. The formula is as follows.

[0250]

[0251] in, This is used to aggregate the model parameters of the i-th layer of the model in the e-th epoch of training. These are the model parameters for the c-th client at the i-th layer in the e-th epoch of training.

[0252] See Figure 10 , Figure 10 This is a schematic diagram of an exemplary embodiment of the present application, showing how a server aggregates the local models of each client based on aggregation weights to obtain an aggregated model.

[0253] like Figure 10 As shown, different clients extract local data features from local training samples in their local training sample sets using local models, and map these local data features to local latent space features using linear layers. Each client sends its local latent space features and local model to server 1010. Server 1010 weights and aggregates the network layers of the same level in each client's local model according to the aggregation weights corresponding to the network layers of the same level in the local models of different clients, resulting in an aggregated model containing network layers of different levels.

[0254] For example, the server according to as well as The first-level network layer of the local models of clients A, B, and C is weighted and aggregated to obtain the first-level network layer of the aggregated model; according to as well as The second-level network layer of the local models of clients A, B, and C is weighted and aggregated to obtain the second-level network layer of the aggregated model; according to as well as The third-level network layer of the local models of clients A, B, and C is weighted and aggregated to obtain the third-level network layer of the aggregated model; according to as well as The fourth-level network layer of the local models of clients A, B, and C is weighted and aggregated to obtain the fourth-level network layer of the aggregated model.

[0255] In this embodiment, local data features are mapped to local latent space features through a linear layer. Since the linear layer is trained based on the consistency of local latent space features of different clients, after consistency training, the consistency of local latent space features among different clients will be higher than the consistency of local data features among different clients. Therefore, the server determines the aggregation weight of the client based on the similarity between the local latent space features and the global latent space features among different clients. Compared with determining the aggregation weight based on local data features, this can increase the weight of clients with more obvious domain offsets during the aggregation process, thereby enhancing the fairness of the aggregation model.

[0256] The design and verification of this application's embodiments utilize three tasks: clinical staging of diabetic retinopathy (a multi-class task), prostate location segmentation in MRI images (an image segmentation task), and breast cancer classification in pathological images (a binary classification task). The clinical staging task for diabetic retinopathy employs six publicly available datasets: DeepDR, Messidor, IDRID, APTOS, FGADR, and RLDR. The prostate location segmentation task uses a multi-center dataset containing data from six centers. The breast cancer classification task in pathological images uses the Camelyon17 dataset, whose data was collected from five hospitals.

[0257] Table 1 below is a comparison table of the effects of the embodiments of this application and related technical solutions in the clinical staging task of diabetic retinopathy; Table 2 below is a comparison table of the effects of the embodiments of this application and related technical solutions in the prostate location segmentation task in MRI images; and Table 3 below is a comparison table of the effects of the embodiments of this application and related technical solutions in the breast cancer classification task in pathological images.

[0258] In the table, A represents the accuracy of the aggregate model applied to the client, and V represents the variance of the accuracy of each client. The smaller V is, the higher the fairness of the aggregate model.

[0259] As shown in Tables 1-3, the aggregation model trained using the method of this application embodiment has significantly higher accuracy and fairness than related technologies.

[0260] Table 1

[0261]

[0262] Table 2

[0263]

[0264]

[0265] Table 3

[0266]

[0267] The federated learning method proposed in this application has a wide range of applications, including but not limited to the following.

[0268] (1) Medical task prediction scenario.

[0269] Medical task prediction scenarios can include medical image classification tasks and medical image segmentation tasks.

[0270] For example, the federated learning method proposed in this application can be used in scenarios such as clinical staging of diabetic retinopathy (a multi-class classification task) and classification of breast cancer in pathological images (a binary classification task). Terminal devices from different hospitals jointly learn an aggregated model for medical image classification tasks. The trained aggregated model can be used to predict the classification results of medical images.

[0271] For example, in the scenario of prostate location segmentation in MRI images, the federated learning method proposed in this application can be used, where terminal devices from different hospitals jointly learn an aggregated model for image segmentation tasks in medical images. The trained aggregated model can be used to predict the segmentation boxes in medical images (e.g., to segment the location of the prostate in medical images).

[0272] (2) Non-medical imaging scenarios.

[0273] For example, in scenarios where different media or advertisers jointly learn about ad placement, the terminal devices of different media or advertisers can use the federated learning method proposed in this application to jointly learn an aggregated model for determining ad placement content.

[0274] For example, in scenarios where different application platforms jointly perform image detection, the different application platforms can use the federated learning method proposed in this application to jointly learn an aggregated model for image detection.

[0275] For example, in scenarios where different insurance companies jointly conduct risk assessments, the different insurance companies can use the federated learning method proposed in this application to jointly learn an aggregated model for risk assessment, which can generate local risk assessment results based on the relevant information of each insurance company's policyholders.

[0276] It should be noted that the above application scenarios are merely examples, and the federated learning method proposed in this application can also be applied to any other possible scenarios without limitation.

[0277] Medical task prediction scenarios

[0278] In one optional example, the federated learning method provided in the embodiments of this application is used as an example in the medical image prediction scenario.

[0279] In some embodiments, the method is executed by a client in federated learning.

[0280] In some embodiments, a local medical task prediction model for the client is obtained, which is trained based on a local medical image sample set.

[0281] Optionally, the local medical task prediction model is used to predict at least one of the following: the lesion staging result corresponding to diabetic retinopathy medical images, the breast cancer identification result corresponding to breast tissue medical images, and the prostate location segmentation box corresponding to MRI images.

[0282] Optionally, in the clinical staging task (multi-class task) of diabetic retinopathy, the local medical task prediction model is used to predict the disease staging results (e.g., stage I, stage II, stage III, etc.) corresponding to diabetic retinopathy medical images. The local medical image sample set includes local medical image samples and corresponding sample result labels. The local medical image samples are locally collected diabetic retinopathy medical images, and the sample result labels are the true values ​​of the disease stage determined based on the actual diagnosis results of doctors.

[0283] Optionally, in the pathological image breast cancer classification task (binary classification task) scenario, the local medical task prediction model is used to predict the breast cancer identification result corresponding to the breast tissue medical image, such as 0 (indicating breast cancer negative) or 1 (indicating breast cancer positive). The local medical image sample set includes local medical image samples and corresponding sample result labels. The local medical image samples are locally collected breast tissue medical images, and the sample result labels are the ground truth values ​​of the breast cancer identification result determined based on the doctor's actual diagnosis.

[0284] Optionally, in the prostate location segmentation task scenario within MRI images, a local medical task prediction model is used to predict the corresponding prostate location bounding box in the MRI image, such as the matrix corresponding to the prostate's position in the MRI image. The local medical image sample set includes local medical image samples and corresponding sample result labels. The local medical image samples are locally collected MRI images, and the sample result labels are the ground truth prostate location bounding boxes determined based on the actual segmentation results from the physician.

[0285] In some embodiments, the client's local medical image data features are extracted from local medical image samples in the local medical image sample set using the client's local medical task prediction model.

[0286] Local medical image data features are used to reflect the data characteristics of local medical image samples in the client's local medical image sample set.

[0287] Regarding the method for determining the features of local medical image data, in some embodiments, the client extracts the training sample data features of each local medical image sample in the local medical image sample set through the trained local medical task prediction model, and determines the local medical image data features based on the training sample data features of each local medical image sample.

[0288] For illustrative purposes only, client c's local medical image sample set contains 1000 breast tissue images. The client uses a trained local medical task prediction model to extract the training sample data features corresponding to each of the 1000 breast tissue images, denoted as f. c1 f c2 ...f c1000 Next, the client will f c1 f c2 ...f c1000 The average value is determined as the local medical image data feature of the local medical image samples in the local medical image sample set.

[0289] In some embodiments, since the local medical image data features are fused with the training sample data features of each local medical image sample in the local medical image sample set, the local medical image data features can reflect the data features of the local medical image sample set corresponding to the client to a certain extent.

[0290] In some embodiments, the local medical image data features of the client are mapped to the local latent space features of the client. The local latent space features of the client are the feature representations of local medical image samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local medical image data features of different clients.

[0291] In one possible implementation, the client can map local medical image data features to the client's local latent space features using a recurrent neural network (RNN) or a deep neural network (DNN).

[0292] In another possible implementation, the client maps the client's local medical image data features to the client's local latent space features through a linear layer, wherein the parameters of the linear layer are obtained based on the consistency training of the local latent space features of different clients.

[0293] A linear layer, also known as a fully connected layer, is a type of neural network layer. Linear layers process the input vector using matrix multiplication and the addition of bias vectors to obtain the output vector. The matrix and bias vector are the learnable parameters of the linear layer.

[0294] In some embodiments, the client will use local medical image data features Inputting a linear layer yields the local latent space feature Z. c The formula is as follows.

[0295]

[0296] Among them, W c Let b be the matrix of the linear layer in client c. c Let be the bias vector of the linear layer in client c. Optionally, the matrix and bias vector of the linear layer in different clients can be the same or different.

[0297] The client's local latent space features are the feature representations of local medical image samples in the latent space.

[0298] Optionally, the client's local latent space features are the latent features of local medical image samples in the latent space.

[0299] Latent space features refer to the latent features of data extracted through data compression and pattern recognition techniques in the latent space. The latent space is a hypothetical, typically not directly observable, multi-dimensional space containing the latent features or attributes of the data. In machine learning and deep learning, latent space features are the latent patterns or features of a dataset learned by the model through training.

[0300] In some embodiments, the local latent space feature Z c It is a linear layer for local medical image data features The mapping is due to the characteristics of local medical image data. To a certain extent, it reflects the data characteristics of the local medical image sample set; therefore, the local latent space feature Z... c It can, to some extent, reflect the feature representation of the local medical image sample set in the latent space (e.g., the brightness of medical images in the local medical image sample set).

[0301] For illustrative purposes only, the local latent space feature Z of client c1 c1 The local medical image samples in the local medical image sample set representing client c1 have low brightness, and the local latent space feature Z of client c2 is... c2 The local medical image samples in the local medical image sample set representing client c2 have higher brightness.

[0302] In some embodiments, clients can update the parameters of the linear layer through consistency training to improve the similarity between local latent space features of different clients, making the similarity between local latent space features of different clients higher than the similarity between local medical image data features of different clients. That is, the purpose of consistency training is to improve the consistency of local latent space features of different clients.

[0303] In some embodiments, prior to consistency training, due to differences in data distribution among different clients (e.g., differences in medical image brightness caused by differences in equipment models and parameters of medical imaging devices), the consistency of feature representations across clients is low, resulting in different local latent space features Z from different clients. c The differences are significant. For example, the local latent space feature of client c1 is Z. c1 The local latent space feature of client c2 is Z. c2 Before consistency training, Z c1 With Z c2 The vector distance between them is relatively large.

[0304] In some embodiments, during the consistency training phase, each client performs consistency training and updates the parameters of the linear layer to improve the consistency of local latent space features among different clients.

[0305] In some embodiments, after consistency training, the differences between the local latent space features of different clients become smaller compared to before consistency training because consistency training improves the consistency of the local latent space features of different clients. For example, Z c1 With Z c2 The vector distance between them becomes smaller compared to before consistency training, meaning that the similarity between local latent space features of different clients is improved, which is higher than the similarity between local medical image data features of different clients.

[0306] Regarding the specific method of updating the parameters of the linear layer through consistency training, in some embodiments, each client can send the local latent space features output by its respective linear layer to the server. The server determines the consistency loss based on the local latent space features of each client and sends a model update instruction to each client based on the consistency loss, instructing each client to update the parameters of the linear layer.

[0307] Regarding the specific method of updating the parameters of the linear layer through consistency training, in some embodiments, the server can determine the consistency loss based on the local latent space features of each client, calculate the model gradient of each client based on the consistency loss, and then return the model gradient to the client so that the client can update the parameters of the linear layer based on the model gradient.

[0308] Regarding the specific method of updating the parameters of the linear layer through consistency training, in some other embodiments, the server can determine the consistency loss based on the local latent space features of each client and return the consistency loss to the client, which then calculates the model gradient based on the consistency loss and updates the parameters of the linear layer based on the model gradient.

[0309] For more information on consistency loss, see [link to relevant documentation]. Figure 4The details and related descriptions will not be repeated here.

[0310] In some embodiments, the local latent space features of the client are sent to the server in the federated learning, so that the server can aggregate the local medical task prediction models of different clients based on the local latent space features of different clients to obtain an aggregated medical image prediction model.

[0311] In some embodiments, the client extracts local latent space features from local medical image data features based on the linear layer obtained by consistency training, and sends the client's local latent space features and the client's local medical task prediction model to the server.

[0312] In one possible implementation, the server can determine the aggregation weight for each client based on the local latent space features sent by each client, and then weight and aggregate the local medical task prediction models of each client based on the aggregation weights to obtain an aggregated model. For example, the model parameters of the local medical task prediction models can be weighted and aggregated according to the aggregation weights corresponding to each client to obtain the aggregated model.

[0313] For more information on aggregate weights, see [link to relevant documentation]. Figure 10 The details and related descriptions will not be repeated here.

[0314] In another possible implementation, the server can also determine whether the aggregation requirement is met based on the local latent space features sent by different clients. If the aggregation requirement is met, the local medical task prediction models of each client are aggregated to obtain an aggregated model. For example, if the differences between the local latent space features of different clients are small (e.g., less than a difference threshold), it is determined that the aggregation requirement is met.

[0315] In some embodiments, the server distributes the aggregated model to each client. Each client can perform multiple iterations based on the aggregated model, that is, update the aggregated model multiple times through steps 301 to 304, until the iteration stops when the iteration stopping condition is met (e.g., the number of iterations reaches a threshold, or the accuracy of the aggregated model meets the accuracy condition), and the final aggregated model is obtained.

[0316] In some embodiments, each client can process its local model prediction task through the final aggregated model. Taking the model prediction task as a classification task of breast tissue images as an example, the client inputs the breast tissue image to be classified into the feature extraction network in the aggregated model to extract the data features of the breast tissue image; then, the data features are input into the task prediction network in the aggregated model to obtain the classification result corresponding to the breast tissue image. For example, the classification result is 0 (indicating breast cancer negative) or the classification result is 1 (indicating breast cancer positive).

[0317] Optionally, mapping the client's local medical image data features to the client's local latent space features includes:

[0318] The local medical image data features of the client are mapped to the local latent space features of the client through a linear layer.

[0319] Optionally, the method further includes:

[0320] The client's local medical image data features are mapped to the client's initial local latent space features through an initial linear layer;

[0321] The initial local latent space features of the client are sent to the server so that the server determines a consistency loss based on the initial local latent space features of the different clients. The consistency loss is used to characterize the differences between the initial local latent space features of the different clients.

[0322] Receive the model update instruction sent by the server based on the consistency loss;

[0323] The client's local medical task prediction model and the parameters of the initial linear layer are updated based on the model update instruction, or the parameters of the initial linear layer are updated based on the model update instruction.

[0324] Optionally, mapping the client's local medical image data features to the client's initial local latent space features through an initial linear layer includes:

[0325] The initial linear layer maps the local medical image data features of the client at different levels to the initial local latent space features of the client at different levels. The local medical image data features of the client at different levels are output by the network layers at different levels in the local medical task prediction model of the client.

[0326] Sending the client's initial local latent space features to the server includes:

[0327] The server sends the initial local latent space features of the clients at different levels, so that the server determines the consistency sub-loss corresponding to the level based on the initial local latent space features of the different clients at the same level, and the sum of the consistency sub-losses corresponding to each level is used as the consistency loss.

[0328] Optionally, the step of extracting local medical image data features from local medical image samples in the local medical image sample set using the client's local medical task prediction model includes:

[0329] The training sample data features of each local medical image sample in the local medical image sample set are extracted using the local medical task prediction model of the client.

[0330] The average value of the element values ​​in the training sample data features of each local medical image sample is determined as the element value in the local medical image data features.

[0331] Optionally, obtaining the client's local medical task prediction model includes:

[0332] Based on the training sample data features extracted from the local medical image samples by the local medical task prediction model of the client, the sample prediction result corresponding to the local medical image sample is determined.

[0333] Based on the sample prediction results and sample result labels, the medical imaging task loss is determined;

[0334] Based on the feature correlation between different feature channels in the training sample data features, a correlation loss is determined. The correlation loss is used to characterize the difference between the correlation of the original features and the correlation of the expected features between different feature channels.

[0335] The client's local medical task prediction model is trained based on the total loss of the correlation loss and the medical imaging task loss.

[0336] By way of example only, in some embodiments, the client determines the medical imaging task loss based on the sample prediction results and sample result labels of local medical image samples.

[0337] Optionally, the medical imaging task loss can be determined based on the sample prediction results and sample result labels of local medical image samples, using cross-entropy loss, mean squared error loss, KL divergence loss, L1 norm loss, or any other possible form. There are no restrictions on this.

[0338] For example, the medical imaging task loss of client c can be characterized as

[0339] Optionally, based on the feature correlation between different feature channels in the training sample data features, a correlation loss is determined, including:

[0340] The training sample data features are grouped according to feature channels to obtain at least two feature groups;

[0341] Based on the correlation between the feature channels in the feature group, the original cross-correlation matrix corresponding to the feature group is determined. The original cross-correlation matrix is ​​used to characterize the correlation between the original features of different feature channels in the feature group.

[0342] Based on the original cross-correlation matrix corresponding to each feature group, an expected cross-correlation matrix is ​​determined. The expected cross-correlation matrix is ​​used to characterize the correlation of the expected features between different feature channels in the feature group.

[0343] Based on the original cross-correlation matrix and the expected cross-correlation matrix corresponding to the feature group, the correlation loss of the feature group is determined;

[0344] The correlation loss is determined based on the sum of the correlation losses of each of the feature groups.

[0345] Optionally, determining the expected cross-correlation matrix based on the original cross-correlation matrix corresponding to each of the feature groups includes:

[0346] The average value of the elements in the original cross-correlation matrix corresponding to each of the feature groups is determined as the element value in the average cross-correlation matrix;

[0347] Based on the difference between the element values ​​at the same position in the original cross-correlation matrix and the average cross-correlation matrix corresponding to the feature group, the expected cross-correlation matrix corresponding to the feature group is determined.

[0348] Specifically, when the difference is greater than the difference threshold or the element value is located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​1; when the difference is less than or equal to the difference threshold and the element value is not located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​0.

[0349] Optionally, the method further includes:

[0350] The number of groups is determined based on the number of different clients in federated learning, wherein the number of groups is positively correlated with the number of clients;

[0351] The training sample data features are grouped according to feature channels to obtain at least two feature groups, including:

[0352] The training sample data features are grouped according to the feature channels and the number of groups to obtain the feature groups of the specified number.

[0353] Optionally, determining the correlation loss based on the feature correlation between different feature channels in the training sample data features includes:

[0354] Based on the feature correlation between different feature channels in the training sample data features at different levels, the correlation loss at different levels is determined, and the training sample data features at different levels are output by the network layers at different levels in the local medical task prediction model.

[0355] The process of training the client's local medical task prediction model based on the total loss of the correlation loss and the medical imaging task loss includes:

[0356] The client's local medical task prediction model is trained based on the total loss of the correlation loss at different levels and the medical imaging task loss.

[0357] In some embodiments, this application also proposes a method performed by a server in federated learning in a medical image prediction scenario.

[0358] In some embodiments, the server receives local latent space features from different clients in federated learning. The local latent space features of a client are obtained by mapping the local medical image data features of the client. The local medical image data features of the client are extracted from local medical image samples in the local medical image sample set through the local medical task prediction model of the client. The local latent space features of the client are the feature representation of the local medical image samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local medical image data features of different clients.

[0359] In some embodiments, the service aggregates the local medical task prediction models of the different clients based on the local latent space features of the different clients to obtain an aggregated medical image prediction model; the local medical task prediction model of the client is trained based on the local medical image sample set.

[0360] Optionally, the aggregation of local medical task prediction models for different clients based on their local latent space features to obtain an aggregated medical image prediction model includes:

[0361] Based on the local latent space features of the different clients, the aggregation weights corresponding to the local medical task prediction models of the different clients are determined.

[0362] The local medical task prediction models of different clients are weighted and aggregated based on the aggregation weights to obtain the aggregated medical image prediction model.

[0363] Optionally, determining the aggregation weights corresponding to the local medical task prediction models of the different clients based on their local latent space features includes:

[0364] The average value of the local latent space features of the different clients is determined as the global latent space feature;

[0365] Based on the similarity between the local latent space features and the global latent space features of the different clients, the aggregation weights corresponding to the local medical task prediction models of the different clients are determined, wherein the aggregation weights are positively correlated with the similarity.

[0366] Optionally, determining the aggregation weights corresponding to the local medical task prediction models of the different clients based on their local latent space features includes:

[0367] Based on the local latent space features of the different clients at the same level, the aggregation weights corresponding to the network layers at the same level in the local medical task prediction model of the different clients are determined. The local latent space features of the different clients at the same level are output by the network layers and linear layers at the same level in the local medical task prediction model of the different clients.

[0368] The weighted aggregation of local medical task prediction models for different clients based on the aggregation weights to obtain the aggregated medical image prediction model includes:

[0369] According to the aggregation weights corresponding to the same level of network layers in the local medical task prediction models of different clients, the same level of network layers in the local medical task prediction models of different clients are weighted and aggregated to obtain the aggregated medical image prediction model containing different levels of network layers.

[0370] Optionally, the client's local latent space features are obtained by mapping the client's local medical image data features through a linear layer, and the method further includes:

[0371] The system receives initial local latent space features from different clients. The initial local latent space features of the clients are obtained by mapping initial local medical image data features to initial linear layers. The initial local medical image data features are extracted from local medical image samples in the local medical image sample set by the local medical task prediction model of the clients.

[0372] A consistency loss is determined based on the initial local latent space features of the different clients, and the consistency loss is used to characterize the differences between the initial local latent space features of the different clients.

[0373] Based on the consistency loss, model update instructions are sent to the different clients. The clients are used to update their local medical task prediction model and the parameters of the initial linear layer based on the model update instructions, or to update the parameters of the initial linear layer.

[0374] This application also proposes a federated learning method jointly executed by a client and a server in federated learning. The method includes:

[0375] In federated learning, the client obtains its local model, which is trained based on a local training sample set.

[0376] The client extracts local data features from the local training samples in the local training sample set using its local model.

[0377] The client maps the client's local data features to the client's local latent space features. The client's local latent space features are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0378] The client sends its local latent space features to the server in the federated learning process.

[0379] The server aggregates the local models of the different clients based on their local latent space features to obtain an aggregated model.

[0380] Optionally, the client maps its local data features to its local latent space features, including:

[0381] The client maps its local data features to its local latent space features through a linear layer.

[0382] Optionally, the method further includes:

[0383] The client maps its local data features to its initial local latent space features through an initial linear layer.

[0384] The client sends its initial local latent space features to the server;

[0385] The server determines a consistency loss based on the initial local latent space features of the different clients, and the consistency loss is used to characterize the differences between the initial local latent space features of the different clients.

[0386] The client receives the model update instruction sent by the server based on the consistency loss;

[0387] The client updates its local model and the parameters of the initial linear layer based on the model update instruction, or updates the parameters of the initial linear layer based on the model update instruction.

[0388] Optionally, the client maps its local data features to its initial local latent space features through an initial linear layer, including:

[0389] The client maps the local data features of the client at different levels to the initial local latent space features of the client at different levels through the initial linear layer. The local data features of the client at different levels are output by the network layers at different levels in the local model of the client.

[0390] The client sends its initial local latent space characteristics to the server, including:

[0391] The client sends its initial local latent space features at different levels to the server;

[0392] The server determines the consistency loss based on the initial local latent space characteristics of the different clients, including:

[0393] The server determines the consistency sub-loss corresponding to the level based on the initial local latent space features of the different clients at the same level.

[0394] The server uses the sum of the consistency sub-losses corresponding to each level as the consistency loss.

[0395] Optionally, the client extracts local data features from the local training samples in the local training sample set using its local model, including:

[0396] The client extracts training sample data features of each local training sample in the local training sample set through the client's local model.

[0397] The client determines the average value of the element values ​​in the training sample data features of each of the local training samples as the element value in the local data features.

[0398] Optionally, the client in the federated learning process obtains the client's local model, including:

[0399] The client determines the sample prediction result corresponding to the local training sample based on the training sample data features extracted from the local training sample by the client's local model.

[0400] The client determines the task loss based on the sample prediction results and sample result labels;

[0401] The client determines the correlation loss based on the feature correlation between different feature channels in the training sample data features. The correlation loss is used to characterize the difference between the correlation of the original features and the correlation of the expected features between different feature channels.

[0402] The client trains its local model based on the total loss of the correlation loss and the task loss.

[0403] Optionally, the client determines the correlation loss based on the feature correlation between different feature channels in the training sample data features, including:

[0404] The client groups the training sample data features according to the feature channels to obtain at least two feature groups;

[0405] The client determines the original cross-correlation matrix corresponding to the feature group based on the correlation between the feature channels in the feature group. The original cross-correlation matrix is ​​used to characterize the correlation between the original features of different feature channels in the feature group.

[0406] The client determines the expected cross-correlation matrix based on the original cross-correlation matrix corresponding to each of the feature groups. The expected cross-correlation matrix is ​​used to characterize the correlation of the expected features between different feature channels in the feature group.

[0407] The client determines the correlation loss of the feature group based on the original cross-correlation matrix and the expected cross-correlation matrix corresponding to the feature group;

[0408] The client determines the correlation loss based on the sum of the correlation losses of each of the feature groups.

[0409] Optionally, the client determines the expected cross-correlation matrix based on the original cross-correlation matrix corresponding to each of the feature groups, including:

[0410] The client determines the average value of the elements in the original cross-correlation matrix corresponding to each of the feature groups as the element value in the average cross-correlation matrix;

[0411] The client determines the expected cross-correlation matrix corresponding to the feature group based on the difference between the element values ​​at the same position in the original cross-correlation matrix and the average cross-correlation matrix corresponding to the feature group.

[0412] Specifically, when the difference is greater than the difference threshold or the element value is located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​1; when the difference is less than or equal to the difference threshold and the element value is not located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​0.

[0413] Optionally, the method further includes:

[0414] The client determines the number of groups based on the number of different clients in the federated learning process, wherein the number of groups is positively correlated with the number of clients.

[0415] The client groups the training sample data features according to feature channels to obtain at least two feature groups, including:

[0416] The client groups the training sample data features according to the feature channels and the number of groups, to obtain the feature groups of the specified number.

[0417] Optionally, the client determines the correlation loss based on the feature correlation between different feature channels in the training sample data features, including:

[0418] The client determines the correlation loss at different levels based on the feature correlation between different feature channels in the training sample data features at different levels, and the training sample data features at different levels are output by the network layers at different levels in the local model.

[0419] The client trains its local model based on the total loss of the relevance loss and the task loss, including:

[0420] The client trains its local model based on the total loss of the correlation loss and the task loss at different levels.

[0421] Optionally, the server aggregates the local models of the different clients based on their local latent space features to obtain an aggregated model, including:

[0422] The server determines the aggregation weights corresponding to the local models of the different clients based on the local latent space features of the different clients.

[0423] The server weights and aggregates the local models of the different clients based on the aggregation weight to obtain the aggregated model.

[0424] Optionally, the server determines the aggregation weights corresponding to the local models of the different clients based on the local latent space features of the different clients, including:

[0425] The server determines the average value of the local latent space features of the different clients as the global latent space features;

[0426] The server determines the aggregation weights corresponding to the local models of different clients based on the similarity between the local latent space features of the different clients and the global latent space features, wherein the aggregation weights are positively correlated with the similarity.

[0427] Optionally, the server determines the aggregation weights corresponding to the local models of the different clients based on the local latent space features of the different clients, including:

[0428] The server determines the aggregation weights corresponding to the network layers at the same level in the local models of the different clients based on the local latent space features of the different clients at the same level. The local latent space features of the different clients at the same level are output by the network layers and linear layers at the same level in the local models of the different clients.

[0429] The server weights and aggregates the local models of the different clients based on the aggregation weight to obtain the aggregated model, including:

[0430] The server weights and aggregates the network layers of the same level in the local models of different clients according to the aggregation weights corresponding to the same level of network layers in the local models of different clients, thereby obtaining the aggregated model containing network layers of different levels.

[0431] See Figure 11 , Figure 11 This is a structural block diagram of a federated learning apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0432] The acquisition module 1101 is used to acquire the local model of the client, which is trained based on the local training sample set;

[0433] The local data feature extraction module 1102 is used to extract the local data features of the client from the local training samples of the local training sample set through the local model of the client.

[0434] The local latent space feature extraction module 1103 is used to map the local data features of the client to the local latent space features of the client. The local latent space features of the client are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0435] The sending module 1104 is used to send the local latent space features of the client to the server in the federated learning, so that the server can aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model.

[0436] Optionally, the local latent space feature extraction module 1103 is used for:

[0437] The local data features of the client are mapped to the local latent space features of the client through a linear layer.

[0438] Optionally, the device further includes a consistency training module, which is used for:

[0439] The client's local data features are mapped to the client's initial local latent space features through an initial linear layer;

[0440] The initial local latent space features of the client are sent to the server so that the server determines a consistency loss based on the initial local latent space features of the different clients. The consistency loss is used to characterize the differences between the initial local latent space features of the different clients.

[0441] Receive the model update instruction sent by the server based on the consistency loss;

[0442] The client's local model and the parameters of the initial linear layer are updated based on the model update instruction, or the parameters of the initial linear layer are updated based on the model update instruction.

[0443] Optional, consistency training module, used for:

[0444] The initial linear layer maps the local data features of the client at different levels to the initial local latent space features of the client at different levels. The local data features of the client at different levels are output by the network layers at different levels in the local model of the client.

[0445] The server sends the initial local latent space features of the clients at different levels, so that the server determines the consistency sub-loss corresponding to the level based on the initial local latent space features of the different clients at the same level, and the sum of the consistency sub-losses corresponding to each level is used as the consistency loss.

[0446] Optionally, the local data feature extraction module 1102 is used for:

[0447] The training sample data features of each local training sample in the local training sample set are extracted using the local model of the client.

[0448] The average value of the element values ​​in the training sample data features of each of the local training samples is determined as the element value in the local data features.

[0449] Optionally, module 1101 is used for:

[0450] Based on the features of the training sample data extracted from the local training samples by the local model of the client, the sample prediction result corresponding to the local training sample is determined.

[0451] Based on the sample prediction results and sample result labels, the task loss is determined;

[0452] Based on the feature correlation between different feature channels in the training sample data features, a correlation loss is determined. The correlation loss is used to characterize the difference between the correlation of the original features and the correlation of the expected features between different feature channels.

[0453] The client's local model is trained based on the total loss of the correlation loss and the task loss.

[0454] Optionally, module 1101 is used for:

[0455] The training sample data features are grouped according to feature channels to obtain at least two feature groups;

[0456] Based on the correlation between the feature channels in the feature group, the original cross-correlation matrix corresponding to the feature group is determined. The original cross-correlation matrix is ​​used to characterize the correlation between the original features of different feature channels in the feature group.

[0457] Based on the original cross-correlation matrix corresponding to each feature group, an expected cross-correlation matrix is ​​determined. The expected cross-correlation matrix is ​​used to characterize the correlation of the expected features between different feature channels in the feature group.

[0458] Based on the original cross-correlation matrix and the expected cross-correlation matrix corresponding to the feature group, the correlation loss of the feature group is determined;

[0459] The correlation loss is determined based on the sum of the correlation losses of each of the feature groups.

[0460] Optionally, module 1101 is used for:

[0461] The average value of the elements in the original cross-correlation matrix corresponding to each of the feature groups is determined as the element value in the average cross-correlation matrix;

[0462] Based on the difference between the element values ​​at the same position in the original cross-correlation matrix and the average cross-correlation matrix corresponding to the feature group, the expected cross-correlation matrix corresponding to the feature group is determined.

[0463] Specifically, when the difference is greater than the difference threshold or the element value is located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​1; when the difference is less than or equal to the difference threshold and the element value is not located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​0.

[0464] Optionally, module 1101 is used for:

[0465] The number of groups is determined based on the number of different clients in federated learning, wherein the number of groups is positively correlated with the number of clients;

[0466] The training sample data features are grouped according to the feature channels and the number of groups to obtain the feature groups of the specified number.

[0467] Optionally, module 1101 is used for:

[0468] Based on the feature correlation between different feature channels in the training sample data features at different levels, the correlation loss at different levels is determined, and the training sample data features at different levels are output by the network layers at different levels in the local model.

[0469] The client's local model is trained based on the total loss of the correlation loss and the task loss at different levels.

[0470] See Figure 12 , Figure 12 This is a structural block diagram of a federated learning apparatus provided in another exemplary embodiment of this application. The apparatus includes:

[0471] The receiving module 1201 is used to receive local latent space features of different clients in federated learning. The local latent space features of a client are obtained by mapping the local data features of the client. The local data features of the client are extracted from the local training samples in the local training sample set through the local model of the client. The local latent space features of the client are the feature representation of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients.

[0472] The aggregation module 1202 is used to aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model; the local models of the clients are trained based on the local training sample set.

[0473] Optionally, aggregation module 1202 is used for:

[0474] Based on the local latent space features of the different clients, the aggregation weights corresponding to the local models of the different clients are determined.

[0475] The aggregated model is obtained by weighting and aggregating the local models of the different clients based on the aggregation weight.

[0476] Optionally, aggregation module 1202 is used for:

[0477] The average value of the local latent space features of the different clients is determined as the global latent space feature;

[0478] Based on the similarity between the local latent space features of the different clients and the global latent space features, the aggregation weights corresponding to the local models of the different clients are determined, wherein the aggregation weights are positively correlated with the similarity.

[0479] Optionally, aggregation module 1202 is used for:

[0480] Based on the local latent space features of the different clients at the same level, the aggregation weights corresponding to the network layers at the same level in the local models of the different clients are determined. The local latent space features of the different clients at the same level are output by the network layers and linear layers at the same level in the local models of the different clients.

[0481] According to the aggregation weights corresponding to the same level of network layers in the local models of different clients, the same level of network layers in the local models of different clients are weighted and aggregated to obtain the aggregated model containing different levels of network layers.

[0482] Optionally, the client's local latent space features are obtained by mapping the client's local data features through a linear layer. The device further includes a consistency training module, which is used for:

[0483] The system receives initial local latent space features from different clients. The initial local latent space features of the clients are obtained by mapping initial local data features to initial linear layers. The initial local data features are extracted from the local training samples in the local training sample set by the local model of the clients.

[0484] A consistency loss is determined based on the initial local latent space features of the different clients, and the consistency loss is used to characterize the differences between the initial local latent space features of the different clients.

[0485] Based on the consistency loss, model update instructions are sent to the different clients. The clients are used to update their local models and the parameters of the initial linear layer based on the model update instructions, or to update the parameters of the initial linear layer.

[0486] See Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. In some embodiments, the computer device can be implemented as a client or server in federated learning in the above embodiments.

[0487] Specifically, the computer device 1300 includes a Central Processing Unit (CPU) 1301, a system memory 1304 including a random access memory 1302 and a read-only memory 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 to facilitate information transfer between various components within the computer, and a mass storage device 1307 for storing the operating system 1312, application programs 1314, and other program modules 1315.

[0488] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0489] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer-readable media (not shown) such as a hard disk or drive.

[0490] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1307 described above can be collectively referred to as memory.

[0491] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1301. The one or more programs contain computer instructions for implementing the methods described above, and the central processing unit 1301 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0492] According to various embodiments of this application, the computer device 1300 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1300 can be connected to a network 1313 via a network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can be used to connect to other types of networks or remote computer systems (not shown).

[0493] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0494] This application also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the method described in the above embodiments. Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0495] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above aspects.

[0496] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A federated learning method, characterized in that, The method is executed by a client in federated learning, and the method includes: Obtain the client's local model, which is trained based on the local training sample set; Using the client's local model, local data features are extracted from the local training samples in the local training sample set. The local data features of the client are mapped to the local latent space features of the client. The local latent space features of the client are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients. The local latent space features of the client are sent to the server in the federated learning process, so that the server can aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model.

2. The method according to claim 1, characterized in that, The step of mapping the client's local data features to the client's local latent space features includes: The local data features of the client are mapped to the local latent space features of the client through a linear layer.

3. The method according to claim 2, characterized in that, The method further includes: The client's local data features are mapped to the client's initial local latent space features through an initial linear layer; The initial local latent space features of the client are sent to the server so that the server determines a consistency loss based on the initial local latent space features of the different clients. The consistency loss is used to characterize the differences between the initial local latent space features of the different clients. Receive the model update instruction sent by the server based on the consistency loss; The client's local model and the parameters of the initial linear layer are updated based on the model update instruction, or the parameters of the initial linear layer are updated based on the model update instruction.

4. The method according to claim 3, characterized in that, The step of mapping the client's local data features to the client's initial local latent space features through an initial linear layer includes: The initial linear layer maps the local data features of the client at different levels to the initial local latent space features of the client at different levels. The local data features of the client at different levels are output by the network layers at different levels in the local model of the client. Sending the client's initial local latent space features to the server includes: The server sends the initial local latent space features of the clients at different levels, so that the server determines the consistency sub-loss corresponding to the level based on the initial local latent space features of the different clients at the same level, and the sum of the consistency sub-losses corresponding to each level is used as the consistency loss.

5. The method according to any one of claims 1 to 4, characterized in that, The step of extracting local data features of the client from the local training samples of the local training sample set using the client's local model includes: The training sample data features of each local training sample in the local training sample set are extracted using the local model of the client. The average value of the element values ​​in the training sample data features of each of the local training samples is determined as the element value in the local data features.

6. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the client's local model includes: Based on the features of the training sample data extracted from the local training samples by the local model of the client, the sample prediction result corresponding to the local training sample is determined. Based on the sample prediction results and sample result labels, the task loss is determined; Based on the feature correlation between different feature channels in the training sample data features, a correlation loss is determined. The correlation loss is used to characterize the difference between the correlation of the original features and the correlation of the expected features between different feature channels. The client's local model is trained based on the total loss of the correlation loss and the task loss.

7. The method according to claim 6, characterized in that, The determination of correlation loss based on the feature correlation between different feature channels in the training sample data features includes: The training sample data features are grouped according to feature channels to obtain at least two feature groups; Based on the correlation between the feature channels in the feature group, the original cross-correlation matrix corresponding to the feature group is determined. The original cross-correlation matrix is ​​used to characterize the correlation between the original features of different feature channels in the feature group. Based on the original cross-correlation matrix corresponding to each feature group, an expected cross-correlation matrix is ​​determined. The expected cross-correlation matrix is ​​used to characterize the correlation of the expected features between different feature channels in the feature group. Based on the original cross-correlation matrix and the expected cross-correlation matrix corresponding to the feature group, the correlation loss of the feature group is determined; The correlation loss is determined based on the sum of the correlation losses of each of the feature groups.

8. The method according to claim 7, characterized in that, The step of determining the expected cross-correlation matrix based on the original cross-correlation matrix corresponding to each of the feature groups includes: The average value of the elements in the original cross-correlation matrix corresponding to each of the feature groups is determined as the element value in the average cross-correlation matrix; Based on the difference between the element values ​​at the same position in the original cross-correlation matrix and the average cross-correlation matrix corresponding to the feature group, the expected cross-correlation matrix corresponding to the feature group is determined. Specifically, when the difference is greater than the difference threshold or the element value is located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​1; when the difference is less than or equal to the difference threshold and the element value is not located on the main diagonal, the element value at the corresponding position in the expected cross-correlation matrix is ​​0.

9. The method according to claim 7, characterized in that, The method further includes: The number of groups is determined based on the number of different clients in federated learning, wherein the number of groups is positively correlated with the number of clients; The training sample data features are grouped according to feature channels to obtain at least two feature groups, including: The training sample data features are grouped according to the feature channels and the number of groups to obtain the feature groups of the specified number.

10. The method according to claim 6, characterized in that, The determination of correlation loss based on the feature correlation between different feature channels in the training sample data features includes: Based on the feature correlation between different feature channels in the training sample data features at different levels, the correlation loss at different levels is determined, and the training sample data features at different levels are output by the network layers at different levels in the local model. The step of training the client's local model based on the total loss of the correlation loss and the task loss includes: The client's local model is trained based on the total loss of the correlation loss and the task loss at different levels.

11. A federated learning method, characterized in that, The method is executed by a server in federated learning, and the method includes: The local latent space features of different clients in federated learning are received. The local latent space features of a client are obtained by mapping the local data features of the client. The local data features of the client are extracted from the local training samples in the local training sample set through the local model of the client. The local latent space features of the client are the feature representation of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients. Based on the local latent space features of the different clients, the local models of the different clients are aggregated to obtain an aggregated model; the local models of the clients are trained based on the local training sample set.

12. The method according to claim 11, characterized in that, The aggregation of local models from different clients based on their local latent space features to obtain an aggregated model includes: Based on the local latent space features of the different clients, the aggregation weights corresponding to the local models of the different clients are determined. The aggregated model is obtained by weighting and aggregating the local models of the different clients based on the aggregation weight.

13. The method according to claim 12, characterized in that, The step of determining the aggregation weights corresponding to the local models of different clients based on the local latent space features of the different clients includes: The average value of the local latent space features of the different clients is determined as the global latent space feature; Based on the similarity between the local latent space features of the different clients and the global latent space features, the aggregation weights corresponding to the local models of the different clients are determined, wherein the aggregation weights are positively correlated with the similarity.

14. The method according to claim 12, characterized in that, The step of determining the aggregation weights corresponding to the local models of different clients based on the local latent space features of the different clients includes: Based on the local latent space features of different clients at the same level, the aggregation weights corresponding to the network layers at the same level in the local models of different clients are determined. The local latent space features of different clients at the same level are output by the network layers and linear layers at the same level in the local models of different clients. The weighted aggregation of the local models of the different clients based on the aggregation weight to obtain the aggregated model includes: According to the aggregation weights corresponding to the same level of network layers in the local models of different clients, the same level of network layers in the local models of different clients are weighted and aggregated to obtain the aggregated model containing different levels of network layers.

15. The method according to any one of claims 11 to 14, characterized in that, The client's local latent space features are obtained by mapping the client's local data features through a linear layer. The method further includes: The system receives initial local latent space features from different clients. The initial local latent space features of the clients are obtained by mapping initial local data features to initial linear layers. The initial local data features are extracted from the local training samples in the local training sample set by the local model of the clients. A consistency loss is determined based on the initial local latent space features of the different clients, and the consistency loss is used to characterize the differences between the initial local latent space features of the different clients. Based on the consistency loss, model update instructions are sent to the different clients. The clients are used to update their local models and the parameters of the initial linear layer based on the model update instructions, or to update the parameters of the initial linear layer.

16. A federated learning device, characterized in that, The device includes: The acquisition module is used to acquire the local model of the client, which is trained based on the local training sample set. The local data feature extraction module is used to extract the local data features of the client from the local training samples of the local training sample set through the client's local model; The local latent space feature extraction module is used to map the local data features of the client to the local latent space features of the client. The local latent space features of the client are the feature representations of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients. The sending module is used to send the local latent space features of the client to the server in the federated learning, so that the server can aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model.

17. A federated learning device, characterized in that, The device includes: The receiving module is used to receive local latent space features from different clients in federated learning. The local latent space features of a client are obtained by mapping the local data features of the client. The local data features of the client are extracted from the local training samples in the local training sample set through the local model of the client. The local latent space features of the client are the feature representation of the local training samples in the latent space. The similarity between the local latent space features of different clients is higher than the similarity between the local data features of different clients. The aggregation module is used to aggregate the local models of the different clients based on the local latent space features of the different clients to obtain an aggregated model; the local models of the clients are trained based on the local training sample set.

18. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one computer instruction, which is executed by the processor to implement the federated learning method as described in any one of claims 1 to 10, or the federated learning method as described in any one of claims 11 to 15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the federated learning method as described in any one of claims 1 to 10, or the federated learning method as described in any one of claims 11 to 15.

20. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the federated learning method as described in any one of claims 1 to 10, or the federated learning method as described in any one of claims 11 to 15.

Images