Federated learning method based on continual learning and related devices

Abstract

The application relates to the technical field of computers, and provides a federated learning method based on continuous learning and related equipment, which is applied to a federated learning system comprising a server and multiple clients. The server and the clients each store a first federated learning model trained based on a first sample data set. The method obtains a second sample data set through the clients, the second sample data set and the first sample data set are respectively used for different learning tasks of the first federated learning model, multiple sample data are extracted from the first sample data as auxiliary sample data, and the local model is trained by minimizing the loss generated when learning the multiple auxiliary sample data while the model learns the second sample data set. According to the embodiment of the application, the model gradient is corrected by incorporating the loss generated when learning multiple old samples while the model learns a new task, the old knowledge learned by the model is protected, and the catastrophic forgetting is alleviated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a federated learning method based on continuous learning and related equipment. Background Technology

[0002] Federated learning is a distributed machine learning technique that breaks down data silos and protects data privacy. It allows multiple centers to jointly train a machine learning model without exchanging local data. Compared to models trained on single-center data, federated learning models often have higher segmentation and generalization performance.

[0003] However, in the process of continuously learning a series of tasks that arrive in sequence, federated learning may experience a situation where the performance of the global model on old tasks significantly declines as new tasks are learned, which is known as catastrophic forgetting. Summary of the Invention

[0004] The main objective of this application is to propose a federated learning method, system, electronic device, and computer-readable storage medium based on continuous learning, which can alleviate the catastrophic forgetting of models in federated learning.

[0005] To achieve the above objectives, a first aspect of this application proposes a federated learning method based on continuous learning. The method is applied to a federated learning system, which includes a server and multiple clients. The server is communicatively connected to each of the multiple clients. Both the server and the clients store a first federated learning model trained based on a first sample dataset. The method includes:

[0006] The client obtains a second sample dataset, and the second sample dataset and the first sample dataset are used for different learning tasks of the first federated learning model, respectively.

[0007] The client extracts multiple sample data from the first sample dataset as auxiliary sample data.

[0008] The client uses the first federated learning model as a local model, with the training objective of minimizing the loss of the local model when learning the second sample dataset and multiple auxiliary sample data, and trains the local model based on the second sample dataset and multiple auxiliary sample data.

[0009] The trained local model is uploaded to the server via the client.

[0010] The server receives local models uploaded by multiple clients, integrates the first federated learning model and the multiple local models, and obtains a second federated learning model.

[0011] According to some embodiments of the federated learning method based on continuous learning provided by the present invention, after extracting multiple sample data as auxiliary sample data from the first sample dataset through the client, the method further includes:

[0012] The client inputs multiple auxiliary sample data into the first federated learning model to obtain the first classification prediction value corresponding to the multiple auxiliary sample data through the first federated learning model;

[0013] The step of training the local model based on the second sample dataset and the multiple auxiliary sample datasets, with the training objective of minimizing the loss incurred by the local model when learning the second sample dataset and the multiple auxiliary sample datasets, includes:

[0014] The second sample dataset and the multiple auxiliary sample data are input into the local model to obtain the second classification prediction value corresponding to each sample data in the second sample dataset and the multiple auxiliary sample data through the local model;

[0015] The true classification label of each sample in the second sample dataset is determined, and the first loss function is determined with the training objective of minimizing the difference between the second classification prediction value and the true classification label of each sample in the second sample dataset.

[0016] The second loss function is determined with the training objective of minimizing the difference between the first and second classification prediction values ​​of each auxiliary sample data.

[0017] The local model is trained based on the first loss function and the second loss function.

[0018] The federated learning method based on continuous learning provided by some embodiments of the present invention,

[0019] The first loss function is determined by the following formula:

[0020]

[0021] Wherein, the L C Let M be the first loss function, where M represents the number of categories in the second sample dataset, and y is the... C For a one-hot encoded vector, if the true classification label of the sample data is the same as classification C, then y CSet p to 1, otherwise set p to 0. C The second-class prediction value that indicates the sample data belongs to class C.

[0022] According to some embodiments of the federated learning method based on continuous learning provided by the present invention, the second loss function is determined by the following formula:

[0023]

[0024] Wherein, the L MSE The second loss function is defined as n, where n is the number of samples in the auxiliary sample data, y′ is the second classification prediction value, and y is the first classification prediction value.

[0025] According to some embodiments of the present invention, the federated learning method based on continuous learning further includes, before integrating the first federated learning model and the plurality of local models to obtain the second federated learning model:

[0026] Obtain the number of samples in the second sample dataset in each client;

[0027] The process of integrating the first federated learning model and the multiple local models to obtain the second federated learning model includes:

[0028] Based on the number of samples in the second sample dataset in each client, determine the first weight coefficient of each local model;

[0029] Based on the first weight coefficient of each local model, the model parameters of multiple local models are weighted and averaged to obtain an intermediate model;

[0030] The model parameters of the first federated learning model and the model parameters of the intermediate model are weighted and averaged to obtain the second federated learning model.

[0031] According to some embodiments of the federated learning method based on continuous learning provided by the present invention, the step of performing a weighted average of the model parameters of the first federated learning model and the model parameters of the intermediate model to obtain a second federated learning model includes:

[0032] Obtain the dynamic second weight coefficient;

[0033] Based on the second weighting coefficient, the model parameters of the first federated learning model and the model parameters of the intermediate model are weighted and averaged to obtain the second federated learning model.

[0034] The second federated learning model is tested. If the test result of the second federated learning model does not meet the preset integration termination condition, the process returns to the step of obtaining the dynamic second weight coefficients until the test result of the second federated learning model meets the integration termination condition.

[0035] According to some embodiments of the federated learning method based on continuous learning provided by the present invention, before testing the second federated learning model, the method further includes:

[0036] Obtain the test sample dataset;

[0037] The testing of the second federated learning model includes:

[0038] The test sample dataset is input into the second federated learning model to obtain the third classification prediction value corresponding to each test sample data in the test sample dataset through the second federated learning model;

[0039] Determine the true classification label of each test sample data in the test sample dataset, and determine and record the evaluation index of the second federated learning model based on the third classification prediction value and the true classification label corresponding to the test sample data;

[0040] The evaluation metrics include at least one of the following: Dessian similarity coefficient, crossover ratio, or accuracy.

[0041] To achieve the above objectives, a second aspect of this application proposes a federated learning system based on continuous learning. The system includes a server and multiple clients, with the server communicating with each client. Both the server and the clients store a first federated learning model trained on a first sample dataset.

[0042] The client includes:

[0043] The acquisition module is used to acquire a second sample dataset, wherein the first sample dataset and the second sample dataset are used for different learning tasks of the first federated learning model, respectively.

[0044] The extraction module is used to extract multiple sample data from the first sample dataset as auxiliary sample data;

[0045] The training module is used to train the local model based on the second sample dataset and the multiple auxiliary sample data, with the goal of minimizing the loss generated by the local model when learning the second sample dataset and the multiple auxiliary sample data.

[0046] A sending module is used to upload the trained local model to the server;

[0047] The server includes:

[0048] A receiving module is used to receive the local models uploaded by multiple clients;

[0049] An integration module is used to integrate the first federated learning model and multiple local models to obtain a second federated learning model.

[0050] To achieve the above objectives, a third aspect of this application provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method described in the first aspect.

[0051] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more computer programs that can be executed by one or more processors to implement the method described in the first aspect.

[0052] This application proposes a federated learning method, system, electronic device, and computer-readable storage medium based on continuous learning. The method is applied to a federated learning system, which includes a server and multiple clients. The server is communicatively connected to each client. Both the server and the clients store a first federated learning model trained on a first sample dataset. The method obtains a second sample dataset from the clients, which is used for a different learning task than the first sample dataset for the first federated learning model. Multiple sample data are extracted from the first sample dataset as auxiliary sample data. The clients use the first federated learning model as a local model, and the training objective is to minimize the loss incurred by the local model when learning the second sample dataset and the multiple auxiliary sample data. The local model is trained based on the second sample dataset and the multiple auxiliary sample data. Then, the server receives the trained local models uploaded by the multiple clients and integrates the first federated learning model and the multiple local models to obtain a second federated learning model. In this embodiment, while the local model learns a new task, the loss incurred when learning multiple old samples is incorporated to correct the model gradient, thereby protecting the model's learned old knowledge and mitigating catastrophic forgetting in federated learning. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a federated learning method based on continuous learning provided in an embodiment of this application;

[0054] Figure 2 This is a flowchart illustrating a federated learning method based on continuous learning, provided in another embodiment of this application.

[0055] Figure 3 This is a flowchart illustrating a federated learning method based on continuous learning, provided in another embodiment of this application.

[0056] Figure 4 This is a flowchart illustrating a federated learning method based on continuous learning, provided in another embodiment of this application.

[0057] Figure 5 This is a flowchart illustrating a federated learning method based on continuous learning, provided in another embodiment of this application.

[0058] Figure 6 This is an implementation environment diagram of a federated learning method based on continuous learning provided in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the structure of a federated learning system based on continuous learning provided in an embodiment of this application;

[0060] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0063] Federated learning is a distributed machine learning technique that breaks down data silos and protects data privacy. It allows multiple centers to jointly train a machine learning model without exchanging local data. Compared to models trained on single-center data, federated learning models often have higher segmentation and generalization performance.

[0064] However, when federated learning is faced with a series of tasks that come in sequence, the performance of the global model on the old tasks will significantly decline as new tasks are learned, which is known as catastrophic forgetting.

[0065] Based on this, embodiments of this application provide a federated learning method, system, electronic device, and computer-readable storage medium based on continuous learning, which can alleviate the catastrophic forgetting of models in federated learning.

[0066] This application provides a federated learning method, system, electronic device, and computer-readable storage medium based on continuous learning, which are specifically described through the following embodiments. First, the federated learning method based on continuous learning in the embodiments of this application is described.

[0067] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0068] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] Please see Figure 6 , Figure 6 This is a diagram illustrating an implementation environment based on continuous learning, as provided in an embodiment of this application. The method is applied to a federated learning system, which includes a server and multiple clients. The server communicates with each of the multiple clients. Both the server and the clients store a first federated learning model trained based on a first sample dataset. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram illustrates a flowchart of a federated learning method based on continuous learning, as provided in an embodiment of this application. Figure 1 As shown, the federated learning method based on continuous learning includes, but is not limited to, steps S110 to S150.

[0071] Step S110: Obtain the second sample dataset through the client. The second sample dataset and the first sample dataset are used for different learning tasks of the first federated learning model, respectively.

[0072] Step S120: Extract multiple sample data from the first sample dataset as auxiliary sample data through the client.

[0073] Step S130: The first federated learning model is used as a local model through the client. The training objective is to minimize the loss of the local model when learning the second sample dataset and the multiple auxiliary sample data. The local model is trained based on the second sample dataset and the multiple auxiliary sample data.

[0074] Step S140: Upload the trained local model to the server via the client.

[0075] Step S150: The server receives the local models uploaded by multiple clients, integrates the first federated learning model and the multiple local models, and obtains the second federated learning model.

[0076] For example, the first federated learning model is a medical image segmentation model, and the first sample dataset is organ image samples. Therefore, after learning from the first sample dataset, the first federated learning model for medical image segmentation can perform relatively accurate image segmentation on other organ image samples. The learning of the first sample dataset by the first federated model is called the old task, and the knowledge learned by the model in the old task is called old knowledge. The second sample dataset, such as cell image samples, is obtained through the client. Then, the first federated learning model is used as a local model, and the local model is trained using the second sample dataset. That is, the local model learns the new task, enabling the local model to perform relatively accurate image segmentation on other cell image samples.

[0077] Understandably, the client obtains a second sample dataset for a different learning task than the first sample dataset, and extracts multiple sample data from the first sample dataset as auxiliary sample data. The first federated learning model is then used as the local model, and the local model is trained based on the second sample dataset and multiple auxiliary sample data. While the local model is learning on the second sample dataset, the loss generated when learning multiple auxiliary sample data is incorporated to correct the model gradient of the local model. The server integrates the first federated learning model obtained from learning the old task and the local model obtained from learning the new task to obtain a second federated learning model that performs well on both the old and new tasks. This enables the global model in federated learning to protect the model's old knowledge while learning the new task, mitigating the catastrophic forgetting of the model in federated learning, and enabling continuous learning under the condition of limited computing and storage resources on the client and server.

[0078] It should be noted that in step S120, a target value can be preset by the client, and sample data of the target value can be randomly extracted from the first sample dataset as auxiliary sample data.

[0079] In some embodiments, the method further includes:

[0080] Obtain the prediction difficulty of each sample in the first sample dataset, and sort each sample in the first sample dataset from largest to smallest according to the prediction difficulty;

[0081] Step S120 includes:

[0082] Multiple sample data points are extracted from the first sample dataset according to the sorting order to serve as auxiliary sample data.

[0083] It should be noted that multiple sample data are extracted from the first sample dataset as auxiliary sample data according to the sorting. In other words, multiple sample data with high prediction difficulty are extracted from the first sample dataset as auxiliary sample data, and the local model is trained based on the second sample dataset and multiple auxiliary sample data.

[0084] Understandably, by selecting representative sample data that is difficult to predict from the first sample dataset as auxiliary sample data, it is possible to ensure the model performance of the local model on the first sample data, that is, to protect the model's old knowledge, while reducing the number of samples in the auxiliary sample data and improving the training efficiency of the model.

[0085] In some embodiments, the prediction difficulty of obtaining each sample data in the first sample dataset includes:

[0086] The first sample dataset is input into the first federated learning model to obtain the classification prediction value corresponding to each sample data in the first sample dataset through the first federated learning model;

[0087] Based on the classification prediction value, determine the information entropy corresponding to each sample data in the first sample dataset;

[0088] The step of sorting each sample data in the first sample dataset according to the prediction difficulty from highest to lowest includes:

[0089] Sort each sample data in the first sample dataset according to its information entropy from largest to smallest.

[0090] In some embodiments, after step S120 and before step S130, the method further includes:

[0091] The client inputs multiple auxiliary sample data into the first federated learning model to obtain a first classification prediction value corresponding to the multiple auxiliary sample data.

[0092] See Figure 2 , Figure 2 The diagram illustrates a flowchart of a federated learning method based on continuous learning, as provided in an embodiment of this application. Figure 2 As shown, the step of training the local model based on the second sample dataset and the multiple auxiliary sample data with the goal of minimizing the loss generated by the local model when learning the second sample dataset and the multiple auxiliary sample data includes, but is not limited to, steps S210 to S240.

[0093] Step S210: Input the second sample dataset and the multiple auxiliary sample data into the local model to obtain the second classification prediction value corresponding to each sample data in the second sample dataset and the multiple auxiliary sample data through the local model.

[0094] Step S220: Determine the true classification label of each sample data in the second sample dataset, and determine the first loss function with the training objective of minimizing the difference between the second classification prediction value and the true classification label of each sample data in the second sample dataset.

[0095] Step S230: Determine a second loss function with the training objective of minimizing the difference between the first classification prediction value and the second classification prediction value of each of the auxiliary sample data.

[0096] Step S240: Train the local model based on the first loss function and the second loss function.

[0097] Understandably, the training objective is to minimize the loss incurred by the local model when learning from the first sample dataset. This means adjusting the local model's gradient using the loss from learning from the first sample dataset to improve its performance on that dataset. Specifically, the local model can obtain the second-class predicted value for each sample in the second sample dataset and determine the true class label for each sample in the second sample dataset. Therefore, minimizing the loss incurred by the local model when learning from the first sample dataset becomes the training objective, which is to minimize the difference between the second-class predicted value and the true class label for each sample in the second sample dataset. A first loss function is then determined, and the local model is trained based on this first loss function.

[0098] Understandably, while minimizing the loss incurred by the local model when learning the second sample dataset, a training objective is also incorporated that minimizes the loss incurred by the local model when learning multiple auxiliary sample datasets. In other words, the losses from learning multiple auxiliary sample datasets are incorporated to jointly correct the local model's gradient, thereby reducing the performance difference between the local model and the first federated learning model on the first sample dataset. Specifically, the first federated learning model can obtain the first classification prediction values ​​corresponding to multiple auxiliary sample datasets, while the local model obtains the second classification prediction values ​​corresponding to multiple auxiliary sample datasets during its learning process on the second sample dataset. Therefore, minimizing the loss incurred by the local model when learning multiple auxiliary sample datasets is the training objective, which is to minimize the difference between the first and second classification prediction values ​​for each auxiliary sample dataset. A second loss function is then determined, and the local model is trained based on this second loss function.

[0099] In some embodiments, the first loss function is determined by the following formula:

[0100]

[0101] Wherein, the L C Let M be the first loss function, where M represents the number of categories in the second sample dataset, and y is the... C For a one-hot encoded vector, if the true classification label of the sample data is the same as classification C, then y C Set p to 1, otherwise set p to 0. C The second-class prediction value that indicates the sample data belongs to class C.

[0102] In some embodiments, the second loss function is determined by the following formula:

[0103]

[0104] Wherein, the L MSE The second loss function is defined as n, where n is the number of samples in the auxiliary sample data, y′ is the second classification prediction value, and y is the first classification prediction value.

[0105] In one specific embodiment, a total loss function is constructed based on a first loss function and a second loss function, and the local model is trained based on the total loss function. The total loss function is determined by the following formula:

[0106]

[0107] It should be noted that, based on the actual application scenario, preset loss weight coefficients can be obtained to determine the proportion of the first loss function and the second loss function in the total loss function, thereby adjusting the impact of the loss generated when learning the first sample dataset and multiple auxiliary sample data on the local model, and improving the adaptability of the federated learning method provided in this application embodiment to different application scenarios.

[0108] It should also be noted that the above embodiments provide a first loss function and a second loss function to determine the difference between the second classification predicted value of each sample data in the second sample dataset and the difference between the true and false labels, as well as the difference between the first classification predicted value and the second classification predicted value of each auxiliary sample data. Specifically, the first loss function is the cross-entropy function and the second loss function is the mean squared error function. It should be understood that other types of loss functions can also be used to determine the difference in this embodiment, which will not be listed here.

[0109] In some embodiments, before integrating the first federated learning model and the plurality of local models to obtain the second federated learning model, the method further includes:

[0110] Obtain the number of samples in the second sample dataset in each client.

[0111] Please see Figure 3 , Figure 3 The diagram illustrates a flowchart of a federated learning method based on continuous learning, as provided in an embodiment of this application. Figure 3 As shown, the process of integrating the first federated learning model and the multiple local models to obtain the second federated learning model includes, but is not limited to, steps S310 to S330.

[0112] Step S310: Determine the first weight coefficient of each local model based on the number of samples in the second sample dataset in each client.

[0113] Step S320: Based on the first weight coefficient of each local model, perform a weighted average of the model parameters of the multiple local models to obtain an intermediate model.

[0114] Step S330: Perform a weighted average of the model parameters of the first federated learning model and the model parameters of the intermediate model to obtain the second federated learning model.

[0115] Understandably, after the local model trained on the client learns from the first sample dataset, it retains some of the old knowledge learned from the first sample dataset. The server, by integrating the first federated learning model and multiple local models, can further protect the old knowledge learned on the first sample dataset in the second federated learning model, compared to only integrating the local model uploaded by the client after learning the new task.

[0116] Understandably, by determining the first weight coefficient for each local model based on the number of samples in each client, the influence of the local model on the second federated learning model on the server can be reasonably adjusted. This avoids the performance degradation of the second federated model obtained by server integration when there is an uneven distribution of sample numbers among multiple clients, resulting in large parameter differences among the trained local models.

[0117] For example, in steps S310 to S330, a total of m clients (set V) uploaded model parameters to the server. The local model, the server obtains the number n samples of the second sample dataset in each client. k The summation yields the total sample data from m clients. Then, based on the number of samples n in the second sample dataset from each client... k Determine the first weight coefficient n for each local model. k / n, and then, based on the first weight coefficient of each local model, the model parameters of multiple local models are weighted and averaged to obtain the intermediate model. The specific formula is as follows:

[0118]

[0119] Wherein, w t+1 These are the model parameters for the intermediate model.

[0120] In some embodiments, see Figure 4 , Figure 4 for Figure 3 The flowchart of the sub-steps in step S330 is as follows: Figure 4 As shown, step S330 includes, but is not limited to, steps S410 to S430.

[0121] Step S410: Obtain the dynamic second weighting coefficient.

[0122] Step S420: Based on the second weight coefficient, perform a weighted average of the model parameters of the first federated learning model and the model parameters of the intermediate model to obtain the second federated learning model.

[0123] Step S430: Test the second federated learning model. If the test result of the second federated learning model does not meet the preset integration termination condition, return to the step of obtaining the dynamic second weight coefficient until the test result of the second federated learning model meets the integration termination condition.

[0124] Understandably, by obtaining dynamic second weight coefficients, the proportion of the intermediate model learning the new task in the second federated learning model is adjusted, and the integrated second federated learning model is tested. If the test result does not meet the preset integration termination condition, the process returns to obtaining dynamic second weight coefficients until the test result meets the integration termination condition. In other words, the second weight coefficients are continuously adjusted until the second federated learning model obtained by integrating the first federated learning model and the intermediate model based on the second weight coefficients meets the integration termination condition.

[0125] It should be noted that the integration can end when the test results of this round of integration are better than those of the previous round, or when the test results reach a preset threshold.

[0126] In some embodiments, the model parameters of the first federated learning model and the model parameters of the intermediate model are weighted and averaged using the following formula:

[0127]

[0128] Wherein, r2 is the model parameter of the second federated learning model, α is the second weight coefficient, and α∈[0,1], w t+1 These are the model parameters for the intermediate model.

[0129] In some embodiments, prior to testing the second federated learning model, the method further includes:

[0130] Obtain the test sample dataset.

[0131] Please see Figure 5 , Figure 5 The diagram illustrates a flowchart of a federated learning method based on continuous learning, as provided in an embodiment of this application. Figure 5 As shown, the testing of the second federated learning model includes, but is not limited to, steps S510 to S520.

[0132] Step S510: Input the test sample dataset into the second federated learning model to obtain the third classification prediction value corresponding to each test sample data in the test sample dataset through the second federated learning model.

[0133] Step S520: Determine the true classification label of each test sample data in the test sample dataset, and determine and record the evaluation index of the second federated learning model based on the third classification prediction value and the true classification label corresponding to the test sample data.

[0134] The evaluation metrics include at least one of the following: Dessian similarity coefficient, crossover ratio, or accuracy.

[0135] Understandably, the test sample dataset can be obtained by the server collecting publicly available, non-privacy sample datasets. Specifically, the test sample dataset contains sample data that matches the task types of the first and second sample datasets. Therefore, the second federated learning model can be tested on both new and old tasks using this test sample dataset to determine the performance of the integrated second federated learning model on both tasks.

[0136] It should be understood that the server can also obtain test sample datasets by receiving test sample datasets uploaded by the client. Similarly, the test sample dataset also contains sample data that matches the task types corresponding to the first and second sample datasets.

[0137] Understandably, in each round of integration, the evaluation metric of the second federated learning model is determined and recorded. If the evaluation metric of the second federated learning model obtained in this round is better than that of the previous round, the process returns to the step of obtaining the dynamic second weight coefficient. The second weight coefficient is then adjusted to perform a weighted summation of the model parameters of the first federated learning model and the intermediate model until the evaluation metric of the second federated learning model obtained in this round is lower than that of the previous round. At this point, the integration stops, and the model with the best evaluation metric is taken as the final second federated learning model.

[0138] This application proposes a federated learning method based on continuous learning. The method is applied to a federated learning system, which includes a server and multiple clients. The server communicates with each client. Both the server and the clients store a first federated learning model trained on a first sample dataset. The method obtains a second sample dataset from the client, which is used for a different learning task than the first sample dataset for the first federated learning model. Multiple sample data are extracted from the first sample dataset as auxiliary sample data. The client uses the first federated learning model as a local model, and the training objective is to minimize the loss incurred by the local model when learning the second sample dataset and the multiple auxiliary sample data. The local model is trained based on the second sample dataset and the multiple auxiliary sample data. Then, the server receives the trained local models uploaded by the multiple clients and integrates the first federated learning model and the multiple local models to obtain a second federated learning model. This embodiment incorporates the loss incurred when learning multiple old samples to correct the model gradient while the local model learns a new task, thereby protecting the model's learned old knowledge and mitigating catastrophic forgetting in federated learning.

[0139] Please see Figure 7 This application also provides a federated learning system 100 based on continuous learning. The federated learning system 100 includes a server 120 and multiple clients 110. The server 120 is communicatively connected to each of the multiple clients 110. Both the server 120 and the clients 110 store a first federated learning model trained based on a first sample dataset.

[0140] The client 110 includes:

[0141] The acquisition module 111 is used to acquire a second sample dataset, wherein the first sample dataset and the second sample dataset are used for different learning tasks of the first federated learning model.

[0142] Extraction module 112 is used to extract multiple sample data from the first sample dataset as auxiliary sample data.

[0143] The training module 113 is used to train the local model based on the second sample dataset and the multiple auxiliary sample data, with the goal of minimizing the loss generated by the local model when learning the second sample dataset and the multiple auxiliary sample data.

[0144] The sending module 114 is used to upload the trained local model to the server.

[0145] The server 120 includes:

[0146] The receiving module 121 is used to receive the local model uploaded by multiple clients.

[0147] The integration module 122 is used to integrate the first federated learning model and the multiple local models to obtain a second federated learning model.

[0148] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment of this application. For details on their specific functions and technical effects, please refer to the method embodiment section, and they will not be repeated here.

[0149] Please see Figure 8 , Figure 8 This application illustrates the hardware structure of an electronic device according to an embodiment of the present application. The electronic device includes:

[0150] The processor 210 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant computer programs to implement the technical solutions provided in the embodiments of this application.

[0151] The memory 220 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 220 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 210 to implement the federated learning method based on continuous learning in the embodiments of this application.

[0152] Input / output interface 230 is used to implement information input and output;

[0153] The communication interface 240 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 250 is used to transmit information between each component of the device (such as processor 210, memory 220, input / output interface 230 and communication interface 240).

[0154] The processor 210, memory 220, input / output interface 230 and communication interface 240 are connected to each other within the device via bus 250.

[0155] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the above-described federated learning method based on continuous learning.

[0156] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0157] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0161] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0162] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0164] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Furthermore, the functional units in each embodiment of this application can be integrated into a single processing unit, or each unit can exist independently, or two or more units can be integrated into a single unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0166] If the assembled units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of each embodiment of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A federated learning method based on continuous learning, characterized in that, The method is applied to a federated learning system, the system comprising a server, a plurality of clients, the server being in communication connection with the plurality of clients respectively, the server and the clients each storing a medical image segmentation model trained based on a first sample data set, the first sample data set being organ image samples, the method comprising: obtaining, by the clients, a second sample data set, the second sample data set and the first sample data set being respectively used for different learning tasks of the medical image segmentation model, the second sample data set being cell image samples; obtaining, by the clients, a predicted difficulty of each sample data in the first sample data set, and sorting each sample data in the first sample data set in descending order of the predicted difficulty, and extracting a plurality of sample data from the first sample data set as auxiliary sample data according to the sorting; inputting, by the clients, the plurality of auxiliary sample data into the medical image segmentation model to obtain first classification prediction values corresponding to the plurality of auxiliary sample data; inputting, by the clients, the medical image segmentation model as a local model, the second sample data set and the plurality of auxiliary sample data into the local model to obtain second classification prediction values corresponding to each sample data in the second sample data set and the plurality of auxiliary sample data; determining real classification labels of each sample data in the second sample data set, taking minimization of differences between the second classification prediction values and the real classification labels of each sample data in the second sample data set as a training target to determine a first loss function; taking minimization of differences between the first classification prediction values and the second classification prediction values of each auxiliary sample data as a training target to determine a second loss function; training the local model based on the first loss function and the second loss function; uploading, by the clients, the trained local model to the server; receiving, by the server, the local models uploaded by the plurality of clients, integrating the medical image segmentation model and the plurality of local models to obtain a second federated learning model.

2. The federated learning method based on continual learning according to claim 1, wherein, The first loss function is determined by the following formula: Among them, the For the first loss function, the This represents the number of categories in the second sample dataset. For a one-hot encoded vector, if the true classification label of the sample data is consistent with the classification... If they are the same, then Select 1 if the value is 1, otherwise select 0. The sample data represents a classification. The second category prediction value. 3.The federated learning method based on continual learning according to claim 1, wherein, The second loss function is determined by the following formula: Wherein, the is a second loss function, the is a sample quantity of the auxiliary sample data, the is the second classification prediction value, is the first classification prediction value. 4.The federated learning method based on continual learning of claim 1, wherein, Before the integrating the medical image segmentation model and the plurality of local models to obtain the second federated learning model, the method further comprises: obtaining a sample quantity of the second sample data set in each client; the integrating the medical image segmentation model and the plurality of local models to obtain the second federated learning model comprises: determining a first weight coefficient of each local model according to the sample quantity of the second sample data set in each client; performing weighted average processing on model parameters of the plurality of local models according to the first weight coefficient of each local model to obtain an intermediate model; performing weighted average processing on model parameters of the medical image segmentation model and the intermediate model to obtain the second federated learning model.

5. The federated learning method based on continual learning according to claim 4, c h a r a c t e r i z e d b y, The model parameters of the medical image segmentation model and the model parameters of the intermediate model are weighted and averaged to obtain a second federated learning model, including: obtaining a dynamic second weight coefficient; According to the second weight coefficient, the model parameters of the medical image segmentation model and the model parameters of the intermediate model are weighted and averaged to obtain a second federated learning model; Before testing the second federated learning model, the method further comprises:

6. The federated learning method based on continual learning according to claim 5, c h a r a c t e r i z e d b y obtaining a test sample data set; The test of the second federated learning model comprises: inputting the test sample data set into the second federated learning model to obtain the third classification prediction value corresponding to each test sample data in the test sample data set through the second federated learning model; Determine the true classification label of each test sample data in the test sample data set, and determine and record the evaluation index of the second federated learning model according to the third classification prediction value and the true classification label of the test sample data. Wherein, the evaluation index includes at least one of the following: Des similarity coefficient, intersection over union coefficient or accuracy. The system comprises a server, a plurality of clients, the server is respectively connected with a plurality of clients in communication, the server and the client are all stored with the medical image segmentation model trained based on the first sample data set, the first sample data set is organ image sample;Wherein, 7. A federated learning system based on continual learning, characterized in that, The client comprises: an acquisition module for acquiring a second sample data set, the first sample data set and the second sample data set are respectively used for different learning tasks of the medical image segmentation model, and the second sample data set is a cell image sample; The client is further used for: obtaining the predicted difficulty of each sample data in the first sample data set, and sorting each sample data in the first sample data set in descending order of the predicted difficulty; an extraction module for extracting a plurality of sample data from the first sample data set as auxiliary sample data according to the sorting; inputting a plurality of the auxiliary sample data into the medical image segmentation model to obtain a plurality of first classification prediction values corresponding to the auxiliary sample data; The client is further used for: inputting a plurality of the auxiliary sample data into the medical image segmentation model to obtain a plurality of first classification prediction values corresponding to the auxiliary sample data through the medical image segmentation model; ​ The training module is configured to input the medical image segmentation model as a local model, the second sample data set, and a plurality of auxiliary sample data into the local model, to obtain second classification prediction values corresponding to each sample data in the second sample data set and the plurality of auxiliary sample data by the local model, to determine true classification labels of each sample data in the second sample data set, to determine a first loss function as a training target for minimizing differences between the second classification prediction values and the true classification labels of each sample data in the second sample data set, to determine a second loss function as a training target for minimizing differences between first classification prediction values and second classification prediction values of each auxiliary sample data, and to train the local model based on the first loss function and the second loss function. The sending module is configured to upload the trained local model to the server. The server comprises: The receiving module is configured to receive the local models uploaded by the plurality of clients. The integration module is configured to perform integration processing on the medical image segmentation model and the plurality of local models to obtain a second federated learning model.

8. An electronic device, comprising: comprise: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program, and the computer program is executed by the at least one processor to enable the at least one processor to execute the federated learning method based on continuous learning according to any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the federated learning method based on continuous learning according to any one of claims 1 to 6.

Images