Lung cancer auxiliary diagnosis method based on federated learning and knowledge distillation

Abstract

The invention discloses a lung cancer auxiliary diagnosis method based on federal learning and knowledge distillation, and the method comprises the following steps: S1, reading and preprocessing data from each medical institution, and extracting key features to construct a local database; s2, each mechanism replaces original data with an encrypted transmission model prediction result, and the communication efficiency and privacy protection of federated learning are optimized; s3, aggregating the global model and evaluating the performance; s4, iteratively optimizing the model according to feedback and supporting a dynamic expansion collaborative network, wherein the whole process takes distributed collaboration, data security and lightweight deployment into account; the problems of communication efficiency optimization and data isomerism challenges in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the field of lung cancer auxiliary diagnosis, and specifically relates to a lung cancer auxiliary diagnosis method based on federated learning and knowledge distillation. Background Art

[0002] 1. High communication overhead. In traditional federated learning methods, each participating institution needs to frequently upload complete local model parameters (such as neural network weights, gradients, etc.) to the central server for aggregation. Since the model parameter dimensions are relatively high (especially for deep learning models), the amount of data to be transmitted for each update is huge.

[0003] 2. Data heterogeneity affects model stability. Data distribution differences (Non-IID): The characteristics of patient groups in different medical institutions (such as age, gender, regional disease spectrum) and detection standards (such as instrument accuracy, operation specifications) are different, resulting in significant differences in the label distribution (such as the positive rate of lung cancer) and feature distribution (such as the normal range of coagulation indicators) of local data. There may be differences in the data dimensions or feature definitions collected by each institution. For example, some hospitals may not detect specific coagulation indicators (such as D-dimer), resulting in missing or redundant features.

[0004] 3. Insufficient cross-institutional collaboration, data island effect: Each medical institution independently trains a local model, lacking an effective data sharing or knowledge interaction mechanism, resulting in the model only being able to reflect the characteristics of local data. Example: Due to the small number of patients in Hospital C, the local model is prone to overfitting; although Hospital D has a large amount of data, it lacks the diverse features of other institutions, and the generalization ability of the model is limited. There may be differences in the inducing factors of lung cancer in different regions (such as environmental pollution, genetic factors), and the model of a single institution cannot capture global features, resulting in a decrease in the accuracy of cross-regional diagnosis.

[0005] The present invention proposes a lung cancer auxiliary diagnosis method based on federated learning and knowledge distillation technology, aiming to solve the following core problems: Optimization of communication efficiency problem: Only exchange the encrypted model prediction results (logits), rather than complete parameters or raw data, significantly reducing the communication overhead.

[0006] Challenge problem of data heterogeneity: Introduce an L2 regularization term based on local logits and global logits to coordinate the model consistency between different institutions and improve the generalization ability of the model. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a lung cancer auxiliary diagnosis method based on federated learning and knowledge distillation, which solves the problems of optimizing communication efficiency and challenging data heterogeneity existing in the prior art.

[0008] The technical solution of the present invention is as follows:

[0009] A lung cancer assisted diagnosis method based on federated learning and knowledge distillation includes the following steps:

[0010] Step S1: After the data is read from each medical institution and preprocessed, key features are extracted to construct a local database;

[0011] Step S2: Each institution optimizes the communication efficiency and privacy protection of federated learning by encrypting and transmitting the model prediction results to replace the original data;

[0012] Step S3: Aggregate the global model and evaluate its performance;

[0013] Step S4: Iteratively optimize the model according to the feedback and support the dynamic expansion of the collaborative network. The overall process takes into account distributed collaboration, data security, and lightweight deployment.

[0014] Preferably, each institution in Step S2 encrypts and transmits the model prediction results, and the results are logits.

[0015] Preferably, Step S2 includes the following sub-steps:

[0016] Sub-step S21: The local model uploads the encrypted logits to the central server;

[0017] Sub-step S22: The server aggregates the logits from each institution to generate global logits;

[0018] Sub-step S23: In the loss function, the Kullback-Leibler divergence loss between the local soft prediction and the global logits, as well as the L2 regularization term, are calculated;

[0019] Sub-step S24: Combine the classification loss of the hard prediction and the regularization term, calculate the total loss and optimize it, thereby completing the model training and update.

[0020] Preferably, sub-step S23 is used to optimize the consistency between the local and global models, while reducing the impact brought by data heterogeneity.

[0021] Preferably, the loss function of sub-step S23 is:

[0022]

[0023] where N represents the sample size of the dataset, C represents the total number of categories, represents the true label of the nth sample, represents that the model predicts that the nth sample belongs to the c category, T represents the total number of features, S represents the number of steps of dynamic routing, K represents the number of attention heads, λ represents the regularization strength, and a ts,k represents the weight of feature t in the s-th routing step and the k-th attention head.

[0024] The beneficial effects of the lung cancer assisted diagnosis method based on federated learning and knowledge distillation of the present invention are as follows:

[0025] 1. The present invention proposes an innovative federated learning framework, which combines knowledge distillation technology and introduces an L2 regularization term based on local logits and global logits, effectively solving the problems of data heterogeneity and communication overhead in the multi-center application of federated learning, and significantly improving the accuracy of the lung cancer diagnosis model. In traditional federated learning, there are differences in the sources and distributions of local data between different medical institutions, which may affect the global model during aggregation, reducing the stability and accuracy of the model. In addition, the communication overhead caused by frequent data transmission and model aggregation is also a limiting factor. To overcome these problems, the present invention innovatively combines knowledge distillation with the federated learning framework. By transferring the knowledge learned by the teacher model from the local data of multiple medical institutions to the student model, the communication overhead is effectively reduced, and at the same time, the adaptability and accuracy of the global model between different institutions are improved.

[0026] 2. To further solve the problem of data heterogeneity, the present invention introduces an L2 regularization term based on local logits and global logits. Local logits represent the outputs obtained by each medical institution through training on local data, while global logits are the outputs obtained by aggregating the model parameters of each institution through the central server. By adding an L2 regularization term to the loss function, the present invention can reduce the difference between the local and global models during the training process, reduce the negative impact of data heterogeneity on the model performance, and improve the stability and adaptability of the global model.

[0027] 3. The experimental results of the present invention show that through this innovative combination of technologies, the accuracy of the lung cancer diagnosis model is improved from 0.8108 to 0.8234, and without increasing the communication overhead, the accuracy of the model is significantly improved. This improvement shows that the solution proposed by the present invention can achieve high accuracy in the field of clinical diagnosis while ensuring high efficiency, providing more reliable technical support for the early diagnosis of lung cancer. Description of the Drawings

[0028] To more clearly elaborate the purpose, design idea and innovation of the multi-source remote sensing cross-domain classification method based on the hierarchical mask adversarial network proposed by the present invention, the present invention will be described in detail below in conjunction with the drawings and tables.

[0029] Figure 1 It is the flow chart of the lung cancer assisted diagnosis system based on federated learning of the present invention.

[0030] Figure 2Flowchart for training a lung cancer auxiliary diagnosis model based on federated learning and knowledge distillation according to the present invention.

[0031] Figure 3 Flowchart of the specific algorithm according to the present invention. Detailed implementation manners

[0032] The present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0033] The following describes the specific implementation manners of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation manners. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

[0034] A lung cancer auxiliary diagnosis method based on federated learning and knowledge distillation includes the following steps:

[0035] Step S1: After the data is read from each medical institution and preprocessed, key features are extracted to construct a local database;

[0036] Step S2: Each institution optimizes the communication efficiency and privacy protection of federated learning by encrypting and transmitting the model prediction results to replace the original data;

[0037] Step S3: Aggregate the global model and evaluate its performance;

[0038] Step S4: Iteratively optimize the model according to the feedback and support dynamic expansion of the collaborative network. The overall process takes into account distributed collaboration, data security, and lightweight deployment.

[0039] In step S2 of this implementation manner, each institution encrypts and transmits the model prediction results, and the results are logits.

[0040] Step S2 of this implementation manner includes the following sub-steps:

[0041] Sub-step S21: The local model uploads the encrypted logits to the central server;

[0042] Sub-step S22: The server aggregates the logits from each institution to generate global logits;

[0043] Sub-step S23: In the loss function, the Kullback-Leibler divergence loss and the L2 regularization term between the local soft prediction and the global logits are calculated;

[0044] Sub-step S24: Combine the classification loss of the hard prediction and the regularization term, calculate the total loss and optimize it, thereby completing the model training and update.

[0045] Sub-step S23 of this implementation scheme is used to optimize the consistency between the local and global models while reducing the impact brought by data heterogeneity.

[0046] The loss function of sub-step S23 of this implementation scheme is:

[0047]

[0048] where N represents the sample size of the dataset, C represents the total number of categories, represents the true label of the nth sample, represents that the model predicts that the nth sample belongs to the c category, T represents the total number of features, S represents the number of steps of dynamic routing, K represents the number of attention heads, λ represents the regularization strength, and a ts,k represents the weight of feature t in the s-th routing step and the k-th attention head.

[0049] When implementing this implementation scheme, the present invention proposes a lung cancer assisted diagnosis method based on federated learning, which combines technologies such as data preprocessing, feature extraction, federated learning framework, and knowledge distillation, and improves the accuracy of early lung cancer diagnosis and the generalization ability of the model through cross-institutional collaborative training. Compared with the prior art, this patent has achieved important innovations and improvements in many aspects, especially in improving the diagnosis efficiency, reducing the communication overhead, and enhancing the model stability.

[0050] First, different from the centralized data training in the prior art, the present invention adopts a federated learning framework. Through local training and parameter aggregation of the central server, it avoids the centralized transmission of raw data, reduces the risk of data privacy leakage, and effectively reduces the communication overhead. The prior art usually relies on frequent global model parameter exchanges, especially in a multi-center medical data environment, where the communication cost is huge. This patent significantly reduces the communication overhead by only exchanging the encrypted logits output instead of the raw data or the complete model parameters, which has significant advantages in the processing and training of large-scale medical data.

[0051] In terms of feature extraction and model training, the prior art often lacks cross-institutional diversified data sharing, resulting in the model training relying on local data from a single source, and thus unable to fully reflect the characteristics of different regions and different patient groups. The present invention screens out features highly relevant to lung cancer diagnosis through a recursive feature elimination method and constructs a multi-center unified database with these features, ensuring the high quality and standardization of the data. At the same time, the optimization of the feature selection process ensures the extraction of key information in the diagnosis process and effectively reduces the negative impact of redundant features on the model performance.

[0052] In terms of federated learning optimization, this patent further combines local logits and global logits through an L2 regularization term to solve the problems brought about by data heterogeneity. Traditional federated learning methods often face challenges such as uneven data distribution and inconsistent features, resulting in difficulties for the global model to converge stably and poor performance. Through regularization adjustment, the model training of each medical institution can be better coordinated in this invention, thereby improving the accuracy and robustness of the global model.

[0053] In addition, this invention also pays attention to the persistence of model evaluation and optimization. In model evaluation, multiple indicators such as accuracy, recall, specificity, and F1 score are used to comprehensively evaluate the model performance. At the same time, the confusion matrix, ROC curve, and AUC value are combined to visually display the performance of the model on various types of data. Compared with the traditional single accuracy evaluation method, this multi-dimensional evaluation can more comprehensively reflect the application effect of the model in actual diagnosis. By continuously optimizing the model algorithm and expanding the multi-center database, this invention can dynamically improve the applicability and generalization ability of the model as the data scale increases, further enhancing the accuracy of early lung cancer diagnosis.

[0054] In summary, the innovation of this patent lies in combining federated learning with knowledge distillation, optimizing communication overhead, and refined feature selection, etc., overcoming the deficiencies of traditional methods in aspects such as communication overhead, data heterogeneity, and model generalization ability, providing a more efficient and accurate lung cancer auxiliary diagnosis solution, with strong clinical application potential.

[0055] Figure 1 Shows a process of a lung cancer auxiliary diagnosis system based on federated learning: After data is read from each medical institution and preprocessed, key features are extracted to construct a local database; each institution optimizes the communication efficiency and privacy protection of federated learning by encrypting and transmitting the model prediction results (logits) instead of the original data; then the global model is aggregated and its performance (such as sensitivity, specificity) is evaluated, and finally the model is iteratively optimized according to the feedback and supports dynamic expansion of the collaborative network. The overall process takes into account distributed collaboration, data security, and lightweight deployment.

[0056] Figure 2Shows a training process of a lung cancer auxiliary diagnosis model based on federated learning and knowledge distillation. Each participating institution uses local data to train the model and calculates soft predictions and hard predictions. The local model uploads the encrypted logits to the central server, and the server aggregates the logits from each institution to generate global logits. In the loss function, the Kullback-Leibler (KL) divergence loss between the local soft predictions and the global logits, as well as the L2 regularization term, are calculated to optimize the consistency between the local and global models and reduce the impact of data heterogeneity. Finally, by combining the classification loss of the hard predictions and the regularization term, the total loss is calculated and optimized to complete the model training and update. This method effectively reduces the communication overhead and improves the model performance and stability in cross-institutional training.

[0057] Figure 3 Shows the training process of the algorithm: starting from model initialization, dividing the dataset and then training, jointly optimizing the parameters by calculating the local loss and the distillation loss. If the model does not converge, the aggregated model output is fed back to the client for retraining; if it converges, the loop is terminated, and finally a stable model is obtained. The process is connected by concise steps to achieve efficient iteration and is suitable for model optimization in distributed or federated learning scenarios.

[0058] Example 1

[0059] A method for training a lung cancer auxiliary diagnosis model based on federated learning includes the following steps:

[0060] (a) Under the federated learning framework, each participating institution initializes and trains the model using local medical data. The local model selects TabNet, a neural network model for tabular data;

[0061] (b) Each institution encrypts the prediction results (logits) of the local model TabNet and transmits them to the central server;

[0062] (c) The central server aggregates the encrypted logits of all institutions and obtains the global logits by weighted averaging;

[0063] (d) Distribute the global logits to each institution and update the local TabNet model in combination with the local data;

[0064] (e) When performing local training, in the local loss function:

[0065]

[0066] where N represents the sample size of the dataset and C represents the total number of classes,

[0067] Add an L2 regularization term and KL divergence based on local logits and global logits to reduce the difference between the two;

[0068] (f) Repeat steps (a)-(e) until each local model TabNet converges.

[0069] The expression of the L2 regularization term is:

[0070]

[0071] where λ represents the regularization strength, generally taken as 0.01, represents the output of the local model, F global (w t ) represents the result after the server takes the weighted average of all client outputs.

[0072] The Chinese expression of the regularization expression: The regularization coefficient multiplies the L2 norm of the difference between the local model output and the global output.

Claims

1. A lung cancer auxiliary diagnosis method based on federated learning and knowledge distillation, characterized in that It includes the following steps: Step S1: After data is read from each medical institution and preprocessed, key features are extracted to construct a local database; Step S2: Each institution replaces the original data with the predicted results of the encrypted transmission model to optimize the communication efficiency and privacy protection of federated learning; Step S3: Aggregate the global model and evaluate its performance; Step S4: Iteratively optimize the model according to the feedback and support the dynamic expansion of the collaborative network. The overall process takes into account distributed collaboration, data security, and lightweight deployment.

2. The lung cancer assisted diagnosis method based on federated learning and knowledge distillation according to claim 1, wherein In Step S2, each institution transmits the predicted results of the encrypted transmission model, and the results are logits.

3. The lung cancer assisted diagnosis method based on federated learning and knowledge distillation according to claim 1, characterized in that, Step S2 includes the following sub-steps: Sub-step S21: The local model uploads the encrypted logits to the central server; Sub-step S22: The server aggregates the logits from each institution to generate global logits; Sub-step S23: In the loss function, the Kullback-Leibler divergence loss between the local soft prediction and the global logits, as well as the L2 regularization term, are calculated; Sub-step S24: Combine the classification loss of the hard prediction and the regularization term, calculate the total loss and optimize it, thereby completing the model training and update.

4. The lung cancer assisted diagnosis method based on federated learning and knowledge distillation according to claim 3, wherein, Sub-step S23 is used to optimize the consistency between the local and global models while reducing the impact of data heterogeneity.

5. The lung cancer assisted diagnosis method based on federated learning and knowledge distillation according to claim 3, wherein The loss function of Sub-step S23 is: Among them, N represents the sample size of the dataset, C represents the total number of categories, represents the true label of the nth sample, represents that the model predicts that the nth sample belongs to the c category, T represents the total number of features, S represents the number of steps of the dynamic routing, K represents the number of attention heads, λ represents the regularization strength, a ts,k represents the weight of feature t in the s-th routing step and the k-th attention head.

Images