A clustering-based dynamic federated optimization method

Abstract

The application provides a clustering-based dynamic federated optimization method, comprising: obtaining a to-be-detected gene; obtaining multiple center medical data; performing data preprocessing on the multiple center medical data to obtain processed multiple center medical data; inputting the multiple center medical data into a pre-constructed unified medical intelligence clustering algorithm to obtain a clustering grouping result; based on the clustering grouping result, federated learning and maximum fairness influence, training a shared large language model according to the processed multiple center medical data to obtain a trained shared large language model; determining a structural entropy corresponding to each medical center data according to the medical center data, and determining a noise intensity of each medical center data according to the structural entropy corresponding to each medical center data; and obtaining a target large language model according to the noise intensity of each medical center data and the model weight of the trained shared large language model.

Description

Technical Field

[0001] This application relates to the medical field, and more specifically, to a clustering-based dynamic federated optimization method. Background Technology

[0002] In the medical field, the integration and utilization of distributed, multi-center hospital data is of great significance. Hospitals accumulate massive amounts of patient data, encompassing a wealth of medical information such as disease diagnoses, treatment plans, and patient recovery status. Effective integration of this data will greatly advance medical research and improve the accuracy of disease diagnosis and the effectiveness of treatment plans. However, achieving this goal faces numerous challenges.

[0003] On the one hand, data security and privacy protection are paramount. Hospital data contains a wealth of sensitive patient information. A leak could cause serious harm to patients. Traditional centralized data processing methods are no longer suitable in this situation, as centralized storage and processing of data introduces higher privacy risks. For example, if a hospital centrally uploads patient data to a central server for analysis, the data may be vulnerable to attack and theft during transmission and storage.

[0004] On the other hand, data from different hospitals often exhibit non-independent and identically distributed characteristics. Due to differences in factors such as region, patient population, and medical technology level, the data from different hospitals vary significantly in terms of feature distribution and data scale. This data heterogeneity severely affects the training effect of the model. If traditional machine learning methods are directly used to uniformly model this data, it will result in poor generalization performance of the model, making it difficult to accurately adapt to the data characteristics of different hospitals. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a clustering-based dynamic federated optimization method. This method adjusts grouping in real time through clustering, grouping hospitals and their data with similar characteristics into the same group to reduce heterogeneity within each group. Within each group, a shared Large Language Model (LLM) is trained through federated learning, and the fairness maximization algorithm ensures fairness during training. Furthermore, personalized privacy noise is added when aggregating model weights to further protect data privacy. This effectively improves the training efficiency and accuracy of the model, providing an efficient and secure solution for the fusion and utilization of distributed multi-center hospital data.

[0006] This application provides a clustering-based dynamic federated optimization method, characterized in that the method includes:

[0007] Acquire medical data from multiple centers, each center's medical data including at least one of medical text data, structured data, and time series data;

[0008] The multiple central medical data are preprocessed to obtain the processed multiple central medical data.

[0009] The medical data from the multiple centers are input into a pre-constructed unified intelligent clustering algorithm to obtain clustering grouping results;

[0010] Based on clustering grouping results, federated learning, and maximizing fairness impact, the shared big oracle model is trained on the processed medical data from multiple centers to obtain the trained shared big language model.

[0011] Based on the data from each medical center, determine the structural entropy corresponding to that medical center's data, and based on the structural entropy corresponding to each medical center's data, determine the noise intensity of each medical center's data.

[0012] The target large language model is obtained based on the noise intensity of the data from each medical center and the model weights of the shared large language model after training.

[0013] Optionally, the medical data for each center can be preprocessed in the following ways:

[0014] The medical data of the center was deduplicated to obtain the first intermediate data;

[0015] The first intermediate data is corrected to obtain the second intermediate data;

[0016] The second intermediate data is standardized and normalized to obtain the processed medical data of the center.

[0017] Optionally, a unified clustering algorithm for intelligent systems can be constructed through the following steps:

[0018] For each center's medical data, multiple data points in the center's medical data are represented as a single-value intelligent set group, and each data point in the single-value intelligent set is represented as a single-value intelligent set;

[0019] Based on a predefined similarity measurement method, calculate the pairwise similarity value between every two data points, and determine the overall similarity value based on multiple pairwise similarities;

[0020] Construct a similarity matrix based on multiple pairwise similarity values;

[0021] The similarity matrix is ​​normalized to obtain a normalized similarity matrix;

[0022] The target matrix is ​​obtained by iterating through the normalized similarity matrix to obtain the target matrix that converges to the normalized similarity matrix. The target matrix is ​​an equivalent similarity matrix.

[0023] Based on a pre-selected confidence level, a cutting matrix is ​​constructed according to the target matrix;

[0024] The similarity metric is determined based on the number of clusters, cluster centroids, and membership values.

[0025] The clustering grouping results are obtained based on the similarity metric, the target matrix, and the cutting matrix.

[0026] Optionally, the unified intelligent clustering algorithm includes an unbiased estimation algorithm for the fair training objective function, a model optimization algorithm, and an RR set quantity calculation algorithm, wherein,

[0027] When 0 < α < 1, the following formula is used to calculate the unbiased estimate of the fair training objective function:

[0028] F α (S)=∑ c∈C n c u c (S) α ;

[0029] Among them, F α (S) is the objective function for fair training, and α is the inequality aversion parameter, which is applied to training groups c and n. c The number of clients is represented by u, and the proportion of clients effectively learned by the model is defined as u. c (S), where S is a set of various parameters and strategies involved in the current training;

[0030] When α = 0, the unbiased estimate of the fair training objective function is calculated using the following formula: F0(S) = ∑ c∈C n c ln(u c (S));

[0031] Optionally, the model optimization algorithm is used to calculate the first marginal fairness impact gain when the inequality aversion parameter is greater than 0 and less than 1, and to calculate the second marginal fairness impact gain when the inequality aversion parameter is equal to 0.

[0032] When 0 < α < 1, the first marginal fairness impact gain is calculated using the following formula:

[0033]

[0034] When α = 0, the second marginal fairness impact gain is calculated using the following formula:

[0035]

[0036] in, κ[v][c] represents the number of RR sets with root node u in training group c covered by S, reflecting the portion of training group c that has been effectively covered by the model under the current policy; κ[v][c] represents the number of RR sets with root node u in training group c that are covered by v but not by S, reflecting the potential of parameter v to provide additional coverage to training group c; θ c This represents the total number of RR sets in training group c.

[0037] Optionally, F is calculated through the following steps. α Number of RR sets in (S) or F0(S) scenarios:

[0038] Set parameters δ1∈(0,1) and ε1∈(0,1);

[0039] Calculate the number of RR sets using the RR set calculation formula;

[0040] The formula for calculating the RR set is:

[0041]

[0042] Where Q is an approximate parameter, b = max(u c (S * )), where s * It is the optimal solution to the problem of maximizing the fairness impact, and the value of b reflects the maximum proportion of clients being learned in each training group under the optimal solution.

[0043] Optionally, the structural entropy corresponding to the data for each medical center is determined through the following steps:

[0044] The data from this medical center is abstracted into a graph structure;

[0045] Calculate the probability vector of the stationary distribution of vertices in the graph structure;

[0046] Calculate the one-dimensional structure entropy based on the one-dimensional structure information formula;

[0047] Calculate the entropy of the two-dimensional structure based on the formula for two-dimensional structure information.

[0048] Optionally, the formula for the one-dimensional structural information is:

[0049]

[0050] Where H1(G) is the one-dimensional structural entropy, and is the probability vector describing the stationary distribution of vertices in graph G. Graph G is an assumed undirected connected graph G = (V, E) with n vertices and m edges. For each vertex i, d i Represents the degree of vertex i in G;

[0051] in,

[0052] The formula for the two-dimensional structural information is as follows:

[0053]

[0054] Where L is the partition The number of modules in |P l | is module P l The number of nodes in d i It is P l The degree of the i-th node, vol(P) l ) is module P l The volume is also the volume of module P. l The sum of the degrees of all nodes in P, e(P) l ) is module P l The number of edges with only one endpoint. For the undirected connected graph G = (V, E), the following assumptions are made. It is a partition of V, called P L For a module or a community.

[0055] Optionally, the noise intensity of the data for each medical center can be determined through the following steps:

[0056] The minimum value of structural entropy in different dimensions is defined as the structural information in different dimensions;

[0057] Based on the structural information under different dimensions, the noise intensity of the data for each medical center is determined.

[0058] Optionally, the steps to obtain the target large language model based on the noise intensity of the data from each medical center and the model weights of the trained shared large language model include:

[0059] The noise intensity of each medical center's data is added to the weight of the shared large language model of each medical center's data to obtain multiple target model weights after adding noise;

[0060] A weighted average aggregation algorithm is used to fuse the weights of multiple target models after adding noise to obtain the global model weights;

[0061] The target large language model is determined based on the global model weights.

[0062] The clustering-based dynamic federated optimization method provided in this application adjusts grouping in real time through clustering, grouping hospitals and their data with similar data characteristics into the same group to reduce heterogeneity within the group. Within each group, a shared Large Language Model (LLM) is trained through federated learning, and the fairness maximization algorithm is used to ensure the fairness of training. Personalized privacy noise is added when aggregating model weights to further protect data privacy. This effectively improves the training efficiency and accuracy of the model, providing an efficient and secure solution for the fusion and utilization of distributed multi-center hospital data.

[0063] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart illustrating a clustering-based dynamic federated optimization method provided in this application embodiment;

[0066] Figure 2 This is a schematic diagram of the framework of a cluster-based dynamic federated optimization system provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0068] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of medical technology.

[0069] Research has shown that the integration and utilization of distributed, multi-center hospital data is of great significance in the medical field. Hospitals accumulate massive amounts of patient data, encompassing a wealth of medical information such as disease diagnoses, treatment plans, and patient recovery status. Effective integration of this data would greatly advance medical research and improve the accuracy of disease diagnosis and the effectiveness of treatment plans. However, achieving this goal faces numerous challenges.

[0070] On the one hand, data security and privacy protection are paramount. Hospital data contains a wealth of sensitive patient information. A leak could cause serious harm to patients. Traditional centralized data processing methods are no longer suitable in this situation, as centralized storage and processing of data introduces higher privacy risks. For example, if a hospital centrally uploads patient data to a central server for analysis, the data may be vulnerable to attack and theft during transmission and storage.

[0071] On the other hand, data from different hospitals often exhibit non-independent and identically distributed characteristics. Due to differences in factors such as region, patient population, and medical technology level, the data from different hospitals vary significantly in terms of feature distribution and data scale. This data heterogeneity severely affects the training effect of the model. If traditional machine learning methods are directly used to uniformly model this data, it will result in poor generalization performance of the model, making it difficult to accurately adapt to the data characteristics of different hospitals.

[0072] Based on this, this application provides a clustering-based dynamic federated optimization method. By adjusting grouping in real time through clustering, hospitals and their data with similar characteristics are grouped together, reducing heterogeneity within each group. Within each group, a shared Large Language Model (LLM) is trained through federated learning, and the fairness maximization algorithm ensures training fairness. Furthermore, personalized privacy noise is added when aggregating model weights to further protect data privacy. This effectively improves the training efficiency and accuracy of the model, providing an efficient and secure solution for the fusion and utilization of distributed multi-center hospital data.

[0073] It should be noted here that federated learning is an emerging artificial intelligence technology that aims to solve the problem of training machine learning models when data is not centralized or shared.

[0074] Suppose there are n participants, and each participant i has a local dataset D. i The goal is to jointly train a global model f(θ), where θ represents the model's parameters. In traditional centralized learning, all datasets are centralized in one place, and the model is trained by optimizing the objective function.

[0075] However, in federated learning, data is distributed among the various participants, making centralized training impossible. Federated learning optimizes the global model through iterative exchange of model parameters among the participants. Specifically, in each training round t, participant i first uses its own dataset D locally. i Train the model to obtain local model parameters. Then, each participant will assign local model parameters. The parameters are uploaded to a central server (or exchanged in a decentralized manner). The central server aggregates these parameters to obtain the global model parameters θ. t+1 .

[0076] Next, the updated global model parameters θ t+1 The data is distributed to each participant, who then uses the new global model parameters to continue local training in the next round. This process is repeated until the model converges. In this way, federated learning can collaboratively train a globally optimal model using data from all parties while protecting data privacy.

[0077] In terms of maximizing the fairness impact, for a given graph G, n G With nodes, the classic influence maximization problem involves selecting a seed set S, containing at most k seeds, to maximize influence propagation σ(S,G). Assume each node belongs to a disjoint set of communities. Where V1∪V2∪…∪V C =V,V c (n c =|V c |) represents the set of nodes belonging to community c. Generally, Fair Influence Maximization (FIM) aims to reduce the influence gap between different communities while maintaining overall influence propagation as constant as possible. For the concept of fairness, a welfare function is used to aggregate the cardinal utility of different communities. The goal is to select at most k seed nodes to maximize the objective function F. α (S) (also known as equity impact), where And F0(S) = When α = 0. Utility u c (S) represents the expected proportion of nodes in community c that are affected under the seed set S. The exponent α is the inequality aversion parameter, controlling the trade-off between fairness and overall impact. When α is close to 1, it favors the propagation of impact; when α is close to 0, it favors fairness. When α = 0, it becomes a unique case, in which all communities are required to be affected by S.

[0078] Please see Figure 1 , Figure 1This is a flowchart illustrating a clustering-based dynamic federated optimization method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the cluster-based dynamic federated optimization method includes:

[0079] S101. Obtain medical data from multiple centers.

[0080] Each center's medical data includes at least one of the following: medical text data, structured data, and time-series data.

[0081] S102. Perform data preprocessing on the multiple central medical data to obtain processed multiple central medical data.

[0082] S103. Input the multiple central medical data into the pre-constructed unified intelligent clustering algorithm to obtain the clustering grouping results.

[0083] S104. Based on clustering grouping results, federated learning, and maximizing fairness impact, the shared big oracle model is trained on the processed medical data from multiple centers to obtain the trained shared big language model.

[0084] S105. Based on the data from each medical center, determine the structural entropy corresponding to that medical center's data, and based on the structural entropy corresponding to each medical center's data, determine the noise intensity of each medical center's data.

[0085] S106. Based on the noise intensity of the data from each medical center and the model weights of the trained shared large language model, the target large language model is obtained.

[0086] based on Figure 1 Please refer to the schematic diagram of the resulting cluster-based dynamic federated optimization system. Figure 2 .

[0087] In step S101, data acquisition is the foundation for building a cluster-based dynamic federated optimization model, and its quality and breadth play a decisive role in the accuracy and applicability of the model.

[0088] The data sources for data collection include: regional medical data sharing platforms, hospital information systems (HIS), research cooperation project data, etc.

[0089] Among them, regional medical data sharing platforms: Under the trend of regional medical collaborative development, these platforms integrate data from multiple hospitals within a region. Their data is broadly representative, showcasing the differences and commonalities in disease diagnosis and treatment methods among different hospitals. For example, for the same disease, different hospitals may adopt different treatment plans and drug choices. Comparing these data can provide a reference for optimizing treatment plans and also provide richer samples for model training.

[0090] Hospital Information System (HIS): A hospital's information system stores massive amounts of data closely related to patients. Medical records cover a patient's detailed medical history, including symptom descriptions, diagnoses, treatments, and medication details for each visit. These records allow for a precise understanding of the progression of a patient's disease, such as fluctuations in the condition of patients with chronic diseases, providing crucial clues for analyzing disease characteristics and treatment needs.

[0091] Research collaboration project data: Numerous research collaboration projects involving medical institutions focus on specific diseases or medical fields. The data collected by these projects undergoes rigorous screening and organization, possessing high research value. For example, research projects targeting rare diseases record detailed patient genetic testing results, unique symptom manifestations, and long-term follow-up data.

[0092] Specifically, each center's medical data includes at least one of the following: medical text data, structured data, and time-series data.

[0093] Text data primarily includes patient consultations, medical records, and diagnostic reports. Text data is the core data for needs assessment; analyzing the text reveals the types and specific content of patient needs.

[0094] Structured data includes patient basic information (age, gender, medical history, etc.) and vital signs data (body temperature, blood pressure, heart rate, etc.). Structured data can supplement textual data, helping to understand the patient's needs more comprehensively.

[0095] Time series data: If data from multiple patient consultations or long-term health monitoring is available, time series data can reflect the changing trends and periodicity of patient needs. For example, the symptoms of some patients with chronic diseases may worsen during specific time periods; by analyzing time series data, patient needs can be predicted in advance.

[0096] Optionally, the medical data of each center can be preprocessed in the following way: the medical data of the center is deduplicated to obtain the first intermediate data; the first intermediate data is corrected to obtain the second intermediate data; the second intermediate data is standardized and normalized to obtain the processed medical data of the center.

[0097] The process of deduplicating the center's medical data, also known as medical data cleaning, involves first importing the original dataset for medical data collected from multiple sources. Duplicate case records are not uncommon in medical data; for example, the same patient's multiple identical test reports may be entered due to system malfunctions or human error. This duplicate data unnecessarily increases the computational burden of model training and can easily lead to overfitting, reducing the model's generalization ability. By using efficient deduplication techniques such as hash algorithms, a comprehensive comparison of the data is performed to ensure that each patient's record appears only once in the dataset, thereby optimizing the model training samples.

[0098] Medical data often contains non-standard expressions, such as inconsistent abbreviations of disease names and typos in drug names. By establishing a standardized medical terminology dictionary and employing string matching algorithms, medical terms in text data can be corrected. For numerical data, such as test indicator values, outliers may exist that exceed normal ranges. By setting reasonable threshold ranges, these outliers can be identified, corrected, or removed to prevent them from misleading model training.

[0099] Correcting the first intermediate data is also known as data standardization and normalization. Specifically, standardization and normalization are crucial when dealing with data from multiple centers of hospitals that have different dimensions and distribution characteristics. The cleaned medical dataset is input into the system, and standardization and normalization operations are performed accordingly.

[0100] For numerical medical data, such as patients' age, height, weight, and various test results, standardization is performed. Using the Z-score standardization method effectively eliminates scale differences between different features, preventing certain feature values ​​from dominating the model training direction and improving the stability and convergence speed of model training.

[0101] Normalization scales data feature values ​​to a specific range, such as the [0,1] interval. For some medical data, such as disease severity scores, the min-max normalization method is used. When different hospitals use different scoring systems for the same disease, normalization allows these data to be compared and analyzed on the same scale, preventing the influence of certain features on the model from becoming unbalanced and enhancing the model's adaptability to multi-source medical data.

[0102] The above data preprocessing steps can effectively improve data quality, significantly enhance model training efficiency and final performance, and lay a solid foundation for the subsequent construction, training, and accurate output of cluster-based dynamic federated optimization models.

[0103] In this way, the above data preprocessing steps can effectively improve data quality, significantly improve the training efficiency and final performance of the model, and lay a solid foundation for the subsequent construction, training and accurate output of the cluster-based dynamic federated optimization model.

[0104] Specifically, a unified intelligent clustering algorithm is constructed through the following steps: For each center's medical data, multiple data points in the center's medical data are represented as single-value intelligent sets, where each data point in a single-value intelligent set is represented as a single-value intelligent set; according to a predefined similarity metric, the pairwise similarity value between every two data points is calculated, and the overall similarity value is determined based on multiple pairwise similarities; a similarity matrix is ​​constructed based on multiple pairwise similarities; the similarity matrix is ​​normalized to obtain a normalized similarity matrix; the normalized similarity matrix is ​​iterated to obtain a target matrix converged by the normalized similarity matrix, where the target matrix is ​​an equivalent similarity matrix; a cutting matrix is ​​constructed based on a pre-selected confidence level value and the target matrix; a similarity metric is determined based on the number of clusters, cluster centroids, and membership values; and the clustering grouping results are obtained based on the similarity metric, the target matrix, and the cutting matrix.

[0105] Among them, the Unified Central Intelligence Clustering Algorithm (UNCA) is specifically designed for clustering analysis of complex datasets such as medical data. Leveraging the logic of Central Intelligence, it deeply analyzes the correlations between data points to achieve more accurate and detailed clustering. Its core process begins with inputting medical data, proceeding through steps such as calculating the similarity matrix, constructing the composite matrix, creating the λ-cutting matrix, initializing Central Intelligence k-means clustering, membership assignment and centroid update, integrating DVN-MST clustering, refining cluster assignments, and finally determining the clustering results. In this series of closely interconnected steps, each step lays the foundation for uncovering the underlying structure of the data and improving clustering quality. These steps collaborate to enable UNCA to effectively capture complex data relationships and provide accurate and robust clustering results.

[0106] As an example, the calculation of the similarity matrix specifically includes the following steps:

[0107] a) SVNS representation of data points: Representing data points as a single-valued set of intelligence (SVNS): Each data point x in the dataset i Represented as an SVNS, including its truth value T i Uncertainty I i And the degree of falsity F i ;

[0108] b) Definition of similarity metric: using

[0109]

[0110] Where Similarity(T) i ,T j ) etc. are used to measure x i and x jThe similarity of authenticity, uncertainty, and falsity;

[0111] c) Calculate pairwise similarity: for each pair of data points (x i ,x j The similarity S(x) is calculated using the defined metric. i ,x j This step involves assessing the similarity of each attribute (T, I, F) and combining these values ​​to obtain an overall similarity score;

[0112] d) Constructing the similarity matrix: Create an n×n similarity matrix C (where n is the number of data points), where each element C ij Represents data point x i and x j The similarity between them, element C ij Given by the following formula: C {ij} =S(x i ,x j );

[0113] The matrix is ​​symmetric because x i and x j The similarity between x j and x i The similarities between them are the same.

[0114] e) Normalized similarity matrix: Depending on the application requirements, the similarity matrix may need to be normalized to ensure that the values ​​are within a specific range, such as [0,1]. The resulting similarity matrix C captures the relationships between data points based on their intrinsic features, thereby facilitating the clustering process by providing a quantitative basis for comparing data points.

[0115] Optionally, this application also requires constructing a synthesis matrix, specifically including the following steps: iteratively constructing the synthesis matrix C,C 2 C 4 ... until matrix C {2(k+1)} Converging to C 2k First, ensure that C is an equivalent similarity matrix; then initialize the composition matrix: starting with the initial similarity matrix C obtained in the previous step. This matrix C serves as the basis for constructing higher-order composition matrices; finally, calculate the higher-order composition matrix: calculate the subsequent power matrix of C. Specifically, it is necessary to calculate C... 2 C 4 This is accomplished through matrix multiplication, such as C. 2 =C×C,C 4 =C 2 ×C 2 Continue this process to obtain C 2kWhere k is the iteration index; Check convergence: The iterative matrix exponentiation process continues until the synthesized matrix converges. Specifically, it is necessary to ensure that C 2(k+1) ≈C 2k Normalizing the final matrix: The final composite matrix is ​​normalized as needed to align the dimensions of the similarity values. This step is accomplished using standard normalization techniques.

[0116] In each iteration, a composite matrix is ​​constructed, which extracts more refined relationships than those represented in the original matrix. The final matrix will be a similarity matrix, summarizing the relationships between all data points for the next clustering task.

[0117] The cut matrix constructed from the target matrix is ​​also called the λ-cut matrix. The λ-cut matrix is ​​constructed based on the selected confidence level λ and is used as a threshold to determine the significant similarity between data points.

[0118] As an example, the λ-cut matrix can be determined using the following steps:

[0119] a) Choosing the confidence level λ: Define a threshold λ between 0 and 1 to determine which similarities are significant in the clustering. A higher λ will include more similarities, while a lower λ is more selective;

[0120] b) Applying the λ threshold: using the confidence level λ from the composite matrix C 2k Create a binary matrix. This involves thresholding the matrix, retaining only similarities above a selected confidence level. Specifically: for matrix C... 2k Each element in Compare it with λ. Construct the λ-cut matrix C. λ ,in:

[0121]

[0122] This will generate a binary matrix C λ , where each element represents whether the similarity between data i and j is significant.

[0123] c) Verify matrix properties: Check if the λ-cut matrix is ​​a valid clustering matrix. The output should preserve basic similarity relationships and allow for the elimination of weak links. Ensure the matrix is ​​not broken and is suitable for the next clustering operation.

[0124] If this λ-cutting matrix is ​​too sparse or too dense, the confidence level λ is adjusted. The relationships between data points are then fine-tuned based on λ, emphasizing the most relevant clustering relationships to prepare for the final clustering step.

[0125] The step of obtaining clustering grouping results based on the similarity measure, the target matrix, and the cutting matrix includes cluster assignment refinement and final determination. The cluster assignment can be refined by utilizing the similarity measure of Single Valued Intelligence Sets (SVNS) and combining the equivalent similarity matrix and the λ-cutting matrix.

[0126] Specifically, the steps in refining cluster assignment include:

[0127] a) Utilizing SVNS similarity metrics:

[0128] Reassess the similarity metric: Based on the SVNS similarity metric calculated in the previous steps, reassess the similarity between data points and clusters. SVNS provides a nuanced similarity metric that combines true values, uncertainty, and false positives, showing the relationship between data points and each cluster from a more detailed perspective.

[0129] Update membership: Refine membership assignment by combining SVNS similarity metric, and recalculate the membership of each data point to each cluster using the updated similarity value, such as T. {ij} It is an updated similarity metric based on SVNS, I {ij} It is a refined measure of uncertainty, F {ij} =1-T {ij} .

[0130] b) Implement an equivalent similarity matrix:

[0131] Constructing an equivalence matrix: Construct an equivalence similarity matrix C using a refined similarity metric. This matrix should reflect the updated clustering relationships based on the latest similarity values.

[0132] Matrix consistency: Ensure that the equivalent similarity matrix is ​​consistent with the previously constructed matrix. It should accurately represent the updated similarity relationships between clusters.

[0133] c) Applying the λ-cutting matrix:

[0134] Constructing the λ-cut matrix: Create a λ-cut matrix based on the selected confidence level λ. This matrix helps define the threshold for membership assignment, filtering out less important relationships.

[0135] Integration with the similarity matrix: The λ-cut matrix is ​​applied to the equivalent similarity matrix. This process involves adjusting the similarity values ​​according to the confidence level, thereby further refining the cluster assignments.

[0136] Thresholding: A λ-cutting matrix is ​​applied to threshold the similarity values. For a given confidence level λ, the similarity measure is filtered.

[0137]

[0138] Update clusters: Update cluster assignments using the filtered similarity values. This ensures that only the most significant similarities are considered in the final clustering results. The clustering process is refined by integrating detailed similarity measures and implementing equivalent similarity matrices and λ-cut matrices.

[0139] When finalizing the clustering, the final cluster assignment for each data point needs to be determined based on the refined membership values, and a defuzzification method is applied to solidify the clustering results. Specific steps include:

[0140] a) Assigning data points to clusters:

[0141] Determine the highest true value: For each data point, identify the value T with the highest true value. ij Clustering. Authenticity reflects the degree to which data points belong to a cluster, based on similarity measures and other factors considered in the clustering process.

[0142] Allocation rule: For each data point x i :

[0143]

[0144] Where j is the data point x i The index of clusters with the highest fidelity.

[0145] b) Applying deblurring methods:

[0146] The purpose of defuzzification is to transform fuzzy membership values ​​into explicit cluster assignments. This step ensures that each data point is explicitly assigned to a cluster.

[0147] Deblurring techniques:

[0148] Maximum membership method: Assign each data point to the cluster with the highest membership value. This is the most straightforward method and is suitable when explicit assignment is required.

[0149]

[0150] Weighted average method: If a more refined method is needed, a centroid-weighted average based on membership degree can be used. This method provides a more balanced distribution when the membership values ​​are not particularly high.

[0151]

[0152] Where c j T represents the centroid of cluster j. ij It is data point x i The degree of truth in cluster j.

[0153] Centroid method: Assigning data points to the cluster whose centroid is closest to the weighted average position of all clusters based on membership values.

[0154]

[0155] In distributed multi-center hospital data fusion scenarios, this method collects and preprocesses data, then uses online clustering technology to assign data points to clusters based on the highest true values, such as disease diagnosis rates. Finally, it uses methods like maximum membership to defuzzify the clusters and obtain the final clustering results. This result is then input into the shared LLM training process within the group, improving the model training effect.

[0156] Specifically, the unified intelligent clustering algorithm includes an unbiased estimation algorithm for the fair training objective function, a model optimization algorithm, and an algorithm for calculating the number of RR sets, wherein,

[0157] When 0 < α < 1, the following formula is used to calculate the unbiased estimate of the fair training objective function:

[0158] F α (S)=∑ c∈C n c u c (S) α ;

[0159] Among them, F α (S) is the objective function for fair training, and α is the inequality aversion parameter, which is applied to training groups c and n. c The number of clients is represented by u, and the proportion of clients effectively learned by the model is defined as u. c (S), where S is a set of various parameters and strategies involved in the current training;

[0160] When α = 0, the unbiased estimate of the fair training objective function is calculated using the following formula:

[0161] F0(S)=∑ c∈C n c ln||(u c (S));

[0162] The model optimization algorithm is used to calculate the first marginal fairness impact gain when the inequality aversion parameter is greater than 0 and less than 1, and to calculate the second marginal fairness impact gain when the inequality aversion parameter is equal to 0.

[0163] When 0 < α < 1, the first marginal fairness impact gain is calculated using the following formula:

[0164]

[0165] When α = 0, the second marginal fairness impact gain is calculated using the following formula:

[0166]

[0167] in, κ[v][c] represents the number of RR sets with root node u in training group c covered by S, reflecting the portion of training group c that has been effectively covered by the model under the current policy; κ[v][c] represents the number of RR sets with root node u in training group c that are covered by v but not by S, reflecting the potential of parameter v to provide additional coverage to training group c; θ c This represents the total number of RR sets in training group c.

[0168] F is calculated through the following steps. α Number of RR sets in (S) or F0(S) scenarios: Set parameters δ1∈(0,1) and ε1∈(0,1); calculate the number of RR sets using the RR set calculation formula.

[0169] The formula for calculating the RR set is:

[0170]

[0171] Where Q is an approximate parameter, b = max(u c (S * )), where s * It is the optimal solution to the problem of maximizing the fairness impact, and the value of b reflects the maximum proportion of clients being learned in each training group under the optimal solution.

[0172] It's important to note that in a federated learning system, after the model clustering and grouping operation is completed, each training group plays a unique and crucial role in the model training process. Considering each training group as an independent research object, let's focus on training group c, where the number of clients is n. c This indicates that, during model training, the proportion of clients in training group c that are effectively learned by the model is defined as u. c (S), where S encompasses the set of various parameters and policies involved in the current training, which has a direct and significant impact on the model training effect and fairness.

[0173] To accurately and comprehensively measure fairness during the training process, a fair training objective function F is constructed here. α (S). When 0 < α < 1, F α (S)=∑ c∈C n c u c (S) α When α = 0, F0(S) = ∑ c∈C n c ln||(u c(S)). In this function system, α, as an inequality aversion parameter, plays a crucial balancing role. When the value of α is closer to 1, more attention and emphasis will be given to improving the training effect during the model training process; conversely, when α is closer to 0, the fairness of training becomes the key consideration, and the model training will pay more attention to the balance of what clients in each training group are learned, avoiding situations where some training groups are over-focused or neglected.

[0174] In actual calculations, directly dealing with u c (S) α and ln||(u c Calculations performed using (S) often introduce biases, which can mislead the evaluation of fairness in model training and subsequent optimization. Therefore, it is necessary to use scientifically sound mathematical methods to obtain unbiased estimates to ensure the accuracy and reliability of the calculation results.

[0175] F is performed through the following steps. α Derivation of the unbiased estimate of (S):

[0176] The binomial series of Taylor expansions provides an effective derivation tool for this method, namely:

[0177]

[0178] Based on the definition of the fair training objective function, the following rigorous derivation is performed:

[0179]

[0180] In this formula, the value of η(n,α) varies depending on the value of n, specifically as follows:

[0181]

[0182] Through this derivation process, F can be calculated more accurately. α The unbiased estimate of (S).

[0183] The unbiased estimate of F0(S) is derived using the following formula:

[0184] Taylor series of logarithms This provides a theoretical basis for the derivation of the unbiased estimate of F0(S). Combining the definition of F0(S), we can obtain:

[0185]

[0186] Based on the above analysis of F αThe derivation of unbiased estimates for F0(S) and F0(S) allows for more accurate calculation methods, providing solid and reliable data support for subsequent model training and optimization, and making the measurement of training fairness more accurate and scientific.

[0187] In the training process of a shared large language model (LLM), unbiased estimation results are used as an important basis, and a greedy algorithm strategy is employed to optimize the model. The aim is to continuously improve the fairness and effectiveness of model training by rationally selecting parameter update and training strategies. Specifically, calculating the marginal gain of different parameter update or training strategy choices on the fair training objective function is a crucial step. This determines the optimal update method and drives the model training towards a better outcome.

[0188] Specifically, the steps for training and optimizing a model based on unbiased estimation include: calculating the marginal fairness impact gain and iteratively optimizing the model.

[0189] The formula for calculating the marginal fairness impact gain includes: when 0 < α < 1,

[0190]

[0191] When α = 0,

[0192]

[0193] In these formulas, κ[v][c] represents the number of RR sets with root node u in training group c covered by S, reflecting the portion of training group c that has been effectively covered by the model under the current strategy; κ[v][c] represents the number of RR sets with root node u in training group c that are covered by v but not by S, reflecting the potential of parameter b to provide additional coverage for training group c; θ c This represents the total number of RR sets in training group c, serving as an important baseline quantity in the calculation. Through the combined calculation of these parameters, the marginal impact of parameter v on the fair training objective function can be accurately measured.

[0194] The iterative optimization model steps include: meticulously recording the marginal gain after different parameter updates or policy adjustments in each training step. By comparing the fair training objective function values ​​under different choices, the scheme with the largest marginal gain is selected for model update. In practice, this means evaluating every possible parameter change or policy adjustment and analyzing its impact on the fair training objective function. For example, when considering updating a parameter v, the marginal gain is calculated... or The value of the parameter update is determined and compared with other possible parameter update choices. Selecting the parameter update or policy adjustment with the largest marginal gain maximizes the value of the fair training objective function in each iteration, thereby driving model training towards improved fairness. Through continuous iteration, the model gradually converges to a relatively better state, making the training process fairer and effectively improving model performance.

[0195] In federated learning training, the number of RR sets (reverse reachable sets) plays a crucial role in accurately approximating the fair training objective function. To ensure an accurate approximation of the fair training objective function with high probability, the required number of RR sets needs to be determined scientifically and rigorously.

[0196] Specifically, in F α The specific steps for calculating the number of RR sets in scenarios (S) or F0(S) include: first, setting parameters δ1∈(0,1) and ε1∈(0,1), and then using the formula... This is used to calculate a key reference value for the number of RR sets. In this formula, Q is used as an approximation parameter, and its value will be reasonably set according to the specific training scenario and accuracy requirements; b = max(u c (S * )), where s * This is the optimal solution to the problem of maximizing fairness, where the value of b reflects the maximum proportion of clients learned in each training group under the optimal solution. When the number of RR sets θ satisfies θ≥Cθ1, it is theoretically guaranteed that there is a probability of at least 1-δ1 such that... This means that under such a number of RR sets, the approximate result of the fair training objective function can reach a certain accuracy requirement with a high probability, providing a reliable guarantee for model training.

[0197] By accurately determining the number of RR sets, fairness assessment and model optimization can be performed based on a sufficient RR set. In actual training, a sufficient RR set can more comprehensively reflect the model's coverage and influence in different training groups, thus making the assessment of model fairness more accurate, providing more targeted directions for model optimization, and ultimately effectively improving the fairness and efficiency of model training, promoting high-quality training of shared LLM in a federated learning environment.

[0198] Optionally, the structural entropy corresponding to the data of each medical center can be determined by the following steps: abstracting the medical center data into a graph structure; calculating the probability vector of the stationary distribution of the vertices of the graph structure; calculating the one-dimensional structural entropy according to the one-dimensional structural information formula; and calculating the two-dimensional structural entropy according to the two-dimensional structural information formula.

[0199] The formula for the one-dimensional structural information is as follows:

[0200]

[0201] Where H1(G) is the one-dimensional structural entropy, and is the probability vector describing the stationary distribution of vertices in graph G. Graph G is an assumed undirected connected graph G = (V, E) with n vertices and m edges. For each vertex i, d i Represents the degree of vertex i in G;

[0202] in,

[0203] The formula for the two-dimensional structural information is as follows:

[0204]

[0205] Where L is the partition The number of modules in |P l | is module P l The number of nodes in d i It is P l The degree of the i-th node, vol(P) l ) is module P l The volume is also the volume of module P. l The sum of the degrees of all nodes in P, e(P) l ) is module P l The number of edges with only one endpoint. For the undirected connected graph G = (V, E), the following assumptions are made. It is a partition of V, called P L For a module or a community.

[0206] The noise intensity of each medical center's data can be determined through the following steps: the minimum value of the structural entropy in different dimensions is defined as the structural information in different dimensions; based on the structural information in different dimensions, the noise intensity of each medical center's data is determined.

[0207] In the real-world scenario of multi-center hospital data, to further enhance data privacy protection, before uploading the LLM models trained by each group, each group constructs a corresponding graph structure based on its own data and generates personalized noise vectors based on structural entropy.

[0208] For each set of hospital datasets, it is abstracted into a graph structure G = (V, E). Vertices V are composed of different data features, such as patient age, gender, disease diagnosis information, and treatment plans; edges E represent the relationships between features, with edge weights set according to the strength of the relationship. For example, regarding disease diagnosis information, if two diseases frequently co-occur in clinical practice, the edge weight between vertices representing these two diseases can be set to a higher value; if the two diseases are unrelated, the edge weight is set to a lower value.

[0209] Specifically, the steps for calculating structural entropy include: assuming G = (V, E) is an undirected connected graph with n vertices and m edges. For each vertex i, use d... i Let represent the degree of vertex i in G. According to the formula... Obtain the probability vector p that describes the stationary distribution of vertices in graph G. i Furthermore, based on the one-dimensional structural information formula... The one-dimensional structural entropy H1(G) is calculated.

[0210] For a two-dimensional structural entropy, given an undirected connected graph G=(V,E), assume... It is a partition of V, called P L For a module or a community. Through

[0211]

[0212] Calculate the entropy of a two-dimensional structure, where L is the partition. The number of modules in |P l | is module P l The number of nodes in d i It is P l The degree of the i-th node, vol(P) l ) is module P l The volume is also the volume of module P. l The sum of the degrees of all nodes in P, e(P) l ) is module P l The number of edges with only one endpoint. Also, according to H... mod (G)= We obtain two-dimensional structural information, namely module entropy H. mod (G), here It is any possible partition module that can be divided in graph G.

[0213] The specific steps for calculating noise intensity include: defining the minimum value of the structural entropy in different dimensions as the structural information of the graph in different dimensions, denoted as H. min (G)=min{H1(G),H mod(G)}. This structural information reflects the complexity and uncertainty of the hospital data graph structure.

[0214] Using structural information H min (G) Generate personalized noise vectors. An encryption algorithm is used, with H... min (G) serves as the seed for the random number generator. Assume a Gaussian distribution is used to generate the noise vector N, with the probability density function of the Gaussian distribution being... Where μ is the mean and σ is the standard deviation. When generating the noise vector, the value of σ can be adjusted according to the sensitivity of the data; the more sensitive the data, the larger σ, and the higher the intensity of the generated noise. For example, if hospital data contains highly sensitive data such as a large amount of patients' genetic information, σ can be appropriately increased; if the data is less sensitive, such as some routine symptom data, σ can be decreased. In this way, the generated noise vector N can meet the requirements of privacy protection while minimizing the impact on model performance.

[0215] Specifically, the steps for obtaining the target large language model based on the noise intensity of each medical center's data and the model weights of the trained shared large language model include: adding the noise intensity of each medical center's data to the weights of the shared large language model for each medical center's data to obtain multiple target model weights with added noise; using a weighted average aggregation algorithm to fuse the multiple target model weights with added noise to obtain the global model weights; and determining the target large language model based on the global model weights.

[0216] In the process of weight fusion, the personalized noise vector N generated locally by each hospital can be added to the local model weight W to obtain the model weight W′ after adding noise.

[0217] Subsequently, each hospital uploads its noise-added model weights W′ to the central server. After receiving the weights from all hospitals, the central server uses a weighted average aggregation algorithm to fuse these weights. Assume there are m hospitals participating in the training, and the noise-added weights uploaded by the i-th hospital are W′. i ′, fusion coefficient is Fusion coefficient ω i The data volume can be determined based on the proportion of each hospital's data volume. Let the data volume of the i-th hospital be n. i The total amount of data is but The final fusion yields the global model weights W. global for:

[0218]

[0219] Among them W i These are the model weights trained locally at the i-th hospital, N.i It is the personalized noise vector generated by the i-th hospital.

[0220] Through the above-described process of personalized noise generation and model weight fusion based on structural entropy, targeted noise is generated by utilizing the characteristics of structural entropy, effectively protecting the privacy of data from various hospitals. At the same time, the fusion of model weights from multiple hospitals is achieved, enabling the fused LLM to comprehensively utilize the advantages of data from all parties, improve model performance, and provide stronger support for the analysis and application of medical data.

[0221] The clustering-based dynamic federated optimization method provided in this application offers a practical and effective solution to the challenges of data fusion and utilization in distributed multi-center hospitals. Through a series of innovative technologies, it has achieved significant results in terms of model training effectiveness, training fairness, and data privacy protection.

[0222] To improve the training effect of large language models, dynamic clustering technology is used to classify data in real time based on data features, effectively reducing intra-group data heterogeneity, accelerating model convergence, and significantly improving training efficiency and accuracy, laying a solid foundation for the precise analysis of medical data. To ensure training fairness, a fairness maximization algorithm is used to reasonably allocate training weights among hospitals, ensuring that data from different hospitals are appropriately integrated into the model, improving model universality, stimulating enthusiasm for participation in federated learning, and maintaining long-term stable cooperative relationships. To strengthen data privacy protection, a personalized privacy noise mechanism based on structural entropy is introduced, generating a unique noise vector for each hospital. This effectively protects data privacy and eliminates potential medical data security risks without affecting the model's aggregation effect.

[0223] From a practical application perspective, this method enables efficient, fair, and secure joint modeling of medical data, promoting advancements in precision diagnosis, disease prediction, and personalized treatment within the medical field. It also facilitates the intelligent transformation of the healthcare industry, yielding significant social benefits. In the future, with the continuous growth of medical data and the ongoing development of federated learning technology, this method is expected to be further optimized, expanding its application scenarios and creating greater value for the medical field, thus making a greater contribution to human health.

[0224] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0225] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cluster-based dynamic federated optimization method, characterized in that, The method comprises: obtaining a plurality of central medical data, each central medical data comprising at least one of medical text data, structured data, and time series data; data preprocessing the plurality of central medical data to obtain a plurality of processed central medical data; inputting the plurality of processed central medical data into a pre-constructed unified medium intelligence clustering algorithm to obtain a clustering grouping result; based on the clustering grouping result, federated learning and fair influence maximization, training a shared large language model according to the plurality of processed central medical data to obtain a trained shared large language model; determining a structural entropy corresponding to each medical center data according to each medical center data, and determining a noise intensity of each medical center data according to the structural entropy corresponding to each medical center data; obtaining a target large language model according to the noise intensity of each medical center data and the model weight of the trained shared large language model; wherein the federated learning and fair influence maximization comprise a fair training objective function unbiased estimation calculation algorithm, a model optimization algorithm, and an RR set quantity calculation algorithm; wherein, when the following formula is used for unbiased estimation of the fair training objective function: ； wherein, is the fair training objective function, is the inequality aversion parameter, for the training group , is used to represent the number of clients, the proportion of clients effectively learned by the model is defined as , is the set of parameters and strategies involved in the current training; When the following formula is used to calculate the unbiased estimate of the fair training objective function: ； wherein the model optimization algorithm is used to calculate a first marginal fair influence gain when the inequality aversion parameter is greater than 0 and less than 1, and to calculate a second marginal fair influence gain when the inequality aversion parameter is equal to 0; wherein, when the first marginal fair influence gain is calculated by the following equation: ； wherein, when the second marginal fair influence gain is calculated by the following equation: ； in, Representative was Coverage of training groups The middle root node is The number of RR sets reflects the training set under the current strategy. The portion that has been effectively covered by the model; Indicates being Covered but not Coverage of training groups The middle root node is The number of RR sets reflects the parameter For the training group The potential for additional coverage; Indicates training group The total number of RR sets in the middle.

2. The method of claim 1, wherein, data preprocessing each central medical data in the following way: de-duplicating the central medical data to obtain first intermediate data; correcting the first intermediate data to obtain second intermediate data; performing data standardization and normalization processing on the second intermediate data to obtain the processed central medical data.

3. The method of claim 1, wherein, The unified medium intelligence clustering algorithm is constructed by the following steps: for each central medical data, a plurality of data points in the central medical data are represented as a single-value medium intelligence set, and each data point in the single-value medium intelligence set is represented as a single-value medium intelligence set; according to a pre-defined similarity measurement method, calculating a pair-wise similarity value between each two data points, and determining an overall similarity value according to a plurality of pair-wise similarity values; constructing a similarity matrix according to a plurality of pair-wise similarity values; performing normalization calculation on the similarity matrix to obtain a normalized similarity matrix; iterating according to the normalized similarity matrix to obtain a target matrix converged by the normalized similarity matrix, the target matrix being an equivalent similarity matrix; based on a pre-selected confidence level value, constructing a cut matrix according to the target matrix; determining a similarity measure according to the number of clusters, the cluster centroid and the membership value; obtaining a clustering grouping result according to the similarity measure, the target matrix and the cut matrix.

4. The method of claim 1, wherein, The RR set number under the scenario is calculated by the following steps or the scene Setting parameters , ; The number of RR sets is calculated by an RR set calculation formula; wherein the RR set calculation formula is: ； wherein is an approximation parameter, wherein is an optimal solution of the fairness impact maximization problem, The value of reflects the maximum proportion of clients learned in each training group under the optimal solution.

5. The method of claim 1, wherein, The structural entropy corresponding to each medical center data is determined by the following steps: abstracting the medical center data into a graph structure; calculating the probability vector of the vertex stationary distribution of the graph structure; calculating one-dimensional structural entropy according to a one-dimensional structure information formula; calculating two-dimensional structural entropy according to a two-dimensional structure information formula.

6. The method of claim 5, wherein, The one-dimensional structure information formula is: ； wherein one-dimensional structure entropy, description graph the probability vector of the stationary distribution of the middle vertex, and the graph G is an undirected connected graph with n vertices and m edges each vertex , the degree of the vertex in G is denoted by ; wherein ; wherein the two-dimensional structure information formula is: ； where is the number of modules in the partition , is the number of nodes in the module , is the degree of the th node in the module , is the volume of the module , which is also the sum of the degrees of all nodes in the module , is the average degree of the nodes in the module , is the average volume of the modules in the partition , is called a module or a community.

7. The method of claim 6, wherein, The noise intensity of each medical center data is determined by the following steps: The minimum value of the entropy of the structure of different dimensions is defined as the structure information under different dimensions; According to the structure information under different dimensions, the noise intensity of each medical center data is determined.

8. The method of claim 7, wherein, According to the noise intensity of each medical center data and the model weight of the trained shared large language model, the steps of obtaining the target large language model include: Add the noise intensity of each medical center data to the weight of the shared large language model of each medical center data to obtain a plurality of target model weights after adding noise; Using a weighted average aggregation algorithm, the plurality of target model weights after adding noise are fused to obtain a global model weight; According to the global model weight, the target large language model is determined.

Images