Dynamic federal optimization method based on clustering

Abstract

The invention provides a clustering-based dynamic federation optimization method, which comprises the following steps: acquiring a to-be-detected gene; acquiring multiple pieces of central medical data; performing data preprocessing on the plurality of pieces of central medical data to obtain a plurality of pieces of processed central medical data; inputting the plurality of central medical data into a pre-constructed unified central intelligent set clustering algorithm to obtain a clustering grouping result; based on a clustering grouping result, federated learning and fair influence maximization, respectively training a shared large oracle model according to the processed multiple pieces of central medical data to obtain a trained shared large language model; according to each piece of medical center data, determining a structure entropy corresponding to the medical center data, and according to the structure entropy corresponding to each piece of medical center data, determining noise intensity of each piece of medical center data; and obtaining a target large language model according to the noise intensity of each piece of medical center data and the model weight of the trained shared large language model.

Description

Technical Field

[0001] The present application relates to the medical field, and in particular, to a clustering-based dynamic federated optimization method. Background Art

[0002] In the healthcare field, the integration and utilization of distributed, multi-center hospital data is of great significance. Hospitals have accumulated vast amounts of patient data, encompassing a wealth of medical information, including disease diagnoses, treatment plans, and patient recovery status. Effectively integrating this data will significantly advance medical research, improving the accuracy of disease diagnoses and the effectiveness of treatment plans. However, achieving this goal faces numerous challenges.

[0003] On the one hand, data security and privacy protection are crucial. Hospital data contains a wealth of sensitive patient information. A leak would severely harm patients. Traditional centralized data processing methods are no longer suitable in this context, as centralized storage and processing of data poses heightened privacy risks. For example, if a hospital uploads patient data to a central server for analysis, the data could be exposed to attack and theft during transmission and storage.

[0004] On the other hand, data from different hospitals often exhibit non-independent and identically distributed characteristics. Due to differences in geography, patient populations, and medical technology levels, data from different hospitals exhibit significant variations in feature distribution and data size. This data heterogeneity severely impacts model training effectiveness. Directly applying traditional machine learning methods to uniformly model this data results in poor generalization performance and difficulty accurately adapting to the data characteristics of different hospitals. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a dynamic federated optimization method based on clustering, which adjusts the grouping in real time through clustering, divides hospitals with similar data characteristics and their data into the same group, and reduces the heterogeneity of data within the group. Within each group, a shared large language model (LLM) is trained through federated learning, and the fairness of training is ensured by using a fairness maximization algorithm. When aggregating model weights, personalized privacy noise is added to further protect data privacy, which can effectively improve the training efficiency and accuracy of the model and provide an efficient and secure solution for the integration and utilization of distributed multi-center hospital data.

[0006] An embodiment of the present application provides a clustering-based dynamic federation optimization method, characterized in that the method includes:

[0007] Acquire multiple central medical data, each central medical data including at least one of medical text data, structured data, and time series data;

[0008] performing data preprocessing on the plurality of central medical data to obtain processed plurality of central medical data;

[0009] Inputting the plurality of central medical data into a pre-built unified Neutral Intelligence clustering algorithm to obtain clustering grouping results;

[0010] Based on clustering results, federated learning, and fair impact maximization, the shared large oracle model is trained separately using the processed medical data from multiple centers to obtain a trained shared large language model.

[0011] Determine, based on each medical center data, the structural entropy corresponding to the medical center data, and determine, based on the structural entropy corresponding to each medical center data, the noise intensity of each medical center data;

[0012] The target large language model is obtained based on the noise intensity of each medical center's data and the model weight of the trained shared large language model.

[0013] Optionally, preprocess the medical data of each center in the following ways:

[0014] Deduplication is performed on the medical data of the center to obtain first intermediate data;

[0015] Correcting the first intermediate data to obtain second intermediate data;

[0016] The second intermediate data is subjected to data standardization and normalization processing to obtain processed medical data of the center.

[0017] Optionally, a unified neutrosophic clustering algorithm is constructed by the following steps:

[0018] For each central medical data, multiple data points in the central medical data are represented as a single-valued neutral intelligence set group, where each data point in the single-valued neutral intelligence set is represented as a single-valued neutral intelligence set;

[0019] Calculate the pairwise similarity value between each two data points according to a predefined similarity metric, and determine the overall similarity value based on multiple pairwise similarities;

[0020] Construct a similarity matrix based on multiple pairwise similarity values;

[0021] Performing normalization calculation on the similarity matrix to obtain a normalized similarity matrix;

[0022] Iterating according to the normalized similarity matrix to obtain a target matrix obtained by convergence of the normalized similarity matrix, wherein the target matrix is an equivalent similarity matrix;

[0023] constructing a cutting matrix from the target matrix based on a preselected confidence level value;

[0024] Determine the similarity measure based on the number of clusters, cluster centroids, and membership values;

[0025] A clustering grouping result is obtained according to the similarity measure, the target matrix and the cutting matrix.

[0026] Optionally, the unified neutrosophic clustering algorithm includes an unbiased estimation calculation algorithm for a fair training objective function, a model optimization algorithm, and an RR set number calculation algorithm, wherein:

[0027] When 0<α<1, the following formula is used to calculate the unbiased estimate of the fair training objective function:

[0028] F α (S)=∑ c∈C n c u c (S) α ;

[0029] Among them, F α (S) is the fair training objective function, α is the inequality aversion parameter, for training groups c, n c The number of clients is represented by , and the proportion of clients effectively learned by the model is defined as u c (S), S is a set of various parameters and strategies involved in the current training;

[0030] When α = 0, the following formula is used to calculate the unbiased estimate of the fair training objective function: F0(S) = ∑ c∈C n c ln(u c (S));

[0031] Optionally, the model optimization algorithm is used to calculate a first marginal fairness impact gain when the inequality aversion parameter is greater than 0 and less than 1, and to calculate a second marginal fairness impact gain when the inequality aversion parameter is equal to 0;

[0032] When 0<α<1, the first marginal fairness impact gain is calculated using the following formula:

[0033]

[0034] When α = 0, the second marginal fairness impact gain is calculated by the following formula:

[0035]

[0036] in, represents the number of RR sets with root node u in the training set c covered by S, which reflects the part of the training set c that has been effectively covered by the model under the current strategy; κ[v][c] represents the number of RR sets with root node u in the training set c covered by v but not covered by S, reflecting the potential of parameter v to provide additional coverage to the training set c; θ c represents the total number of RR sets in training group c.

[0037] Alternatively, calculate F by the following steps α Number of RR sets in (S) or F0(S) scenarios:

[0038] Set parameters δ1∈(0,1), ε1∈(0,1);

[0039] Calculate the number of RR sets using the RR set calculation formula;

[0040] The RR set calculation formula is:

[0041]

[0042] Among them, Q is the approximate parameter, b=max(u c (S * )), where s * It is the optimal solution to the fairness impact maximization problem. The value of b reflects the maximum proportion of clients learned in each training group under the optimal solution.

[0043] Optionally, determine the structural entropy corresponding to each medical center data by the following steps:

[0044] Abstract the medical center data into a graph structure;

[0045] Calculating a probability vector of a stationary distribution of vertices of the graph structure;

[0046] According to the one-dimensional structure information formula, calculate the one-dimensional structure entropy;

[0047] The two-dimensional structure entropy is calculated according to the two-dimensional structure information formula.

[0048] Optionally, the one-dimensional structure information formula is:

[0049]

[0050] Among them, H1(G) is the one-dimensional structural entropy, which is the probability vector describing the stable distribution of vertices in the graph G. The graph G is a hypothetical undirected connected graph G = (V, E) with n vertices and m edges. For each vertex i, d i represents the degree of vertex i in G;

[0051] in,

[0052] The two-dimensional structure information formula is:

[0053]

[0054] Where L is the partition The number of modules in |P l | is module P l The number of nodes in d i It's P l The degree of the i-th node in , vol(P l ) is module P l The volume of module P l The sum of the degrees of all nodes in the network, e(P l ) is module P l The number of edges with only one endpoint, For the undirected connected graph G = (V, E) assumed, It is a partition of V, called P L For a module or a community.

[0055] Optionally, determine the noise intensity of each medical center's data by:

[0056] The minimum value of the structural entropy of different dimensions is defined as the structural information in different dimensions;

[0057] The noise intensity of each medical center's data is determined based on the structural information in different dimensions.

[0058] Optionally, the step of obtaining a target large language model according to the noise intensity of each medical center data and the model weight of the trained shared large language model includes:

[0059] The noise intensity of each medical center data is added to the weight of the shared large language model of each medical center data to obtain multiple target model weights after adding noise;

[0060] The weighted average aggregation algorithm is used to fuse the weights of multiple target models after adding noise to obtain the global model weight;

[0061] A target large language model is determined according to the global model weight.

[0062] The clustering-based dynamic federated optimization method provided in the embodiment of the present application adjusts the grouping in real time through clustering, classifying hospitals with similar data characteristics and their data into the same group, thereby reducing the heterogeneity of the data within the group. Within each group, a shared large language model (LLM) is trained through federated learning, and the fairness of training is ensured by using a fairness maximization algorithm. When aggregating model weights, personalized privacy noise is added to further protect data privacy. This can effectively improve the training efficiency and accuracy of the model, and provide an efficient and secure solution for the integration and utilization of distributed multi-center hospital data.

[0063] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 A schematic diagram of a flow chart of a clustering-based dynamic federated optimization method provided in an embodiment of the present application;

[0066] Figure 2 A schematic diagram of the framework of the clustering-based dynamic federated optimization system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0068] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of medical technology.

[0069] Research has found that the integration and utilization of distributed, multi-center hospital data is of great significance in the medical field. Hospitals have accumulated vast amounts of patient data, encompassing a rich array of medical information, including disease diagnoses, treatment plans, and patient recovery status. Effectively integrating this data would significantly advance medical research, improving the accuracy of disease diagnoses and the effectiveness of treatment plans. However, achieving this goal faces numerous challenges.

[0070] On the one hand, data security and privacy protection are crucial. Hospital data contains a wealth of sensitive patient information. A leak would severely harm patients. Traditional centralized data processing methods are no longer suitable in this context, as centralized storage and processing of data poses heightened privacy risks. For example, if a hospital uploads patient data to a central server for analysis, the data could be exposed to attack and theft during transmission and storage.

[0071] On the other hand, data from different hospitals often exhibit non-independent and identically distributed characteristics. Due to differences in geography, patient populations, and medical technology levels, data from different hospitals exhibit significant variations in feature distribution and data size. This data heterogeneity severely impacts model training effectiveness. Directly applying traditional machine learning methods to uniformly model this data results in poor generalization performance and difficulty accurately adapting to the data characteristics of different hospitals.

[0072] Based on this, the embodiment of the present application provides a dynamic federated optimization method based on clustering. By clustering and adjusting the grouping in real time, hospitals with similar data characteristics and their data are divided into the same group to reduce the heterogeneity of the data within the group. Within each group, a shared large language model (LLM) is trained through federated learning, and the fairness of the training is ensured by using a fairness maximization algorithm. When aggregating the model weights, personalized privacy noise is added to further protect data privacy. This can effectively improve the training efficiency and accuracy of the model, and provide an efficient and secure solution for the integration and utilization of distributed multi-center hospital data.

[0073] It should be noted here that federated learning is an emerging artificial intelligence technology that aims to solve the problem of training machine learning models when data is not centralized or shared.

[0074] Assume there are n participants, each participant i has a local dataset D i , whose goal is to jointly train a global model f(θ), where θ is the parameter of the model. In traditional centralized learning, all data are concentrated in one place, and then the model is trained by optimizing the objective function.

[0075] However, in federated learning, data is scattered among various participants and cannot be trained directly in a centralized manner. Federated learning optimizes the global model by iteratively exchanging model parameters between participants. Specifically, in each round of training t, participant i first uses its own dataset D locally to train the model. i Train the model to obtain local model parameters Then, each participant will set the local model parameters Upload to the central server (or use a decentralized method to exchange parameters), the central server aggregates these parameters to obtain the global model parameters θ t+1 .

[0076] Next, the updated global model parameters θ t+1 The new global model parameters are distributed to all participants, who then use them to continue the next round of local training, repeating this cycle until the model converges. In this way, federated learning can leverage data from all parties to jointly train a globally optimal model while protecting data privacy.

[0077] In terms of maximizing the fairness impact, for a given graph G, with n G The classic influence maximization problem is to select a seed set S, which contains at most k seeds, to maximize the influence spread σ(S,G). Assume that each node belongs to a disjoint community set where V1∪V2∪…∪V C =V, V c (n c =|V c |) represents the set of nodes belonging to community c. Generally speaking, fair influence maximization (FIM) aims to narrow the influence gap between different communities while keeping the overall influence spread as unchanged as possible. For the fairness concept, the welfare function is used to aggregate the cardinality utilities of different communities. The goal is to select at most k seed nodes to maximize the objective function F α (S) (also called equity impact), where And F0(S)= When α=0. Utility u c (S) represents the expected proportion of nodes in community c that are affected given the seed set S. The exponent α is an inequality aversion parameter that controls the trade-off between fairness and overall impact. When α is close to 1, it favors the spread of influence; when α is close to 0, it favors fairness. When α = 0, it becomes a unique case, in which all communities are required to be affected by S.

[0078] See also Figure 1 , Figure 1The flowchart of a clustering-based dynamic federation optimization method provided in the embodiment of the present application is as follows. Figure 1 As shown in , the clustering-based dynamic federated optimization method provided in the embodiment of the present application includes:

[0079] S101. Acquire medical data from multiple centers.

[0080] Each central medical data includes at least one of medical text data, structured data, and time series data.

[0081] S102: Perform data preprocessing on the plurality of central medical data to obtain processed plurality of central medical data.

[0082] S103: Input the plurality of central medical data into a pre-built unified Neutral Intelligence clustering algorithm to obtain clustering grouping results.

[0083] S104. Based on the clustering grouping results, federated learning and maximization of fairness impact, the shared large oracle model is trained separately according to the processed medical data of multiple centers to obtain a trained shared large language model.

[0084] S105 . Determine the structural entropy corresponding to each medical center data according to the medical center data, and determine the noise intensity of each medical center data according to the structural entropy corresponding to each medical center data.

[0085] S106. Obtain a target large language model based on the noise intensity of each medical center data and the model weight of the trained shared large language model.

[0086] based on Figure 1 The framework diagram of the clustering-based dynamic federated optimization system is shown in Figure 2 .

[0087] In step S101 , data collection is the foundation for building a clustering-based dynamic federated optimization model, and its quality and breadth play a decisive role in the accuracy and applicability of the model.

[0088] The data sources for data collection include: regional medical data sharing platform, hospital information system (HIS), scientific research cooperation project data, etc.

[0089] Among them, regional medical data sharing platforms: Driven by the trend of collaborative regional medical development, these platforms integrate data from multiple hospitals within a region. These data are broadly representative, revealing both differences and commonalities in disease diagnosis and treatment approaches across hospitals. For example, for the same disease, different hospitals may employ different treatment plans and medications. Comparing these data can provide a reference for optimizing treatment plans and a richer sample for model training.

[0090] Hospital Information Systems (HIS): Hospital information systems store vast amounts of patient-related data. Medical records contain detailed information about a patient's medical history, including descriptions of symptoms, diagnoses, treatments, and medications for each visit. These records allow for a precise understanding of a patient's disease progression, such as fluctuations in the condition of chronic illnesses, providing key insights for analyzing disease characteristics and treatment needs.

[0091] Data from collaborative research projects: Collaborative research projects involving numerous medical institutions focus on specific diseases or medical fields. The data collected from these projects has been rigorously screened and organized, and possesses high research value. For example, research projects targeting rare diseases provide detailed records of patients' genetic testing results, specific symptoms, and long-term follow-up data.

[0092] Specifically, each central medical data includes at least one of medical text data, structured data, and time series data.

[0093] Among them, text data mainly includes patient consultation content, medical records, diagnosis reports, etc. Text data is the core data for demand judgment. By analyzing the text, we can understand the type and specific content of the patient's needs.

[0094] Structured data: Such as basic patient information (age, gender, medical history, etc.) and vital signs data (temperature, blood pressure, heart rate, etc.). Structured data can provide supplementary information to text data, helping to more fully understand the patient's needs.

[0095] Time series data: If there is data from multiple patient consultations or long-term health monitoring, time series data can reflect the changing trends and cyclical nature of patient needs. For example, symptoms of certain chronic disease patients may worsen within a specific time period. By analyzing time series data, patients' needs can be predicted in advance.

[0096] Optionally, each central medical data is preprocessed in the following manner: deduplication of the central medical data is performed to obtain first intermediate data; the first intermediate data is corrected to obtain second intermediate data; and the second intermediate data is standardized and normalized to obtain the processed central medical data.

[0097] The center's medical data is deduplicated, also known as medical data cleaning. Specifically, for medical data collected from multiple sources, the original dataset is first imported. Duplicate case records are not uncommon in medical data. For example, due to system failures or human errors, multiple identical test reports for the same patient may be entered. This duplicate data increases the computational burden of model training and can easily lead to overfitting, reducing the model's generalization ability. Using efficient deduplication techniques such as hashing algorithms, the data is comprehensively compared to ensure that each patient's record appears only once in the dataset, thereby optimizing the model training sample.

[0098] Medical data often contains non-standard expressions, such as inconsistent abbreviations for disease names and typos in drug names. By establishing a standardized medical terminology dictionary and employing string matching algorithms, we can correct medical terminology in text data. Numerical data, such as test index values, may contain outliers that fall outside the normal range. By setting appropriate thresholds, we can identify, correct, or delete these outliers to prevent them from misleading model training.

[0099] Correcting the first intermediate data is also known as data standardization and normalization. Specifically, standardization and normalization are crucial for data from multiple hospitals with varying dimensions and distribution characteristics. The cleaned medical dataset is input into the system, where standardization and normalization operations are performed.

[0100] Numerical medical data, such as a patient's age, height, weight, and various test index values, is standardized. Using the Z-score standardization method, we can effectively eliminate scale differences between different features, preventing excessively large or small feature values from dominating model training, and improving the stability and convergence speed of model training.

[0101] Normalization involves scaling data feature values to a specific range, such as [0, 1]. For some medical data, such as disease severity scores, min-max normalization is used. When different hospitals use different scoring systems for the same disease, normalization allows these data to be compared and analyzed on the same scale, preventing unbalanced effects of certain features on the model and enhancing the model's adaptability to multi-source medical data.

[0102] The above data preprocessing steps can effectively improve data quality, significantly enhance the model's training efficiency and final performance, and lay a solid foundation for the subsequent construction, training, and accurate output of clustering-based dynamic federated optimization models.

[0103] In this way, after the above data preprocessing steps, the data quality can be effectively improved, and the training efficiency and final performance of the model can be significantly improved, laying a solid foundation for the subsequent construction, training and accurate output of the clustering-based dynamic federated optimization model.

[0104] Specifically, a unified neutral intelligence clustering algorithm is constructed by the following steps: for each central medical data, multiple data points in the central medical data are represented by a single-valued neutral intelligence set group, and each data point in the single-valued neutral intelligence set is represented as a single-valued neutral intelligence set; according to a predefined similarity measurement method, the pairwise similarity value between each two data points is calculated, and the overall similarity value is determined based on multiple pairwise similarities; a similarity matrix is constructed based on the multiple pairwise similarity values; the similarity matrix is normalized to obtain a normalized similarity matrix; it is iterated based on the normalized similarity matrix to obtain a target matrix obtained by convergence of the normalized similarity matrix, and the target matrix is an equivalent similarity matrix; based on a preselected confidence level value, a cutting matrix is constructed based on the target matrix; the similarity measurement is determined based on the number of clusters, the cluster centroid and the membership value; the cluster grouping result is obtained based on the similarity measurement, the target matrix and the cutting matrix.

[0105] Among them, the Unified CICI Clustering Algorithm (UNCA) is specifically designed for cluster analysis of complex datasets such as medical data. It leverages CICI logic to deeply analyze the correlations between data points, achieving more accurate and detailed clustering. Its core process begins with the input of medical data, and goes through the calculation of the similarity matrix, the construction of the composite matrix, the creation of the λ-cut matrix, the initialization of CICI k-means clustering, the assignment of membership and centroid updates, the integration of DVN-MST clustering, the refinement of cluster assignments, and finally the determination of the clustering results. In this series of closely connected steps, each step lays the foundation for exploring the potential structure of the data and improving the quality of clustering. The collaboration of these steps enables UNCA to effectively capture complex data relationships and provide accurate and robust clustering results.

[0106] As an example, when calculating the similarity matrix, the following steps are specifically included:

[0107] a) SVNS representation of data points: Represent the data points as single-valued neutrosophic sets (SVNS): Each data point x in the dataset i Represented as an SVNS, including its truthfulness T i , uncertainty I i and falsehood F i ;

[0108] b) Similarity metric definition: using

[0109]

[0110] Similarity(T i ,T j ) etc. are used to measure x i and x jsimilarity of truth, uncertainty and falsehood;

[0111] c) Calculate pairwise similarity: For each pair of data points (x i ,x j ), calculate the similarity S(x i ,x j This step involves evaluating the similarity of each attribute (T, I, F) and combining these values to obtain an overall similarity score;

[0112] d) Construct similarity matrix: Create a similarity matrix C of size n×n (n is the number of data points), where each element C ij Represents the data point x i and x j The similarity between elements C ij Given by the following formula: C {ij} =S(x i ,x j );

[0113] The matrix is symmetric because x i and x j The similarity between j and x i The similarities between them are the same.

[0114] e) Normalized Similarity Matrix: Depending on the application requirements, it may be necessary to normalize the similarity matrix to ensure that the values are within a specific range, such as [0, 1]. The resulting similarity matrix C captures the relationships between data points based on their neural network features and facilitates the clustering process by providing a quantitative basis for comparing data points.

[0115] Optionally, the present application also requires the construction of a composite matrix, and the specific steps include: iteratively constructing the composite matrix C, C 2 ,C 4 ,… until the matrix C {2(k+1)} Converges to C 2k , ensuring that C becomes an equivalent similarity matrix; Initialize the composite matrix: Start with the initial similarity matrix C calculated in the previous step. This matrix C serves as the basis for constructing a higher-order composite matrix; Calculate a higher-order composite matrix: Calculate the subsequent power matrix of C. Specifically, it is necessary to calculate C 2 ,C 4 etc. This is done by matrix multiplication, as in C 2 =C×C,C 4 =C 2 ×C 2 , continue this process to obtain C 2k, where k is the iteration index; Check convergence: Iterate the matrix exponentiation process until the resultant matrix converges. Specifically, it is necessary to ensure that C 2(k+1) ≈C 2k Normalize the final matrix: Normalize the final composite matrix as needed to align the dimensions of the similarity values. This is done using standard normalization techniques.

[0116] In each iteration, a synthetic matrix is constructed, which extracts more refined relationships than those represented in the original matrix. The final matrix will be a similarity matrix that summarizes the relationships between all data points for the next step of clustering.

[0117] The target matrix is constructed based on the cut matrix, also known as the lambda cut matrix, which is used as a threshold for determining significant similarity between data points.

[0118] As an example, the lambda cut matrix can be determined by the following steps:

[0119] a) Choose the confidence level λ: Define a threshold λ between 0 and 1 to determine which similarities are significant in the cluster. A higher λ will include more similarities, while a lower λ will be more selective;

[0120] b) Apply λ threshold: Use confidence level λ to get the value from the composite matrix C 2k This involves thresholding the matrix, retaining only similarities above a chosen confidence level. Specifically: for the matrix C 2k Each element in Compare it with λ. Construct the λ cut matrix C λ ,in:

[0121]

[0122] This will generate a binary matrix C λ , where each element indicates whether the similarity between data i and j is significant.

[0123] c) Verify Matrix Properties: Check that the lambda cut matrix is a valid clustering matrix. The output representation should preserve basic similarity relationships and allow for the elimination of weak links. Ensure that the matrix is not broken and is suitable for the next step of clustering.

[0124] If this λ-cut matrix is too sparse or too dense, the confidence level λ is changed. According to λ, the relationship between data points is fine-tuned to emphasize the most relevant clustering relationships in preparation for the final clustering step.

[0125] In the step of obtaining the cluster grouping result according to the similarity measure, the target matrix and the cut matrix, including cluster allocation refinement and finalization, the cluster allocation can be refined by utilizing the similarity measure of the single valued neutral set (SVNS) and combining the equivalent similarity matrix and the lambda cut matrix.

[0126] Specifically, the steps of refining cluster assignment include:

[0127] a) Using SVNS similarity measurement:

[0128] Re-evaluate the similarity measure: Re-evaluate the similarity between the data point and the cluster based on the SVNS similarity measure calculated in the previous steps. SVNS provides a detailed similarity measure that combines truth, uncertainty, and falsehood, providing a more detailed view of the relationship between the data point and each cluster.

[0129] Update membership: Combine the SVNS similarity metric to refine the membership assignment, and use the updated similarity value to recalculate the membership of each data point to each cluster, such as T {ij} is the updated similarity measure based on SVNS, I {ij} is a refined measure of uncertainty, F {ij} =1-T {ij} .

[0130] b) Implementing the Equivalent Similarity Matrix:

[0131] Construct equivalence matrix: Use the refined similarity measure to construct the equivalence similarity matrix C. This matrix should reflect the updated clustering relationship based on the latest similarity values.

[0132] Matrix consistency: Ensure that the equivalence similarity matrix is consistent with the previously constructed matrix. It should accurately represent the updated similarity relationships between clusters.

[0133] c) Apply the lambda cut matrix:

[0134] Construct Lambda Cut Matrix: Create a lambda cut matrix based on the selected confidence level lambda. This matrix helps define the threshold for membership assignment and filters out less important relationships.

[0135] Integration with Similarity Matrix: Apply the lambda cut matrix to the equivalent similarity matrix. This process involves adjusting the similarity values based on the confidence level, thus further refining the cluster assignments.

[0136] Thresholding: Applying the lambda cut matrix to threshold the similarity values. For a given confidence level lambda, the similarity measure is filtered,

[0137]

[0138] Update Clusters: Update cluster assignments using filtered similarity values. This ensures that only the most significant similarities are considered in the final clustering results. The clustering process is refined by incorporating detailed similarity measures and implementing equivalent similarity matrices and lambda cut matrices.

[0139] When finalizing the clustering, it is necessary to determine the final cluster assignment of each data point based on the refined membership value and apply the defuzzification method to consolidate the clustering results. The specific steps include:

[0140] a) Assigning data points to clusters:

[0141] Determine the highest truth value: For each data point, identify the value with the highest truth value T ij The trueness reflects the degree to which a data point belongs to a cluster, based on the similarity measure and other factors considered in the clustering process.

[0142] Assignment rule: For each data point x i :

[0143]

[0144] where j is the data point x i The index of the cluster with the highest truth.

[0145] b) Apply defuzzification methods:

[0146] The goal of defuzzification is to convert fuzzy membership values into clear cluster assignments. This step ensures that each data point is unambiguously assigned to a cluster.

[0147] Defuzzification technology:

[0148] Maximum Membership: Assign each data point to the cluster with the largest membership value. This is the most direct method and is suitable when a clear assignment is required.

[0149]

[0150] Weighted average: If you need a more nuanced approach, you can use a weighted average of the cluster centroids based on membership. This method provides a more balanced distribution when the membership values are not particularly high.

[0151]

[0152] where c j represents the centroid of cluster j, T ij is the data point x i The degree of truth in cluster j.

[0153] Centroid method: Assigns a data point to the cluster whose centroid is closest to the weighted average location of all clusters based on membership values.

[0154]

[0155] In a distributed multi-center hospital data fusion scenario, this method collects and preprocesses data, then uses online clustering techniques to assign data points to clusters based on the highest true value, such as disease confirmation rate. Defuzzification is then performed using methods such as the maximum membership method to obtain the final clustering result. This result is then fed into the group's shared LLM training process to improve model training effectiveness.

[0156] Specifically, the unified neutrino clustering algorithm includes an unbiased estimation calculation algorithm for a fair training objective function, a model optimization algorithm, and an RR set number calculation algorithm, wherein:

[0157] When 0<α<1, the following formula is used to calculate the unbiased estimate of the fair training objective function:

[0158] F α (S)=∑ c∈C n c u c (S) α ;

[0159] Among them, F α (S) is the fair training objective function, α is the inequality aversion parameter, for training groups c, n c The number of clients is represented by , and the proportion of clients effectively learned by the model is defined as u c (S), S is a set of various parameters and strategies involved in the current training;

[0160] When α = 0, the following formula is used to calculate the unbiased estimate of the fair training objective function:

[0161] F0(S)=∑ c∈C n c ln||(u c (S));

[0162] The model optimization algorithm is used to calculate the first marginal fairness impact gain when the inequality aversion parameter is greater than 0 and less than 1, and to calculate the second marginal fairness impact gain when the inequality aversion parameter is equal to 0.

[0163] When 0<α<1, the first marginal fairness impact gain is calculated using the following formula:

[0164]

[0165] When α = 0, the second marginal fairness impact gain is calculated by the following formula:

[0166]

[0167] in, represents the number of RR sets with root node u in the training set c covered by S, which reflects the part of the training set c that has been effectively covered by the model under the current strategy; κ[v][c] represents the number of RR sets with root node u in the training set c covered by v but not covered by S, reflecting the potential of parameter v to provide additional coverage to the training set c; θ c represents the total number of RR sets in training group c.

[0168] Among them, F is calculated by the following steps: α Number of RR sets in the (S) or F0(S) scenario: Set parameters δ1∈(0,1), ε1∈(0,1); calculate the number of RR sets using the RR set calculation formula.

[0169] The RR set calculation formula is:

[0170]

[0171] Among them, Q is the approximate parameter, b=max(u c (S * )), where s * It is the optimal solution to the fairness impact maximization problem. The value of b reflects the maximum proportion of clients learned in each training group under the optimal solution.

[0172] It should be noted that in the federated learning system, after completing the model clustering operation, each training group plays a unique and critical role in the model training process. Each training group is regarded as an independent research object. For training group c, the number of clients is n. c During the model training process, the proportion of clients in the training group c that are effectively learned by the model is defined as u c (S), where S covers the various parameters and strategy sets involved in the current training, which has a direct and important impact on the model training effect and fairness.

[0173] In order to accurately and comprehensively measure the fairness of the training process, a fair training objective function F is constructed here α (S). When 0<α<1, F α (S)=∑ c∈C n c u c (S) α ; When α=0, F0(S)=∑ c∈C n c ln||(u c(S)). In this function system, α, as the inequality aversion parameter, plays a crucial balancing role. When the value of α approaches 1, during model training, greater attention and emphasis will be placed on improving training results. Conversely, when α approaches 0, training fairness becomes a key consideration. Model training will place greater emphasis on the balance of clients learned within each training group, avoiding situations where certain training groups are overly focused or neglected.

[0174] In the actual calculation process, directly c (S) α and ln||(u c Calculations based on the (S) often produce biases that can mislead fairness assessments of model training and subsequent optimization. Therefore, it is necessary to use scientific and sound mathematical methods to obtain unbiased estimates to ensure the accuracy and reliability of the calculation results.

[0175] Among them, F is carried out by the following steps α Derivation of unbiased estimation of (S):

[0176] The binomial series of Taylor expansion provides an effective derivation tool for this method, namely:

[0177]

[0178] Combined with the definition of the fair training objective function, the following rigorous derivation is performed:

[0179]

[0180] In this formula, the value of η(n,α) varies according to n, specifically:

[0181]

[0182] Through this derivation process, F can be calculated more accurately α An unbiased estimate of (S).

[0183] Among them, the unbiased estimation of F0(S) is derived by the following formula:

[0184] Taylor expansion of the logarithmic series This provides a theoretical basis for the derivation of unbiased estimation of F0(S). Combined with the definition of F0(S), we can obtain:

[0185]

[0186] Through the above αThe unbiased estimation derivation of F0(S) and F0(S) can obtain a more accurate calculation method, providing solid and reliable data support for subsequent model training optimization, making it more accurate and scientific when measuring training fairness.

[0187] During the training of a shared large language model (LLM), unbiased estimation results are used as the key basis for model optimization using a greedy algorithm. This aims to continuously improve the fairness and effectiveness of model training by rationally selecting parameter updates and training strategies. Specifically, calculating the marginal gain of different parameter updates or training strategy choices on the fairness training objective function is a key step in determining the optimal update method and driving model training towards a more optimal direction.

[0188] Specifically, the steps of model training optimization based on unbiased estimation include: calculating marginal fairness impact gain and iteratively optimizing the model.

[0189] The formula for calculating the marginal equity impact gain includes: when 0<α<1,

[0190]

[0191] When α=0,

[0192]

[0193] In these formulas, represents the number of RR sets with root node u in the training group c covered by S, which reflects the part of the training group c that has been effectively covered by the model under the current strategy; κ[v][c] represents the number of RR sets with root node u in the training group c covered by v but not covered by S, reflecting the potential of parameter b to provide additional coverage for the training group c; θ c is the total number of RR sets in the training group c, which is an important benchmark in the calculation. Through the comprehensive operation of these parameters, the marginal impact of parameter v on the fair training objective function can be accurately measured.

[0194] The iterative optimization model step includes: in each training step, detailed record of the marginal gains after different parameter updates or policy adjustments. By comparing the values of the fair training objective function under different choices, the solution with the largest marginal gain is selected to implement the model update. In practice, this means that every possible parameter change or policy adjustment is evaluated and analyzed for its impact on the fair training objective function. For example, when considering updating a parameter v, the calculation or and compare it with other possible parameter update choices. Selecting the parameter update or policy adjustment with the largest marginal gain can maximize the value of the fairness training objective function in each iteration, thereby driving model training towards continuous improvement in fairness. Through continuous iteration, the model will gradually converge to a relatively better state, making the training process more fair and effectively improving model performance.

[0195] During federated learning training, the number of reverse reachable sets (RR sets) plays a key role in accurately approximating the fair training objective function. To ensure accurate approximation of the fair training objective function with high probability, the required number of RR sets needs to be determined scientifically and rigorously.

[0196] Specifically, in F α The specific steps for calculating the number of RR sets in the (S) or F0(S) scenario include: first setting parameters δ1∈(0,1), ε1∈(0,1), and then using the formula To calculate a key reference value for the number of RR sets. In this formula, Q is an approximate parameter, and its value will be reasonably set according to the specific training scenario and accuracy requirements; b=max(u c (S * )), where s * is the optimal solution to the fairness impact maximization problem. The value of b reflects the maximum proportion of clients in each training group that are learned under the optimal solution. When the number of RR sets θ satisfies θ≥Cθ1, it can be theoretically guaranteed that with a probability of at least 1-δ1, This means that under such a large number of RR sets, the approximate result of the fair training objective function can meet certain accuracy requirements with high probability, providing a reliable guarantee for model training.

[0197] By precisely determining the number of RR sets, fairness assessment and model optimization can be performed based on sufficient RR sets. In actual training, sufficient RR sets can more comprehensively reflect the model's coverage and impact in different training groups, thereby making model fairness assessments more accurate and providing more targeted guidance for model optimization. Ultimately, this effectively improves the fairness and efficiency of model training and promotes high-quality training of shared LLMs in a federated learning environment.

[0198] Optionally, the structural entropy corresponding to each medical center data can be determined by the following steps: abstracting the medical center data into a graph structure; calculating the probability vector of the stationary distribution of the vertices of the graph structure; calculating the one-dimensional structural entropy according to the one-dimensional structural information formula; and calculating the two-dimensional structural entropy according to the two-dimensional structural information formula.

[0199] Wherein, the one-dimensional structure information formula is:

[0200]

[0201] Among them, H1(G) is the one-dimensional structural entropy, which is the probability vector describing the stable distribution of vertices in the graph G. The graph G is a hypothetical undirected connected graph G = (V, E) with n vertices and m edges. For each vertex i, d i represents the degree of vertex i in G;

[0202] in,

[0203] The two-dimensional structure information formula is:

[0204]

[0205] Where L is the partition The number of modules in |P l | is module P l The number of nodes in d i It's P l The degree of the i-th node in , vol(P l ) is module P l The volume of module P l The sum of the degrees of all nodes in the network, e(P l ) is module P l The number of edges with only one endpoint, For the undirected connected graph G = (V, E) assumed, It is a partition of V, called P L For a module or a community.

[0206] The noise intensity of each medical center data can be determined by the following steps: defining the minimum value of the structural entropy of different dimensions as the structural information in different dimensions; and determining the noise intensity of each medical center data based on the structural information in different dimensions.

[0207] In the actual scenario of multi-center hospital data, in order to further strengthen data privacy protection, before uploading the LLM model trained by each group, each group constructs a corresponding graph structure based on its own data and generates a personalized noise vector based on structural entropy.

[0208] For each hospital dataset, we abstract it into a graph structure G = (V, E). Vertices V are composed of different data features, such as the patient's age, gender, disease diagnosis, and treatment plan. Edges E represent the associations between features, and edge weights are determined based on the closeness of the association. For example, if two diseases frequently coexist in clinical practice, the edge weight between the vertices representing these two diseases can be set high; if the two diseases are unrelated, the edge weight can be set low.

[0209] Specifically, the steps for calculating the structural entropy include: assuming that G = (V, E) is an undirected connected graph with n vertices and m edges. For each vertex i, use d i Represents the degree of vertex i in G. According to the formula Get the probability vector p describing the stationary distribution of vertices in graph G i . Then, according to the one-dimensional structural information formula The one-dimensional structural entropy H1(G) is calculated.

[0210] For the two-dimensional structural entropy, given an undirected connected graph G = (V, E), assume It is a partition of V, called P L For a module or a community.

[0211]

[0212] Calculate the two-dimensional structural entropy, where L is the partition The number of modules in |P l | is module P l The number of nodes in d i It's P l The degree of the i-th node in , vol(P l ) is module P l The volume of module P l The sum of the degrees of all nodes in the network, e(P l ) is module P l The number of edges with only one endpoint. At the same time, according to H mod (G)= Get the two-dimensional structural information, that is, the module entropy H mod (G) Here It is all possible partition modules in graph G.

[0213] The specific steps for calculating the noise intensity include: defining the minimum value of the structural entropy of different dimensions as the structural information of the graph in different dimensions, denoted as H min (G) = min{H1(G),H mod(G)}. This structural information reflects the complexity and uncertainty of the hospital data graph structure.

[0214] Using structural information H min (G) Generate personalized noise vector. Use encryption algorithm to generate personalized noise vector. min (G) is used as the seed of the random number generator. Assume that the noise vector N is generated using Gaussian distribution, and the probability density function of Gaussian distribution is Where μ is the mean and σ is the standard deviation. When generating the noise vector, the value of σ can be adjusted based on the sensitivity of the data. The more sensitive the data, the larger σ, and the higher the noise intensity generated. For example, if hospital data contains highly sensitive data such as a large amount of patient genetic information, σ can be appropriately increased. If the data is less sensitive, such as some common symptom data, σ can be reduced. In this way, the generated noise vector N can meet privacy protection requirements while minimizing the impact on model performance.

[0215] Specifically, the step of obtaining a target large language model based on the noise intensity of each medical center data and the model weight of the trained shared large language model includes: adding the noise intensity of each medical center data to the weight of the shared large language model of each medical center data to obtain multiple target model weights after adding noise; using a weighted average aggregation algorithm to fuse the multiple target model weights after adding noise to obtain a global model weight; and determining the target large language model based on the global model weight.

[0216] When performing weight fusion, the personalized noise vector N generated locally by each hospital can be added to the local model weight W to obtain the model weight W′ after adding noise.

[0217] Subsequently, each hospital uploads the model weight W′ after adding noise to the central server. After receiving the weights uploaded by all hospitals, the central server uses a weighted average aggregation algorithm to fuse these weights. Assuming that there are m hospitals participating in the training, the weight after adding noise uploaded by the i-th hospital is W i ′, the fusion coefficient is Fusion coefficient ω i It can be determined based on the proportion of data volume of each hospital. Let the data volume of the i-th hospital be n i The total amount of data is but The final fused global model weight W global for:

[0218]

[0219] Where W i is the model weight trained locally in the i-th hospital, Ni is the personalized noise vector generated by the i-th hospital.

[0220] Through the above-mentioned personalized noise generation and model weight fusion process based on structural entropy, the characteristics of structural entropy are used to generate targeted noise, effectively protecting the privacy of each hospital's data, and realizing the fusion of multi-center hospital model weights. The fused LLM can comprehensively utilize the data advantages of all parties, improve model performance, and provide stronger support for the analysis and application of medical data.

[0221] The clustering-based dynamic federated optimization method provided in this application provides a practical and effective solution to the problem of data integration and utilization in distributed multi-center hospitals. Through a series of innovative technologies, it has achieved significant results in model training effectiveness, training fairness, and data privacy protection.

[0222] In terms of improving the training effect of large language models, with the help of dynamic clustering technology, real-time classification based on data features is carried out, which effectively reduces the heterogeneity of data within the group, accelerates model convergence, and greatly improves training efficiency and accuracy, laying a solid foundation for the accurate analysis of medical data. In terms of ensuring training fairness, the fairness maximization algorithm is used to reasonably allocate the training weights of each hospital, ensuring that data from different hospitals are reasonably integrated into the model, improving the versatility of the model, stimulating the enthusiasm of all parties to participate in federated learning, and maintaining a long-term and stable cooperative relationship. In terms of strengthening data privacy protection, a personalized privacy noise mechanism based on structural entropy is introduced. Each hospital generates a unique noise vector, which effectively protects data privacy and eliminates medical data security risks without affecting the aggregation effect of the model.

[0223] In terms of practical application value, this method enables efficient, fair, and secure joint modeling of medical data, promoting advancements in precision diagnosis, disease prediction, and personalized treatment, facilitating the intelligent transformation of the medical industry, and possessing significant social benefits. In the future, with the continued growth of medical data and the continued development of federated learning technology, this method is expected to be further optimized and expanded in application scenarios, creating greater value for the medical field and making further contributions to human health.

[0224] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0225] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A clustering-based dynamic federated optimization method, characterized in that: The method comprises: Acquire multiple central medical data, each central medical data including at least one of medical text data, structured data, and time series data; performing data preprocessing on the plurality of central medical data to obtain processed plurality of central medical data; Inputting the plurality of central medical data into a pre-built unified Neutral Intelligence clustering algorithm to obtain clustering grouping results; Based on clustering results, federated learning, and fair impact maximization, the shared large oracle model is trained separately using the processed medical data from multiple centers to obtain a trained shared large language model. Determine, based on each medical center data, the structural entropy corresponding to the medical center data, and determine, based on the structural entropy corresponding to each medical center data, the noise intensity of each medical center data; The target large language model is obtained based on the noise intensity of each medical center's data and the model weight of the trained shared large language model.

2. The method according to claim 1, characterized in that Data preprocessing of each center's medical data is performed in the following ways: Deduplication is performed on the medical data of the center to obtain first intermediate data; Correcting the first intermediate data to obtain second intermediate data; The second intermediate data is subjected to data standardization and normalization processing to obtain processed medical data of the center.

3. The method according to claim 1, characterized in that The unified neutrosophic clustering algorithm is constructed by the following steps: For each central medical data, multiple data points in the central medical data are represented as a single-valued neutral intelligence set group, where each data point in the single-valued neutral intelligence set is represented as a single-valued neutral intelligence set; Calculate the pairwise similarity value between each two data points according to a predefined similarity metric, and determine the overall similarity value based on multiple pairwise similarities; Construct a similarity matrix based on multiple pairwise similarity values; Performing normalization calculation on the similarity matrix to obtain a normalized similarity matrix; Iterating according to the normalized similarity matrix to obtain a target matrix obtained by convergence of the normalized similarity matrix, wherein the target matrix is an equivalent similarity matrix; constructing a cutting matrix from the target matrix based on a preselected confidence level value; Determine the similarity measure based on the number of clusters, cluster centroids, and membership values; A clustering grouping result is obtained according to the similarity measure, the target matrix and the cutting matrix.

4. The method according to claim 1, wherein The unified neutrino clustering algorithm includes an unbiased estimation calculation algorithm for a fair training objective function, a model optimization algorithm, and an RR set number calculation algorithm, wherein: When 0<α<1, the following formula is used to calculate the unbiased estimate of the fair training objective function: F α (S)=∑ c∈C n c u c (S) α ; Among them, F α (S) is the fair training objective function, α is the inequality aversion parameter, for training groups c, n c The number of clients is represented by , and the proportion of clients effectively learned by the model is defined as u c (S), S is a set of various parameters and strategies involved in the current training; When α = 0, the following formula is used to calculate the unbiased estimate of the fair training objective function: F0(S)=∑ c∈C n c ln(u c (S)).

5. The method according to claim 4, characterized in that The model optimization algorithm is used to calculate a first marginal equity impact gain when the inequality aversion parameter is greater than 0 and less than 1, and is used to calculate a second marginal equity impact gain when the inequality aversion parameter is equal to 0; When 0<α<1, the first marginal fairness impact gain is calculated using the following formula: When α = 0, the second marginal fairness impact gain is calculated by the following formula: in, represents the number of RR sets with root node u in the training group c covered by S, which reflects the part of the training group c that has been effectively covered by the model under the current strategy; k[v][c] represents the number of RR sets with root node u in the training group c covered by v but not covered by S, reflecting the potential of parameter v to provide additional coverage to the training group c; θ c represents the total number of RR sets in training group c.

6. The method according to claim 5, characterized in that Calculate F by the following steps α Number of RR sets in (S) or F0(S) scenarios: Set parameters δ1∈(0,1), ε1∈(0,1); Calculate the number of RR sets using the RR set calculation formula; The RR set calculation formula is: Among them, Q is the approximate parameter, b=max(u c (S * )), where s * It is the optimal solution to the fairness impact maximization problem. The value of b reflects the maximum proportion of clients learned in each training group under the optimal solution.

7. The method according to claim 1, characterized in that Determine the structural entropy corresponding to each medical center's data by following these steps: Abstract the medical center data into a graph structure; Calculating a probability vector of a stationary distribution of vertices of the graph structure; According to the one-dimensional structure information formula, calculate the one-dimensional structure entropy; The two-dimensional structure entropy is calculated according to the two-dimensional structure information formula.

8. The method according to claim 7, characterized in that The one-dimensional structural information formula is: Among them, H1(G) is the one-dimensional structural entropy, which is the probability vector describing the stable distribution of vertices in the graph G. The graph G is a hypothetical undirected connected graph G = (V, E) with n vertices and m edges. For each vertex i, d i represents the degree of vertex i in G; in, The two-dimensional structure information formula is: Where L is the partition The number of modules in |P l | is module P l The number of nodes in d i It's P l The degree of the i-th node in , vol(P l ) is module P l The volume of module P l The sum of the degrees of all nodes in the network, e(P l ) is module P l The number of edges with only one endpoint, For the undirected connected graph G = (V, E) assumed, It is a partition of V, called P L For a module or a community.

9. The method according to claim 8, characterized in that Determine the noise intensity of each medical center's data using the following steps: The minimum value of the structural entropy of different dimensions is defined as the structural information in different dimensions; The noise intensity of each medical center's data is determined based on the structural information in different dimensions.

10. The method according to claim 9, characterized in that Based on the noise intensity of each medical center's data and the model weights of the trained shared large language model, the steps for obtaining the target large language model include: The noise intensity of each medical center data is added to the weight of the shared large language model of each medical center data to obtain multiple target model weights after adding noise; The weighted average aggregation algorithm is used to fuse the weights of multiple target models after adding noise to obtain the global model weight; A target large language model is determined according to the global model weight.

Images