A privacy-preserving collaborative training method for multi-source heterogeneous data based on federated learning
By employing methods such as multi-source data processing, secure communication, gradient adaptive calibration, dynamic differential noise compensation, and system architecture collaborative scheduling, the training challenge of multi-source heterogeneous data in federated learning is solved, enabling efficient and accurate multi-agency joint modeling and improving model performance and privacy protection capabilities.
Patent Information
- Application Number
- CN202611293606.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-17
AI Technical Summary
Existing federated learning methods struggle to simultaneously address issues such as multi-source heterogeneous data perception, adaptive gradient calibration, dynamic privacy budget allocation, noise bias compensation, and communication efficiency optimization, resulting in model performance that fails to meet practical requirements.
A multi-source data processing module is used to unify the data format, a secure communication module is used for encrypted transmission, a gradient adaptive calibration module is used for gradient calibration, a dynamic differential noise compensation module is used for privacy protection, a communication compression transmission module is used to optimize transmission, and a system architecture collaborative scheduling module is used for centralized training to achieve global model optimization and convergence.
While ensuring that data does not leave the domain and strictly protects privacy, it significantly improves the accuracy and efficiency of multi-institutional joint modeling, effectively handles the training difficulty in multi-source heterogeneous data scenarios, and improves the convergence speed and accuracy of the model.
Smart Images

Figure CN123122912A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data security, and in particular to a collaborative training method for privacy protection of multi-source heterogeneous data based on federated learning. Background Technology
[0002] With the deep integration and development of artificial intelligence (AI) technology and the big data industry, data has become a core production factor driving socio-economic transformation and technological innovation. In key areas such as healthcare, fintech, smart government, and industrial manufacturing, the accumulation of massive amounts of data has provided unprecedentedly rich materials for training machine learning models, promoting the implementation of a series of high-value AI applications such as disease-assisted diagnosis, credit risk assessment, intelligent traffic scheduling, and industrial fault prediction. However, realizing the value of data faces an increasingly severe contradiction: on the one hand, the performance of machine learning models is highly dependent on the scale and diversity of training data. Data held by a single institution often suffers from limitations such as insufficient sample size, skewed distribution, and single feature dimensions, making it difficult to train high-precision models with strong generalization capabilities. On the other hand, with the successive promulgation and increasingly stringent data privacy protection laws and regulations in various countries, it is clearly required that personal and sensitive data cannot be transferred and shared across institutions without full authorization. The direct aggregation and centralized processing of raw data faces significant compliance risks at both legal and ethical levels. This contradiction makes the collaborative utilization of data among multiple institutions a key scientific problem that urgently needs to be solved in the field of artificial intelligence.
[0003] Federated learning, as a distributed machine learning paradigm where the data remains stationary while the model moves, provides a practical technical path to address the aforementioned contradictions. The core idea of federated learning is that each data holder trains the model locally and only uploads model parameters or gradient update information to the central server. The central server aggregates the update results from all parties and then distributes the global model. This process iterates until the model converges. Throughout the training process, the original data remains on the local servers of each participant and does not leave its respective data domain, thus satisfying the compliance requirement of data not leaving its domain to a certain extent.
[0004] However, in practical applications, the data from different participating institutions often exhibit significant distributional differences, which manifest in multiple dimensions: at the label distribution level, due to the different business focuses and service targets of different institutions, the proportions of each category in their data may vary drastically. For example, the disease distribution in large general hospitals is relatively uniform, while the disease distribution in specialized hospitals is highly concentrated. At the feature space level, the data collection equipment, testing standards, and recording specifications of different regions or institutions may differ, leading to a systematic shift in the numerical distribution of features with the same semantic meaning. At the sample size level, large institutions may have millions of records, while small institutions may have only tens of thousands or even fewer, and the data scale between participants may differ by several orders of magnitude. These three types of heterogeneity often coexist and overlap in real-world scenarios, greatly increasing the training difficulty of federated learning. Existing methods either only handle a single type of heterogeneity or treat different types of heterogeneity separately, lacking a unified framework to simultaneously perceive and process the overlapping effects of the three types of heterogeneity, resulting in unsatisfactory model performance in real-world multi-source heterogeneous scenarios.
[0005] In summary, the current field of federated learning lacks a comprehensive method that can simultaneously address multiple coupled problems such as multi-source heterogeneous sensing, adaptive gradient calibration, dynamic privacy budget allocation, noise bias compensation, and communication efficiency optimization. This invention addresses these shortcomings by proposing a privacy-preserving collaborative training method for multi-source heterogeneous data based on federated learning. This method significantly improves the accuracy, efficiency, and practicality of multi-institutional joint modeling while ensuring data remains within its domain and strictly protecting privacy. Summary of the Invention
[0006] The purpose of this invention is to provide a privacy-preserving collaborative training method for multi-source heterogeneous data based on federated learning, aiming to solve the problems mentioned in the background above.
[0007] To achieve the above objectives, this invention proposes a privacy-preserving collaborative training method for multi-source heterogeneous data based on federated learning. The method comprises a multi-source data processing module, a secure communication module, a gradient adaptive calibration module, a dynamic differential noise compensation module, a communication compression and transmission module, a global model optimization and convergence module, and a system architecture collaborative scheduling module. In the multi-source data processing module, different clients perform unified data format processing locally. In the secure communication module, the client and the aggregation server exchange passwords for encrypted data transmission. The gradient adaptive calibration module uses a gradient adaptive calibration algorithm to perform gradient calibration on the multi-source heterogeneous data. The dynamic differential noise compensation module uses a dynamic differential algorithm to improve data privacy during the federated learning process. The communication compression and transmission module compresses the data during transmission. The global model optimization and convergence module determines the degree of iterative optimization in federated learning based on the convergence conditions of the aggregation server. The system architecture collaborative scheduling module adopts a centralized aggregation distributed training architecture for multi-client model training.
[0008] Furthermore, the multi-source data processing module allows the client to perform format standardization processing on the raw data locally, and to statistically analyze the category distribution, feature mean, and total sample size locally before encrypting and uploading them to the aggregation server.
[0009] Furthermore, the secure communication module aggregates the generation of asymmetric key pairs between the server and the client and exchanges public keys. All public keys are bound to identities through digital certificates and share a random seed for mask generation through a key exchange protocol. Each round of communication uses an independent session key and is accompanied by a message authentication code and digital signature to ensure the confidentiality, integrity, and source authentication of the transmitted content.
[0010] Furthermore, the gradient adaptive calibration algorithm calculates the overall heterogeneity based on the number of client samples, the mean of feature dimensions, and the total number of samples. It defines the distribution overlap coefficient between the client and the global model, and uses the product of three factors—temporal annealing factor, global bias direction, and distribution gating factor—as the gradient correction amount for gradient calibration. It also updates the aggregate weights of the client in the federated learning global model using nonlinear distribution alignment factor, heterogeneity penalty factor, and data volume reward factor.
[0011] Furthermore, the client updates the aggregate weights in the federated learning global model, including the following steps:
[0012] S1. Collect basic statistical information from all clients, including the number of samples in each category, the mean of feature dimensions, and the total number of samples. Based on the global category distribution ratio, the global weighted mean of feature dimensions, and the global average sample size, calculate the label distribution offset, feature space drift, and sample size loss to obtain the comprehensive heterogeneity.
[0013] S2. Define the distribution overlap coefficient between the client and the global distribution. Using the original local gradient of the client in each dimension, the global weighted average of the gradients of all clients is used as the global consensus direction. Based on the distribution overlap coefficient, a distribution gating factor is constructed. The gradient correction amount is combined with the product of the time-series annealing factor, the global deviation direction, and the distribution gating factor to perform gradient calibration.
[0014] S3. Use the square of the client distribution overlap coefficient as the non-linear distribution alignment factor, the client comprehensive heterogeneity as the heterogeneity penalty factor, and the client sample size as the data volume reward factor to update the client's aggregation weight in the federated learning global model.
[0015] First, for different clients, calculate the label distribution offset based on the sample size, mean of feature dimensions, total sample size, global class distribution ratio, global weighted mean of feature dimensions, and global average sample size. Feature space drift Sample size measurement Among them, the label distribution offset , Indicates the total number of data categories. Indicates the first degree of isomerism coefficient. Indicates the first The first client Number of class samples Indicates the first Total number of samples per client Represents the global scope of the first Total number of samples in each class Represents the total number of global samples and the feature space drift. , Indicates the total number of feature dimensions. This represents the second degree of isomerism coefficient. Indicates the first The client in the first The sample mean on the dimensional feature. Indicates the global number Weighted mean on dimensional features, sample size loss measurement , Indicates the third degree of isomerism coefficient. This represents the total number of clients participating in federated training. Represents the smoothing constant, based on the label distribution offset. Feature space drift Sample size measurement Define the degree of comprehensive heterogeneity , The heterogeneity score is naturally encoded within the heterogeneity step, and the distribution shift present in clients with sufficient sample size does not affect the heterogeneity score. Then, based on the client sample ratio, the distribution overlap coefficient between the client and the global distribution is defined. , The distribution overlap coefficient is calculated by summing the smaller values among the proportions of each category, reflecting the consistency between client data and global data, and thus forming the distribution gating factor. The global weighted mean of all client gradients is used as the global consensus direction, and the global bias direction is formed by the original local gradients of each client in each dimension. Finally, a time-series annealing factor is constructed based on time to perform gradient calibration, resulting in the calibrated gradient components. , ,in, Indicates the first The client in the first The original gradient components in dimension, Indicates the first The globally weighted mean of all client gradients on the dimension. Indicates the first Federal training rounds, This indicates the preset total number of federal training rounds. The gradient correction vector represents the smoothing gradient constant, which dynamically changes with the federated training process and adaptively adjusts according to distribution consistency. This allows the global model to fully utilize the local information from each party in the early stages and quickly converge to the global optimum in the later stages. Then, based on the client distribution overlap coefficient, client comprehensive heterogeneity, and client sample size, the nonlinear distribution alignment factor, heterogeneity penalty factor, and data volume reward factor are obtained, which are used to update the aggregation weights of the clients in the federated learning global model. ,in, Indicates the first The aggregate weights of each client in the federated learning global model Indicates the first The overall heterogeneity of each client Indicates the first The total number of samples from each client and the aggregated weights ensure that, during the global model update in federated learning, the client with the most consistent distribution with the global model, the lowest heterogeneity, and the largest amount of data has the greatest say, accelerating the convergence of the global model and improving accuracy.
[0016] Furthermore, the dynamic difference algorithm, based on the client's original gradient vector and the global weighted mean, uses the client's gradient range to obtain the gradient sensitivity. Based on the client's gradient sensitivity and overall heterogeneity, it allocates privacy weights for the client, calculates the client's dynamic privacy fitness, and injects noise through the client's own gradient range using independently sampled random noise with a standard normal distribution mechanism.
[0017] Further, noise injection is performed, including the following steps:
[0018] S1. Calculate the consensus deviation between the client's original gradient vector and the global weighted mean. Use the client's gradient range to obtain the normalization factor. Based on the consensus deviation and the normalization factor, obtain the gradient sensitivity.
[0019] S2. Based on the gradient sensitivity and overall heterogeneity of the client, define a privacy allocation joint factor, allocate privacy weights to the client based on the reciprocal of the privacy allocation joint factor, and calculate the client's dynamic privacy fitness.
[0020] S3. By using the gradient range of the client itself to replace the traditional global pruning threshold, and based on the client's dynamic privacy fitness, distribution overlap coefficient, and sample size, noise injection is performed using random noise independently sampled by the standard normal distribution mechanism.
[0021] First, the consensus bias is calculated using the difference between the client's original gradient vector and the globally weighted mean of the client's gradients. Then, the gradient sensitivity is calculated using the client's gradient range as a normalization factor. , ,in, Indicates the first The first round of federal training The client in the first The original gradient components in dimension, Indicates the first The first round of federal training The globally weighted mean of all client gradients on the dimension. The total dimension is represented by the product of gradient sensitivity and overall heterogeneity to obtain the privacy allocation joint factor. Based on the reciprocal of this joint factor, privacy weights are assigned to the client, and the privacy fitness is calculated. ,in Indicates the first The first round of federal training Privacy tolerance of each client Indicates the first The first round of federal training The overall heterogeneity of each client Indicates the first The first round of federal training The overall heterogeneity of each client Indicates the first The first round of federal training Gradient sensitivity of each client, This represents the overall privacy fitness, ensuring that privacy resources are allocated to clients with gradients consistent with global consensus and data distribution aligned with global standards. It maximizes global model accuracy while maintaining an overall level of privacy protection. Then, using the client gradient range as the local gradient domain, random noise is independently sampled from a standard normal distribution and injected to complete the encryption. ,in, Indicates the first The first round of federal training The client in the first The gradient after injecting differential privacy noise on the dimension, Indicates the first The first round of federal training The client in the first The calibrated gradient components on the dimension, Indicates random noise. The comprehensive coefficient represents the adaptive personalized differential privacy protection based on the client's own local gradient range, distribution overlap coefficient, sample size, and privacy adaptability.
[0022] Furthermore, in the communication compression and transmission module, after the client generates a noisy gradient locally, it sorts the gradient dimensions by absolute value and performs sparsification. The sparsification ratio adaptively decreases as training progresses. The retained gradients are transmitted to the aggregation server. The gradient components discarded in each round are stored in a buffer and recompressed after being added to the new gradient in the next round, ensuring that all gradient information is transmitted completely after a sufficient number of rounds. After the client completes local calculation, it asynchronously uploads the compressed gradient. The aggregation server starts aggregation after receiving the preset ratio.
[0023] Furthermore, in the global model optimization convergence module, after aggregation, the client calculates the loss value on the local validation set and uploads it in encryption. The aggregation server calculates the global validation loss by weighting the values according to the size of the validation set. When the relative rate of change of this loss is lower than a preset threshold in multiple consecutive rounds, and the loss is lower than the preset target, the federated learning training is terminated. When the global validation loss does not improve for several consecutive rounds, the global base learning rate is automatically multiplied by a decay factor, and federated learning training is restarted.
[0024] Furthermore, the system architecture collaborative scheduling module adopts a centralized aggregated distributed training architecture, which includes three types of roles: aggregation server, client local computing node, and audit and supervision node. The aggregation server is responsible for global model storage updates and training process scheduling, the client local computing node is responsible for local data processing and model training, and the audit and supervision node monitors the compliance metadata of the training process.
[0025] Beneficial effects
[0026] 1. The gradient adaptive calibration algorithm proposed in this invention calculates the label distribution offset, feature space drift, and sample size loss based on the sample size, feature dimension mean, total sample size, global class distribution ratio, global feature dimension weighted mean, and global average sample size, thereby obtaining the comprehensive heterogeneity. It defines the distribution overlap coefficient between the client and the global distribution, uses the client's original local gradients in each dimension, and employs the global weighted mean of all client gradients as the global consensus direction. A distribution gating factor is constructed based on the distribution overlap coefficient. The gradient correction is then performed by combining the temporal annealing factor, the global deviation direction, and the distribution gating factor as the gradient correction amount. This invention utilizes the square of the client distribution overlap coefficient as a nonlinear distribution alignment factor, the client comprehensive heterogeneity as a heterogeneity penalty factor, and the client sample size as a data volume reward factor to update the aggregate weights of clients in the federated learning global model. The gradient adaptive calibration algorithm proposed in this invention intelligently corrects the gradients of different clients in terms of direction, neither completely preserving the local direction nor forcing global alignment. It adaptively balances the two based on the consistency of data distribution and training progress, ensuring that the client with the most consistent distribution with the global model, the lowest heterogeneity, and the largest data volume has the greatest say in global model updates, accelerating convergence and improving accuracy, thus balancing convergence speed and global accuracy.
[0027] 2. The dynamic differencing algorithm proposed in this invention calculates the consensus deviation between the client's original gradient vector and the global weighted mean. It uses the client's gradient range to calculate gradient sensitivity. Based on the client's gradient sensitivity and overall heterogeneity, a privacy allocation joint factor is defined to calculate the client's dynamic privacy fitness. The algorithm replaces the traditional global pruning threshold with the client's own gradient range. Based on the client's dynamic privacy fitness, distribution overlap coefficient, and sample size, it injects noise using independently sampled random noise obtained through a standard normal distribution mechanism. The proposed dynamic differencing algorithm simultaneously achieves two novel characteristics: scale invariance and consensus awareness. Regardless of the magnitude of the client's original gradient, the sensitivity score remains within a reasonable range and can be directly compared across clients. While ensuring the overall privacy protection level remains unchanged, it allocates limited privacy resources to clients with gradient safety and aligned data distributions, maximizing global model accuracy. It replaces global pruning with the client's local gradient range. For clients with smaller gradient magnitudes, the local range is much smaller than the global pruning threshold, requiring less noise and reducing unnecessary accuracy loss. The introduction of a distribution overlap coefficient reduces noise without significantly increasing privacy risks. Based on the client's own gradient and data characteristics, it maximizes model accuracy under strict privacy guarantees. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0029] Figure 1 This is a system schematic diagram of the present invention; Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0031] To achieve the above objectives, this invention proposes a privacy-preserving collaborative training method for multi-source heterogeneous data based on federated learning. The method comprises a multi-source data processing module, a secure communication module, a gradient adaptive calibration module, a dynamic differential noise compensation module, a communication compression and transmission module, a global model optimization and convergence module, and a system architecture collaborative scheduling module. In the multi-source data processing module, different clients perform unified data format processing locally. In the secure communication module, the client and the aggregation server exchange passwords for encrypted data transmission. The gradient adaptive calibration module uses a gradient adaptive calibration algorithm to perform gradient calibration on the multi-source heterogeneous data. The dynamic differential noise compensation module uses a dynamic differential algorithm to improve data privacy during the federated learning process. The communication compression and transmission module compresses the data during transmission. The global model optimization and convergence module determines the degree of iterative optimization in federated learning based on the convergence conditions of the aggregation server. The system architecture collaborative scheduling module adopts a centralized aggregation distributed training architecture for multi-client model training.
[0032] Specifically, the multi-source data processing module, located on the client side, performs format standardization processing on the raw data, and locally calculates the category distribution, feature mean, and total sample size before encrypting and uploading it to the aggregation server.
[0033] Specifically, the secure communication module aggregates the generation of asymmetric key pairs between the server and the client and exchanges public keys. All public keys are bound to identities through digital certificates and share a random seed for mask generation through a key exchange protocol. Each round of communication uses an independent session key and is accompanied by a message authentication code and digital signature to ensure the confidentiality, integrity and source authentication of the transmitted content.
[0034] Specifically, the gradient adaptive calibration algorithm calculates the overall heterogeneity based on the number of client samples, the mean of feature dimensions, and the total number of samples. It defines the distribution overlap coefficient between the client and the global model, and uses the product of three factors—temporal annealing factor, global bias direction, and distribution gating factor—as the gradient correction amount for gradient calibration. It updates the aggregation weights of the client in the federated learning global model using a nonlinear distribution alignment factor, a heterogeneity penalty factor, and a data volume reward factor. First, for different clients, considering the number of samples, the mean of feature dimensions, the total number of samples, the global class distribution ratio, the global weighted mean of feature dimensions, and the global average sample size, it calculates the label distribution offset. Feature space drift Sample size measurement Among them, the label distribution offset , Indicates the total number of data categories. Indicates the first degree of isomerism coefficient. Indicates the first The first client Number of class samples Indicates the first Total number of samples per client Represents the global scope of the first Total number of samples in each class Represents the total number of global samples and the feature space drift. , Indicates the total number of feature dimensions. This represents the second degree of isomerism coefficient. Indicates the first The client in the first The sample mean on the dimensional feature. Indicates the global number Weighted mean on dimensional features, sample size loss measurement , Indicates the third degree of isomerism coefficient. This represents the total number of clients participating in federated training. Represents the smoothing constant, based on the label distribution offset. Feature space drift Sample size measurement Define the degree of comprehensive heterogeneity , The heterogeneity score is naturally encoded within the heterogeneity step, and the distribution shift present in clients with sufficient sample size does not affect the heterogeneity score. Then, based on the client sample ratio, the distribution overlap coefficient between the client and the global distribution is defined. , The distribution overlap coefficient is calculated by summing the smaller values among the proportions of each category, reflecting the consistency between client data and global data, and thus forming the distribution gating factor. The global weighted mean of all client gradients is used as the global consensus direction, and the global bias direction is formed by the original local gradients of each client in each dimension. Finally, a time-series annealing factor is constructed based on time to perform gradient calibration, resulting in the calibrated gradient components. , ,in, Indicates the first The client in the first The original gradient components in dimension, Indicates the first The globally weighted mean of all client gradients on the dimension. Indicates the first Federal training rounds, This indicates the preset total number of federal training rounds. The gradient correction vector represents the smoothing gradient constant, which dynamically changes with the federated training process and adaptively adjusts according to distribution consistency. This allows the global model to fully utilize the local information from each party in the early stages and quickly converge to the global optimum in the later stages. Then, based on the client distribution overlap coefficient, client comprehensive heterogeneity, and client sample size, the nonlinear distribution alignment factor, heterogeneity penalty factor, and data volume reward factor are obtained, which are used to update the aggregation weights of the clients in the federated learning global model. ,in, Indicates the first The aggregate weights of each client in the federated learning global model Indicates the first The overall heterogeneity of each client Indicates the first The total number of samples from each client and the aggregated weights ensure that, during the global model update in federated learning, the client with the most consistent distribution with the global model, the lowest heterogeneity, and the largest amount of data has the greatest say, accelerating the convergence of the global model and improving accuracy.
[0035] Specifically, the dynamic difference algorithm, based on the client's original gradient vector and the global weighted mean, uses the client's gradient range to obtain gradient sensitivity. Based on the client's gradient sensitivity and overall heterogeneity, it assigns privacy weights to the client and calculates the client's dynamic privacy fitness. Noise injection is performed using the client's own gradient range and independently sampled random noise through a standard normal distribution mechanism. First, the difference between the client's original gradient vector and the global weighted mean of the client's gradient is used as the consensus bias. Then, the client's gradient range is used as a normalization factor to calculate the gradient sensitivity. , ,in, Indicates the first The first round of federal training The client in the first The original gradient components in dimension, Indicates the first The first round of federal training The globally weighted mean of all client gradients on the dimension. The total dimension is represented by the product of gradient sensitivity and overall heterogeneity to obtain the privacy allocation joint factor. Based on the reciprocal of this joint factor, privacy weights are assigned to the client, and the privacy fitness is calculated. ,in Indicates the first The first round of federal training Privacy tolerance of each client Indicates the first The first round of federal training The overall heterogeneity of each client Indicates the first The first round of federal training The overall heterogeneity of each client Indicates the first The first round of federal training Gradient sensitivity of each client, This represents the overall privacy fitness, ensuring that privacy resources are allocated to clients with gradients consistent with global consensus and data distribution aligned with global standards. It maximizes global model accuracy while maintaining an overall level of privacy protection. Then, using the client gradient range as the local gradient domain, random noise is independently sampled from a standard normal distribution and injected to complete the encryption. ,in, Indicates the first The first round of federal training The client in the first The gradient after injecting differential privacy noise on the dimension, Indicates the first The first round of federal training The client in the first The calibrated gradient components on the dimension, Indicates random noise. The comprehensive coefficient represents the adaptive personalized differential privacy protection based on the client's own local gradient range, distribution overlap coefficient, sample size, and privacy adaptability.
[0036] Specifically, in the communication compression and transmission module, after the client generates a noisy gradient locally, it sorts the gradient dimensions by absolute value and performs sparsification. The sparsification ratio decreases adaptively as training progresses. The retained gradients are transmitted to the aggregation server. The gradient components discarded in each round are stored in a buffer and recompressed after being added to the new gradient in the next round, ensuring that all gradient information is transmitted completely after a sufficient number of rounds. After the client completes local calculation, it asynchronously uploads the compressed gradient. The aggregation server starts aggregation after receiving the preset ratio.
[0037] Specifically, in the global model optimization convergence module, after aggregation, the client calculates the loss value on the local validation set and uploads it in encryption. The aggregation server calculates the global validation loss by weighting the values according to the size of the validation set. When the relative rate of change of this loss is lower than a preset threshold in multiple consecutive rounds, and the loss is lower than the preset target, the federated learning training is terminated. When the global validation loss does not improve for several consecutive rounds, the global base learning rate is automatically multiplied by a decay factor, and federated learning training is restarted.
[0038] Specifically, the system architecture collaborative scheduling module adopts a centralized aggregated distributed training architecture, which includes three types of roles: aggregation server, client local computing node, and audit and supervision node. The aggregation server is responsible for global model storage updates and training process scheduling, the client local computing node is responsible for local data processing and model training, and the audit and supervision node monitors the compliance metadata of the training process.
[0039] This embodiment uses multi-institutional medical data federated collaborative modeling as its application scenario. Three medical institutions from different regions are selected as participants in the federated learning, with patient disease risk prediction as the modeling objective. Participant A is a large tertiary general hospital in eastern China, holding approximately 1.2 million electronic medical records covering multiple departments. The data includes 56 clinical laboratory indicators, covering 12 common chronic diseases. The data distribution is relatively balanced, but the testing equipment is high-end imported, and the numerical range deviates somewhat from domestic standards. Participant B is a secondary specialized hospital in central China, holding approximately 350,000 electronic medical records, primarily for cardiovascular disease patients, with cardiovascular-related categories accounting for 65%. The testing equipment is domestically produced mid-range equipment, and the numerical range of some indicators shows a systematic deviation from that of Participant A. Participant C is a primary healthcare consortium in western China, holding approximately 80,000 electronic medical records. Due to limited testing conditions at the primary level, 11 of the 56 features are missing for a long time, resulting in the smallest data volume and incomplete features. Due to restrictions imposed by patient privacy regulations, none of the three parties can transmit original medical records to external sources. However, they hope to jointly train a high-precision disease risk prediction model through federated learning. The aggregation server is deployed on a secure cloud platform designated by the health authorities, with a maximum training epoch of 200 epochs. The target validation loss is 0.25, sparsity is 30% in the first epoch, decreasing by 5% every 10 epochs, with a minimum of 5%, and asynchronous aggregation ratio is 60%.
[0040] Before the federated training began, each of the three parties performed data preprocessing locally. Institution A standardized field names to the system's default naming conventions, converted units of measurement and date formats, and marked missing values. Institution B converted different units of measurement for indicators such as blood glucose to a unified standard. Each party encrypted and uploaded its feature field list to the aggregation server. The aggregation server calculated the intersection of the three parties' feature fields, obtaining 45 common features, and then distributed the alignment scheme. Institutions A and B discarded their unique feature columns locally. Institution C, naturally possessing the 45 common features, required no additional processing. Subsequently, each party performed data cleaning locally. Each party calculated the mean and standard deviation of the 45 features locally, encrypted the statistics, and uploaded them. The aggregation server calculated the global mean and global standard deviation weighted by sample size and then distributed them. The three parties then re-performed standardization using the global statistics. Each of the three parties also locally compiled and uploaded the sample distribution, mean features, and total sample size for 12 disease categories. Among them, Institution A had 1.19 million samples with a relatively even distribution, Institution B had 348,000 samples with a very high proportion of coronary heart disease, and Institution C had 79,000 samples with a severe category skewness. Each of the three parties locally divided the training set, validation set, and test set in an 8:1:1 ratio.
[0041] On the same day, the three parties and the aggregation server completed secure initialization, including key exchange, digital certificate identity binding, and negotiation of a shared random seed between each pair. Subsequently, in each round of training, each institution uses the shared seed to generate mask vectors that are opposites of each other, which are then superimposed on the gradient. This ensures that the masks automatically cancel each other out when the aggregation server aggregates the data, making it impossible for any single institution to see its original gradient throughout the entire process.
[0042] The aggregation server calculates the multidimensional heterogeneity score for each institution based on the statistics uploaded by the three parties. The first heterogeneity coefficient is 0.4, the second is 0.4, the third is 0.2, and the comprehensive coefficient is 0.5. Institution A has a large and evenly distributed dataset with a small label distribution shift, and its sample size is much higher than the average, resulting in a negative sample size loss value, thus further reducing its heterogeneity. Institution B has a much higher proportion of coronary heart disease cases than the global average, leading to a severe label distribution shift. Institution C has the smallest dataset but the most missing features.
[0043] After the aggregation server distributes the global initial model to the three parties, they begin the first round of training in parallel locally. The learning rate decay factor is 0.5. Each party completes mini-batch gradient descent on its local training set and then calculates its local gradient. The temporal annealing factor in the first round is only 0.005, with minimal correction, and the gradient almost maintains its original direction, allowing each party to explore freely. Subsequently, due to its distribution overlap coefficient of approximately 0.88, low heterogeneity, and large data volume, organization A has an aggregation weight as high as 0.854; while organization C, due to its distribution overlap coefficient of only 0.65, high heterogeneity, and small data volume, has a weight of only 0.020.
[0044] During training, the global validation loss decreased to 0.249 in the 168th round, below the target value of 0.25, thus meeting the early stopping condition and terminating training. The final model achieved a weighted average score of 0.91 on the three local test sets, a significant improvement over the best scores obtained by each of the three organizations through independent training. Throughout the entire process, the original medical record data from the three organizations remained stored on their respective local servers and never left their respective data domains. The only content transmitted externally was the gradient vector, compressed and encrypted after injecting differential privacy noise, and a small amount of statistical information. The aggregation server could not see the original gradients of any individual institution, nor could it reverse-engineer the original data. This fully verifies the effectiveness and practicality of this invention in achieving joint modeling without data leaving its domain in multi-source heterogeneous data scenarios.
[0045] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0046] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0047] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A privacy-preserving collaborative training method for multi-source heterogeneous data based on federated learning, characterized in that, Includes the following steps: S1 Each participating client completes standardized preprocessing of multi-source heterogeneous data locally, statistically analyzes local data distribution characteristics, and encrypts and uploads the data to the aggregation server. The client and the aggregation server complete security key negotiation and establish an encrypted communication link; S2 employs a gradient adaptive calibration mechanism for gradient correction and dynamic allocation of aggregate weights: it calculates the overall heterogeneity based on the differences in label distribution, feature distribution, and sample size of each client, and combines the degree of overlap between the client and global data distributions. It corrects the local gradient direction through the combined effect of time-series annealing factor, global bias direction, and distribution gating factor. At the same time, it dynamically updates the aggregate weights of each client based on the degree of distribution alignment, heterogeneity level, and sample size. S3 employs a dynamic differential privacy mechanism for personalized noise injection: it calculates gradient sensitivity based on the deviation between local gradients and global consensus, dynamically assigns privacy weights to each client and calculates privacy fitness by combining comprehensive heterogeneity, and injects differentiated random noise according to the client's own gradient value range scale to maximize model accuracy under global privacy budget constraints. S4 performs adaptive sparse compression transmission on the noisy gradients, and the discarded gradient components are cached and accumulated for retransmission in subsequent rounds. The aggregation server starts asynchronous aggregation after receiving the client uploads of a preset proportion. After S5 aggregates and generates the global model, it determines the convergence status based on multi-party validation loss. If the convergence status is not met, it issues an updated global model to start the next iteration. If the status is met, it terminates training and outputs the final global model.
2. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The local data distribution characteristics include category distribution, mean of feature dimensions, and total sample size; The secure key negotiation uses asymmetric key exchange and digital certificate identity binding. Each round of communication uses an independent session key and is accompanied by a message authentication code and digital signature to ensure the confidentiality, integrity and source authentication of the transmission.
3. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The comprehensive heterogeneity integrates three types of heterogeneity: label distribution shift, feature space drift, and sample size imbalance. It uniformly quantifies the degree of difference between each client and the global data distribution. The more abundant the sample size of a client, the lower the heterogeneity score corresponding to the same distribution shift.
4. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, In the gradient direction correction, the higher the data distribution overlap coefficient, the more local directions the gradient retains; the temporal annealing factor gradually enhances the global alignment as training progresses, achieving an adaptive balance of fully utilizing local information from all parties in the early stage of training and rapidly converging towards the global optimum in the later stage.
5. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The dynamic aggregation weight assigns higher weights to clients with high distribution alignment, low heterogeneity, and large sample size, causing global model updates to favor participants with better data quality, accelerating convergence and improving the final model accuracy.
6. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The dynamic differential privacy mechanism replaces the globally unified pruning threshold with the client's own gradient range. The smaller the gradient magnitude, the lower the noise intensity injected into the client. Privacy resources are tilted towards clients whose gradients are consistent with the global consensus and whose data distribution is aligned, thereby reducing the overall accuracy loss under the same level of privacy protection.
7. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The sparsity ratio of the adaptive sparsity compression gradually decreases with each training round. High sparsity in the early stage of training ensures communication efficiency, while low sparsity in the later stage of training ensures convergence accuracy. The cache resending mechanism ensures that all gradient components fully participate in aggregation after multiple iterations, avoiding permanent information loss.
8. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The convergence determination is based on the global validation loss calculated by weighting the local validation loss of each client according to the sample size. If the rate of change of loss is lower than the threshold for multiple consecutive rounds, the training is terminated; if the loss does not improve for multiple consecutive rounds, the global learning rate is dynamically reduced to improve convergence stability.
9. The multi-source heterogeneous data privacy-preserving collaborative training method according to claim 1, characterized in that, The training adopts a centralized aggregation and distributed training architecture, which includes three types of roles: aggregation server, local computing node, and audit and supervision node. The audit and supervision node independently monitors the compliance metadata of the entire training process without touching any original business data.
10. A privacy-preserving collaborative training system for multi-source heterogeneous data based on federated learning, characterized in that, The method described in any one of claims 1 to 9 is used to implement the system, which includes a multi-source data processing module, a secure communication module, a gradient adaptive calibration module, a dynamic differential noise compensation module, a communication compression and transmission module, a global model optimization and convergence module, and a system architecture collaborative scheduling module. The modules work together to achieve high-precision privacy-preserving collaborative training under the premise that multi-source heterogeneous data does not leave the domain.
