Distributed system performance evaluation method based on federated learning and storage medium
Through the multi-dimensional evaluation model and dynamic optimization strategy, the problem of single evaluation dimensions and incomplete privacy protection of the federated learning system is solved, and a comprehensive, real-time and efficient evaluation and optimization of the federated learning system is achieved.
Patent Information
- Application Number
- CN202510436877.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-08
- Publication Date
- 2025-07-25
AI Technical Summary
The performance evaluation method of the federated learning system in the prior art has the problems of single evaluation dimensions, lack of dynamic feedback mechanisms and incomplete privacy protection, and it is impossible to comprehensively measure system performance and dynamic optimization.
A distributed system performance evaluation method based on federated learning is proposed. Through a multi-dimensional evaluation model (computing efficiency, communication overhead, privacy protection capability, node heterogeneity and model accuracy), a comprehensive performance evaluation framework is established, and the system parameters are automatically adjusted through dynamic optimization strategies to improve performance.
It realizes a multi-dimensional comprehensive assessment of the federated learning system, provides real-time monitoring and feedback, dynamically optimizes system configuration, improves system efficiency and optimizes privacy protection, and solves the trade-off between privacy protection and performance.
Smart Images
Figure CN120371668A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and particularly to a method for evaluating the effectiveness of a distributed system based on federated learning and a storage medium. Background Art
[0002] When the federated learning technology is applied to a distributed prediction system, the effectiveness and stability of the system are affected by multiple factors, such as the number of clients, data distribution, aggregation methods, etc. At present, the evaluation of the performance of federated learning systems mostly focuses on single scenarios or specific data sets, lacking a comprehensive multi-factor evaluation mechanism. In the fields of intelligent manufacturing, supply chain optimization, network security, etc., how to measure and optimize the overall performance of federated learning is the key to improving the robustness and prediction accuracy of the system.
[0003] Currently, the effectiveness evaluation methods of most federated learning systems mainly focus on two dimensions: computational efficiency and communication overhead. Computational efficiency is usually measured by calculating the time consumed in each round of training, and communication overhead is calculated by the parameter upload frequency and size. However, these evaluation methods have the following deficiencies:
[0004] 1) Single evaluation dimension: Existing methods usually only consider computational and communication efficiency, while ignoring factors such as privacy protection capabilities, model accuracy, and heterogeneity among nodes, resulting in the evaluation results being unable to fully reflect the actual effectiveness of the system.
[0005] 2) Lack of dynamic feedback mechanism: The current effectiveness evaluations are mostly offline and static. They cannot automatically adjust the evaluation strategy and system configuration in real time according to changes in the system state, resulting in an inability to cope with rapid fluctuations in system performance.
[0006] 3) Incomplete privacy protection evaluation: For technologies that protect data privacy in federated learning (such as differential privacy, homomorphic encryption, etc.), existing evaluation methods fail to fully consider the impact of privacy protection on computational efficiency and communication overhead, lacking a dedicated privacy protection evaluation model.
[0007] Therefore, the existing technologies cannot comprehensively consider the system effectiveness in multiple dimensions and provide dynamic optimization, and cannot meet the requirements for efficiently and comprehensively evaluating the performance of distributed systems. Summary of the Invention
[0008] The present invention proposes a method for evaluating the effectiveness of a distributed system based on federated learning, which solves the problems of single evaluation dimension, lack of feedback mechanism, and imperfect privacy protection existing in the prior art for the method of evaluating the effectiveness of a distributed system based on federated learning. The technical solution of the present invention is implemented as follows:
[0009] A method for evaluating the efficiency of a distributed system based on federated learning includes the following steps: determining multiple evaluation dimensions and establishing corresponding evaluation models, establishing a data collection and node monitoring mechanism to periodically collect the computing consumption, communication overhead, and privacy protection status data of each node, and providing accurate performance evaluation for the system; calculating the efficiency of each evaluation dimension based on the collected data; establishing a comprehensive efficiency evaluation framework, which combines the results of each evaluation dimension to calculate the comprehensive efficiency score of the entire federated learning system, and this score will be used as the basis for system optimization to guide the system to make dynamic adjustments under different operating conditions; determining a dynamic optimization strategy, automatically adjusting the operating parameters of the system according to the real-time efficiency evaluation results of the system, thereby improving the system efficiency, optimizing the privacy protection technology, and reducing the communication overhead and computing time.
[0010] As a preferred technical solution, the multiple evaluation dimensions include computing efficiency, communication overhead, privacy protection ability, model accuracy, and node heterogeneity.
[0011] As a preferred technical solution, define the computing efficiency as the computing resources consumed by each node for local training, and its calculation formula is as follows:
[0012]
[0013] where is the training time of the i-th node, and is the number of nodes participating in the training;
[0014] The communication overhead evaluation formula is as follows:
[0015]
[0016] where S i is the size of the model parameters uploaded by the i-th node, B i is the bandwidth of the node, W node,i is the weight coefficient of the node, reflecting the importance of the computing task of the node in the overall system.
[0017] As a preferred technical solution, by comparing the leakage ratios of the original data and the encrypted data, evaluate the effect of the privacy protection measures. This evaluation model measures the privacy protection ability by calculating the ratio of the leaked data volume to the total data volume owned by the node. Specifically, the evaluation formula for the privacy protection ability is:
[0018]
[0019] where L leaked is the leaked data volume, L total is all the data volume owned by the node, D privacy is the level coefficient of the privacy protection measure, reflecting the strength of the privacy protection technology.
[0020] As a preferred technical solution, the accuracy of the model is evaluated by calculating the error of the model on the test set. Specifically, the accuracy evaluation method usually uses the following formula for calculation:
[0021]
[0022] where y j is the true value, is the predicted value of the model, and N test is the number of samples in the test set.
[0023] As a preferred technical solution, by monitoring the hardware configuration and load conditions of each node, and combining the resource differences between nodes, a heterogeneity evaluation model is established. The impact of the resource differences between nodes on the system performance is quantified through the evaluation model, and the training strategy is adjusted to minimize the performance bottleneck caused by heterogeneity.
[0024] As a preferred technical solution, the real-time collected data includes: computing consumption, communication data, privacy protection status, and node status monitoring.
[0025] As a preferred technical solution, the overall performance score is obtained by weighted average of the results of all evaluation dimensions. The calculation formula is as follows:
[0026] E total = w1·C efficiency + w2·C comm + w3·P privacy + w4·M accuracy
[0027] where w1, w2, w3, and w4 respectively represent the weight coefficients of each dimension, representing the contributions of each evaluation dimension to the overall system performance.
[0028] As a preferred technical solution, the dynamic optimization strategy is as follows: According to the comprehensive performance score, the system will automatically adjust key parameters such as communication frequency, node task assignment, and model update frequency; when the evaluation result shows that the privacy protection is relatively weak, the system ensures privacy security by adding noise and strengthening homomorphic encryption.
[0029] A non-transitory storage medium is used to store a program that executes the above-mentioned method for evaluating the performance of a distributed system based on federated learning.
[0030] Compared with the prior art, the present solution has the following beneficial effects:
[0031] (1) Multi-dimensional performance evaluation: The present invention comprehensively considers multiple dimensions such as computing efficiency, communication overhead, privacy protection ability, node heterogeneity, and model accuracy, providing a more accurate and comprehensive performance evaluation for distributed systems.
[0032] (2) Real-time monitoring and data feedback: By real-time monitoring the node status and feeding back real-time data, the system can dynamically adjust the system configuration to ensure optimal performance under different conditions.
[0033] (3) Intelligent optimization mechanism: The dynamic optimization strategy proposed by the present invention can intelligently adjust the system operation parameters according to the real-time evaluation results, thereby improving the system performance and optimizing the privacy protection measures.
[0034] (4) Efficient privacy protection evaluation: The present invention effectively combines privacy protection with system performance evaluation, proposes an evaluation method that can measure the impact of privacy protection ability on the overall performance, and effectively solves the trade-off problem between privacy protection and performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.
[0036] Figure 1 It is a method flow chart of a method for evaluating the performance of a distributed system based on federated learning according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0037] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.
[0038] Refer to Figure 1 , the present invention provides a method for evaluating the performance of a distributed system based on federated learning. The present invention has been optimized from multiple aspects such as the definition of evaluation dimensions, data collection and node status monitoring, calculation methods for each evaluation dimension, establishment of a comprehensive performance evaluation framework, and application of a dynamic optimization strategy. The following details the technical details and implementation methods of each part:
[0039] 1. Determination of evaluation dimensions and system modeling
[0040] To comprehensively evaluate the performance of a distributed system, this invention introduces multi-dimensional evaluation criteria such as privacy protection capabilities, node heterogeneity, and model accuracy on the basis of traditional computing efficiency and communication overhead. Each dimension evaluates the system from a different perspective, as follows:
[0041] 1.1 Computational efficiency evaluation: Computational efficiency refers to the computing resources consumed by each node during local training, usually measured by the time taken for each round of training. To more accurately reflect the computing load, the evaluation of computational efficiency takes into account multiple factors such as the hardware performance of the node, the amount of data, and the algorithm complexity.
[0042] 1.1.1 Time consumption model: The computing time depends not only on the hardware configuration of the node but also on the size of the training data, the model complexity, and the choice of optimization algorithm. Specifically, the computational efficiency is calculated by the following formula:
[0043]
[0044] where T i is the training time of the i-th node, and N nodes is the number of nodes participating in the training.
[0045] 1.2 Communication overhead evaluation: Communication overhead refers to the bandwidth and time consumed when nodes exchange information. During the federated learning process, each node needs to upload its model updates and receive the global model parameters. The communication overhead depends not only on the size of the uploaded model update parameters but also on factors such as bandwidth, network latency, and upload frequency.
[0046] 1.2.1 Relationship between bandwidth and upload frequency: The evaluation of communication overhead takes into account the bandwidth, parameter size, and upload frequency of each node. The specific evaluation formula is:
[0047]
[0048] where S i is the size of the model parameters uploaded by the i-th node, B i is the bandwidth of the node, and W node,i is the weight coefficient of the node, reflecting the importance of the node's computing task in the overall system.
[0049] 1.3 Privacy protection capability evaluation: A key advantage of federated learning is its ability to protect the data privacy of participants. The evaluation of privacy protection capabilities mainly considers the effectiveness of privacy protection technologies, especially how to balance the trade-off between privacy protection and computing and communication efficiency when using technologies such as differential privacy and homomorphic encryption.
[0050] 1.3.1 Measurement of data leakage volume: By comparing the leakage ratios of the original data and the encrypted data, the effectiveness of privacy protection measures is evaluated. This evaluation model measures the privacy protection ability by calculating the ratio of the leaked data volume to the total data volume owned by the node. The evaluation formula for privacy protection ability is:
[0051]
[0052] Among them, L leaked is the leaked data volume, L total is the total data volume owned by the node, D privacy is the level coefficient of the privacy protection measure, reflecting the strength of the privacy protection technology.
[0053] 1.4 Model accuracy evaluation: Model accuracy is the core indicator for evaluating the effectiveness of a federated learning system. This indicator is measured by calculating the error of the model on the test set, usually using methods such as mean squared error, accuracy, and F1 value. Model accuracy evaluation reflects the performance of the final model in the prediction task. Especially when there are multiple types of nodes in the system, how to maintain a high accuracy through model fusion is the key to the evaluation.
[0054] 1.4.1 Error calculation: The specific accuracy evaluation method is usually calculated using the following formula:
[0055]
[0056] Among them, y j is the true value, is the predicted value of the model, and N test is the number of samples in the test set.
[0057] 1.5 Node heterogeneity evaluation: Since there are differences in the hardware configuration, network bandwidth, storage capacity, etc. of each node in a distributed system, node heterogeneity has an important impact on system performance. The present invention designs a heterogeneity evaluation model by monitoring the hardware configuration (such as CPU, memory, bandwidth) and load conditions of each node and combining the resource differences between nodes. This evaluation model can quantify the impact of resource differences between nodes on system efficiency and minimize the performance bottleneck caused by heterogeneity by adjusting the training strategy.
[0058] 2. Data collection and node status monitoring
[0059] In order to ensure that the system can perceive and analyze the status of each node in real time, the present invention proposes an efficient data collection and node monitoring mechanism. This mechanism can periodically collect data such as the computing consumption, communication overhead, and privacy protection status of each node and provide accurate performance evaluation for the system.
[0060] 2.1 Real-time data collection: The system continuously monitors the following data on each node through a lightweight monitoring module:
[0061] 2.1.1 Computational consumption: The time spent on each round of training, the load situation of the node, and the computational resource utilization rate (CPU, GPU occupancy rate, etc.).
[0062] 2.1.2 Communication data: The size of the model update uploaded by the node, the upload frequency, and the network bandwidth usage.
[0063] 2.1.3 Privacy protection status: The usage of encryption technologies and the monitoring data of data leakage.
[0064] 2.2 Node status monitoring: The monitoring module also continuously tracks the resource usage (CPU, memory, storage) and bandwidth utilization rate of the node, and evaluates the running status of the node. If the node resources are tense, the system will dynamically adjust the task allocation of the node to ensure the effective utilization of resources.
[0065] 3. Efficiency calculation of each evaluation dimension
[0066] Based on the data collected above, the present invention calculates the efficiency of each evaluation dimension through formulas. The calculation formula for each dimension is based on specific data inputs and is adjusted in combination with actual business requirements. For example, the calculation formulas for computing efficiency and communication overhead consider the different configurations and network conditions of the nodes; the privacy protection ability is evaluated based on the strength of differential privacy technology.
[0067] 3.1 Comprehensive efficiency calculation: The results of all evaluation dimensions are finally aggregated, and the overall efficiency score is obtained through weighted average. This comprehensive score can provide a comprehensive evaluation of the system efficiency and provide a basis for system optimization.
[0068] 4. Establishment of a comprehensive efficiency evaluation framework
[0069] The present invention proposes a comprehensive efficiency evaluation framework, which calculates the comprehensive efficiency score of the entire federated learning system by combining the results of each evaluation dimension through a weighted average method. This score will be used as the basis for system optimization to guide the system to make dynamic adjustments under different operating conditions.
[0070] 4.1 Comprehensive efficiency evaluation model: By aggregating the weighted calculation results of the efficiency of each dimension, the comprehensive efficiency score is obtained:
[0071] E total = w1·C efficiency + w2·C comm + w3·P privacy + w4·M accuracy
[0072] Among them, w1, w2, w3, and w4 respectively represent the weight coefficients of each dimension, representing the contributions of each evaluation dimension to the overall system performance.
[0073] 5. Application of Dynamic Optimization Strategy
[0074] The dynamic optimization strategy proposed by the present invention can automatically adjust the operating parameters of the system according to the real-time performance evaluation results of the system, thereby improving the system performance, optimizing the privacy protection technology, and reducing the communication overhead and computing time.
[0075] 5.1 Parameter Tuning: According to the comprehensive performance score, the system will automatically adjust key parameters such as communication frequency, node task allocation, and model update frequency. For example, when the communication overhead is too high, the system will reduce the model update frequency or use compression technology to reduce the size of the uploaded data.
[0076] 5.2 Privacy Protection Optimization: For privacy protection measures, when the evaluation results show that the privacy protection is relatively weak, the system will strengthen the privacy protection mechanism (such as adding noise, strengthening homomorphic encryption, etc.) to ensure privacy security without affecting the overall performance.
[0077] Compared with the prior art, the present application has the following beneficial effects:
[0078] (1) Multi-dimensional Performance Evaluation: The present invention comprehensively considers multiple dimensions such as computing efficiency, communication overhead, privacy protection ability, node heterogeneity, and model accuracy, providing a more accurate and comprehensive performance evaluation for distributed systems.
[0079] (2) Real-time Monitoring and Data Feedback: The system can dynamically adjust the system configuration by real-time monitoring the node status and feedbacking real-time data to ensure the best performance in different situations.
[0080] (3) Intelligent Optimization Mechanism: The dynamic optimization strategy proposed by the present invention can intelligently adjust the system operating parameters according to the real-time evaluation results, thereby improving the system performance and optimizing the privacy protection measures.
[0081] (4) Efficient Privacy Protection Evaluation: The present invention effectively combines privacy protection and system performance evaluation, and proposes an evaluation method that can measure the impact of privacy protection ability on the overall performance, effectively solving the trade-off problem between privacy protection and performance.
[0082] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for evaluating the performance of a distributed system based on federated learning, characterized in that It includes the following steps: Determine multiple evaluation dimensions and establish corresponding evaluation models. Establish a data collection and node monitoring mechanism to periodically collect the computing consumption, communication overhead, and privacy protection status data of each node, and provide accurate performance evaluation for the system. Calculate the effectiveness of each evaluation dimension based on the collected data. Establish a comprehensive effectiveness evaluation framework that combines the results of each evaluation dimension to calculate the comprehensive effectiveness score of the entire federated learning system. This score will be used as the basis for system optimization to guide the system to make dynamic adjustments under different operating conditions. Determine a dynamic optimization strategy. According to the real-time effectiveness evaluation results of the system, automatically adjust the operating parameters of the system to improve system effectiveness, optimize privacy protection technologies, and reduce communication overhead and computing time.
2. The method for evaluating the performance of a distributed system based on federated learning according to claim 1, wherein The multiple evaluation dimensions include computing efficiency, communication overhead, privacy protection ability, model accuracy, and node heterogeneity.
3. The method for evaluating the efficiency of a distributed system based on federated learning according to claim 2, wherein Define the computing efficiency as the computing resources consumed by each node during local training. The calculation formula is as follows: where is the training time of the i-th node, and N nodes is the number of nodes participating in the training; The communication overhead evaluation formula is as follows: Among them, S i is the size of the model parameters uploaded by the i-th node, B i is the bandwidth of the node, W node,i is the weight coefficient of the node, reflecting the importance of the computing task of the node in the overall system.
4. A method for evaluating the performance of a distributed system based on federated learning according to claim 2, characterized in that, By comparing the leakage ratios of the original data and the encrypted data, evaluate the effectiveness of the privacy protection measures. This evaluation model measures the privacy protection ability by calculating the ratio of the leaked data volume to the total data volume owned by the node. Specifically, the evaluation formula for privacy protection ability is: Among them, L leaked is the amount of leaked data, and L total is the total amount of data owned by the node, and D privacy is the level coefficient of privacy protection measures, reflecting the strength of privacy protection technology.
5. The method for evaluating the performance of a distributed system based on federated learning according to claim 2, wherein Evaluate the model accuracy by calculating the error of the model on the test set. The specific accuracy evaluation method is usually calculated using the following formula: Among them, y j is the true value, is the predicted value of the model, and N test is the number of samples in the test set.
6. The method for evaluating the effectiveness of a distributed system based on federated learning according to claim 2, wherein By monitoring the hardware configuration and load conditions of each node and combining the resource differences between nodes, establish a heterogeneity evaluation model, and quantify the impact of the resource differences between nodes on the system effectiveness through the evaluation model. Minimize the performance bottleneck caused by heterogeneity by adjusting the training strategy.
7. A method for evaluating the effectiveness of a distributed system based on federated learning according to claim 1, characterized in that, The real-time collected data includes: computing consumption, communication data, privacy protection status, and node status monitoring.
8. A method for evaluating the performance of a distributed system based on federated learning according to claim 1, wherein The overall effectiveness score is obtained by weighted average of the results of all evaluation dimensions. The calculation formula is as follows: E total = w1·C efficiency + w2·C comm + w3·P privacy + w4·M accuracy Among them, w1, w2, w3, and w4 respectively represent the weight coefficients of each dimension, representing the contributions of each evaluation dimension to the overall system effectiveness.
9. A method for evaluating the effectiveness of a distributed system based on federated learning according to claim 1, characterized in that, The dynamic optimization strategy is as follows: According to the comprehensive effectiveness score, the system will automatically adjust key parameters such as communication frequency, node task allocation, and model update frequency. When the evaluation results show that privacy protection is relatively weak, the system ensures privacy security by adding noise and strengthening homomorphic encryption.
10. A non-transitory storage medium, characterized in that, It is used to store a program that executes a method for evaluating the effectiveness of a distributed system based on federated learning as described in any one of claims 1 to 9 above.