Multi-target automatic hyper-parameter configuration method and system for distributed machine learning

Abstract

The invention discloses a distributed machine learning-oriented multi-target automatic hyper-parameter configuration method and system, which are used for simultaneously optimizing model precision, training efficiency, safety and bandwidth efficiency in single training. The method comprises the steps that gradient noise scale (GNS) is used as a unified index, and a mathematical model between hyper-parameters and multi-target performance is established; training index data and GNS are monitored in real time in the training process, and hyper-parameters are dynamically adjusted; hyper-parameter optimization in single training is realized through lightweight analysis and closed optimization; a user-friendly API is provided, and the configuration process is simplified. According to the method, an efficient and safe automatic hyper-parameter configuration solution is provided, the overall performance of distributed machine learning is remarkably improved, experiments show that the training time of distributed machine learning can be remarkably shortened, and meanwhile model precision and safety are improved.

Description

Technical Field

[0001] This invention relates to the field of automated hyperparameter optimization technology in machine learning, specifically to the field of multi-objective automated hyperparameter optimization technology in distributed machine learning (Geo-Distributed ML) and federated learning (FL) scenarios, and particularly to a multi-objective automated hyperparameter configuration method and system for distributed machine learning. Background Technology

[0002] Distributed machine learning, as a scalable paradigm for large-scale model training, significantly improves model training speed and overcomes the limitations of single-machine resources by distributing computational tasks across multiple computing nodes for parallel execution. In this architecture, the distributed storage characteristics of training data and the collaborative scheduling mechanism of computing resources have a crucial impact on the model's convergence speed, resource utilization, and system robustness.

[0003] Hyperparameters play a crucial role in distributed machine learning. They are set before training begins and influence model behavior and performance, including learning rate, batch size, number of training epochs, and model-specific configurations such as the number of layers in a neural network. Proper hyperparameter configuration is essential for efficient distributed machine learning, but existing work faces several challenges in automating hyperparameter configuration, limiting the efficiency and security of distributed machine learning in practical applications: ① Lack of systematic guidance: Existing hyperparameter configuration methods often rely on expert experience and grid search, lacking systematic mathematical model support and failing to fully consider the trade-offs between model performance, resource consumption, and data security. ② Single-objective optimization: Existing work typically focuses only on a single performance objective (such as model accuracy), neglecting the combined impact of hyperparameters on training efficiency, security, and bandwidth efficiency, leading to performance imbalances. ③ Insufficient security considerations: Existing hyperparameter configuration methods mostly focus on model performance and resource utilization efficiency, ignoring the impact of hyperparameter configuration on data security, potentially leading to security risks such as data leakage. ④ Poor dynamic adaptability: Static hyperparameter configuration cannot adapt to changes in gradient noise during training, making it difficult to achieve optimal performance balance.

[0004] Therefore, there is an urgent need for a multi-objective automatic hyperparameter configuration method and system for distributed machine learning to address the aforementioned shortcomings. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-objective automatic hyperparameter configuration method and system for distributed machine learning. This invention can optimize multi-objective performance while protecting data privacy.

[0006] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a multi-objective automatic hyperparameter configuration method for distributed machine learning, applicable to distributed machine learning scenarios. The method includes the following steps: Step 1: Construct a mathematical model: Establish a quantitative relationship model between hyperparameters and performance objectives as a mathematical model; among which, performance objectives include system throughput, statistical efficiency, security, and bandwidth efficiency; Step 2, Dynamic Hyperparameter Tuning: During training, training metric data is collected in real time to monitor gradient noise scale and dynamically adjust hyperparameters to balance multi-objective performance. Step 3, Communication Optimization: Filter gradient noise scales through statistical efficiency thresholds to reduce low-value gradient transmission; Step 4: Simplified User Configuration: Provide an API interface to allow users to start an automated hyperparameter optimization process through simple configuration.

[0007] Furthermore, step one specifically includes the following sub-steps: (a1) The training time per iteration is modeled by analyzing each stage of the forward and backward propagation processes in each training iteration. The training efficiency is quantified by the number of training samples processed per unit time, which is used as the system throughput. The calculation formula is as follows:

[0008]

[0009]

[0010] In the formula, Let M represent the system throughput in the t-th iteration of training, and M represent the batch size in the hyperparameters. This represents the total forward propagation time. This indicates the time it takes for participant B to perform reasoning. This indicates the time when participant B transmits the intermediate results to participant A. This indicates the time it takes for participant A to calculate the loss based on the intermediate results. This represents the total time for backpropagation. This represents the time it takes for participant A to calculate the gradient based on the loss. This indicates the time it takes for participant A to transfer the gradient to participant B. This indicates the time it takes for participant B to update its local model using gradients. These are parameters used to balance transmission overlap time; (a2) Statistical efficiency is calculated based on gradient noise scaling to quantify the progress of a single iteration of distributed machine learning and obtain the progress of model training in each round; the formula for calculating statistical efficiency is:

[0011] In the formula, This represents the statistical efficiency of the t-th iteration of training. Indicates the gradient noise scale. Indicates the user-configured baseline batch size; (a3) The attack effect of the attack model is calculated based on the gradient noise scale to quantify security. The calculation formula is as follows:

[0012] In the formula, This indicates the safety of the t-th iteration of training. Parameters indicating the degree of importance attached to safety; (a4) Calculate bandwidth efficiency using the bandwidth usage in each training iteration. The calculation formula is as follows:

[0013]

[0014] In the formula, This represents the bandwidth efficiency of the t-th iteration of training. This represents the average bandwidth cost per sample. For indicator functions, Indicates the statistical efficiency threshold; (a5) Based on the above system throughput, statistical efficiency, security, and bandwidth efficiency, a mathematical model is constructed, the expression of which is:

[0015] In the formula, Let represent the mathematical model for the t-th iteration of training.

[0016] Furthermore, step two specifically includes the following sub-steps: (b1) Real-time acquisition of training metric data during each round of training iterations, including batch size, training time, and gradient, and calculation of gradient noise scale based on gradient; (b2) Based on the training index data and gradient noise scale fitting mathematical model collected in step (b1), calculate the batch size when the mathematical model reaches its maximum value, and set it as the batch size for the next round of training iterations; after adjusting the batch size, adjust the learning rate synchronously based on the square root learning rate scaling rule.

[0017] Furthermore, the formula for calculating the gradient noise scale is as follows:

[0018] In the formula, This represents the gradient noise scale in the t-th iteration of training. Let be the covariance matrix of the gradient. The covariance matrix of the gradient traces, This represents the gradient.

[0019] Furthermore, step four specifically includes: When the gradient noise scale is less than the statistical efficiency threshold, skip the remaining gradient aggregation and downstream training steps in this round of training, that is, skip the update of the model in this round and start the next round of training directly, so as to reduce the transfer of low-value gradients.

[0020] Furthermore, the API interface supports the following configurations: initial batch size, security priority parameter, and statistical efficiency threshold.

[0021] A second aspect of the present invention provides a multi-objective automatic hyperparameter configuration system for distributed machine learning, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described multi-objective automatic hyperparameter configuration method for distributed machine learning.

[0022] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described multi-objective automatic hyperparameter configuration method for distributed machine learning.

[0023] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention constructs a quantitative relationship model between hyperparameters and multi-objective performance based on gradient noise scale (GNS), comprehensively considering model performance, computational cost, communication cost and data security.

[0024] (2) The present invention collects and calculates the training time, gradient noise scale, statistical efficiency and communication bandwidth of each stage during the training process, thereby realizing real-time performance monitoring of the training process.

[0025] (3) This invention combines the quantization model with the collected data to realize the function of dynamic adjustment of hyperparameters, and simultaneously optimizes the model accuracy, training efficiency, security and bandwidth efficiency in a single training session.

[0026] (4) The present invention uses API interface to simplify the configuration process and realizes the hyperparameter tuning setting function at the user level, so that users with different technical levels can easily configure hyperparameters.

[0027] (5) This invention achieves real-time performance monitoring and dynamic hyperparameter adjustment of the training layer by continuously measuring the gradient noise scale of each iteration and the time required for each process, so as to optimize the performance of federated learning jobs. Attached Figure Description

[0028] Figure 1 This is a flowchart of the multi-objective automatic hyperparameter configuration method for distributed machine learning according to the present invention; Figure 2 This is an attack flowchart in the distributed machine learning of this invention; Figure 3 This is the logical process of fitting parameters and adapting hyperparameters in the training layer of this invention; Figure 4 This is a schematic diagram of a multi-objective automatic hyperparameter configuration system for distributed machine learning according to the present invention. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0032] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0033] This invention relates to a multi-objective automatic hyperparameter configuration method and system for distributed machine learning. Applicable to distributed machine learning scenarios, it aims to improve efficiency and security in distributed machine learning by optimizing hyperparameter configuration through mathematical modeling and analysis to balance model performance, computational cost, communication cost, and security. Figure 1 As shown, the specific implementation steps include: Step 1: Construct a mathematical model: Establish a quantitative relationship model between hyperparameters and performance objectives as the mathematical model. The performance objectives include system throughput, statistical efficiency, security, and bandwidth efficiency.

[0034] Furthermore, corresponding quantitative relationship models are established for training time (corresponding to system throughput), statistical efficiency, security, and communication bandwidth (corresponding to bandwidth efficiency) in distributed machine learning, thereby completing the construction of the mathematical model. Taking a two-party distributed machine learning (A and B jointly perform machine learning, where A is the active party and holds the training data labels, and B holds the training data features) as an example, the specific steps may include the following: (a1) The training time per iteration is modeled by analyzing each stage of the forward and backward propagation processes in each training iteration. The training efficiency is quantified by the number of training samples processed per unit time, which is used as the system throughput. The calculation formula is as follows:

[0035]

[0036]

[0037] In the formula, Let M represent the system throughput in the t-th iteration of training, and M represent the batch size in the hyperparameters. This represents the total forward propagation time. This indicates the time it takes for participant B to perform reasoning. This indicates the time when participant B transmits the intermediate results to participant A. This indicates the time it takes for participant A to calculate the loss based on the intermediate results. This represents the total time for backpropagation. This represents the time it takes for participant A to calculate the gradient based on the loss. This indicates the time it takes for participant A to transfer the gradient to participant B. This indicates the time it takes for participant B to update its local model using gradients. The parameters required for equalizing transmission overlap time are as follows: Considering that computation and transmission may overlap, a value is used to approximate the degree of overlap when... When equals the sum of the two items inside the parentheses, when As it approaches infinity, it equals the larger of the two values ​​within the parentheses. Each of the above T represents the time required for one training phase in the forward and backward propagation processes, and these times are each represented by a linear function of the batch size M, i.e. All the parameters that need to be fitted in the formula can be simply defined as a quadruple. , where * represents , , , , and , representing each of the training phases mentioned above.

[0038] (a2) Statistical efficiency is calculated based on Gradient Noise Scale (GNS) to quantify the progress of a single iteration of distributed machine learning, obtain the progress of the model in each round of training, and approximate the accuracy improvement in each round of training. The formula for calculating statistical efficiency is:

[0039] In the formula, This represents the statistical efficiency of the t-th iteration of training. Indicates the gradient noise scale. This represents the user-configured baseline batch size, which is a constant value, while M represents the batch size to be adjusted.

[0040] (a3) See also Figure 2 Based on experiments and analysis of common attack methods in distributed machine learning, this paper summarizes and abstracts the process of common attack methods. This process includes the main characteristics of widely popular attack algorithms such as label inference attacks, feature inference attacks, and model extraction attacks, guiding the quantification of model security. Specifically, the attack effect of the attack model can be calculated based on GNS to quantify security. The calculation formula is as follows:

[0041] In the formula, This indicates the safety of the t-th iteration of training. This indicates an adjustable safety priority parameter. .

[0042] (a4) To quantify the impact of gradient filtering on WAN bandwidth consumption, bandwidth efficiency is introduced to describe the bandwidth usage in a single iteration. Specifically, bandwidth efficiency can be calculated using the bandwidth usage in each training iteration, and the calculation formula is as follows:

[0043]

[0044] In the formula, This represents the bandwidth efficiency of the t-th iteration of training. This represents the average bandwidth cost per sample. For indicator functions, The statistical efficiency threshold represents the minimum statistical efficiency threshold required to trigger communication. The statistical efficiency term... This reflects the informational value of the current step.

[0045] Furthermore, to make the model smooth and differentiable, thus suitable for adaptive parameter tuning, a smooth approximation function (such as the sigmoid function) can be used to replace the indicator function, resulting in an improved bandwidth-efficient model, whose expression is:

[0046] In the formula, This represents the Sigmoid function. These are parameters used to control the sharpness of the filtering boundary. The improved bandwidth efficiency model allows for end-to-end differentiability optimization.

[0047] (a5) Based on the above system throughput, statistical efficiency, security, and bandwidth efficiency, a mathematical model is constructed, the expression of which is:

[0048] In the formula, This represents the mathematical model for the t-th iteration of training. This represents the system throughput during the t-th round of training iterations. This represents the statistical efficiency of the t-th iteration of training. This indicates the safety of the t-th iteration of training. Let M represent the bandwidth efficiency of the t-th iteration of training, and M represent the batch size in the hyperparameters.

[0049] Step 2, Dynamic Hyperparameter Tuning: During training, training metric data is collected in real time to monitor GNS and dynamically adjust hyperparameters (such as batch size and learning rate) to balance multi-objective performance.

[0050] It's important to note that during the first round of training, i.e., at the very beginning, there's a lack of sufficient historical data for model-based optimization. Therefore, process initialization is necessary. This can be achieved using heuristics, gradually increasing the initial batch size by adding a small positive constant x to stimulate learning dynamics and collect necessary data. Heuristics refer to a class of problem-solving methods that, within limited time, information, or resources, utilize empirical rules, intuition, and common sense to find a sufficiently good (but not necessarily optimal) solution. Hyperparameter dynamic tuning specifically includes the following sub-steps: (b1) After the user-configured process starts, the training index data of each round of training iteration is collected in real time, including but not limited to batch size, training time, gradient and other training index data. That is, data such as batch size, training time, gradient and other data are collected in each round of training. Specifically, a data collection module can be used to implement this through a callback mechanism and calculate GNS based on gradient. In this way, the necessary data can be collected and calculated. These training index data will be recorded and used to fit the mathematical model in subsequent modules.

[0051] The formula for calculating GNS is as follows:

[0052] In the formula, This represents the gradient noise scale in the t-th iteration of training. Let be the covariance matrix of the gradient. The covariance matrix of the gradient traces, This represents the gradient.

[0053] (b2) Based on the training index data collected in step (b1) and the GNS fitting mathematical model, calculate the batch size M when the mathematical model reaches its maximum value, and set it as the batch size for the next round of training iterations; after adjusting the batch size, adjust the learning rate synchronously based on the square root learning rate scaling rule to further optimize the training effect.

[0054] Specifically, such as Figure 3 As shown, after data collection is complete, hyperparameters are dynamically adjusted based on the training results of each round to improve model performance, training efficiency, and security. First, the pre-established quantized mathematical model is fitted using the training metric data collected in step (b1) and the GNS (Generative Navier-Stokes System), and the optimal batch size is derived based on this mathematical model for application in subsequent iterations. Specifically, the time parameters are fitted based on the training metric data and the GNS. This allows for the prediction of throughput in each round using a mathematical model, combined with the prediction gradient noise scale. and initial batch size The statistical efficiency, security, and bandwidth efficiency of each round are predicted, and these four factors jointly predict the overall performance of the task. After each training round, based on the collected training time, batch size, gradient, and gradient noise scale, a system throughput model is fitted. Calculate the statistical efficiency model, security model, and bandwidth efficiency model. And combined with the pre-configured initial batch size Safety importance level parameter Average bandwidth cost per sample and statistical efficiency threshold These parameters form a function with batch size M as the independent variable. By calculating the value of M when this function is maximized, the optimal hyperparameter values ​​are predicted, rounded, and applied to subsequent training iterations. This process is repeated from the second round until the optimal batch size is determined. Simultaneously, the learning rate is adjusted in tandem with a square root learning rate scaling rule, i.e., the learning rate is scaled by the square root of the gradient scaling ratio.

[0055] Step 3, Communication Optimization: Filter GNS by statistical efficiency threshold to reduce low-value gradient transmissions and lower bandwidth consumption.

[0056] Specifically, when the GNS is less than the statistical efficiency threshold, the remaining gradient aggregation and downstream training steps in that round of training (i.e., the gradients or intermediate results calculated by each node are transmitted to the central node for aggregation and then transmitted back to each node to update the local model) are skipped. This means skipping the model update for that round and directly starting the next round of training. This effectively reduces redundant communication, reduces the transmission of low-value gradients, and lowers bandwidth consumption. More specifically, model update control is needed during training. A statistical efficiency threshold is used to mitigate the additional time and bandwidth overhead in distributed machine learning; that is, a significance threshold controls the transmission of gradients and model updates, thereby reducing time and bandwidth consumption. Before each gradient aggregation step, the collected and calculated gradient noise scale is compared with a dynamic threshold: by evaluating the GNS of the local gradients, if the GNS is lower than the statistical efficiency threshold dynamically determined by the current batch, the gradient aggregation and downstream training stages for that iteration are skipped, i.e., this model update is skipped. As training progresses, the gradient noise scale gradually increases, and the optimal batch size for model computation also increases accordingly. The model benefits from gradients with greater information content, at which point gradient propagation and aggregation are resumed. For large-batch scenarios, a larger noise scale allows the model to better utilize massive amounts of data. This method avoids redundant computation and significantly reduces WAN bandwidth consumption (especially in the early stages of training or inefficient phases) by filtering out low-value gradient steps.

[0057] Step 4: Simplified User Configuration: Provide an API interface to allow users to start an automated hyperparameter optimization process through simple configuration.

[0058] Furthermore, the API interface supports the following configuration: initial batch size. Safety importance level parameter and statistical efficiency threshold .

[0059] Corresponding to the aforementioned embodiments of the multi-objective automatic hyperparameter configuration method for distributed machine learning, the present invention also provides embodiments of a multi-objective automatic hyperparameter configuration system for distributed machine learning.

[0060] See Figure 4The present invention provides a multi-objective automatic hyperparameter configuration system for distributed machine learning, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-objective automatic hyperparameter configuration method for distributed machine learning in the above embodiment.

[0061] The embodiments of the multi-objective automatic hyperparameter configuration system for distributed machine learning of this invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical system, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities where the multi-objective automatic hyperparameter configuration system for distributed machine learning of this invention is located. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in which the system is located in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0062] The implementation process of the functions and roles of each unit in the above system is detailed in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0063] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0064] Furthermore, the system output includes: dynamically adjusted batch size and learning rate; model accuracy (expressed as accuracy measured in real time), security (expressed as the effect of the attack algorithm, i.e., the accuracy and loss of the attack model), and bandwidth efficiency (expressed as bandwidth consumption metrics) during the training process; and performance optimization results presented in a visual form.

[0065] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the multi-objective automatic hyperparameter configuration method for distributed machine learning described in the above embodiments.

[0066] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-objective automatic hyperparameter configuration method for distributed machine learning, characterized in that, Applicable to distributed machine learning scenarios, the method includes the following steps: Step 1: Construct a mathematical model: Establish a quantitative relationship model between hyperparameters and performance objectives as a mathematical model; among which, performance objectives include system throughput, statistical efficiency, security, and bandwidth efficiency; Step 2, Dynamic Hyperparameter Tuning: During training, training metric data is collected in real time to monitor gradient noise scale and dynamically adjust hyperparameters to balance multi-objective performance. Step 3, Communication Optimization: Filter gradient noise scales through statistical efficiency thresholds to reduce low-value gradient transmission; Step 4: Simplified User Configuration: Provide an API interface to allow users to start an automated hyperparameter optimization process through simple configuration.

2. The multi-objective automatic hyperparameter configuration method for distributed machine learning according to claim 1, characterized in that, Step one specifically includes the following sub-steps: (a1) The training time per iteration is modeled by analyzing each stage of the forward and backward propagation processes in each training iteration. The training efficiency is quantified by the number of training samples processed per unit time, which is used as the system throughput. The calculation formula is as follows: ； ； ； In the formula, Let M represent the system throughput in the t-th iteration of training, and M represent the batch size in the hyperparameters. This represents the total forward propagation time. This indicates the time it takes for participant B to perform reasoning. This indicates the time when participant B transmits the intermediate results to participant A. This indicates the time it takes for participant A to calculate the loss based on the intermediate results. This represents the total time for backpropagation. This represents the time it takes for participant A to calculate the gradient based on the loss. This indicates the time it takes for participant A to transfer the gradient to participant B. This indicates the time it takes for participant B to update its local model using gradients. These are parameters used to balance transmission overlap time; (a2) Statistical efficiency is calculated based on gradient noise scaling to quantify the progress of a single iteration of distributed machine learning and obtain the progress of model training in each round; the formula for calculating statistical efficiency is: ； In the formula, This represents the statistical efficiency of the t-th iteration of training. Indicates the gradient noise scale. Indicates the user-configured baseline batch size; (a3) The attack effect of the attack model is calculated based on the gradient noise scale to quantify security. The calculation formula is as follows: ； In the formula, This indicates the safety of the t-th iteration of training. Parameters indicating the degree of importance attached to safety; (a4) Calculate bandwidth efficiency using the bandwidth usage in each training iteration. The calculation formula is as follows: ； ； In the formula, This represents the bandwidth efficiency of the t-th iteration of training. This represents the average bandwidth cost per sample. For indicator functions, Indicates the statistical efficiency threshold; (a5) Based on the above system throughput, statistical efficiency, security, and bandwidth efficiency, a mathematical model is constructed, the expression of which is: ； In the formula, Let represent the mathematical model for the t-th iteration of training.

3. The multi-objective automatic hyperparameter configuration method for distributed machine learning according to claim 1, characterized in that, Step two specifically includes the following sub-steps: (b1) Real-time acquisition of training metric data during each round of training iterations, including batch size, training time, and gradient, and calculation of gradient noise scale based on gradient; (b2) Based on the training index data and gradient noise scale fitting mathematical model collected in step (b1), calculate the batch size when the mathematical model reaches its maximum value, and set it as the batch size for the next round of training iterations; after adjusting the batch size, adjust the learning rate synchronously based on the square root learning rate scaling rule.

4. The multi-objective automatic hyperparameter configuration method for distributed machine learning according to claim 3, characterized in that, The formula for calculating the gradient noise scale is: ； In the formula, This represents the gradient noise scale in the t-th iteration of training. Let be the covariance matrix of the gradient. The covariance matrix of the gradient traces, This represents the gradient.

5. The multi-objective automatic hyperparameter configuration method for distributed machine learning according to claim 1, characterized in that, Step four specifically includes: When the gradient noise scale is less than the statistical efficiency threshold, skip the remaining gradient aggregation and downstream training steps in this round of training, that is, skip the update of the model in this round and start the next round of training directly, so as to reduce the transfer of low-value gradients.

6. The multi-objective automatic hyperparameter configuration method for distributed machine learning according to claim 1, characterized in that, The API interface supports the following configurations: initial batch size, security priority parameter, and statistical efficiency threshold.

7. A multi-objective automatic hyperparameter configuration system for distributed machine learning, comprising one or more processors and memory, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-objective automatic hyperparameter configuration method for distributed machine learning as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the multi-objective automatic hyperparameter configuration method for distributed machine learning as described in any one of claims 1-6.

Images