Distributed training method, system, medium and device based on client autonomy

Abstract

The present disclosure provides a client autonomy-based distributed training method, system, medium and device, wherein the client autonomy-based distributed training method comprises: determining the characteristics of the client local training, the characteristics of the client local training including system characteristics and statistical characteristics; determining the local training process of the client by adopting an early stopping strategy according to the characteristics of the client local training; and optimizing the communication process of the local training of the client by adopting an error feedback-based hierarchical early transmission strategy according to the statistical characteristics of the client local training. Through the present disclosure, the state change in the client is perceived and the autonomous decision of the client is realized based on the guidance of the characteristics of the client local training, the computing resources are saved, the model training efficiency is improved, the computing-communication overlap time is increased, and the resource utilization is improved.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and more specifically, to a distributed training method, system, medium, and device based on client autonomy. Background Technology

[0002] Federated learning is a privacy-preserving distributed model training architecture that allows multiple edge devices to jointly train a global model without leaking local data. The most common algorithm for federated learning is FedAvg, and the entire process involves training multiple rounds until the model converges. Each client round of training repeats the following process: First, at the beginning of each round, participating clients obtain the latest global model and update their local models; then, each client performs multiple calculations based on local mini-batch samples and updates its local model; finally, after the local training round is completed, the client uploads its local model updates to the server, which aggregates all updates and updates the global model.

[0003] Low training efficiency is one of the most significant problems faced by real-world federated learning applications. Edge devices participating in federated learning (such as mobile phones and IoT devices) typically have heterogeneous hardware resources and limited network bandwidth, and these resources and bandwidth are often dynamically changing. This affects the computation and communication time of the devices, thus impacting the overall training process. In synchronous federated learning architectures, devices with slow computation or communication can slow down the entire training process, severely impacting training efficiency. Asynchronous federated learning systems usually set a tolerance interval, where the server addresses the performance issues caused by device heterogeneity by accepting outdated updates from outside the current round within the interval. However, outdated information can affect the convergence speed and performance of the global model.

[0004] To address the challenges posed by heterogeneity in computation and communication, several optimization methods exist. For example, one existing technique, FedProx, is proposed as a generalization and reparameterization method of FedAvg to address the difficulties brought by heterogeneity in federated learning. In each training round, FedProx adds a proximate term to the local optimization objective of each client. For instance, on the k-th device participating in training, its local optimization objective is changed from F... k (w) becomes Where w represents the initial model parameters, w t Let μ represent the model parameters in the t-th round, and μ represent the algorithm hyperparameters. Adding proximal terms can reduce the impact of heterogeneity in two ways: (1) tolerating the impact of system heterogeneity, for example, allowing different training progress on the device; (2) ensuring that local updates do not deviate too much from the initial global model, thus mitigating the impact of statistical heterogeneity of local client data. At the same time, FedProx has theoretically proven that this method can guarantee model convergence performance.

[0005] While FedProx simulates device system heterogeneity by setting different training epochs and experimentally verified the algorithm's tolerance to this heterogeneity, it does not provide an effective strategy for selecting appropriate training epochs. Furthermore, the training epochs only address coarse-grained workload control on the client side; however, the client state changes not only between epochs but also rapidly within each epoch, and the algorithm ignores the potential for finer-grained control optimization within each epoch. Additionally, the algorithm can negatively impact the model's convergence speed to some extent.

[0006] Existing Technology 2: FedAda dynamically and adaptively allocates workload to clients participating in training by combining device system performance and local data statistical characteristics. Specifically, FedAda uses training time to represent device system characteristics and local loss values ​​to represent the statistical characteristics of local data. In each round, the server dynamically allocates different mini-batch iteration training times to each client based on the training time and local loss values ​​reported by the clients. FedAda defines a multi-objective optimization formula for the algorithm: Where T represents the sum of the computation time and communication time for each client in each round, T idle This represents the idle time that the client that completes training faster waits for all other clients to finish. Let T represent the maximum value in the initial T, while C is the value obtained after applying the sigmoid function to the difference in local loss between the two rounds. Therefore, C sum This represents the sum of all clients C, and the sum of C. max This represents the maximum value. FedAda does not directly solve for the optimization objective. Instead, it starts with a set initial number of iterations, continuously reduces the number of iterations for each client, and calculates the value of P until P is minimized. After calculation, FedAda usually allocates fewer iterations to clients that train more slowly in each round so that all clients can complete training in a similar amount of time.

[0007] While FedAda can adaptively and dynamically adjust the number of iterations per round for each client by balancing system and data heterogeneity, it still has certain limitations. FedAda relies on the server for optimization decisions, and the server typically only receives the state data after each client training round, unable to perceive the dynamic changes in the client's training state within each round. However, these changes are crucial; for example, a client's computing power might decrease due to high load or power-saving mode. If such changes could be detected, strategies could be implemented to prevent that client from slowing down the overall training process. Furthermore, the convergence speed of local model parameters varies across clients. Clients can optimize computation-communication time overlap by pre-transmitting convergence parameters, further improving training efficiency. Summary of the Invention

[0008] In view of the deficiencies in the prior art, the purpose of this disclosure is to provide a distributed training method, system, medium and device based on client autonomy.

[0009] According to one aspect of this disclosure, a distributed training method based on client autonomy is provided, comprising:

[0010] Determine the characteristics of client-side local training, including system characteristics and statistical characteristics;

[0011] Based on the characteristics of the client's local training, an early stopping strategy is adopted to determine the local training process of the client;

[0012] Based on the statistical characteristics of the client's local training, a hierarchical advance transmission strategy based on error feedback is adopted to optimize the communication process of the client's local training.

[0013] Optionally, determining the features of client-side local training includes:

[0014] The cumulative gradient of the client after each iteration of training within the sampling training round is sampled using a periodic sampling method;

[0015] Based on the cumulative gradient of the client during the sampling training rounds, the statistical characteristics of the client's local training are determined by calculating the statistical progress of the model layer by layer and as a whole.

[0016] Optionally, determining the features of local training on the client further includes:

[0017] The system characteristics of the client's local training are represented by the time it takes for the client to complete each iteration in each round of training.

[0018] Optionally, the step of deciding on the local training process of the client based on the characteristics of the client's local training using an early stopping strategy includes:

[0019] The marginal benefit generated by the client in each iteration of each training round is quantified to determine the marginal benefit of the client in each iteration of each training round.

[0020] The marginal cost of the client completing each iteration in each training round is quantified to determine the marginal cost of the client completing each iteration in each training round.

[0021] Based on the marginal benefit of the client completing each iteration in each training round and the marginal cost of the client completing each iteration in each training round, determine the net benefit of the client completing each iteration in each training round.

[0022] Based on the net gain of the client after each iteration in each training round, the client autonomously decides whether to terminate the training round early.

[0023] Optionally, the step of the client autonomously deciding whether to terminate the training round early based on the net gain of the client after each iteration in each training round includes:

[0024] If the net gain of the client after completing this iteration in the training round is negative, the client will terminate the training round in advance.

[0025] If the net gain of the client after completing this iteration in the training round is non-negative, the client continues the training round.

[0026] Optionally, optimizing the communication process of the client's local training using a hierarchical advance transmission strategy based on error feedback, based on the statistical characteristics of the client's local training, includes:

[0027] In each round of training on the client, the converged layer parameters of the client are transmitted to the server in advance.

[0028] When the client completes each round of training, the cosine similarity is determined based on the cumulative gradient snapshot of the client during the advance transmission and the corresponding cumulative gradient after the client completes each round of training.

[0029] If the cosine similarity is less than a preset first threshold, after the client completes the training round, the hierarchical parameters transmitted in advance in the training round will be re-uploaded to the server.

[0030] Optionally, the step of transmitting the converged layer parameters of the client to the server in advance during each round of training on the client side includes:

[0031] If the statistical characteristic index value of the same number of iterations in adjacent periodically sampled training rounds is not less than a preset second threshold when the hierarchical parameters of the client complete each iteration in each training round, it is determined that the hierarchical parameters of the client have reached a convergence state when completing that iteration in that training round.

[0032] The client uploads the level parameters that reach convergence at each iteration in each training round to the server in advance.

[0033] According to a second aspect of this disclosure, a distributed training system based on client autonomy is provided, comprising:

[0034] A feature determination module is used to determine the features of client-side local training, wherein the features of client-side local training include system features and statistical features;

[0035] The first optimization module is used to decide the local training process of the client based on the characteristics of the client's local training and by adopting an early stopping strategy.

[0036] The second optimization module is used to optimize the communication process of the client's local training by adopting a hierarchical advance transmission strategy based on error feedback, according to the statistical characteristics of the client's local training.

[0037] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method provided in the first aspect of this disclosure.

[0038] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0039] A memory on which computer programs are stored;

[0040] A processor for executing the computer program in the memory to implement the steps of the method provided in the first aspect of this disclosure.

[0041] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects:

[0042] Through the above technical solutions, the characteristics of client-side local training are determined, including system characteristics and statistical characteristics, enabling rapid perception of changes in client state within training rounds. Guided by these local training characteristics, the client adopts an early stopping strategy, autonomously deciding whether to stop the current training round. This saves computational resources and reduces the impact of slower-training clients on the global training process, thereby improving training efficiency, while minimizing the impact on model accuracy. Furthermore, guided by these local training characteristics, the client employs an error feedback-based layer-by-layer early transmission strategy. This strategy transmits converged layer parameters in advance to increase computation-communication overlap time, and introduces an error feedback mechanism to reduce the impact of early transmission on global accuracy, thereby improving resource utilization and enhancing client-side local training efficiency. Attached Figure Description

[0043] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0044] Figure 1 This is a flowchart illustrating a distributed training method based on client autonomy according to an exemplary embodiment.

[0045] Figure 2 This is a schematic diagram illustrating the change of gradient vectors during local training on a client side, according to an exemplary embodiment.

[0046] Figure 3 This is a flowchart illustrating a method for determining features of client-side local training according to an exemplary embodiment.

[0047] Figure 4 This is a flowchart illustrating a method for making decisions on the local training process of a client using an early stopping strategy, according to an exemplary embodiment.

[0048] Figure 5 This is a schematic diagram illustrating an error feedback-based advance transmission strategy according to an exemplary embodiment.

[0049] Figure 6 This is a flowchart illustrating a method for optimizing the communication process of client-side local training using a hierarchical advance transmission strategy based on error feedback, according to an exemplary embodiment.

[0050] Figure 7 This is a flowchart illustrating a method for pre-transmitting converged hierarchical parameters according to an exemplary embodiment.

[0051] Figure 8 This is a schematic diagram of the statistical process curves of multiple models according to an exemplary embodiment.

[0052] Figure 9 This is a schematic diagram of the statistical progress curve of a model over five consecutive rounds, according to an exemplary embodiment.

[0053] Figure 10 This is a schematic diagram of the statistical process curve of model sampling according to an exemplary embodiment.

[0054] Figure 11 This is a schematic diagram illustrating the comparison of the effects of implementing a method according to an exemplary embodiment.

[0055] Figure 12 This is a block diagram illustrating a distributed training system based on client autonomy, according to an exemplary embodiment.

[0056] Figure 13 This is a schematic diagram illustrating the architecture of an intelligent medical image analysis system according to an exemplary embodiment.

[0057] Figure 14 This is a schematic diagram of the architecture of a smart community service system according to an exemplary embodiment.

[0058] Figure 15 This is a schematic diagram of the architecture of an intelligent driving assistance system according to an exemplary embodiment. Detailed Implementation

[0059] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all fall within the protection scope of the present disclosure.

[0060] Figure 1 This is a flowchart illustrating a distributed training method based on client autonomy according to an exemplary embodiment.

[0061] like Figure 1 As shown, this disclosure provides a distributed training method based on client autonomy, including S11 to S13.

[0062] S11, determine the features of the client's local training.

[0063] Among them, the features of local training on the client side include system features and statistical features.

[0064] During local training, 70% of the devices remain available for no more than 10 minutes. The unavailability of a device can be considered an extreme case of device state change, indicating that in federated learning, the state of the client is constantly changing in each round of local training.

[0065] In one possible embodiment, the system characteristics of client-local training are represented by the time it takes for the client to complete each iteration in each round of training.

[0066] In another possible embodiment, the statistical characteristics of client-side local training are described using the client's local statistical process in each round of training.

[0067] Figure 2 This is a schematic diagram illustrating the change of gradient vectors during local training on a client side, according to an exemplary embodiment.

[0068] As an example, this disclosure may use the cosine similarity and modulus similarity of gradient vectors to compute the client's local statistical process.

[0069] like Figure 2 As shown, in the local training of the local model, it takes 7 iterations to reach the local optimum. In the initial stage, such as the first 3 iterations, the local model parameters deviate far from the local optimum, and the update steps of each iteration are large and the direction is consistent. In the later stage of training, such as the last 4 iterations, the local model parameters are close to the local optimum, and the update steps of each iteration are small and the directions conflict. Therefore, the contribution of each iteration to the overall training of the local model is different.

[0070] like Figure 2 As shown, the cumulative gradients after the first three iterations of training are represented by the green dashed line, and the cumulative gradients after all seven iterations of training in this round are represented by the blue dashed line. The two cumulative gradient vectors are very similar, with high cosine similarity and similar magnitudes. Therefore, the cosine similarity and magnitude similarity of the gradient vectors can be used together to represent the local statistical progress of the local model, i.e.:

[0071]

[0072] Among them, P i G represents the statistical progress of the local model after the i-th iteration of the current training round. i G represents the cumulative gradient of the local model after the i-th iteration of the current training round. K This represents the cumulative gradient of the local model after completing this round of training locally.

[0073] P can be used i -P i-1This represents the gain of the local model in the i-th iteration during the current round of training locally.

[0074] S12, based on the characteristics of local training on the client, adopts an early stopping strategy to decide the local training process of the client.

[0075] The early termination strategy means that the client makes an autonomous decision on whether to terminate the local training process based on the net gains in each round of local training.

[0076] Among them, the statistical process P is adopted. i -P i-1 The benefit of each iteration is calculated, using system characteristics, i.e., the time consumed in each iteration, to represent the cost of each iteration. The net benefit is determined by calculating the difference between the benefit and the cost. Based on the net benefit, the client autonomously decides whether to terminate the local training process in the current training round early. If the net benefit at the end of the current training round is non-negative, the client can continue the local training process in this round; if the net benefit at the end of the current training round is negative, the client can terminate the local training process in this round.

[0077] By employing an early stopping strategy, the client can dynamically and autonomously decide on its local training process, thereby saving computing resources and improving training efficiency.

[0078] S13. Based on the statistical characteristics of local training on the client, a hierarchical advance transmission strategy based on error feedback is adopted to optimize the communication process of local training on the client.

[0079] Among them, the hierarchical advance transmission strategy based on error feedback includes advance transmission strategy and error feedback mechanism.

[0080] The advance transmission strategy involves transmitting the converged layer parameters from the client to the server in advance during each round of training on the client side.

[0081] The error feedback mechanism includes verifying, after each round of training is completed on the client side, whether the pre-transmitted layer parameters deviate significantly from the corresponding layer parameters after the completion of that round of training. If the parameter deviation exceeds a preset threshold, the corresponding layer parameters will be retransmitted after the completion of that round of training.

[0082] Through the above technical solutions, the characteristics of client-side local training are determined, including system characteristics and statistical characteristics, enabling rapid perception of changes in client state within training rounds. Guided by these local training characteristics, the client adopts an early stopping strategy, autonomously deciding whether to stop the current training round. This saves computational resources and reduces the impact of slower-training clients on the global training process, thereby improving training efficiency, while minimizing the impact on model accuracy. Furthermore, guided by these local training characteristics, the client employs an error feedback-based layer-by-layer early transmission strategy. This strategy transmits converged layer parameters in advance to increase computation-communication overlap time, and introduces an error feedback mechanism to reduce the impact of early transmission on global accuracy, thereby improving resource utilization and enhancing client-side local training efficiency.

[0083] Figure 3 This is a flowchart illustrating a method for determining features of client-side local training according to an exemplary embodiment.

[0084] like Figure 3 As shown, in one possible embodiment, S11, determining the features of the client-side local training includes S21 to S22.

[0085] S21 uses a periodic sampling method to sample the cumulative gradient of the client after each iteration within the sampling training round.

[0086] The periodic sampling method disclosed herein determines the sampling period of the client's periodic sampling based on pre-configured parameters, and obtains the number of sampling training rounds based on the sampling period. Within the corresponding sampling training rounds, the client performs complete training without executing the early stopping strategy or the early transmission strategy, and executes the random sampling strategy to sample and record the cumulative gradient after each iteration within the round.

[0087] As an example, if a sampling period is 10 rounds, sampling is performed for rounds 1, 10, 20, 30, etc. During these sampling training rounds, the client does not implement early stopping or early transmission strategies, and the cumulative gradient after each iteration is sampled and recorded. The sampling ratio for the parameters of each layer of the model is 50%, and the maximum number of samples is 100. This periodic sampling method can significantly reduce the memory overhead of the statistical acquisition process.

[0088] S22, based on the cumulative gradient of the client during the sampling training rounds, calculate the statistical progress of the model layer by layer and as a whole, and determine the statistical characteristics of the client's local training.

[0089] Based on the cumulative gradient obtained after each iteration within the sampling training rounds using the periodic sampling method, the statistical characteristics of the client within the sampling training rounds are determined.

[0090] The statistical characteristics of the client trained locally, i.e., the statistical process.

[0091] In this disclosure, the above formulas for cosine similarity and modulus similarity based on gradient vectors are used to calculate the statistical features of the client during local training, i.e., the statistical process:

[0092]

[0093] Among them, P i G represents the statistical progress of the client's local training after the i-th iteration. i G represents the cumulative gradient on the client side after the i-th iteration of the current training round. K This represents the cumulative gradient after the client completes this round of training locally.

[0094] Through the above technical solution, a periodic sampling method is adopted. The client uses statistical processes to represent the statistical characteristics of local training. When making autonomous decisions, the statistical processes of adjacent sampling training rounds can be used to guide decision-making. This solves the problem that the client cannot predict the final state after each round of local training, i.e., it cannot obtain the cumulative gradient corresponding to the client after completing the current round of training locally. Furthermore, by adopting a periodic sampling method, the client does not need to record the local training data of each training round, which effectively reduces the algorithm's memory overhead.

[0095] like Figure 3 As shown, in one possible embodiment, S11, determining the features of the client-side local training, may also include S23.

[0096] S23 uses the time it takes for the client to complete each iteration in each round of training to represent the system characteristics of local training on the client side.

[0097] Figure 4 This is a flowchart illustrating a method for making decisions on the local training process of a client using an early stopping strategy, according to an exemplary embodiment.

[0098] like Figure 4 As shown, in one possible embodiment, S12 may include S31 to S34.

[0099] S31 quantifies the marginal benefit generated by the client in each iteration of each round of local training, and determines the marginal benefit of the client in each iteration of each round of training.

[0100] As an example, the marginal benefit of the τth iteration in the Rth round of training for the quantization client is:

[0101]

[0102] Among them, b R,τ P represents the marginal reward of the client when it completes the τth iteration in the Rth training round, where T represents the nearest periodically sampled training round to the Rth round, τ represents the number of iterations completed within the training round, and P represents the marginal reward of the client when it completes the τth iteration. T,τ P represents the statistical progress corresponding to the τth iteration within the periodic sampling training round closest to the Rth round. T,τ-1 This represents the statistical progress corresponding to the (τ-1)th iteration within the periodic sampling training round closest to the Rth round, and K represents the total number of iterations per round initially set by the algorithm.

[0103] S32 quantifies the marginal cost of the client completing each iteration in each round of local training to determine the marginal cost of the client completing each iteration in each round of local training.

[0104] As an example, the marginal cost of the τth iteration of the Rth round of training for the quantization client is:

[0105]

[0106] Among them, c R,τ t represents the marginal cost of completing the τth iteration in the Rth round of training for the client. R,τ T represents the time consumed by the client to complete the τth iteration of the Rth round of training. R This represents the global time limit for the client's training in the Rth round, and β represents the pre-set hyperparameters.

[0107] The global time limit for the client in the Rth round of training is set by the server based on the local training time of each client that participated in the previous round of training.

[0108] The hyperparameter β << 1. The hyperparameter β represents the value within the global time period T. R Previous training cost ratio, global time duration T R The calculation is performed using the methods employed in FedBalancer.

[0109] S33. Based on the marginal benefit and marginal cost of the client completing each iteration in each training round, determine the net benefit of the client completing each iteration in each training round.

[0110] Following the example above, based on the benefit-cost trade-off, determine the net benefit for the client to complete the τth iteration in the Rth round of training:

[0111]

[0112] Where, n R,τ b represents the net gain of the client completing the τth iteration in the Rth round of training. R,τc represents the marginal reward for the client to complete the τth iteration in the Rth round of training. R,τ This represents the marginal cost for the client to complete the τth iteration in the Rth round of training.

[0113] S34. Based on the net gain of the client after each iteration in each training round, the client decides whether to terminate the training round early.

[0114] As an example, if the net gain of the client after completing this iteration in the training round is negative, the client will terminate the training round early.

[0115] Following the example above, when net income n R,τ If the value is negative, the client terminates the current training session early.

[0116] As another example, if the net gain of the client after completing this iteration in the training round is non-negative, the client continues the training round.

[0117] Following the example above, when net income n R,τ If the value is non-negative, the client continues this round of training.

[0118] By employing the above technical solution and adopting an early stopping strategy, we can save computing resources, minimize the impact on the global model, reduce the impact of slower-training clients on the global training process, and thus improve overall training efficiency.

[0119] Figure 5 This is a schematic diagram illustrating an error feedback-based advance transmission strategy according to an exemplary embodiment.

[0120] Figure 6 This is a flowchart illustrating a method for optimizing the communication process of client-side local training using a hierarchical advance transmission strategy based on error feedback, according to an exemplary embodiment.

[0121] like Figure 5 , Figure 6 As shown, in one possible embodiment, S13 includes S41 to S43.

[0122] S41, in each round of training on the client side, transmits the converged layer parameters of the client side to the server side in advance.

[0123] Figure 7 This is a flowchart illustrating a method for pre-transmitting converged hierarchical parameters according to an exemplary embodiment.

[0124] like Figure 7 As shown in one possible embodiment, S41 may include S51 to S52.

[0125] S51, if the statistical characteristic index value of the same number of iterations in adjacent periodically sampled training rounds is not less than the preset second threshold when the client's hierarchical parameters complete each iteration in each training round, it is determined that the client's hierarchical parameters have reached the convergence state when completing that iteration in that training round.

[0126] As an example, the statistical characteristic index value of the client's layer l parameter after completing the τth iteration in the Rth round of training is... The statistical characteristic index value of the client's layer l parameter in the τth iteration of the Tth round of periodic sampling training is Among them, the Tth round of periodic sampling training is the most recent periodic sampling training round of the Rth round of training.

[0127] The client's layer l parameter advance transmission strategy is described as follows:

[0128]

[0129] Among them, T e This indicates the preset second threshold.

[0130] In this disclosure, the preset second threshold T e ≤1.

[0131] The statistical characteristic index of the client's layer l parameters in the τth iteration of the Tth round of periodic sampling training. When the value exceeds the preset second threshold, it is equivalent to the statistical characteristic index value of the client's layer l parameter completing the τth iteration in the Rth round of training being [value missing]. If the value is greater than the preset second threshold, it indicates that the client's layer l parameter has reached a convergence state.

[0132] S52 uploads the layer parameters from the client to the server in advance after each iteration in each training round, once the layer has reached convergence.

[0133] When uploading the level parameters that have reached convergence to the server in advance, a snapshot of the cumulative gradient at that time can be recorded.

[0134] S42, when the client completes each round of training, the cosine similarity is determined based on the cumulative gradient snapshot of the client during the advance transmission and the corresponding cumulative gradient after the client completes each round of training.

[0135] As an example, the cumulative gradient snapshot of the client's layer l parameters during advance transmission in the Rth round of training is: After the client completes the Rth round of training, the cumulative gradient corresponding to the parameters of layer l is: The cosine similarity between the cumulative gradient snapshot of layer l parameters on the client during pre-transmission and the corresponding cumulative gradient of layer l parameters after the client completes the Rth round of training is:

[0136]

[0137] Among them, T r This indicates the preset first threshold.

[0138] S43. If the cosine similarity is less than the preset first threshold, after the client completes this round of training, the hierarchical parameters transmitted in advance in this round of training will be re-uploaded to the server.

[0139] After each round of training is completed on the client side, an error feedback mechanism is used to check whether the hierarchical parameters transmitted to the server in advance have deviated significantly from the hierarchical parameters at the time of completing this round of training. If there is a significant deviation, the client re-uploads the hierarchical parameters.

[0140] Following the example above, if the cosine similarity between the cumulative gradient snapshot of the client's layer l parameters transmitted in advance and the corresponding cumulative gradient of the layer l parameters after the client completes the training round is less than a preset first threshold, it is determined that the client's layer l parameters need to be re-uploaded, and the layer l parameters are re-uploaded after the client completes the training round.

[0141] The above technical solution enables the retransmission of hierarchical parameters based on an error feedback mechanism to minimize the impact on the global model when there is a difference between the actual statistical progress of the current training round and the statistical progress of the most recent periodic sampling training round.

[0142] This disclosure employs a hierarchical advance transmission strategy based on error feedback, which can increase computation-communication overlap time, improve resource utilization, and enhance the training efficiency of the client and the global model.

[0143] Figure 8 This is a schematic diagram of the statistical process curves of multiple models according to an exemplary embodiment.

[0144] In one possible embodiment, the effectiveness of the statistical features of client-side local training proposed in this disclosure is verified:

[0145] like Figure 8 As shown, the CNN model was trained using the CIFAR-10 dataset; the LSTM model was trained using the KWS dataset; and the WideResNet28 model was trained using the CIFAR-100 dataset; each model underwent 200 training iterations.

[0146] like Figure 8 As shown, Figure 8The first column of images shows the statistical progress curves for each model and two randomly selected clients participating in the training in the early stages of model training, such as round 10. The second column of images shows the statistical progress curves for each model and two randomly selected clients participating in the training in the later stages of model training, such as round 200.

[0147] Depend on Figure 8 Therefore, the statistical characteristic index used in this disclosure, namely statistical process, effectively describes the diminishing marginal returns effect existing in each round of local training, and... Figure 2 The scenarios depicted are the same, and they also show differences on different clients, indicating the statistical heterogeneity of client-side local data.

[0148] In another possible embodiment, statistical characteristics at different levels in each model, i.e., statistical progress curves, can also be plotted, and the results are compared with... Figure 8 Similarly, the statistical features of local training on the client side, namely the statistical process, can also represent the differences between model layers during the training process.

[0149] Figure 9 This is a schematic diagram of the statistical progress curve of a model over five consecutive rounds, according to an exemplary embodiment.

[0150] In one possible embodiment, the effectiveness of the periodic sampling method is verified:

[0151] Periodic sampling methods can be used to address the challenges of statistical feature acquisition, namely, the difficulties faced by statistical processes: First, the client cannot predict the final state after each training round when making autonomous decisions, that is, the client cannot predict the cumulative gradient corresponding to each training round; Second, when obtaining the cumulative gradient corresponding to each iteration in each training round, a large amount of memory is required to store intermediate values, that is, to store the cumulative gradient corresponding to each iteration in each training round.

[0152] Taking the WideResNet28 model as an example, this model has more than 30 million parameters, and the memory overhead required to store intermediate values ​​in each round of client training is about 14GB.

[0153] like Figure 9 As shown, this disclosure uses a periodic sampling method to collect statistical progress curves of the CNN model, LSTM model, and WRN model for five consecutive rounds.

[0154] like Figure 9 As shown, the first column of images represents the statistical progress curves for five consecutive rounds starting from the 10th round of training in the early stage of model training, and the second column of images represents the statistical progress curves for five consecutive rounds starting from the 196th round of training in the later stage of model training.

[0155] Depend on Figure 9 The statistical progress curves for each of the five consecutive rounds in each image show a high degree of similarity, indicating that during local training on the client side, the statistical progress of the previous rounds can provide guidance for the training of subsequent rounds.

[0156] In one possible embodiment, the validity of parameter sampling is verified by determining the client's sampling period using a preset sampling period hyperparameter.

[0157] Figure 10 This is a schematic diagram of the statistical process curve of sampling according to an exemplary embodiment of a model.

[0158] like Figure 10 As shown, the training of CNN, LSTM, and WRN models is performed. The first column of images represents the comparison between the statistical progress curve obtained from sampling and the statistical progress curve of the complete model parameters in the early training stage of each model, such as the 10th epoch. The solid line represents the statistical progress curve of the accumulated gradient of the complete parameters in the 10th epoch, and the dashed line represents the statistical progress curve of the accumulated gradient of the sampled parameters in the 10th epoch. The second column of images represents the comparison between the statistical progress curve obtained from sampling and the statistical progress curve of the complete model parameters in the later training stage of each model, such as the 200th epoch. The solid line represents the statistical progress curve of the accumulated gradient of the complete parameters in the 200th epoch, and the dashed line represents the statistical progress curve of the accumulated gradient of the sampled parameters in the 200th epoch.

[0159] Depend on Figure 10 It can be seen that the statistical process curves of the sampled parameters are highly similar to those of the complete parameters, indicating that the statistical process obtained after sampling the parameters is representative compared to the complete parameters. Furthermore, sampling can effectively reduce memory overhead.

[0160] In one possible implementation, a parameter server with 32 cores, 64GB of memory, and an EC2 instance is used, and 128 EC2 instances with 2 cores, 4GB of memory, and two instances are used as clients for federated learning training. The bandwidth of the parameter server is limited to 10Gbps, and the uplink and downlink bandwidth of each client is limited to 13.7Mbps.

[0161] Referring to the related work FedScale, two Gamma distributions were used to randomly generate the normal training state time interval and the state change time interval for the client. When the client training entered the state change time interval, a delay ratio was randomly generated by uniform distribution. The delay ratio was used to simulate the situation where the client's computing resources were reduced and the computing power was decreased.

[0162] Figure 11This is a schematic diagram illustrating the comparison of the effects of implementing a method according to an exemplary embodiment.

[0163] This disclosure employs the existing FedProx method, the existing FedAda method, and the client-autonomy-based distributed training method proposed in this disclosure to train three models, including: training a CNN model using the CIFAR-10 dataset, training an LSTM model using the KWS dataset, and training a WideResNet28 model using the CIFAR-100 dataset.

[0164] The heterogeneity distribution of the dataset across different clients was simulated using a Dirichlet distribution with a Dirichlet factor set to 0.1.

[0165] like Figure 11 As shown, the time required for the prior art FedProx method, the prior art FedAda method, and the client-autonomy-based distributed training method proposed in this disclosure to train a CNN model to achieve the target accuracy of 0.55 is 1.89h, 0.86h, and 0.71h, respectively; the time required for the prior art FedProx method, the prior art FedAda method, and the client-autonomy-based distributed training method proposed in this disclosure to train an LSTM model to achieve the target accuracy of 0.85 is 1.28h, 0.75h, and 0.61h, respectively; and the time required for the prior art FedProx method, the prior art FedAda method, and the client-autonomy-based distributed training method proposed in this disclosure to train a WideResNet28 model to achieve the target accuracy of 0.55 is 294.7h, 228.9h, and 125.2h, respectively.

[0166] Therefore, compared with the prior art FedProx method, the client-autonomy-based distributed training method proposed in this disclosure improves the training efficiency by more than 50% on all three models; compared with the prior art FedAda method, the client-autonomy-based distributed training method proposed in this disclosure improves the training efficiency by 17.4% to 45.3% on all three models.

[0167] This disclosure quickly detects state changes within client training rounds by determining the in-round characteristics of local training on the client. Based on the in-round characteristics of local training on the client, it adopts an early stopping strategy with autonomous decision-making and a hierarchical early transmission strategy based on error feedback to achieve fine-grained optimization within client training rounds. This overcomes the problems of increased communication overhead and increased server pressure in existing server-side optimization techniques, as well as the inability of inter-round optimization methods to cope with rapid changes in training state within rounds.

[0168] Figure 12This is a block diagram illustrating a client-autonomy-based distributed training system according to an exemplary embodiment. Based on the same concept, this disclosure also provides a client-autonomy-based distributed training system, such as... Figure 12 As shown, it includes a feature determination module 110, a first optimization module 120, and a second optimization module 130.

[0169] Feature determination module 110 is used to determine the features of client-side local training, wherein the features of client-side local training include system features and statistical features;

[0170] The first optimization module 120 is used to make decisions on the local training process of the client based on the characteristics of the client's local training and by adopting an early stopping strategy.

[0171] The second optimization module 130 is used to optimize the communication process of the client's local training by adopting a hierarchical advance transmission strategy based on error feedback, according to the characteristics of the client's local training.

[0172] Through the above technical solutions, the characteristics of client-side local training are determined, including system characteristics and statistical characteristics, enabling rapid perception of changes in client state within training rounds. Guided by these local training characteristics, the client adopts an early stopping strategy, autonomously deciding whether to stop the current training round. This saves computational resources and reduces the impact of slower-training clients on the global training process, thereby improving training efficiency, while minimizing the impact on model accuracy. Furthermore, guided by these local training characteristics, the client employs an error feedback-based layer-by-layer early transmission strategy. This strategy transmits converged layer parameters in advance to increase computation-communication overlap time, and introduces an error feedback mechanism to reduce the impact of early transmission on global accuracy, thereby improving resource utilization and enhancing client-side local training efficiency.

[0173] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0174] Furthermore, based on the aforementioned distributed training method based on client autonomy, federated learning systems for different domains are constructed and explained.

[0175] In one embodiment, federated learning in a smart healthcare system is used as an example. In existing healthcare systems, data between different medical institutions is independent and highly private. This data can help doctors make rapid and reasonable analyses of patients' conditions, predict the risks of early-stage illnesses, and promote important biomedical research by researchers. By applying the client-autonomous distributed training system of this disclosure, data silos can be broken down while protecting privacy. This allows for joint learning of the data characteristics of various medical institutions to complete tasks such as examination, prediction, and analysis in smart healthcare, effectively improving diagnostic efficiency, alleviating pressure on medical services, providing personal health management services, and enhancing research capabilities. The following description illustrates a smart healthcare image analysis system built based on the client-autonomous distributed training system provided in this disclosure.

[0176] Figure 13 This is a schematic diagram of the architecture of an intelligent medical image analysis system according to an exemplary embodiment.

[0177] like Figure 13 As shown, the system consists of multiple clients participating in training and a centralized cloud server. Clients are deployed on medical imaging equipment or edge servers in various hospitals, such as medical imaging equipment in public health centers, clinics, hospitals, and personal mobile devices. The medical imaging equipment includes ultrasound imaging equipment, X-ray imaging equipment, computed tomography (CT) scanners, magnetic resonance imaging (MRI) equipment, and nuclear medicine equipment. Medical imaging equipment generates various types of medical image data, i.e., local data. By extracting and learning features from medical images, it can help doctors diagnose and predict conditions, effectively improving efficiency. The cloud server is typically deployed in a cloud data center, aggregating the models trained by each client and monitoring and displaying the training and analysis process.

[0178] The intelligent medical image analysis system disclosed herein can help process heterogeneous, large-scale medical image data more accurately. Specifically, the system works as follows: Users set up medical image analysis tasks, and the cloud server initializes the model, parameters, and configuration. Clients on various medical institutions and personal mobile devices perform multiple rounds of training based on local medical image data according to the global configuration. At the beginning of each training round, the client downloads the latest global model from the cloud server and performs multiple iterations of training locally. The client trains using the client-autonomous distributed training method proposed in this disclosure and acquires statistical progress based on a periodic sampling strategy. Based on the statistical progress acquired in the most recent sampling round of the current training and the time consumed in each iteration, the system calculates the net benefit and cost, dynamically adjusting the number of local training iterations to improve training efficiency. Simultaneously, during training, the client identifies the parameters of the layers that have converged ahead of schedule based on the statistical progress indicators of each layer of the model and uploads them to the cloud server. After each training round, the cloud server aggregates the model updates received from each client and updates the global model before proceeding to the next training round. Simultaneously, the client selects and sets the global deadline for the next round based on the training time of each client in that round. Ultimately, all clients and cloud servers complete the pre-configured training and aggregation rounds to obtain the trained global model. Users then process and analyze the medical images they need to analyze based on the training results.

[0179] Based on the federalized intelligent medical image analysis system disclosed herein, the models used for training on each client and for aggregation and updating on the cloud server are universal, meaning they can use convolutional neural networks, support vector machines, deep neural networks, etc. Selecting appropriate models for training, or comparing multiple trained models, can further improve the accuracy and efficiency of the analysis system.

[0180] In another embodiment, federated learning in a smart community is used as an example. In existing community management systems, resident behavioral data is diverse and privacy-sensitive, making unified collection and management impossible. Community managers typically communicate and manage the community manually. Resident behavioral data can improve community service and management efficiency, facilitating more intelligent suggestions for residents. By applying the client-autonomous distributed training system of this disclosure, data features from various communities can be jointly learned while protecting privacy. This enables tasks such as analysis, early warning, management, and suggestions within the smart community, effectively improving work efficiency and providing residents with more intelligent services. The following description illustrates a smart community service system built based on the client-autonomous distributed training system provided in this disclosure.

[0181] Figure 14 This is a schematic diagram of the architecture of a smart community service system according to an exemplary embodiment.

[0182] like Figure 14 As shown, the system consists of multiple clients participating in training and a centralized cloud server. The clients are deployed on various IoT devices belonging to multiple residents within multiple communities, including smart TVs, smart routers, and smart cameras. These IoT devices generate diverse user behavior data. By extracting and learning the characteristics of this user behavior data, community workers can provide more efficient services and management to users, such as community traffic prediction, resident preference surveys, and community risk warnings. The cloud server is typically deployed in a cloud data center, aggregating the models trained by each client and updating the global model.

[0183] The smart community service system disclosed herein can help process heterogeneous user behavior data more accurately. Specifically, the cloud server initializes the model, parameters, and configuration. Clients on each IoT device perform multiple rounds of training based on local behavior data and global configuration. At the start of each training round, the client downloads the latest global model from the cloud server and performs multiple iterative training locally. The client trains using the client-autonomous distributed training method proposed in this disclosure and acquires statistical progress based on a periodic sampling strategy. Based on the statistical progress acquired in the most recent sampling round of the current training and the time consumed in each iteration, the client calculates the net benefit by calculating the cost and benefits, and dynamically adjusts the number of local training iterations to improve training efficiency. Simultaneously, during training, the client identifies the parameters of the layers that have converged ahead of schedule based on the statistical progress indicators of each layer of the model and uploads them to the cloud server. After each training round, the cloud server aggregates the model updates received from each client and updates the global model before proceeding to the next training round. Simultaneously, the client selects and sets the global deadline for the next round based on the training time of each client in that round. Ultimately, all clients and cloud servers complete the pre-configured training and aggregation rounds to obtain the trained global model. Users then analyze the training results to perform necessary tasks, such as traffic prediction in residential areas, conducting community activities based on resident preference analysis, mining and managing user characteristics, and providing community risk warnings. This improves community service efficiency, reduces management costs, and enhances the ability to respond to abnormal situations.

[0184] Based on the publicly disclosed Federal Smart Community Service System, the models used for training on each client and for aggregation and updating on the cloud server are universal, meaning they can utilize convolutional neural networks, support vector machines, deep neural networks, etc. Selecting appropriate models for training, or comparing multiple trained models, can further improve the accuracy and efficiency of the analysis system.

[0185] In another embodiment, federated learning in intelligent driving is used as an example. With the development of vehicle-to-everything (V2X) technology, vehicle behavior data is diverse and privacy-sensitive, making unified collection and management impossible. Traditional driver assistance systems often fail to provide effective assistance to drivers. However, utilizing driving data and other road traffic data can provide higher-performance and more accurate intelligent driver assistance functions and more accurate driving suggestions, improving the accuracy and safety of autonomous driving and further enhancing road safety. The client-autonomous distributed training system of this disclosure enables joint learning of driving and sensing data features from various vehicles while protecting privacy, thereby completing functions such as route suggestion, driving optimization, autonomous driving, and safe driving in intelligent driving. The following description illustrates an intelligent driving assistance system built based on the client-autonomous distributed training system provided in this disclosure.

[0186] Figure 15 This is a schematic diagram of the architecture of an intelligent driving assistance system according to an exemplary embodiment.

[0187] like Figure 15 As shown, the system consists of multiple clients participating in training and a centralized cloud server. The clients are deployed on the terminal systems of multiple vehicles and road traffic monitoring equipment. The vehicle management system acquires vehicle behavior data, vehicle status data, environmental data captured by cameras, and data from other temperature and pressure sensors. The road monitoring equipment provides users with normal or abnormal driving information based on various road conditions. By extracting and learning features from various vehicle data sources, the system can help drivers drive more safely, such as providing more efficient route suggestions, more accurate autonomous driving, abnormal warnings, and emergency avoidance functions. The cloud server is typically deployed in a cloud data center, aggregating the models trained by each client and updating the global model.

[0188] The intelligent driving assistance system based on this disclosure can help process heterogeneous vehicle and road data more accurately. Specifically, the cloud server initializes the model, parameters, and configuration. Clients on each vehicle and road monitoring system perform multiple rounds of training based on local behavioral data according to the global configuration. At the start of each training round, the client downloads the latest global model from the cloud server and performs multiple iterative training locally. The client trains using the client-autonomy-based distributed training method proposed in this disclosure and acquires statistical progress based on a periodic sampling strategy. Based on the statistical progress acquired in the most recent sampling round of the current training and the time consumed in each iteration, the client calculates the net benefit based on the cost and dynamically adjusts the number of local training iterations to improve training efficiency. Simultaneously, during training, the client identifies the parameters of the layers that have converged ahead of schedule based on the statistical progress indicators of each layer of the model and uploads them to the cloud server. After each training round is completed, the cloud server aggregates the model updates received from each client and updates the global model, then proceeds to the next training round. Simultaneously, the client selects and sets the global deadline for the next round based on the training time of each client in that round. Ultimately, all clients and cloud servers complete the pre-configured training and aggregation rounds to obtain the trained global model. Based on the training results, the system then provides driver assistance services, such as more efficient route suggestions, more accurate autonomous driving, anomaly warnings, and emergency avoidance functions, thereby improving vehicle and road safety.

[0189] Based on the intelligent driving assistance system disclosed herein, the models used for training on various clients and for aggregation and updating on the cloud server are universal, meaning they can utilize convolutional neural networks, support vector machines, deep neural networks, etc. Selecting appropriate models for training, or comparing multiple trained models, can further improve the accuracy and efficiency of the analysis system.

[0190] Based on the same concept described above, in another embodiment of this disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes a distributed training method based on client autonomy when executing the program.

[0191] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0192] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0193] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0194] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0195] In this embodiment of the disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of a distributed training method based on client autonomy in any of the above embodiments.

[0196] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] Although preferred embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0201] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A distributed training method based on client autonomy, characterized in that, include: The characteristics of client-side local training are determined, including system characteristics and statistical characteristics. The determination of client-side local training characteristics includes: sampling the cumulative gradient of the client after each iteration of training in the sampling training round using a periodic sampling method; and calculating the statistical progress of the model layer by layer and as a whole based on the cumulative gradient of the client in the sampling training round to determine the statistical characteristics of client-side local training. Based on the characteristics of the client's local training, an early stopping strategy is adopted to determine the client's local training process. This includes: quantifying the marginal benefit generated by the client in each iteration of each training round to determine the marginal benefit of the client in each iteration of each training round; quantifying the marginal cost corresponding to the client in each iteration of each training round to determine the marginal cost of the client in each iteration of each training round; determining the net benefit of the client in each iteration of each training round based on the marginal benefit and the marginal cost of the client in each training round; and, based on the net benefit of the client in each iteration of each training round, the client autonomously decides whether to terminate the training round early. The step of the client autonomously deciding whether to terminate the training round early based on the net gain of the client after completing each iteration in each training round includes: if the net gain of the client after completing the iteration in the training round is negative, the client terminates the training round early; if the net gain of the client after completing the iteration in the training round is non-negative, the client continues the training round. Based on the statistical characteristics of the client's local training, a hierarchical advance transmission strategy based on error feedback is adopted to optimize the communication process of the client's local training. This optimization includes: in each round of training on the client, transmitting the converged hierarchical parameters of the client to the server in advance; when the client completes each round of training, determining a cosine similarity based on the client's cumulative gradient snapshot at the time of advance transmission and the corresponding cumulative gradient after the client completes each round of training; if the cosine similarity is less than a preset first threshold, after the client completes that round of training, re-uploading the hierarchical parameters transmitted in advance in that round of training to the server. Specifically, in each round of training on the client side, the converged hierarchical parameters of the client side are transmitted to the server side in advance. This includes: if the statistical characteristic index value of the same number of iterations in adjacent periodically sampled training rounds is not less than a preset second threshold when the hierarchical parameters of the client side complete each iteration in each round of training, it is determined that the hierarchical parameters of the client side have reached a convergent state when completing that iteration in that round of training; and the hierarchical parameters of the client side that have reached a convergent state when completing each iteration in each round of training are uploaded to the server side in advance.

2. The method according to claim 1, characterized in that, The determination of the features of local training on the client also includes: The system characteristics of the client's local training are represented by the time it takes for the client to complete each iteration in each round of training.

3. A distributed training system based on client autonomy, characterized in that, include: A feature determination module is used to determine the features of client-side local training. The features of client-side local training include system features and statistical features. The feature determination module is used to sample the cumulative gradient of the client after each iteration of training in the sampling training round using a periodic sampling method. Based on the cumulative gradient of the client in the sampling training round, the statistical progress of the model is calculated layer by layer and as a whole to determine the statistical features of client-side local training. The first optimization module is used to decide on the local training process of the client based on the characteristics of the client's local training, using an early termination strategy. Specifically, the first optimization module quantifies the marginal benefit generated by the client in each iteration of each training round to determine the marginal benefit of the client in each iteration of each training round; quantifies the marginal cost corresponding to the client in each iteration of each training round to determine the marginal cost of the client in each training round; determines the net benefit of the client in each iteration of each training round based on the marginal benefit and marginal cost of the client in each training round; and, based on the net benefit of the client in each iteration of each training round, the client autonomously decides whether to terminate the training round early. The step of the client autonomously deciding whether to terminate the training round early based on the net gain of the client after completing each iteration in each training round includes: if the net gain of the client after completing the iteration in the training round is negative, the client terminates the training round early; if the net gain of the client after completing the iteration in the training round is non-negative, the client continues the training round. The second optimization module is used to optimize the communication process of the client's local training by adopting a hierarchical advance transmission strategy based on error feedback, according to the statistical characteristics of the client's local training. Specifically, the second optimization module is used to advance the converged hierarchical parameters of the client to the server in each round of training; when the client completes each round of training, it determines the cosine similarity based on the client's cumulative gradient snapshot at the time of advance transmission and the corresponding cumulative gradient after the client completes each round of training; if the cosine similarity is less than a preset first threshold, the hierarchical parameters advanced in that round of training are re-uploaded to the server after the client completes that round of training. Specifically, in each round of training on the client side, the converged hierarchical parameters of the client side are transmitted to the server side in advance. This includes: if the statistical characteristic index value of the same number of iterations in adjacent periodically sampled training rounds is not less than a preset second threshold when the hierarchical parameters of the client side complete each iteration in each round of training, it is determined that the hierarchical parameters of the client side have reached a convergent state when completing that iteration in that round of training; and the hierarchical parameters of the client side that have reached a convergent state when completing each iteration in each round of training are uploaded to the server side in advance.

4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-2.

5. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-2.

Images