User side training scheduling method and system for federated learning

Abstract

The invention provides a user side training scheduling method and system for federated learning, and the method comprises the steps: determining the local training performance characteristics of user sides based on the local training logs of all user sides, thereby selecting a part of user sides to form a user side cluster, and executing the next federated learning; based on respective local training process characteristics of all the user sides in the cluster, a plurality of training data subsets in one-to-one correspondence with the user sides are obtained through division, and based on local training states of the user sides on the training data subsets in next federated learning, a local model is returned to the server; after the local models with defects are returned to the user sides for local training again, all the local models received by the server are fused, and a plurality of user sides with balanced performance are identified and screened through local training performance to participate in federated learning, so that time waste caused by mutual waiting of the user sides can be effectively reduced; the federal learning process is optimized according to the local training performance of the user side, and the federal learning training speed and precision are improved.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to a client-side training scheduling method and system for federated learning. Background Technology

[0002] Federated learning, also known as federated machine learning, allows multiple client machines (participants) to collaboratively train a shared machine learning model while maintaining data privacy and locality. Traditional federated learning utilizes clients for local training; however, the performance of different clients is uneven, leading to significant differences in the time it takes for them to complete local training. This results in longer time intervals between the returned partial models to the server, causing clients that finish local training earlier to wait for those that finish later, increasing the overall time consumption of federated learning. The performance inconsistency among all client machines involved in federated learning affects the efficiency and reliability of model training. To effectively reduce the time wasted due to significant differences in local training time among client machines, it is necessary to select suitable client machines to participate in federated learning based on their local training performance, thereby reducing the time consumption and improving the training speed and accuracy of federated learning. Summary of the Invention

[0003] The purpose of this invention is to provide a client-side training scheduling method and system for federated learning. Based on the local training logs of all client-side computers, the local training performance characteristics of each client are determined, and a subset of client-side computers are selected to form a client-side cluster for the next federated learning iteration. Based on the local training process characteristics of each client-side computer within the cluster, several training data subsets corresponding to each client-side computer are obtained. Based on the local training status of each client-side computer for the next federated learning iteration, a local model is returned to the server. Defective local models are returned to the client-side computers for retraining. All local models received by the server are then fused. By identifying and selecting client-side computers with balanced performance based on their local training performance, the time wasted due to client-side waiting is effectively reduced. The federated learning process is optimized based on the local training performance of the client-side computers, improving the training speed and accuracy of federated learning.

[0004] This invention is achieved through the following technical solution:

[0005] Client-side training scheduling methods for federated learning include:

[0006] Based on the local training logs of each user terminal, the local training performance characteristics of the user terminal in the historical federated learning process are determined; based on the local training performance characteristics, a portion of the user terminals are selected to form a user terminal cluster to execute the next federated learning.

[0007] Based on the local training process characteristics of each user terminal in the user terminal cluster, the training dataset is divided into several training data subsets that correspond one-to-one with each user terminal; based on the local training state of the user terminal for the training data subset in the next federated learning, a local model is returned to the server.

[0008] After returning the defective local model to the corresponding user terminal for retraining locally, all local models received by the server are fused together.

[0009] Optionally, based on the local training logs of each user terminal, the local training performance characteristics of the user terminals in the historical federated learning process are determined; based on the local training performance characteristics, a subset of user terminals are selected to form a user terminal cluster to perform the next federated learning, including:

[0010] Extract local training task records from the local training logs of each user terminal, analyze the local training task records, and obtain the time consumption characteristics of the local training tasks executed by the user terminal in different federated learning processes.

[0011] Based on the time consumption characteristics of the local training task, the time difference information of the user terminal completing the local training task is determined; based on the time difference information, a portion of the user terminals are selected to form a user terminal cluster to perform the next federated learning.

[0012] Optionally, based on the local training process characteristics of each user terminal within the user terminal cluster, the training dataset is divided into several training data subsets corresponding one-to-one with each user terminal; based on the local training state of the user terminal for the training data subset in the next federated learning, a local model is returned to the server, including:

[0013] Obtain the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data, wherein the training error rate characteristics include the error frequency of each user terminal during local training of each type of training data; based on the local training error rate characteristics of each user terminal, divide the training dataset into several training data subsets that correspond one-to-one with each user terminal.

[0014] Based on the local training progress of the user client on the training data subset in the next federated learning, it is determined whether the user client has completed local training, and the local model generated after the local training is completed is returned to the server.

[0015] Optionally, after returning the defective local models to the corresponding user terminals for retraining locally, all local models received by the server are fused, including:

[0016] The system performs model performance identification on all local models received by the server to determine the compatibility of each local model with other local models. This determines whether the local model has defects, and the defective local models are returned to the corresponding user terminal for retraining. Finally, all local models received by the server are fused together.

[0017] Optionally, model performance identification is performed on all local models received by the server to obtain the compatibility degree between the local models and other local models, including:

[0018] Extract the model parameter distribution features, sample prediction probability distribution features, and model contribution weights for each local model received by the server;

[0019] The parameter distribution fit between each local model and other models is obtained by utilizing the model parameter distribution characteristics of each local model and other models.

[0020] The parameter distribution fit is obtained by the following formula:

[0021]

[0022] Where A represents the parameter distribution fit between each local model and other models; n represents the number of core parameter layers of the model; σ bi σ represents the standard deviation of the parameters of the i-th layer of each other model corresponding to the local model; ai μ represents the standard deviation of the parameters of the i-th layer in each local model. bi μ represents the mean of the parameters of the i-th layer of each other model corresponding to the local model; ai This represents the mean of the parameters of the i-th layer in each local model;

[0023] Retrieve the predicted probability vectors of each local model and other models on the same key samples, and use the predicted probability vectors of each local model and other models on the same key samples to obtain the corresponding cosine similarity.

[0024] Retrieve the model contribution weights of each local model to other models, and use these weights to obtain the contribution correlation between each local model and other models.

[0025] The contribution correlation between each local model and other models is obtained by the following formula;

[0026]

[0027] Where B represents the contribution correlation between each local model and other models; W aW represents the model contribution weight corresponding to each local model; b These represent the model contribution weights of other models corresponding to each local model;

[0028] The compatibility coefficient between each local model and other models is obtained by using the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models. The larger the compatibility coefficient, the higher the compatibility.

[0029] The compatibility coefficient is obtained using the following formula:

[0030]

[0031] Where J represents the compatibility coefficient; x 01 x 02 and x 03 , respectively, represent the weight values ​​corresponding to the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models; A, S, and B represent the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models, respectively.

[0032] The client-side training scheduling system for federated learning includes:

[0033] The local training performance identification module is used to determine the local training performance characteristics of each user terminal in the historical federated learning process based on the local training logs of each user terminal.

[0034] The cluster building module is used to select a portion of user terminals to form a user terminal cluster based on the local training performance characteristics, so as to perform the next federated learning.

[0035] The training data partitioning module is used to divide the training dataset into several training data subsets that correspond one-to-one with all user terminals based on the local training process characteristics of each user terminal in the user terminal cluster.

[0036] The local model return module is used to return a local model to the server based on the local training status of the training data subset on the user terminal in the next federated learning.

[0037] The local model adjustment and fusion module is used to return defective local models to the corresponding user terminals for retraining locally, and then fuse all local models received by the server.

[0038] Optionally, the local training performance identification module is used to determine the local training performance characteristics of each user terminal during the historical federated learning process based on the local training logs of each user terminal, including:

[0039] Extract local training task records from the local training logs of each user terminal, analyze the local training task records, and obtain the time consumption characteristics of the local training tasks executed by the user terminal in different federated learning processes.

[0040] The cluster building module is used to select a portion of user terminals to form a user terminal cluster based on the local training performance characteristics, and then perform the next federated learning, including:

[0041] Based on the time consumption characteristics of the local training task, the time difference information of the user terminal completing the local training task is determined; based on the time difference information, a portion of the user terminals are selected to form a user terminal cluster to perform the next federated learning.

[0042] Optionally, the training data partitioning module is used to partition the training dataset into several training data subsets corresponding one-to-one with all user terminals based on the local training process characteristics of each user terminal within the user terminal cluster, including:

[0043] Obtain the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data, wherein the training error rate characteristics include the error frequency of each user terminal during local training of each type of training data; based on the local training error rate characteristics of each user terminal, divide the training dataset into several training data subsets that correspond one-to-one with each user terminal.

[0044] The local model return module is used to return a local model to the server based on the local training state of the training data subset on the user terminal in the next federated learning, including:

[0045] Based on the local training progress of the user client on the training data subset in the next federated learning, it is determined whether the user client has completed local training, and the local model generated after the local training is completed is returned to the server.

[0046] Optionally, the local model adjustment and fusion module is used to return defective local models to the corresponding user terminal for retraining locally, and then fuse all local models received by the server, including:

[0047] The system performs model performance identification on all local models received by the server to determine the compatibility of each local model with other local models. This determines whether the local model has defects, and the defective local models are returned to the corresponding user terminal for retraining. Finally, all local models received by the server are fused together.

[0048] Optionally, model performance identification is performed on all local models received by the server to obtain the compatibility degree between the local models and other local models, including:

[0049] Extract the model parameter distribution features, sample prediction probability distribution features, and model contribution weights for each local model received by the server;

[0050] The parameter distribution fit between each local model and other models is obtained by utilizing the model parameter distribution characteristics of each local model and other models.

[0051] The parameter distribution fit is obtained by the following formula:

[0052]

[0053] Where A represents the parameter distribution fit between each local model and other models; n represents the number of core parameter layers of the model; σ bi σ represents the standard deviation of the parameters of the i-th layer of each other model corresponding to the local model; ai μ represents the standard deviation of the parameters of the i-th layer in each local model. bi μ represents the mean of the parameters of the i-th layer of each other model corresponding to the local model; ai This represents the mean of the parameters of the i-th layer in each local model;

[0054] Retrieve the predicted probability vectors of each local model and other models on the same key samples, and use the predicted probability vectors of each local model and other models on the same key samples to obtain the corresponding cosine similarity.

[0055] Retrieve the model contribution weights of each local model to other models, and use these weights to obtain the contribution correlation between each local model and other models.

[0056] The contribution correlation between each local model and other models is obtained by the following formula;

[0057]

[0058] Where B represents the contribution correlation between each local model and other models; W a W represents the model contribution weight corresponding to each local model; b These represent the model contribution weights of other models corresponding to each local model;

[0059] The compatibility coefficient between each local model and other models is obtained by using the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models. The larger the compatibility coefficient, the higher the compatibility.

[0060] The compatibility coefficient is obtained using the following formula:

[0061]

[0062] Where J represents the compatibility coefficient; x 01 x 02 and x 03 , respectively, represent the weight values ​​corresponding to the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models; A, S, and B represent the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models, respectively.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The client-side training scheduling method and system for federated learning provided in this application determine the local training performance characteristics of each client based on the local training logs of all clients. This allows for the selection of a subset of clients to form a client cluster for the next federated learning iteration. Based on the local training process characteristics of each client within the cluster, several training data subsets corresponding to each client are obtained. Based on the local training status of each client's training data subset for the next federated learning iteration, a local model is returned to the server. Defective local models are returned to the clients for retraining. All local models received by the server are then merged. By identifying and selecting clients with balanced performance based on their local training performance, the federated learning process can be effectively reduced, minimizing time wasted due to clients waiting for each other. The federated learning process is optimized based on the local training performance of each client, improving both the training speed and accuracy of federated learning. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0066] Figure 1 This is a flowchart illustrating the client-side training scheduling method for federated learning provided by the present invention.

[0067] Figure 2This is a schematic diagram of the structure of the client-side training scheduling system for federated learning provided by the present invention. Detailed Implementation

[0068] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not the entire structure. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0069] The terms “comprising” and “having”, and any variations thereof, used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0070] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0071] Please see Figure 1 As shown, an embodiment of this application provides a client-side training scheduling method for federated learning. This client-side training scheduling method for federated learning includes:

[0072] Based on the local training logs of each user terminal, the local training performance characteristics of the user terminals in the historical federated learning process are determined; based on the local training performance characteristics, a portion of the user terminals are selected to form a user terminal cluster, which is then used to execute the next federated learning.

[0073] Based on the local training process characteristics of each user terminal in the user terminal cluster, the training dataset is divided into several training data subsets that correspond one-to-one with each user terminal; based on the local training state of the user terminal for the training data subset in the next federated learning, a local model is returned to the server.

[0074] After returning the defective local model to the corresponding user terminal for retraining locally, all local models received by the server are fused together.

[0075] The beneficial effects of the above embodiments are as follows: the client-side training scheduling method for federated learning determines the local training performance characteristics of each client based on the local training logs of all clients, thereby selecting a portion of clients to form a client cluster for the next federated learning iteration; based on the local training process characteristics of each client within the cluster, several training data subsets corresponding one-to-one with each client are obtained, and based on the local training status of the client's training data subsets in the next federated learning iteration, a local model is returned to the server; after returning the defective local models to the client for retraining, all local models received by the server are fused, and several client-side models with balanced performance are selected through local training performance identification to participate in federated learning, which can effectively reduce the time wasted due to client-side waiting for each other; the federated learning process is optimized according to the local training performance of the client, improving the training speed and accuracy of federated learning.

[0076] In another embodiment, based on the local training logs of each user terminal, the local training performance characteristics of the user terminals in the historical federated learning process are determined; based on the local training performance characteristics, a subset of user terminals are selected to form a user terminal cluster to perform the next federated learning, including:

[0077] Extract local training task records from the local training logs of all users, analyze the local training task records, and obtain the time consumption characteristics of local training tasks executed by users in different federated learning processes.

[0078] Based on the time consumption characteristics of local training tasks, the time difference information of user terminals completing local training tasks is determined; based on the time difference information, a portion of user terminals are selected to form a user terminal cluster to execute the next federated learning.

[0079] The beneficial effects of the above embodiments are that, as participants in federated learning, user terminals can utilize training data for local training, thereby participating in the local training of machine learning models and obtaining a completed local model. The local training performance of different user terminals varies, not only between different user terminals but also particularly within the same user terminal for different types of training data. For example, some user terminals exhibit high training efficiency and accuracy for certain types of training data, completing local training in a shorter time, while exhibiting lower efficiency and accuracy for other types of training data, requiring a longer time to complete local training. Furthermore, due to differences in their own computing and memory resources, different user terminals also show differences in training efficiency and accuracy for the same type of training data. Each user terminal generates a corresponding local training task record and uploads it to the local training log each time it performs local training. Therefore, the local training log comprehensively records all historical local training process information of the user terminal, including but not limited to the start and end times of each historical local training session, and the usage of computing and memory resources for local training. The better the local training performance of a user terminal, the less time it takes to complete local training. The time it takes for a user terminal to complete local training directly reflects its local training performance. Therefore, by analyzing the local training time of each user terminal, we can determine whether the local training performance is balanced across different user terminals. Specifically, we extract the local training logs of each user terminal from their respective local training logs, perform local training time-domain feature identification on the local training task records, obtain the time spent by each user terminal on all local training tasks executed in different federated learning projects throughout history, and determine the average local training time of all local training tasks. This average local training time is then used as the time consumption feature of the user terminal's local training tasks. Furthermore, we calculate the difference between the average local training times of all user terminals to obtain the difference between the average local training times of any two user terminals. This difference serves as the time difference information for all user terminals completing local training tasks. The smaller the difference in average local training time between two user terminals, the smaller the difference in local training performance between the two user terminals. Based on the time difference information of all user terminals completing local training tasks, a portion of user terminals are selected to form a user terminal cluster. This ensures that the difference in average local training time between any two user terminals in the selected user terminal cluster is less than a preset difference threshold, and that the offset of the average local training time of each selected user terminal relative to the predetermined target time is less than a preset offset threshold. This guarantees that all user terminals in the user terminal cluster have high local training performance and relatively balanced local training performance, reducing the time wasted by the user terminal cluster waiting for each other during the next federated learning process.

[0080] In another embodiment, based on the local training process characteristics of each user terminal within the user terminal cluster, the training dataset is divided into several training data subsets corresponding one-to-one with each user terminal; based on the local training state of the user terminal for the training data subset in the next federated learning, a local model is returned to the server, including:

[0081] Obtain the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data. The training error rate characteristics include the error frequency of each type of training data during local training. Based on the local training error rate characteristics of each user terminal, divide the training dataset into several training data subsets that correspond one-to-one with each user terminal.

[0082] Based on the user's local training progress on the training data subset in the next federated learning iteration, determine whether the user has completed local training, and return the local model generated after local training to the server.

[0083] The beneficial effects of the above embodiments are that each user terminal in the user terminal cluster performs local training independently. Different user terminals require different computing and memory resources for local training on different types of training data. If a user terminal provides insufficient computing and / or memory resources for local training on a certain type of training data, it cannot guarantee that the user terminal will correctly complete the entire local training, thereby reducing the reliability of the local model obtained through local training. Therefore, it is necessary to allocate appropriate training data to all user terminals under the user terminal cluster to ensure the accuracy of local training. Specifically, the computing and memory resources required by each user terminal in the user terminal cluster for local training on different types of training data are first obtained. Combined with the computing and memory resources that each user terminal can provide, the temporal variation characteristics of the computing and memory resource gaps for each user terminal during local training on different types of training data are determined. This allows for the determination of the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data. Generally, the longer the duration during which the computing power and / or memory resource gap values ​​exceed a preset gap threshold during local training of a certain type of training data on a user end, the higher the error frequency during local training of that type of training data on the user end. Based on the error rate characteristics of local training on all types of training data by all user ends within the user end cluster, the training dataset is divided into several training data subsets corresponding to each user end. This ensures that each user end can provide sufficient computing power and memory resources for local training on its corresponding training data subset. Furthermore, the local training progress of the user end's training data subset in the next federated learning iteration is obtained to determine whether the user end has completed local training. If completed, the locally generated local model is returned to the server; otherwise, the local model is not returned to the server, thus ensuring that the server can quickly obtain local models returned from different user ends within a relatively concentrated time interval.

[0084] In another embodiment, after returning the defective local model to the corresponding user terminal for retraining locally, all local models received by the server are fused, including:

[0085] The system performs model performance identification on all local models received from the server to determine the compatibility of each local model with other local models. This helps determine if any local models have defects. Defective local models are then returned to the corresponding user terminals for retraining. Finally, all local models received from the server are fused together.

[0086] The beneficial effect of the above embodiments is that the server needs to fuse all the local models generated by the user terminals through local training to construct a complete machine learning model. The performance parameters of each local model, such as data processing speed and data processing reliability, directly affect the overall performance of the machine learning model. To ensure good overall performance of the machine learning model, it is necessary to identify the performance parameters of all local models received by the server, such as data processing speed and data processing reliability, to determine whether the data processing speed and reliability of each local model are compatible with its neighboring local models, obtaining a compatibility value between the local models and other local models. If the compatibility value is less than a preset compatibility threshold, the local model is determined to have a defect; otherwise, it is determined that the local model has no defect. The defective local models are then returned to their corresponding user terminals for retraining to improve the local models. The user terminals that have completed retraining are then returned to the server. In this way, the server can fuse all the non-defective local models to obtain a complete machine learning model, ensuring the reliability of the machine learning model.

[0087] In another embodiment, model performance identification is performed on all local models received by the server to obtain the compatibility degree of the local models with other local models, including:

[0088] Extract the model parameter distribution features, sample prediction probability distribution features, and model contribution weights for each local model received by the server;

[0089] The parameter distribution fit between each local model and other models is obtained by utilizing the model parameter distribution characteristics of each local model and other models.

[0090] The parameter distribution fit is obtained by the following formula:

[0091]

[0092] Where A represents the parameter distribution fit between each local model and other models; n represents the number of core parameter layers of the model; σ bi σ represents the standard deviation of the parameters of the i-th layer of each other model corresponding to the local model; ai μ represents the standard deviation of the parameters of the i-th layer in each local model. bi μ represents the mean of the parameters of the i-th layer of each other model corresponding to the local model; ai This represents the mean of the parameters of the i-th layer in each local model;

[0093] Retrieve the predicted probability vectors of each local model and other models on the same key samples, and use the predicted probability vectors of each local model and other models on the same key samples to obtain the corresponding cosine similarity.

[0094] Retrieve the model contribution weights of each local model to other models, and use these weights to obtain the contribution correlation between each local model and other models.

[0095] The contribution correlation between each local model and other models is obtained by the following formula;

[0096]

[0097] Where B represents the contribution correlation between each local model and other models; W a W represents the model contribution weight corresponding to each local model; b These represent the model contribution weights of other models corresponding to each local model;

[0098] The compatibility coefficient between each local model and other models is obtained by using the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models. The larger the compatibility coefficient, the higher the compatibility.

[0099] The compatibility coefficient is obtained using the following formula:

[0100]

[0101] Where J represents the compatibility coefficient; x 01 x 02 and x 03 , respectively, represent the weight values ​​corresponding to the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models; A, S, and B represent the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models, respectively.

[0102] The beneficial effects of the above embodiments are that, by extracting three core types of information—model parameter distribution features, sample prediction probability distribution features, and model contribution weights—the limitations of traditional single-dimensional evaluation are overcome, achieving a comprehensive characterization of model compatibility characteristics and avoiding misjudgments of compatibility due to incomplete information. Simultaneously, calculating parameter distribution fit based on the mean and standard deviation of the core parameter layer accurately measures the synergy at the parameter level between models, reducing interference from parameter conflicts in subsequent fusion. Furthermore, calculating cosine similarity through the prediction probability vectors of the same key samples effectively assesses the consistency of prediction logic between models, ensuring the fused model performs well in core task scenarios. The system improves the predictive stability under different scenarios. Simultaneously, by calculating the contribution correlation using model contribution weights, the system can balance the fit between different value models, avoiding the drag on the fusion effect of low-contribution models on high-contribution models. Finally, by weighting the parameter distribution fit, cosine similarity, and contribution correlation using preset weights, a compatibility coefficient is obtained. This achieves the orderly integration of multi-dimensional indicators, ensuring the accuracy and objectivity of the compatibility assessment. This provides a reliable basis for subsequent identification of defective models and optimization of the model fusion process, thereby improving the performance stability of the final fused model and the efficiency of server-side model fusion, and reducing the risk of fusion failure caused by defective models.

[0103] Please see Figure 2 As shown, an embodiment of this application provides a client-side training scheduling system for federated learning. This client-side training scheduling system for federated learning includes:

[0104] The local training performance identification module is used to determine the local training performance characteristics of each user terminal in the historical federated learning process based on the local training logs of each user terminal.

[0105] The cluster building module is used to select a portion of user terminals to form a user terminal cluster based on local training performance characteristics, so as to execute the next federated learning;

[0106] The training data partitioning module is used to divide the training dataset into several training data subsets that correspond one-to-one with each user terminal based on the local training process characteristics of each user terminal in the user terminal cluster.

[0107] The local model return module is used to return a local model to the server based on the local training status of the user's training data subset in the next federated learning;

[0108] The local model adjustment and fusion module is used to return defective local models to the corresponding user terminals for retraining locally, and then fuse all local models received from the server.

[0109] The beneficial effects of the above embodiments are as follows: the client-side training scheduling system for federated learning determines the local training performance characteristics of each client based on the local training logs of all clients, and selects a portion of clients to form a client cluster to execute the next federated learning iteration; based on the local training process characteristics of each client within the cluster, several training data subsets corresponding one-to-one with each client are obtained, and based on the local training status of the client's training data subset in the next federated learning iteration, a local model is returned to the server; after returning the defective local model to the client for retraining, all local models received by the server are fused, and several client-side systems with balanced performance are selected to participate in federated learning through local training performance identification, which can effectively reduce the time wasted due to client-side waiting for each other; the federated learning process is optimized according to the local training performance of the client, improving the training speed and accuracy of federated learning.

[0110] In another embodiment, the local training performance identification module is used to determine the local training performance characteristics of each user terminal during the historical federated learning process based on the local training logs of each user terminal, including:

[0111] Extract local training task records from the local training logs of all users, analyze the local training task records, and obtain the time consumption characteristics of local training tasks executed by users in different federated learning processes.

[0112] The cluster building module is used to select a subset of user terminals to form a user terminal cluster based on local training performance characteristics, in order to perform the next federated learning iteration, including:

[0113] Based on the time consumption characteristics of local training tasks, the time difference information of user terminals completing local training tasks is determined; based on the time difference information, a portion of user terminals are selected to form a user terminal cluster to execute the next federated learning.

[0114] The beneficial effects of the above embodiments are that, as participants in federated learning, user terminals can utilize training data for local training, thereby participating in the local training of machine learning models and obtaining a completed local model. The local training performance of different user terminals varies, not only between different user terminals but also particularly within the same user terminal for different types of training data. For example, some user terminals exhibit high training efficiency and accuracy for certain types of training data, completing local training in a shorter time, while exhibiting lower efficiency and accuracy for other types of training data, requiring a longer time to complete local training. Furthermore, due to differences in their own computing and memory resources, different user terminals also show differences in training efficiency and accuracy for the same type of training data. Each user terminal generates a corresponding local training task record and uploads it to the local training log each time it performs local training. Therefore, the local training log comprehensively records all historical local training process information of the user terminal, including but not limited to the start and end times of each historical local training session, and the usage of computing and memory resources for local training. The better the local training performance of a user terminal, the less time it takes to complete local training. The time it takes for a user terminal to complete local training directly reflects its local training performance. Therefore, by analyzing the local training time of each user terminal, we can determine whether the local training performance is balanced across different user terminals. Specifically, we extract the local training logs of each user terminal from their respective local training logs, perform local training time-domain feature identification on the local training task records, obtain the time spent by each user terminal on all local training tasks executed in different federated learning projects throughout history, and determine the average local training time of all local training tasks. This average local training time is then used as the time consumption feature of the user terminal's local training tasks. Furthermore, we calculate the difference between the average local training times of all user terminals to obtain the difference between the average local training times of any two user terminals. This difference serves as the time difference information for all user terminals completing local training tasks. The smaller the difference in average local training time between two user terminals, the smaller the difference in local training performance between the two user terminals. Based on the time difference information of all user terminals completing local training tasks, a portion of user terminals are selected to form a user terminal cluster. This ensures that the difference in average local training time between any two user terminals in the selected user terminal cluster is less than a preset difference threshold, and that the offset of the average local training time of each selected user terminal relative to the predetermined target time is less than a preset offset threshold. This guarantees that all user terminals in the user terminal cluster have high local training performance and relatively balanced local training performance, reducing the time wasted by the user terminal cluster waiting for each other during the next federated learning process.

[0115] In another embodiment, the training data partitioning module is used to partition the training dataset into several training data subsets corresponding one-to-one with each user terminal based on the local training process characteristics of each user terminal within the user terminal cluster, including:

[0116] Obtain the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data. The training error rate characteristics include the error frequency of each type of training data during local training. Based on the local training error rate characteristics of each user terminal, divide the training dataset into several training data subsets that correspond one-to-one with each user terminal.

[0117] The local model return module is used to return a local model to the server based on the user's local training state of a subset of training data for the next federated learning iteration. This model includes:

[0118] Based on the user's local training progress on the training data subset in the next federated learning iteration, determine whether the user has completed local training, and return the local model generated after local training to the server.

[0119] The beneficial effects of the above embodiments are that each user terminal in the user terminal cluster performs local training independently. Different user terminals require different computing and memory resources for local training on different types of training data. If a user terminal provides insufficient computing and / or memory resources for local training on a certain type of training data, it cannot guarantee that the user terminal will correctly complete the entire local training, thereby reducing the reliability of the local model obtained through local training. Therefore, it is necessary to allocate appropriate training data to all user terminals under the user terminal cluster to ensure the accuracy of local training. Specifically, the computing and memory resources required by each user terminal in the user terminal cluster for local training on different types of training data are first obtained. Combined with the computing and memory resources that each user terminal can provide, the temporal variation characteristics of the computing and memory resource gaps for each user terminal during local training on different types of training data are determined. This allows for the determination of the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data. Generally, the longer the duration during which the computing power and / or memory resource gap values ​​exceed a preset gap threshold during local training of a certain type of training data on a user end, the higher the error frequency during local training of that type of training data on the user end. Based on the error rate characteristics of local training on all types of training data by all user ends within the user end cluster, the training dataset is divided into several training data subsets corresponding to each user end. This ensures that each user end can provide sufficient computing power and memory resources for local training on its corresponding training data subset. Furthermore, the local training progress of the user end's training data subset in the next federated learning iteration is obtained to determine whether the user end has completed local training. If completed, the locally generated local model is returned to the server; otherwise, the local model is not returned to the server, thus ensuring that the server can quickly obtain local models returned from different user ends within a relatively concentrated time interval.

[0120] In another embodiment, the local model adjustment and fusion module is used to return the defective local model to the corresponding user terminal for retraining locally, and then fuse all local models received by the server, including:

[0121] The system performs model performance identification on all local models received from the server to determine the compatibility of each local model with other local models. This helps determine if any local models have defects. Defective local models are then returned to the corresponding user terminals for retraining. Finally, all local models received from the server are fused together.

[0122] The beneficial effect of the above embodiments is that the server needs to fuse all the local models generated by the user terminals through local training to construct a complete machine learning model. The performance parameters of each local model, such as data processing speed and data processing reliability, directly affect the overall performance of the machine learning model. To ensure good overall performance of the machine learning model, it is necessary to identify the performance parameters of all local models received by the server, such as data processing speed and data processing reliability, to determine whether the data processing speed and reliability of each local model are compatible with its neighboring local models, obtaining a compatibility value between the local models and other local models. If the compatibility value is less than a preset compatibility threshold, the local model is determined to have a defect; otherwise, it is determined that the local model has no defect. The defective local models are then returned to their corresponding user terminals for retraining to improve the local models. The user terminals that have completed retraining are then returned to the server. In this way, the server can fuse all the non-defective local models to obtain a complete machine learning model, ensuring the reliability of the machine learning model.

[0123] In another embodiment, model performance identification is performed on all local models received by the server to obtain the compatibility degree of the local models with other local models, including:

[0124] Extract the model parameter distribution features, sample prediction probability distribution features, and model contribution weights for each local model received by the server;

[0125] The parameter distribution fit between each local model and other models is obtained by utilizing the model parameter distribution characteristics of each local model and other models.

[0126] The parameter distribution fit is obtained by the following formula:

[0127]

[0128] Where A represents the parameter distribution fit between each local model and other models; n represents the number of core parameter layers of the model; σ bi σ represents the standard deviation of the parameters of the i-th layer of each other model corresponding to the local model; ai μ represents the standard deviation of the parameters of the i-th layer in each local model. bi μ represents the mean of the parameters of the i-th layer of each other model corresponding to the local model; ai This represents the mean of the parameters of the i-th layer in each local model;

[0129] Retrieve the predicted probability vectors of each local model and other models on the same key samples, and use the predicted probability vectors of each local model and other models on the same key samples to obtain the corresponding cosine similarity.

[0130] Retrieve the model contribution weights of each local model to other models, and use these weights to obtain the contribution correlation between each local model and other models.

[0131] The contribution correlation between each local model and other models is obtained by the following formula;

[0132]

[0133] Where B represents the contribution correlation between each local model and other models; W a W represents the model contribution weight corresponding to each local model; b These represent the model contribution weights of other models corresponding to each local model;

[0134] The compatibility coefficient between each local model and other models is obtained by using the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models. The larger the compatibility coefficient, the higher the compatibility.

[0135] The compatibility coefficient is obtained using the following formula:

[0136]

[0137] Where J represents the compatibility coefficient; x 01 x 02 and x 03 , respectively, represent the weight values ​​corresponding to the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models; A, S, and B represent the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models, respectively.

[0138] The beneficial effects of the above embodiments are that, by extracting three core types of information—model parameter distribution features, sample prediction probability distribution features, and model contribution weights—the limitations of traditional single-dimensional evaluation are overcome, achieving a comprehensive characterization of model compatibility characteristics and avoiding misjudgments of compatibility due to incomplete information. Simultaneously, calculating parameter distribution fit based on the mean and standard deviation of the core parameter layer accurately measures the synergy at the parameter level between models, reducing interference from parameter conflicts in subsequent fusion. Furthermore, calculating cosine similarity through the prediction probability vectors of the same key samples effectively assesses the consistency of prediction logic between models, ensuring the fused model performs well in core task scenarios. The system improves the predictive stability under different scenarios. Simultaneously, by calculating the contribution correlation using model contribution weights, the system can balance the fit between different value models, avoiding the drag on the fusion effect of low-contribution models on high-contribution models. Finally, by weighting the parameter distribution fit, cosine similarity, and contribution correlation using preset weights, a compatibility coefficient is obtained. This achieves the orderly integration of multi-dimensional indicators, ensuring the accuracy and objectivity of the compatibility assessment. This provides a reliable basis for subsequent identification of defective models and optimization of the model fusion process, thereby improving the performance stability of the final fused model and the efficiency of server-side model fusion, and reducing the risk of fusion failure caused by defective models.

[0139] In summary, this client-side training scheduling method and system for federated learning determines the local training performance characteristics of each client based on their local training logs. This allows for the selection of a subset of clients to form a client cluster for the next federated learning iteration. Based on the local training process characteristics of each client within the cluster, several training data subsets corresponding to each client are obtained. Based on the local training status of these subsets for the next federated learning iteration, local models are returned to the server. Defective local models are returned to the clients for retraining. All local models received by the server are then merged. By identifying and selecting clients with balanced performance based on their local training performance, the system effectively reduces time wasted due to client waiting for each other. Finally, the federated learning process is optimized based on the local training performance of each client, improving both the training speed and accuracy of federated learning.

[0140] The above is only one specific embodiment of the present invention, and any improvements made based on the concept of the present invention shall be considered within the scope of protection of the present invention.

Claims

1. A client-side training scheduling method for federated learning, characterized in that, include: Based on the local training logs of each user terminal, the local training performance characteristics of the user terminal in the historical federated learning process are determined. Based on the local training performance characteristics, a portion of the user terminals are selected to form a user terminal cluster, which is then used to perform the next federated learning. Based on the local training process characteristics of each user terminal in the user terminal cluster, the training dataset is divided into several training data subsets that correspond one-to-one with each user terminal; based on the local training state of the user terminal for the training data subset in the next federated learning, a local model is returned to the server. After returning the defective local model to the corresponding user terminal for retraining locally, all local models received by the server are fused together.

2. The client-side training scheduling method for federated learning as described in claim 1, characterized in that: Based on the local training logs of each user terminal, the local training performance characteristics of the user terminal in the historical federated learning process are determined. Based on the local training performance characteristics, a subset of user terminals are selected to form a user terminal cluster for the next federated learning iteration, including: Extract local training task records from the local training logs of each user terminal, analyze the local training task records, and obtain the time consumption characteristics of the local training tasks executed by the user terminal in different federated learning processes. Based on the time consumption characteristics of the local training task, the time difference information of the user terminal completing the local training task is determined; based on the time difference information, a portion of the user terminals are selected to form a user terminal cluster to perform the next federated learning.

3. The client-side training scheduling method for federated learning as described in claim 1, characterized in that: Based on the local training process characteristics of each user terminal in the user terminal cluster, the training dataset is divided into several training data subsets that correspond one-to-one with each user terminal. Based on the local training state of the training data subset on the user terminal in the next federated learning iteration, a local model is returned to the server, including: Obtain the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data, wherein the training error rate characteristics include the error frequency of each user terminal during local training of each type of training data; based on the local training error rate characteristics of each user terminal, divide the training dataset into several training data subsets that correspond one-to-one with each user terminal. Based on the local training progress of the user client on the training data subset in the next federated learning, it is determined whether the user client has completed local training, and the local model generated after the local training is completed is returned to the server.

4. The client-side training scheduling method for federated learning as described in claim 1, characterized in that: After returning the defective local model to the corresponding user terminal for retraining locally, all local models received by the server are fused, including: The system performs model performance identification on all local models received by the server to determine the compatibility of each local model with other local models. This determines whether the local model has defects, and the defective local models are returned to the corresponding user terminal for retraining. Finally, all local models received by the server are fused together.

5. The client-side training scheduling method for federated learning as described in claim 1, characterized in that: Perform model performance identification on all local models received by the server to obtain the compatibility degree of the local models with other local models, including: Extract the model parameter distribution features, sample prediction probability distribution features, and model contribution weights for each local model received by the server; The parameter distribution fit between each local model and other models is obtained by utilizing the model parameter distribution characteristics of each local model and other models. The parameter distribution fit is obtained by the following formula: Where A represents the parameter distribution fit between each local model and other models; n represents the number of core parameter layers of the model; σ bi σ represents the standard deviation of the parameters of the i-th layer of each other model corresponding to the local model; ai μ represents the standard deviation of the parameters of the i-th layer in each local model. bi μ represents the mean of the parameters of the i-th layer of each other model corresponding to the local model; ai This represents the mean of the parameters of the i-th layer in each local model; Retrieve the predicted probability vectors of each local model and other models on the same key samples, and use the predicted probability vectors of each local model and other models on the same key samples to obtain the corresponding cosine similarity. Retrieve the model contribution weights of each local model to other models, and use these weights to obtain the contribution correlation between each local model and other models. The contribution correlation between each local model and other models is obtained by the following formula; Where B represents the contribution correlation between each local model and other models; W a W represents the model contribution weight corresponding to each local model; b These represent the model contribution weights of other models corresponding to each local model; The compatibility coefficient between each local model and other models is obtained by using the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models. The larger the compatibility coefficient, the higher the compatibility. The compatibility coefficient is obtained using the following formula: Where J represents the compatibility coefficient; x 01 x 02 and x 03 , respectively, represent the weight values ​​corresponding to the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models; A, S, and B represent the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models, respectively.

6. A client-side training scheduling system for federated learning, characterized in that, include: The local training performance identification module is used to determine the local training performance characteristics of each user terminal in the historical federated learning process based on the local training logs of each user terminal. The cluster building module is used to select a portion of user terminals to form a user terminal cluster based on the local training performance characteristics, so as to perform the next federated learning. The training data partitioning module is used to divide the training dataset into several training data subsets that correspond one-to-one with all user terminals based on the local training process characteristics of each user terminal in the user terminal cluster. The local model return module is used to return a local model to the server based on the local training status of the training data subset on the user terminal in the next federated learning. The local model adjustment and fusion module is used to return defective local models to the corresponding user terminals for retraining locally, and then fuse all local models received by the server.

7. The client-side training scheduling system for federated learning as described in claim 6, characterized in that: The local training performance identification module is used to determine the local training performance characteristics of each user terminal during the historical federated learning process based on the local training logs of each user terminal, including: Extract local training task records from the local training logs of each user terminal, analyze the local training task records, and obtain the time consumption characteristics of the local training tasks executed by the user terminal in different federated learning processes. The cluster building module is used to select a portion of user terminals to form a user terminal cluster based on the local training performance characteristics, and then perform the next federated learning, including: Based on the time consumption characteristics of the local training task, the time difference information of the user terminal completing the local training task is determined; based on the time difference information, a portion of the user terminals are selected to form a user terminal cluster to perform the next federated learning.

8. The client-side training scheduling system for federated learning as described in claim 6, characterized in that: The training data partitioning module is used to divide the training dataset into several training data subsets corresponding one-to-one with each user terminal, based on the local training process characteristics of each user terminal in the user terminal cluster, including: Obtain the local training error rate characteristics of each user terminal in the user terminal cluster for different types of training data, wherein the training error rate characteristics include the error frequency of each user terminal during local training of each type of training data; based on the local training error rate characteristics of each user terminal, divide the training dataset into several training data subsets that correspond one-to-one with each user terminal. The local model return module is used to return a local model to the server based on the local training state of the training data subset on the user terminal in the next federated learning, including: Based on the local training progress of the user client on the training data subset in the next federated learning, it is determined whether the user client has completed local training, and the local model generated after the local training is completed is returned to the server.

9. The client-side training scheduling system for federated learning as described in claim 6, characterized in that: The local model adjustment and fusion module is used to return defective local models to the corresponding user terminals for retraining locally, and then fuse all local models received by the server, including: The system performs model performance identification on all local models received by the server to determine the compatibility of each local model with other local models. This determines whether the local model has defects, and the defective local models are returned to the corresponding user terminal for retraining. Finally, all local models received by the server are fused together.

10. The client-side training scheduling system for federated learning as described in claim 6, characterized in that: Perform model performance identification on all local models received by the server to obtain the compatibility degree of the local models with other local models, including: Extract the model parameter distribution features, sample prediction probability distribution features, and model contribution weights for each local model received by the server; The parameter distribution fit between each local model and other models is obtained by utilizing the model parameter distribution characteristics of each local model and other models. The parameter distribution fit is obtained by the following formula: Where A represents the parameter distribution fit between each local model and other models; n represents the number of core parameter layers of the model; σ bi σ represents the standard deviation of the parameters of the i-th layer of each other model corresponding to the local model; ai μ represents the standard deviation of the parameters of the i-th layer in each local model. bi μ represents the mean of the parameters of the i-th layer of each other model corresponding to the local model; ai This represents the mean of the parameters of the i-th layer in each local model; Retrieve the predicted probability vectors of each local model and other models on the same key samples, and use the predicted probability vectors of each local model and other models on the same key samples to obtain the corresponding cosine similarity. Retrieve the model contribution weights of each local model to other models, and use these weights to obtain the contribution correlation between each local model and other models. The contribution correlation between each local model and other models is obtained by the following formula; Where B represents the contribution correlation between each local model and other models; W a W represents the model contribution weight corresponding to each local model; b These represent the model contribution weights of other models corresponding to each local model; The compatibility coefficient between each local model and other models is obtained by using the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models. The larger the compatibility coefficient, the higher the compatibility. The compatibility coefficient is obtained using the following formula: Where J represents the compatibility coefficient; x 01 x 02 and x 03 , respectively, represent the weight values ​​corresponding to the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models; A, S, and B represent the parameter distribution fit, cosine similarity, and contribution correlation between each local model and other models, respectively.

Images