A federated directive tuning method and system based on data quality control
By conducting data quality assessment and hierarchical training in federated learning, filtering out low-quality data and gradually introducing complex data samples, the problem of inconsistent data quality in federated learning is solved, improving the robustness and training efficiency of the model, while protecting data privacy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-08-15
- Publication Date
- 2026-07-24
AI Technical Summary
In a federated learning environment, large language models face the problem of inconsistent data quality, which leads to low training efficiency and slowed convergence speed. Existing technologies lack effective data quality control methods, which are particularly difficult to apply in generation tasks.
The server sends global initial model parameters and quality thresholds to the client, performs instruction-response consistency scoring, filters low-quality data, trains high-quality data in layers, gradually introduces complex data samples, and updates global model parameters through weighted averaging.
It improves the robustness and training efficiency of the model, reduces the negative impact of low-quality data, enhances the overall performance and stability of the model, and protects data privacy.
Smart Images
Figure CN119026704B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information technology, and more specifically, to a method and system for optimizing federated instructions based on data quality control. Background Technology
[0002] With the development of big data and artificial intelligence, the performance of machine learning models increasingly depends on the quantity and quality of training data. However, in many privacy-sensitive fields, such as healthcare and finance, data privacy issues limit the centralized sharing and utilization of large-scale data. Federated learning, as a distributed training method, solves the data privacy problem by allowing multiple clients to collaboratively train a unified model without sharing their data, and has broad application prospects. Large language models are a type of model that has made significant progress in the field of natural language processing in recent years. Through training on large-scale instruction-response data, models are able to better understand and perform complex tasks.
[0003] However, instruction tuning for large language models in a federated learning environment faces challenges in data quality control. Due to the diverse data sources from various clients and the varying quality of the data, it is difficult to detect and filter low-quality data, directly impacting the model's training effectiveness and final performance. Current data quality control schemes for large language models focus on evaluating data quality. Ignoring these issues, especially the importance of data quality, during training can lead to low training efficiency and a significant slowdown in convergence speed. If the model encounters data of varying difficulty levels during training, particularly a large number of complex samples in the initial stages, this can exacerbate the instability of the learning process and negatively affect the training convergence speed.
[0004] Existing federated learning methods primarily focus on expanding data volume and protecting privacy, while lacking in-depth research on data quality control. Current work on data quality in federated learning mainly concentrates on classification tasks, which are largely difficult to transfer to language model generation tasks and adapt to real-world scenarios in federated learning. Summary of the Invention
[0005] In view of the deficiencies in the prior art, the purpose of this invention is to provide a method and system for optimizing federal instructions based on data quality control.
[0006] According to one aspect of the present invention, a method for optimizing federated instructions based on data quality control is provided, comprising:
[0007] The server sends global initial model parameters and quality thresholds to the client;
[0008] The global initial model parameters are used to score the command-response consistency of local data samples on each client to evaluate the quality of each data sample.
[0009] Low-quality data is filtered based on the quality threshold, and only data samples with an instruction-response consistency score higher than the quality threshold are retained.
[0010] The filtered sample data are sorted from high to low according to the instruction-response consistency score and divided into multiple levels, each level containing the same number of data samples;
[0011] Each client uses the data samples for local training in descending order of hierarchy.
[0012] After each training round, the server receives the locally updated parameters from all participating clients and updates the global model parameters by weighted averaging.
[0013] Repeat the above steps until the preset number of training rounds is reached.
[0014] Preferably, the step of using the global initial model parameters to perform instruction-response consistency scoring on the local data samples of each client to evaluate the quality of each data sample specifically involves:
[0015] f(q i a i θ 0 )=L(a i ;θ 0 )-L((q i a i );θ 0 )
[0016] Where f is a function for calculating data quality, i.e., a high instruction-response consistency score; (q i a i )∈D n D n Let L(a) be the local client dataset for user n, where n ∈ [1, N], and N is the number of users participating in federated learning; i ;θ 0 ) indicates that there is no instruction q i In the case of generating response a i The loss, L((q) i a i );θ 0 ) indicates that in the given instruction q i In the case of generating response a i The loss; θ 0 These are the initial global model parameters.
[0017] Preferably, the step of filtering low-quality data according to the quality threshold and retaining only data samples whose instruction-response consistency score is higher than the quality threshold specifically involves:
[0018] D′ n ={(q i a i )∈D n |f(q i a i θ 0 )≥λ}
[0019] Where D n Let D′ be the local dataset of user n. n The filtered dataset is λ, which is the set quality threshold.
[0020] Preferably, the filtered sample data is sorted from high to low according to the instruction-response consistency score and divided into multiple levels, each level containing the same number of data samples, specifically:
[0021]
[0022] Where (q) i a i Taken from the dataset after ascending sorting. m is the filtered local dataset D′ n The amount of data, k represents the level, and n represents the client.
[0023] Preferably, each client uses data samples for local training sequentially according to hierarchical order, gradually introducing data samples from lower-scoring hierarchical levels, specifically as follows:
[0024] The client n selects the data H corresponding to the layer k based on the training round. nk Perform local training and update local model parameters.
[0025] Assuming the total number of training rounds is R, and the number of training rounds for each layer except the last layer is... The number of training rounds for the last layer is
[0026] Preferably, updating the local model parameters is achieved using gradient descent:
[0027]
[0028] Where η is the learning rate. For local loss function, For parameters gradient, Initialize to t is the current local update step number, and local updates stop when t = T, where T is the number of steps in each round of local updates.
[0029] Preferably, after each round of training, the server receives the locally updated parameters from all participating clients and updates the global model parameters using a weighted average, specifically as follows:
[0030]
[0031] Where |D n |For dataset D n The size of the data, The weights, representing the weighting, are typically assigned based on the size of the dataset for each client, S. r It is the set of users participating in this round of training. If all users participate in this round of training, then it is S. r ={1,2,…,N}.
[0032] According to a second aspect of the present invention, a federated instruction tuning system based on data quality control is provided, comprising:
[0033] Initialization module: The server sends global initial model parameters and quality thresholds to the client;
[0034] Data quality assessment module: Uses the global initial model parameters to score the command-response consistency of local data samples on each client, and evaluates the quality of each data sample;
[0035] Data filtering module: Filters low-quality data according to the quality threshold, and retains only data samples whose instruction-response consistency score is higher than the quality threshold;
[0036] Data sorting and stratification module: Sorts the filtered sample data from high to low according to the command-response consistency score, and divides it into multiple strata, each containing the same number of data samples;
[0037] Layered training module: Each client uses data samples for local training in hierarchical order, gradually introducing data samples from lower-scoring layers;
[0038] Global model aggregation module: After each round of training, the server receives the locally updated parameters from all participating clients and updates the global model parameters by weighted average.
[0039] Repeated modules: Repeatedly run the data quality assessment module, data filtering module, data sorting and stratification module, stratified training module, and global model aggregation module until the preset training rounds are reached.
[0040] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform the method described therein, or to run the system described therein.
[0041] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described thereon, or to perform the system described thereon.
[0042] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:
[0043] This invention provides a federated instruction tuning method based on data quality control, which implements globally consistent and efficient data quality assessment. Combined with a hierarchical training method, it can improve the model's robustness to noise and enhance the model's training effect and performance.
[0044] A federated instruction tuning method based on data quality control in this invention embodiment can effectively reduce the negative impact of low-quality data: by filtering low-quality data, it ensures that the data participating in training has high consistency, thereby improving the overall performance of the model.
[0045] A federated instruction tuning method based on data quality control in this embodiment of the invention can improve the efficiency and robustness of model training: the quality-aware hierarchical training method enables the model to gradually adapt to the complexity of the data, prevents the model from overfitting to noisy data, and improves the robustness and training efficiency of the model. Attached Figure Description
[0046] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0047] Figure 1 This is a flowchart of a preferred embodiment of the federated instruction tuning method based on data quality control according to the present invention;
[0048] Figure 2 This is a schematic diagram of a federal instruction tuning method based on data quality control in a preferred embodiment of the present invention. Detailed Implementation
[0049] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0050] The federated instruction tuning method based on data quality control in this invention is based on the federated averaging method, a fundamental method in federated learning, which is used as a reference. Data quality assessment and data training methods are then added to this foundation. The process of the federated averaging method is as follows:
[0051] A global model needs to be initialized before federated training begins. That is, at the start of training, the server initializes a global model parameter θ. 0 This model will be used in the subsequent federated training process.
[0052] In each round of training:
[0053] First, client selection is required. The server randomly selects a subset of clients n∈S from all clients. r It participates in the training of the current round r. The purpose of doing this is to reduce communication overhead and computational burden.
[0054] Next, during local model training, each selected client n receives the current global model parameters from the server.
[0055] Then, the client uses its local dataset D n The model parameters are trained and updated locally.
[0056] The training process typically minimizes the local loss function L through multiple local gradient descent operations (or other optimization methods). n (θ):
[0057]
[0058] Where η is the learning rate. For local loss function, For parameters gradient, Initialize to t is the current local update step number, and local updates stop when t = T, where T is the number of steps in each round of local updates.
[0059] After each client completes local training, it will update the model parameters. Uploaded to the server. At this point, the server will receive updated model parameters from multiple clients. After each round, the server performs a global model aggregation. The server receives the local update parameters from all participating clients and updates the global model parameters using a weighted average.
[0060]
[0061] in The weights, representing the weighting, are typically assigned based on the size of the dataset for each client, S. r It is the set of users participating in this round of training. If all users participate in this round of training, then it is S. r ={1,2,…,N},|D n |For dataset D n The amount of data. Repeat the above steps until the training round r = R.
[0062] The federated instruction tuning method based on data quality control proposed in this embodiment of the invention has two innovative adjustments compared to the federated averaging method. First, a data quality assessment and filtering part is added after initializing the global model and before training begins. Second, during the model training phase, the data is divided into multiple subsets for hierarchical training. Specifically, in one embodiment of the invention, a federated instruction tuning method based on data quality control is provided, such as... Figure 1 and Figure 2 As shown, its main process is as follows:
[0063] S1, the server sends the global initial model parameters and quality threshold to the client;
[0064] S2 uses global initial model parameters to score the command-response consistency of local data on each client, evaluating the quality of each data sample.
[0065] S3 filters low-quality data based on a preset quality threshold, retaining only data samples whose instruction-response consistency score calculated by S2 is higher than the threshold.
[0066] S4 sorts the sample data after S3 filtering according to the instruction-response consistency score from high to low and divides them into multiple levels, each level containing the same number of data samples.
[0067] S5, each client uses data samples for local training in the order of the levels divided in S4, and gradually introduces data samples of lower-scoring levels.
[0068] S6. After each training round, the server receives the locally updated parameters from all participating clients and updates the global model parameters by weighted averaging.
[0069] Repeat steps S1-S6 until the preset number of training rounds is reached.
[0070] In the above embodiments, by implementing globally consistent and efficient data quality assessment during training, combined with a hierarchical training method, the model's robustness to noise can be improved, and the model's training effect and performance can be enhanced. This changes the situation in existing technologies where the importance of data quality is ignored during training.
[0071] In a preferred embodiment of the present invention, step S2 is performed to evaluate the quality of each data sample. This embodiment proposes an instruction-response consistency metric to address data quality issues. This metric evaluates data quality by calculating the difference between the unconditional response generation loss and the conditional response generation loss for each data sample.
[0072] Specifically, the command-response consistency score is calculated as follows: Using a global initial model, inference is performed on the local data of each client, and the loss L(a) for generating a response for each sample under conditions without a command is calculated. i ;θ) and the loss L((q) for generating a response under given instructions i ,a i The instruction-response consistency score is defined as the difference between these two values, i.e., θ).
[0073] f(q i ,a i ,θ)=L(a i ;θ)-L((q i ,a i );θ),
[0074] Where f is a function for calculating data quality, it takes data points and the model as input and outputs a vector representing the data quality of that data point. The high instruction-response consistency score represents instruction q. i and response a i A high correlation between the data indicates high data quality; a low command-response consistency score indicates low data quality, which may be due to noisy data.
[0075] After completing the above data quality assessment, the data needs to be filtered. In a preferred embodiment of the present invention, implementation S2 provides a superior filtering method. That is, each client filters data samples according to a globally set quality threshold λ. Specifically, samples below the threshold are filtered out, retaining only high-quality data. This process is performed entirely locally on the client to ensure data privacy.
[0076] The formula representing the filtering process is:
[0077] D′ n ={(q i ,a i )∈D n |f(q i ,a i ,θ 0 )≥λ}
[0078] Where D n For the local dataset of customer n, D′ nThis is the filtered dataset.
[0079] In the above embodiments, by filtering low-quality data, the consistency of the data used for training is ensured, thereby improving the overall performance of the model.
[0080] In a preferred embodiment of the present invention, step S4 is performed, in which the filtered data is sorted from high to low according to the instruction-response consistency score, and further divided into multiple levels. Each level contains approximately the same number of data samples, aiming to help the model learn more effectively by gradually introducing complexity.
[0081] The formulas representing sorting and stratification are as follows:
[0082]
[0083] Where (q) i ,a i Taken from the dataset after ascending sorting. m is the filtered local dataset D ′ n The amount of data.
[0084] In a preferred embodiment of the present invention, step S5 is implemented to perform layer-by-layer training. The client uses the data for local training sequentially according to the layer order.
[0085] First, use the highest-rated level. Train the program, then use the lower-scoring layers sequentially. Conduct training.
[0086] Assuming the total number of training rounds is R, then the number of training rounds for each layer, except the last layer, is... The number of training rounds for the last layer is The design aims to ensure that each layer of data is processed with the same number of steps as much as possible during training, thereby achieving a more balanced and efficient training effect.
[0087] In the above embodiments, a tiered training strategy was employed to ensure that the model first builds a solid foundation on high-quality, highly relevant data. Subsequently, the model gradually adapts to more complex data environments, effectively avoiding overfitting and noise interference that might occur if the entire dataset were used directly. This training process not only significantly improves training efficiency and effectiveness but also significantly enhances the model's robustness. Furthermore, all data processing (including evaluation, filtering, and sorting) is performed locally on the client side, without involving the uploading or sharing of raw data, thus fully protecting data privacy.
[0088] In a preferred embodiment of the present invention, step S6 is performed to aggregate parameters. Specifically, after each round of training, the server receives the locally updated parameters from all participating clients and updates the global model parameters using a weighted average method, as follows:
[0089]
[0090] in The weights, representing the weighting, are typically assigned based on the dataset size for each client, |D n |For dataset D n Data size, S r It is the set of users participating in this round of training. If all users participate in this round of training, then it is S. r ={1,2,…,N}.
[0091] The local updates and global aggregations in each round are the same as the federated averaging method, and the entire training process is repeated until the training round r = R.
[0092] Based on the same inventive concept, other embodiments of the present invention provide a federated instruction tuning system based on data quality control, including:
[0093] Initialization module: The server sends global initial model parameters and quality thresholds to the client;
[0094] Data quality assessment module: Uses the global initial model parameters to score the command-response consistency of local data samples on each client, and evaluates the quality of each data sample;
[0095] Data filtering module: Filters low-quality data according to the quality threshold, and retains only data samples whose instruction-response consistency score is higher than the quality threshold;
[0096] Data sorting and stratification module: Sorts the filtered sample data from high to low according to the command-response consistency score, and divides it into multiple strata, each containing the same number of data samples;
[0097] Layered training module: Each client uses data samples for local training in hierarchical order, gradually introducing data samples from lower-scoring layers;
[0098] Global model aggregation module: After each round of training, the server receives the locally updated parameters from all participating clients and updates the global model parameters by weighted average.
[0099] Repeated modules: Repeatedly run the data quality assessment module, data filtering module, data sorting and stratification module, stratified training module, and global model aggregation module until the preset training rounds are reached.
[0100] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the federated instruction tuning method based on data quality control in the above embodiments, and will not be repeated here.
[0101] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the aforementioned federated instruction tuning method based on data quality control, or to run the aforementioned federated instruction tuning system based on data quality control.
[0102] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to execute the aforementioned data quality control-based federated instruction tuning method, or to execute the aforementioned data quality control-based federated instruction tuning system.
[0103] The technical solutions described in the above embodiments of the present invention have numerous industrial application scenarios, especially since many real-world applications require federated instruction tuning methods based on data quality control. For example, in the medical field, federated learning can be used to analyze patient data from different hospitals or clinics to improve disease diagnosis, treatment planning, and drug discovery, while protecting patient privacy.
[0104] For example, federated learning might be used to predict a patient's resistance to a particular treatment. The steps are as follows:
[0105] S100. In each participating healthcare institution, the global initial model is used to score the instruction-response consistency of the local data (patient data samples) of each institution to evaluate the quality of each patient's data sample.
[0106] S200. At each medical facility, patient data is filtered, retaining only data with an instruction-response consistency score higher than a set quality threshold.
[0107] S300. At each medical institution, the screened patient data is divided into levels, with the amount of data in each level remaining the same.
[0108] S400: Perform tiered training locally at each medical institution and update the model parameters for each medical institution.
[0109] S500 aggregates the updated model parameters from various medical institutions to update the global model.
[0110] The S600 method, which repeats the S100-S500 method, is a federated instruction tuning method based on data quality control. It is used to train and iterate the model, optimize the model parameters, and improve the model performance.
[0111] S700: Deploy the trained model to medical institutions with relevant application needs for actual medical diagnosis and treatment.
[0112] In the aforementioned process, all patient data processing (including assessment, screening, and sorting) was strictly performed locally on the client side, without any uploading or sharing of raw data. This was to ensure full protection of data privacy. Simultaneously, by implementing a globally unified and efficient data quality assessment mechanism, combined with a hierarchical training method, the model's tolerance to noise was effectively improved, thereby enhancing the accuracy of predicting patient resistance to specific treatment drugs.
[0113] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.
Claims
1. A method for optimizing federated instructions based on data quality control, characterized in that, include: The server sends global initial model parameters and quality thresholds to the client; The global initial model parameters are used to score the command-response consistency of local data samples on each client to evaluate the quality of each data sample. Low-quality data is filtered based on the quality threshold, and only data samples with an instruction-response consistency score higher than the quality threshold are retained. The filtered sample data are sorted from high to low according to the instruction-response consistency score and divided into multiple levels, each level containing the same number of data samples. Each client uses the data samples for local training in descending order of hierarchy. After each training round, the server receives the locally updated parameters from all participating clients and updates the global model parameters by weighted averaging. Repeat the above steps until the preset number of training rounds is reached; The step of using the global initial model parameters to perform instruction-response consistency scoring on the local data samples of each client, and evaluating the quality of each data sample, specifically involves: ； in, This is a function for calculating data quality, specifically a high instruction-response consistency score. , For user n's local client dataset, N is the number of users participating in federated learning; Indicates no instruction Generate response in the case The loss, Indicates that in the given instruction Generate response in the case The loss; These are the initial global model parameters.
2. The federated instruction tuning method based on data quality control according to claim 1, characterized in that, The step of filtering low-quality data according to the quality threshold and retaining only data samples whose instruction-response consistency score is higher than the quality threshold specifically involves: ； in For user n's local dataset, For the filtered dataset, The set quality threshold.
3. The federated instruction tuning method based on data quality control according to claim 1, characterized in that, The filtered sample data is sorted from high to low according to the instruction-response consistency score and divided into multiple levels, each containing the same number of data samples, specifically: ； in Data taken from ascending sorted dataset , For the filtered local dataset The amount of data, k represents the level, and n represents the client.
4. The federated instruction tuning method based on data quality control according to claim 1, characterized in that, Each client uses data samples for local training sequentially according to hierarchical order, gradually introducing data samples from lower-scoring hierarchical levels, specifically as follows: Client Select the corresponding level based on the training rounds. Corresponding data Perform local training and update local model parameters. ; Assume the total number of training rounds is Except for the last layer, the number of training rounds for each layer is... The number of training rounds for the last layer is .
5. The federated instruction tuning method based on data quality control according to claim 4, characterized in that, Updating local model parameters is achieved using gradient descent. ； in For learning rate, For local loss function, For parameters gradient, Initialize to , This represents the current local update step count, and the local update is in... Stop when T is the number of steps in each round of local updates.
6. The federated instruction tuning method based on data quality control according to claim 1, characterized in that, After each training round, the server receives the locally updated parameters from all participating clients and updates the global model parameters using a weighted average. Specifically: ； in For dataset The size of the data, The weights, representing the weighting, are typically assigned based on the size of the dataset for each client. It is the set of users participating in this round of training. If all users participate in this round of training, then it is... .
7. A federated instruction tuning system based on data quality control, used to implement the method of claim 1, characterized in that, include: Initialization module: The server sends global initial model parameters and quality thresholds to the client; Data quality assessment module: Uses the global initial model parameters to score the command-response consistency of local data samples on each client, and evaluates the quality of each data sample; Data filtering module: Filters low-quality data according to the quality threshold, and retains only data samples whose instruction-response consistency score is higher than the quality threshold; Data sorting and stratification module: Sorts the filtered sample data from high to low according to the command-response consistency score, and divides it into multiple strata, each containing the same number of data samples; Layered training module: Each client uses data samples for local training in hierarchical order, gradually introducing data samples from lower-scoring layers; Global model aggregation module: After each round of training, the server receives the locally updated parameters from all participating clients and updates the global model parameters by weighted average. Repeated modules: Repeatedly run the data quality assessment module, data filtering module, data sorting and stratification module, stratified training module, and global model aggregation module until the preset training rounds are reached.
8. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-6.