Data quality scoring method, device, equipment, storage medium and product
By scoring data quality based on preset benchmark models, the inefficiency problem caused by relying on manual rules and simple statistical features in the prior art is solved, and efficient and accurate data quality evaluation and filtering are achieved.
Patent Information
- Application Number
- CN202411870445.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-04
AI Technical Summary
The prior art relies on manual rules and simple statistical features when identifying and filtering low-quality data, resulting in inefficiency and poor adaptability.
By determining the model update parameters of the data to be scored based on the preset benchmark model, calculating the change in the model parameter, and then grading the data quality. This method avoids the subjectivity of manually formulating rules, and uses internal information during model training to capture deep features and complex relationships in the data.
It realizes efficient and accurate data quality evaluation, improves the accuracy and efficiency of data filtering, reduces computational overhead, and does not need to rely on external discriminant models.
Smart Images
Figure CN119760318A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data quality scoring method, device, equipment, storage medium and product. Background Art
[0002] In fields such as machine learning, data quality directly affects the training effect and performance of the model. With the expansion of data scale and the increasing complexity of application scenarios, how to effectively identify and filter low-quality data has become a key issue that needs to be solved. Traditional data filtering methods often rely on manually formulated rules or simple statistical features, which not only requires a lot of human resources, but also has difficulty adapting to the data characteristics in different scenarios. Summary of the invention
[0003] In view of this, the present disclosure proposes a data quality scoring method, device, equipment, storage medium and product to solve the above technical problems.
[0004] In order to achieve the above objectives, the technical solution adopted by the present disclosure is:
[0005] According to a first aspect of an embodiment of the present disclosure, a data quality scoring method is proposed, the method comprising:
[0006] In response to obtaining the data set, each piece of data in the data set is used as data to be scored;
[0007] Determine the model update parameter corresponding to each piece of the to-be-scored data based on a preset benchmark model;
[0008] Determine the model parameter change corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored, wherein the model initial parameters include the initial parameters of the preset benchmark model;
[0009] The quality score of each piece of data to be scored is determined based on the model parameter change corresponding to each piece of data to be scored.
[0010] In some embodiments, determining the model update parameter corresponding to each piece of the to-be-scored data based on a preset benchmark model includes:
[0011] Cloning a preset benchmark model to obtain a cloned benchmark model;
[0012] The cloned benchmark model is trained based on each piece of the data to be scored, respectively, to obtain a model update parameter corresponding to each piece of the data to be scored.
[0013] In some embodiments, the step of training the cloned benchmark model based on each piece of the data to be scored to obtain a model update parameter corresponding to each piece of the data to be scored includes:
[0014] Perform forward propagation calculation based on each piece of the to-be-scored data and the cloned benchmark model to obtain a model prediction value;
[0015] Constructing a loss function based on the difference between the model prediction value and the actual label value;
[0016] Back propagation is performed based on the gradient of the loss function with respect to the parameters, the preset learning rate, and the initial parameters of the model to obtain the model update parameters corresponding to each piece of the data to be scored.
[0017] In some embodiments, the model initial parameters and the model update parameters are parameters of a target key layer, and the target key layer is a key layer that needs to be monitored and is pre-specified in the cloned benchmark model.
[0018] In some embodiments, the parameters of the target key layer include a weight matrix and a bias vector of the target key layer.
[0019] In some embodiments, determining the model parameter change amount corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored includes:
[0020] For each layer of the cloned benchmark model, determine a weight change matrix corresponding to each piece of the data to be scored based on an initial weight matrix and an updated weight matrix corresponding to each piece of the data to be scored, wherein the initial weight matrix is a weight matrix of the cloned benchmark model before training, and the updated weight matrix is a weight matrix of the cloned benchmark model after training;
[0021] For each layer of the cloned benchmark model, the gradient change matrix corresponding to each piece of the data to be scored is determined based on the initial gradient matrix and the updated gradient matrix corresponding to each piece of the data to be scored, wherein the initial gradient matrix is the gradient matrix of the cloned benchmark model before training, and the updated gradient matrix is the gradient matrix of the cloned benchmark model after training.
[0022] In some embodiments, determining the quality score of each piece of data to be scored based on the model parameter change corresponding to each piece of data to be scored includes:
[0023] For each piece of the to-be-scored data, respectively determine a first average value of a model weight change matrix of each layer of the cloned benchmark model and a second average value of a model gradient change matrix of each layer;
[0024] A quality score of each piece of the to-be-scored data is determined based on the first average value and the second average value.
[0025] In some embodiments, determining the quality score of each piece of the to-be-scored data based on the first average value and the second average value includes:
[0026] Determine the sum of the products of the first average value and the second average value of each layer of the cloned benchmark model as the initial score of each piece of the data to be scored;
[0027] The initial score of each piece of data to be scored is normalized to obtain a quality score of each piece of data to be scored.
[0028] In some embodiments, the method further comprises:
[0029] Each piece of data in the data set is filtered based on a preset ratio threshold and the quality score.
[0030] According to a second aspect of an embodiment of the present disclosure, a data quality scoring device is provided, the device comprising:
[0031] A data determination module, configured to, in response to acquiring a data set, use each piece of data in the data set as data to be scored;
[0032] A parameter determination module, used to determine the model update parameter corresponding to each piece of the to-be-scored data based on a preset benchmark model;
[0033] A variable determination module, used to determine the model parameter change corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored, wherein the model initial parameters include the initial parameters of the preset benchmark model;
[0034] The scoring determination module is used to determine the quality score of each piece of data to be scored based on the model parameter change corresponding to each piece of data to be scored.
[0035] In some embodiments, the parameter determination module includes:
[0036] A model cloning unit, used for cloning a preset benchmark model to obtain a cloned benchmark model;
[0037] The parameter determination unit is used to train the clone benchmark model based on each piece of the data to be scored, and obtain the model update parameter corresponding to each piece of the data to be scored.
[0038] In some embodiments, the parameter determination unit is further configured to:
[0039] Perform forward propagation calculation based on each piece of the to-be-scored data and the cloned benchmark model to obtain a model prediction value;
[0040] Constructing a loss function based on the difference between the model prediction value and the actual label value;
[0041] Back propagation is performed based on the gradient of the loss function with respect to the parameters, the preset learning rate, and the initial parameters of the model to obtain the model update parameters corresponding to each piece of the data to be scored.
[0042] In some embodiments, the model initial parameters and the model update parameters are parameters of a target key layer, and the target key layer is a key layer that needs to be monitored and is pre-specified in the cloned benchmark model.
[0043] In some embodiments, the parameters of the target key layer include a weight matrix and a bias vector of the target key layer.
[0044] In some embodiments, the variable determination module includes:
[0045] A first matrix determination unit is used to determine, for each layer of the cloned benchmark model, a weight change matrix corresponding to each piece of the data to be scored based on an initial weight matrix and an updated weight matrix corresponding to each piece of the data to be scored, wherein the initial weight matrix is a weight matrix of the cloned benchmark model before training, and the updated weight matrix is a weight matrix of the cloned benchmark model after training;
[0046] The second matrix determination unit is used to determine, for each layer of the cloned benchmark model, the gradient change matrix corresponding to each piece of the data to be scored based on the initial gradient matrix and the updated gradient matrix corresponding to each piece of the data to be scored, wherein the initial gradient matrix is the gradient matrix of the cloned benchmark model before training, and the updated gradient matrix is the gradient matrix of the cloned benchmark model after training.
[0047] In some embodiments, the score determination module includes:
[0048] a mean value determination unit, used to determine, for each piece of the to-be-scored data, a first mean value of a model weight change matrix of each layer of the cloned benchmark model and a second mean value of a model gradient change matrix of each layer;
[0049] A score determination unit is used to determine a quality score of each piece of the to-be-scored data based on the first average value and the second average value.
[0050] In some embodiments, the score determination unit is further configured to:
[0051] Determine the sum of the products of the first average value and the second average value of each layer of the cloned benchmark model as the initial score of each piece of the data to be scored;
[0052] The initial score of each piece of data to be scored is normalized to obtain a quality score of each piece of data to be scored.
[0053] In some embodiments, the apparatus further comprises:
[0054] The data filtering module is used to filter each piece of data in the data set based on a preset ratio threshold and the quality score.
[0055] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:
[0056] processor;
[0057] memory for storing computer programs;
[0058] Wherein, the processor is configured to implement any of the above-mentioned data quality scoring methods when executing the computer program.
[0059] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is processed by a processor, the data quality scoring method described in any one of the above items is implemented.
[0060] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is proposed, on which a computer program is stored, and when the computer program is processed by a processor, the data quality scoring method as described in any one of the above items is implemented.
[0061] Compared with the prior art, the present invention, in response to obtaining a data set, takes each data in the data set as data to be scored, and determines the model update parameters corresponding to each data to be scored based on a preset benchmark model, and then determines the model parameter change corresponding to each data to be scored based on the model initial parameters and the model update parameters corresponding to each data to be scored, and then determines the quality score of each data to be scored based on the model parameter change corresponding to each data to be scored. This can realize the use of the influence of the data to be scored on the model training to determine the quality score of the data, avoid the subjectivity of manually formulating rules, and achieve efficient and accurate data quality assessment, thereby improving the accuracy and efficiency of subsequent data filtering, without relying on an external discriminant model, and reducing computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 A flow chart showing a data quality scoring method according to an exemplary embodiment of the present disclosure is shown;
[0063] Figure 2 A flowchart showing how to determine the model update parameter corresponding to each piece of data to be scored based on a preset benchmark model according to an exemplary embodiment of the present disclosure;
[0064] Figure 3 A schematic diagram showing how to obtain the model update parameter corresponding to each piece of data to be scored according to an exemplary embodiment of the present disclosure;
[0065] Figure 4 A schematic diagram showing how to determine the quality score of each piece of data to be scored based on the model parameter change amount corresponding to each piece of data to be scored according to an exemplary embodiment of the present disclosure;
[0066] Figure 5 A schematic diagram showing how to filter each piece of data in the data set based on a preset ratio threshold and the quality score according to an exemplary embodiment of the present disclosure;
[0067] Figure 6 A flow chart of a method for data quality scoring and filtering according to an exemplary embodiment of the present disclosure is shown;
[0068] Figure 7 A structural block diagram of a data quality scoring device according to an exemplary embodiment of the present disclosure is shown;
[0069] Figure 8 A structural block diagram of a data quality scoring device according to another exemplary embodiment of the present disclosure is shown;
[0070] Fig. 9 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0071] The present disclosure will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present disclosure, and any structural, methodological, or functional changes made by a person of ordinary skill in the art based on these embodiments are included in the protection scope of the present disclosure. In addition, the technical features involved in the different embodiments of the present disclosure described below can be combined with each other as long as they do not conflict with each other.
[0072] In fields such as machine learning, data quality directly affects the training effect and performance of the model. With the expansion of data scale and the increasing complexity of application scenarios, how to effectively identify and filter low-quality data has become a key issue that needs to be solved urgently.
[0073] At present, the commonly used data filtering methods in the industry can be mainly divided into the following categories: the first category is rule-based filtering methods, which judge the quality of data through artificially pre-defined rules, such as checking the integrity or consistency of data; the second category is statistical feature-based filtering methods, which identify abnormal data by calculating the statistical features of data (such as mean, variance, etc.); the third category is model-based filtering methods, which distinguish high-quality and low-quality data by training simple discriminant models. Although these methods can work in specific scenarios, they all have obvious limitations:
[0074] Rule-based filtering methods require a lot of manual experience and are difficult to cover all possible situations; filtering methods based on statistical features are often too simple and difficult to capture deep-level features and complex relationships in the data; and existing model-based methods usually use external discriminant models, which adds additional computational overhead.
[0075] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a data quality scoring method, device, equipment, storage medium and product, which directly evaluates the contribution of the data to be scored to the model training by analyzing the parameter changes of the model during the training process, and achieves efficient and accurate data quality scoring. It can not only avoid the subjectivity of manually formulating rules, but also make full use of the internal information in the model training process to capture the deep-level features and complex relationships in the data, thereby improving the accuracy and efficiency of data filtering, and there is no need to rely on external discriminant models, which can reduce computational overhead.
[0076] Figure 1 A flow chart of a data quality scoring method according to an exemplary embodiment of the present disclosure is shown. The method can be performed by a data quality scoring device, which can be configured in an electronic device. This embodiment does not limit the type of electronic device, which can be any device type suitable for implementation, such as a terminal device or a server.
[0077] like Figure 1 As shown, the method includes the following steps S101-S104:
[0078] In step S101, in response to obtaining a data set, each piece of data in the data set is used as data to be scored.
[0079] In this embodiment, after the electronic device obtains the data set whose data quality is to be evaluated, each piece of data in the data set can be used as data to be evaluated.
[0080] For example, if the electronic device receives a data set D to be evaluated for data quality input by a user, it can extract the i-th data d from the data set. i(i＝1,2,…,n), as the data to be scored, where n is the total number of data in the data set.
[0081] In step S102, a model update parameter corresponding to each piece of data to be scored is determined based on a preset benchmark model.
[0082] In this embodiment, after each piece of data in the data set is used as data to be scored, the electronic device can determine the model update parameter corresponding to each piece of data to be scored based on a preset reference model.
[0083] It is worth noting that the type of the above-mentioned preset benchmark model can be selected from relevant technologies based on actual business needs. For example, it can be selected as a model of a Transformer architecture that only includes the decoder part (such as the Liama model or the Qwen model, etc.). This embodiment does not limit this.
[0084] In some embodiments, a preset benchmark model may be trained with each piece of the data to be scored to determine a model update parameter corresponding to each piece of the data to be scored, wherein the model update parameter includes a parameter updated after the preset benchmark model is trained.
[0085] In some other embodiments, the above method of determining the model update parameter corresponding to each piece of the to-be-scored data based on the preset benchmark model can also refer to the following Figure 2 The illustrated embodiment will not be described in detail here.
[0086] In step S103, based on the model initial parameters and the model update parameters corresponding to each piece of the data to be scored, the model parameter change corresponding to each piece of the data to be scored is determined.
[0087] In this embodiment, after the model update parameters corresponding to each piece of data to be scored are determined based on the preset benchmark model, the model parameter change corresponding to each piece of data to be scored can be determined based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored.
[0088] The above-mentioned model initial parameters may include the initial parameters of the preset benchmark model, and the above-mentioned model parameter change amount may be used to characterize the difference between the model initial parameters and the model update parameters corresponding to each piece of the data to be scored.
[0089] It can be understood that the model parameter change corresponding to each of the above-mentioned data to be scored can be used to reflect the degree of influence of each of the data to be scored on the training of the preset benchmark model. That is, the larger the model parameter change, the higher the influence of the data to be scored on the training of the preset benchmark model; conversely, the smaller the model parameter change, the lower the influence of the data to be scored on the training of the preset benchmark model.
[0090] In some embodiments, the above method of determining the model parameter change amount corresponding to each piece of the to-be-scored data can also refer to the following Figure 2 The illustrated embodiment will not be described in detail here.
[0091] In step S104, the quality score of each piece of the data to be scored is determined based on the model parameter change amount corresponding to each piece of the data to be scored.
[0092] In this embodiment, after determining the model parameter change corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored, the quality score of each piece of data to be scored can be determined based on the model parameter change corresponding to each piece of data to be scored.
[0093] It is understandable that the model parameter change corresponding to each of the above-mentioned data to be scored can be positively correlated with the quality score of each of the data to be scored. That is, the larger the model parameter change, the higher the impact of the data to be scored on the training of the preset benchmark model, and thus the higher the quality score of the data to be scored; conversely, the smaller the model parameter change, the lower the impact of the data to be scored on the training of the preset benchmark model, and thus the lower the quality score of the data to be scored.
[0094] In some embodiments, the above method for determining the quality score of each piece of data to be scored can also refer to the following Figure 4 The illustrated embodiment will not be described in detail here.
[0095] On the other hand, after determining the quality score of each piece of data to be scored, each piece of data in the data set may be filtered based on a preset ratio threshold and the quality score. Figure 5 The illustrated embodiment will not be described in detail here.
[0096] From the above description, it can be seen that the method of this embodiment, in response to obtaining a data set, takes each data in the data set as data to be scored, and determines the model update parameters corresponding to each data to be scored based on a preset benchmark model, and then determines the model parameter change corresponding to each data to be scored based on the model initial parameters and the model update parameters corresponding to each data to be scored, and then determines the quality score of each data to be scored based on the model parameter change corresponding to each data to be scored. It can be realized to determine the quality score of the data by utilizing the degree of influence of the data to be scored on the model training, avoid the subjectivity of manually formulating rules, and achieve efficient and accurate data quality assessment, thereby improving the accuracy and efficiency of subsequent data filtering, without relying on an external discriminant model, and reducing computational overhead.
[0097] Figure 2 A flowchart is shown of how to determine the model update parameters corresponding to each piece of data to be scored based on a preset benchmark model according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to determine the model update parameters corresponding to each piece of data to be scored based on a preset benchmark model.
[0098] like Figure 2 As shown, the above step S102 of determining the model update parameter corresponding to each piece of the to-be-scored data based on the preset benchmark model may include the following steps S201-S202:
[0099] In step S201, a preset reference model is cloned to obtain a cloned reference model.
[0100] In this embodiment, when the model update parameter corresponding to each piece of the to-be-scored data is determined based on the preset benchmark model, the preset benchmark model may be cloned to obtain a cloned benchmark model.
[0101] For example, assuming that the neural network model used as the baseline model is M, cloning (i.e., copying) the model can obtain a temporary model M temp , which is the above-mentioned clone benchmark model.
[0102] It can be understood that the baseline model M and the cloned baseline model M temp For example, if the parameter set of the baseline model M is θ, then the parameter set of the cloned baseline model M is also θ.
[0103] In step S202, the cloned benchmark model is trained based on each piece of the data to be scored, and a model update parameter corresponding to each piece of the data to be scored is obtained.
[0104] In this embodiment, after obtaining the above clone benchmark model, based on the i-th data d i Train the clone benchmark model to obtain the i-th data d i The corresponding model update parameter θ i .
[0105] In some embodiments, the clone benchmark model may be trained separately based on each piece of the data to be scored using a preset method in the related art. This embodiment does not limit the specific training method.
[0106] In some other embodiments, the above method of obtaining the model update parameter corresponding to each piece of the to-be-scored data can also refer to the following Figure 3 The illustrated embodiment will not be described in detail here.
[0107] In this embodiment, the model initial parameters and the model update parameters may both be parameters of the target key layer, wherein the target key layer is a key layer that needs to be monitored and is pre-specified in the cloned benchmark model.
[0108] For example, the target key layer can be a feed-forward layer and / or a self-attention layer in a neural network model. It is understandable that in a deep learning model, not all layers are equally important. Some layers (such as layers close to the input layer) may contribute less to the final output. In this embodiment, by focusing on those layers that have a greater impact on model performance (i.e., target key layers), it is helpful to deeply understand the contribution of data to model training, thereby improving the rationality and accuracy of subsequent data quality evaluation.
[0109] Furthermore, the parameters of the target key layer may include the weight matrix W and the bias vector b of the target key layer. On this basis, when determining the model parameter change corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored, the following steps (1) and (2) may be performed:
[0110] (1) For each layer l of the cloned benchmark model, based on the initial weight matrix And the updated weight matrix corresponding to each piece of data to be scored Determine the weight change matrix △W corresponding to each piece of data to be scored l .
[0111] Among them, the above initial weight matrix The weight matrix of the cloned benchmark model before training may be the weight matrix of the updated weight matrix It can be the weight matrix of the cloned benchmark model after training.
[0112] Exemplarily, the weight change matrix ΔW can be determined based on the following formula (2-1): l :
[0113]
[0114] (2) For each layer l of the cloned benchmark model, based on the initial gradient matrix And the updated gradient matrix corresponding to each piece of data to be scored Determine the gradient change matrix ΔG corresponding to each piece of data to be scored l .
[0115] Among them, the above initial gradient matrix The gradient matrix of the cloned benchmark model before training may be the gradient matrix of the updated gradient matrix It can be the gradient matrix of the cloned benchmark model after training.
[0116] For example, the gradient change matrix ΔG can be determined based on the following formula (2-2): l :
[0117]
[0118] On this basis, the method of determining the quality score of each piece of data to be scored based on the model parameter change amount corresponding to each piece of data to be scored described in the above step S104 can also be referred to as follows Figure 4 The illustrated embodiment will not be described in detail here.
[0119] As can be seen from the above description, this embodiment obtains a cloned benchmark model by cloning the preset benchmark model, and trains the cloned benchmark model based on each of the data to be scored to obtain the model update parameters corresponding to each of the data to be scored. This can accurately determine the model update parameters corresponding to each of the data to be scored based on the preset benchmark model, and then subsequently determine the model parameter change corresponding to each of the data to be scored based on the model update parameters and determine the quality score of each of the data to be scored based on the model parameter change. This can determine the quality score of the data by utilizing the degree of influence of the data to be scored on the model training, avoid the subjectivity of manually formulating rules, and achieve efficient and accurate data quality assessment, thereby improving the accuracy and efficiency of subsequent data filtering without relying on an external discriminant model, thereby reducing computational overhead.
[0120] Figure 3 A schematic diagram is shown of how to obtain the model update parameters corresponding to each piece of data to be scored according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to obtain the model update parameters corresponding to each piece of data to be scored as an example to exemplify.
[0121] like Figure 3 As shown, the above step S202, based on each of the data to be scored, respectively trains the clone benchmark model to obtain the model update parameter corresponding to each of the data to be scored, which may include the following steps S301-S303:
[0122] In step S301, forward propagation calculation is performed based on each piece of the to-be-scored data and the cloned benchmark model to obtain a model prediction value.
[0123] In this embodiment, when the reference model is cloned to obtain the cloned reference model M temp After that, the i-th data d can be used i To M temp Perform training, including forward propagation calculation based on the following formula (3-1):
[0124] y i =M temp (d i ); (3-1)
[0125] Among them, y i is the model prediction value, which is the output of the forward propagation calculation.
[0126] In step S302, a loss function is constructed based on the difference between the model prediction value and the actual label value.
[0127] In this embodiment, when the model prediction value y is obtained i Then, the value y can be predicted based on the model i and the actual label value y true The loss function is constructed by taking the difference between .
[0128] For example, the above loss function can be expressed as follows:
[0129] L=Loss(y i ,y true ); (3-2)
[0130] In step S303, back propagation is performed based on the gradient of the loss function with respect to the parameters, the preset learning rate and the initial parameters of the model to obtain the model update parameters corresponding to each piece of the data to be scored.
[0131] In this embodiment, after constructing a loss function based on the difference between the model prediction value and the actual label value, back propagation can be performed based on the gradient of the loss function with respect to the parameters, the preset learning rate, and the initial parameters of the model to obtain the model update parameters corresponding to each piece of data to be scored.
[0132] For example, suppose the gradient of the above loss function with respect to the parameters is The preset learning rate is α and the initial model parameter is θ init , then the model updates the parameter θ new It can be shown as the following formula (3-3):
[0133]
[0134] It is worth noting that the value of the above preset learning rate α can be set based on actual business needs, such as setting it to 1*10 -5 Etc., and this embodiment does not limit this.
[0135] As can be seen from the above description, this embodiment performs forward propagation calculations based on each of the data to be scored and the clone benchmark model to obtain a model prediction value, and constructs a loss function based on the difference between the model prediction value and the actual label value, and then performs back propagation based on the gradient of the loss function with respect to the parameters, the preset learning rate, and the initial parameters of the model to obtain the model update parameters corresponding to each of the data to be scored. It is possible to train the clone benchmark model based on each of the data to be scored to accurately obtain the model update parameters corresponding to each of the data to be scored, so that the model parameter change corresponding to each of the data to be scored can be determined based on the model update parameters, and the quality score of each of the data to be scored can be determined based on the model parameter change. It is possible to determine the quality score of the data by using the degree of influence of the data to be scored on the model training, avoid the subjectivity of manually formulating rules, and achieve efficient and accurate data quality assessment, thereby improving the accuracy and efficiency of subsequent data filtering, without relying on an external discriminant model, and reducing computational overhead.
[0136] Figure 4 A schematic diagram is shown of how to determine the quality score of each piece of data to be scored based on the change in model parameters corresponding to each piece of data to be scored according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to determine the quality score of each piece of data to be scored based on the change in model parameters corresponding to each piece of data to be scored.
[0137] like Figure 4 As shown, the determination of the quality score of each piece of data to be scored based on the model parameter change corresponding to each piece of data to be scored in the above step S104 may include the following steps S401-S402:
[0138] In step S401, for each piece of the to-be-scored data, a first average value of the model weight change matrix of each layer of the cloned benchmark model and a second average value of the model gradient change matrix of each layer are determined respectively.
[0139] In this embodiment, based on the above Figure 2 After the illustrated embodiment determines the model weight change matrix of each layer and the model gradient change matrix of each layer of the cloned benchmark model, the first average value of the model weight change matrix of each layer of the cloned benchmark model and the second average value of the model gradient change matrix of each layer can be determined for each piece of data to be scored.
[0140] For example, suppose the weight change matrix of each layer l of the cloned baseline model is ΔW l , then the corresponding first average value can be mean(ΔW l ).
[0141] For another example, suppose the gradient change matrix of each layer l of the cloned benchmark model is ΔG l , then the corresponding second mean value can be mean(ΔG l ).
[0142] Among them, mean() is the mean value of the matrix.
[0143] In step S402, a quality score of each piece of the to-be-scored data is determined based on the first average value and the second average value.
[0144] In this embodiment, after determining the first average value of the model weight change matrix of each layer of the cloned benchmark model and the second average value of the model gradient change matrix of each layer, the quality score of each piece of the data to be scored can be determined based on the first average value and the second average value.
[0145] That is to say, this embodiment comprehensively considers the parameter changes and gradient changes of the model to determine the quality score of each piece of data.
[0146] For example, the sum of the products of the first average value and the second average value of each layer of the cloned benchmark model can be determined as the initial score of each piece of the data to be scored. i The initial score score i :
[0147]
[0148] In the above formula, L=N, where N is the total number of model layers.
[0149] On this basis, the initial score of each piece of data to be scored can be normalized to obtain a quality score of each piece of data to be scored.
[0150] For example, the i-th data d can be calculated based on the following formula (4-2): iThe initial score score i Normalize:
[0151]
[0152] In the above formula, score norm_i is the i-th data d i Quality rating of; scores = score 0,1,…,N .
[0153] It can be seen from the above description that this embodiment determines the first average value of the model weight change matrix of each layer of the cloned benchmark model and the second average value of the model gradient change matrix of each layer for each piece of the data to be scored, and determines the quality score of each piece of the data to be scored based on the first average value and the second average value. This can accurately determine the quality score of each piece of the data to be scored based on the model parameter change amount corresponding to each piece of the data to be scored. Since the subjectivity of manually formulated rules can be avoided, efficient and accurate data quality assessment can be achieved, thereby improving the accuracy and efficiency of subsequent data filtering, and there is no need to rely on an external discriminant model, which can reduce computational overhead.
[0154] Figure 5 A schematic diagram is shown of how to filter each piece of data in the data set based on a preset ratio threshold and the quality score according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to filter each piece of data in the data set based on a preset ratio threshold and the quality score as an example to exemplify.
[0155] like Figure 5 As shown, the data quality scoring method of this embodiment may also include filtering each piece of data in the data set based on a preset ratio threshold and the quality score based on the following steps S501-S502:
[0156] In step S501, the quality scores of the data in the data set are sorted in descending order to obtain a sorting result.
[0157] In step S502, data exceeding a preset ratio threshold in the sorting result is removed to obtain filtered data.
[0158] For example, after the quality score of each piece of data to be scored is determined based on the model parameter variation corresponding to each piece of data to be scored, a filtering rule may be set based on a preset ratio threshold.
[0159] It is worth noting that the above threshold can be set based on actual business needs, such as being set to 0.25 (i.e., retaining the first 75% of the data in the sorting result and removing the last 25% of the data), etc. This embodiment does not limit this.
[0160] On this basis, when the filtered data set D is obtained filtered Afterwards, the filtering effect can be verified. For example, the filtered data set D filtered To train a new model M new , and evaluate the performance of the new model on the test set, and calculate the performance improvement of the model on the downstream test set.
[0161] From the above description, it can be seen that this embodiment uses the degree of influence of the data to be scored on the model training to determine the quality score of the data, which can avoid the subjectivity of manually formulating rules and achieve efficient and accurate data quality assessment. Therefore, based on the preset ratio threshold and the quality score, each data in the data set is filtered, which can improve the accuracy and efficiency of data filtering. Compared with the traditional statistical value-based solution, it can greatly improve the dirty data identification results, thereby ensuring the performance of the subsequent trained model.
[0162] The following is a complete example to introduce the data quality scoring and filtering method flow of the present disclosure, but it is not intended to limit the scope of the present disclosure. Figure 6 A flowchart of a method for data quality scoring and filtering according to an exemplary embodiment of the present disclosure is shown; wherein the specific explanation of each step can be found in the above Figures 1 to 5 The embodiments shown are not described in detail here. Figure 6 As shown, the method includes the following steps S601-606:
[0163] In step S601, a benchmark model is established.
[0164] A pre-trained neural network model can be selected as the benchmark model, denoted as M, and its parameter set is θ.
[0165] Exemplarily, the above-mentioned benchmark model can be selected as a model of a Transformer architecture that only includes a decoder part (eg, a Liama model or a Qwen model, etc.).
[0166] In step S602, initial parameters are recorded.
[0167] After the benchmark model M is established, the initial parameters θ can be recorded init , and then the key layer parameters that need to be monitored can be determined, including the weight matrix W and the bias vector b.
[0168] Among them, the above-mentioned key layer can be selected as a feed-forward layer and / or a self-attention layer.
[0169] In step S603, single data training and parameter variable calculation are performed.
[0170] For the i-th data d in the data set D to be filtered i(i＝1,2,…,n) , you can perform the following steps:
[0171] a) Clone the baseline model M to obtain the temporary model M temp ;
[0172] b) Using a single piece of data d i For model M temp Conduct training, including:
[0173] b1) Forward propagation calculation output: y i =M temp (d i );
[0174] b2) Calculate the loss function: L = Loss(y i ,y true );
[0175] b3) Back propagation update parameters:
[0176] Among them, α is the preset learning rate, for example, 1*10 -5 ; is the gradient of the loss function with respect to the parameters.
[0177] c) Calculate parameter variables:
[0178] For each layer l of the model, the weight change matrix ΔW can be calculated l And the gradient change matrix
[0179] ΔG l :
[0180]
[0181] In step S604, a quality score is calculated.
[0182] Considering the parameter changes and gradient changes comprehensively, the quality score of each data is calculated. i The initial score score i :
[0183]
[0184] It is recommended to set L = N, where N is the total number of model layers and mean() is the average value of the matrix.
[0185] In step S605, data is filtered.
[0186] a) Normalize the quality scores of all data:
[0187]
[0188] Among them, score norm_i is the i-th data d i Quality rating of; scores = score 0,1,…,N .
[0189] b) Setting a preset ratio threshold for data filtering.
[0190] For example, the threshold can be set to 0.25, which means that the data with the top 75% quality scores in the dataset are retained.
[0191] In step S606, a high-quality data set is output.
[0192] The data with quality scores in the top 75% in the data set can be output as high-quality data.
[0193] On this basis, when the filtered data set D is obtained filtered Afterwards, the filtering effect can be verified. For example, the filtered data set D filtered To train a new model M new , and evaluate the performance of the new model on the test set, and calculate the performance improvement of the model on the downstream test set.
[0194] From the above description, it can be seen that, on the one hand, this embodiment obtains the influence of data samples on model training by monitoring the parameter changes of the model during training, including the degree of change of the weight matrix and bias term, and the gradient change during forward propagation. This evaluation scheme based on internal changes of the model can more directly and accurately reflect the quality of the data.
[0195] On the other hand, this embodiment proposes a new data quality scoring scheme, which comprehensively considers the parameter changes and gradient information of each layer of the model, and calculates the quality score of the data sample by weighted summation. This scoring mechanism not only considers the overall impact of the data on the model, but also reflects the contribution of the data at different layers, making the data quality assessment more comprehensive and accurate.
[0196] Based on the cooperation of the above two aspects, this embodiment can improve the effect and efficiency of data filtering. Compared with the traditional calculation method based on statistical values, it can greatly improve the recognition effect of dirty data, thereby ensuring the performance of subsequent training models.
[0197] Figure 7 A structural block diagram of a data quality scoring device according to an exemplary embodiment of the present disclosure is shown; the data quality scoring device of this embodiment can be configured in an electronic device. This embodiment does not limit the type of electronic device, which can be any type of device suitable for implementation, such as a terminal device or a server.
[0198] like Figure 7 As shown, the device includes: a data determination module 110, a parameter determination module 120, a variable determination module 130 and a score determination module 140, wherein:
[0199] A data determination module 110 is used for, in response to acquiring a data set, taking each piece of data in the data set as data to be scored;
[0200] A parameter determination module 120, configured to determine a model update parameter corresponding to each piece of data to be scored based on a preset benchmark model;
[0201] A variable determination module 130, configured to determine a model parameter change corresponding to each piece of data to be scored based on model initial parameters and model update parameters corresponding to each piece of data to be scored, wherein the model initial parameters include initial parameters of the preset benchmark model;
[0202] The score determination module 140 is used to determine the quality score of each piece of the data to be scored based on the model parameter change amount corresponding to each piece of the data to be scored.
[0203] From the above description, it can be seen that the device of this embodiment, in response to obtaining a data set, takes each data in the data set as data to be scored, and determines the model update parameters corresponding to each data to be scored based on a preset benchmark model, and then determines the model parameter change corresponding to each data to be scored based on the model initial parameters and the model update parameters corresponding to each data to be scored, and then determines the quality score of each data to be scored based on the model parameter change corresponding to each data to be scored. It can be realized to determine the quality score of the data by utilizing the degree of influence of the data to be scored on the model training, avoid the subjectivity of manually formulating rules, and achieve efficient and accurate data quality assessment, thereby improving the accuracy and efficiency of subsequent data filtering, and there is no need to rely on an external discriminant model, which can reduce computational overhead.
[0204] Figure 8A structural block diagram of a data quality scoring device according to another exemplary embodiment of the present disclosure is shown; the data quality scoring device of this embodiment can be configured in an electronic device, and this embodiment does not limit the type of electronic device, which can be any type of device suitable for implementation, such as a terminal device or a server, etc.
[0205] The data determination module 210, the parameter determination module 220, the variable determination module 230 and the score determination module 240 are the same as those described above. Figure 7 The functions of the data determination module 110 , the parameter determination module 120 , the variable determination module 130 and the score determination module 140 in the illustrated embodiment are the same and will not be described in detail herein.
[0206] like Figure 8 As shown, the parameter determination module 220 may include:
[0207] A model cloning unit 221, used for cloning a preset reference model to obtain a cloned reference model;
[0208] The parameter determination unit 222 is used to train the clone benchmark model based on each piece of the data to be scored, and obtain the model update parameter corresponding to each piece of the data to be scored.
[0209] In some embodiments, the parameter determination unit 222 may also be used to:
[0210] Perform forward propagation calculation based on each piece of the to-be-scored data and the cloned benchmark model to obtain a model prediction value;
[0211] Constructing a loss function based on the difference between the model prediction value and the actual label value;
[0212] Back propagation is performed based on the gradient of the loss function with respect to the parameters, the preset learning rate, and the initial parameters of the model to obtain the model update parameters corresponding to each piece of the data to be scored.
[0213] In some embodiments, the model initial parameters and the model update parameters are parameters of a target key layer, and the target key layer is a key layer that needs to be monitored and is pre-specified in the cloned benchmark model.
[0214] In some embodiments, the parameters of the target key layer may include a weight matrix and a bias vector of the target key layer.
[0215] In some embodiments, the variable determination module 230 may include:
[0216] A first matrix determination unit 231 is used to determine, for each layer of the cloned benchmark model, a weight change matrix corresponding to each piece of the data to be scored based on an initial weight matrix and an updated weight matrix corresponding to each piece of the data to be scored, wherein the initial weight matrix is a weight matrix of the cloned benchmark model before training, and the updated weight matrix is a weight matrix of the cloned benchmark model after training;
[0217] The second matrix determination unit 232 is used to determine the gradient change matrix corresponding to each piece of data to be scored for each layer of the cloned benchmark model based on the initial gradient matrix and the updated gradient matrix corresponding to each piece of data to be scored, wherein the initial gradient matrix is the gradient matrix of the cloned benchmark model before training, and the updated gradient matrix is the gradient matrix of the cloned benchmark model after training.
[0218] In some embodiments, the score determination module 240 may include:
[0219] A mean value determination unit 241 is used to determine, for each piece of the to-be-scored data, a first mean value of a model weight change matrix of each layer of the cloned benchmark model and a second mean value of a model gradient change matrix of each layer;
[0220] The score determination unit 242 is configured to determine a quality score of each piece of the to-be-scored data based on the first average value and the second average value.
[0221] In some embodiments, the score determination unit 242 may also be used to:
[0222] Determine the sum of the products of the first average value and the second average value of each layer of the cloned benchmark model as the initial score of each piece of the data to be scored;
[0223] The initial score of each piece of data to be scored is normalized to obtain a quality score of each piece of data to be scored.
[0224] In some embodiments, the above apparatus may further include:
[0225] The data filtering module 250 is used to filter each piece of data in the data set based on a preset ratio threshold and the quality score.
[0226] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.
[0227] Fig. 9 1 is a schematic diagram of a structure of an electronic device according to an exemplary embodiment of the present disclosure. The electronic device may be, for example, a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, etc. Fig. 9 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an inspection device for the grayscale environment at the logical level. Of course, in addition to the software implementation, the present disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0228] The embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is processed by a processor, the above-mentioned data quality scoring method is implemented.
[0229] The computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present disclosure is not limited thereto.
[0230] The embodiment of the present disclosure also provides a computer program product, on which a computer program is stored, and when the program is processed by a processor, the above-mentioned data quality scoring method is implemented.
[0231] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims of the present disclosure.
[0232] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A data quality scoring method, characterized in that: The method comprises: In response to obtaining the data set, each piece of data in the data set is used as data to be scored; Determine a model update parameter corresponding to each piece of data to be scored based on a preset benchmark model; Determine the model parameter change corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored, wherein the model initial parameters include the initial parameters of the preset benchmark model; The quality score of each piece of data to be scored is determined based on the model parameter change corresponding to each piece of data to be scored.
2. The method according to claim 1, characterized in that The step of determining the model update parameter corresponding to each piece of the to-be-scored data based on the preset benchmark model includes: Cloning a preset benchmark model to obtain a cloned benchmark model; The cloned benchmark model is trained based on each piece of the data to be scored, respectively, to obtain a model update parameter corresponding to each piece of the data to be scored.
3. The method according to claim 2, characterized in that The step of training the cloned benchmark model based on each piece of the data to be scored to obtain a model update parameter corresponding to each piece of the data to be scored includes: Perform forward propagation calculation based on each piece of the to-be-scored data and the cloned benchmark model to obtain a model prediction value; Constructing a loss function based on the difference between the model prediction value and the actual label value; Back propagation is performed based on the gradient of the loss function with respect to the parameters, the preset learning rate, and the initial parameters of the model to obtain the model update parameters corresponding to each piece of the data to be scored.
4. The method according to claim 2, characterized in that: The model initial parameters and the model update parameters are both parameters of a target key layer, and the target key layer is a key layer that needs to be monitored and is pre-specified in the cloned benchmark model.
5. The method according to claim 4, characterized in that The parameters of the target key layer include a weight matrix and a bias vector of the target key layer.
6. The method according to claim 2, characterized in that The determining, based on the model initial parameters and the model update parameters corresponding to each piece of the data to be scored, the model parameter change corresponding to each piece of the data to be scored comprises: For each layer of the cloned benchmark model, determine a weight change matrix corresponding to each piece of the data to be scored based on an initial weight matrix and an updated weight matrix corresponding to each piece of the data to be scored, wherein the initial weight matrix is a weight matrix of the cloned benchmark model before training, and the updated weight matrix is a weight matrix of the cloned benchmark model after training; For each layer of the cloned benchmark model, the gradient change matrix corresponding to each piece of the data to be scored is determined based on the initial gradient matrix and the updated gradient matrix corresponding to each piece of the data to be scored, wherein the initial gradient matrix is the gradient matrix of the cloned benchmark model before training, and the updated gradient matrix is the gradient matrix of the cloned benchmark model after training.
7. The method according to claim 6, characterized in that The determining the quality score of each piece of data to be scored based on the model parameter change corresponding to each piece of data to be scored comprises: For each piece of the to-be-scored data, respectively determine a first average value of a model weight change matrix of each layer of the cloned benchmark model and a second average value of a model gradient change matrix of each layer; A quality score of each piece of the to-be-scored data is determined based on the first average value and the second average value.
8. The method according to claim 7, characterized in that The determining the quality score of each piece of the to-be-scored data based on the first average value and the second average value comprises: Determine the sum of the products of the first average value and the second average value of each layer of the cloned benchmark model as the initial score of each piece of the data to be scored; The initial score of each piece of data to be scored is normalized to obtain a quality score of each piece of data to be scored.
9. The method according to claim 1, characterized in that: The method further comprises: Each piece of data in the data set is filtered based on a preset ratio threshold and the quality score.
10. A data quality scoring device, characterized in that: The device comprises: A data determination module, configured to, in response to acquiring a data set, use each piece of data in the data set as data to be scored; A parameter determination module, used to determine the model update parameter corresponding to each piece of the to-be-scored data based on a preset benchmark model; A variable determination module, used to determine the model parameter change corresponding to each piece of data to be scored based on the model initial parameters and the model update parameters corresponding to each piece of data to be scored, wherein the model initial parameters include the initial parameters of the preset benchmark model; The scoring determination module is used to determine the quality score of each piece of data to be scored based on the model parameter change corresponding to each piece of data to be scored.
11. An electronic device, characterized in that: The electronic device comprises: processor; memory for storing computer programs; Wherein, the processor is configured to implement the data quality scoring method described in any one of claims 1 to 9 when executing the computer program.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is processed by a processor, the data quality scoring method described in any one of claims 1 to 9 is implemented.
13. A computer program product having a computer program stored thereon, characterized in that: When the computer program is processed by a processor, the data quality scoring method according to any one of claims 1 to 9 is implemented.