Data quality evaluation method and device, computing equipment and storage medium

Abstract

The invention provides a data quality evaluation method and device, computing equipment and a storage medium, and the method comprises the steps: carrying out the multi-round training of a target model based on each piece of target data in to-be-evaluated target data, obtaining a plurality of target parameter sets for evaluating the data quality of each piece of target data on the basis of the plurality of generated training parameter sets; evaluating the effect of training the target model based on each piece of target data from multiple dimensions such as the parameter change oscillation degree in the training process, the accumulated parameter change trend of the multi-round training process and the consistency degree of the parameter change directions of the multi-round training process to obtain a plurality of first evaluation parameters, thereby obtaining a plurality of second evaluation parameters based on the plurality of first evaluation parameters; and evaluating the data quality of each piece of target data. By integrating the multi-round model training effect of the single data, the data quality evaluation is realized from multiple dimensions, and the accuracy of the data quality evaluation result is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly to a method, device, computing device and storage medium for data quality assessment. Background Art

[0002] In the field of artificial intelligence, especially in the training process of deep learning models such as natural language processing models, the quality of training data has an important impact on the training effect of the model. Accurately evaluating the quality of training data to screen out high-quality training data from a large amount of training data is a key step in improving the model performance. High-quality training data can ensure that the model learns true and effective processing ideas, reduce the risk of overfitting or underfitting, and improve the generalization ability of the model. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present application provides a method, device, computing device and storage medium for data quality assessment.

[0004] According to the first aspect of the embodiments of the present application, a method for data quality assessment is provided. The method includes:

[0005] Based on multiple training parameter sets generated by respectively training a target model with each target data in the target data to be evaluated for multiple rounds, obtaining multiple target parameter sets for evaluating the data quality of each of the target data;

[0006] Based on the multiple target parameter sets, evaluating the training effect of training the target model based on each of the target data from multiple dimensions to obtain multiple first evaluation parameters. The multiple dimensions at least include a first dimension, a second dimension and a third dimension. The first dimension is used to indicate the degree of parameter change oscillation during the training process, the second dimension is used to indicate the cumulative parameter change trend during multiple rounds of training, and the third dimension is used to indicate the degree of consistency of the parameter change direction during multiple rounds of training;

[0007] Based on the multiple first evaluation parameters, evaluating the data quality of each of the target data.

[0008] According to the second aspect of the embodiments of the present application, a data quality assessment device is provided. The device includes:

[0009] An obtaining module, configured to obtain multiple target parameter sets for evaluating the data quality of each of the target data based on multiple training parameter sets generated by respectively training a target model with each target data in the target data to be evaluated for multiple rounds;

[0010] An evaluation module, configured to evaluate the effect of training the target model based on each of the target data from multiple dimensions based on the multiple sets of target parameters, to obtain multiple first evaluation parameters. The multiple dimensions at least include a first dimension, a second dimension, and a third dimension. The first dimension is used to indicate the degree of parameter change oscillation during the training process. The second dimension is used to indicate the cumulative parameter change trend during multiple rounds of training. The third dimension is used to indicate the degree of consistency of the parameter change direction during multiple rounds of training.

[0011] The evaluation module is further configured to evaluate the data quality of each of the target data based on the multiple first evaluation parameters.

[0012] According to a third aspect of the embodiments of the present application, there is provided a computing device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the operations performed by the above data quality evaluation method are implemented.

[0013] According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a program is stored. When the program is executed by the processor, the operations performed by the above data quality evaluation method are implemented.

[0014] According to a fifth aspect of the embodiments of the present application, there is provided a computer program product, including a computer program. When the computer program is executed by the processor, the operations performed by the above data quality evaluation method are implemented.

[0015] The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

[0016] By separately training the target model in multiple rounds based on each of the target data in the target data to be evaluated, multiple sets of target parameters for evaluating the data quality of each target data are obtained based on the multiple sets of training parameters generated during the training process of each target data. Based on the multiple sets of target parameters, the effect of training the target model based on each target data is evaluated from multiple dimensions such as the degree of parameter change oscillation during the training process, the cumulative parameter change trend during multiple rounds of training, and the degree of consistency of the parameter change direction during multiple rounds of training, to obtain multiple first evaluation parameters. Thus, based on the multiple first evaluation parameters, the data quality of each target data is evaluated. According to the solution provided by the present application, the multiple-round model training effects of individual data can be comprehensively considered, and the data quality can be evaluated from multiple dimensions such as the degree of parameter change oscillation during the training process, the cumulative parameter change trend during multiple rounds of training, and the degree of consistency of the parameter change direction during multiple rounds of training, improving the accuracy of the data quality evaluation result.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and form a part of this application, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0019] Figure 1 is a schematic structural diagram of a target model shown according to an exemplary embodiment of this application.

[0020] Figure 2 is a flowchart of a data quality assessment method shown according to an exemplary embodiment of this application.

[0021] Figure 3 is a schematic flowchart of a data quality assessment process shown according to an exemplary embodiment of this application.

[0022] Figure 4 is a schematic flowchart of a data quality assessment process shown according to an exemplary embodiment of this application.

[0023] Figure 5 is a block diagram of a data quality assessment device shown according to an exemplary embodiment of this application.

[0024] Figure 6 is a schematic structural diagram of a computing device shown according to an exemplary embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed herein.

[0026] The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit this application. The singular forms "a", "the", and "said" used in this application are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0028] This application provides a data quality assessment method for comprehensively assessing the data quality of target data used to train a target model, so as to help screen out high-quality data in the target data for subsequent model training processes.

[0029] In some embodiments, the target data may be text data, image data, audio data, time series data, etc., but is not limited thereto.

[0030] In some embodiments, the target model may be used to perform at least one of natural language processing tasks, computer vision tasks, audio processing tasks, and time series data analysis tasks. That is, the target model may be a natural language processing model (such as a large language model (LLM)), a computer vision model, an audio processing model, a time series data analysis model, etc., but is not limited thereto.

[0031] In some embodiments, the target model may be a model based on the transformer architecture, but is not limited thereto. See Figure 1 , Figure 1 is a schematic structural diagram of a target model shown according to an exemplary embodiment of this application, Figure 1 Taking the target model as a natural language processing model used to perform natural language processing tasks as an example, the target model may be a pre-trained model based on the transformer architecture, such as Figure 1As shown, the target model mainly consists of two parts: an encoder and a decoder. Among them, the role of the encoder is to process the input sequence and convert it into a context-related representation. Each encoder can include two main parts: a multi-head self-attention mechanism and a feed-forward neural network. The role of the decoder is to generate the output sequence. Each decoder can include a masked multi-head self-attention mechanism, a multi-head self-attention mechanism, and a feed-forward neural network. In addition, the target model can also include an embedding layer, a position encoder, and a fully connected layer.

[0032] In some embodiments, the input sequence can be converted into word vectors through the embedding layer, and then, based on the word vectors, the position encoder can be used to superimpose and obtain a representation with position information. Moreover, the input sequence can be processed by the encoder to output a context-related representation, and the decoder can generate the target sequence based on the output of the encoder, and finally, the final prediction can be output through the fully connected layer.

[0033] It should be noted that the above is only an exemplary description of the application scenario of the present application, and does not constitute a limitation on the application scenario of the present application. In more possible implementation manners, the present application can be applied to a variety of other scenarios involving data quality assessment.

[0034] The above data quality assessment method can be executed by a computing device. The computing device can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, an advertising machine, an all-in-one computer, etc. The present application does not limit the specific type of the computing device.

[0035] After introducing the application scenario of the present application, next, in combination with the embodiments of the present application, the data quality assessment method provided by the present application will be described in detail.

[0036] As Figure 2 shown, Figure 2 is a flowchart of a data quality assessment method shown according to an exemplary embodiment of the present application. The method includes the following steps:

[0037] Step 201: Based on multiple training parameter sets generated by performing multiple rounds of training on a target model using each target data in the target data to be evaluated, obtain multiple target parameter sets for evaluating the data quality of each target data.

[0038] As described above, the target data can be text data, image data, audio data, time series data, etc., but is not limited thereto. Correspondingly, the target model can be used to perform natural language processing tasks, computer vision tasks, audio processing tasks, time series data analysis tasks, etc., but is not limited thereto.

[0039] In some embodiments, there can be multiple target data to be evaluated. For each target data among the multiple target data, the target data can be used to perform multiple rounds of training on the target model. Each round of training updates the model parameters of the target model once, obtaining a set of training parameters. Then, multiple rounds of training generate multiple sets of training parameters. Thus, multiple sets of target parameters can be obtained from the multiple sets of training parameters generated by training the target model using the target data. These multiple sets of target parameters can be used to evaluate the data quality of the target data.

[0040] Step 202: Based on multiple sets of target parameters, evaluate the effects of training the target model based on each target data from multiple dimensions, obtaining multiple first evaluation parameters. The multiple dimensions at least include a first dimension, a second dimension, and a third dimension. The first dimension is used to indicate the degree of parameter change oscillation during the training process, the second dimension is used to indicate the cumulative parameter change trend during multiple rounds of training, and the third dimension is used to indicate the degree of consistency of the parameter change direction during multiple rounds of training.

[0041] In some embodiments, for each target data, the training effects of multiple rounds of training on the target model can be evaluated from multiple dimensions. The evaluation of each dimension can generate a first evaluation parameter, so that multiple first evaluation parameters can be obtained.

[0042] In some embodiments, the multiple dimensions can include a first dimension for indicating the degree of parameter change oscillation during the training process, a second dimension for indicating the cumulative parameter change trend during multiple rounds of training, and a third dimension for indicating the degree of consistency of the parameter change direction during multiple rounds of training.

[0043] It should be noted that the first dimension is used to indicate the degree of parameter change oscillation during the training process to measure whether the parameter changes in each training process are balanced, so as to determine the stability of the degree of parameter change in each training process. The second dimension is used to indicate the cumulative parameter change trend during multiple rounds of training to measure the parameter change direction between the finally trained model and the initial model, so as to determine the overall trend of model parameter changes after the entire training process. The third dimension is used to indicate the degree of consistency of the parameter change direction during multiple rounds of training to measure whether the parameter change directions in each training process are consistent, so as to determine the consistency of the parameter change trends in each training process.

[0044] Step 203: Evaluate the data quality of each target data based on multiple first evaluation parameters.

[0045] In some embodiments, for each target data, the data quality score of the target data can be determined based on multiple first evaluation parameters corresponding to the target data, so as to evaluate the data quality of the target data.

[0046] According to the solution provided by the present application, the multi-round model training effects of a single data can be comprehensively considered, and the data quality can be evaluated from multiple dimensions such as the degree of parameter change oscillation during the training process, the cumulative parameter change trend of the multi-round training process, and the consistency degree of the parameter change directions in the multi-round training process, so as to improve the accuracy of the data quality evaluation result.

[0047] After introducing the basic implementation process of the present application, the various non-limiting implementation manners of the present application will be specifically introduced below.

[0048] In some embodiments, the target model can be trained multiple rounds based on each target data respectively. That is, for the first data which is one of the target data, the target model can be trained multiple rounds based on the first data, and the model parameters of the target model can be updated once in each round of training, so as to obtain multiple training parameter sets, and through step 201, multiple target parameter sets for evaluating the data quality of the first data can be obtained based on these multiple training parameter sets.

[0049] In some embodiments, a pre-trained model based on the transformer architecture can be used as the initial target model (denoted as M0), and its parameter set is the initial training parameter set (denoted as θ0).

[0050] In some embodiments, all the target data to be evaluated can form a training data set, and each target data can have its own corresponding label data for indicating the true processing result of the data. Taking the target model as a neural network model for identifying items in an image as an example, the label data is the true category of the items included in the image which is the target data. Optionally, the label data can be used to compare with the result output by the model based on the target data, so as to update the model parameters.

[0051] For any piece of data in the training dataset (i.e., the first data), the first data can be used as the input to model M0 to process the first data through model M0, obtaining a first output result. Based on the first output result and the label data of the first data, a loss function is calculated, and then the model parameters are updated through backpropagation according to the calculated loss function, obtaining a target model (denoted as M1) after the first parameter update, and its parameter set can be denoted as θ1. Then, the first data can be used as the input to model M1 to process the first data through model M1, obtaining a second output result. Based on the second output result and the label data of the first data, a loss function is calculated, and then the model parameters are updated through backpropagation according to the calculated loss function, obtaining a target model (denoted as M2) after the second parameter update, and its parameter set can be denoted as θ2. Then, the first data is used as the input to model M2 to process the first data through model M2, obtaining a third output result. Based on the third output result and the label data of the first data, a loss function is calculated, and then the model parameters are updated through backpropagation according to the calculated loss function, obtaining a target model (denoted as M3) after the third parameter update, and its parameter set can be denoted as θ3, and so on, to perform multiple rounds of training on the target model based on the first data.

[0052] In some embodiments, when calculating the loss function, the cross-entropy loss function can be used, but it is not limited to this, and other types of loss functions can also be used.

[0053] Taking the use of the cross-entropy loss function to calculate the loss function as an example, the following formula (1) can be used to calculate the loss function:

[0054] L(y i ,y true )=GrossEntropyLoss(y i ,y true ) (1)

[0055] Wherein, L() represents the loss function, CrossEntropyLoss() represents the cross-entropy loss function, y i represents the output result of the model, and y true represents the label data.

[0056] In some embodiments, when updating the model parameters through backpropagation, it can be implemented through the following formula:

[0057]

[0058] Wherein, θ n+1 represents the updated model parameter set (i.e., the training parameter set of model M n+1 ), θn represents the set of model parameters before update (i.e., the training parameter set of model M n ), and α represents the learning rate.

[0059] Optionally, the value of the learning rate can be 1e - 5, but is not limited thereto, and the learning rate can also be other values.

[0060] In some embodiments, the target model can be trained for a set number of rounds based on the first data, and the set of training parameters generated after the set number of rounds of training can be used as multiple target parameter sets for evaluating the data quality of the first data.

[0061] Optionally, the set number of rounds can be any value. For example, the set number of rounds can be 3, and the target model can be trained for 3 rounds based on the first data, so that 3 training parameter models can be obtained, and these 3 training parameter models can be used as 3 target parameter sets for evaluating the data quality of the first data.

[0062] In some embodiments, multiple sets of training parameters generated by training the target model for multiple rounds using the first data can also be sampled according to a set step size to obtain multiple target parameter sets for evaluating the data quality of the first data.

[0063] Optionally, the set step size can be any value. For example, the set step size can be 10, but is not limited thereto, and the set step size can also be other values.

[0064] For example, the target model can be trained for 100 rounds using the first data to obtain 100 sets of training parameters. Then, with a set step size of 10, parameter set collection can be performed every 10 rounds of training, and 10 sets of training parameters can be extracted from the 100 sets of training parameters as the target parameter sets for evaluating the data quality of the first data.

[0065] The above embodiments provide two alternative implementation methods for obtaining the target parameter sets. For the first data, which is one of the target data, no matter which method is used to obtain the target parameter set corresponding to the first data, after obtaining the target parameter set, step 202 can be used to evaluate the effect of training the target model based on the first data from multiple dimensions based on multiple target parameter sets, and multiple first evaluation parameters can be obtained.

[0066] In some embodiments, step 202 can include the following multiple implementation methods to evaluate the model training effect for the first data from multiple dimensions:

[0067] Step 2021: Based on multiple sets of target parameters corresponding to the first data, evaluate the effect of training the target model based on the first data from the first dimension to obtain a first evaluation parameter.

[0068] In some embodiments, a first parameter change amount between sets of target parameters obtained in two adjacent training processes of the target model based on the first data can be determined, so as to evaluate the effect of training the target model based on the first data from the first dimension and obtain a first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

[0069] That is, two adjacent training processes can be taken as a group to determine the first parameter change amount between the sets of target parameters obtained in these two rounds of training, and so on. Each two adjacent training processes can obtain a corresponding first parameter change amount, thus obtaining multiple first parameter change amounts, and based on these multiple first parameter change amounts, evaluate the model training effect of the first data from the first dimension.

[0070] In a possible implementation, a first vector difference between vectors corresponding to sets of target parameters obtained in two adjacent training processes of the target model based on the first data can be determined, so as to determine the modulus value of each first vector difference as the first parameter change amount; determine the standard deviation of all the first parameter change amounts, and use the reciprocal of the determined standard deviation as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

[0071] That is, after determining the first parameter change amount through the following formula (3), the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension can be determined through the following formula (4):

[0072] ΔW n =|θ n -θ n-1 | (3)

[0073] stability=1 / std([ΔW1,ΔW2,…,ΔW n ) (4)

[0074] where θ n represents the set of parameters of model M n , θ n-1 represents the set of parameters of model M n-1 , ΔW represents the first parameter change amount between two adjacent training processes. Taking ΔW n as an example, ΔW n represents the set of parameters of model M n relative to the set of parameters of model M n-1The first parameter change amount of the parameter set, and stability represents the first evaluation parameter corresponding to the model training effect in the first dimension.

[0075] Step 2022: Based on multiple target parameter sets corresponding to the first data, evaluate the effect of training the target model based on the first data from the second dimension to obtain a first evaluation parameter.

[0076] In some embodiments, it is possible to determine the second parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, and the third parameter change amount between the target parameter set obtained through one round of training of the target model based on the first data and the target parameter set of the initial target model, so as to evaluate the effect of training the target model based on the first data from the second dimension based on the second parameter change amount and the third parameter change amount, and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension.

[0077] That is, for the case of training the target model M through n rounds based on the first data n it is possible to determine the second parameter change amount between the finally trained model M n and the initial model M0, and the third parameter change amount between the model M1 obtained through one round of training and the initial model M0, so as to evaluate the model training effect of the first data from the second dimension based on the second parameter change amount and the third parameter change amount.

[0078] In a possible implementation, it is possible to respectively determine the third vector difference between the vector corresponding to the target parameter set of the finally trained target model and the vector corresponding to the target parameter set of the initial target model, and the fourth vector difference between the vector corresponding to the target parameter set of the target model obtained through one round of training and the vector corresponding to the target parameter set of the initial target model, so as to determine the modulus value of the third vector difference and the modulus value of the fourth vector difference as the second parameter change amount and the third parameter change amount; determine the difference between the second parameter change amount and the third parameter change amount, and use the ratio between the determined difference and the third parameter change amount as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension.

[0079] That is, after determining the second parameter change amount through the following formula (5) and determining the third parameter change amount through the following formula (6), the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension can be determined through the following formula (7):

[0080] ∑W n =|θn -θ0| (5)

[0081] ∑W1 = |θ1 - θ0| (6)

[0082]

[0083] where θ n represents the parameter set of model M n , θ0 represents the parameter set of the initial model M0, ∑W n represents the second parameter change amount between the target parameter set of the finally trained target model and the target parameter set of the initial target model, ∑W1 represents the third parameter change amount between the target parameter set of the target model obtained after one round of training and the target parameter set of the initial target model, and trend represents the first evaluation parameter corresponding to the model training effect in the second dimension.

[0084] Step 2023: Based on multiple target parameter sets corresponding to the first data, evaluate the effect of training the target model from the third dimension to obtain a first evaluation parameter.

[0085] In some embodiments, the similarity of the first parameter change direction between the target parameter sets obtained from three adjacent rounds of training of the target model based on the first data can be determined, so as to evaluate the training effect of training the target model based on the first data from the third dimension and obtain the first evaluation parameter corresponding to the training effect of training the target model based on the first data in the third dimension.

[0086] That is, three adjacent rounds of training processes can be taken as a group to determine the similarity of the first parameter change direction between the target parameter sets obtained from these three rounds of training. By analogy, each group of three adjacent training processes can obtain a corresponding first parameter change similarity, so as to obtain multiple first parameter change similarities, and based on these multiple first parameter change similarities, evaluate the model training effect of the first data from the third dimension.

[0087] In a possible implementation, the first vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent rounds of training of the target model based on the first data can be determined, and the cosine similarity between every two adjacent first vector differences can be determined as the similarity of the first parameter change direction; determine the average value of all the similarities of the first parameter change direction, and use the determined average value as the first evaluation parameter corresponding to the training effect of training the target model based on the first data in the third dimension.

[0088] That is, after determining the similarity of the change direction of the first parameter through the following formula (8), the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension can be determined through the following formula (9):

[0089] cos n =cos<θ n -θ n-1 ,θ n+1 -θ n > (8)

[0090] direction=mean([cos1,cos2,…,cos n ) (9)

[0091] where, θ n represents the parameter set of model M n , θ n-1 represents the parameter set of model M n-1 , θ n+1 represents the parameter set of model M n+1 , cos n represents the similarity of the change direction of the first parameter between the target parameter sets obtained in the adjacent three rounds of training processes (i.e., the (n - 1)-th round of training, the n-th round of training, and the (n + 1)-th round of training) corresponding to the n-th round of training process, and direction represents the first evaluation parameter corresponding to the model training effect in the third dimension.

[0092] It should be noted that the labels of steps 2021, 2022, and 2023 do not constitute a limitation on the execution order of the above steps. In more possible implementation manners, the execution order of the above steps can be adjusted as needed, or the above steps can be executed simultaneously, and this application does not make any limitation in this regard.

[0093] Through the above embodiments, the evaluation of the model training effect for the first data can be achieved from multiple dimensions. Optionally, when training the target model for a set number of rounds based on the first data to obtain a set number of training parameter sets as multiple target parameter sets, the evaluation of the model training effect for the first data can be achieved through the above embodiments, but not limited thereto. It is also possible to achieve the evaluation of the model training effect for the first data after obtaining multiple target parameter sets by sampling from the multiple training parameter sets obtained by training the target model for multiple rounds based on the first data according to a set step size. It should be noted that when obtaining multiple target parameter sets by sampling from the multiple training parameter sets obtained by training the target model for multiple rounds based on the first data according to a set step size and then achieving the evaluation of the model training effect for the first data through the above embodiments, θ1, θ2, …, θ nThat is the set of target parameters sampled. There is an interval of k-1 training processes between every two adjacent sets of target parameters, where k is the set step size.

[0094] In some embodiments, after achieving the multi-dimensional evaluation of the model training effect for the first data through the above embodiments, the data quality of the first data can be evaluated based on multiple first evaluation parameters corresponding to the first data through step 203.

[0095] In some embodiments, for step 203, when evaluating the data quality of the first data based on multiple first evaluation parameters corresponding to the first data, it can be achieved in the following manner:

[0096] Based on the preset first weight parameter, second weight parameter, and third weight parameter, perform weighted summation on the first evaluation parameters corresponding to the effect of training the target model on the first data in the first dimension, second dimension, and third dimension respectively, to obtain a second evaluation parameter for evaluating the data quality of the first data.

[0097] Optionally, the determination of the second evaluation parameter for evaluating the data quality of the first data can be achieved through the following formula (10):

[0098] score = w1 * stability + w2 * trend + w3 * direction (10)

[0099] Among them, score represents the second evaluation parameter, stability represents the first evaluation parameter corresponding to the model training effect in the first dimension, trend represents the first evaluation parameter corresponding to the model training effect in the second dimension, direction represents the first evaluation parameter corresponding to the model training effect in the third dimension, w1 represents the first weight parameter, w2 represents the second weight parameter, and w3 represents the third weight parameter.

[0100] Optionally, the values of the first weight parameter, second weight parameter, and third weight parameter can be any values, as long as the sum of the three is 1. For example, the first weight parameter can be 0.3, the second weight parameter can be 0.4, and the third weight parameter can be 0.3, but it is not limited thereto.

[0101] It should be noted that the above embodiments are only an exemplary implementation manner. In more possible implementation manners, the multi-dimensional evaluation of the model training effect for the first data can also be achieved through the following embodiments.

[0102] In some embodiments, for step 2021, when evaluating the effect of training the target model based on the first data from the first dimension and obtaining a first evaluation parameter based on multiple target parameter sets corresponding to the first data, it can also be implemented in the following manner:

[0103] Determine the first parameter change direction similarity between the target parameter sets obtained from the adjacent three rounds of training processes of the target model based on the first data, and the second parameter change direction similarity between the target parameter sets obtained from the adjacent three rounds of training processes of the target model based on each target data, so as to evaluate the effect of training the target model based on the first data from the first dimension and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

[0104] That is, for the training process based on the first data, taking the adjacent three rounds of training processes as a group, determine the first parameter change direction similarity between the target parameter sets obtained from these three rounds of training, and so on. Each set of three adjacent training processes can obtain a corresponding first parameter change similarity, thus obtaining multiple first parameter change similarities; similarly, for the training process based on other target data, taking the adjacent three rounds of training processes as a group, determine the second parameter change direction similarity between the target parameter sets obtained from these three rounds of training, thus obtaining multiple second parameter change similarities. Then, based on these multiple first parameter change similarities and multiple second parameter change similarities, the model training effect of the first data can be evaluated from the first dimension.

[0105] In a possible implementation, the first vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent rounds of training of the target model based on the first data, and the second vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent rounds of training of the target model based on each target data can be determined respectively; the cosine similarity between every two adjacent first vector differences and the cosine similarity between every two adjacent second vector differences corresponding to each target data are determined respectively as the first parameter change direction similarity and the second parameter change direction similarity corresponding to each target data; based on the difference between the first value and each first parameter change direction similarity, and the difference between the first value and the second parameter change direction similarity corresponding to each target data, they are used as the first difference and the second difference corresponding to each target data; the sum value of all the first differences and the sum value of all the second differences corresponding to each target data are determined as the first parameter value and the second parameter value corresponding to each target data; the ratio of the first parameter value to the maximum second parameter value among the second parameter values corresponding to all the target data is determined, and the difference between the first value and the determined ratio is used as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension. Among them, the first value can be 1.

[0106] That is to say, after determining the first parameter change direction similarity through the following formula (11) and determining the second parameter change direction similarity through the following formula (12), the first parameter value and the second parameter value can be determined respectively through the following formula (13) and formula (14), so as to determine the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension through formula (15):

[0107] cos j =cos<θ j -θ j-1 ,θ j+1 -θ j > (11)

[0108] cos ′ h,j =cos<θ ′ h,j -θ ′ h,j-1 ,θ ′ h,j+1 -θ ′ h,j > (12)

[0109]

[0110]

[0111] Among them, θ j represents the set of parameters of model M trained based on the first data, θ j represents the set of parameters of model M trained based on the first data, θ j-1 represents the set of parameters of model M trained based on the first data, θ j-1 represents the set of parameters of model M trained based on the first data, θ j+1 represents the set of parameters of model M trained based on the first data, θ j+1 represents the set of parameters of model M trained based on the first data, cos j represents the first parameter change direction similarity between the target parameter sets obtained from the adjacent three rounds of training processes (i.e., the (j - 1)-th round of training, the j-th round of training, and the (j + 1)-th round of training) corresponding to the j-th round of training during the model training based on the first data, θ ′ h,j represents the set of parameters of model M trained based on the h-th target data, θ h,j represents the set of parameters of model M trained based on the h-th target data, θ ′ h,j-1 represents the set of parameters of model M trained based on the h-th target data, θ h,j-1 represents the set of parameters of model M trained based on the h-th target data, θ ′ h,j+1 represents the set of parameters of model M trained based on the h-th target data, θ h,j+1 represents the set of parameters of model M trained based on the h-th target data, cos ′ h,j represents the second parameter change direction similarity between the target parameter sets obtained from the adjacent three rounds of training processes (i.e., the (j - 1)-th round of training, the j-th round of training, and the (j + 1)-th round of training) corresponding to the j-th round of training during the model training based on the h-th target data, Oscillation represents the first parameter value, Oscillation ′ h represents the second parameter value corresponding to the h-th target data, Oscillation score represents the first evaluation parameter corresponding to the model training effect in the third dimension, and n represents the total number of training rounds.

[0112] In some embodiments, for step 2022, when evaluating the effect of training the target model based on the first data from the second dimension and obtaining a first evaluation parameter, it can also be achieved in the following manner:

[0113] Determine the fourth parameter change amount between the target parameter sets obtained from the adjacent two rounds of training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter sets obtained from the adjacent two rounds of training processes of the target model based on each target data, so as to evaluate the effect of training the target model based on the first data from the second dimension based on the fourth parameter change amount and the fifth parameter change amount corresponding to each target data, and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension.

[0114] That is, for the training process based on the first data, two adjacent rounds of the training process can be taken as a group to determine the change amount of the fourth parameter between the sets of target parameters obtained from these two rounds of training, and so on. For each two adjacent rounds of the training process, a corresponding change amount of the fourth parameter can be obtained, thus obtaining multiple change amounts of the fourth parameter; similarly, for the training process based on other target data, two adjacent rounds of the training process can be taken as a group to determine the change amount of the fifth parameter between the sets of target parameters obtained from these two rounds of training, and so on. For each two adjacent rounds of the training process, a corresponding change amount of the fifth parameter can be obtained, thus obtaining multiple change amounts of the fifth parameter. Then, based on these multiple change amounts of the fourth parameter and multiple change amounts of the fifth parameter, the training effect of the model for the first data can be evaluated from the second dimension.

[0115] In a possible implementation, the first vector difference between the vectors corresponding to the sets of target parameters obtained from two adjacent rounds of training of the target model based on the first data, and the second vector difference between the vectors corresponding to the sets of target parameters obtained from two adjacent rounds of training of the target model based on each target data can be determined respectively, so as to determine the second-order norm of each first vector difference and the second-order norm of each second vector difference corresponding to each target data as the change amount of the fourth parameter and the change amount of the fifth parameter corresponding to each target data; determine the sum value of all the change amounts of the fourth parameter and the sum value of all the change amounts of the fifth parameter corresponding to each target data as the third parameter value and the fourth parameter value; determine the ratio of the third parameter value to the maximum fourth parameter value among all the fourth parameter values corresponding to the target data as the first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension.

[0116] That is, the change amount of the fourth parameter can be determined by the following formula (16) and the third parameter value can be determined according to the change amount of the fourth parameter, and the change amount of the fifth parameter can be determined by the following formula (17) and the fourth parameter can be determined according to the change amount of the fifth parameter, so as to determine the first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension through formula (18):

[0117]

[0118] where θ j represents the set of parameters of the model M trained based on the first data j and θ j-1 represents the set of parameters of the model M trained based on the first data j-1 and θ ′ h,j represents the set of parameters of the model M trained based on the h-th target data h,jThe parameter set, θ ′ h,j-1 represents the model M trained based on the h-th target data h,j-1 The parameter set, Path represents the third parameter value corresponding to the first data, Path ′ h represents the fourth parameter value corresponding to the h-th target data, Path length represents the first evaluation parameter corresponding to the model training effect in the second dimension, and n represents the total number of training rounds.

[0119] In some embodiments, for step 2023, when evaluating the effect of training the target model based on the first data from the third dimension and obtaining a first evaluation parameter for a plurality of target parameter sets corresponding to the first data, it can also be implemented in the following manner:

[0120] Determine the fourth parameter change amount between the target parameter sets obtained from two adjacent training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, so as to evaluate the effect of training the target model based on the first data from the third dimension and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension.

[0121] That is, for the training process based on the first data, two adjacent training processes can be taken as a group to determine the fourth parameter change amount between the target parameter sets obtained from these two rounds of training. By analogy, each two adjacent training processes can obtain a corresponding fourth parameter change amount, thus obtaining multiple fourth parameter change amounts. And, the finally trained model M n and the fifth parameter change amount between the initial model M0 can be determined, so as to evaluate the model training effect of the first data from the third dimension based on the fourth parameter change amount and the fifth parameter change amount.

[0122] In a possible implementation manner, the first vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent training processes of the target model based on the first data can be determined to determine the second-order norm of each first vector difference as the fourth parameter change amount; determine the third vector difference between the vector corresponding to the target parameter set of the finally trained first model and the vector corresponding to the target parameter set of the initial target model, and determine the second-order norm of the third vector difference as the fifth parameter change amount; determine the sum value of all the fourth parameter change amounts, and use the ratio of the fifth parameter change amount to the determined sum value as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension.

[0123] That is, when the sum value of all the fourth parameter changes (i.e., the third parameter value) is determined by the formula (16) as described above, the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension can be determined by the following formula (19):

[0124]

[0125] where θ n represents the parameter set of the finally trained model M0 based on the first data, θ0 represents the parameter set of the initial model M0, Path represents the sum value of all the fourth parameter changes (i.e., the third parameter value), ‖θ n - θ0‖2 is the fifth parameter change, and Consistency score represents the first evaluation parameter corresponding to the model training effect in the third dimension, and n represents the total number of training rounds.

[0126] In some embodiments, after the multi-dimensional evaluation of the model training effect on the first data is achieved through the above embodiments, the data quality of the first data can be evaluated based on the multiple first evaluation parameters corresponding to the first data through step 203.

[0127] In some embodiments, for step 203, when evaluating the data quality of the first data based on the multiple first evaluation parameters corresponding to the first data, it can be achieved in the following manner:

[0128] Based on the preset first weight parameter, second weight parameter, and third weight parameter, weighted summation is performed on the first evaluation parameters corresponding to the effect of training the target model based on the first data in the first dimension, second dimension, and third dimension respectively, to obtain a second evaluation parameter for evaluating the data quality of the first data.

[0129] Optionally, the determination of the second evaluation parameter for evaluating the data quality of the first data can be achieved through the following formula (20):

[0130] score = w1 * Oscillation score + w2 * Path length + w3 * Consistency score (20)

[0131] where score represents the second evaluation parameter, Oscillation score represents the first evaluation parameter corresponding to the model training effect in the first dimension, Path length represents the first evaluation parameter corresponding to the model training effect in the second dimension, and Consistency scoreThe first evaluation parameter corresponding to the model training effect in the third dimension is denoted as, w1 represents the first weight parameter, w2 represents the second weight parameter, and w3 represents the third weight parameter.

[0132] Optionally, the values of the first weight parameter, the second weight parameter, and the third weight parameter can be any values, as long as the sum of the three is 1. For example, the first weight parameter can be 0.3, the second weight parameter can be 0.3, and the third weight parameter can be 0.4, but not limited thereto.

[0133] In some embodiments, after determining the second evaluation parameter corresponding to each target data through the above embodiments, the second evaluation parameters of each target data can also be sorted in descending order to obtain multiple target data before the set position, as the training data for model training.

[0134] Optionally, the set position can be determined according to actual needs. For example, it can be determined according to the total number of target data and the proportion of high-quality data to be retained. For example, it can be set that the proportion of high-quality data to be retained is p. If the total number of target data is x, then the set position is x*p. Optionally, p can be any positive value less than 1. For example, p can be 0.75 or 0.8, but not limited thereto.

[0135] In some embodiments, the filtered training data can also be used to retrain the model, and the performance metrics (such as accuracy, F1 score, etc.) of the retrained model can be evaluated, so as to verify the data quality evaluation effect of the training data by comparing the performance metrics of the models trained based on the data before and after filtering.

[0136] In some embodiments, when using the LLaMA-7B model as the initialization model and the Alpaca dataset as the training dataset, the accuracy of the filtered model is increased by an average of 4.2%, and the F1 score is increased by 0.035, indicating that the model performance can be significantly improved through data filtering, that is, the solution provided by the present application can effectively identify unstable data samples in the training process.

[0137] According to the solution provided by the present application, for a single target data, the target model can be trained for a set number of rounds based on the target data, so as to use the set number of training parameter sets generated after the set number of rounds of training as multiple target parameter sets for evaluating the data quality of the first data, thereby realizing the evaluation of the data quality based on these multiple target parameter sets.

[0138] See Figure 3 , Figure 3This is a schematic flowchart of a data quality assessment process shown according to an exemplary embodiment of the present application. Taking 3 rounds as an example, as Figure 3 shown, after obtaining the initial model M0, the model M0 can be trained in three rounds based on a single piece of data. The first round of training obtains the model M1, the first round of training obtains the model M2, and the first round of training obtains the model M3. Thus, the analysis of the model training effect can be realized based on the model parameters of the model M1, the model M2, and the model M3, to calculate the changes in the model parameters between adjacent rounds, the cumulative effect of the parameter changes, and the consistency of the multi-round parameter change directions. Thus, by calculating the stability index (corresponding to the first dimension), the cumulative effect index (corresponding to the second dimension), and the directionality index (corresponding to the third dimension), the model training effect can be evaluated from multiple dimensions. Thus, based on the multi-dimensional evaluation results of the model training effect, the final score can be obtained through weighted calculation, and then the data can be screened by sorting in descending order of the score to obtain high-quality training data.

[0139] Through the solution provided by the present application, by setting a multi-round training mechanism, the same data sample can be used to train the model repeatedly, and the parameter changes after each round of training can be recorded to observe the long-term impact of the data on the model, so as to more accurately evaluate the data quality. In addition, the present application can also provide a multi-dimensional evaluation system including change quantity stability, cumulative effect trend, and direction consistency, which can not only more comprehensively evaluate the data quality, but also effectively identify samples that may cause model instability. All in all, through this model data quality assessment method based on multi-round cumulative training, not only can the effect and efficiency of data cleaning be improved, but also a higher-quality data set can be provided for model training, thus significantly improving the performance and generalization ability of the machine learning model.

[0140] In addition, for a single piece of target data, multiple target parameter sets for evaluating the data quality of the first data can be sampled according to a set step size from multiple training parameter sets generated by training the target model with the first data in multiple rounds, so as to realize the evaluation of the data quality based on these multiple target parameter sets.

[0141] See Figure 4 , Figure 4 This is a schematic flowchart of a data quality assessment process shown according to an exemplary embodiment of the present application, such as Figure 4As shown, after the model is initialized, the model can be trained for multiple rounds based on a single piece of data and sampled regularly to obtain multiple parameter sets starting from the initial parameters with a fixed sampling interval, forming a parameter sequence. Based on the sampled parameter sequence, by calculating the stability index (corresponding to the first dimension), the cumulative effect index (corresponding to the second dimension), and the directionality index (corresponding to the third dimension), the training effect of the model can be evaluated from multiple dimensions. Thus, based on the multi-dimensional evaluation results of the model training effect, the final score can be obtained through weighted calculation, and then the data can be screened in descending order of the score to obtain high-quality training data.

[0142] Through the solution provided in this application, a reasonable sampling strategy can be designed to record the change trajectory of the model parameters during the training process and construct a complete dynamic training path. This dynamic recording method can capture process information that cannot be obtained by traditional static comparison methods, providing a richer basis for data quality evaluation. In addition, by providing a multi-dimensional analysis method including dimensions such as the parameter change speed, direction consistency, path length, and oscillation degree, the impact of data on model training can be comprehensively evaluated, and problem data can be effectively identified.

[0143] Corresponding to the embodiments of the foregoing method, this application also provides embodiments of an apparatus and a computing device to which the apparatus is applied.

[0144] As Figure 5 shown, Figure 5 is a block diagram of a data quality evaluation apparatus shown according to an exemplary embodiment of this application. The apparatus includes:

[0145] An acquisition module 501, configured to obtain multiple target parameter sets for evaluating the data quality of each of the target data based on multiple training parameter sets generated by training a target model multiple times using each of the target data in the target data to be evaluated.

[0146] An evaluation module 502, configured to evaluate the effect of training the target model based on each of the target data from multiple dimensions based on the multiple target parameter sets, to obtain multiple first evaluation parameters. The multiple dimensions include at least a first dimension, a second dimension, and a third dimension. The first dimension is used to indicate the degree of parameter change oscillation during the training process, the second dimension is used to indicate the cumulative parameter change trend during multiple rounds of training, and the third dimension is used to indicate the degree of consistency of the parameter change direction during multiple rounds of training.

[0147] The evaluation module 502 is further configured to evaluate the data quality of each of the target data based on the multiple first evaluation parameters.

[0148] In some embodiments, for the first data which is one of the target data, when the evaluation module 502 is used to evaluate the effect of training the target model based on the first data from a first dimension based on the multiple target parameter sets and obtain a first evaluation parameter, it is used for any one of the following:

[0149] Determine a first parameter change amount between the target parameter sets obtained from two adjacent training processes of the target model based on the first data, so as to evaluate the effect of training the target model based on the first data from a first dimension based on the first parameter change amount, and obtain a first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension;

[0150] Determine a first parameter change direction similarity between the target parameter sets obtained from three adjacent training processes of the target model based on the first data, and a second parameter change direction similarity between the target parameter sets obtained from three adjacent training processes of the target model based on each target data, so as to evaluate the effect of training the target model based on the first data from a first dimension based on the first parameter change direction similarity and the second parameter change direction similarity corresponding to each target data, and obtain a first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

[0151] In some embodiments, when the evaluation module 502 is used to determine a first parameter change amount between the target parameter sets obtained from two adjacent training processes of the target model based on the first data, and evaluate the effect of training the target model based on the first data from a first dimension based on the first parameter change amount, and obtain a first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension, it is used for:

[0152] Determine a first vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent training processes of the target model based on the first data, so as to determine the modulus value of each first vector difference as the first parameter change amount;

[0153] Determine the standard deviation of all the first parameter change amounts, and use the reciprocal of the determined standard deviation as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

[0154] In some embodiments, when the evaluation module 502 is used to determine the similarity of the first parameter change directions between the target parameter sets obtained from three adjacent training processes of the target model based on the first data, and the similarity of the second parameter change directions between the target parameter sets obtained from three adjacent training processes of the target model based on each target data, so as to evaluate the effect of training the target model based on the first data from the first dimension and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension, it is used for:

[0155] Determine the first vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent training processes of the target model based on the first data, and the second vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent training processes of the target model based on each target data;

[0156] Respectively determine the cosine similarity between every two adjacent first vector differences and the cosine similarity between every two adjacent second vector differences corresponding to each target data as the similarity of the first parameter change direction and the similarity of the second parameter change direction corresponding to each target data;

[0157] Based on the difference between the first value and each similarity of the first parameter change direction, and the difference between the first value and the similarity of the second parameter change direction corresponding to each target data, as the first difference and the second difference corresponding to each target data;

[0158] Determine the sum value of all the first differences and the sum value of all the second differences corresponding to each target data as the first parameter value and the second parameter value corresponding to each target data;

[0159] Determine the ratio of the first parameter value to the maximum second parameter value among the second parameter values corresponding to all the target data, and use the difference between the first value and the determined ratio as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

[0160] In some embodiments, for the first data which is one of the target data, when the evaluation module 502 is used to evaluate the effect of training the target model based on the first data from the second dimension based on the multiple target parameter sets and obtain a first evaluation parameter, it is used for any one of the following:

[0161] Determine a second parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, and a third parameter change amount between the target parameter set obtained from one round of training of the target model based on the first data and the target parameter set of the initial target model, so as to evaluate the training effect of the target model based on the first data from a second dimension, and obtain a first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension;

[0162] Determine a fourth parameter change amount between the target parameter sets obtained from two adjacent rounds of training of the target model based on the first data, and a fifth parameter change amount between the target parameter sets obtained from two adjacent rounds of training of the target model based on each target data, so as to evaluate the training effect of the target model based on the first data from a second dimension, and obtain a first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension.

[0163] In some embodiments, when the evaluation module 502 is used to determine a second parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, and a third parameter change amount between the target parameter set obtained from one round of training of the target model based on the first data and the target parameter set of the initial target model, so as to evaluate the training effect of the target model based on the first data from a second dimension, and obtain a first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension, it is used for:

[0164] Respectively determine a third vector difference between the vector corresponding to the target parameter set of the target model finally trained and the vector corresponding to the target parameter set of the initial target model, and a fourth vector difference between the vector corresponding to the target parameter set of the target model obtained after one round of training and the vector corresponding to the target parameter set of the initial target model, so as to determine the modulus of the third vector difference and the modulus of the fourth vector difference as the second parameter change amount and the third parameter change amount;

[0165] Determine the difference between the second parameter change amount and the third parameter change amount, and use the ratio between the determined difference and the third parameter change amount as the first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension.

[0166] In some embodiments, the evaluation module 502 is configured to determine a fourth parameter change amount between target parameter sets obtained from two adjacent training processes of the target model based on the first data, and a fifth parameter change amount between target parameter sets obtained from two adjacent training processes of the target model based on each target data, so as to evaluate the effect of training the target model based on the first data from a second dimension based on the fourth parameter change amount and the fifth parameter change amount corresponding to each target data, and when obtaining a first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension, is used for:

[0167] respectively determine a first vector difference between vectors corresponding to target parameter sets obtained from two adjacent training processes of the target model based on the first data, and a second vector difference between vectors corresponding to target parameter sets obtained from two adjacent training processes of the target model based on each target data, so as to respectively determine the second-order norm of each first vector difference and the second-order norm of each second vector difference corresponding to each target data, as the fourth parameter change amount and the fifth parameter change amount corresponding to each target data;

[0168] determine the sum value of all the fourth parameter change amounts and the sum value of all the fifth parameter change amounts corresponding to each target data, as a third parameter value and a fourth parameter value;

[0169] determine the ratio of the third parameter value to the maximum fourth parameter value among the fourth parameter values corresponding to all the target data, as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension.

[0170] In some embodiments, for the first data which is one of the target data, when the evaluation module 502 is configured to evaluate the effect of training the target model based on the first data from a third dimension based on the multiple target parameter sets and obtain a first evaluation parameter, it is used for any one of the following:

[0171] determine the similarity of the first parameter change direction between target parameter sets obtained from three adjacent training processes of the target model based on the first data, so as to evaluate the training effect of training the target model based on the first data from the third dimension based on the similarity of the first parameter change direction, and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension;

[0172] Determine the fourth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, so as to evaluate the training effect of training the target model based on the first data from the third dimension, and obtain the first evaluation parameter corresponding to the training effect of training the target model based on the first data in the third dimension.

[0173] In some embodiments, when the evaluation module 502 is used to determine the first parameter change direction similarity between the target parameter sets obtained in three adjacent training processes of the target model based on the first data, and evaluate the training effect of training the target model based on the first data from the third dimension based on the first parameter change direction similarity, and obtain the first evaluation parameter corresponding to the training effect of training the target model based on the first data in the third dimension, it is used for:

[0174] Determine the first vector difference between the vectors corresponding to the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and determine the cosine similarity between every two adjacent first vector differences as the first parameter change direction similarity;

[0175] Determine the average value of all the first parameter change direction similarities, and use the determined average value as the first evaluation parameter corresponding to the training effect of training the target model based on the first data in the third dimension.

[0176] In some embodiments, when the evaluation module 502 is used to determine the fourth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, and evaluate the training effect of training the target model based on the first data from the third dimension based on the fourth parameter change amount and the fifth parameter change amount, and obtain the first evaluation parameter corresponding to the training effect of training the target model based on the first data in the third dimension, it is used for:

[0177] Determine the first vector difference between the vectors corresponding to the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and determine the second-order norm of each first vector difference as the fourth parameter change amount;

[0178] Determine the third vector difference between the vector corresponding to the set of target parameters of the finally trained first model and the vector corresponding to the set of target parameters of the initial target model, so as to determine the second-order norm of the third vector difference as the fifth parameter change amount;

[0179] Determine the sum value of all the fourth parameter change amounts, and use the ratio of the fifth parameter change amount to the determined sum value as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension.

[0180] In some embodiments, when the evaluation module 502 is used to evaluate the data quality of each target data based on the multiple first evaluation parameters, it is used for:

[0181] For the first data that is one of the target data, based on the preset first weight parameter, second weight parameter, and third weight parameter, perform weighted summation on the first evaluation parameters corresponding to the effects of training the target model based on the first data in the first dimension, the second dimension, and the third dimension respectively, to obtain a second evaluation parameter for evaluating the data quality of the first data.

[0182] In some embodiments, the device further includes:

[0183] A screening module, configured to perform a descending order sorting based on the second evaluation parameter of each target data;

[0184] The screening module is further configured to obtain multiple target data before the set position in the sorting as the training data for model training.

[0185] In some embodiments, when the obtaining module 501 is used to obtain multiple sets of target parameters for evaluating the data quality of each target data based on multiple sets of training parameters generated by performing multiple rounds of training on the target model using each target data in the target data to be evaluated, it is used for any one of the following:

[0186] For the first data that is one of the target data, perform a set number of rounds of training on the target model based on the first data, and use the set number of sets of training parameters generated after the set number of rounds of training as multiple sets of target parameters for evaluating the data quality of the first data;

[0187] For the first data that is one of the target data, sample the multiple sets of training parameters generated by performing multiple rounds of training on the target model using the first data at a set step size to obtain multiple sets of target parameters for evaluating the data quality of the first data.

[0188] In some embodiments, the target data includes at least one of text data, image data, audio data, and time series data;

[0189] The target model is used to perform at least one of natural language processing tasks, computer vision tasks, audio processing tasks, and time series data analysis tasks.

[0190] The implementation processes of the functions and roles of each module in the above device are specifically described in the implementation processes of the corresponding steps in the above method, and will not be elaborated here.

[0191] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0192] This application also provides a computing device. Refer to Figure 6 , Figure 6 which is a schematic structural diagram of a computing device shown by this application according to an exemplary embodiment. As Figure 6 shown, the computing device includes a processor 610, a memory 620, and a network interface 630. The memory 620 is used to store computer instructions that can run on the processor 610. The processor 610 is used to implement the data quality assessment method provided by any embodiment of this application when executing the computer instructions. The network interface 630 is used to implement input and output functions. In more possible implementation manners, the computing device may further include other hardware, and this application does not make any limitations thereto.

[0193] This application also provides a computer-readable storage medium. The computer-readable storage medium can be in various forms. For example, in different examples, the computer-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof. Specifically, the computer-readable medium can also be paper or other suitable media that can print programs. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the data quality assessment method provided by any embodiment of this application.

[0194] The present application also provides a computer program product, including a computer program which, when executed by a processor, implements the data quality assessment method provided in any embodiment of the present application.

[0195] Those skilled in the art should understand that one or more embodiments of the present application may be provided as a method, an apparatus, a computing device, a computer-readable storage medium, or a computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0196] The embodiments in the present application are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the embodiment corresponding to the computing device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment.

[0197] The specific embodiments of the present application are described above. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application may be executed in a different order from that in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0198] The embodiments of the subject matter and the functional operations described in the present application can be implemented in the following: digital electronic circuits, tangible computer software or firmware, computer hardware including the structures disclosed in the present application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in the present application can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data quality assessment device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0199] The processes and logical flows described in this application can be executed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the apparatus can also be implemented as special-purpose logic circuitry.

[0200] Computers suitable for executing computer programs include, by way of example, general and / or special-purpose microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. Basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, etc., or the computer will be operatively coupled to such mass storage devices to receive data therefrom or transfer data thereto, or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few.

[0201] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0202] Although this application contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of what is claimed, but rather are mainly used to describe the features of specific embodiments of a particular invention. Certain features described in multiple embodiments in this application can also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may operate in certain combinations as described above and are even initially claimed as such, one or more features from a claimed combination can in some cases be removed from that combination, and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0203] Similarly, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or sequentially, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system modules and components in the above embodiments should not be understood as required in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0204] Accordingly, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the present application. In some cases, the acts recited in the present application may be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the drawings are not necessarily in the particular order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0205] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention herein. The present application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not claimed in the present application. That is, the present application is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.

[0206] The foregoing are only alternative embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of protection of the present application.

Claims

1. A data quality assessment method, characterized in that, The method includes: Obtaining a plurality of target parameter sets for evaluating the data quality of each of the target data based on a plurality of training parameter sets generated by performing multiple rounds of training on a target model respectively using each piece of target data to be evaluated; Evaluating the effects of training the target model based on each of the target data from multiple dimensions based on the plurality of target parameter sets, obtaining a plurality of first evaluation parameters, the multiple dimensions at least including a first dimension, a second dimension, and a third dimension, the first dimension being used to indicate the degree of parameter change oscillation during the training process, the second dimension being used to indicate the cumulative parameter change trend during multiple rounds of training, and the third dimension being used to indicate the degree of consistency of the parameter change direction during multiple rounds of training; Evaluating the data quality of each of the target data based on the plurality of first evaluation parameters.

2. The method according to claim 1, characterized in that, For a first piece of data that is one of the target data, when evaluating the effect of training the target model based on the first piece of data from the first dimension based on the plurality of target parameter sets and obtaining a first evaluation parameter, the method includes any one of the following: Determining a first parameter change amount between the target parameter sets obtained from two adjacent rounds of training of the target model based on the first piece of data, so as to evaluate the effect of training the target model based on the first piece of data from the first dimension based on the first parameter change amount, and obtaining a first evaluation parameter corresponding to the effect of training the target model based on the first piece of data in the first dimension; Determining a first parameter change direction similarity between the target parameter sets obtained from three adjacent rounds of training of the target model based on the first piece of data, and a second parameter change direction similarity between the target parameter sets obtained from three adjacent rounds of training of the target model based on each piece of target data, so as to evaluate the effect of training the target model based on the first piece of data from the first dimension based on the first parameter change direction similarity and the second parameter change direction similarity corresponding to each piece of target data, and obtaining a first evaluation parameter corresponding to the effect of training the target model based on the first piece of data in the first dimension.

3. The method according to claim 2, characterized in that, The determining a first parameter change amount between the target parameter sets obtained from two adjacent rounds of training of the target model based on the first piece of data, so as to evaluate the effect of training the target model based on the first piece of data from the first dimension based on the first parameter change amount, and obtaining a first evaluation parameter corresponding to the effect of training the target model based on the first piece of data in the first dimension, includes: Determining a first vector difference between the vectors corresponding to the target parameter sets obtained from two adjacent rounds of training of the target model based on the first piece of data, so as to determine the modulus value of each first vector difference as the first parameter change amount; Determining the standard deviation of all the first parameter change amounts, and taking the reciprocal of the determined standard deviation as the first evaluation parameter corresponding to the effect of training the target model based on the first piece of data in the first dimension.

4. The method according to claim 2, wherein Determining a first parameter change direction similarity between target parameter sets obtained from three adjacent rounds of training of a target model based on the first data, and a second parameter change direction similarity between target parameter sets obtained from three adjacent rounds of training of the target model based on each target data, to evaluate the effect of training the target model based on the first data from a first dimension, and obtaining a first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension, includes: Determining a first vector difference between vectors corresponding to target parameter sets obtained from two adjacent rounds of training of the target model based on the first data, and a second vector difference between vectors corresponding to target parameter sets obtained from two adjacent rounds of training of the target model based on each target data; Respectively determining the cosine similarity between every two adjacent first vector differences and the cosine similarity between every two adjacent second vector differences corresponding to each target data, as the first parameter change direction similarity and the second parameter change direction similarity corresponding to each target data; Using the difference between a first value and each first parameter change direction similarity, and the difference between the first value and the second parameter change direction similarity corresponding to each target data, as a first difference and the second difference corresponding to each target data; Determining the sum value of all the first differences and the sum value of all the second differences corresponding to each target data, as a first parameter value and the second parameter value corresponding to each target data; Determining the ratio of the first parameter value to the maximum second parameter value among the second parameter values corresponding to all the target data, and using the difference between the first value and the determined ratio as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the first dimension.

5. The method according to claim 1, characterized in that For the first data which is one of the target data, when evaluating the effect of training the target model based on the first data from a second dimension based on the multiple target parameter sets and obtaining a first evaluation parameter, the method includes any one of the following: Determining a second parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, and a third parameter change amount between the target parameter set obtained from one round of training of the target model based on the first data and the target parameter set of the initial target model, to evaluate the effect of training the target model based on the first data from the second dimension, and obtaining a first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension; Determine the fourth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on each target data, so as to evaluate the training effect of the target model based on the first data from a second dimension, and obtain a first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension.

6. The method according to claim 5, characterized in that, The method for determining the second parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, and the third parameter change amount between the target parameter set obtained in one training process of the target model based on the first data and the target parameter set of the initial target model, so as to evaluate the training effect of the target model based on the first data from a second dimension, and obtain a first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension, includes: Respectively determine the third vector difference between the vector corresponding to the target parameter set of the target model finally trained and the vector corresponding to the target parameter set of the initial target model, and the fourth vector difference between the vector corresponding to the target parameter set of the target model obtained after one round of training and the vector corresponding to the target parameter set of the initial target model, so as to determine the modulus of the third vector difference and the modulus of the fourth vector difference as the second parameter change amount and the third parameter change amount; Determine the difference between the second parameter change amount and the third parameter change amount, and use the ratio between the determined difference and the third parameter change amount as the first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension.

7. The method according to claim 5, wherein The method for determining the fourth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on each target data, so as to evaluate the training effect of the target model based on the first data from a second dimension, and obtain a first evaluation parameter corresponding to the training effect of the target model based on the first data in the second dimension, includes: Determine the first vector difference between the vectors corresponding to the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the second vector difference between the vectors corresponding to the target parameter sets obtained in two adjacent training processes of the target model based on each target data, so as to determine the second-order norm of each first vector difference and the second-order norm of each second vector difference corresponding to each target data, as the fourth parameter change amount and the fifth parameter change amount corresponding to each target data; Determine the sum value of all the fourth parameter change amounts and the sum value of all the fifth parameter change amounts corresponding to each target data, as the third parameter value and the fourth parameter value; Determine the ratio of the third parameter value to the maximum fourth parameter value among the fourth parameter values corresponding to all the target data, as the first evaluation parameter corresponding to the effect of training the target model based on the first data in the second dimension.

8. The method according to claim 1, wherein For the first data that is one of the target data, when evaluating the effect of training the target model based on the first data from the third dimension based on the multiple target parameter sets and obtaining a first evaluation parameter, the method includes any one of the following: Determine the first parameter change direction similarity between the target parameter sets obtained in three adjacent training processes of the target model based on the first data, so as to evaluate the training effect of training the target model based on the first data from the third dimension based on the first parameter change direction similarity, and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension; Determine the fourth parameter change amount between the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the fifth parameter change amount between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, so as to evaluate the effect of training the target model based on the first data from the third dimension based on the fourth parameter change amount and the fifth parameter change amount, and obtain the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension.

9. The method according to claim 8, wherein The determining the first parameter change direction similarity between the target parameter sets obtained in three adjacent training processes of the target model based on the first data, so as to evaluate the training effect of training the target model based on the first data from the third dimension based on the first parameter change direction similarity, and obtaining the first evaluation parameter corresponding to the effect of training the target model based on the first data in the third dimension includes: Determine the first vector difference between the vectors corresponding to the target parameter sets obtained in two adjacent training processes of the target model based on the first data, so as to determine the cosine similarity between every two adjacent first vector differences, as the first parameter change direction similarity; Determine the average of the similarity degrees of the change directions of all the first parameters, and use the determined average value as the first evaluation parameter corresponding to the third dimension of the effect of training the target model based on the first data.

10. The method according to claim 8, characterized in that, Determine the change amount of the fourth parameter between the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and the change amount of the fifth parameter between the target parameter set of the target model finally trained based on the first data and the target parameter set of the initial target model, so as to evaluate the effect of training the target model based on the first data from the third dimension, and obtain the first evaluation parameter corresponding to the third dimension of the effect of training the target model based on the first data, including: Determine the first vector difference between the vectors corresponding to the target parameter sets obtained in two adjacent training processes of the target model based on the first data, and determine the second-order norm of each first vector difference as the change amount of the fourth parameter; Determine the third vector difference between the vector corresponding to the target parameter set of the first model finally trained and the vector corresponding to the target parameter set of the initial target model, and determine the second-order norm of the third vector difference as the change amount of the fifth parameter; Determine the sum value of all the change amounts of the fourth parameter, and use the ratio of the change amount of the fifth parameter to the determined sum value as the first evaluation parameter corresponding to the third dimension of the effect of training the target model based on the first data.

11. The method according to claim 1, characterized in that, Based on the multiple first evaluation parameters, evaluate the data quality of each target data, including: For the first data that is one of the target data, based on the preset first weight parameter, second weight parameter, and third weight parameter, perform weighted summation on the first evaluation parameters corresponding to the first dimension, second dimension, and third dimension of the effect of training the target model based on the first data, and obtain the second evaluation parameter for evaluating the data quality of the first data.

12. The method according to claim 11, wherein After evaluating the data quality of each target data based on the multiple first evaluation parameters, the method further includes: Perform descending sorting based on the second evaluation parameter of each target data; Obtain multiple target data before the set position in the sorting as the training data for model training.

13. The method according to claim 1, wherein Based on the multiple training parameter sets generated by training the target model with each target data in the target data to be evaluated respectively, obtain multiple target parameter sets for evaluating the data quality of each target data, including any one of the following: For the first data that is one of the target data, train the target model for a set number of rounds based on the first data, and use the set number of training parameter sets generated after the set number of rounds of training as the multiple target parameter sets for evaluating the data quality of the first data; For a first piece of data that is one of the target data, sample multiple sets of training parameters generated by training a target model multiple times using the first piece of data at a set step size to obtain multiple sets of target parameters for evaluating the data quality of the first piece of data.

14. The method according to claim 1, wherein The target data includes at least one of text data, image data, audio data, and time series data; The target model is used to perform at least one of natural language processing tasks, computer vision tasks, audio processing tasks, and time series data analysis tasks.

15. A data quality evaluation device, characterized in that, The device includes: An acquisition module, configured to obtain multiple sets of target parameters for evaluating the data quality of each piece of target data based on multiple sets of training parameters generated by training a target model multiple times using each piece of target data to be evaluated; An evaluation module, configured to evaluate the effects of training the target model based on each piece of target data from multiple dimensions based on the multiple sets of target parameters to obtain multiple first evaluation parameters. The multiple dimensions at least include a first dimension, a second dimension, and a third dimension. The first dimension is used to indicate the degree of parameter change oscillation during the training process, the second dimension is used to indicate the cumulative parameter change trend during the multiple rounds of training, and the third dimension is used to indicate the degree of consistency of the parameter change direction during the multiple rounds of training; The evaluation module is further configured to evaluate the data quality of each piece of target data based on the multiple first evaluation parameters.

16. A computing device, characterized in that, The computing device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, when the processor executes the program, it implements the operations performed by the data quality evaluation method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, A program is stored on the computer-readable storage medium, and the program is executed by the processor to perform the operations performed by the data quality evaluation method according to any one of claims 1 to 14.

Images