Method, device and equipment for testing effectiveness of data set quality assessment model

Abstract

The present disclosure relates to a method, device and equipment for validity test of a data set quality evaluation model, and belongs to the technical field of data set evaluation. The method comprises: dividing a training data set and a test data set from a reference data set; performing data change processing on the training data set to obtain n-1 derivative data sets, the training data set and the n-1 derivative data sets together forming n data sets; performing quality evaluation on the n data sets by a data set quality evaluation model to obtain n evaluation results; training n artificial intelligence (AI) models respectively by using the n data sets to obtain n trained AI models; processing the test data set by the n trained AI models respectively to obtain n test results; and determining the validity of the data set quality evaluation model based on the n test results and the n evaluation results. The present disclosure can effectively and accurately test and verify the evaluation capability of the data set quality evaluation model itself.

Description

Technical Field

[0001] This disclosure relates to the field of dataset evaluation technology, and in particular to a method, apparatus and equipment for testing the effectiveness of a dataset quality evaluation model. Background Technology

[0002] A dataset is a collection of data, also known as a data set, data collection, or information set. With the development of artificial intelligence (AI), the use of datasets is becoming increasingly frequent, and a wide variety of datasets are emerging.

[0003] In the field of artificial intelligence applications, datasets are the foundation for training various intelligent algorithms, and the quality of the dataset determines the theoretical upper limit of the algorithm's effectiveness. Dataset quality assessment is a core task in data quality evaluation within the AI ​​field. High-quality or highly relevant datasets are extremely helpful for model training, effectively improving its accuracy. Related technologies utilize dataset quality assessment models to evaluate dataset quality. These models typically set evaluation metrics across multiple dimensions, calculate the values ​​of each metric using methods such as hierarchical analysis, and then synthesize the values ​​of all metrics to ultimately determine the dataset's quality.

[0004] However, if the evaluation metrics or techniques used in the dataset quality assessment model itself are flawed, the assessment results may be inaccurate or erroneous, thus failing to achieve effective dataset evaluation. Therefore, before a dataset quality assessment model is put into application, it is urgent to test and verify its evaluation effectiveness; however, relevant technologies have not yet provided effective and accurate testing methods. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a method, apparatus, and device for testing the effectiveness of a dataset quality assessment model, which can effectively and accurately test and verify the assessment capability of the dataset quality assessment model itself.

[0006] According to a first aspect of the present disclosure, a method for testing the effectiveness of a dataset quality assessment model is provided, comprising:

[0007] The training dataset and the test dataset are divided from the reference dataset, and the data type of the reference dataset is consistent with the data type of the evaluation object of the dataset quality assessment model to be tested for effectiveness.

[0008] The training dataset is subjected to data transformation processing to obtain n-1 derived datasets; the training dataset and the n-1 derived datasets together constitute n datasets;

[0009] The quality of the n datasets is evaluated using the dataset quality assessment model, resulting in n evaluation results.

[0010] n AI models are trained using the n datasets respectively, resulting in n AI models that have been trained.

[0011] The test dataset is processed by the n trained AI models to obtain n test results;

[0012] Based on the n test results and the n evaluation results, the effectiveness of the dataset quality evaluation model is determined.

[0013] Optionally, the step of performing data transformation processing on the training dataset to obtain n-1 derived datasets includes: acquiring the data content of the training dataset; determining at least one method of data transformation processing based on the data content; and performing data transformation processing on the training dataset with different degrees of processing according to the target method among the at least one method to obtain multiple derived datasets.

[0014] Optionally, based on the data content of the image classification dataset, at least one method of data modification is determined, including: adding noise to the image, deleting image-level labels, tampering with image-level labels, adding multiple image-level labels to the image, increasing the repetition rate of a specific image, adding conflicting labels to copies of the same image, and / or deleting some images under a specific category to cause class imbalance; based on the data content of the text classification dataset, at least one method of data modification is determined, including: embedding typos in the text, shuffling the text sequence, deleting classification labels, tampering with classification labels, adding multiple classification labels to the text, increasing the repetition rate of a specific text, adding conflicting labels to copies of the same text, and / or deleting some text under a specific category to cause class imbalance; based on the data content of the audio classification dataset, at least one method of data modification is determined, including: adding background noise to the audio. The data modification methods determined based on the data content of the video segmentation dataset include: adding incorrect segmentation markers to the original video, deleting video segments, deleting shot segmentation location labels, tampering with shot segmentation location labels, adding multiple shot segmentation location labels to the original video, increasing the repetition rate of a specific original video, and / or adding conflicting labels to copies of the same original video; the data modification methods determined based on the data content of the text-to-image dataset include: adding mismatched text-image pairs, deleting text or image from text-image pairs, scrambling the matching relationship of text-image pairs, and / or increasing the repetition rate of specific text-image pairs.

[0015] Optionally, the n AI models are AI models with the same network structure and initial parameters, and the n AI models are matched with the data type and application task of the training dataset.

[0016] Optionally, the dataset quality assessment model includes: a target layer, a feature layer, and an indicator layer; the step of assessing the quality of the n datasets respectively through the dataset quality assessment model to obtain n assessment results includes: obtaining the dataset quality conclusion calculated by the target layer, wherein the dataset quality conclusion is a score value or evaluation level; obtaining at least one dimension of quality characteristics calculated by the feature layer to indicate the quality of the dataset, wherein the quality characteristics include: completeness, consistency, confidentiality, accuracy, normalization, unbiasedness, diversity, and / or timeliness; obtaining the assessment indicators of each quality characteristic calculated by the indicator layer; wherein the assessment indicators of completeness include: data element completeness, data record completeness, metadata completeness, null value rate, data category completeness, and / or data quantity completeness; the assessment indicators of accuracy include: logical conflict degree. The evaluation metrics for data quality include: quantity and / or annotation metrics; the confidentiality evaluation metrics include: raw data metrics, annotated data metrics, data sharing metrics, data analysis metrics, data usage metrics, and / or data obsolescence metrics; the accuracy evaluation metrics include: data duplication rate and / or outlier metrics; the standardization evaluation metrics include: data standard metrics, authoritative reference data metrics, business rule metrics, security standard metrics, standardization metrics, and / or data format compliance; the unbiasedness evaluation metrics include: distribution unbiasedness and / or historical unbiasedness; the diversity evaluation metrics include: data source diversity and / or data specification diversity; the timeliness evaluation metrics include: time period correctness, timeliness, and / or temporal sequence; the evaluation result of the corresponding dataset is obtained by combining the dataset quality conclusions, the quality characteristics, and one or more of the evaluation metrics.

[0017] Optionally, determining the validity of the dataset quality assessment model based on the n test results and the n evaluation results includes: calculating the correlation coefficient between the n evaluation results and the n test results; if the correlation coefficient is greater than a preset threshold, the n evaluation results are consistent with the n test results, and the dataset quality assessment model is determined to be effective in assessing the quality of the dataset; if the correlation coefficient is less than the preset threshold, the n evaluation results are inconsistent with the n test results, and the dataset quality assessment model is determined to be ineffective in assessing the quality of the dataset.

[0018] Optionally, calculating the correlation coefficient between the n evaluation results and the n test results includes: converting the n test results into an n*m matrix, where m represents the number of samples included in the test dataset, and each row of the n*m ​​matrix represents a test result; converting the n evaluation results into an n*k matrix, where k represents the number of data dimensions calculated by the dataset quality assessment model during the evaluation process, and each row of the n*k matrix represents an evaluation result, and the same row in the n*m ​​matrix and the n*k matrix corresponds to the same dataset in the n datasets; calculating the weighted rank R1 of each row element in the n*m ​​matrix; calculating the weighted rank R2 of each row element in the n*k matrix; and calculating the Sporman rank correlation coefficient between the weighted rank R1 and the weighted rank R2.

[0019] According to a second aspect of the present disclosure, an apparatus for testing the effectiveness of a dataset quality assessment model is provided, the apparatus comprising:

[0020] The dataset partitioning module is used to partition the reference dataset into a training dataset and a test dataset. The data type of the reference dataset is consistent with the data type of the evaluation object of the dataset quality assessment model to be tested for effectiveness.

[0021] The data transformation module is used to perform data transformation processing on the training dataset to obtain n-1 derived datasets; the training dataset and the n-1 derived datasets together constitute n datasets;

[0022] The quality assessment module is used to assess the quality of the n datasets respectively using the dataset quality assessment model, and obtain n assessment results;

[0023] The model training module is used to train n AI models using the n datasets respectively, so as to obtain n AI models that have been trained.

[0024] The model testing module is used to process the test dataset using the n trained AI models respectively to obtain n test results;

[0025] The validity assessment module is used to determine the validity of the dataset quality assessment model based on the n test results and the n assessment results.

[0026] According to a third aspect of the present disclosure, a computer device is provided, the computer device comprising: a processor and a memory storing computer program instructions; wherein the processor executes the computer program instructions to implement a method for testing the validity of a dataset quality assessment model provided in the first aspect of the present disclosure.

[0027] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer program instructions are stored on the computer-readable storage medium, and the computer program instructions, when executed by a processor, implement the validity testing method for the dataset quality assessment model provided in the first aspect of the present disclosure.

[0028] The technical solutions provided in this disclosure have at least the following beneficial effects:

[0029] This invention provides an automated method for testing and validating the effectiveness of a dataset quality assessment model. On one hand, the invention processes n datasets using the dataset quality assessment model to obtain n assessment results; on the other hand, it trains n AI models on each of the n datasets, and then processes the test datasets using each of the trained AI models to obtain n test results. The n assessment results indicate the dataset quality assessment model's evaluation of the quality differences among the n datasets, while the n test results indirectly and accurately indicate these quality differences. Using the n test results as a reference, the consistency between the n assessment results and the n test results is compared and analyzed to determine the effectiveness of the dataset quality assessment model. This automated testing and validation of the dataset quality assessment model's effectiveness before its application avoids inaccuracies or errors in the assessment results due to inherent problems within the model itself, thus contributing to improved accuracy in dataset quality assessment.

[0030] Regarding the technical problem of comparing the consistency between evaluation results and test results to determine the effectiveness of the dataset quality assessment model, the conventional approach in this field is to sort the n evaluation results and n test results separately according to their overall values, and compare whether the evaluation results and test results in the same order correspond to the same dataset to determine whether the n evaluation results and n test results are consistent. However, this conventional approach is not perfect or accurate enough for comparing and analyzing the consistency between evaluation results and test results, and can only determine whether the dataset quality assessment model is effective. In this invention, on the one hand, when obtaining the n evaluation results, intermediate data calculated by the dataset quality assessment model during the evaluation process is incorporated to obtain the most comprehensive and rich evaluation data for consistency comparison analysis, thus fully and accurately verifying the effectiveness of the dataset quality assessment model; on the other hand, when comparing and analyzing the consistency between evaluation results and test results, the correlation coefficient between the n evaluation results and n test results is calculated. The correlation coefficient indicates the degree of effectiveness of the dataset quality assessment model, achieving a more granular verification of effectiveness.

[0031] Furthermore, this invention employs two methods: firstly, it modifies the training dataset using various data processing techniques; secondly, it applies the same data processing techniques to the training dataset with varying degrees of modification, resulting in multiple derived datasets to construct n datasets. Consequently, these n datasets not only contain different types of data errors but also data errors of the same type but with varying intensities, resulting in a richer diversity of quality differences among the n datasets. The effectiveness of the dataset quality assessment model is then tested using these n datasets, analyzing whether the model can accurately assess quality differences across different dimensions, thus achieving a more comprehensive and accurate effectiveness test for the dataset quality assessment model.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0034] Figure 1 This is a flowchart illustrating a method for testing the effectiveness of a dataset quality assessment model according to an exemplary embodiment.

[0035] Figure 2 This is a schematic diagram illustrating a dataset quality assessment model according to an exemplary embodiment.

[0036] Figure 3 This is a block diagram illustrating an effectiveness testing apparatus for a dataset quality assessment model according to an exemplary embodiment.

[0037] Figure 4 This is a block diagram illustrating a computer device 400 according to an exemplary embodiment. Detailed Implementation

[0038] The exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0039] like Figure 1 As shown, the effectiveness testing method in this invention includes the following steps (steps 1 to 6).

[0040] Step 1: Divide the reference dataset into training and test datasets.

[0041] The reference dataset is a publicly available dataset with high data reliability, which can be downloaded from the internet by computer devices. The data type of the reference dataset is consistent with the data type of the dataset quality assessment model being evaluated for validity testing. For example, if the dataset quality assessment model is used to evaluate the quality of an image dataset, then the data type of the reference dataset is images; if the dataset quality assessment model is used to evaluate the quality of a fused dataset, then the data type of the reference dataset includes, but is not limited to, images, text, audio, and video.

[0042] After obtaining the reference dataset, the computer device divides it into training and test datasets for subsequent effectiveness testing. This invention does not limit the specific method of dividing the training and test datasets. Optionally, the computer device divides them proportionally, for example, allocating 80% of the reference dataset to the training dataset and 20% to the test dataset; or, the computer device divides them quantitatively, for example, allocating 1000 data sets from the reference dataset to construct the test dataset and using the remaining data to construct the training dataset; or, the computer device first classifies the data in the reference dataset and then proportionally allocates each category to the training and test datasets.

[0043] Step 2: Perform data transformation processing on the training dataset to obtain n-1 derived datasets.

[0044] In this invention, on the one hand, a dataset quality assessment model is used to process n datasets to obtain n assessment results; on the other hand, n AI models are trained on the n datasets respectively, and the trained n AI models are used to process the test dataset to obtain n test results. Then, the effectiveness of the dataset quality assessment model is determined by comparing and analyzing the n assessment results and the n test results. It is evident that the effectiveness testing of the dataset quality assessment model requires n datasets and one test dataset, resulting in a large data requirement. If all n datasets are derived from the reference dataset, the amount of data in each dataset will be significantly reduced, potentially hindering effective AI model training and affecting the effectiveness test results of the dataset quality assessment model. Therefore, this invention first divides the reference dataset into a training dataset and a test dataset, and then performs data transformation processing on this training dataset to obtain n-1 derived datasets, thus obtaining the n datasets required for the effectiveness testing and verification process of this invention.

[0045] In one example, the specific method of data change processing is matched with the data content of the training dataset, so step 2 above may include the following sub-steps (steps 2.1 to 2.3).

[0046] Step 2.1: Obtain the data content of the training dataset.

[0047] The training dataset includes not only the main data content (i.e., the data types mentioned above), but also the corresponding annotation information (i.e., labels, etc.). It should be understood that for different application tasks, even if the datasets have the same data type, the annotation information may differ.

[0048] For example, the data content of an image classification dataset includes images and image-level labels; the data content of an image semantic segmentation dataset includes images and pixel-level labels; the data content of a text classification dataset includes text and classification labels; the data content of an audio classification dataset includes audio and classification labels; the data content of a video segmentation dataset includes the original video and shot segmentation location labels; and the data content of a broad-modal text-to-image dataset includes image-text pairs.

[0049] Step 2.2: Based on the data content, determine at least one method for handling data changes.

[0050] Computer devices need to acquire one or more data change processing methods that match the data content. Optionally, the computer device may have multiple pre-defined matching relationships between data content and data change processing methods. After acquiring the specific data content of the training dataset, the specific data change processing method can be determined based on the matching relationship.

[0051] For example, data change processing methods that match the data content of an image classification dataset include, but are not limited to: adding noise to images, deleting image-level labels, tampering with image-level labels, adding multiple image-level labels to images, increasing the repetition rate of specific images, adding conflicting labels to copies of the same image, and deleting some images under a specific category to cause class imbalance.

[0052] For example, data change processing methods that match the data content of a text classification dataset include, but are not limited to: embedding typos in the text, shuffling the text sequence, deleting classification labels, tampering with classification labels, adding multiple classification labels to the text, increasing the repetition rate of a specific text, adding conflicting labels to copies of the same text, and deleting parts of the text under a specific category to cause class imbalance.

[0053] For example, data change processing methods that match the data content of the audio classification dataset include, but are not limited to: adding background noise to the audio, deleting audio segments, deleting classification labels, tampering with classification labels, adding multiple classification labels to the audio, increasing the repetition rate of a specific audio, adding conflicting labels to copies of the same audio, and deleting some audio in a specific category to cause class imbalance, etc.

[0054] For example, data change processing methods that match the data content of the video segmentation dataset include, but are not limited to: adding incorrect segmentation tags to the original video, deleting video segments, deleting shot segmentation location labels, tampering with shot segmentation location labels, adding multiple shot segmentation location labels to the original video, increasing the repetition rate of a specific original video, and adding conflicting labels to copies of the same original video.

[0055] For example, data change processing methods that match the data content of the text-to-text dataset include, but are not limited to: adding mismatched text-to-text pairs, deleting text or images from text-to-text pairs, shuffling the matching relationship of text-to-text pairs, and increasing the repetition rate of specific text-to-text pairs.

[0056] Step 2.3: For the target method in at least one of the methods, perform data transformation processing on the training dataset with different degrees of processing according to the target method to obtain multiple derivative datasets.

[0057] In this invention, not only can the training dataset be processed in various ways to obtain multiple derived datasets, facilitating subsequent analysis of the dataset quality assessment model to evaluate its effectiveness in the face of different types of data errors in the dataset; but also, the training dataset can be processed in the same way with multiple degrees of processing to obtain multiple derived datasets, facilitating subsequent analysis of the dataset quality assessment model to evaluate its effectiveness in the face of data errors of the same type but different intensities in the dataset. It should be understood that in practical applications, computer devices can simultaneously employ multiple different methods to process the training dataset.

[0058] Based on this, for the target method obtained in at least one of the methods in step 2.2 above, the computer device performs data transformation processing on the training dataset with various degrees of processing according to the target method to obtain multiple derived datasets. This invention does not limit the specific settings of different processing degrees. Optionally, different processing degrees include at least five processing degrees. For higher and lower processing degrees, at least two similar processing degrees are set respectively, thereby constructing multiple derived datasets with small quality differences to analyze whether the dataset quality assessment model can accurately evaluate small differences in dataset quality. For example, for the data processing method of deleting labels, different processing degrees can be set as: deleting 10% of labels, deleting 15% of labels, deleting 50% of labels, deleting 85% of labels, and deleting 90% of labels.

[0059] Optionally, the aforementioned target method can be any of the at least one method, that is, for each of the at least one method, the training dataset is subjected to various degrees of data transformation according to that method; or, the aforementioned target method can also be a specific method among the at least one methods, such as the target method matching the evaluation metrics used by the dataset quality assessment model, thereby more specifically analyzing the effectiveness of the dataset quality assessment model in the face of specific types of data errors in the dataset. For example, if the evaluation metrics used by the dataset quality assessment model include normalization, diversity, and accuracy, then the target method can include deleting labels, deleting some data under a specific category to cause class imbalance, or tampering with labels.

[0060] Step 3: Use the dataset quality assessment model to assess the quality of each of the n datasets and obtain n assessment results.

[0061] The datasets include the training dataset and n-1 derived datasets. The dataset quality assessment model evaluates the quality of each dataset to obtain an evaluation result. This invention does not limit the specific content or format of the evaluation results. Optionally, to more accurately verify the effectiveness of the dataset quality assessment model, comprehensive and rich evaluation data is needed. Therefore, each evaluation result may include not only the final conclusion output by the dataset quality assessment model but also intermediate data calculated by the model during the evaluation process.

[0062] For example, the dataset quality assessment model is a multi-layered model, including at least three layers: an objective layer, a feature layer, and an indicator layer. Wherein, such as Figure 2As shown, the dataset quality conclusion calculated by the target layer can be a numerical score or an evaluation level (such as excellent, average, poor); the characteristic layer calculates quality characteristics of at least one dimension used to indicate the quality of the dataset, including but not limited to completeness, self-consistency, confidentiality, accuracy, normalization, unbiasedness, diversity, and timeliness; the indicator layer calculates evaluation indicators for each quality characteristic. Evaluation indicators for completeness include but are not limited to data element completeness, data record completeness, metadata completeness, null value rate, data category completeness, and data quantity completeness; evaluation indicators for self-consistency include but are not limited to logical conflict measures and annotation measures; evaluation indicators for confidentiality include but are not limited to... The evaluation metrics are not limited to raw data metrics, labeled data metrics, data sharing metrics, data analysis metrics, data usage metrics, and data obsolescence metrics. Accuracy evaluation metrics include, but are not limited to, data duplication rate and outlier metrics. Normative evaluation metrics include, but are not limited to, data standard metrics, authoritative reference data metrics, business rule metrics, security standard metrics, standardization metrics, and data format compliance metrics. Unbiased evaluation metrics include, but are not limited to, distribution unbiasedness and historical unbiasedness. Diversity evaluation metrics include, but are not limited to, data source diversity and data specification diversity. Timeliness evaluation metrics include, but are not limited to, time period accuracy, timeliness, and time sequence.

[0063] Based on this, each evaluation result obtained in step 3 can include at least one of the following: the dataset quality conclusion output by the target layer of the dataset quality assessment model, each quality characteristic output by the feature layer, and the value of each evaluation indicator output by the indicator layer. For example, each evaluation result obtained in step 3 includes the value of each evaluation indicator output by the indicator layer of the dataset quality assessment model. If the indicator layer includes k evaluation indicators, the n evaluation results obtained by performing quality assessments on n datasets can be represented as an n*k matrix. The element in the i-th row and j-th column of this matrix represents the value of the j-th evaluation indicator calculated by the dataset quality assessment model when performing quality assessment on the i-th dataset among the n datasets.

[0064] Step 4: Train n AI models using n datasets respectively, to obtain n AI models that have been trained.

[0065] The n datasets include the aforementioned training dataset and n-1 derived datasets. Each AI model is trained using one of these n datasets to obtain a fully trained AI model. To ensure consistency in the dataset quality assessment and model validation process, when training n AI models using the n datasets, it is necessary to select n identical AI models (including identical network structure and initial parameters) and perform training using the same method to obtain n fully trained AI models.

[0066] Optionally, to achieve better training results, the AI ​​model used may vary depending on the data type of the training dataset and the application task. For example, if the training dataset is an image classification dataset, the AI ​​model used may be a ResNet50 model; if the training dataset is a text classification dataset, the AI ​​model used may be a Transformer model; if the training dataset is an audio classification dataset, the AI ​​model used may be a torchaudio framework; if the training dataset is a video segmentation dataset, the AI ​​model used may be a TransNet model; and if the training dataset is a text-to-image dataset, the AI ​​model used may be an OpenAI Clip model.

[0067] Step 5: Process the test dataset using each of the n trained AI models to obtain n test results.

[0068] Although n AI models have the same network structure, initial parameters, and training methods, their performance varies due to the different training datasets used. This performance difference reflects the quality differences of the training datasets. In this invention, the n trained AI models are tested using a test dataset, yielding n test results. These n test results directly demonstrate the performance differences among the n trained AI models and indirectly demonstrate the quality differences among the n datasets.

[0069] Each trained AI model processes the test dataset and obtains a processing result for each sample in the test dataset. If the test dataset contains m samples, then each test result contains m processing results. Based on this, the n test results obtained by processing the test dataset with n trained AI models can be represented as an n*m matrix. The element in the p-th row and q-th column of this matrix represents the processing result of the p-th AI model among the n trained AI models for the q-th sample in the test dataset.

[0070] Step 6: Based on n test results and n evaluation results, determine the effectiveness of the dataset quality assessment model.

[0071] On the one hand, n evaluation results can indicate the quality differences among the n datasets identified by the dataset quality assessment model; on the other hand, n test results can accurately and indirectly indicate the quality differences among the n datasets. Using the n test results as a reference, the consistency between the n evaluation results and the n test results is analyzed. If the n evaluation results are consistent with the n test results, the dataset quality assessment model is deemed effective in assessing the quality of the datasets; if the n evaluation results are inconsistent with the n test results, the dataset quality assessment model is deemed ineffective in assessing the quality of the datasets.

[0072] In one example, step 6 above includes the following sub-steps (steps 6.1 to 6.3).

[0073] Step 6.1: Calculate the correlation coefficient between the n evaluation results and the n test results.

[0074] This example uses the Spearman rank correlation coefficient to indicate the consistency between n evaluation results and n test results. The Spearman rank correlation coefficient ranges from 0 to 1, with a higher value indicating a stronger correlation.

[0075] As can be seen from the above embodiments, n test results can be represented as an n*m matrix, where each row of the matrix represents a test result; n evaluation results can be represented as an n*k matrix, where each row of the matrix represents an evaluation result; wherein, the same row in the n*m ​​matrix and the n*k matrix corresponds to the same dataset in the n datasets. Based on this, step 6.1 above includes: calculating the weighted rank R1 of each row element in the n*m ​​matrix; calculating the weighted rank R2 of each row element in the n*k matrix; and calculating the value of the Sporman rank correlation coefficient between R1 and R2.

[0076] Step 6.2: If the correlation coefficient is greater than the preset threshold, then the n evaluation results are consistent with the n test results, confirming that the dataset quality assessment model is effective in assessing the quality of the dataset.

[0077] This example does not specify a particular value for the preset threshold. In practical applications, it can be determined based on the required effectiveness of the dataset quality assessment model. If the effectiveness requirement is strict, a higher preset threshold (e.g., 0.8) can be set; if the effectiveness requirement is lenient, a lower preset threshold (e.g., 0.5) can be set. When the correlation coefficient is greater than the preset threshold, the larger the correlation coefficient, the more consistent the n evaluation results are with the n test results, and the higher the effectiveness of the dataset quality assessment model.

[0078] Step 6.3: If the correlation coefficient is less than the preset threshold, then the n evaluation results are inconsistent with the n test results, and it is determined that the dataset quality assessment model is invalid for the quality assessment of the dataset.

[0079] If the correlation coefficient is less than the preset threshold, it means that the consistency between the n evaluation results and the n test results is too low, and they can be considered inconsistent. The computer equipment determines that the dataset quality assessment model is not effective.

[0080] Optionally, the present invention can further classify the effectiveness of the dataset quality assessment model into multiple levels based on the range of correlation coefficient values. For example, if the correlation coefficient is greater than or equal to 0 and less than 0.5, the dataset quality assessment model is not effective; if the correlation coefficient is greater than or equal to 0.5 and less than 0.7, the dataset quality assessment model is effective, but with low effectiveness; if the correlation coefficient is greater than or equal to 0.7 and less than 0.9, the dataset quality assessment model is effective, but with moderate effectiveness; and if the correlation coefficient is greater than or equal to 0.9 and less than or equal to 1, the dataset quality assessment model is effective, but with high effectiveness.

[0081] In summary, this invention provides an automated method for testing and verifying the effectiveness of a dataset quality assessment model. On one hand, the invention processes n datasets using the dataset quality assessment model to obtain n assessment results; on the other hand, it trains n AI models on each of the n datasets, and then processes the test datasets using each of the trained AI models to obtain n test results. The n assessment results indicate the dataset quality assessment model's evaluation of the quality differences among the n datasets, while the n test results indirectly and accurately indicate these quality differences. Using the n test results as a reference, the consistency between the n assessment results and the n test results is compared and analyzed to determine the effectiveness of the dataset quality assessment model in evaluating the datasets. This automated testing and verification of the dataset quality assessment model's effectiveness before its application avoids inaccuracies or errors in the assessment results due to inherent problems within the model itself, thus contributing to improved accuracy in dataset quality assessment.

[0082] Regarding the technical problem of comparing the consistency between evaluation results and test results to determine the effectiveness of the dataset quality assessment model, the conventional approach in this field is to sort the n evaluation results and n test results separately according to their overall values, and compare whether the evaluation results and test results in the same order correspond to the same dataset to determine whether the n evaluation results and n test results are consistent. However, this conventional approach is not perfect or accurate enough for comparing and analyzing the consistency between evaluation results and test results, and can only determine whether the dataset quality assessment model is effective. In this invention, on the one hand, when obtaining the n evaluation results, intermediate data calculated by the dataset quality assessment model during the evaluation process is incorporated to obtain the most comprehensive and rich evaluation data for consistency comparison analysis, thus fully and accurately verifying the effectiveness of the dataset quality assessment model; on the other hand, when comparing and analyzing the consistency between evaluation results and test results, the correlation coefficient between the n evaluation results and n test results is calculated. The correlation coefficient indicates the degree of effectiveness of the dataset quality assessment model, achieving a more granular verification of effectiveness.

[0083] Furthermore, this invention employs two methods: firstly, it modifies the training dataset using various data processing techniques; secondly, it applies the same data processing techniques to the training dataset with varying degrees of modification, resulting in multiple derived datasets to construct n datasets. Consequently, these n datasets not only contain different types of data errors but also data errors of the same type but with varying intensities, resulting in a richer diversity of quality differences among the n datasets. The effectiveness of the dataset quality assessment model is then tested using these n datasets, analyzing whether the model can accurately assess quality differences across different dimensions, thus achieving a more comprehensive and accurate effectiveness test for the dataset quality assessment model.

[0084] Exemplary device

[0085] Figure 3 This is an exemplary embodiment illustrating a validity testing apparatus for a dataset quality assessment model, with reference to... Figure 3 The device includes: a dataset partitioning module 310, a data transformation module 320, a quality assessment module 330, a model training module 340, a model testing module 350, and an effectiveness assessment module 360.

[0086] The dataset partitioning module 310 is used to partition the reference dataset into a training dataset and a test dataset, wherein the data type of the reference dataset is consistent with the data type of the evaluation object of the dataset quality evaluation model to be tested for effectiveness.

[0087] The data transformation module 320 is used to perform data transformation processing on the training dataset to obtain n-1 derived datasets; the training dataset and the n-1 derived datasets together constitute n datasets;

[0088] The quality assessment module 330 is used to assess the quality of the n datasets respectively using the dataset quality assessment model, and obtain n assessment results;

[0089] The model training module 340 is used to train n AI models using the n datasets respectively, so as to obtain n AI models that have been trained.

[0090] The model testing module 350 is used to process the test dataset using the n trained AI models respectively to obtain n test results;

[0091] The validity assessment module 360 ​​is used to determine the validity of the dataset quality assessment model based on the n test results and the n assessment results.

[0092] In one embodiment of this disclosure, the data change module 320 is further configured to: acquire the data content of the training dataset; determine at least one method of data change processing based on the data content; and perform data change processing on the training dataset with different degrees of processing according to the target method of the at least one method to obtain multiple derived datasets.

[0093] In one embodiment of this disclosure, based on the data content of an image classification dataset, at least one method of data change processing is determined, including: adding noise to an image, deleting image-level labels, tampering with image-level labels, adding multiple image-level labels to an image, increasing the repetition rate of a specific image, adding conflicting labels to copies of the same image, and / or deleting some images under a specific category, resulting in category imbalance; based on the data content of a text classification dataset, at least one method of data change processing is determined, including: embedding typos in the text, shuffling the text sequence, deleting classification labels, tampering with classification labels, adding multiple classification labels to the text, increasing the repetition rate of a specific text, adding conflicting labels to copies of the same text, and / or deleting some text under a specific category, resulting in category imbalance; based on the data content of an audio classification dataset, at least one method of data change processing is determined, including: adding noise to an image, deleting image-level labels, tampering with image-level labels, adding multiple image-level labels to an image ... The data modification methods determined based on the data content of the video segmentation dataset include: adding background noise, deleting audio segments, deleting classification labels, tampering with classification labels, adding multiple classification labels to audio, increasing the repetition rate of specific audio, adding conflicting labels to copies of the same audio, and / or deleting some audio under a specific category, resulting in class imbalance; adding incorrect segmentation markers to the original video, deleting video segments, deleting shot segmentation location labels, tampering with shot segmentation location labels, adding multiple shot segmentation location labels to the original video, increasing the repetition rate of a specific original video, and / or adding conflicting labels to copies of the same original video; adding mismatched text-image pairs, deleting images or text in text-image pairs, scrambling the matching relationship of text-image pairs, and / or increasing the repetition rate of specific text-image pairs.

[0094] In one embodiment of this disclosure, the n AI models are AI models with the same network structure and initial parameters, and the n AI models are matched with the data type and application task of the training dataset.

[0095] In one embodiment of this disclosure, the dataset quality assessment model includes: a target layer, a feature layer, and an indicator layer; the quality assessment module 330 is further configured to: obtain the dataset quality conclusion calculated by the target layer, wherein the dataset quality conclusion is a score value or evaluation level; obtain quality characteristics calculated by the feature layer, including at least one dimension for indicating dataset quality, wherein the quality characteristics include: completeness, self-consistency, confidentiality, accuracy, normalization, unbiasedness, diversity, and / or timeliness; obtain the evaluation indicators for each quality characteristic calculated by the indicator layer; wherein the evaluation indicators for completeness include: data element completeness, data record completeness, metadata completeness, null value rate, data category completeness, and / or data quantity completeness; the evaluation indicators for accuracy include: logical conflict measurement and / or annotation measurement; The evaluation indicators for confidentiality include: raw data metrics, labeled data metrics, data sharing metrics, data analysis metrics, data usage metrics, and / or data obsolescence metrics; the evaluation indicators for accuracy include: data duplication rate and / or outlier metrics; the evaluation indicators for standardization include: data standard metrics, authoritative reference data metrics, business rule metrics, security standard metrics, standardization metrics, and / or data format compliance; the evaluation indicators for unbiasedness include: distributional unbiasedness and / or historical unbiasedness; the evaluation indicators for diversity include: data source diversity and / or data specification diversity; the evaluation indicators for timeliness include: time period accuracy, timeliness, and / or temporal sequence; the evaluation result for the corresponding dataset is obtained by combining the dataset quality conclusions, the quality characteristics, and one or more of the evaluation indicators.

[0096] In one embodiment of this disclosure, the validity assessment module 360 ​​is further configured to: calculate the correlation coefficient between the n assessment results and the n test results; if the correlation coefficient is greater than a preset threshold, the n assessment results are consistent with the n test results, and the dataset quality assessment model is determined to be effective in assessing the quality of the dataset; if the correlation coefficient is less than the preset threshold, the n assessment results are inconsistent with the n test results, and the dataset quality assessment model is determined to be ineffective in assessing the quality of the dataset.

[0097] In one embodiment of this disclosure, the validity evaluation module 360 ​​is further configured to: convert the n test results into an n*m matrix, where m represents the number of samples included in the test dataset, and each row of the n*m ​​matrix represents a test result; convert the n evaluation results into an n*k matrix, where k represents the number of data dimensions calculated by the dataset quality evaluation model during the evaluation process, each row of the n*k matrix represents an evaluation result, and the same row in the n*m ​​matrix and the n*k matrix corresponds to the same dataset in the n datasets; calculate the weighted rank R1 of each row element in the n*m ​​matrix; calculate the weighted rank R2 of each row element in the n*k matrix; and calculate the value of the Sporman rank correlation coefficient between the weighted rank R1 and the weighted rank R2.

[0098] The exemplary device is an embodiment of the device corresponding to the exemplary method described above. The specific operation of each module can be understood with reference to the description of the method embodiment, and will not be repeated here.

[0099] Exemplary electronic devices

[0100] Figure 4 This is a block diagram illustrating a computer device 400 according to an exemplary embodiment. The computer device 400 may be a terminal, a laptop computer, a desktop computer, a server, a computer cluster, or other types of electronic equipment.

[0101] Reference Figure 4 The computer device 400 may include at least one processor 410 and a memory 420. The processor 410 can execute instructions stored in the memory 420. The processor 410 is communicatively connected to the memory 420 via a data bus. In addition to the memory 420, the processor 410 may also be communicatively connected to an input device 430, an output device 440, and a communication device 450 via the data bus.

[0102] Processor 410 can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0103] The memory 420 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0104] In this embodiment of the present disclosure, the memory 420 stores executable instructions, and the processor 410 can read the executable instructions from the memory 420 and execute the instructions to implement all or part of the steps of the validity testing method for the dataset quality assessment model in the exemplary embodiment described above.

[0105] Exemplary computer-readable storage media

[0106] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0107] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0108] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0109] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for testing the effectiveness of a dataset quality assessment model, characterized in that, The method includes: The training dataset and the test dataset are divided from the reference dataset, and the data type of the reference dataset is consistent with the data type of the evaluation object of the dataset quality assessment model to be tested for effectiveness. The training dataset is subjected to data transformation processing to obtain n-1 derived datasets; the training dataset and the n-1 derived datasets together constitute n datasets; The quality of the n datasets is evaluated using the dataset quality assessment model, resulting in n evaluation results. n artificial intelligence (AI) models are trained using the n datasets respectively, resulting in n AI models that have been trained. The test dataset is processed by the n trained AI models to obtain n test results; Based on the n test results and the n evaluation results, the effectiveness of the dataset quality assessment model is determined; The step of determining the effectiveness of the dataset quality assessment model based on the n test results and the n evaluation results includes: Calculate the correlation coefficient between the n evaluation results and the n test results; If the value of the correlation coefficient is greater than the preset threshold, then the n evaluation results are consistent with the n test results, and it is determined that the dataset quality assessment model is effective in assessing the quality of the dataset. If the value of the correlation coefficient is less than the preset threshold, then the n evaluation results are inconsistent with the n test results, and it is determined that the dataset quality assessment model is invalid for the quality assessment of the dataset. The calculation of the correlation coefficient between the n evaluation results and the n test results includes the following: Transform the n test results into n The m-matrix represents the number of samples included in the test dataset, and the n-matrix represents the number of samples included in the test dataset. Each row of the m-matrix represents a test result; Transform the n evaluation results into n The k matrix represents the number of data dimensions calculated by the dataset quality assessment model during the assessment process, and the n... Each row of the k matrix represents an evaluation result, and the n The m matrix and the n In matrix k, the same row corresponds to the same dataset in n datasets; Calculate the n The weighted rank of each row element in matrix m ; Calculate the n The weighted rank of each row element in the k matrix ; Calculate the weighted rank With the weighted rank The values ​​of the Spoman rank correlation coefficient between them; The datasets include image classification datasets, text classification datasets, audio classification datasets, video segmentation datasets, or text-to-image datasets.

2. The method according to claim 1, characterized in that, The training dataset is subjected to data transformation processing to obtain n-1 derived datasets, including: Obtain the data content of the training dataset; Based on the data content, determine at least one method for processing data changes; For the target method in at least one of the methods, the training dataset is subjected to various data transformation processes with different degrees of processing according to the target method to obtain multiple derivative datasets.

3. The method according to claim 2, characterized in that, Based on the data content of the image classification dataset, at least one of the determined methods for data change processing includes: adding noise to the image, deleting image-level labels, tampering with image-level labels, adding multiple image-level labels to the image, increasing the repetition rate of a specific image, adding conflicting labels to copies of the same image, and / or deleting some images under a specific category, resulting in class imbalance. Based on the data content of the text classification dataset, at least one of the determined methods for data change processing includes: embedding typos in the text, shuffling the text sequence, deleting classification labels, tampering with classification labels, adding multiple classification labels to the text, increasing the repetition rate of a specific text, adding conflicting labels to copies of the same text, and / or deleting parts of the text under a specific category, resulting in class imbalance. Based on the data content of the audio classification dataset, at least one of the determined data change processing methods includes: adding background noise to the audio, deleting audio segments, deleting classification labels, tampering with classification labels, adding multiple classification labels to the audio, increasing the repetition rate of a specific audio, adding conflicting labels to copies of the same audio, and / or deleting some audio in a specific category, resulting in class imbalance. Based on the data content of the video segmentation dataset, at least one of the determined methods for data change processing includes: adding incorrect segmentation tags to the original video, deleting video segments, deleting shot segmentation location labels, tampering with shot segmentation location labels, adding multiple shot segmentation location labels to the original video, increasing the repetition rate of a specific original video, and / or adding conflicting labels to copies of the same original video. Based on the data content of the text-to-image dataset, at least one of the determined methods for data change processing includes: adding mismatched text-image pairs, deleting text or image from text-image pairs, shuffling the matching relationship of text-image pairs, and / or increasing the repetition rate of specific text-image pairs.

4. The method according to claim 1, characterized in that, The n AI models are AI models with the same network structure and initial parameters, and the data type and application task of the training dataset are matched with the n AI models.

5. The method according to claim 1, characterized in that, The dataset quality assessment model includes: a target layer, a feature layer, and an indicator layer; The quality of the n datasets is evaluated using the dataset quality assessment model to obtain n evaluation results, including: Obtain the dataset quality conclusion calculated by the target layer, wherein the dataset quality conclusion is a score value or evaluation level; Obtain at least one dimension of quality characteristics calculated by the feature layer to indicate the quality of the dataset, the quality characteristics including: completeness, self-consistency, confidentiality, accuracy, normalization, unbiasedness, diversity and / or timeliness; Obtain the evaluation indices for each quality characteristic calculated by the index layer; wherein, The integrity assessment metrics include: data element integrity, data record integrity, metadata integrity, null value rate, data category integrity, and / or data quantity integrity. The accuracy evaluation metrics include: logical conflict metrics and / or annotation metrics; The confidentiality assessment metrics include: raw data metrics, labeled data metrics, data sharing metrics, data analysis metrics, data usage metrics, and / or data obsolescence metrics; The accuracy assessment metrics include: data duplication rate and / or outlier measurement; The normative evaluation indicators include: data standard metrics, authoritative reference data metrics, business rule metrics, security specification metrics, standardization metrics, and / or data format compliance. The evaluation metrics for unbiasedness include: distributional unbiasedness and / or historical unbiasedness; The evaluation metrics for diversity include: diversity of data sources and / or diversity of data specifications; The current timeliness assessment indicators include: accuracy of time period, timeliness and / or chronology; The evaluation result of the corresponding dataset is obtained by combining one or more of the dataset quality conclusions, the quality characteristics, and the evaluation indicators.

6. A validity testing device for a dataset quality assessment model, characterized in that, The device includes: The dataset partitioning module is used to partition the reference dataset into a training dataset and a test dataset. The data type of the reference dataset is consistent with the data type of the evaluation object of the dataset quality assessment model to be tested for effectiveness. The data transformation module is used to perform data transformation processing on the training dataset to obtain n-1 derived datasets; the training dataset and the n-1 derived datasets together constitute n datasets; The quality assessment module is used to assess the quality of the n datasets respectively using the dataset quality assessment model, and obtain n assessment results; The model training module is used to train n AI models using the n datasets respectively, so as to obtain n AI models that have been trained. The model testing module is used to process the test dataset using the n trained AI models respectively to obtain n test results; The validity assessment module is used to determine the validity of the dataset quality assessment model based on the n test results and the n assessment results; Specifically, the effectiveness evaluation module is used for: Calculate the correlation coefficient between the n evaluation results and the n test results; If the value of the correlation coefficient is greater than the preset threshold, then the n evaluation results are consistent with the n test results, and it is determined that the dataset quality assessment model is effective in assessing the quality of the dataset. If the value of the correlation coefficient is less than the preset threshold, then the n evaluation results are inconsistent with the n test results, and it is determined that the dataset quality assessment model is invalid for the quality assessment of the dataset. The effectiveness evaluation module calculates the correlation coefficient between the n evaluation results and the n test results in the following way: Transform the n test results into n The m-matrix represents the number of samples included in the test dataset, and the n-matrix represents the number of samples included in the test dataset. Each row of the m-matrix represents a test result; Transform the n evaluation results into n The k matrix represents the number of data dimensions calculated by the dataset quality assessment model during the assessment process, and the n... Each row of the k matrix represents an evaluation result, and the n The m matrix and the n In matrix k, the same row corresponds to the same dataset in n datasets; Calculate the n The weighted rank of each row element in matrix m ; Calculate the n The weighted rank of each row element in the k matrix ; Calculate the weighted rank With the weighted rank The values ​​of the Spoman rank correlation coefficient between them; The datasets include image classification datasets, text classification datasets, audio classification datasets, video segmentation datasets, or text-to-image datasets.

7. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the validity testing method of the dataset quality assessment model as described in any one of claims 1-5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed, they implement the validity testing method for the dataset quality assessment model according to any one of claims 1 to 5.

Images