Data set quality evaluation test method oriented to artificial intelligence

Abstract

The invention discloses an artificial intelligence-oriented data set quality evaluation test method, which relates to the technical field of artificial intelligence, and comprises the following steps: S1, preparing a test data set; s2, initializing test conditions; s3, pre-detection is carried out; s4, carrying out post detection; and S5, calculating a data set quality evaluation score. According to the artificial intelligence-oriented data set quality evaluation test method, targeted innovative design, adjustment and expansion are carried out on artificial intelligence data set quality evaluation indexes, methods and technical processes according to unique requirements of artificial intelligence data set quality evaluation methods and technologies; it is ensured that the method not only can fit the specific stage and scene of AI model training and performance evaluation, but also can promote the iterative progress of artificial intelligence related technologies and the improvement of practical application efficiency, and high scene suitability and technical perspectiveness are embodied.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-oriented data set quality assessment test method. Background Art

[0002] In the era of big models, high-quality training data sets are essential for building accurate, efficient, and reliable models. High-quality data sets can not only improve the generalization ability of the model, but also reduce the risk of overfitting, making the AI ​​system more robust. However, the data collected in practice often have many problems, such as missing data, outliers, label errors, data imbalance, noise interference, etc. These problems directly affect the performance and generalization ability of the model. How to comprehensively, objectively, detailedly, and quantitatively evaluate the quality performance of artificial intelligence data sets is very important.

[0003] Therefore, there is an urgent need to improve this shortcoming. The present invention studies and improves the existing technology and its shortcomings, and provides a data set quality assessment test method for artificial intelligence. Summary of the invention

[0004] The purpose of the present invention is to provide an artificial intelligence-oriented data set quality assessment test method to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a data set quality assessment test method for artificial intelligence, comprising the following steps: S1. Test data set preparation: Use standardized, diversified, and integrated data generation technology to achieve standardized production of data sets required for large model training; S2. Test condition initialization: Based on the basic situation of the test data set and the test objectives, screen and determine the quality evaluation indicators required for the actual test, and design the weight structure of each core indicator to quantify the final quality score of the test data set; Since different industry data sets have different focuses on the evaluation dimensions, it is necessary to design and determine the corresponding indicator weights according to the actual business scenario requirements, which specifically includes two major components: importing the basic situation of the test data set and designing the indicator weights of the test data set; S3. Pre-test: Based on the established dataset knowledge and rule management system, use the AI ​​dataset quality test rules or empirical formulas to conduct preliminary tests on the quality of the AI ​​dataset, which specifically includes two parts: rule-based autonomous detection and manual-assisted detection; S4. Post-test: Use different technical means to detect the specific role of high-quality AI datasets in the performance of large model training, and use model performance feedback to quantify the specific role of different evaluation dimensions of AI datasets in improving the performance of large models, mainly including model benchmark feedback and dataset ablation testing; S5. Calculation of data set quality assessment score: Based on the quality assessment technical process of the first four steps, the specific calculation of the final AI data set quality assessment score is as follows: In the formula, Represents the final total score, Represents the weight of the pre-test evaluation score in the total score, Represents the specific score of the i-th indicator in the rule-based autonomous detection of the pre-detection, The representative evaluation service object autonomously detects the specific weight of the i-th indicator according to the rules set in the business scenario. Represents the specific index score of the jth index in the manual assisted detection of the previous detection, Represents the specific weight of the jth indicator of manual assisted detection set by the evaluation service object according to the business scenario, Represents the weight of the post-detection evaluation method score in the total score, Represents the specific indicator score of the kth indicator in the model performance feedback test of the post-detection, Represents the specific weight of the kth indicator detected by the evaluation service object based on the model performance feedback set in the business scenario. Represents the specific indicator score of the index in the data ablation test of the post-detection, Represents the specific weight of the th indicator of the data set ablation test set by the evaluation service object according to the business scenario. Represents the total number of rule-based autonomous detection indicators for pre-detection. Represents the total number of manual assisted detection indicators for pre-detection, Represents the total number of model performance feedback test indicators for post-testing, Represents the total number of ablation test metrics for the dataset for post-detection.

[0006] Furthermore, in step S1, the specific implementation method of test data set preparation includes two parts: a high-quality data set preparation technical link based on intelligent data sampling technology and a high-quality data set preparation technical link based on synthetic data technology. The generation and preparation of high-quality data sets are achieved through the technical integration of the two methods.

[0007] Furthermore, the high-quality data preparation link based on intelligent data collection technology specifically targets the existing data in the current physical space, and establishes a set of standardized intelligent data collection technology processes to achieve high-quality preparation of test data sets, including four core technical links: automated collection, deep cleaning, intelligent labeling, and fusion quality inspection. Among them, the implementation methods of automated collection include Web crawlers, API interfaces, Internet of Things access, database synchronization, and multimodal data collection. The implementation methods of deep cleaning include Apache NiFi ETL technology, automatic cleaning shell scripts, machine learning cleaning models, Google Cloud, and Dataflow cloud service technology. The implementation methods of intelligent labeling include MFT multimodal fusion labeling, cross-modal data alignment, transfer learning labeling, and human-computer collaborative interaction labeling. The implementation methods of fusion quality inspection include K-Means clustering, SVM intelligent classification, density anomaly LOF algorithm, SOS probability anomaly detection, PCA linear dimensionality reduction detection, and AutoEncoder nonlinear detection.

[0008] Furthermore, the high-quality data preparation link based on synthetic data technology specifically generates a test data set that meets the requirements of large model training through data synthesis technical methods, which specifically includes two technical links: multi-mode data synthesis and synthetic data quality inspection; among them, the multi-mode data synthesis method includes the LLM-P-Cot large model data synthesis technical method based on LLMs large model prompt engineering and Cot intermediate thinking chain reasoning steps, the BigGAN-StyleGAN fusion algorithm model based on adversarial neural network, and the distribution function generation method based on normal distribution and exponential distribution. The synthetic data quality inspection specifically includes the five major technical indicators of artificial intelligence data set fidelity, privacy, coverage, interpretability, and availability.

[0009] Furthermore, in step S2, the basic information of the test data set is imported to provide a complete and detailed description of the basic information of the data set for the test data set to be tested, which is the basis for the selection of dimensions and weight design of the subsequent test data set quality assessment, and the basic information required for the test data set includes the following aspects: 1) Field structure of the data set: clarify the specific field composition of the test data set and the specific meaning of each field; 2) Dataset size: Specify the specific data size and number of data items in the test dataset. Generally, the number of data items in the test dataset must be greater than 20,000. 3) Modality type of the dataset: clarify the specific modality and type of the test dataset, including text, voice, image, video, audio, multi-modality, etc.; 4) Task types supported by the dataset: clarify the types of training tasks that the test dataset needs to support in the future, such as unsupervised learning, supervised learning, reinforcement learning, etc. 5) Dataset application scenario classification system: For the quality assessment of high-quality industry data sets, clarify the key industry fields and business scenario systems of the data sets. The key industry fields include science, manufacturing, agriculture, energy, transportation, finance, medical care, education, consumption, Internet governance, human resources, public security and other key industry fields. The industry business scenario classification system refers to the actual business scenario classification structure system in different industry fields; 6) Training stages supported by the dataset: pre-training stage, fine-tuning stage, feedback reinforcement stage, evaluation and verification stage, reasoning stage, etc.; 7) Dataset security technical requirements: clarify the specific requirements for test confidentiality and security of the test dataset, and specify those with special security and confidentiality requirements; 8) Description of other issues: If there are other special requirements, additional explanation is required.

[0010] Furthermore, in step S2, the weight design of the test data set indicators is different according to the subject scope and professional level. The specific definition of the artificial intelligence data set is divided into three categories: general data set, industry general data set and industry specialized data set. The three categories of data sets have different evaluation requirements for the specific 12 types of first-level evaluation indicators and 36 types of second-level evaluation indicators; The first-level evaluation indicators include completeness, standardization, accuracy, timeliness, consistency, density, diversity, balance, relevance, originality, traceability, and accessibility, and the second-level indicators included in each first-level evaluation indicator are as follows: Secondary indicators of completeness include: scale completeness, structural completeness, mission completeness, and domain completeness; The secondary indicators of normativity include: formal normativity, privacy normativity, and security normativity; Secondary indicators of accuracy include: content authenticity and field expertise; The secondary indicators of timeliness include: generation timeliness, collection timeliness, processing timeliness, and release and update timeliness; Secondary indicators of consistency include: label consistency, concept consistency; Secondary indicators of density include: sample uniqueness and content information; The secondary indicators of diversity include: feature diversity, type diversity, source diversity, task diversity, domain diversity, and cultural diversity; The secondary indicators of balance include: category balance, source balance; Secondary indicators of relevance include: logical relevance, task relevance, and domain relevance; The secondary indicators of originality include: source originality, content originality; The secondary indicators of traceability include: source traceability, link traceability, and version control traceability; The secondary indicators of accessibility include: ease of acquisition, ease of use, and technical support.

[0011] Furthermore, in step S3, the technical process of autonomous rule detection is: a specific automated operation algorithm, program or model is written through a deterministic rule system or empirical formula for artificial intelligence data set quality assessment to automatically check the quality of the data set, screen out data sets with too low quality in advance, and reduce the cost for subsequent sampling detection and model verification, specifically including: S31. Autonomous identification of dataset knowledge information It aims to efficiently identify and analyze the detailed knowledge and information content of the data set and improve the utilization rate of the data set for the model. It specifically includes two major components: one is shallow information recognition, which is responsible for defining and organizing the classification standards of the data set, including using keyword search, text encoding and decoding, optical character recognition and other technologies to identify data set modality, industry and application scenario label information; the other is deep information extraction, using statistical and machine learning techniques to deeply explore and analyze the inherent patterns and laws in the data set, including but not limited to the correlation between features, abnormal patterns, data distribution characteristics and time series characteristics; S32. Dataset quality rule detection It aims to realize the automated rule detection of basic quality and quality assessment indicators of data sets, including four major components: missing value detection, duplicate value detection, data set format detection, and automated rule detection of data set quality assessment indicators. Specific technical implementation methods include rule-based detection scripts, data quality tools, ETL tools, built-in functions of database management systems, API calls, and machine learning generation models.

[0012] Furthermore, in step S3, manual assisted detection is performed on the data set that has initially met all the standards in the rule-based autonomous detection. With the help of the professional knowledge and industry experience of exclusive industry experts, some errors or anomalies in the data set are manually assisted to identify omitted complex problems that cannot be covered by the rule-based autonomous detection method. Specific detection methods include simple random sampling, priority sampling, mixed sampling, stratified sampling, systematic sampling, probability proportionate sampling (PPS), convenience sampling, judgment sampling and other methods.

[0013] Furthermore, in step S4, model performance feedback: select a suitable model according to the business scenario and task requirements, and send the evaluated data set into the model for training, and select appropriate model performance verification indicators to verify the model performance. Through the core performance indicators of large model training, the fluctuation of model performance is discovered in time, and the specific quality performance of the evaluated data set is determined by judging the performance of the model on the evaluation indicators; the training performance indicators of the large model specifically include three parts: model training accuracy performance (precision, recall rate, accuracy, F1 Score, root mean square error MSE), model training speed performance (training time, number of iterations, model throughput, model delay, etc.), model training resource consumption (computing resource utilization, memory occupancy, energy consumption per unit time, etc.).

[0014] Furthermore, in step S4, the data set ablation test is performed: multiple different data sets are obtained by changing one or more evaluation dimensions of the original data set, and the same model is trained with different data sets obtained by changing these indicators, and the changes in the model benchmark testing capabilities are tested and verified. The data set ablation test can be used to evaluate and verify the specific contribution of one or more indicator combinations to the model performance; the data set ablation test technical means specifically include data set subset experiments (removing samples of specific categories, reducing the amount of data, disabling certain features, changing data distribution), data set feature screening analysis, etc.

[0015] The present invention provides a data set quality assessment test method for artificial intelligence, which has the following beneficial effects: In response to the unique needs of artificial intelligence dataset quality assessment methods and technologies, the present invention has carried out targeted innovative design, adjustment and expansion of artificial intelligence dataset quality assessment indicators, methods and technical processes, ensuring that they can not only fit the specific stages and scenarios of AI model training and performance evaluation, but also promote the iterative progress of artificial intelligence-related technologies and the improvement of practical application efficiency, reflecting a high degree of scenario suitability and technical foresight.

[0016] The present invention can systematically check the multi-dimensional key attributes of artificial intelligence datasets, such as integrity, accuracy, consistency, timeliness and diversity, to ensure that the datasets can comprehensively cover the target application scenarios of artificial intelligence large models. By establishing a detailed dataset quality assessment quantitative indicator system and a standardized testing process, it is possible to implement multi-dimensional quality indicator testing and quantitative scoring of high-quality artificial intelligence datasets, complete multi-level analysis of datasets, grasp the deep knowledge value of datasets, ensure high-quality supply of datasets, and form a data closed loop throughout the entire life cycle of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A logical block diagram of a data set quality assessment test method for artificial intelligence of the present invention; Figure 2 A schematic diagram of a test data set preparation technology implementation flow of an artificial intelligence-oriented data set quality assessment test method of the present invention; Figure 3 A schematic diagram of a high-quality data preparation technology link based on intelligent data standard collection technology for an artificial intelligence-oriented data set quality assessment test method of the present invention; Figure 4 A schematic diagram of a high-quality data preparation technology link based on synthetic data technology of a data set quality assessment test method for artificial intelligence of the present invention; Figure 5 A schematic diagram of an artificial intelligence data set quality assessment test index and weight design for an artificial intelligence-oriented data set quality assessment test method of the present invention; Figure 6 A table of specific evaluation requirements classification and weight design for different quality evaluation indicators of an artificial intelligence-oriented data set quality evaluation test method of the present invention; Figure 7 This is a table showing the importance of different quality assessment indicators for different categories of data in an artificial intelligence-oriented data set quality assessment test method of the present invention; Figure 8 A schematic diagram of a pre-detection technology flow chart of an artificial intelligence-oriented data set quality assessment test method of the present invention; Fig. 9 The present invention is a schematic diagram of the implementation process of the post-detection technology of an artificial intelligence-oriented data set quality assessment test method. DETAILED DESCRIPTION

[0018] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0019] like Figure 1-Figure 9As shown, a data set quality assessment test method for artificial intelligence includes the following steps: S1. Test data set preparation: Standardized production of data sets required for large model training is achieved by using standardized, diversified, and integrated data generation technologies. The specific implementation of test data set preparation includes two parts: the technical link of high-quality data set preparation based on intelligent data standard collection technology and the technical link of high-quality data set preparation based on synthetic data technology. The generation and preparation of high-quality data sets are achieved through the technical integration of the two methods. The overall technical implementation principle and process are as follows: Figure 2 As shown; In this embodiment, the high-quality data preparation link based on intelligent data standard collection technology specifically targets the existing data in the current physical space, and establishes a set of standardized intelligent data standard collection technology processes to achieve high-quality preparation of test data sets, which specifically includes four core technical links: automated collection, deep cleaning, intelligent labeling, and fusion quality inspection. The specific technical implementation methods of each part are as follows: Figure 3 As shown; Among them, the implementation methods of automated collection include Web crawlers, API interfaces, IoT access, database synchronization, and multimodal data collection; the implementation methods of deep cleaning include Apache NiFi ETL technology, automatic cleaning shell scripts, machine learning cleaning models, Google Cloud, and Dataflow cloud service technology; the implementation methods of intelligent labeling include MFT multimodal fusion labeling, cross-modal data alignment, transfer learning labeling, and human-computer collaborative interaction labeling; the implementation methods of fusion quality inspection include K-Means clustering, SVM intelligent classification, density anomaly LOF algorithm, SOS probability anomaly detection, PCA linear dimensionality reduction detection, and AutoEncoder nonlinear detection; In this embodiment, the high-quality data preparation link based on synthetic data technology specifically generates a test data set that meets the requirements of large model training through data synthesis technology methods, which specifically includes two technical links: multi-mode data synthesis and synthetic data quality inspection; among them, the multi-mode data synthesis method includes the LLM-P-Cot large model data synthesis technology method based on LLMs large model prompt engineering and Cot intermediate thinking chain reasoning steps, the BigGAN-StyleGAN fusion algorithm model based on adversarial neural network, and the distribution function generation method based on normal distribution and exponential distribution. The synthetic data quality inspection specifically includes the fidelity, privacy, coverage, interpretability, and availability of artificial intelligence data sets. The five major technical indicators of detection, the specific technical implementation path Figure 4 As shown; S2. Test condition initialization: Based on the basic situation of the test data set and the test objectives, screen and determine the quality evaluation indicators required for the actual test, and design the weight structure of each core indicator to quantify the final quality score of the test data set; since different industry data sets have different focuses on the evaluation dimensions, it is necessary to design and determine the corresponding indicator weights according to the actual business scenario requirements. Specifically, it includes two major components: importing the basic situation of the test data set and designing the indicator weights of the test data set. The specific technical principle flow chart is as follows Figure 5 As shown; In this embodiment, the import of the basic information of the test data set is intended to provide a complete and detailed description of the basic information of the data set for the test. It is the basis for the selection of dimensions and weight design for the subsequent test data set quality assessment. The basic information required for the specific test data set includes the following aspects: 1) Field structure of the data set: clarify the specific field composition of the test data set and the specific meaning of each field; 2) Dataset size: Specify the specific data size and number of data items in the test dataset. Generally, the number of data items in the test dataset must be greater than 20,000. 3) Modality type of the dataset: clarify the specific modality and type of the test dataset, including text, voice, image, video, audio, multi-modality, etc.; 4) Task types supported by the dataset: clarify the types of training tasks that the test dataset needs to support in the future, such as unsupervised learning, supervised learning, reinforcement learning, etc. 5) Dataset application scenario classification system: For the quality assessment of high-quality industry data sets, clarify the key industry fields and business scenario systems of the data sets. The key industry fields include science, manufacturing, agriculture, energy, transportation, finance, medical care, education, consumption, Internet governance, human resources, public security and other key industry fields. The industry business scenario classification system refers to the actual business scenario classification structure system in different industry fields; 6) Training stages supported by the dataset: pre-training stage, fine-tuning stage, feedback reinforcement stage, evaluation and verification stage, reasoning stage, etc.; 7) Dataset security technical requirements: clarify the specific requirements for test confidentiality and security of the test dataset, and specify those with special security and confidentiality requirements; 8) Other issues: If there are other special requirements, additional explanation is required; In this embodiment, the weight design of the test data set indicators is based on the different subject scopes and professional levels. The specific definition of the artificial intelligence data set is divided into three categories: general knowledge data set, industry general knowledge data set, and industry specialized knowledge data set. The three categories of data sets have different evaluation requirements for the specific 12 types of first-level evaluation indicators and 36 types of second-level evaluation indicators. The specific evaluation requirements classification and weight design are as follows: Figure 6 As shown, Figure 7 Specifically shows the importance of different types of data sets to 12 specific first-level evaluation indicators and 36 second-level evaluation indicators; The first-level evaluation indicators include completeness, standardization, accuracy, timeliness, consistency, density, diversity, balance, relevance, originality, traceability, and accessibility. The second-level indicators included in each first-level evaluation indicator are as follows: The secondary indicators of completeness include: scale completeness (whether the amount of data in the dataset meets the minimum amount required for the application of the AI ​​model), structural completeness (whether the information recorded by the samples in the dataset is complete, with no missing values ​​or the missing values ​​are within a reasonable range), task completeness (the extent to which the data in the dataset can cover the number of AI tasks), and field completeness (the extent to which the data in the dataset can cover the number of industry fields); The secondary indicators of normativeness include: formal normativeness (the extent to which the data in the dataset conforms to certain data form standards, such as standards that need to be followed in naming, creation, definition, updating and archiving, including international standards, national standards, industry standards, industry practice experience or relevant regulations, etc.), privacy normativeness (the extent to which the data in the dataset conforms to privacy protection-related norms such as legal and regulatory requirements, industry standards, and internal corporate policies), and security normativeness (the data in the dataset conforms to the security requirements of AI model training to ensure the accuracy and reliability of the model); Secondary indicators of accuracy include: content authenticity (whether the information recorded in the data set is consistent with and true to the actual object, event or fact it represents), domain expertise (the applicability, professional depth and accuracy of the data in the data set in a specific field, and should be closely integrated with the standards, terminology, business logic and expertise of a specific industry); Secondary indicators of timeliness include: generation timeliness (the generation time of data in a dataset should be as close to the current moment as possible, especially in scenarios that require real-time analysis and response), collection timeliness (the frequency of data collection in a dataset determines the sensitivity of the dataset to changes. High-frequency collection can improve the timeliness of data), processing timeliness (the time required for the entire processing flow from data collection to data analysis and use in a dataset also affects the timeliness of data, including steps such as cleaning, conversion, and loading), and release and update timeliness (the frequency of dataset updates and when they are released to the public directly affect whether data users can obtain the latest information). Secondary indicators of consistency include: label consistency (for annotated datasets, whether different annotators maintain consistent annotations for the same or similar instances), conceptual consistency (the terms, classification standards, and definitions used in the data in the dataset are consistent across the dataset); The secondary indicators of density include: sample uniqueness (whether there are a large number of duplicate records in the sample in the data set, removing duplicates can improve information density), content information volume (detection of the amount of information covered by a unit sample in the data set); The secondary indicators of diversity include: feature diversity (the breadth and comprehensiveness of the types of features contained in the dataset, whether it covers different attributes and angles of describing entities), type diversity (the breadth and comprehensiveness of the types contained in the dataset, whether it covers data in different formats, including structured, semi-structured and unstructured data), source diversity (the breadth and comprehensiveness of the data sources or channels in the dataset to ensure comprehensive information, reduce bias, and increase the breadth of application scenarios), task diversity (the breadth and comprehensiveness of the dataset to support artificial intelligence tasks), field diversity (the breadth and comprehensiveness of the dataset to cover industry fields to promote cross-domain knowledge transfer and application), and cultural diversity (the breadth and comprehensiveness of the dataset to reflect different social groups, cultural backgrounds, viewpoints, and diverse values ​​of stakeholders); The secondary indicators of balance include: category balance (the balance of the number of data in each category in the data set, ensuring that the number of samples of each type of label in the data set is close to prevent the model from being biased towards the majority class), source balance (the balance of the number of data from each source in the data set, ensuring that the amount of data from different sources is appropriate, and improving the comprehensiveness and generalization ability of the model); Secondary indicators of relevance include: logical relevance (the degree of relevance between the data in the dataset and a specific research field or application field), task relevance (the degree of relevance between the data in the dataset and the AI ​​task to be completed (such as classification, regression, clustering, etc.)), and domain relevance (the data in the dataset contains context-independent content); The secondary indicators of originality include: source originality (the originality of the data collection source in the dataset. That is, the data is not simply copied or adapted from existing public datasets, but is obtained for the first time through special experimental design, sensor networks, customized surveys, unique observations, etc.), content originality (the originality of the data in the dataset. Even if the data comes from known channels, its content may be an unprecedented combination, a record in a specific situation, or a description of a unique phenomenon); The secondary indicators of traceability include: source traceability (whether the data in the dataset can trace its original source, including how the data was collected, by whom, when and where it was collected. This is crucial to verify the authenticity and legality of the data, especially in scientific research and legal compliance), link traceability (whether the data in the dataset can trace the entire transformation process from the original form to the final processed form, including the records of each step of the data cleaning, conversion, aggregation, analysis, etc. Understanding the evolution path of the data helps to quickly locate the problem when errors are found or retrospective analysis is required, and it also facilitates the reproduction of the research process), version control traceability (whether the dataset can track the version records of each version during its update and iteration process, including the change history, modification reasons and responsible persons. This is particularly critical for team collaboration, error recovery and auditing needs, and can ensure that data users can obtain and use the correct data version while maintaining the historical integrity of the data); Secondary indicators of accessibility include: ease of acquisition (whether the dataset has a clear license agreement, whether users can easily understand the license terms, and whether special permissions or application procedures are required to obtain the data), ease of use (whether the dataset is equipped with detailed documentation, including data dictionaries, metadata descriptions (data sources, collection methods, update frequency, etc.), user guides and sample codes, etc.), technical support (whether the dataset provides official or community technical support channels, and whether users can get timely and effective help when they encounter problems in the process of using the data); S3. Preliminary testing: Based on the established dataset knowledge and rule management system, use the AI ​​dataset quality testing rules or empirical formulas to conduct preliminary testing of the AI ​​dataset quality. This includes two parts: rule-based autonomous testing and manual-assisted testing. The specific technical process is as follows: Figure 8 As shown; In this embodiment, the technical process of rule autonomous detection is: a specific automated operation algorithm, program or model is written through a deterministic rule system or empirical formula for artificial intelligence data set quality assessment to automatically check the quality of the data set, screen out data sets with too low quality in advance, and reduce the cost for subsequent sampling detection and model verification, specifically including: S31. Autonomous identification of dataset knowledge information It aims to efficiently identify and analyze the detailed knowledge and information content of the data set and improve the utilization rate of the data set for the model. It specifically includes two major components: one is shallow information recognition, which is responsible for defining and organizing the classification standards of the data set, including using keyword search, text encoding and decoding, optical character recognition and other technologies to identify data set modality, industry and application scenario label information; the other is deep information extraction, using statistical and machine learning techniques to deeply explore and analyze the inherent patterns and laws in the data set, including but not limited to the correlation between features, abnormal patterns, data distribution characteristics and time series characteristics; S32. Dataset quality rule detection It aims to realize the automated rule detection of basic quality and quality assessment indicators of data sets, including four major components: missing value detection, duplicate value detection, data set format detection, and automated rule detection of data set quality assessment indicators. Specific technical implementation methods include rule-based detection scripts, data quality tools, ETL tools, built-in functions of database management systems, API calls, and machine learning generation models.

[0020] In this embodiment, manual assisted detection is aimed at the data set that has initially met the standards in the rule-based autonomous detection. With the help of the professional knowledge and industry experience of dedicated industry experts, some errors or anomalies in the data set are manually assisted to identify the omitted complex problems that cannot be covered by the rule-based autonomous detection method. Specific detection methods include simple random sampling, priority sampling, mixed sampling, stratified sampling, systematic sampling, probability proportional sampling (PPS), convenience sampling, judgment sampling and other methods; S4. Post-test: Use different technical means to detect the specific role of high-quality AI datasets in the performance of large model training, and quantify the specific role of different evaluation dimensions of AI datasets in improving the performance of large models through model performance feedback. This mainly includes model benchmark feedback and dataset ablation testing. The specific technical process is as follows: Fig. 9 As shown; In this embodiment, model performance feedback: select a suitable model according to the business scenario and task requirements, and send the evaluated data set into the model for training, and select appropriate model performance verification indicators to verify the model performance. Through the core performance indicators of large model training, the fluctuation of model performance is discovered in time, and the specific quality performance of the evaluated data set is determined by judging the performance of the model on the evaluation indicators; the training performance indicators of the large model specifically include three parts: model training accuracy performance (precision, recall rate, accuracy, F1 Score, root mean square error MSE), model training speed performance (training time, number of iterations, model throughput, model delay, etc.), model training resource consumption (computing resource utilization, memory usage, energy consumption per unit time, etc.); In this embodiment, the data set ablation test: by changing one or more evaluation dimensions of the original data set to obtain multiple different data sets, the same model is trained with different data sets obtained by changing these indicators, and the change of the model benchmark test capability is tested and verified. Through the data set ablation test, the specific contribution of one or more indicator combinations to the model performance can be evaluated and verified; the data set ablation test technical means specifically include data set subset experiments (removing samples of a specific category, reducing the amount of data, disabling certain features, changing data distribution), data set feature screening analysis, etc.; S5. Dataset quality assessment score calculation: Based on the quality assessment technical process of the first four steps, the specific calculation and public disclosure of the final AI data set quality assessment score is as follows: Represents the final total score, Represents the weight of the pre-test evaluation score in the total score, Represents the specific score of the i-th indicator in the rule-based autonomous detection of the pre-detection, The representative evaluation service object autonomously detects the specific weight of the i-th indicator according to the rules set in the business scenario. Represents the specific index score of the jth index in the manual assisted detection of the previous detection, Represents the specific weight of the jth indicator of manual assisted detection set by the evaluation service object according to the business scenario, Represents the weight of the post-detection evaluation method score in the total score, Represents the specific indicator score of the kth indicator in the model performance feedback test of the post-detection, Represents the specific weight of the kth indicator detected by the evaluation service object based on the model performance feedback set in the business scenario. Represents the specific indicator score of the index in the data ablation test of the post-detection, Represents the specific weight of the th indicator of the data set ablation test set by the evaluation service object according to the business scenario. Represents the total number of rule-based autonomous detection indicators for pre-detection. Represents the total number of manual assisted detection indicators for pre-detection, Represents the total number of model performance feedback test indicators for post-testing, Represents the total number of ablation test metrics for the dataset for post-detection.

[0021] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A data set quality assessment test method for artificial intelligence, characterized in that: The following steps are involved: S1. Test data set preparation: Use standardized, diversified, and integrated data generation technology to achieve standardized production of data sets required for large model training; S2. Test condition initialization: Based on the basic situation of the test data set and the test objectives, screen and determine the quality evaluation indicators required for the actual test, and design the weight structure of each core indicator to quantify the final quality score of the test data set. It specifically includes two major components: importing the basic situation of the test data set and designing the weight of the test data set indicators; S3. Pre-test: Based on the established dataset knowledge and rule management system, use the AI ​​dataset quality test rules or empirical formulas to conduct preliminary tests on the quality of the AI ​​dataset, which specifically includes two parts: rule-based autonomous detection and manual-assisted detection; S4. Post-test: Use different technical means to detect the specific role of high-quality AI datasets in the performance of large model training, and use model performance feedback to quantify the specific role of different evaluation dimensions of AI datasets in improving the performance of large models, mainly including model benchmark feedback and dataset ablation testing; S5. Calculation of data set quality assessment score: Based on the quality assessment technical process of the first four steps, the specific calculation of the final AI data set quality assessment score is as follows: In the formula, Represents the final total score, Represents the weight of the pre-test evaluation score in the total score, Represents the specific indicator score of the i-th indicator in the rule-based autonomous detection of the pre-detection, The representative evaluation service object autonomously detects the specific weight of the i-th indicator according to the rules set in the business scenario. Represents the specific index score of the jth index in the manual assisted detection of the previous detection, Represents the specific weight of the jth indicator of manual assisted detection set by the evaluation service object according to the business scenario, Represents the weight of the post-detection evaluation method score in the total score, Represents the specific indicator score of the kth indicator in the model performance feedback test of the post-detection, Represents the specific weight of the kth indicator detected by the evaluation service object based on the model performance feedback set in the business scenario. Represents the specific indicator score of the index in the data ablation test of the post-detection, Represents the specific weight of the th indicator of the data set ablation test set by the evaluation service object according to the business scenario. Represents the total number of rule-based autonomous detection indicators for pre-detection. Represents the total number of manual assisted detection indicators for pre-detection, Represents the total number of model performance feedback test indicators for post-testing, Represents the total number of ablation test metrics for the dataset for post-detection.

2. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In step S1, the specific implementation method of the test data set preparation includes two parts: a high-quality data set preparation technical link based on intelligent data sampling technology and a high-quality data set preparation technical link based on synthetic data technology. The generation and preparation of high-quality data sets are achieved through the technical integration of the two methods.

3. The data set quality assessment test method for artificial intelligence according to claim 2, characterized in that: The high-quality data preparation link based on intelligent data collection technology is specifically aimed at the existing data in the current physical space, and specifically includes four core technical links: automated collection, deep cleaning, intelligent labeling, and fusion quality inspection. Among them, the implementation methods of automated collection include Web crawlers, API interfaces, Internet of Things access, database synchronization, and multimodal data collection. The implementation methods of deep cleaning include Apache NiFi ETL technology, automatic cleaning shell scripts, machine learning cleaning models, Google Cloud, and Dataflow cloud service technology. The implementation methods of intelligent labeling include MFT multimodal fusion labeling, cross-modal data alignment, transfer learning labeling, and human-computer collaborative interaction labeling. The implementation methods of fusion quality inspection include K-Means clustering, SVM intelligent classification, density anomaly LOF algorithm, SOS probability anomaly detection, PCA linear dimensionality reduction detection, and AutoEncoder nonlinear detection.

4. The method for quality assessment of data sets for artificial intelligence according to claim 2, characterized in that: The high-quality data preparation link based on synthetic data technology specifically generates a test data set that meets the requirements of large model training through data synthesis technology methods, which specifically includes two technical links: multi-mode data synthesis and synthetic data quality inspection; among them, the multi-mode data synthesis method includes the LLM-P-Cot large model data synthesis technology method based on LLMs large model prompt engineering and Cot intermediate thinking chain reasoning steps, the BigGAN-StyleGAN fusion algorithm model based on adversarial neural network, and the distribution function generation method based on normal distribution and exponential distribution. The synthetic data quality inspection specifically includes the five major technical indicators of artificial intelligence data set fidelity, privacy, coverage, interpretability, and availability.

5. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In step S2, the import of the basic information of the test data set is intended to provide a complete and detailed description of the basic information of the data set required for the test, and the basic information required for the test data set includes the following aspects: 1) Field structure of the data set: clarify the specific field composition of the test data set and the specific meaning of each field; 2) Dataset size: Specify the specific data size and number of data items in the test dataset; 3) Modality type of the dataset: clarify the specific modality and type of the test dataset, including text, voice, image, video, audio, and multi-modality; 4) Task types supported by the dataset: clarify the types of training tasks that the test dataset needs to support in the future; 5) Dataset application scenario classification system: evaluate the quality of high-quality industry data sets, and clarify the key industry fields and industry business scenario systems to which the data sets belong; 6) Training stages supported by the dataset: pre-training stage, fine-tuning stage, feedback reinforcement stage, evaluation and verification stage, and reasoning stage; 7) Dataset security technical requirements: clarify the specific requirements for test confidentiality and security of the test dataset, and specify those with special security and confidentiality requirements; 8) Description of other issues: If there are other special requirements, additional explanation is required.

6. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In step S2, the weight design of the test data set indicators is different according to the subject scope and professional level. The specific definition of the artificial intelligence data set is divided into three categories: general data set, industry general data set and industry special data set. There are differences in the evaluation requirements of the three categories of data sets for the specific 12 categories of first-level evaluation indicators and 36 categories of second-level evaluation indicators; The first-level evaluation indicators include completeness, standardization, accuracy, timeliness, consistency, density, diversity, balance, relevance, originality, traceability, and accessibility, and the second-level indicators included in each first-level evaluation indicator are as follows: Secondary indicators of completeness include: scale completeness, structural completeness, mission completeness, and domain completeness; The secondary indicators of normativity include: formal normativity, privacy normativity, and security normativity; Secondary indicators of accuracy include: content authenticity and field expertise; The secondary indicators of timeliness include: generation timeliness, collection timeliness, processing timeliness, and release and update timeliness; Secondary indicators of consistency include: label consistency, concept consistency; Secondary indicators of density include: sample uniqueness and content information; The secondary indicators of diversity include: feature diversity, type diversity, source diversity, task diversity, domain diversity, and cultural diversity; The secondary indicators of balance include: category balance, source balance; Secondary indicators of relevance include: logical relevance, task relevance, and domain relevance; The secondary indicators of originality include: source originality, content originality; The secondary indicators of traceability include: source traceability, link traceability, and version control traceability; The secondary indicators of accessibility include: ease of acquisition, ease of use, and technical support.

7. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In step S3, the technical process of autonomous rule detection is to write a specific automated operation algorithm, program or model through a deterministic rule system or empirical formula for artificial intelligence data set quality assessment to automatically check the quality of the data set, screen out data sets with too low quality in advance, and reduce the cost for subsequent sampling detection and model verification, which specifically includes: S31. Autonomous identification of dataset knowledge information It aims to efficiently identify and analyze the detailed knowledge and information content of the data set and improve the utilization rate of the data set for the model. It specifically includes two major components: one is shallow information identification, which is responsible for defining and organizing the classification standards of the data set; the other is deep information extraction, which uses statistical and machine learning techniques to deeply explore and analyze the inherent patterns and laws in the data set; S32. Dataset quality rule detection It aims to realize the automated rule detection of basic quality and quality assessment indicators of data sets, including four major components: missing value detection, duplicate value detection, data set format detection, and automated rule detection of data set quality assessment indicators. Specific technical implementation methods include rule-based detection scripts, data quality tools, ETL tools, built-in functions of database management systems, API calls, and machine learning generation models.

8. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In the step S3, manual assisted detection is performed on the data set that has initially met the standards in the rule-based autonomous detection. With the help of the professional knowledge and industry experience of exclusive industry experts, manual assisted detection of some errors or anomalies in the data set is performed to identify omitted complex problems that cannot be covered by the rule-based autonomous detection method. Specific detection methods include simple random sampling, priority sampling, mixed sampling, stratified sampling, systematic sampling, probability proportion size sampling, convenience sampling, and judgment sampling.

9. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In step S4, model performance feedback: select a suitable model according to the business scenario and task requirements, and send the evaluated data set into the model for training. At the same time, select appropriate model performance verification indicators to verify the model performance. Through the core performance indicators of large model training, the fluctuation of model performance is discovered in time. The specific quality performance of the evaluated data set is determined by judging the performance of the model on the evaluation indicators. The training performance indicators of the large model specifically include three parts: model training accuracy performance, model training speed performance, and model training resource consumption.

10. The method for quality assessment of data sets for artificial intelligence according to claim 1, characterized in that: In step S4, the data set ablation test is performed by changing one or more evaluation dimensions of the original data set to obtain multiple different data sets, using these indicators to change the different data sets to train the same model respectively, and testing and verifying the changes in the model benchmarking capabilities to evaluate and verify the specific contribution of one or more indicator combinations to the model performance. The data set ablation test technical means specifically include data set subset experiments and data set feature screening analysis.

Images