Data quality detection method and system based on large model

Abstract

The invention discloses a data quality detection method and system based on a large model. The data quality detection method based on the large model comprises the following steps: S1, obtaining to-be-detected data; s2, preprocessing the to-be-detected data, and extracting data features; dividing the data into a training set, a verification set and a test set to prepare for subsequent model training; s3, selecting a suitable large model of the data quality detection task for pre-training; performing fine adjustment on the selected large model on the training set so as to adapt to specific data quality detection requirements; s4, detecting the data quality of the to-be-detected data according to the data features and a preset quality detection rule by using the fine-tuned large model; and S5, comparing a detection result with a manual auditing result, and evaluating the accuracy and effectiveness of the large model. According to the invention, efficient and intelligent data quality detection can be realized, the labor cost is reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data quality detection method and system based on a large model. Background Art

[0002] With the rapid development of information technology, the amount of data is growing exponentially. In the process of data-driven decision-making, enterprises are faced with the problem of poor data quality. Data quality problems usually include missing data, duplication, errors, and inconsistencies, which will directly affect the reliability and effectiveness of data analysis results.

[0003] Traditional data quality detection methods mainly rely on rule engines and manual review, which have the following shortcomings:

[0004] Complex rule formulation: Traditional methods require manual formulation of a large number of rules, which makes it difficult to cover all possible abnormal situations.

[0005] Poor adaptability: Faced with emerging data types and data patterns, it is very difficult to update and maintain rules.

[0006] High labor costs: Manual review is time-consuming and inefficient, and it is difficult to meet the needs of large-scale data processing.

[0007] Low intelligence: Lack of understanding of data context makes it difficult to identify potential quality issues.

[0008] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a data quality detection method and system based on a large model to achieve efficient and intelligent data quality detection, reduce labor costs, and improve data processing efficiency.

[0010] The technical solution of the present invention is as follows: a data quality detection method based on a large model is provided, comprising the following steps.

[0011] S1: Obtain the data to be detected.

[0012] S2: Preprocess the data to be detected and extract data features; and divide the data into a training set, a validation set, and a test set to prepare for subsequent model training.

[0013] S3: Select a large model suitable for the data quality detection task for pre-training; fine-tune the selected large model on the training set to adapt to specific data quality detection requirements.

[0014] S4: Using the fine-tuned large model, the data to be detected is subjected to quality detection according to the data features and preset quality detection rules.

[0015] S5: Compare the test results with the manual review results to evaluate the accuracy and effectiveness of the large model; and further optimize the model based on the feedback information to improve the detection performance.

[0016] The data preprocessing step includes: cleaning the data to be detected, removing noise and standardizing the format; specifically including:

[0017] Detect missing values: Detect null values ​​or missing values ​​in the data; handle missing values: use appropriate filling methods to supplement continuous data; such as mean, median, mode filling, or use interpolation to supplement continuous data; consider deleting data columns with a high proportion of missing values.

[0018] Detect outliers: Use statistical methods or models to detect outliers in data; Process outliers: Correct outliers or eliminate unreasonable data based on business logic.

[0019] Data format unification: standardize data in different formats, such as standardizing the date format to YYYY-MM-DD; clean up text fields, including removing spaces, special characters, HTML tags and other useless information.

[0020] Noise removal: Filter out meaningless random noise in the data (such as incorrectly entered data or invalid entries in logs); for time series data, smoothing techniques (such as sliding average) can be used to reduce data fluctuations.

[0021] Duplicate data processing: Remove duplicate data rows by checking unique identifiers (such as primary keys or business identifiers); merge records with duplicate information to reduce redundant data.

[0022] Feature normalization and / or standardization: normalization and / or standardization of numerical data to ensure that the data range of different features is consistent; Normalization method (Min-Max Scaling): map the data to the range [0,1]. Standardization method: convert the data to a normal distribution with a mean of 0 and a variance of 1.

[0023] Data consistency check: Ensure the consistency between multiple fields, for example, the order date cannot be later than the delivery date; verify the data type (such as integer, string, etc.) to ensure that it is consistent with the expected type.

[0024] The large model is a pre-trained deep learning model, including but not limited to BERT, GPT, etc. The steps for training the large model are as follows:

[0025] Dataset division: training set, validation set, and test set account for 70%-80%, 10%-15%, and 10%-15%, respectively, ensuring consistent data distribution and random sampling.

[0026] Data preprocessing: Perform missing value filling, normalization, text vectorization and other preprocessing on the training set, validation set and test set, and use a large model to extract text and image features.

[0027] Model selection and initialization: Select a suitable deep learning model or self-supervised learning model, and load pre-trained model parameters or start training from scratch.

[0028] Model training: Train the model using the training set, select a suitable loss function and update the parameters through back propagation.

[0029] Hyperparameter optimization: Use the validation set to evaluate model performance and optimize hyperparameters through methods such as grid search or random search.

[0030] Model validation: Monitor model performance through the validation set. If overfitting occurs, stop training or adjust the regularization strategy.

[0031] Model testing: Evaluate model performance on the test set to ensure that the generalization ability meets expectations.

[0032] Model deployment and feedback: Deploy the trained model and use actual data and feedback mechanisms to continuously optimize model performance.

[0033] The purpose of fine-tuning the large model is to improve its accuracy and adaptability in specific data quality detection tasks. The specific method is to optimize its performance by adjusting the large model parameters based on the data characteristics of the data to be detected. Fine-tuning the large model is usually performed before the large model is initially built and applied to actual detection tasks or when a large number of false positives / missing negatives occur during the detection process, to ensure that the large model can more accurately identify quality problems in the data, thereby improving the efficiency of data quality detection and reducing the cost of manual intervention.

[0034] The data characteristics include the following types:

[0035] Numerical features appear in the form of continuous or discrete values, such as temperature, pressure, time, etc. These features can be directly used in mathematical calculations and analysis and are the basis of quantitative research.

[0036] Categorical features are represented as discrete values ​​or strings and usually have a limited set of values, such as city, country, product category, etc. They are more often used for classification and labeling.

[0037] Text features contain a large amount of natural language information and are usually unstructured data, such as user comments, product descriptions, etc., which require the use of natural language processing technology for analysis.

[0038] Image-based features, represented in the form of pixels or high-dimensional matrices, involve spatial and visual features, such as product pictures, scanned documents, etc. They require image processing technology to extract useful information.

[0039] Time series features, whose characteristic values ​​depend on the time sequence and show certain time trends and laws, such as stock prices and temperature changes, play an important role in time series analysis and prediction.

[0040] Set quality detection rules for data feature types, as follows:

[0041] For numerical features, such as temperature, pressure, time and other continuous values, the distribution characteristics of the data are analyzed through large models to detect whether there are outliers, missing values ​​or data drift, to ensure the accuracy and rationality of the values.

[0042] For categorical features, such as city, country, product category, etc., verify whether their values ​​are within the predefined set and check whether there is unreasonable distribution deviation between categories to maintain data consistency and balance.

[0043] For text features, such as user comments and product descriptions, natural language processing technology is used to detect the standardization of language, the clarity of semantics, and the completeness of information to ensure the readability and validity of text data.

[0044] For image-based features, such as product images and scanned documents, the clarity, completeness, and format compliance of the images are evaluated, and the quality and availability of image data are ensured through pixel-level analysis.

[0045] For time series features, such as stock prices, temperature changes, etc., analyze the time trend, periodicity and stationarity of the data, detect abnormal fluctuations and pattern changes, and ensure the stability and predictability of the time series data.

[0046] The steps for data quality testing are as follows:

[0047] (1) Data loading and preliminary check: Load data and perform basic statistics, and generate a preliminary report to identify missing, anomaly, and format issues.

[0048] (2) Missing value detection and processing: The large model learns the pattern of the data to predict the missing location, classify the missing values ​​and select the filling method.

[0049] (3) Outlier detection and processing: Combine statistical methods with large models to identify outliers and determine the processing method based on quality inspection rules.

[0050] (4) Duplicate data detection and deduplication: Duplicate data is checked through hashing and other means, marked, and deduplicated as needed.

[0051] (5) Data consistency detection: The large model analyzes data relationships and detects and corrects or marks inconsistent data.

[0052] (6) Data format verification and standardization: Check whether the data format meets expectations and convert or modify those that do not meet the standards.

[0053] (7) Text and nested data detection: Use NLP models to detect text features and parsing tools to verify nested data structures.

[0054] (8) Data distribution and correlation analysis: Analyze the distribution of numerical data and feature correlation to identify potential quality issues.

[0055] (9) Generate quality inspection report: summarize the inspection results, generate a report and feed it back to the system interface to optimize the data input source or preprocessing process.

[0056] Outlier detection and processing specifically include: In the process of data quality detection, common problems can be summarized into the following categories: missing values, outliers, duplicate data, data inconsistency and data format errors; specific solutions are as follows: For missing values, the mean, median or large model prediction can be used to fill in; for outliers, use statistical methods (such as the 3σ rule) or large model-based anomaly detection algorithms to identify and correct; for duplicate data, redundant data can be deleted through data deduplication algorithms or unique identifier verification; for data inconsistencies, data normalization is required according to preset rules and large model entity alignment technology; and for data format errors, they can be repaired through formatting and regular expression matching; these methods can effectively improve data quality and ensure the reliability of data analysis and model training.

[0057] In step S5, after comparing the detection results with the manual review results, the differences between the two are analyzed to identify the deficiencies of the large model in data quality detection, thereby evaluating the accuracy and effectiveness of the large model; the specific methods are as follows: for the detected errors and problem categories (such as missed detection, false detection, etc.), optimize the large model according to the feedback information; first, expand the training set by introducing more diverse and accurately labeled data to improve the generalization ability of the model for different scenarios and abnormal types; second, adjust the model's hyperparameters (such as learning rate, regularization coefficient, etc.) to optimize model performance; in addition, introduce transfer learning or fine-tuning technology, and optimize specific data features in combination with high-quality pre-trained models; at the same time, use the feedback results to redefine or enhance the quality detection rules to increase the model's sensitivity to specific data features; finally, continue to iterate training, and ensure the steady improvement of model performance through continuous evaluation of validation sets and test sets, so as to achieve more efficient and accurate data quality detection.

[0058] Further optimize the large model based on the feedback information to improve the detection performance. The steps include: dynamically adjust the data processing flow according to the detection results to improve the efficiency of subsequent data quality detection;

[0059] According to the test results, the data processing flow is dynamically adjusted to improve the efficiency of subsequent data quality testing; specifically, by analyzing the quality test report output by the model, the main problems in the data and their distribution patterns are identified, and the data processing flow is optimized in a targeted manner.

[0060] For data sources that often have missing values ​​or outliers, you can dynamically adjust the data collection strategy, strengthen real-time verification during the collection process, or directly optimize the data cleaning rules, such as defining the filling method and exception handling logic of specific fields in advance.

[0061] For data with large feature distribution deviations, dynamically optimize data preprocessing steps, such as adjusting normalization and discretization methods or introducing new feature engineering strategies.

[0062] Dynamically update quality detection rules based on feedback results, such as adding new rules to the rule base, or increasing sensitivity to specific issues by adjusting model weights; dynamic adjustment of the overall process continuously improves the adaptability and processing capabilities of the large model to different data scenarios through a closed-loop feedback mechanism, thereby improving the accuracy and efficiency of data quality detection.

[0063] The present invention also provides a data quality detection system based on a large model, which is used to automatically implement the aforementioned data quality detection method based on a large model, including: a data input module, used to receive data to be detected; a data preprocessing module, used to clean and extract features of the data to be detected; a data detection module, used to perform quality detection on the data according to a fine-tuned large model; and a result output module, used to output the detection results and perform feedback control based on the results.

[0064] The present invention provides a data quality detection method and system based on a large model, which aims to automatically identify quality problems in data through deep learning technology. The method includes: obtaining data to be detected; preprocessing the data to be detected based on the large model and extracting data features; detecting data quality according to data features and preset quality detection rules; outputting the detection results, and performing feedback control according to the detection results. The method and system of the present invention improve the accuracy and efficiency of data quality detection through the application of large models, reduce the cost of manual intervention, and provide strong support for data quality management.

[0065] By adopting the above scheme, the present invention provides a data quality detection method and system based on a large model, which has the following technical effects:

[0066] 1. Improve detection efficiency: Automated detection reduces the need for manual review and greatly increases the speed of data processing.

[0067] 2. Reduce labor costs: Through intelligent data quality detection, the investment in human resources is reduced.

[0068] 3. Improve detection accuracy: Utilizing the deep learning capabilities of large models, a variety of complex data quality issues can be accurately identified.

[0069] 4. Enhanced adaptability: The system can continuously learn new data features, update detection rules in a timely manner, and improve adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The figure is a flow chart of the method for detecting data quality based on a large model of the present invention. DETAILED DESCRIPTION

[0071] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0072] See also Figure 1 The present invention provides a data quality detection method based on a large model, comprising the following steps.

[0073] S1: Obtain the data to be detected.

[0074] S2: Preprocess the data to be detected and extract data features; and divide the data into a training set, a validation set, and a test set to prepare for subsequent model training.

[0075] S3: Select a large model suitable for the data quality detection task for pre-training; fine-tune the selected large model on the training set to adapt to specific data quality detection requirements.

[0076] S4: Using the fine-tuned large model, the data to be detected is subjected to quality detection according to the data features and preset quality detection rules.

[0077] S5: Compare the test results with the manual review results to evaluate the accuracy and effectiveness of the large model; and further optimize the model based on the feedback information to improve the detection performance.

[0078] The data preprocessing step includes: cleaning the data to be detected, removing noise and standardizing the format; specifically including:

[0079] Detect missing values: Detect null values ​​or missing values ​​in the data; handle missing values: use appropriate filling methods to supplement continuous data; such as mean, median, mode filling, or use interpolation to supplement continuous data; consider deleting data columns with a high proportion of missing values.

[0080] Detect outliers: Use statistical methods or models to detect outliers in data; Process outliers: Correct outliers or eliminate unreasonable data based on business logic.

[0081] Data format unification: standardize data in different formats, such as standardizing the date format to YYYY-MM-DD; clean up text fields, including removing spaces, special characters, HTML tags and other useless information.

[0082] Noise removal: Filter out meaningless random noise in the data (such as incorrectly entered data or invalid entries in logs); for time series data, smoothing techniques (such as sliding average) can be used to reduce data fluctuations.

[0083] Duplicate data processing: Remove duplicate data rows by checking unique identifiers (such as primary keys or business identifiers); merge records with duplicate information to reduce redundant data.

[0084] Feature normalization and / or standardization: normalization and / or standardization of numerical data to ensure that the data range of different features is consistent; Normalization method (Min-Max Scaling): map the data to the range [0,1]. Standardization method: convert the data to a normal distribution with a mean of 0 and a variance of 1.

[0085] Data consistency check: Ensure the consistency between multiple fields, for example, the order date cannot be later than the delivery date; verify the data type (such as integer, string, etc.) to ensure that it is consistent with the expected type.

[0086] The large model is a pre-trained deep learning model, including but not limited to BERT, GPT, etc. The steps for training the large model are as follows:

[0087] Dataset division: training set, validation set, and test set account for 70%-80%, 10%-15%, and 10%-15%, respectively, ensuring consistent data distribution and random sampling.

[0088] Data preprocessing: Perform missing value filling, normalization, text vectorization and other preprocessing on the training set, validation set and test set, and use a large model to extract text and image features.

[0089] Model selection and initialization: Select a suitable deep learning model or self-supervised learning model, and load pre-trained model parameters or start training from scratch.

[0090] Model training: Train the model using the training set, select a suitable loss function and update the parameters through back propagation.

[0091] Hyperparameter optimization: Use the validation set to evaluate model performance and optimize hyperparameters through methods such as grid search or random search.

[0092] Model validation: Monitor model performance through the validation set. If overfitting occurs, stop training or adjust the regularization strategy.

[0093] Model testing: Evaluate model performance on the test set to ensure that the generalization ability meets expectations.

[0094] Model deployment and feedback: Deploy the trained model and use actual data and feedback mechanisms to continuously optimize model performance.

[0095] The purpose of fine-tuning the large model is to improve its accuracy and adaptability in specific data quality detection tasks. The specific method is to optimize its performance by adjusting the large model parameters based on the data characteristics of the data to be detected. Fine-tuning the large model is usually performed before the large model is initially built and applied to actual detection tasks or when a large number of false positives / missing negatives occur during the detection process, to ensure that the large model can more accurately identify quality problems in the data, thereby improving the efficiency of data quality detection and reducing the cost of manual intervention.

[0096] The data characteristics include the following types:

[0097] Numerical features appear in the form of continuous or discrete values, such as temperature, pressure, time, etc. These features can be directly used in mathematical calculations and analysis and are the basis of quantitative research.

[0098] Categorical features are represented as discrete values ​​or strings and usually have a limited set of values, such as city, country, product category, etc. They are more often used for classification and labeling.

[0099] Text features contain a large amount of natural language information and are usually unstructured data, such as user comments, product descriptions, etc., which require the use of natural language processing technology for analysis.

[0100] Image-based features, represented in the form of pixels or high-dimensional matrices, involve spatial and visual features, such as product pictures, scanned documents, etc. They require image processing technology to extract useful information.

[0101] Time series features, whose characteristic values ​​depend on the time sequence and show certain time trends and laws, such as stock prices and temperature changes, play an important role in time series analysis and prediction.

[0102] Set quality detection rules for data feature types, as follows:

[0103] For numerical features, such as temperature, pressure, time and other continuous values, the distribution characteristics of the data are analyzed through large models to detect whether there are outliers, missing values ​​or data drift, to ensure the accuracy and rationality of the values.

[0104] For categorical features, such as city, country, product category, etc., verify whether their values ​​are within the predefined set and check whether there is unreasonable distribution deviation between categories to maintain data consistency and balance.

[0105] For text features, such as user comments and product descriptions, natural language processing technology is used to detect the standardization of language, the clarity of semantics, and the completeness of information to ensure the readability and validity of text data.

[0106] For image-based features, such as product images and scanned documents, the clarity, completeness, and format compliance of the images are evaluated, and the quality and availability of image data are ensured through pixel-level analysis.

[0107] For time series features, such as stock prices, temperature changes, etc., analyze the time trend, periodicity and stationarity of the data, detect abnormal fluctuations and pattern changes, and ensure the stability and predictability of the time series data.

[0108] The steps for data quality testing are as follows:

[0109] (1) Data loading and preliminary check: Load data and perform basic statistics, and generate a preliminary report to identify missing, anomaly, and format issues.

[0110] (2) Missing value detection and processing: The large model learns the pattern of the data to predict the missing location, classify the missing values ​​and select the filling method.

[0111] (3) Outlier detection and processing: Combine statistical methods with large models to identify outliers and determine the processing method based on quality inspection rules.

[0112] (4) Duplicate data detection and deduplication: Duplicate data is checked through hashing and other means, marked, and deduplicated as needed.

[0113] (5) Data consistency detection: The large model analyzes data relationships and detects and corrects or marks inconsistent data.

[0114] (6) Data format verification and standardization: Check whether the data format meets expectations and convert or modify those that do not meet the standards.

[0115] (7) Text and nested data detection: Use NLP models to detect text features and parsing tools to verify nested data structures.

[0116] (8) Data distribution and correlation analysis: Analyze the distribution of numerical data and feature correlation to identify potential quality issues.

[0117] (9) Generate quality inspection report: summarize the inspection results, generate a report and feed it back to the system interface to optimize the data input source or preprocessing process.

[0118] Outlier detection and processing specifically include: In the process of data quality detection, common problems can be summarized into the following categories: missing values, outliers, duplicate data, data inconsistency and data format errors; specific solutions are as follows: For missing values, the mean, median or large model prediction can be used to fill in; for outliers, use statistical methods (such as the 3σ rule) or large model-based anomaly detection algorithms to identify and correct; for duplicate data, redundant data can be deleted through data deduplication algorithms or unique identifier verification; for data inconsistencies, data normalization is required according to preset rules and large model entity alignment technology; and for data format errors, they can be repaired through formatting and regular expression matching; these methods can effectively improve data quality and ensure the reliability of data analysis and model training.

[0119] In step S5, after comparing the detection results with the manual review results, the differences between the two are analyzed to identify the deficiencies of the large model in data quality detection, thereby evaluating the accuracy and effectiveness of the large model; the specific methods are as follows: for the detected errors and problem categories (such as missed detection, false detection, etc.), optimize the large model according to the feedback information; first, expand the training set by introducing more diverse and accurately labeled data to improve the generalization ability of the model for different scenarios and abnormal types; second, adjust the model's hyperparameters (such as learning rate, regularization coefficient, etc.) to optimize model performance; in addition, introduce transfer learning or fine-tuning technology, and optimize specific data features in combination with high-quality pre-trained models; at the same time, use the feedback results to redefine or enhance the quality detection rules to increase the model's sensitivity to specific data features; finally, continue to iterate training, and ensure the steady improvement of model performance through continuous evaluation of validation sets and test sets, so as to achieve more efficient and accurate data quality detection.

[0120] Further optimize the large model based on the feedback information to improve the detection performance. The steps include: dynamically adjust the data processing flow according to the detection results to improve the efficiency of subsequent data quality detection;

[0121] According to the test results, the data processing flow is dynamically adjusted to improve the efficiency of subsequent data quality testing; specifically, by analyzing the quality test report output by the model, the main problems in the data and their distribution patterns are identified, and the data processing flow is optimized in a targeted manner.

[0122] For data sources that often have missing values ​​or outliers, you can dynamically adjust the data collection strategy, strengthen real-time verification during the collection process, or directly optimize the data cleaning rules, such as defining the filling method and exception handling logic of specific fields in advance.

[0123] For data with large feature distribution deviations, dynamically optimize data preprocessing steps, such as adjusting normalization and discretization methods or introducing new feature engineering strategies.

[0124] Dynamically update quality detection rules based on feedback results, such as adding new rules to the rule base, or increasing sensitivity to specific issues by adjusting model weights; dynamic adjustment of the overall process continuously improves the adaptability and processing capabilities of the large model to different data scenarios through a closed-loop feedback mechanism, thereby improving the accuracy and efficiency of data quality detection.

[0125] The present invention also provides a data quality detection system based on a large model, which is used to automatically implement the aforementioned data quality detection method based on a large model, including: a data input module for receiving data to be detected; a data preprocessing module for cleaning and feature extraction of the data to be detected; a data detection module for performing quality detection on the data according to the fine-tuned large model; and a result output module for outputting the detection results and performing feedback control according to the results. The system of the present invention can realize an automated data quality detection process and support batch processing of large-scale data.

[0126] The present invention is applicable to various data management scenarios, including but not limited to database management, data warehouses, data lakes, enterprise resource planning (ERP) systems, etc. It can effectively improve data quality and assist in data-driven decision making.

[0127] In summary, the present invention provides a data quality detection method and system based on a large model, which has the following technical effects:

[0128] 1. Improve detection efficiency: Automated detection reduces the need for manual review and greatly increases the speed of data processing.

[0129] 2. Reduce labor costs: Through intelligent data quality detection, the investment in human resources is reduced.

[0130] 3. Improve detection accuracy: Utilizing the deep learning capabilities of large models, a variety of complex data quality issues can be accurately identified.

[0131] 4. Enhanced adaptability: The system can continuously learn new data features, update detection rules in a timely manner, and improve adaptability.

[0132] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data quality detection method based on a large model, characterized in that: The following steps are involved: S1: Obtain the data to be detected; S2: preprocessing the data to be detected and extracting data features; And divide the data into training set, validation set and test set to prepare for subsequent model training; S3: Select a large model suitable for the data quality detection task for pre-training; fine-tune the selected large model on the training set to adapt to specific data quality detection requirements; S4: Using the fine-tuned large model, the data to be detected is subjected to data quality detection according to the data features and preset quality detection rules; S5: Compare the test results with the manual review results to evaluate the accuracy and effectiveness of the large model; and further optimize the model based on the feedback information to improve the detection performance.

2. According to the large model-based data quality detection method of claim 1, it is characterized in that: The data preprocessing step includes: cleaning the data to be detected, removing noise and standardizing the format; specifically including: Detect missing values: detect null or missing values ​​in the data; handle missing values: use appropriate filling methods to supplement continuous data; consider deleting columns with high missing data percentages; Detect outliers: Use statistical methods or models to detect outliers in data; Process outliers: Correct outliers or eliminate unreasonable data based on business logic; Data format unification: standardize data in different formats; clean up text fields, including removing spaces, special characters, and HTML tags; Noise removal: filter out meaningless random noise in the data; Duplicate data processing: remove duplicate data rows by checking unique identifiers; merge records with duplicate information to reduce redundant data; Feature normalization and / or standardization: normalization and / or standardization methods are used on numerical data to ensure that the data ranges of different features are consistent; Data consistency check: Ensure the consistency between multiple fields; verify the data type to ensure that it is consistent with the expected type.

3. The data quality detection method based on a large model according to claim 1 is characterized in that: The large model is a pre-trained deep learning model; the steps for training the large model are as follows: Dataset division: training set, validation set, and test set account for 70%-80%, 10%-15%, and 10%-15% respectively, ensuring consistent data distribution and random sampling; Data preprocessing: perform missing value filling, normalization, and text vectorization preprocessing on the training set, validation set, and test set, and use a large model to extract text and image features; Model selection and initialization: select a suitable deep learning model or self-supervised learning model, and load pre-trained model parameters or start training from scratch; Model training: Use the training set to train the model, select a suitable loss function and update the parameters through back propagation; Hyperparameter optimization: Use the validation set to evaluate model performance and optimize hyperparameters through grid search or random search. Model validation: Monitor model performance through validation sets. If overfitting occurs, stop training or adjust the regularization strategy. Model testing: Evaluate model performance on the test set to ensure that the generalization ability meets expectations; Model deployment and feedback: Deploy the trained model and use actual data and feedback mechanisms to continuously optimize model performance.

4. The data quality detection method based on a large model according to claim 1 is characterized in that: The purpose of fine-tuning the large model is to improve its accuracy and adaptability in specific data quality detection tasks. The specific method is to optimize its performance by adjusting the large model parameters based on the data characteristics of the data to be detected. Fine-tuning the large model is usually performed before the large model is initially built and applied to actual detection tasks or when a large number of false positives / missing negatives occur during the detection process, to ensure that the large model can more accurately identify quality problems in the data, thereby improving the efficiency of data quality detection and reducing the cost of manual intervention.

5. The data quality detection method based on a large model according to claim 1 is characterized in that: The data characteristics include the following type: Numerical features appear in the form of continuous or discrete values ​​and are directly used for mathematical calculations and analysis. They are the basis of quantitative research. Categorical features are represented as discrete values ​​or strings, usually with a limited set of values, and are used for classification and labeling; Text features contain a large amount of natural language information, which is usually unstructured data and needs to be analyzed with the help of natural language processing technology; Image-based features are represented in the form of pixels or high-dimensional matrices. They involve spatial and visual features and require image processing techniques to extract useful information. Time series characteristics, whose characteristic values ​​depend on the time sequence, show certain time trends and laws, and play an important role in time series analysis and prediction.

6. A data quality detection method based on a large model according to claim 5, characterized in that: Set quality detection rules for data feature types, as follows: For numerical features, we use large models to analyze the distribution characteristics of the data and detect whether there are outliers, missing values, or data drift to ensure the accuracy and rationality of the values. For categorical features, verify whether their values ​​are within the predefined set and check whether there is unreasonable distribution deviation between categories to maintain data consistency and balance; For text features, natural language processing technology is used to detect the standardization of language, the clarity of semantics, and the integrity of information to ensure the readability and effectiveness of text data; For image-based features, the clarity, integrity, and format of the image are evaluated, and the quality and availability of image data are ensured through pixel-level analysis; For time series characteristics, the time trend, periodicity and stationarity of the data are analyzed, and abnormal fluctuations and pattern changes are detected to ensure the stability and predictability of the time series data.

7. The data quality detection method based on a large model according to claim 1 is characterized in that: The steps for data quality testing are as follows: (1) Data loading and preliminary check: Load data and perform basic statistics, generate preliminary reports to identify missing, anomalies and format issues; (2) Missing value detection and processing: The large model learns the data pattern to predict the missing location, classifies the missing values ​​and selects the filling method; (3) Outlier detection and processing: Combine statistical methods with large models to identify outliers and determine the processing method based on quality inspection rules; (4) Duplicate data detection and deduplication: Check duplicate data through hashing and other means, mark and dedupe as needed; (5) Data consistency detection: The big model analyzes data relationships, detects and corrects or marks inconsistent data; (6) Data format verification and standardization: Check whether the data format meets expectations and convert or modify those that do not meet the standards; (7) Text and nested data detection: Use NLP models to detect text features and parsing tools to verify nested data structures; (8) Data distribution and correlation analysis: Analyze the distribution and feature correlation of numerical data to identify potential quality issues; (9) Generate quality inspection report: summarize the inspection results, generate a report and feed it back to the system interface to optimize the data input source or preprocessing process; Outlier detection and processing specifically include: In the process of data quality detection, common problems can be summarized into the following categories: missing values, outliers, duplicate data, data inconsistency and data format errors; specific solutions are as follows: For missing values, the mean, median or large model prediction can be used to fill in the gaps; for outliers, statistical methods or large model-based anomaly detection algorithms are used to identify and correct them; for duplicate data, redundant data can be deleted through data deduplication algorithms or unique identifier verification; for data inconsistencies, data normalization is required according to preset rules and large model entity alignment technology; and for data format errors, they can be repaired through formatting and regular expression matching.

8. The data quality detection method based on a large model according to claim 1 is characterized in that: In step S5, after comparing the detection results with the manual review results, the differences between the two are analyzed to identify the deficiencies of the large model in data quality detection, thereby evaluating the accuracy and effectiveness of the large model; the specific method is as follows: Based on the detected errors and problem categories, the large model is optimized according to the feedback information. First, the training set is expanded by introducing more diverse and accurately labeled data to improve the model's generalization ability for different scenarios and anomaly types. Secondly, adjust the model's hyperparameters to optimize model performance. In addition, introduce transfer learning or fine-tuning technology, and combine high-quality pre-trained models to optimize specific data features. At the same time, use feedback results to redefine or enhance quality detection rules to increase the model's sensitivity to specific data features. Finally, continue iterative training and continuously evaluate the validation set and test set to ensure steady improvement of model performance and achieve more efficient and accurate data quality detection.

9. The data quality detection method based on a large model according to claim 1 is characterized in that: Further optimize the large model based on the feedback information to improve the detection performance. The steps include: dynamically adjust the data processing flow according to the detection results to improve the efficiency of subsequent data quality detection; According to the test results, the data processing flow is dynamically adjusted to improve the efficiency of subsequent data quality testing; specifically, by analyzing the quality test report output by the model, the main problems in the data and their distribution patterns are identified, and the data processing flow is optimized in a targeted manner; For data sources that often have missing values ​​or outliers, you can dynamically adjust the data collection strategy, strengthen real-time verification during the collection process, or directly optimize the data cleaning rules; Dynamically optimize data preprocessing steps for data with large feature distribution deviations; Dynamically update quality inspection rules based on feedback results; dynamic adjustment of the overall process continuously improves the adaptability and processing capabilities of the large model to different data scenarios through a closed-loop feedback mechanism, thereby improving the accuracy and efficiency of data quality inspection.

10. A data quality detection system based on a large model, characterized in that: The method for automatically implementing the data quality detection method based on a large model according to any one of claims 1 to 9 comprises: A data input module, used for receiving data to be detected; A data preprocessing module, used for cleaning and feature extraction of the data to be detected; A data detection module, used for performing quality detection on the data according to the fine-tuned large model; The result output module is used to output the test results and perform feedback control based on the results.

Images