Structured data quality evaluation and optimization system using large language model

Abstract

The invention discloses a structured data quality evaluation and optimization system using a large language model, which comprises a data access module used for formatting and preprocessing data, a data quality detection module used for establishing data quality evaluation indexes, performing statistical analysis and rule verification to detect the data and calculating the overall quality grade, and a big language model, the semantic analysis and SQL generation module is used for analyzing a database structure through a large language model, converting a natural language instruction into an executable SQL statement and is based on the large language model, and the anomaly detection and optimization suggestion module is used for positioning anomaly by utilizing a mixed detection algorithm and generating natural language interpretation and optimization suggestions in combination with the large language model. And the user feedback and dynamic adjustment module is used for absorbing adjustment suggestions through a feedback loop mechanism, and optimizing a prompt project and an SQL generation strategy. According to the method, semantic analysis is carried out on a database structure by adopting a large language model, and high-precision SQL query is automatically generated.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and in particular to a structured data quality assessment and optimization system utilizing a large language model. Background Technology

[0002] In today's data-driven era, structured data plays a crucial role in business decision-making, operational optimization, and new technology development. However, data quality issues consistently hinder the accuracy and timeliness of data insights. Traditional data quality management methods, such as data cleaning, manual verification, and rule writing, often face challenges related to large data volumes, high dimensionality and complexity, and dynamic business needs.

[0003] Data quality directly impacts the accuracy of corporate decision-making and operational efficiency. Low-quality data can lead to erroneous analytical results, wasted resources, and biased decision-making. According to relevant research reports, low-quality data can cost companies millions of dollars annually. Traditional data quality management relies on manually writing rules, data cleaning, and validation processes, which often proves inadequate when dealing with large-scale data. Furthermore, data heterogeneity, data silos, and data redundancy further complicate data management, forcing companies to seek more efficient and automated solutions.

[0004] With the emergence of Large Language Models (LLMs), such as Qwen and DeepSeek, their remarkable capabilities in natural language understanding, generation, and reasoning have provided new insights for data quality optimization. Large Language Models can parse complex semantics, dynamically generate SQL queries, and provide natural language interpretations, thereby enabling automatic detection and remediation suggestions for data quality anomalies. However, existing research and practical applications show that relying solely on LLMs to generate SQL queries often results in inaccurate or abstract query results due to a lack of in-depth understanding of structured data. Therefore, a semantic reasoning layer is needed to compensate for this deficiency. Summary of the Invention

[0005] To address the aforementioned shortcomings, this invention proposes a structured data quality assessment and optimization system utilizing a large language model. This system employs a large language model to perform deep semantic analysis of the database structure, enabling the automatic generation of highly accurate SQL queries and overcoming the problem of inaccurate semantics in traditional SQL generation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a structured data quality assessment and optimization system utilizing a large language model, comprising, in sequence, the following components: The data access module is used to access structured data from multiple source systems and perform preliminary formatting and preprocessing using extraction, transformation, and loading techniques. The data quality detection module establishes data quality assessment indicators, including accuracy, completeness, consistency, timeliness, uniqueness, and effectiveness. It uses data quality detection algorithms for statistical analysis and rule verification to conduct preliminary data detection, and calculates the overall quality level through a weighted comprehensive score formula. The semantic parsing and SQL generation module based on the large language model parses the database structure, namely tables, fields and relationships, and transforms natural language instructions into executable SQL statements. The generation process includes: constructing a database knowledge graph, calculating entity relationship matrix and field semantic vectors; and dynamically generating SQL templates based on intent probability distribution and semantic matching degree. The anomaly detection and optimization suggestion module uses a hybrid detection algorithm, namely statistical methods and machine learning algorithms, to locate anomalies, and combines a large language model to generate natural language explanations and optimization suggestions. The user feedback and dynamic adjustment module incorporates user feedback through a feedback loop mechanism to optimize prompts and SQL generation strategies, enabling the system to learn autonomously and continuously improve.

[0007] As an improvement, preprocessing specifically includes the following steps: Transform structured data into a standard format to unify the data format and perform preliminary verification; Basic marking is performed on anomalies, including but not limited to missing data and null values, to complete data integrity checks and generate a preliminary quality report; Record the data source, format, update frequency, and field description to complete metadata collection, thereby providing basic information for subsequent semantic parsing.

[0008] As an improvement, the accuracy of the data is checked by comparing it with external benchmark data or historical data to verify its true reflection; the integrity of the data is checked by checking for missing records or abnormal null values; the consistency of the data is checked by ensuring that the data format, units, and logical relationships between fields are consistent; the timeliness of the data is checked by evaluating the frequency and real-time nature of data updates to ensure that it reflects the latest information; the uniqueness of the data is checked by checking for duplicate data and merging or deduplicating it; and the validity of the data is checked by checking whether the data conforms to the preset format and range restrictions.

[0009] As an improvement, the data quality detection algorithm specifically includes the following steps: Let the dataset... Include 1 record One field: Accuracy testing for numeric fields Calculate the accuracy based on statistical anomalies For fields with reference standards, the accuracy of calculations based on reference values ​​is... The final accuracy quality score is calculated. : For numeric fields Using the IQR method: Among them, the set of outliers ; For fields with reference standards: in, For reference only. , It is a collection of numeric fields. A set of standard fields with parameters; Integrity checks begin with the fields. Single field integrity calculation is performed to obtain , and then through Calculate the overall integrity of the dataset Ultimately based on The integrity quality score was calculated. : in, For fields The number of null values ​​in the data. , For indicator functions, This is the integrity threshold, with a value ranging from 0.9 to 0.98. Consistency checks begin with the fields. Consistency measurement is obtained by detecting format consistency. Secondly, regarding the relevant fields and Detect cross-field consistency The final consistency quality score is obtained. : Among them, for the field Let its format pattern set be Dominant mode , ; Timeliness detection first checks the freshness of the data. Secondly, calculate the proportion of expired data. Finally, the timeliness quality score is calculated. : in, For timestamp fields, For the current time, For attenuation parameters, This is the expiration threshold; Uniqueness check, for fields Calculate the uniqueness of a single field; for combinations of fields... Calculate the uniqueness of the combined fields, and finally calculate the uniqueness quality score. : in, And remove null values ​​from the set. This is the set of values ​​after deduplication. Validation checks, for fields and verification rule set Calculate rule-based effectiveness For fields Overall effectiveness calculation Finally, the effectiveness quality score is calculated. : in, For rules The weights; The overall quality score is obtained first based on the accuracy, completeness, consistency, timeliness, uniqueness, and effectiveness of the tests. The weighted composite score is calculated based on the dimensional weights. , and then according to Classify by quality level: in, , For dimension weights, satisfying .

[0010] As an improvement, , , , , , .

[0011] As an improvement, the data quality detection algorithm also includes anomaly score calculation and dynamic threshold adjustment, specifically including the following steps: Classify the severity of problems based on the quality dimension d and field j: Abnormal score calculation: in, , ; Thresholds based on historical data: Confidence interval for quality score: in, This represents the historical average quality score. The standard deviation of historical quality scores. To adjust the parameters, a value of 1.5-2.0 is typically used. This is the critical value for the standard normal distribution.

[0012] As an improvement, the semantic parsing and SQL generation module based on the large language model includes a database structure parsing unit, a user instruction semantic interpretation unit, a dynamic SQL generation unit, and an SQL query verification and optimization unit. Based on the above units, the system is integrated and optimized, and evaluation indicators are generated.

[0013] As an improvement, the semantic parsing and SQL generation module based on a large language model specifically includes the following steps: The database structure parsing unit constructs a database knowledge graph and evaluates database complexity. Set up a database Containing table collection Each table Contains a collection of fields ; Constructing an entity relationship matrix : in, For table With table The strength of the relationship between them This is the semantic association threshold. For semantic similarity between tables; The semantic vector is represented as: in For table Middle field semantic vectors, For field name characteristics, For field type characteristics, Annotate features and weight parameters for fields. ; Semantic similarity between fields is represented as follows: in, :surface fields in , :surface fields in , : Semantic vector of the corresponding field, similarity range: ; Edge weights in a knowledge graph: in, The structure refers to the weights of primary key and foreign key relationships. The semantic similarity weight is the similarity weight of the field names. To use frequency weights, ; Structured weights: Semantic weight: Frequency weighting: Database structure complexity: in, This represents the total number of tables in the database. For table Number of fields Elements of the inter-table relationship matrix; The user command semantic interpretation unit identifies natural language intent and calculates semantic similarity: First, classify the query intent, assuming the user query... The probability distribution of the intent is as follows: in User query text, Intent category, The set of all possible intent categories. Large language model for queries Belongs to intention The rating; Intent category definition: Next, entity recognition and mapping are performed, with entity recognition probability as follows: in, For the first One candidate entity, For entity identification weight matrix, For bias vectors, For BERT model queries Middle position The expression, It is the sigmoid activation function; Entity-to-database field mapping score: in, For the identified entities, For database fields, For the entity's embedding vector, This is the semantic vector of the field.

[0014] Context weight function: Entity importance calculation: The constraints are Then, multi-layer semantic matching is performed to determine the semantic match between the user query and the database elements: in, , The semantic field consists of three layers: lexical, syntactic, and semantic. For the first Layer weights; The constraints are ; The dynamic SQL generation unit yields a probabilistic model for SQL structure generation and an algorithm for constructing SQL statements. First, select an SQL template. The probability of selecting an SQL template is as follows: in, For the first One SQL template, For user intent, The set of identified entities; Template scoring function: in, For SQL templates, For user intent, For a collection of entities; The constraints are ; Next, the JOIN operation decision is made, using a multi-table join decision function: in, For table and The hidden state is represented by, This is a vector representing the relationships between tables. The JOIN decision weight matrix is... For bias vectors, This is a vector concatenation operation; The SQL query validation and optimization unit validates syntax correctness, optimizes query performance, and provides feedback for learning. First, check for syntax errors. Syntax error probability: in, For the first in the SQL statement One token, For the front A sequence of tokens The total number of tokens used in the SQL statements; Conditional probability calculation Next, query the execution cost model: in, For operation unit cost The amount of data processed. For the selectivity of operation; And evaluate index utilization. Index utilization: in, For the first The number of times an index is used. For the first The weight of each index, For the first The availability of each index, For the first The maximum number of times an index can be used; Generate evaluation metrics, including SQL generation accuracy. Semantic understanding accuracy System response time User satisfaction : Where N is the total number of test samples, I(·) is an indicator function that returns 1 when the condition is true and 0 otherwise, Execute(SQL_i) is the execution result of the i-th SQL statement, and Expected_i is the expected result of the i-th test sample; The value range is [0,1], and the closer the value is to 1, the higher the accuracy. Where Correct is the set of samples with correct semantic understanding, Predicted is the set of samples with semantic understanding predicted by the system, |·| is the cardinality (number of elements) of the set, and ∩ is the intersection operation of the sets; Among them, T parse For natural language parsing time, T generate T represents the SQL generation time. validate For SQL verification time, T execute SQL execution time; Where M is the number of users participating in the evaluation, and Relevance j For the j-th user's rating of the relevance of the result, Usability j Let Accuracy be the rating of the system's usability by the j-th user. j Let be the rating of the j-th user for the accuracy of the result.

[0015] As an improvement, basic statistical anomaly detection is performed using statistical methods, including Z-Score anomaly detection, interquartile range anomaly detection, and quantile-based anomaly detection. Advanced machine learning anomaly detection is then performed using machine learning algorithms. This advanced machine learning anomaly detection uses the Isolation Forest algorithm to calculate anomaly scores and expected path lengths for anomaly determination. Subsequently, it detects autoencoder anomalies by calculating reconstruction errors and anomaly scores. For anomalies with significant deviations from standard values, the data input source is corrected, and data is re-collected. For anomalies with a large number of null values ​​or missing records, a data imputation mechanism is added. For anomalies where the same field has inconsistent formats in different tables, a unified data format is established, and standard specifications are defined. For anomalies causing data redundancy due to duplicate records, a deduplication algorithm is implemented, and primary key constraints are established. The system generates anomaly reports and suggestions, which are then fed back to the user interface, allowing users to manually adjust the data processing flow. This feedback is further used for model fine-tuning and prompting engineering parameter updates to ensure that the next round of detection is more targeted and accurate.

[0016] As an improvement, a user feedback interface is embedded in the system, allowing users to evaluate the query results and optimization suggestions generated by the system. Based on the feedback, the system makes the following self-adjustments: First, based on the shortcomings described by users, the prompt engineering is adjusted and the prompt content is redesigned to improve the accuracy of LLM understanding; the detection rules are fine-tuned and the rule engine is updated to improve the statistical analysis algorithm; the LLM is retrained using feedback data to fine-tune the model and enhance its adaptability to specific industry data.

[0017] Compared with the prior art, the advantages of the present invention are as follows: This invention achieves automated processing without human intervention by integrating a closed-loop process design encompassing data access, preprocessing, quality detection with an automatic quality scoring algorithm, anomaly optimization, and feedback adjustment. The LLM semantic hub utilizes knowledge graph construction, entity relation matrix calculation, and semantic vector matching technologies to achieve deep analysis of the database structure and precise conversion of natural language commands, automatically generating highly accurate SQL queries. This solves the problem of inaccurate queries caused by semantic ambiguity in traditional SQL generation (such as field mapping and time constraint parsing), reducing the rate of missed anomaly detection. Simultaneously, it realizes an end-to-end automated data governance system, significantly improving data cleaning and verification efficiency. The fusion of statistical rigor and semantic interpretability relies on the quantitative analysis capabilities of Z-Score / IQR statistical testing and the qualitative reasoning capabilities of LLM semantic parsing, achieving a dual breakthrough in anomaly detection: precise numerical localization and business interpretability. Furthermore, it upgrades traditional algorithms by combining a rule engine with a large model, such as dynamic weight configuration, overcoming the limitations of fixed thresholds. The quality score is obtained through accuracy, completeness, consistency, timeliness, uniqueness, and validity checks. The weighted composite score is calculated based on the dimensional weights. This approach enables a six-dimensional quality scoring system to quantify quality assessment, utilizing a hybrid detection algorithm combining statistical and machine learning methods to locate anomalies and improve the identification rate of unknown anomalies. The system employs a three-pronged approach: structured data transformation (JSON to SQL tables), anomaly marking (missing value imputation strategy), and metadata recording (data lineage tracing), building an automated ETL pipeline to reduce manual intervention. Furthermore, a dual-channel feedback mechanism—explicit user feedback (marking unreasonable suggestions) + implicit system feedback (query success rate statistics)—links model fine-tuning and uses feedback data to update the knowledge graph for continuous LLM training, thereby maintaining optimal data quality management performance in various environments. Through architectural innovation, algorithmic breakthroughs, and process refactoring, this patent constructs a three-in-one technological barrier of "automation-intelligence-adaptation" in the field of data quality management, particularly suitable for high-value, high-complexity, and highly dynamic structured data processing scenarios. Attached Figure Description

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Figure 1 This is a diagram illustrating the overall architecture of a structured data quality assessment and optimization system utilizing a large language model. Figure 2 This is a schematic diagram of the workflow of a structured data quality assessment and optimization system that utilizes a large language model. Figure 3 This is a flowchart of a structured data quality assessment and optimization system that utilizes a large language model. Detailed Implementation

[0019] like Figures 1 to 3As shown, a structured data quality assessment and optimization system utilizing a large language model includes, in sequence, a data access module, a data quality detection module, a semantic parsing and SQL generation module based on a large language model, an anomaly detection and optimization suggestion module, and a user feedback and dynamic adjustment module. The data access module accesses structured data from multiple source systems and performs preliminary formatting and preprocessing using extraction, transformation, and loading techniques. The data quality detection module establishes data quality assessment indicators, including accuracy, completeness, consistency, timeliness, uniqueness, and validity. It uses data quality detection algorithms for statistical analysis and rule validation to perform preliminary data detection and calculates the overall quality using a weighted comprehensive score formula. The system is divided into several modules: a semantic parsing and SQL generation module based on a large language model, which parses the database structure (tables, fields, and relationships) using the large language model and converts natural language instructions into executable SQL statements. The generation process includes: constructing a database knowledge graph, calculating the entity relationship matrix and field semantic vectors; and dynamically generating SQL templates based on intent probability distribution and semantic matching degree. The anomaly detection and optimization suggestion module uses a hybrid detection algorithm (statistical methods and machine learning algorithms) to locate anomalies and combines this with the large language model to generate natural language explanations and optimization suggestions. The user feedback and dynamic adjustment module incorporates user feedback through a feedback loop mechanism to optimize the prompting process and SQL generation strategy, enabling the system to learn autonomously and continuously optimize.

[0020] I. Data Access and Preprocessing The system first connects to multiple data sources, such as internal enterprise databases, data warehouses, and external data interfaces, through the data access module, and uses ETL (Extract, Transform, Load) technology to perform preliminary data cleaning and standardization.

[0021] During the preprocessing stage, the system employs the following steps: Data format standardization: Transform structured data into a standard format and perform preliminary verification.

[0022] Data integrity check: Marks out anomalies such as missing data and null values, and generates a preliminary quality report.

[0023] Metadata collection: Records data source, format, update frequency, and field descriptions to provide basic information for subsequent semantic parsing.

[0024] Table 1. Main Steps and Functions of Data Access and Preprocessing II. Data Quality Detection Algorithm After the data is preprocessed, the system will use its built-in rule engine and statistical methods to perform multi-dimensional quality checks on the data. The detection dimensions mainly include: Accuracy: Verify the true reflectiveness of the data by comparing it with external benchmark data or historical data; Completeness: Check if there are missing records or abnormal null values ​​in the dataset; Consistency: Ensure that the data format, units, and logical relationships between fields are consistent; Timeliness: Assess the frequency and real-time nature of data updates to ensure that they reflect the latest information; Uniqueness: Detect duplicate data and perform merging or deduplication; Validity: Check whether the data conforms to the preset format and range limits; The system uses a combination of SQL queries and statistical analysis to sample data and detect anomalies for various indicators, outputting detailed quality reports and problem lists. The data quality detection algorithm formula is as follows: 1. Basic Definitions Given a dataset Include 1 record One field: in, Indicates the first 1 record Indicates the first The first record Each field value.

[0025] 2. Completeness Inspection 2.1 Single Field Integrity For fields The integrity metric is: in, = field The number of null values ​​in the data. , This is an indicator function.

[0026] 2.2 Overall Integrity Overall completeness of the dataset: 2.3 Integrity Quality Score in, This is the integrity threshold (set to 0.95 in this project).

[0027] 3. Accuracy Testing 3.1 Accuracy based on statistical anomalies For numeric fields Using the IQR method: Outlier set: Accuracy measurement: 3.2 Accuracy based on reference values For fields with reference standards: in, For reference only.

[0028] 3.3 Accuracy Quality Score in, , It is a collection of numeric fields. This is a set of standard fields with parameters.

[0029] 4. Consistency Check 4.1 Format consistency For fields Let its format pattern set be : Dominant mode: Consistency measure: 4.2 Cross-field consistency For the relevant fields and Consistency constraints : 4.3 Consistency Quality Score 5. Timeliness Detection 5.1 Data Freshness Let the timestamp field be The current time is : in, This is the attenuation parameter.

[0030] 5.2 Proportion of Expired Data The expiration threshold is : 5.3 Timeliness Quality Score 6. Uniqueness Detection 6.1 Single-field uniqueness For fields : in, (Remove null values) This is the set of values ​​after deduplication.

[0031] 6.2 Uniqueness of Combined Fields For field combinations : 6.3 Uniqueness Quality Score 7. Validity Check 7.1 Rule-based validity For fields and verification rule set : Fields Overall effectiveness: in, For rules The weight.

[0032] 7.2 Effectiveness Quality Score 8. Calculation of Overall Quality Score 8.1 Weighted Composite Score in, , For dimension weights, satisfying 8.2 Recommendation Weight Allocation 8.3 Quality Grade Classification 9. Anomaly detection threshold 9.1 Classification of Problem Severity For quality dimension d and field j: Recommended threshold , .

[0033] 9.2 Calculation of Outlier Scores in: 10. Dynamic threshold adjustment 10.1 Thresholds based on historical data in, This represents the historical average quality score. The standard deviation of historical quality scores. To adjust the parameters (usually set to 1.5-2.0) 10.2 Confidence Interval Calculation Confidence interval for quality score: in, This is the critical value for the standard normal distribution.

[0034] III. Semantic Parsing and SQL Generation Based on Large Language Models Traditional SQL generation often suffers from inaccurate semantics and simplistic query structures. This system introduces a large language model to perform deep semantic analysis of the structured information in the database, enabling dynamic data quality detection SQL generation and execution for database tables. The main steps include: 1. Database structure parsing module 1.1 Database Knowledge Graph Construction Set up a database Containing table collection Each table Contains a collection of fields .

[0035] 1.1.1 Entity Relationship Matrix Constructing an entity relationship matrix : in, For table With table The strength of the relationship between them The semantic association threshold is suggested to be 0.75. For semantic similarity between tables; 1.1.2 Field Semantic Vectorization For fields Its semantic vector representation is as follows: in For table Middle field semantic vectors, For field name characteristics, For field type characteristics, Annotate features and weight parameters for fields. ; Suggested weights (Field name weight) (Field type weight) (Field annotation weight); Semantic similarity between fields: in, :surface fields in , :surface fields in , : Semantic vector of the corresponding field, similarity range: ; 1.1.3 Knowledge Graph Weight Calculation Edge weights in a knowledge graph: in, For structural weights (primary key and foreign key relationships). Semantic weight (field name similarity). To use frequency weights, .

[0036] Structured weights: Semantic weight: Frequency weighting: 1.2 Database Complexity Assessment Database structure complexity: in This represents the total number of tables in the database. For table Number of fields Elements of the inter-table relationship matrix; 2. User command semantic interpretation module 2.1 Natural Language Intent Recognition 2.1.1 Query Intent Classification Suppose a user query The probability distribution of the intent is as follows: in, User query text, Intent category, The set of all possible intent categories. Large language model for queries Belongs to intention The rating; Intent category definition: 2.1.2 Entity Recognition and Mapping Entity recognition probability: in, For the first One candidate entity, For entity identification weight matrix, For bias vectors, For BERT model queries Middle position The expression, It is the sigmoid activation function; Entity-to-database field mapping score: in, For the identified entities, For database fields, For the entity's embedding vector, This is the semantic vector of the field.

[0037] Context weight function: Entity importance calculation: The constraints are .

[0038] 2.2 Semantic Similarity Calculation 2.2.1 Multi-layer semantic matching Semantic match between user queries and database elements: in, , The semantic field consists of three layers: lexical, syntactic, and semantic. For the first Layer weights; The constraints are 3. Dynamic SQL Generation Module 3.1 SQL Structure Generation Probabilistic Model 3.1.1 SQL Template Selection SQL template selection probability: in, For the first One SQL template, For user intent, For the identified entity set Template scoring function: in, For SQL templates, For user intent, For entity collection The constraints are ; 3.1.2 JOIN Operation Decision Multi-table join decision function: in, For table and The hidden state is represented by, This is a vector representing the relationships between tables. The JOIN decision weight matrix is... For bias vectors, This is a vector concatenation operation; 4. SQL Query Validation and Optimization Module 4.1 Syntax Correctness Verification 4.1.1 Syntax Error Detection Syntax error probability: in, For the first in the SQL statement One token, For the front A sequence of tokens The total number of tokens used in the SQL statements; Conditional probability calculation 4.2 Query Performance Optimization 4.2.1 Execution Cost Estimation Query execution cost model: in, For operation unit cost The data base for the operation, Selectivity factor 4.2.2 Index Utilization Evaluation Index utilization: in, For the first The number of times an index is used. For the first The weight of each index, For the first The availability of each index, For the first The maximum number of times an index can be used; 5. Evaluation Indicators 5.1 SQL generation accuracy Where N is the total number of test samples, I(·) is an indicator function that returns 1 when the condition is true and 0 otherwise. Execute(SQL i ) represents the execution result of the i-th SQL statement, Expected i Let be the expected result for the i-th test sample; 5.2 Semantic Understanding Accuracy Where Correct is the set of samples with correct semantic understanding, Predicted is the set of samples with semantic understanding predicted by the system, |·| is the cardinality (number of elements) of the set, and ∩ is the intersection operation of the sets; The value range is [0,1], and the closer the value is to 1, the higher the precision. 5.3 System Response Time Among them, T parseT represents the natural language parsing time (in milliseconds). generate T represents the SQL generation time (in milliseconds). validate T represents the SQL validation time (in milliseconds). execute SQL execution time (in milliseconds); 5.4 User Satisfaction Where M is the number of users participating in the evaluation, and Relevance j For the j-th user's rating of the relevance of the result, Usability j Let Accuracy be the rating of the system's usability by the j-th user. j Let j be the rating of the accuracy of the result for the j-th user; IV. Anomaly Detection and Optimization Suggestion Generation After executing SQL queries and obtaining data quality results, the system uses built-in anomaly detection algorithms to perform in-depth analysis of data anomalies. The main strategies include: Outlier detection: Statistical methods (such as Z-Score) and machine learning algorithms (such as IsolationForest and autoencoder anomaly detection) are used to detect outliers in the data. The following are algorithms for detecting outlier data quality: 1. Basic Statistical Anomaly Detection Methods 1.1 Z-Score Anomaly Detection 1.1.1 Standard Z-Score For data points Its Z-Score is defined as: in, (Sample mean) (Sample standard deviation).

[0039] Anomaly detection criteria: generally or 1.1.2 Modify Z-Score (MAD-based) Robust Z-Score based on median absolute deviation: in, = Median of the data, (Absolute deviation of median) 1.2 Interquartile Range (IQR) Anomaly Detection 1.2.1 Standard IQR Method Abnormal boundary: Anomaly detection: generally (Mild abnormality) or (Extreme anomaly) 1.2.2 Adjusting the IQR Method Considering the adjustment factor for data distribution skewness: Including skewness: 1.3 Quantile-based anomaly detection 1.3.1 Percentile Method Exception definition: generally (i.e., P5 and P95) 1.3.2 Dynamic quantile threshold in The adjustment factor is typically set between 1.5 and 2.0. 2. Advanced Machine Learning Anomaly Detection Algorithms 2.1 Isolation Forest Algorithm 2.1.1 Calculation of Abnormal Scores For data points Its abnormal score is: in, For data points Average path length across all isolation trees The average path length of a binary search tree. For harmonic numbers ( (Euler's constant) 2.1.2 Path Length Expectation Path length in a single isolation tree: in, This represents the current path length. The number of samples in the current node; 2.1.3 Anomaly Detection generally 2.2 Anomaly Detection of Self-Encoder 2.2.1 Reconstruction Error 2.2.2 Abnormal Score in, and These are the mean and standard deviation of the reconstruction error, respectively.

[0040] Root cause analysis: Based on the detected anomaly information, the system automatically generates a natural language description through LLM to explain the possible causes of the anomaly and provides corresponding optimization suggestions.

[0041] Table 2 Examples of Anomaly Detection Indicators and Optimization Suggestions Feedback loop mechanism: The system will send the generated anomaly reports and suggestions to the user interface, allowing users to manually adjust the data processing flow. This feedback will also be used to fine-tune the system's model and prompt engineering parameter updates, ensuring that the next round of detection is more targeted and accurate.

[0042] V. User Feedback and Dynamic System Optimization The system includes a user feedback interface, allowing users to evaluate the query results and optimization suggestions generated by the system. Based on the feedback, the system makes the following self-adjustments: The prompts have been redesigned based on user feedback to improve the accuracy of LLM comprehension. Rule engine update: Fine-tuning of detection rules and improvement of statistical analysis algorithms; Model fine-tuning: Retraining the LLM using feedback data to enhance its adaptability to industry-specific data.

[0043] This dynamic feedback mechanism ensures that the system maintains a high level of data quality management in different data environments and continuously evolves to adapt to changes in business needs.

[0044] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0045] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor, and the functions described above can be performed at least in part by one or more hardware logic components.

[0046] The above description only illustrates the preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. All modifications made within the scope of the independent claims of this invention are also within the scope of protection of this invention.

Claims

1. A structured data quality assessment and optimization system utilizing a large language model, characterized in that: In order to include The data access module is used to access structured data from multiple source systems and perform preliminary formatting and preprocessing using extraction, transformation, and loading techniques. The data quality detection module establishes data quality assessment indicators, which include accuracy, completeness, consistency, timeliness, uniqueness, and effectiveness. It uses data quality detection algorithms for statistical analysis and rule verification to perform preliminary detection on the data, and calculates the overall quality level through a weighted comprehensive score formula. The semantic parsing and SQL generation module based on the large language model parses the database structure, namely tables, fields and relationships, and transforms natural language instructions into executable SQL statements. The generation process includes: constructing a database knowledge graph, calculating entity relationship matrix and field semantic vectors; and dynamically generating SQL templates based on intent probability distribution and semantic matching degree. The anomaly detection and optimization suggestion module uses a hybrid detection algorithm, namely statistical methods and machine learning algorithms, to locate anomalies, and combines a large language model to generate natural language explanations and optimization suggestions. The user feedback and dynamic adjustment module incorporates user feedback through a feedback loop mechanism to optimize prompts and SQL generation strategies, enabling the system to learn autonomously and continuously improve.

2. The structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: Preprocessing specifically includes the following steps: Transform structured data into a standard format to unify the data format and perform preliminary verification; Basic marking is performed on anomalies, including but not limited to missing data and null values, to complete data integrity checks and generate a preliminary quality report; Record the data source, format, update frequency, and field description to complete metadata collection, thereby providing basic information for subsequent semantic parsing.

3. The structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: The accuracy of the data is verified by comparing it with external benchmark data or historical data; the completeness of the data is verified by checking for missing records or abnormal null values ​​in the dataset. Data consistency is checked by ensuring consistency in data format, units, and logical relationships between fields; timeliness is checked by evaluating the frequency and real-time nature of data updates to ensure that the data reflects the latest information; uniqueness is checked by checking for duplicate data and merging or deduplicating it; and validity is checked by checking whether the data conforms to preset format and range restrictions.

4. The structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: The data quality detection algorithm specifically includes the following steps: (Set the dataset...) Include 1 record One field, , , Accuracy testing for numeric fields Calculate the accuracy based on statistical anomalies For fields with reference standards, the accuracy of calculations based on reference values ​​is... The final accuracy quality score is calculated. : For numeric fields Using the IQR method: Among them, the set of outliers ； For fields with reference standards: in, For reference only. , It is a collection of numeric fields. A set of standard fields with parameters; Integrity checks begin with the fields. Single field integrity calculation is performed to obtain , and then through Calculate the overall integrity of the dataset Ultimately based on The integrity quality score was calculated. : in, For fields The number of null values ​​in the data. , For indicator functions, This is the integrity threshold, with a value ranging from 0.9 to 0.

98. Consistency checks begin with the fields. Consistency measurement is obtained by detecting format consistency. Secondly, regarding the relevant fields and Detect cross-field consistency The final consistency quality score is obtained. : Among them, for the field Let its format pattern set be Dominant mode , ; Timeliness detection first checks the freshness of the data. Secondly, calculate the proportion of expired data. Finally, the timeliness quality score is calculated. : in, For timestamp fields, For the current time, For attenuation parameters, This is the expiration threshold; Uniqueness check, for fields Calculate the uniqueness of a single field; for combinations of fields... Calculate the uniqueness of the combined fields, and finally calculate the uniqueness quality score. : in, And remove null values ​​from the set. This is the set of values ​​after deduplication. Validation checks, for fields and verification rule set Calculate rule-based effectiveness For fields Overall effectiveness calculation Finally, the effectiveness quality score is calculated. : in, For rules The weights; The overall quality score is obtained first based on the accuracy, completeness, consistency, timeliness, uniqueness, and effectiveness of the tests. The weighted composite score is calculated based on the dimensional weights. Then according to Classify by quality level: in, , For dimension weights, satisfying .

5. The structured data quality assessment and optimization system utilizing a large language model according to claim 4, characterized in that: ， ， ， ， ， 。 6. The structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: The data quality detection algorithm also includes anomaly score calculation and dynamic threshold adjustment, specifically including the following steps: Classify the severity of problems based on the quality dimension d and field j: Abnormal score calculation: in, , ; Thresholds based on historical data: Confidence interval for quality score: in, This represents the historical average quality score. The standard deviation of historical quality scores. To adjust the parameters, a value of 1.5-2.0 is typically used. This is the critical value for the standard normal distribution.

7. The structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: The semantic parsing and SQL generation module based on the large language model includes a database structure parsing unit, a user instruction semantic interpretation unit, a dynamic SQL generation unit, and an SQL query verification and optimization unit. Based on the above units, the system is integrated and optimized, and evaluation indicators are generated.

8. A structured data quality assessment and optimization system utilizing a large language model according to claim 7, characterized in that: The semantic parsing and SQL generation module based on the large language model specifically includes the following steps: The database structure parsing unit constructs a database knowledge graph and evaluates database complexity. Set up a database Containing table collection Each table Contains a collection of fields ; Constructing an entity relationship matrix : in, For table With table The strength of the relationship between them This is the semantic association threshold. For semantic similarity between tables; The semantic vector is represented as: in For table Middle field semantic vectors, For field name characteristics, For field type characteristics, Annotate features and weight parameters for fields. ; Semantic similarity between fields is represented as follows: in, :surface fields in , :surface fields in , : Semantic vector of the corresponding field, similarity range: ; Edge weights in a knowledge graph: in, The structure refers to the weights of primary key and foreign key relationships. The semantic similarity weight is the similarity weight of the field names. To use frequency weights, ; Structured weights: Semantic weight: Frequency weighting: Database structure complexity: in, This represents the total number of tables in the database. For table Number of fields Elements of the inter-table relationship matrix; The user command semantic interpretation unit identifies natural language intent and calculates semantic similarity: First, classify the query intent, assuming the user query... The probability distribution of the intent is as follows: in User query text, Intent category, The set of all possible intent categories. Large language model for queries Belongs to intention The rating; Intent category definition: Next, entity recognition and mapping are performed, with entity recognition probability as follows: in, For the first One candidate entity, Weight matrix for entity identification, For bias vectors, For BERT model queries Middle position The expression, It is the sigmoid activation function; Entity-to-database field mapping score: in, For the identified entities, For database fields, For the entity's embedding vector, This is the semantic vector of the field; Context weight function: Entity importance calculation: The constraints are Then, multi-layer semantic matching is performed to determine the semantic match between the user query and the database elements: in, , The semantic field consists of three layers: lexical, syntactic, and semantic. For the first Layer weights; The constraints are ; The dynamic SQL generation unit yields a probabilistic model for SQL structure generation and an algorithm for constructing SQL statements. First, select an SQL template. The probability of selecting an SQL template is as follows: in, For the first One SQL template, For user intent, The set of identified entities; Template scoring function: in, For SQL templates, For user intent, For a collection of entities; The constraints are ; Next, the JOIN operation decision is made, using a multi-table join decision function: in, For table and The hidden state is represented by, This is a vector representing the relationships between tables. The JOIN decision weight matrix is... For bias vectors, This is a vector concatenation operation; The SQL query validation and optimization unit validates syntax correctness, optimizes query performance, and provides feedback for learning. First, check for syntax errors. Syntax error probability: in, For the first in the SQL statement One token, For the front A sequence of tokens The total number of tokens used in the SQL statements; Conditional probability calculation Next, query the execution cost model: in, For operation unit cost The amount of data processed. For the selectivity of operation; And evaluate index utilization. Index utilization: in, For the first The number of times an index is used. For the first The weight of each index, For the first The availability of each index, For the first The maximum number of times an index can be used; Generate evaluation metrics, including SQL generation accuracy. Semantic understanding accuracy System response time User satisfaction : Where N is the total number of test samples, I(·) is an indicator function that returns 1 when the condition is true and 0 otherwise, Execute(SQL_i) is the execution result of the i-th SQL statement, and Expected_i is the expected result of the i-th test sample; The value range is [0,1], and the closer the value is to 1, the higher the accuracy. Where Correct is the set of samples with correct semantic understanding, Predicted is the set of samples with semantic understanding predicted by the system, |·| is the cardinality (number of elements) of the set, and ∩ is the intersection operation of the sets; Among them, T parse For natural language parsing time, T generate T represents the SQL generation time. validate T represents the SQL validation time. execute SQL execution time; Where M is the number of users participating in the evaluation, and Relevance j For the j-th user's rating of the relevance of the result, Usability j Let Accuracy be the rating of the system's usability by the j-th user. j Let be the score given by the j-th user for the accuracy of the result.

9. A structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: Basic statistical anomaly detection is performed using statistical methods, including Z-Score anomaly detection, interquartile range anomaly detection, and quantile-based anomaly detection. Advanced machine learning anomaly detection is then employed, using the Isolation Forest algorithm to calculate anomaly scores and expected path lengths for anomaly determination. Subsequently, reconstruction errors and anomaly scores are calculated to detect autoencoder anomalies. For anomalies with significant deviations from standard values, the data input source is corrected, and data is re-collected. For anomalies with a large number of null values ​​or missing records, a data imputation mechanism is added. For anomalies where the same field has inconsistent formats across different tables, a standardized data format is established. For anomalies causing data redundancy due to duplicate records, a deduplication algorithm is implemented, and primary key constraints are established. The generated anomaly reports and suggestions are fed back to the user interface, allowing users to manually adjust the data processing flow. This feedback is further used for model fine-tuning and prompting engineering parameter updates to ensure more targeted and accurate detection in the next round.

10. The structured data quality assessment and optimization system utilizing a large language model according to claim 1, characterized in that: The system embeds a user feedback interface, allowing users to evaluate the query results and optimization suggestions generated by the system. Based on the feedback, the system makes the following self-adjustments: First, based on the shortcomings described by users, the system adjusts the prompts and redesigns the prompt content to improve the accuracy of LLM understanding. Fine-tune the detection rules and update the rule engine to improve the statistical analysis algorithm; The LLM model is retrained using feedback data, and the model is fine-tuned to enhance its adaptability to specific industry data.

Images