A data quality detection method and system

Abstract

This application relates to the field of big data governance technology, specifically providing a data quality detection method and system, including: acquiring sample data for multimodal feature extraction, generating a data fingerprint vector based on statistical feature vectors and semantic feature vectors; constructing complete prompt words containing role settings, task instructions, feature context, and output constraints, inputting the complete prompt words into an LLM (Learning Logic Model), adaptively generating data quality verification rule code adapted to the target business scenario, forming a dynamic rule base; using the dynamic rule base to drive an execution engine to scan the data to be detected, capturing abnormal data items that do not conform to the rules to construct an initial abnormal state vector; mapping the initial abnormal state vector to a pre-constructed data lineage knowledge graph, performing context tracing and propagation analysis, and determining the anomaly identification result. This application utilizes a large language model to automate the closed loop from data feature perception to verification rule generation, achieving adaptive and intelligent data quality management.

Description

Technical Field

[0001] This application relates to the field of big data governance technology, and in particular to a data quality detection method and system. Background Technology

[0002] While massive, multi-source, and heterogeneous data assets bring enormous value to enterprises, they also present severe data quality challenges. The quality of data directly affects the training effect of artificial intelligence models, the accuracy of business decisions, and the efficiency of enterprise operations.

[0003] In existing technologies, data quality inspection mainly relies on rule-based methods. This involves business experts or data engineers pre-defining a series of static validation rules based on experience, such as null value checks, value range constraints, regular expression format matching, and uniqueness constraints. These rules are then scanned and verified using ETL (Extract-Transform-Load) tools or dedicated data quality management platforms. However, this method, which heavily relies on hard-coded rules based on human experience, has significant limitations: firstly, manually compiling rules for massive amounts of business tables and fields is time-consuming, labor-intensive, lacks scalability, and is prone to omissions; secondly, when business logic changes or data distribution naturally shifts, static rules often cannot be updated in a timely manner, leading to numerous false positives or false negatives and extremely high maintenance costs.

[0004] Furthermore, existing data quality inspection technologies are typically based on statistical characteristics or fixed format constraints, lacking a deep understanding and generalization ability regarding the semantics of data business. Existing monitoring systems often view single-point data anomalies in isolation. When an anomaly occurs, the system can only issue an alarm, requiring maintenance personnel to spend a significant amount of time investigating whether it is a source data entry error, data loss during transmission, or logical changes in the business system, resulting in low governance efficiency. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a data quality inspection method and system. By utilizing a large language model to automate the process from data feature perception to verification rule generation, and integrating data lineage knowledge graph technology, the system accurately locates the root causes of anomalies, thereby achieving adaptive and intelligent data quality management.

[0006] To achieve the objectives of this application, the following technical solution is provided: Firstly, this application provides a data quality detection method, including: Acquire sample data and perform multimodal feature extraction on the sample data to generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors; Each component in the data fingerprint vector is converted into natural language description text, and a complete prompt word containing role settings, task instructions, feature context and output constraints is constructed; the complete prompt word is input into LLM to adaptively generate data quality verification rule code adapted to the target business scenario, forming a dynamic rule base; The dynamic rule base drives the execution engine to scan the data to be detected, capture abnormal data items that do not conform to the rules, and parameterize the abnormality type, deviation degree and time dimension of the abnormal data items to construct an initial abnormal state vector. The initial abnormal state vector is mapped to a pre-constructed data lineage knowledge graph; the initial abnormal state vector is then subjected to contextual tracing and propagation analysis based on the data lineage knowledge graph, the root cause confidence of the abnormality is calculated, and the abnormality identification result is determined.

[0007] A further improvement of this application is that the step of extracting multimodal features from the sample data to generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors includes: constructing a basic statistical vector. In the formula, The mean of the sample data. Standard deviation To be the minimum value, The maximum value, The null value rate The number of unique values ​​is used; for discrete data, the Shannon entropy formula is used to calculate the information entropy; for numerical fields, the KL divergence between the distribution of the first digit and the theoretical distribution of Benford's law is calculated; the basic feature vector, the information entropy, and the KL divergence are concatenated to form the statistical feature vector; the context information of the table is constructed into a text sequence; non-empty values ​​are randomly extracted from the sample data and concatenated to form a content sequence; the concatenated text sequence and the content sequence are input into a pre-trained BERT-base-chinese model to obtain the last hidden layer, and the semantic feature vector is obtained by using the vector representation of CLS Token; the statistical feature vector and the semantic feature vector are fused to generate a data fingerprint vector that uniquely represents the sample data; wherein, after the statistical feature vector and the semantic feature vector are concatenated, a fully connected layer is used to map the statistical features to the same high-dimensional space as the semantic features; a self-attention mechanism is introduced to calculate the fusion weights of the statistical feature vector and the semantic feature vector to obtain the weighted fused data fingerprint vector.

[0008] A further improvement of this application is that the step of converting each component in the data fingerprint vector into natural language descriptive text and constructing a complete prompt word containing role settings, task instructions, feature context, and output constraints includes: defining a feature description function. The components in the data fingerprint vector are converted into natural language descriptive text. Specifically, for statistical feature components, their distribution pattern is described; for semantic feature components, they are mapped to business meaning; for entropy features, they are described as information entropy values; and a complete prompt word is constructed, including role settings, task instructions, feature context, and output constraints. Specifically: ; In the formula, This is a complete dynamic Prompt template for the final input LLM; The role setting module is used to determine the system identity, responsibilities, and behavioral constraints undertaken in a task; This is the task instruction module, used to specify the specific task content, objectives, and requirements that need to be executed. For the feature context module, where, This section provides task description information, which explains the object, scenario, and specific content of the current data quality inspection. This is a reference example for sample learning; For modules with output format requirements, specify that the output must be in the format of a Python function or an SQL statement; This represents a text concatenation operation; specifically, a pre-trained embedding model is used to map text descriptions from a pre-built general data quality rule base into a high-dimensional dense vector set, which is then stored in a vector database to establish a knowledge base vector index; subsequently, the data fingerprint vector is used as the query vector. Perform a K-nearest neighbor search on the vector database and compute the query vector. With knowledge base vectors The cosine similarity between the samples is used to select multiple rule samples whose cosine similarity is greater than a preset threshold, forming a reference example for sample learning.

[0009] A further improvement in this application is that the step of inputting the complete prompt word into the LLM to adaptively generate data quality verification rule code adapted to the target business scenario and form a dynamic rule base includes: analyzing the causal relationship between the data features corresponding to the complete prompt word and potential quality problems, and drawing inference conclusions; then generating corresponding data verification function code based on the inference conclusions; performing code syntax parsing and security review on the data verification function code based on abstract syntax tree, and serializing the verified code into binary objects, combining it with the metadata to encapsulate it into standard rule objects, and storing it in the dynamic rule base.

[0010] A further improvement in this application is that the parameterized encoding of the anomaly type, deviation degree, and time dimension of the anomalous data item to construct an initial anomaly state vector includes: for numerical anomaly deviation, under the condition that the rules define an effective interval, using a normalized distance formula based on standard deviation to calculate the relative deviation of the anomaly value; for textual or enumerated anomaly distance, under the condition that the rules define an allowed set of strings or regular expressions, calculating the edit distance between the anomalous text and the expected pattern, and normalizing it to a similarity difference to obtain the deviation of the anomalous text; analyzing the temporal anomalies of the anomalous data item based on a sliding window to obtain the local anomaly rate and the burst index; and constructing the initial anomaly state vector of the anomalous data item. Specifically: ; The definition is as follows: To leverage Graph Embedding to map the table ID and field ID of an anomaly to node vectors in the graph, thus establishing location features; This is a type-based one-hot encoding method used to perform one-hot encoding on the types of rules that are violated. These are the normalized outlier values. The severity features are specifically the relative deviation of the outliers and the deviation of the outlier text. The local anomaly rate and the suddenness index are respectively.

[0011] A further improvement in this application is that the step of combining the data lineage knowledge graph to perform contextual source tracing and propagation analysis on the initial abnormal state vector, calculating the root cause confidence, and completing the anomaly identification includes: parsing the positional features in the initial abnormal state vector and locking the positional features as abnormal anchor nodes in the data lineage knowledge graph; traversing in the reverse direction of the edges starting from the abnormal anchor nodes, setting a maximum hop count, and extracting all upstream dependent nodes corresponding to the starting point; simultaneously performing a pruning strategy to obtain a source tracing subgraph; using a Bayesian inference model on the source tracing subgraph to calculate the posterior probability of each upstream node causing the current anomaly, obtaining the root cause confidence corresponding to each upstream node; and performing Softmax normalization on the root cause confidence of all nodes in the source tracing subgraph to filter out the nodes with the highest probabilities. Each node is used as a candidate root cause set.

[0012] A further improvement in this application is that the step of calculating the posterior probability of each upstream node causing the current anomaly using a Bayesian inference model on the source subgraph to obtain the anomaly root cause confidence level corresponding to each upstream node includes: for each node in the source subgraph, collecting key state feature vectors within the time window before the anomaly occurs. The key state feature vector Includes change characteristics, load characteristics, and quality characteristics; defines nodes. The confidence score of the root cause The confidence score This includes the change impact factor, distance attenuation factor, and propagation path reachability, and its calculation formula is as follows: ; In the formula, This is an indicator function; it is 1 if the node has recently changed, and 0 otherwise. As a time decay term, the closer the change occurred to the anomaly, the greater its impact. This represents the shortest path distance from the node to the anomaly point; the closer the distance, the greater the impact. The historical stability probability of a node; These are the weighting coefficients corresponding to the change impact factor, distance attenuation factor, and propagation path reachability, respectively.

[0013] A further improvement in this application is that the acquisition of sample data includes: reading system tables or metadata dictionaries from the data source to be detected to obtain the basic attributes of the object to be detected; calculating the frequency distribution histogram of the full data to be detected, adding sampling weights to the long-tailed data intervals, and extracting the sample data based on the updated sampling rate weights; wherein, the dynamic calculation of the sampling rate is as follows: ; In the formula, Sampling rate, This represents the total number of rows in the data. Based on the sampling coefficient, The maximum sampling limit is preset; the sample data is then cleaned and standardized preprocessed.

[0014] A further improvement of this application is that it also includes: performing hierarchical governance on the data to be detected based on the anomaly identification results, and displaying anomaly details and governance suggestions to the user through an interactive interface; constructing a feedback dataset based on real-time recorded user feedback, introducing a human feedback reinforcement learning mechanism, and continuously optimizing the complete prompt word template or LLM parameters using the feedback dataset; wherein, the hierarchical governance on the data to be detected based on the anomaly identification results includes: mapping the root cause node type and confidence level in the anomaly identification results to specific governance execution actions based on a governance strategy mapping model.

[0015] Secondly, this application provides a data quality inspection system for implementing the above-mentioned data quality inspection method, the system comprising: The feature extraction module is used to acquire metadata and sample data from the data source to be detected, perform multimodal feature extraction on the sample data, and generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors. The code generation module is used to convert each component in the data fingerprint vector into natural language description text, and construct a complete prompt word containing role settings, task instructions, feature context and output constraints; and input the complete prompt word into LLM to adaptively generate data quality verification rule code adapted to the target business scenario, forming a dynamic rule base; The anomaly encoding module is used to drive the execution engine to scan the data to be detected using a dynamic rule base, capture abnormal data items that do not conform to the rules, and parameterize the anomaly type, deviation degree and time dimension of the abnormal data items to construct an initial anomaly state vector. An anomaly identification module is used to map the initial anomaly state vector to a pre-constructed data lineage knowledge graph; combine the data lineage knowledge graph to perform contextual tracing and propagation analysis on the initial anomaly state vector, calculate the anomaly root cause confidence, and determine the anomaly identification result.

[0016] Thirdly, this application provides a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for performing the methods described above; the processors are used to execute the programs.

[0017] Fourthly, this application provides a computer-readable storage medium having stored thereon a plurality of instructions adapted to be loaded and executed by a processor to implement the steps of the above-described method.

[0018] Compared with the prior art, this application has the following beneficial effects: The data quality inspection method and system provided in this application extracts data fingerprint vectors and constructs complete prompt words containing role settings, task instructions, feature context, and output constraints. These complete prompt words are then input into an LLM (Local Language Model) to adaptively generate data quality verification rule codes adapted to the target business scenario, forming a dynamic rule base. This allows for the automatic judgment of the rationality of fields in specific business scenarios by combining context, and the automatic inference of potential data logic using the general knowledge of a large language model. This effectively discovers implicit logical anomalies that are difficult to define using traditional manual rules, improving rule configuration efficiency by several orders of magnitude. It achieves a leap from passive manual configuration to proactive system discovery. Furthermore, when an anomaly occurs, the intelligent agent extracts a source subgraph by searching and combines it with a Bayesian inference model to calculate the change probability and propagation confidence of each node. This allows for the penetration of complex data links, accurately locating the source of the anomaly, and significantly shortening the troubleshooting time for operations and maintenance personnel. Attached Figure Description

[0019] The accompanying drawings are provided to further understand this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. Figure 1 A schematic diagram of an optional first process for the data quality detection method provided in the embodiments of this application; Figure 2 This is a schematic diagram of an optional second process for the data quality detection method provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this application, unless otherwise stated, "multiple" means two or more.

[0022] With the development of internet technology and the deepening of enterprise digital transformation, data has become the fifth major factor of production after land, labor, capital, and technology. While massive, multi-source, and heterogeneous data assets bring enormous value to enterprises, they also present severe data quality challenges. The quality of data directly affects the training effect of artificial intelligence models, the accuracy of business decisions, and the efficiency of enterprise operations. Therefore, how to efficiently and accurately conduct data quality detection and anomaly identification has become a core issue that urgently needs to be addressed in the field of big data governance.

[0023] In existing technologies, data quality inspection primarily relies on rule-based methods. This involves business experts or data engineers pre-defining a series of static validation rules based on experience, such as null value checks, value range constraints, regular expression format matching, and uniqueness constraints. These rules are then scanned and verified using ETL tools or dedicated data quality management platforms. However, this approach, which heavily relies on hard-coded rules based on human experience, has significant limitations. Firstly, manually compiling rules for massive amounts of business tables and fields is time-consuming, labor-intensive, lacks scalability, and is prone to omissions. Secondly, when business logic changes or data distribution naturally shifts, static rules often fail to update in a timely manner, leading to numerous false positives or false negatives and extremely high maintenance costs.

[0024] Furthermore, existing data quality inspection technologies are typically based on statistical characteristics or fixed format constraints, lacking a deep understanding and generalization ability regarding the semantics of data business. Simultaneously, existing monitoring systems often view single-point data anomalies in isolation, failing to effectively integrate data quality issues with data lineage, system mapping relationships, and upstream / downstream dependencies. When anomalies occur, the system can only issue alarms but cannot provide root cause analysis. Operations personnel must spend significant time investigating whether the anomaly stems from source data entry errors, data loss during transmission, or logical changes in the business system, resulting in inefficient governance.

[0025] To address the aforementioned technical problems, this application proposes the following technical solutions and corresponding embodiments.

[0026] The following is combined Figures 1 to 2 The embodiments shown illustrate the technical solutions of this application: Example 1 This application provides a data quality detection and anomaly identification method, referring to... Figure 1 As shown, the process includes the following steps S101 to S104: Step S101: Obtain sample data and perform multimodal feature extraction on the sample data to generate a data fingerprint vector based on statistical feature vector and semantic feature vector.

[0027] In the embodiments of this application, reference is made to Figure 2As shown, the agent automatically connects to the data source to be detected (heterogeneous data source), cleans and preprocesses the metadata and sample data, and extracts statistical features, semantic features, and structural features through a multimodal feature extraction algorithm, mapping and fusing them into a standardized data fingerprint vector. The agent connects to heterogeneous data sources (including relational databases, NoSQL databases, and API interfaces) through a multi-protocol adapter and executes an adaptive sampling strategy based on the total amount and distribution characteristics of the data to balance computational efficiency and sample representativeness. In this embodiment, the agent is the core execution unit.

[0028] In this embodiment, the intelligent agent connects to heterogeneous data sources through a multi-protocol adapter and executes an adaptive sampling strategy based on the total amount and distribution characteristics of the data. This can be achieved through the following steps S1011 to S1012: Step S10111: The agent reads the system table or metadata dictionary from the heterogeneous data source to obtain the basic attributes of the object to be detected. The basic attributes include table name, field name, data type and comments.

[0029] For fields lacking explicit metadata, this embodiment uses a regular expression library for regular expression matching to automatically infer their physical type. , such as Integer, Float, String, Date, Boolean.

[0030] Step S10112: The agent performs adaptive hierarchical sampling based on the data distribution to obtain a sample set.

[0031] In this embodiment, to avoid the loss of edge cases due to random sampling, an adaptive hierarchical sampling algorithm (corresponding to the adaptive sampling strategy in this embodiment) is used for hierarchical sampling. Specifically, firstly, the frequency distribution histogram of the full dataset to be detected (the data to be detected) is calculated. For data intervals with long-tail distribution, the sampling weight is increased, and the dynamic calculation formula for the sampling rate is as follows: ; In the formula, Sampling rate, This represents the total number of rows in the data. Based on the sampling coefficient, For example, the preset maximum sampling limit is... It is 10%.

[0032] Therefore, based on this sampling rate Extracting sample set This ensures that the samples can cover the main features and sparsity anomalies of the data.

[0033] In this embodiment of the application, the sample set is obtained by the intelligent agent. Then, the sample set Denoising and standardization preprocessing are performed to eliminate the influence of dimensions and provide clean input for feature extraction. First, Named Entity Recognition (NER) technology is used to identify sensitive information in the samples and perform masking desensitization; simultaneously, missing values ​​(Null / NaN) are marked to generate a mask matrix. Instead of directly filling in missing data, the missing feature itself is retained as the basis for quality detection. Furthermore, for numerical fields, zero-mean normalization (Z-Score Normalization) is performed to map them to a standard normal distribution, eliminating differences in magnitude between different fields; specifically, the transformation formula is: ; In the formula, These are the original values. The mean of the sample data. The standard deviation of the sample data. To prevent small constants with a denominator of zero (such as...) ).

[0034] In this embodiment, an intelligent agent is used to extract multimodal features from denoised and standardized sample data to establish a data fingerprint model. Specifically, the multimodal feature extraction involves in-depth statistical feature extraction based on statistical physics, including basic statistical feature calculation, information entropy calculation, and Benford's Law fit feature calculation. As a feasible implementation, in the in-depth statistical feature extraction process based on physical statistics, not only are conventional statistics calculated, but information theory and numerical law features are also introduced to quantify the disorder and conformity to natural laws of the data. Specifically, in the calculation of conventional statistics, the skewness and kurtosis of the sample data are calculated to determine the asymmetry and steepness of the data distribution. Simultaneously, basic statistical vectors are constructed. In the formula, The mean of the sample data. The standard deviation of the sample data. The minimum value of the sample data. The maximum value of the sample data. The null value rate The number of unique values. In this embodiment, the Shannon entropy formula is used to quantify the information content and disorder of this field. For discrete data, information entropy... for: ; In the formula, The number of unique values. value The frequency of occurrence; here, excessively high or low information entropy may indicate abnormal data quality, such as single-value padding or random garbled characters.

[0035] For numeric fields of financial or natural count types, calculate the Kullback-Leibler divergence between the distribution of its first digit and the distribution based on Benford's Law. Specifically: ; In the formula, This represents the actual distribution of the first digit of the sample. This characteristic is a Benford-style distribution. Used to identify whether data has been forged or tampered with.

[0036] In this embodiment, semantic feature embedding is performed based on a pre-trained large model. Specifically, a deep learning model is used to understand the business meaning of the data, solving the problem of traditional methods not recognizing numbers. In this embodiment, the semantic feature embedding is achieved through the following steps S10121 to S10123: Step S10121: Metadata text serialization; This involves constructing the table's context information into a text sequence. : "Table name is [A], Columnname is [B], Column comment is [C]".

[0037] Step S10122: Sample content serialization; In this embodiment, non-empty values ​​are randomly selected from the samples and concatenated into a content sequence. : "Sample values ​​are [v1, v2, v3...]".

[0038] Step S10123, BERT semantic encoding; In this embodiment, the text sequence and content sequence After concatenation, the data is input into a pre-trained BERT-base-chinese model to obtain the output of the last hidden state; the vector representation of the CLS token is used as the global semantic feature vector for this field. : ; In the formula, BERT (Bidirectional Encoder Representations from Transformers) refers to a pre-trained language model based on the bidirectional Transformer architecture, which serves as the core semantic encoding tool for deep semantic understanding and vector transformation of input text information; subscripts This indicates the corresponding output of the BERT model. Vectors with special markings.

[0039] In this embodiment of the application, the statistical feature vector is... With global semantic feature vector The features are then concatenated. Since the two features have significantly different dimensions (statistical features are low-dimensional, semantic features are high-dimensional), a fully connected layer maps the statistical features to the same high-dimensional space as the semantic features. To enable the agent to automatically determine which features are more important for the current quality detection (e.g., for the "amount" field, statistical distribution is more important; for the "address" field, semantic format is more important), a self-attention mechanism is introduced to calculate the fusion weights.

[0040] Let the concatenated feature matrix be... It can be linearly mapped to a query matrix using a learnable parameter matrix. Key matrix Sum matrix Substituting into the self-attention formula yields the weighted feature matrix. :

[0041] In the formula, Represents the key vector Feature dimensions, It is a normalized exponential function.

[0042] because Given a two-dimensional matrix containing multiple feature sequences, to generate a single vector representation, [the following is done] along the dimension of the feature sequences. Perform global average pooling (GAP) for dimensionality reduction and compression:

[0043] In the formula, The total length of the feature sequence. The first in the weighted characteristic matrix Each feature vector. The final output is a compressed one-dimensional dense vector. This is the data fingerprint vector of the data object, which serves as the sole input reference for large model inference in the subsequent step S102.

[0044] Step S102: Convert each component in the data fingerprint vector into natural language description text, and construct a complete prompt word containing role settings, task instructions, feature context and output constraints; input the complete prompt word into LLM, adaptively generate data quality verification rule code adapted to the target business scenario, and form a dynamic rule base.

[0045] In the embodiments of this application, by utilizing the reasoning capabilities of Retrieval Enhancement (RAG) and Large Language Model (LLM), it is possible to adaptively perceive the data distribution patterns and generate suitable Python / SQL validation scripts.

[0046] To mitigate the knowledge illusion inherent in large language models within specific data governance scenarios, the agent first uses the data fingerprint vector generated in step S101 to retrieve the most matching historical rule template from the rule knowledge base as a contextual reference. Specifically, a general data quality rule base is pre-constructed, containing national standards, industry specifications, and historical high-quality rule codes; a pre-trained embedding model is then used to map the text descriptions in the general data quality rule base into a high-dimensional dense vector set. The data fingerprint vector generated in step S101 is then stored in a vector database to establish a knowledge base vector index; subsequently, the data fingerprint vector generated in step S101 is... as query vector Perform K-Nearest Neighbor (KNN) search on the vector database and compute the query vector. With knowledge base vectors Cosine similarity between : ; In the formula, For query vector, This is a knowledge base vector.

[0047] Therefore, cosine similarity is selected. Greater than the preset threshold Top-K rule sample set This serves as a reference example for few-shot learning.

[0048] As a feasible implementation method, the embedding model can be BERT (a bidirectional encoder representation based on Transformer) or Text2Vec (Text to Vector).

[0049] In this embodiment, the intelligent agent deserializes abstract mathematical feature vectors into natural language descriptions understandable by a large language model, and assembles a structured prompt engineering template. Specifically, it first performs semantic mapping from features to text, then constructs complete prompt words containing role settings, task instructions, feature context, and output constraints, and assembles them into a dynamic Prompt template. Specifically, a feature description function is defined. The components of the fingerprint vector are converted into natural language description text. For statistical feature components, describe their distribution pattern, for example: "The data in this field exhibits a normal distribution with a mean of..." "Long-tail truncation exists"; for semantic feature components, they are mapped to business meanings, such as: "This field is semantically identified as a 'unique identifier' and should not contain special characters"; for entropy features, they are described as: "Information entropy value is..." "The data has high dispersion." The structure of the dynamic Prompt template is as follows: ; In the formula, This provides a complete dynamic Prompt template for the final input large language model; The role setting module is used to determine the system identity, responsibilities, and behavioral constraints of the intelligent agent in the task; This is a task instruction module, used to clearly define the specific task content, objectives, and requirements that the agent needs to perform. For the feature context module, where, This section provides task description information, which explains the object, scenario, and specific content of the current data quality inspection. This is a reference example for sample learning; For modules with output format requirements, specify that the output must be in the format of a Python function or an SQL statement; This indicates a text concatenation operation.

[0050] In this embodiment of the application, the above-obtained The data is input into a Large Language Model (LLM). The model utilizes a thought chain mechanism to guide the step-by-step reasoning of the quality constraints the data should meet, and generates code. Specifically, the model first outputs the intermediate reasoning process (Thought Trace), analyzes the causal relationship between data features and potential quality problems, and draws a reasoning conclusion; that is, the LLM infers the data step-by-step. Specific quality constraints must be met; for example, given that field A is a continuous numerical value and conforms to Benford's law, any outlier that deviates significantly from the theoretical distribution should be marked as an anomaly. Based on this reasoning, corresponding data validation function code is then generated. This code uses data processing libraries (such as Pandas or Spark SQL) to perform logical judgments on the input data columns and returns a Boolean mask sequence.

[0051] To prevent syntax errors or security risks in the code generated by the model, the intelligent agent introduces compiler principles to verify the data function code. A rigorous review process was conducted. On the one hand, a language parser was used to... Parse into an abstract syntax tree The system traverses the tree nodes, checking for SyntaxErrors or undefined variable references. If parsing fails, it captures the exception stack trace and feeds it back to the LLM for self-correction. On the other hand, security auditing is performed based on a whitelist; this involves first defining a whitelist of allowed functions and libraries. Then traverse the abstract syntax tree. All function call nodes :

[0052] If a high-risk operation is detected (such as the system call os.system or network request requests), the rule will be discarded and the process will be terminated.

[0053] In this embodiment, the verified code is serialized into a binary object and encapsulated into a standard rule object by combining metadata information (generation time, applicable field types, version number), and stored in the dynamic rule base. middle.

[0054] In this way, by constructing a dual security mechanism of generation and verification, the potential illusions and security risks that may arise from large models in industrial-grade data governance are effectively addressed. To address the potential instability of code generated from large models, this application innovatively introduces a static code analysis and sandbox execution mechanism based on AST. Through whitelist auditing and syntax tree parsing, the system can automatically intercept high-risk operations and syntax errors before rules are deployed to production, ensuring the security and executability of intelligently generated rules in enterprise production environments, and solving the last-mile trust challenge in the field of generative AI implementation in data governance.

[0055] Step S103: Use the dynamic rule base to drive the execution engine to scan the data to be detected, capture abnormal data items that do not conform to the rules, and parameterize the abnormality type, deviation degree and time dimension of the abnormal data items to construct an initial abnormal state vector.

[0056] In this embodiment of the application, a data quality verification script is executed, and a dynamic rule base is used to drive the execution engine to scan the full amount of data to be detected, identify abnormal data items, and construct an initial abnormal vector based on the verification results (abnormality type, degree of deviation and time dimension).

[0057] In this embodiment, to address the need for detecting massive amounts of data, the agent constructs an execution environment based on a distributed computing framework. The rule code generated in step S102 is distributed to each computing node for distributed rule execution based on sandbox isolation. For example, the distributed computing framework could be Apache Spark or Flink.

[0058] The agent parses multiple independent rule scripts into a Directed Acyclic Graph (DAG). For rules involving the same field, their data read operations (IO) are merged, and columnar storage technology is used for batch reading to minimize IO overhead.

[0059] The rule code is executed using SIMD (Single Instruction Multiple Data) technology. The rule logic is then... Acting on data column vectors Generate Boolean mask vector : ; In the formula, 0 represents normal, and 1 represents abnormal. All values ​​are quickly filtered out using bitwise operations. abnormal index set .

[0060] In this embodiment of the application, for the filtered abnormal data (abnormal index set) The system no longer simply flags errors, but calculates the degree to which they deviate from the rule threshold, distinguishing between minor fluctuations and serious errors. Specifically, for numerical anomalies, if the rule defines a valid range... For outliers Using standard deviation The normalized distance formula is used to calculate its relative deviation. : ; This formula quantifies how many standard deviations out of the normal range an outlier is; the larger the value, the more severe the outlier.

[0061] For text-based / enumerated anomaly distances, if the rule defines a set of allowed strings or regular expressions, then for the anomaly text... Calculate the edit distance (Levenshtein Distance) between it and the desired pattern, and normalize it to the similarity difference: ; In the formula, The deviation of the abnormal text. This is the matching score between the abnormal text and the string with the highest similarity in the legal set. A predefined set of valid strings for the rules. The length of the string containing the abnormal text.

[0062] In this embodiment, after completing the multi-dimensional quantification of anomaly deviations, the agent combines the time dimension to analyze whether the anomaly is an occasional isolated point or a continuous outbreak. Specifically, sliding window statistics are first performed: Set time window (e.g., the most recent 1 hour or the most recent 1000 records), maintain a first-in-first-out (FIFO) anomaly counter. Combine this with the data anomaly detection results. Calculate the current time Local anomaly rate :

[0063] In the formula, For the first in the sliding window The valid judgment value of each record (1 for abnormal, 0 for normal). This represents the total number of records contained within the window. Therefore, by summing the anomaly marker values ​​within the window, the frequency of anomalies in the current data stream within a local time period can be quantified.

[0064] Furthermore, the anomalous rate is tested using the Poisson distribution hypothesis to determine whether it is significantly higher than the historical baseline, and the suddenness index is calculated. : ; In the formula, This is the historical average outlier rate for this field. It is a very small positive constant; if This indicates that we are currently in a period of abnormal outbreaks, such as a large number of errors caused by system failures.

[0065] In this embodiment, all the feature parameters calculated above are encoded and concatenated to construct a state vector that can comprehensively characterize various attributes of the abnormal event. This serves as the input entity for the graph reasoning in the next step, S104.

[0066] Wherein, the state vector It includes the following five core components: ; The specific definitions are as follows: For location features (location embedding), use Graph Embedding (such as Node2Vec) to map the table ID and field ID of the anomaly to node vectors in the graph; One-hot encoding is used for type-specific violations of rule types (integrity / uniqueness / consistency / timeliness). These are numerical features, which are normalized outlier values. The severity feature is specifically the deviation calculated using the aforementioned multi-dimensional abnormal deviation quantification. ( ); The temporal characteristics are specifically the suddenness index calculated above. and local abnormality rate .

[0067] Here, the above-constructed... The primary key index of the accompanying raw data is stored in a time-series database or feature store to form an abnormal event log, which can be subscribed to and consumed by the subsequent root cause analysis module.

[0068] Step S104: Map the initial abnormal state vector to a pre-constructed data lineage knowledge graph; combine the data lineage knowledge graph to perform contextual tracing and propagation analysis on the initial abnormal state vector, calculate the root cause confidence of the abnormality, and determine the abnormality identification result.

[0069] In this embodiment of the application, the initial abnormal state vector is mapped to a pre-constructed enterprise-level data lineage knowledge graph. Based on the subgraph extraction and probability propagation algorithm, the abnormal propagation path is traced in reverse in the enterprise-level data lineage knowledge graph, the root cause confidence of each associated node is calculated, and the source node causing the quality problem is accurately identified.

[0070] In this embodiment, the intelligent agent constructs and maintains in real time a global topology network (data lineage knowledge graph) covering the business source end, data middle platform, and application end through multi-source metadata collection and parsing. Specifically, the intelligent agent accesses database system tables, ETL scheduling logs (such as Airflow / Dolphin Scheduler logs), and SQL execution scripts, using an SQL parser to extract data flow relationships and identify input and output tables. Simultaneously, it employs entity alignment technology to map entities with the same name in different systems (such as user_id in CRM and uid in the data warehouse) to the same semantic node in the graph. 。 Among them, the set of node types that define the graph = {Business System, Physical Table, Field, API Interface, ETL Task, Report Metrics}; Define the set of edge types. = {Data_Flow (data flow direction), Trigger_By (task trigger), Depend_On (logical dependency), Belong_To (belonging relationship)}. In this way, the constructed graph not only includes a static structure but also dynamic runtime attributes, such as the task's most recent execution time and duration.

[0071] In this embodiment, to reduce the computational complexity of reasoning on large-scale graphs, the agent employs a focus-aware strategy to extract local subgraphs directly related to the current anomaly. First, the aforementioned anomaly state vector is parsed. Positional features in This will identify it as an anomalous anchor node in the graph. Then with Starting from the edge, along the opposite direction (i.e., the direction of data source) Perform a traversal and set the maximum number of jumps. (like ） Extract all upstream dependent nodes; simultaneously perform a pruning strategy: if a node on a certain path has been in the past time window... If there are no data update records, the path is considered to be in a "dormant" state and will not participate in the current fault propagation, thus yielding a simplified source subgraph. Furthermore, as a feasible implementation method, in the source tracing subgraph... The intelligent agent combines the "change events" and "historical stability" of each node, uses a Bayesian inference model to calculate the posterior probability of each upstream node causing the current anomaly, sorts the calculation results, locks the fault source with the highest probability, and generates a human-readable analysis report.

[0072] Specifically, for each node in the subgraph (the origin subgraph) The agent collects key state feature vectors within the time window prior to the occurrence of the anomaly. : Change characteristics: Whether there are DDL changes, code commits (Git Commit), or configuration modifications; Load characteristics: volatility of data volume and volatility of task execution time; Quality Feature: The historical data quality score of this node itself.

[0073] Among them, define nodes The confidence score of the root cause The score is composed of three weighted components: change impact factor, distance attenuation factor, and propagation path reachability, and its calculation formula is as follows: ; In the formula, This is an indicator function; it is 1 if the node has recently changed, and 0 otherwise. As a time decay term, the closer the change occurred to the anomaly, the greater its impact. This is the shortest path distance from the node to the anomaly point; the closer the distance, the greater the impact (distance decay). represents the historical stability probability of a node.

[0074] In this embodiment of the application, for all nodes in the subgraph Perform Softmax normalization: ; In the formula, For the current subgraph structure, the first... Nodes The probability of being the root node; This indicates that the subgraph structure being processed is a local graph structure composed of several nodes and edges. Therefore, the one with the highest probability is selected. Each node (e.g., Top-3) is used as a candidate root cause set. .

[0075] Therefore, the intelligent agent utilizes the text generation capabilities of LLM to... To the root cause node The graph path is converted into a natural language explanation chain. For example, missing data in the "User Table" was detected (anomaly anchor). Through source tracing analysis, the upstream "System-Client Synchronization Task" (root cause node, 98% confidence) underwent code changes at 10:00 AM that day, and the subsequent execution time of this task was abnormally shortened, presumably due to data extraction interruption caused by code logic changes.

[0076] In this embodiment, a hierarchical governance strategy is generated based on the root cause identification result (anomaly identification result), and human feedback operations are captured through a reinforcement learning mechanism (Reinforcement Learning from Human Feedback, LHF) to dynamically update the inference parameters or prompt templates of the large model, thereby realizing the continuous evolution of the agent detection capability.

[0077] Among them, the root cause node type output in step S104 With confidence level Mapped to specific execution actions The intelligent agent displays anomaly details and governance recommendations to the user through an interactive interface (UI) or dialog window. It provides operation options, and the system records the user's explicit feedback and implicit behavior in real time to build a feedback dataset. Once the strategy is confirmed for execution, the agent calls the execution layer interface to complete data governance and verify the effect.

[0078] As a feasible implementation method, the rule generation model (Prompt or LLM parameters) in step S102 is fine-tuned using the collected user feedback reward signals.

[0079] This approach introduces a human-machine co-evolutionary mechanism based on RLHF (Human Feedback Reinforcement Learning), endowing the system with the ability to continuously learn and adaptively optimize. This transforms the proposed solution from a static detection tool into an intelligent agent system with evolutionary capabilities. By capturing user feedback signals such as "adoption," "rejection," or "ignoring" of governance strategies, the system dynamically fine-tunes the Prompt weights and inference parameters of the large model using the Proximity Policy Optimization (PPO) algorithm. Thus, over time, the agent can increasingly accurately understand the business specifics of the enterprise and the user's fault-tolerance preferences, automatically reducing false positives and achieving dynamic evolution and personalized adaptation of governance strategies.

[0080] The data quality detection method provided in this embodiment extracts the data fingerprint vector of the data and constructs a complete prompt word that includes role settings, task instructions, feature context, and output constraints. The complete prompt word is input into the LLM (Local Language Model) to adaptively generate data quality verification rule code adapted to the target business scenario, forming a dynamic rule base. By combining the context, the rationality of fields in specific business scenarios is automatically judged, and the general knowledge of the large language model is used to automatically infer potential data logic. This effectively discovers implicit logical anomalies that are difficult to define with traditional manual rules, improving the efficiency of rule configuration by several orders of magnitude. It realizes a leap from manual passive configuration to system proactive discovery. When an anomaly occurs, the intelligent agent extracts the source subgraph by searching and combines it with the Bayesian inference model to calculate the change probability and propagation confidence of each node. This can penetrate complex data links, accurately locate the source of the anomaly, and significantly shorten the troubleshooting time for operation and maintenance personnel.

[0081] Example 2 Based on the above embodiments, this embodiment also provides a data quality detection system for implementing the above data quality detection method. The system of this embodiment includes: The feature extraction module is used to acquire metadata and sample data from the data source to be detected, perform multimodal feature extraction on the sample data, and generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors. The code generation module is used to convert each component in the data fingerprint vector into natural language description text, and construct a complete prompt word containing role settings, task instructions, feature context and output constraints; and input the complete prompt word into LLM to adaptively generate data quality verification rule code adapted to the target business scenario, forming a dynamic rule base; The anomaly encoding module is used to drive the execution engine to scan the data to be detected using a dynamic rule base, capture abnormal data items that do not conform to the rules, and parameterize the anomaly type, deviation degree and time dimension of the abnormal data items to construct an initial anomaly state vector. An anomaly identification module is used to map the initial anomaly state vector to a pre-constructed data lineage knowledge graph; combine the data lineage knowledge graph to perform contextual tracing and propagation analysis on the initial anomaly state vector, calculate the anomaly root cause confidence, and determine the anomaly identification result.

[0082] This application utilizes a large language model to achieve an automated closed loop from data feature perception to verification rule generation, and integrates data lineage knowledge graph technology to accurately locate the root cause of anomalies through graph propagation algorithms, thereby achieving adaptive and intelligent data quality management.

[0083] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method in any of the embodiments of this application. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU (Central Processing Unit) or MPU (Microprocessor Unit) of the system or apparatus may read and execute the program code stored in the storage medium.

[0084] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined above in the system of this application.

[0085] It should be noted that the computer-readable storage medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF (Radio Frequency), etc., or any suitable combination thereof.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0087] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0088] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed systems, modules, and methods can be implemented in other ways. For example, the module embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between modules or units, and may be electrical, mechanical, or other forms.

[0090] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. This application is not limited to the exact structures described above and illustrated in the accompanying drawings, and it should not be considered that the specific implementation of this application is limited to these descriptions. For those skilled in the art, various changes and modifications made without departing from the concept of this application should be considered to fall within the protection scope of this application.

Claims

1. A data quality inspection method, characterized in that, include: Acquire sample data and perform multimodal feature extraction on the sample data to generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors; Each component in the data fingerprint vector is converted into natural language description text, and a complete prompt word containing role settings, task instructions, feature context and output constraints is constructed; the complete prompt word is input into LLM to adaptively generate data quality verification rule code adapted to the target business scenario, forming a dynamic rule base; The dynamic rule base drives the execution engine to scan the data to be detected, capture abnormal data items that do not conform to the rules, and parameterize the abnormality type, deviation degree and time dimension of the abnormal data items to construct an initial abnormal state vector. The initial abnormal state vector is mapped to a pre-constructed data lineage knowledge graph; the initial abnormal state vector is then subjected to contextual tracing and propagation analysis based on the data lineage knowledge graph, the root cause confidence of the abnormality is calculated, and the abnormality identification result is determined.

2. The data quality detection method according to claim 1, characterized in that, The step of extracting multimodal features from the sample data to generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors includes: Constructing basic statistical vectors In the formula, The mean of the sample data. Standard deviation To be the minimum value, The maximum value, The null value rate, The number of unique values; for discrete data, the information entropy is calculated using the Shannon entropy formula; for numerical fields, the KL divergence between the distribution of its first digit and the distribution based on Benford's law is calculated. The statistical feature vector is formed by concatenating the basic feature vector, the information entropy, and the KL divergence. The context information of the table is constructed into a text sequence; non-null values ​​are randomly extracted from the sample data and concatenated to form a content sequence; The text sequence and the content sequence are concatenated and input into the pre-trained BERT-base-chinese model to obtain the last hidden layer, and the semantic feature vector is obtained by using the vector representation of CLS Token. The statistical feature vector and the semantic feature vector are fused to generate a data fingerprint vector that uniquely represents the sample data; Specifically, after concatenating the statistical feature vector and the semantic feature vector, a fully connected layer is used to map the statistical features to the same high-dimensional space as the semantic features; a self-attention mechanism is introduced to calculate the fusion weights of the statistical feature vector and the semantic feature vector to obtain the weighted fusion data fingerprint vector.

3. The data quality detection method according to claim 2, characterized in that, The process of converting each component of the data fingerprint vector into natural language description text and constructing a complete prompt word containing role settings, task instructions, feature context, and output constraints includes: Define feature description function The components in the data fingerprint vector are converted into natural language descriptive text. Among them, statistical feature components describe their distribution pattern; semantic feature components are mapped to business meaning; and entropy features are described as information entropy values. Construct complete prompt words that include character settings, task instructions, feature context, and output constraints. Specifically: ； In the formula, This is a complete dynamic Prompt template for the final input LLM; The role setting module is used to determine the system identity, responsibilities, and behavioral constraints undertaken in a task; This is the task instruction module, used to specify the specific task content, objectives, and requirements that need to be executed. For the feature context module, where, This section provides task description information, which explains the object, scenario, and specific content of the current data quality inspection. This is a reference example for sample learning; For modules with output format requirements, specify that the output must be in the format of a Python function or an SQL statement; This indicates a text concatenation operation; Specifically, a pre-trained embedding model is used to map text descriptions from a pre-built general data quality rule base into a high-dimensional dense vector set, which is then stored in a vector database to establish a knowledge base vector index; subsequently, the data fingerprint vector is used as the query vector. Perform a K-nearest neighbor search on the vector database and compute the query vector. With knowledge base vectors The cosine similarity between the samples is used to select multiple rule samples whose cosine similarity is greater than a preset threshold, forming a reference example for sample learning.

4. The data quality detection method according to claim 3, characterized in that, The step of inputting the complete prompt word into the LLM to adaptively generate data quality verification rule code adapted to the target business scenario and form a dynamic rule base includes: Analyze the causal relationship between the data features corresponding to the complete prompt words and potential quality problems to draw inference conclusions; then, based on the inference conclusions, generate the corresponding data verification function code; The data verification function code is subjected to code syntax parsing and security review based on abstract syntax tree, and the verified code is serialized into a binary object, combined with the metadata and encapsulated into a standard rule object, which is then stored in the dynamic rule library.

5. The data quality detection method according to claim 4, characterized in that, The parameterized encoding of the anomaly type, deviation degree, and time dimension of the abnormal data items to construct an initial anomaly state vector includes: For numerical outlier deviations, given that the rules define an effective range, the normalized distance formula based on standard deviation is used to calculate the relative deviation of outliers. For text-based or enumerated anomaly distances, if the rules define an allowed set of strings or regular expressions, the edit distance between the anomalous text and the expected pattern is calculated and normalized to a similarity difference to obtain the deviation of the anomalous text. Based on the sliding window analysis of the time-series anomalies of the anomalous data items, the local anomaly rate and the burst index are obtained; Construct the initial abnormal state vector of the abnormal data item Specifically: ； The definition is as follows: To leverage Graph Embedding to map the table ID and field ID of an anomaly to node vectors in the graph, thus establishing location features; This is a type-based one-hot encoding method used to perform one-hot encoding on the types of rules that are violated. These are the normalized outlier values. The severity features are specifically the relative deviation of the outliers and the deviation of the outlier text. The local anomaly rate and the suddenness index are respectively.

6. The data quality detection method according to claim 5, characterized in that, The process of combining the data lineage knowledge graph to perform contextual source tracing and propagation analysis on the initial abnormal state vector, calculating the root cause confidence of the abnormality, and completing the abnormality identification includes: The positional features in the initial abnormal state vector are analyzed, and the positional features are locked as abnormal anchor nodes in the data lineage knowledge graph; Starting from the abnormal anchor node, traverse in the opposite direction of the edge, set the maximum number of hops, and extract all upstream dependent nodes corresponding to the starting point; at the same time, perform a pruning strategy to obtain the source subgraph; On the source subgraph, the posterior probability of each upstream node causing the current anomaly is calculated using a Bayesian inference model, and the anomaly root cause confidence level corresponding to each upstream node is obtained. The confidence scores of the abnormal root causes corresponding to all nodes in the source subgraph are subjected to Softmax normalization to select the nodes with the highest probabilities. Each node is used as a candidate root cause set.

7. The data quality detection method according to claim 6, characterized in that, The step of calculating the posterior probability of each upstream node causing the current anomaly using a Bayesian inference model on the source subgraph, and obtaining the anomaly root cause confidence level corresponding to each upstream node, includes: For each node in the source tracing subgraph, key state feature vectors are collected within the time window before the anomaly occurs. The key state feature vector This includes change characteristics, load characteristics, and quality characteristics; Define nodes The confidence score of the root cause The confidence score This includes the change impact factor, distance attenuation factor, and propagation path reachability, and its calculation formula is as follows: ； In the formula, This is an indicator function; it is 1 if the node has recently changed, and 0 otherwise. As a time decay term, the closer the change occurred to the anomaly, the greater its impact. This represents the shortest path distance from the node to the anomaly point; the closer the distance, the greater the impact. The historical stability probability of a node; These are the weighting coefficients corresponding to the change impact factor, distance attenuation factor, and propagation path reachability, respectively.

8. The data quality detection method according to any one of claims 1-7, characterized in that, The acquisition of sample data includes: Read the system tables or metadata dictionary from the data source to be detected to obtain the basic attributes of the object to be detected; Calculate the frequency distribution histogram of the full dataset to be detected, add sampling weights to data intervals with long-tail distributions, and extract sample data based on the updated sampling rate weights; wherein, the dynamic calculation of the sampling rate is as follows: ； In the formula, Sampling rate, This represents the total number of rows in the data. Based on the sampling coefficient, This is the preset maximum sampling limit; The sample data is then cleaned and standardized.

9. The data quality detection method according to claim 8, characterized in that, Also includes: Based on the anomaly identification results, the data to be detected is classified and managed, and the anomaly details and management suggestions are displayed to the user through an interactive interface; A feedback dataset is constructed based on real-time recorded user feedback. A human feedback reinforcement learning mechanism is introduced, and the complete prompt word template or LLM parameters are continuously optimized using the feedback dataset. The step of classifying and managing the data to be detected based on the anomaly identification results includes: The governance strategy mapping model maps the root cause node type and confidence level in the anomaly identification results to specific governance execution actions.

10. A data quality inspection system, used to implement the data quality inspection method according to any one of claims 1-9, characterized in that, The system includes: The feature extraction module is used to acquire metadata and sample data from the data source to be detected, perform multimodal feature extraction on the sample data, and generate a data fingerprint vector based on statistical feature vectors and semantic feature vectors. The code generation module is used to convert each component in the data fingerprint vector into natural language description text, and construct a complete prompt word containing role settings, task instructions, feature context and output constraints; and input the complete prompt word into LLM to adaptively generate data quality verification rule code adapted to the target business scenario, forming a dynamic rule base; The anomaly encoding module is used to drive the execution engine to scan the data to be detected using a dynamic rule base, capture abnormal data items that do not conform to the rules, and parameterize the anomaly type, deviation degree and time dimension of the abnormal data items to construct an initial anomaly state vector. An anomaly identification module is used to map the initial anomaly state vector to a pre-constructed data lineage knowledge graph; combine the data lineage knowledge graph to perform contextual tracing and propagation analysis on the initial anomaly state vector, calculate the anomaly root cause confidence, and determine the anomaly identification result.

Images