A large model-based data quality detection and management method

Abstract

The application provides a kind of data quality detection and management method based on large model, it is related to electric digital data processing field, comprising: step S1, data preprocessing: first using large model to the original data of input is deconstructed and modal labeling, then pass through business knowledge graph driving entity link and semantic enhancement, finally realize cross-modal unified semantic representation coding;Step S2, feature extraction: including semantic rule mining and feature mode extraction, rule automatic generation and priority sorting, rule and data feature dynamic binding;Step S3, data quality detection: first, the joint coding of multimodal data quality characteristics is carried out, and then the context-aware quality anomaly detection is carried out;Step S4, root cause analysis and repair.The method can solve the problems of strong rule dependency, insufficient universality, inaccurate root cause analysis and poor interpretability of existing data quality detection methods.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and specifically to a data quality detection and management method based on a large model. Background Technology

[0002] With the rapid development of the big data era, data quality inspection technology is widely used in artificial intelligence model training, data standardization, and data governance scenarios. Enterprises and organizations face the challenge of managing massive amounts of data, and data quality issues are becoming increasingly prominent. Data quality inspection technology is a technical system that integrates algorithmic models, business rules, and engineering tools. Its core objective is to conduct multi-dimensional, automated or semi-automated evaluation, verification, and anomaly identification of data throughout its entire lifecycle (collection, storage, processing, transmission, and application). It determines whether the data meets preset quality standards and business requirements, and outputs quality reports and anomaly location results, providing core evidence for data quality optimization, data governance, and data value mining.

[0003] Currently, existing data quality management technologies have made some progress in basic format checks and simple statistical analysis. These methods are effective in addressing fundamental quality issues of structured data, but they still have significant shortcomings in practical applications: 1. Lack of semantic understanding: Existing methods struggle to understand the business semantics of data and cannot identify semantic-level data quality problems; 2. Limited ability to recognize complex patterns, strong rule dependence, and poor compatibility: Existing methods typically rely on static rules or statistical models, lacking the ability to recognize complex data relationships and patterns across fields and tables; 3. Insufficient context awareness and inability to understand semantic context: Existing methods mainly rely on rule matching or statistical calculations, lacking a deep understanding of the semantics and context of data content, and cannot combine business context to make data quality judgments, leading to difficulties in handling complex quality indicators (such as content authenticity and type consistency). The existing methods suffer from several drawbacks: 1. Low accuracy in identifying data types (e.g., low accuracy in identifying data types); 2. Insufficient versatility and scalability, and weak adaptability to dynamic data: Existing methods are poorly adaptable to data distribution and business rules, and are difficult to quickly expand or migrate to new scenarios (e.g., a data quality detection method and system based on a large model disclosed in patent document CN119829965A, or a data quality detection method, system, and storage medium based on a large model for the ecological environment field disclosed in patent document CN120561117A; although they use large models for fine-tuning, they are mostly targeted at specific fields or basic detection, lacking a cross-domain general framework); 3. Insufficient ability to process unstructured data: Existing methods are mostly targeted at structured data, and have limited ability to detect the quality of unstructured data such as text and images; 4. Poor interpretability: The detection results of existing methods lack interpretability, and it is difficult to clearly explain the causes and scope of the problem to users. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention aims to provide a data quality detection and management method based on a large model. This method can overcome modal barriers and achieve semantic-level processing of structured, semi-structured, and unstructured data, thereby solving the problem of insufficient support for semi-structured or unstructured data in the prior art. At the same time, this method realizes a closed-loop process of detection-root cause-repair-verification, avoiding the problems of strong rule dependence, insufficient universality, inaccurate root cause analysis, and poor interpretability of existing data quality detection methods.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A data quality inspection and management method based on a large model includes:

[0007] Step S1, Data Preprocessing: First, the input raw data is deconstructed and modally labeled using a large model. Then, entity linking and semantic enhancement are driven by business knowledge graph. Finally, cross-modal unified semantic representation encoding is achieved.

[0008] Step S2, Feature Extraction: This includes semantic rule mining and feature pattern extraction, automatic rule generation and priority ranking, and dynamic binding of rules and data features;

[0009] Step S3, Data Quality Detection: First, perform joint encoding of multimodal data quality features, and then perform context-aware quality anomaly detection;

[0010] Step S4, Root Cause Analysis and Remediation: This includes the fusion of multi-source detection results, deep root cause analysis, interpretability report generation, adaptive remediation execution, and closed-loop verification.

[0011] Based on further optimization of the above scheme, the large model adopts either GPT-4V or Llama 3-Vision.

[0012] Based on further optimization of the above scheme, the original data includes structured tables, semi-structured JSON / XML, unstructured text or images, etc.

[0013] Based on further optimization of the above scheme, in step S1, the deconstruction and modality labeling of the input raw data using a large model specifically involves:

[0014] For structured data: extract field names, field types, field values, and inter-table join keys;

[0015] For semi-structured data: parsing the hierarchical structure and extracting node semantics and attributes;

[0016] For unstructured data: extract keywords and semantic roles from text, and extract semantic content from images;

[0017] The final output is a deconstructed dataset with modal labels and basic semantics. D :

[0018] ;

[0019] In the formula: S Representing structured data, SS This represents semi-structured data. U This represents unstructured data.

[0020] Based on further optimization of the above scheme, in step S1, driving entity linking and semantic enhancement through business knowledge graph specifically involves:

[0021] First, a business knowledge graph is constructed using a large model. B-KG ;

[0022] Then deconstruct the data set Entity and Business Knowledge Graph B-KG Linking entities within the destructuring process involves extracting the entities to be linked (such as field names, field descriptions, and the business entities corresponding to field values) from the destructuring results to form a set of entities to be linked. The large model considers each entity to be linked. e data,m Perform preliminary semantic matching and filter business knowledge graphs. B-KG Entities with a string similarity of not less than 0.5 or a preliminary semantic similarity of not less than 0.4 are selected as the candidate set. ;

[0023] Then, by leveraging the text embedding capabilities of the large model, the entities to be linked are calculated. e data,m With candidate entities e hx,n Semantic similarity (cosine similarity):

[0024] ;

[0025] In the formula: LLM-Emb() This represents a large model text embedding function used to convert input text into a high-dimensional semantic vector;

[0026] Preset similarity threshold Sim X If the semantic similarity is not less than the similarity threshold, it is considered a valid match, and the corresponding candidate entity is the link result.

[0027] Finally, the business knowledge graph B-KGThe entity attributes linked in the middle are directly mapped to the entities in the deconstructed data set, outputting an enhanced data set with the desired business semantic attributes: ,in, s i Represents a set of business semantic attributes.

[0028] Based on further optimization of the above scheme, in step S1, the implementation of cross-modal unified semantic representation encoding specifically involves:

[0029] By using cross-modal unified semantic representation, augmented data from different modalities are transformed into unified high-dimensional semantic vectors:

[0030] ;

[0031] In the formula: These represent the weight coefficients of the large model embedding and the weight coefficients of the context embedding, respectively. e i express Link to business knowledge graph B-KG The entity in KG-Emb() Representing business knowledge graphs B-KG Entity embedding function; c i express The business context in which it is located Context-Emb() This indicates a context embedding function that outputs a context vector based on the business scenario in which the data resides.

[0032] Noise filtering is used to remove noisy data that does not match the semantics of the business or the scenario.

[0033] ;

[0034] Preset noise threshold ,like If the expression is 0, it is removed; otherwise, the corresponding unified semantic vector is retained. v i ;

[0035] Finally, a unified semantic representation vector set across modalities is obtained:

[0036] , ( d (For vector dimensions).

[0037] Based on further optimization of the above scheme, in step S2, semantic rule mining and feature pattern extraction include business document rule mining and historical data feature pattern extraction. Specifically, business document rule mining involves: first, inputting enterprise business documents and splitting them into semantically coherent business topic blocks; then, constructing a system that includes "business topic block + rule type + output format requirements". Promptand will Prompt The data is fed into a large model to obtain preliminary rule candidates; then, the semantic parsing capabilities of the large model are used to map natural language constraints into structured expressions.

[0038] ;

[0039] In the formula: LLM-Parse() This represents a semantic parsing function for large models, which converts natural language into structured conditions; Constraint n Represents the natural language constraints of the large model output; Entity type Indicates the type of business entity;

[0040] Calculate the confidence level of the rule:

[0041] ;

[0042] In the formula: Represents the weight coefficients output by the large model; This indicates the probability that the large model outputs this rule; This indicates that the rules are consistent with industry or business common sense;

[0043] Preset rule confidence threshold Conf thrt ,reserve The rules are used to filter out low-confidence rules;

[0044] Resolve conflicts arising from extraction rules across different documents or topics by using conflict resolution rules:

[0045] ;

[0046] In the formula: ConfSet A set of rules representing conflicts; This indicates the authority of the source document for the rules;

[0047] Finally, the filtered and resolved rules are organized into a set of rule templates: ;in, rt f-j Includes rule type, business entity, and constraints;

[0048] Historical feature data feature pattern extraction: Input high-quality historical dataset D hist Compared with historical quality problem datasets D error Completed through pre-training HGNNHeterogeneous Graph Neural Networks (HGNNs) extract feature pattern sets across fields / tables. .

[0049] Based on further optimization of the above scheme, in step S2, the automatic rule generation and priority sorting specifically involve: obtaining the rule template... RT With feature pattern set FP Input a large model, and the large model calculates the probability of rule generation:

[0050] ;

[0051] In the formula: LLM-Predict() This represents the prediction score of the large model for the combination of rule templates and feature pattern sets.

[0052] Arrange the rules in descending order of their generation probability and select the rules with the highest generation probability. L Combining these elements generates a set of structured rules: Each rule r i It includes rule ID, rule name, applicable modality, applicable scenario, constraints, business semantic interpretability, confidence level, etc.

[0053] Structured rules are evaluated based on business impact, data coverage, and confidence level. r i Weighted priority calculations are performed to ensure that high-value rules are executed first.

[0054] ;

[0055] In the formula: Imp(r i ) This indicates the business impact of the rule; Cover(r i ) This indicates the data coverage of the rule; Confi (r i ) Indicates confidence in the rule; These represent the corresponding weight coefficients;

[0056] According to the rules P(r i ) Sort from highest to lowest, and rules with higher priority are executed first in data quality inspection;

[0057] The dynamic binding of rules and data features specifically involves: first, generating the rules... r i With cross-modal unified semantic representation vectorv i Bind:

[0058] ;

[0059] Preset binding threshold Match th ,like Then the corresponding rule will be categorized into the applicable set of data;

[0060] Real-time monitoring of data distribution and business document changes, dynamically updating, deprecating, and adding rules:

[0061] Calculate the current data distribution P sent With high-quality data distribution Q ant KL divergence:

[0062] ;

[0063] Preset KL divergence threshold KL th If greater than KL th This proves that the data distribution changes significantly;

[0064] Calculate the current document Doc new Compared with the original document Doc old Semantic similarity between them:

[0065] ;

[0066] Preset document similarity threshold Sim Dth If less than Sim Dth This proves that the business documents have been significantly updated;

[0067] Evolutionary rule setting: If the statistical pattern corresponding to the rule exceeds KL th Furthermore, the KL divergence of adjacent windows in subsequent consecutive windows (usually 3) all exceed [a certain value]. KL th If the corresponding rule is removed, then remove the rule; if the statistical pattern corresponding to the rule exceeds [a certain threshold], then remove the rule. KL th Furthermore, the KL divergence of adjacent windows in subsequent consecutive windows (usually 3) does not exceed KL th If the document similarity corresponding to the rule is less than 1, then adjust the constraints of the rule; if the document similarity corresponding to the rule is less than 1. Sim DthThen, a new template generation rule is established based on the newly added constraints in the business document;

[0068] The evolved rules are recalculated for priority and dynamically bound to data features to ensure the timeliness of the rules.

[0069] Based on further optimization of the above scheme, step S3 specifically includes:

[0070] Joint encoding of multimodal data quality features:

[0071] First, the cross-modal unified semantic representation vector v i Mapped to corresponding text description v text,i , set of rules R i Convert to text description R text,i It employs a semantic encoding function of a large model to output semantic feature vectors. f s,i :

[0072] ;

[0073] In the formula: LLM-Encode() A text embedding function representing a large model;

[0074] For structured data, construct a data relationship graph. G s,i =( J , ),in, J The union of the table node feature set and the field node feature set. To encode the graph structure using the GNN encoding method, we use the union of three types of edges: containing edges, associated edges, and computed edges.

[0075] ;

[0076] In the formula: GNN-Sencode() Representing heterogeneous graphs GNN (like HAN The structure encoding function;

[0077] For unstructured data, the intrinsic structure of the content is encoded through a structure encoding function of a large model (which transforms the content structure of unstructured data into a fixed-dimensional structure feature vector):

[0078] ;

[0079] In the formula: LLM-Sencode() The structural encoding function represents the large model; Uns c,iRepresents the content structure of unstructured data;

[0080] semantic feature vector f s,i With structural feature vectors f t,i Concatenate the features to obtain the joint feature vector:

[0081] ;

[0082] Context-aware quality anomaly detection:

[0083] Use context embedding functions to embed business scenarios c i Encoded as scene vectors c emb,i and joint features f i Concatenate the data, input it into the large model context inference module, and calculate the anomaly probability:

[0084] ;

[0085] In the formula: LLM-Infer() This represents the classification inference function for a large model, outputting anomaly probabilities ranging from 0 to 1;

[0086] Preset anomaly detection threshold Determine the abnormal state based on the abnormal probability and output the abnormal confidence level. Conf(o i ) :

[0087] , ;

[0088] Output quality anomaly detection set And the corresponding anomaly confidence level.

[0089] Based on further optimization of the above scheme, the fusion of multi-source detection results in step S4 is specifically as follows:

[0090] For each data v i Check each one to see if it meets the requirements. R i If each rule constraint is satisfied, then an abnormal rule matching result will occur. r m,i =0, if the data v i Not satisfied R i The constraints of any rule, then r m,i =1, generate a set of rule matching exceptions. ;

[0091] Obtain data v i In the rules r j Single rule anomaly confidence Conf j,i :

[0092] ;

[0093] In the formula: Confidence(r j ) This represents the confidence level of the rule itself (i.e., the confidence level output from the rule generation stage, specifically...). Conf(rt j ) ); Vio j,i Representing data v i On the rules r j Severity of violation:

[0094] ;

[0095] For data { v i The set of applicable rules for} R i All rules in the algorithm are merged to obtain the rule matching anomaly confidence score:

[0096] ;

[0097] Based on the quality anomaly detection set, the rule-matched anomaly set, and the corresponding confidence scores, the fusion anomaly score is calculated:

[0098] ;

[0099] In the formula: w oi , w ri These represent the corresponding weight coefficients;

[0100] Preset score threshold :

[0101] ;

[0102] The final set of quality anomaly determination results is obtained as follows: .

[0103] Based on further optimization of the above scheme, in step S4, the deep root cause analysis specifically includes:

[0104] Based on the business knowledge graph B-KG and historical quality problem data, a causal graph containing "node-causal edge-causal probability" is constructed. Nodes include data entities, rules, business processes, system components, and abnormal events; causal edges are directed edges.

[0105] Causal probability P ( J v | J u ) represents a node J u When it happens, node J v Conditional probability of occurrence:

[0106] ;

[0107] In the formula: Indicating historical quality issues in the data J u and J v The number of times they occur simultaneously; Indicating historical data J u The number of times it occurs; LLM-Causal(J u ,J v ) Represents the reasoning from the large model J u arrive J v The causal strength; Represents the weighting coefficients, used to balance data statistics and large model inference;

[0108] Based on the constructed causal graph, the final abnormal results are analyzed. O final Root cause localization: This involves identifying the final abnormal results. O final Anomalous events are mapped to target nodes in the causal graph. J mv (i.e., the node corresponding to the anomaly); search the causal graph for the target node. J mv Obtain the set of all directed paths to the destination. P(J e ) And for each path p lj ∈ P(J e ) Obtain the product of the probabilities of all causal edges on the path:

[0109] ;

[0110] In the formula: Indicates traversing the causal path p lj All causal edges ( J u , J v ), and perform a multiplication operation on the causal probability of each edge;

[0111] The optimal root cause path is selected by using the product of maximum probabilities.

[0112] ;

[0113] Finally, the rightmost root cause path PathM(p e ) The corresponding node sequence is transformed into a root cause chain described in natural language.

[0114] Based on further optimization of the above scheme, in step S4, the generation of the interpretability report specifically involves: collecting anomalous data (such as anomalous results). O final ), root cause analysis (such as optimal root cause path) PathM(p e ) ), the impact of abnormal business operations obtained based on the business knowledge graph (i.e., the impact on downstream operations), and the applicable rule-level content that triggered the abnormality (from the rule set). R i ), generate an interpretable report;

[0115] The adaptive repair execution and closed-loop verification are as follows: Based on the root cause analysis type and business knowledge graph, the large model extracts the appropriate repair strategy, which includes automatic correction and manual review; the repaired data is then re-entered into the data preprocessing stage for full-process quality inspection to verify the repair results; if the anomaly rate is still higher than the threshold, rule updates or root cause analysis are triggered for repeated iterative repair until the requirements are met.

[0116] The following are the technical effects of the present invention:

[0117] This invention uses a large model as its core driver, integrating technologies such as knowledge graphs, heterogeneous graph neural network models, and cross-modal representations to construct a full-process data quality control system encompassing preprocessing, feature extraction, detection, root cause identification, and repair. This system achieves multimodal compatibility, deep business integration, dynamic adaptation, accurate and efficient detection, and interpretable root causes. It effectively addresses the problems of existing data quality detection methods, such as lack of semantic understanding, limited ability to recognize complex patterns, insufficient context awareness, inadequate versatility and scalability, insufficient unstructured data processing capabilities, and poor interpretability.

[0118] In the data processing stage: This invention refines the structure of multimodal data, applying differentiated structural logic to structured, semi-structured, and unstructured data to accurately extract core information from various types of data, achieving comprehensive coverage of all data types; through semantic enhancement driven by business knowledge graphs, it achieves precise entity linking, facilitating strong binding of subsequent detection rules with business logic and avoiding misjudgments that are technically compliant but business-ineffective; through cross-modal unified semantic representation, it transforms data from different modalities (i.e., large model embedding, knowledge graph entity embedding, and context embedding) into unified high-dimensional vectors, solving the pain point of semantic heterogeneity and inability to jointly analyze multimodal data. In the feature extraction stage, this invention ensures that the source of rules takes into account both business specifications (explicit rules) and data patterns (implicit rules) through semantic rule mining and feature pattern extraction, avoiding problems such as rule omissions or inaccurate judgments caused by rules failing to cover implicit data. Through automatic rule generation and priority sorting, it optimizes detection efficiency and focuses on high-value issues, avoiding resource waste and increased computing power caused by differentially executing all rules. Through dynamic binding of rules and data features, it ensures that the rule system is dynamically optimized with business iterations and data drift, thereby maintaining the long-term effectiveness of the detection and management methods and improving the overall scalability of the method. In the data quality detection stage: through the joint encoding of multimodal data quality features, taking into account both semantic connotation and structural form, it accurately characterizes data quality problems (such as missing cross-table associations in structured data, semantic conflicts in unstructured text, etc.), thereby comprehensively capturing quality features. Through context-aware quality anomaly detection, business scenarios are encoded into scenario vectors, concatenated with joint features, and input into the large model inference module, reducing scenario misjudgments, lowering the probability of misjudgment, and improving the accuracy of anomaly detection. In the root cause analysis and remediation phase: By fusing multi-source detection results, the bias of a single detection dimension is avoided, the reliability of anomaly judgment is improved, and high-quality input is provided for subsequent root cause analysis; through deep root cause analysis, using the construction of causal maps, the underlying root causes are penetrated from surface anomalies to accurately locate the essence of the problem, thereby providing a precise direction for remediation; through the generation of interpretable reports and the coordination of adaptive remediation execution and closed-loop verification, a complete closed loop is formed to ensure the continuous achievement of data quality standards and the stability of detection. Attached Figure Description

[0119] Figure 1 This is a flowchart of the data quality detection and management method in an embodiment of the present invention. Detailed Implementation

[0120] The technical solutions in the embodiments of the present invention will be clearly and completely described below. In the following description, specific details such as specific system structures and technologies are presented for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention.

[0121] Example 1:

[0122] A data quality inspection and management method based on a large model includes:

[0123] Step S1, Data Preprocessing:

[0124] Step S11: First, use a large model to deconstruct and modally label the input raw data. The large model can be either GPT-4V or Llama 3-Vision (GPT-4V is preferred). The raw data includes structured tables, semi-structured JSON / XML, unstructured text or images, etc.; specifically:

[0125] For structured data: extract field names, field types, field values, and inter-table join keys; first, identify the structure carrier and format, and the large model reads the data's metadata (such as database table DDL statements, Excel sheet structures, etc.; taking "MySQL table DDL" as an input, the large model outputs: format type: MySQL relational table, containing 1 sheet (table), table name: orders); then, parse the table structure metadata, and the large model parses the table's core metadata (table name, field list); then, perform refined extraction of field attributes, and the large model combines DDL and sample data to extract the field name, type, typical value, and business constraints of each field; finally, perform inter-table join key inference, and the large model analyzes the semantics and type consistency of fields across multiple tables to infer join keys (foreign keys).

[0126] For semi-structured data: parsing the hierarchical structure and extracting node semantics and attributes; the large model first identifies the format type and hierarchical separator (taking JSON data as an example, the large model outputs: format type: JSON, hierarchical separator: {} (object node), [] (array node)); the large model then parses the hierarchical relationship and generates a tree structure (the output includes root node, child nodes, leaf nodes, etc.); after that, the large model extracts the business semantics, attribute name, attribute value, and attribute type of each node.

[0127] For unstructured data: extract keywords and semantic roles from text, and extract semantic content from images;

[0128] For unstructured data that is primarily text-based: First, the large model performs long text segmentation and semantic clustering, breaking it down into business topic blocks (taking a Word document fragment as an example, the large model outputs: Business Topic 1: Order Creation Rules, content: After a user submits an order, the system generates a 6-digit order_id; Business Topic Block 2: User Information Specifications, content: User information must include user_id and user_name, with user_name being 2-10 characters long); then, the large model extracts keywords and business entities from the topic blocks; finally, the large model identifies the semantic roles of the entities (agent, patient, constraint, etc.).

[0129] For unstructured data that is primarily image-based: First, the large model preprocesses the images (e.g., converting them to Base64 encoding using OpenCV and adjusting the resolution to 1024×1024 to fit the large model's input); then, the large model identifies the core business objects by inputting the images and business scenario prompts; finally, the large model maps the identified structures to a business terminology database.

[0130] The final output is a deconstructed dataset with modal labels and basic semantics. D :

[0131] ;

[0132] In the formula: This indicates the first element in the destructured data set. i Deconstructing data; N This indicates the total number of data items to be deconstructed in the dataset; S Representing structured data, SS This represents semi-structured data. U This represents unstructured data.

[0133] Step S12: Then, entity linking and semantic enhancement are driven by the business knowledge graph, specifically as follows:

[0134] First, a business knowledge graph is constructed using a large model. B-KG ;

[0135] Then deconstruct the data set Entity and Business Knowledge Graph B-KG Linking entities within the destructuring process involves extracting the entities to be linked (such as field names, field descriptions, and the business entities corresponding to field values) from the destructuring results to form a set of entities to be linked. , N data This represents the total number of entities in the Edata set of entities to be linked; the large model considers each entity to be linked... e data,m Perform preliminary semantic matching and filter business knowledge graphs.B-KG Entities with a string similarity (calculated using Levenshtein distance similarity or cosine similarity) of not less than 0.5 or a preliminary semantic similarity (calculated using cosine similarity from large model embeddings) of not less than 0.4 are selected as candidates. , N c In the business knowledge graph B-KG The candidate entity set selected from the data C m The number of entities;

[0136] Then, by leveraging the text embedding capabilities of the large model, the entities to be linked are calculated. e data,m With candidate entities e hx,n Semantic similarity (cosine similarity):

[0137] ;

[0138] In the formula: LLM-Emb() This represents a large model text embedding function used to convert the input text into a high-dimensional semantic vector (e.g., text-embedding-ada-002 of GPT-4V or embedding layers of Llama3).

[0139] Preset similarity threshold Sim X (Generally 0.7), if the semantic similarity is not less than the similarity threshold, it is considered a valid match, and the corresponding candidate entity is the link result;

[0140] Finally, the business knowledge graph B-KG The entity attributes linked in the middle are directly mapped to the entities in the deconstructed data set, outputting an enhanced data set with the desired business semantic attributes: ,in, s i Represents a set of business semantic attributes (i.e., through a business knowledge graph). B-KG (Business semantic attributes supplemented by entity links).

[0141] Specifically, building a business knowledge graph through a large model involves:

[0142] First, collect enterprise business documents (including SOPs, data dictionaries, business reports, SQL table structures, etc.), classify them by format (i.e., categorize them into structured data, semi-structured data, and unstructured data), and then use a large model to perform semantic segmentation and topic clustering on unstructured documents (i.e., break long documents into text blocks of 200-500 characters, and use the large model's topic recognition capabilities to label each text block with a business topic, and then use the text blocks...). d tWith hashtags top k (Semantic similarity is used for clustering), topic clustering is specifically as follows:

[0143] ;

[0144] In the formula: Represents the cosine similarity function;

[0145] Preset clustering decision threshold Sim th (Usually 0.6), if Then the text block d t Categorize into the corresponding topic top k If it is inside, otherwise it will not be included;

[0146] Output a normalized set of document blocks labeled with business topics. D doc :

[0147] ;

[0148] in, top Indicates business topic tags; T Represents a set of normalized document blocks D doc The total number of document blocks contained, that is, the number of elements in the collection;

[0149] Next, pre-defined business entity types (such as "Business Object: Customer / Order", "Data Field: Customer ID / Order Amount", "Business Constraint: Not Null / Positive Number", etc.) are input into the large model along with document blocks and entity type constraints to extract business document entities. e i And calculate the corresponding confidence level:

[0150] ;

[0151] In the formula: This represents the weighting coefficient (usually 0.7). Indicates the output entity of the large model e i The probability (e.g.: GPT of logits (normalized value) Representing entities e i industry-standard terminology database IndusttyTerm semantic similarity;

[0152] Preset entity confidence threshold Conf the (Usually 0.8), retained The entities form an entity set: , N e Represents a set of entities E The total number of entities included;

[0153] Given a predefined entity relationship type (e.g., "Association: A→B", "Containment: A→B", "Constraint: A→B", etc.), input a document block + entity set + relationship type, extract the relationships between entities and calculate the corresponding confidence scores.

[0154] ;

[0155] In the formula: This represents the weighting coefficient (usually 0.8). Represents the output relationship of the large model r j The probability (obtained by converting the logits vector of the output layer of the large model into a probability distribution); Representing relations r j Semantic consistency with the entity set (e.g., 1 if the association "Customer ID → Order ID" conforms to the business logic, 0 otherwise);

[0156] Preset relationship confidence threshold Conf thr (Usually 0.75), retained Relationships that form a set of relationships ;

[0157] Preset entity attribute types (such as "data type", "business constraint", "formatting requirements", etc.), input document blocks + entities, extract entity attributes and calculate corresponding confidence scores:

[0158] ;

[0159] In the formula: This represents the weighting coefficient (typically 0.75). Represents the output attributes of the large model The probability (obtained through the probability distribution of the model's output layer); representing the attribute With entity e i The matching validity (e.g., if the "order amount" constraint > 0 is valid, then it is 1);

[0160] Preset attribute confidence threshold Conf tha (Usually 0.8), retained The attributes form an attribute set. , Represents a set of attributes A The total number of attributes included;

[0161] entity collection E Relationship combination R Attribute set A Import a graph database (such as Neo4j) and build an initial business knowledge graph: B-KG = ( E,R,A ), and add metadata (source document, extraction time, confidence level, etc.) to each element.

[0162] Step S13: Finally, implement cross-modal unified semantic representation encoding, specifically as follows:

[0163] By using cross-modal unified semantic representation, augmented data from different modalities are transformed into unified high-dimensional semantic vectors:

[0164] ;

[0165] In the formula: Let represent the weight coefficients of the large model embedding and the weight coefficients of the context embedding, respectively (where, It embodies semantic dominance. (Reflecting the relevance of the scenario) e i express Link to business knowledge graph B-KG The entity in KG-Emb() Representing business knowledge graphs B-KG Entity embedding functions (such as) TransE Algorithm or TransR algorithm); c i express The business context in which it is located (i.e., the first) i The data entries are located in business scenarios such as "order payment scenario" or "customer registration scenario". Context-Emb() The context embedding function (which can use general text embedding models such as BERT-base-chinese, Llama 3-8B, and text-embedding-ada-002) outputs a context vector based on the business scenario in which the data is located.

[0166] Noise filtering is used to remove noisy data that does not match the semantics of the business or the scenario.

[0167] ;

[0168] Preset noise threshold (Usually 0.3), if If the expression is 0, it is removed; otherwise, the corresponding unified semantic vector is retained. vi ;

[0169] Finally, a unified semantic representation vector set across modalities is obtained:

[0170] , ( d (This refers to the vector dimension, typically 512 / 1024).

[0171] Step S2, Feature Extraction:

[0172] Step S21, Semantic rule mining and feature pattern extraction: This includes business document rule mining and historical data feature pattern extraction.

[0173] Specifically, business document rule mining involves: first, inputting enterprise business documents and breaking them down into semantically coherent business topic blocks; then, constructing a rule structure that includes "business topic block + rule type + output format requirements". Prompt and will Prompt The data is fed into a large model to obtain preliminary rule candidates; then, the semantic parsing capabilities of the large model are used to map natural language constraints into structured expressions.

[0174] ;

[0175] In the formula: LLM-Parse() Represents the semantic parsing function of a large model (e.g.) Prompt Guided parsing, Llama 2-7B, Qwen-1.8B parsing, etc., convert natural language into structured conditions; Constraint n Represents the natural language constraints of the large model output (e.g., "order date is later than customer registration date"); Entity type Indicates the type of the business entity (e.g., "Order Date" is a time type);

[0176] Calculate the confidence level of the rule:

[0177] ;

[0178] In the formula: rt j Indicates the first j The business rules extracted from the document; The weight coefficients representing the output of the large model (reflecting the reliability of extraction, generally...) ); This represents the probability that the large model outputs the rule (in generative models, it is the joint probability of the rule and the corresponding token sequence; in classification models, it is the Softmax probability). This indicates the consistency between the rule and common sense in the industry or business (values ​​range from 0 to 1, obtained through reasoning from a large model, such as "customer mobile phone number is 11 digits" conforming to common sense, its consistency is 1);

[0179] Preset rule confidence threshold Conf thrt (Usually 0.7), retained The rules are used to filter out low-confidence rules;

[0180] Resolve conflicts arising from extraction rules across different documents or topics by using conflict resolution rules:

[0181] ;

[0182] In the formula: ConfSet A set of rules representing conflicts; rt f Representing the set of conflict rules ConfSet The optimal rule that is ultimately retained after conflict resolution; rt Representing the set of conflict rules ConfSet Any one of the candidate business rules in the set (i.e., the traversal variable of all rules in the set); This indicates the authority of the document from which the rule is sourced (e.g., the authority of the "Data Quality Standard" is 1, while that of a regular SOP is 0.8). rt sou Representation rules rt The source document;

[0183] Finally, the filtered and resolved rules are organized into a set of rule templates: ;in, rt f-j It includes rule types (such as length, range, association, etc.), business entities, and constraints; M Representation of rule template set RT The total number of rule templates included;

[0184] Historical feature data feature pattern extraction: Input high-quality historical dataset D hist Compared with historical quality problem datasets D error Completed through pre-training HGNN Heterogeneous Graph Neural Networks (HGNNs) extract feature pattern sets across fields / tables. Specifically, this involves: first, processing high-quality historical datasets... D hist Compared with historical quality problem datasets D errorThe transformation into a heterogeneous graph structure includes node feature initialization and edge construction.

[0185] Node feature initialization includes table nodes and field nodes. Table node features: x table =[Number of records (i.e., ...] D hist [Number of records in the table), number of fields (i.e., the number of fields contained in the table), record anomaly rate (i.e., the record anomaly rate in the table); Field node characteristics: from D hist Extracting statistical features x field =[mean (i.e.) D hist The mean and variance of the middle field. D hist The variance of the middle field), distribution entropy (i.e. D hist The distribution entropy of the middle field reflects the diversity of values, and the anomaly rate (i.e., D hist [Abnormality rate of the field], field type encoding (e.g., vertical is 1, string is 2, time is 3, etc.);

[0186] Edges include contain edges (directed edges constructed directly from the table's field list, connecting "table node → field node"; indicating that the table contains a certain field), association edges (directed edges constructed from database foreign keys and association rules in business documents, connecting "table A.field → table B.field"; indicating cross-table field relationships), and computed edges (edges constructed from the table's field list, connecting "table A.field → table B.field"). D hist For pairs of fields with numerical relationships, construct directed edges "field A → field B" and label the edge type as "ratio", "summation", etc.; representing the numerical relationship between fields.

[0187] By integrating the metadata of the table fields with the above features, a heterogeneous graph is constructed: ;in, J The union of the table node feature set and the field node feature set. It is the union of three types of edges: edges, associated edges, and computed edges. The corresponding type of the node (i.e., table node or field node). The edge type (i.e., any one of the following: containing edge, associated edge, or computed edge).

[0188] Then, a Heterogeneous Graph Attention Network (HAN) is used to learn the embedded representation of nodes. HAN is a deep learning model for processing heterogeneous graph data, which refers to a graph composed of different types of nodes or edges, and uses an attention mechanism to perform node-level and semantic-level aggregation. HAN can assign different attention weights to different types of nodes / edges, thereby accurately capturing cross-table / cross-field association patterns.

[0189] Node type-level attention aggregation: For a certain type of node F i (Such as field nodes or table nodes), group their neighbor nodes by edge type, i.e. Then, aggregate each group of neighboring nodes separately to obtain the aggregated features of that group of neighboring nodes:

[0190] ;

[0191] In the formula: Represents a node F i In edge type Next, aggregate its neighboring nodes. u The intermediate feature vector obtained after feature extraction; The activation function is typically LeakyReLU. Represents a node F i In edge type The set of neighboring nodes; This represents the corresponding feature transformation matrix (obtained through learning), used to transform neighboring nodes. u The original features are mapped to the current node. F i Matching feature dimensions; Representing neighboring nodes u The original feature vector;

[0192] Representing edge type Downstream and neighboring nodes u For nodes F i The attention weights reflect the contribution of neighboring nodes to the current node's pattern:

[0193] ;

[0194] In the formula: Representing edge type The corresponding attention learning vector (i.e., attention score, which is learned during model training); This represents the vector concatenation operation, i.e., the transformed vector... F i Features and Transformed u The features are concatenated into a longer vector;

[0195] By concatenating the aggregation degree features under different edge types, nodes can be obtained. F i Final embedding:

[0196] ;

[0197] Then, in heterogeneous graphs G HGNN In the middle, excavation D hist Frequently appearing in and D error A rare subgraph structure in China, where the frequency of a subgraph is defined as:

[0198] ;

[0199] In the formula: Indicates that subgraph S is in D hist The actual number of times it appears in the text express D hist The total number of all possible subgraphs in the graph;

[0200] like Supp(S) >0.7 and Supp(S,D error ) If the value is less than 0.2, it is considered as part of the candidate pattern set. FieldSet k ;in, Supp(S,D error ) Subgraph S In the error sample set D error The frequency of occurrence, i.e.:

[0201] ;

[0202] In the formula: Indicates that subgraph S is in D error The actual number of times it appears in the text express D error The total number of all possible subgraphs in the graph;

[0203] Embedding nodes enter ​In clustering algorithms, fields with similar embeddings are grouped into one class to obtain clustered fields (fields in the same cluster are...). D hist (A stable cross-field pattern exists within the cluster), analyze the statistical interaction characteristics of fields within each cluster, and obtain the constraints corresponding to the feature pattern. ​ k (For example, if "the ratio of fields A to B within a cluster is stable between 0.8 and 1.2", then the constraint is: ratio ∈ [0.8, 1.2]), ultimately forming a feature pattern set:

[0204] ;

[0205] in, Representation pattern in D hist The stability of the statistic (obtained by the degree of fluctuation of the statistic); ​ k This is the set of candidate patterns.

[0206] Step S22, Automatic Rule Generation and Priority Sorting: The obtained rule templates are then automatically generated and prioritized. ​ With feature pattern set ​ Input a large model, and the large model calculates the probability of rule generation:

[0207] ;

[0208] In the formula: ​ This represents the prediction score of the large model for the combination of rule templates and feature pattern sets (obtained through the semantic reasoning ability of the large model, such as in GPT-4V, which uses generative probability output, i.e., logits normalization + Prompt-guided reasoning; first, construct the Prompt: "Rule template: { ​}, Feature Pattern: { ​ The process involves evaluating the suitability of the combination of the two methods for data quality detection, outputting a score from 0 to 1 (higher scores indicate better suitability); secondly, extracting the model's output score, or calculating the probability score of the combined semantic consistency using logits vectors, as... ​ (Results)

[0209] Arrange the rules in descending order of their generation probability and select the rules with the highest generation probability. L Combining these elements generates a set of structured rules: Each rule r i It includes rule ID, rule name, applicable modality, applicable scenario, constraints, business semantic interpretability, confidence level, etc.

[0210] Structured rules are evaluated based on business impact, data coverage, and confidence level. ri Weighted priority calculations are performed to ensure that high-value rules are executed first.

[0211] ;

[0212] In the formula: ​ i ) The business impact of the rule is represented by the business semantic reasoning of the large model. The input is the business process description corresponding to the rule, such as the order amount rule being associated with transaction settlement. The output is a score between 0 and 1, such as: the order amount rule has a score of 0.9, and the customer remarks rule has a score of 0.3. ​ i ) The data coverage of the rule (represented by "Applicable Rule") r i (obtained by "data volume / total data volume"). ​ i ) The confidence level of a rule (which can be directly obtained from the generation probability of the corresponding rule); These represent the corresponding weight coefficients (generally) );

[0213] According to the rules P(r i ) Sort by high to low priority, and rules with higher priority are executed first in data quality inspection.

[0214] Step S23, Dynamic Binding of Rules and Data Features: First, bind the generated rules... r i With cross-modal unified semantic representation vector v i Bind:

[0215] ;

[0216] Preset binding threshold ​ th (Usually 0.6), if Then the corresponding rule will be categorized into the applicable set of data;

[0217] Real-time monitoring of data distribution and business document changes, dynamically updating, deprecating, and adding rules:

[0218] Calculate the current data distribution P sent With high-quality data distribution Q ant KL divergence:

[0219] ;

[0220] Preset ​ divergence threshold ​ th (Usually 0.3), if greater than ​ th This proves that the data distribution changes significantly;

[0221] Calculate the current document ​ new Compared with the original document ​ old Semantic similarity between them:

[0222] ;

[0223] Preset document similarity threshold ​ Dth (Generally 0.7), if less than ​ Dth This proves that the business documents have been significantly updated;

[0224] Evolutionary rule setting: If the statistical pattern corresponding to the rule exceeds ​ th Furthermore, the KL divergence of adjacent windows in subsequent consecutive windows (usually 3) all exceed [a certain value]. ​ th If the corresponding rule is removed, then remove the rule; if the statistical pattern corresponding to the rule exceeds [a certain threshold], then remove the rule. ​ th Furthermore, the KL divergence of adjacent windows in subsequent consecutive windows (usually 3) does not exceed ​ th If the document similarity corresponding to the rule is less than 1, then adjust the constraints of the rule; if the document similarity corresponding to the rule is less than 1. ​ Dth Then, a new template generation rule is established based on the newly added constraints in the business document;

[0225] The evolved rules are recalculated for priority and dynamically bound to data features to ensure the timeliness of the rules.

[0226] Step S3, data quality inspection, specifically:

[0227] Step S31, Joint encoding of multimodal data quality features:

[0228] First, the cross-modal unified semantic representation vector v i Mapped to corresponding text description v text,i , set of rules R i Convert to text description R text,i It employs a semantic encoding function of a large model to output semantic feature vectors.f s,i :

[0229] ;

[0230] In the formula: ​ Text embedding functions for large models (such as semantic encoding of GPT-4V, using the text-embedding-ada-002 interface, directly calling the API, inputting text description, such as "Rule: Order amount > 0; Data: Order amount = 500", outputting a 1536-dimensional semantic vector to meet the requirement of "high-dimensional semantic representation"; or using the built-in embedding layer Llama 3 Embedding Layer of Meta Llama3).

[0231] For structured data, construct a data relationship graph. G s,i =( J , ),in, J The union of the table node feature set and the field node feature set. To encode the graph structure using the GNN encoding method, we use the union of three types of edges: containing edges, associated edges, and computed edges.

[0232] ;

[0233] In the formula: ​ Representing heterogeneous graphs ​ (like ​ The structure encoding function can be used (existing DGL, HANConv (heterogeneous graph attention network layer) in the Deep Graph Library; for example: first construct the heterogeneous graph object ( ​ Define node types (tables, fields, etc.) and edge types (containment, association, computation, etc.); then call... ​ The layer aggregates neighbor features of different edge types through an attention mechanism, outputting the structural embedding vector of the node, which is directly used as... ​ (Results)

[0234] For unstructured data, the intrinsic structure of the content is encoded through a structure encoding function of a large model (which transforms the content structure of unstructured data into a fixed-dimensional structure feature vector):

[0235] ;

[0236] In the formula: ​The structure encoding function for large models (for text structure encoding, it uses BERT's hierarchical encoding + sequence aggregation; for example, for a text sentence sequence, such as "order creation → payment → shipment", BERT's Transformer layer encodes the positional information and contextual relationships of each sentence; then, it uses "mean pooling" or "max pooling" to aggregate the sequence vectors, outputting fixed-dimensional structural features, corresponding to...) ​ For image structure encoding, GPT-4V region encoding is used. For example, the input image and target bounding box coordinates are given, such as "in an order screenshot, the product name box is (100, 200, 300, 400), and the amount box is (500, 200, 600, 400)". Then, the region feature encoding function of the multimodal model is called, such as GPT-4V's image_region_embedding(), to output the spatial relationship and associated structure vector of the target bounding box. ​ c,i Represents the content structure of unstructured data (such as a sequence of sentences in text, a set of bounding boxes in an image, etc.).

[0237] semantic feature vector f s,i With structural feature vectors f t,i Concatenate the features to obtain the joint feature vector:

[0238] ;

[0239] Step S32, Context-Aware Quality Anomaly Detection:

[0240] Use context embedding functions to embed business scenarios c i Encoded as scene vectors c emb,i and joint features f i Concatenate the data, input it into the large model context inference module, and calculate the anomaly probability:

[0241] ;

[0242] In the formula: ​ This represents the classification inference function of the large model, outputting anomaly probabilities from 0 to 1 (obtained using the GPT-4V function call pattern + classification Prompt; for example, first construct the Prompt: "Business scenario: order payment; Joint features: { f i}; Scene vector: { c emb,iThe algorithm determines whether the data has quality anomalies and outputs an anomaly probability of 0 to 1. Then, it extracts the probability value from the model output or obtains the normalized probability through the API's logprobs parameter, as... ​ (Results)

[0243] Preset anomaly detection threshold (Typically 0.5), determine the abnormal state based on the abnormal probability and output the abnormal confidence level. ​ i ) :

[0244] , ;

[0245] Output quality anomaly detection set And the corresponding anomaly confidence level.

[0246] Step S4, Root Cause Analysis and Repair:

[0247] Step S41, Fusion of multi-source detection results: For each data v i Check each one to see if it meets the requirements. R i If each rule constraint is satisfied, then an abnormal rule matching result will occur. r m,i =0, if the data v i Not satisfied R i The constraints of any rule, then r m,i =1, generate a set of rule matching exceptions. ;

[0248] Obtain data v i In the rules r j Single rule anomaly confidence ​ j,i :

[0249] ;

[0250] In the formula: ​ j ) This represents the confidence level of the rule itself (i.e., the confidence level output from the rule generation stage, specifically...). ​ j ) ); ​ j,i Representing data v iOn the rules r j Severity of violation:

[0251] ;

[0252] For data { v i The set of applicable rules for} R i All rules in the algorithm are merged to obtain the rule matching anomaly confidence score:

[0253] ;

[0254] Based on the quality anomaly detection set, the rule-matched anomaly set, and the corresponding confidence scores, the fusion anomaly score is calculated:

[0255] ;

[0256] In the formula: w oi , w ri These represent the corresponding weighting coefficients (generally) w oi =0.6, w ri =0.4);

[0257] Preset score threshold (Usually 0.5):

[0258] ;

[0259] The final set of quality anomaly determination results is obtained as follows: .

[0260] Step S42, Deep Root Cause Analysis: Based on the Business Knowledge Graph (B-KG) and historical quality issue data, construct a causal graph containing "nodes-causal edges-causal probabilities". Nodes include data entities (such as "customer name", "order date", etc.), rules (such as "name length rule", "cross-table join rule", etc.), business processes (such as "customer registration process", "order settlement process", etc.), system components (such as "input module", "time synchronization component", etc.), and abnormal events (such as "semantic inconsistency", "rule matching anomaly", etc.). Causal edges are directed edges (e.g., x→y means node x is the cause of node y).

[0261] Causal probability P ( J v | J u ) represents a node Ju When it happens, node J v Conditional probability of occurrence:

[0262] ;

[0263] In the formula: Indicating historical quality issues in the data J u and J v The number of times they occur simultaneously; Indicating historical data J u The number of times it occurs; ​ u ,J v ) Represents the reasoning from the large model J u arrive J v The causal strength (values ​​[0,1], with 1 for strong causality and 0 for weak causality); obtained using GPT-4V's causal inference Prompt+score output, for example, first constructing the Prompt: "Known nodes..." J u :{ J u Description}, Node J v :{ J v Description}, based on common business sense (such as customer registration process), determine J u Is it J v The model identifies the cause and outputs a causal strength value of 0 to 1, where 1 represents strong causality and 0 represents weak causality. The model's output score is then extracted, or the results are corrected through multiple rounds of follow-up questions to ensure that the causal judgment aligns with business logic. This represents the weighting coefficient, used to balance data statistics with large model inference (typically 0.7).

[0264] Based on the constructed causal graph, the final abnormal results are analyzed. O final Root cause localization: This involves identifying the final abnormal results. O final Anomalous events are mapped to target nodes in the causal graph. J mv (i.e., the node corresponding to the anomaly); search the causal graph for the target node. J mv Obtain the set of all directed paths to the destination. ​e ) And for each path p lj ∈ ​ e ) Obtain the product of the probabilities of all causal edges on the path:

[0265] ;

[0266] In the formula: Indicates traversing the causal path p lj All causal edges ( J u , J v ), and perform a multiplication operation on the causal probability of each edge;

[0267] The optimal root cause path is selected by using the product of maximum probabilities.

[0268] ;

[0269] Finally, the rightmost root cause path ​ e ) The corresponding node sequence is transformed into a root cause chain described in natural language.

[0270] Step S43, Explainability Report Generation: Collect data including anomalous data (such as anomalous results). O final ), root cause analysis (such as optimal root cause path) ​ e ) ), the impact of abnormal business operations obtained based on the business knowledge graph (i.e., the impact on downstream operations), and the applicable rule-level content that triggered the abnormality (from the rule set). R i ), generate an interpretability report.

[0271] Specifically, the impact of abnormal business operations on downstream processes, obtained based on business knowledge graphs, is as follows:

[0272] Based on the structured association of entities, relationships, and attributes in the business knowledge graph, the abnormal events output in step S42 are first mapped to the corresponding nodes in the business knowledge graph to form the abnormal starting node, providing a starting point for tracing downstream impacts.

[0273] Then, starting from the anomaly initiation node, and relying on the directed edges (associations, inclusions, constraints, etc.) of the business knowledge graph, the downstream associated nodes are recursively traversed. The impact is deduced hierarchically by combining node attributes and metadata, including: 1. Starting from the anomaly initiation node, extracting all downstream nodes corresponding to the directed edges in the business relationship graph to form an entity set; 2. Each entity in the business knowledge graph contains business attributes and metadata, added during the construction in step S12. These attributes are used to quantify the scope, level, and type of the impact; 3. Using the semantic description of the relationships in the business knowledge graph, combined with the business semantic reasoning capabilities of the large model, the specific consequences for each affected entity are clarified; 4. The scope, level, and type of impact in point 3 are integrated with the specific consequences of the impact in point 4 and incorporated into the interpretability report.

[0274] Specific example: Taking the abnormal starting node "order amount = -100" as an example, its downstream directed relationships in the business knowledge graph are: order amount → financial reconciliation amount (association), order amount → order settlement process (constraint), order amount → financial accounting rules (dependency). Then the set of affected entities is {financial reconciliation amount, order settlement process, financial accounting rules}.

[0275] The specific consequences clearly defined by the large model are as follows: Financial reconciliation amount → Derived consequence: The calculation of the financial reconciliation amount shows a negative difference, resulting in an imbalance in the reconciliation; Order settlement process → Derived consequence: The order settlement process is blocked due to abnormal amounts, and payment confirmation cannot be completed; Financial accounting rules → Derived consequence: The financial accounting rules cannot generate compliant accounting vouchers due to invalid order amounts;

[0276] Final output: Impact of abnormal business operations:

[0277] Scope of impact: Financial system, payment system; Covered business scenarios: Order settlement, financial reconciliation, monthly accounting;

[0278] Impact level: High risk (based on the "Order Amount" entity in the business knowledge graph having a priority of 0.9 and a constraint level of mandatory).

[0279] Impact types: Data consistency impact + Process blocking impact + Rule failure impact;

[0280] Specific consequences: negative discrepancies in financial reconciliation, resulting in unbalanced accounts; blocked order settlement process, leading to payment confirmation failure; inability to trigger financial accounting rules, resulting in missing compliant vouchers; and impact on monthly closing progress.

[0281] Step S44, Adaptive Repair Execution and Closed-Loop Verification: Based on the root cause analysis type and business knowledge graph, the large model extracts suitable repair strategies, including automatic correction and manual review (e.g., if the ID card interface can be directly called to match and replace abnormal names, then automatic correction is performed; if automatic correction is not possible, the abnormal data is pushed to the operations personnel for correction). The repaired data is then re-entered into the data preprocessing stage for full-process quality inspection to verify the repair results: if the abnormality rate is still higher than the threshold, rule updates or root cause analysis are triggered for iterative repair until the requirements are met.

[0282] Example 2:

[0283] As another preferred embodiment of the technical solution of the present invention, based on the solution of Embodiment 1 above, in step S12 above, if the entity to be linked matches multiple candidate entities, the optimal entity is selected by multi-dimensional weighted similarity:

[0284] ;

[0285] In the formula: w 1. w 2. w 3 represents the corresponding weighting coefficients (generally) ); This indicates the similarity between the entity type to be linked and the candidate entity type (e.g., the type similarity between "data field" and "data field" is 1). This indicates the similarity between the relationship between the entity to be linked and the candidate entity (e.g., the similarity between "User ID associated with Order ID" and "Customer ID associated with Order ID" is 0.9).

[0286] Selecting multiple candidate entities Maximum and The candidate entities are used as the final linking results.

[0287] Example 3:

[0288] As another preferred embodiment of the technical solution of the present invention, this embodiment provides a data quality management system based on a large model, including a data access layer, a data processing layer, a feature extraction layer, a quality detection layer, a result processing layer, and an application service layer. The data access layer includes structured data sources, semi-structured data sources, unstructured data sources, and data adapters. The data processing layer corresponds to the data preprocessing method in step S1 of embodiment 1. The feature extraction layer corresponds to the feature extraction method in step S2 of embodiment 1. The quality detection layer corresponds to the data quality detection method in step S3 of embodiment 1. The result processing layer corresponds to the root cause analysis and repair method in step S4 of embodiment 1. The application service layer includes API interfaces, web interfaces, early warning notifications, mobile terminals, etc.

Claims

1. A data quality detection and management method based on a large model, characterized in that: include: Step S1, Data Preprocessing: First, a large model is used to deconstruct and modally label the input raw data, specifically as follows: For structured data: extract field names, field types, field values, and inter-table join keys; For semi-structured data: parsing the hierarchical structure and extracting node semantics and attributes; For unstructured data: extract keywords and semantic roles from text, and extract semantic content from images; The final output is a deconstructed dataset with modal labels and basic semantics. D : ； In the formula: S Representing structured data, SS This represents semi-structured data. U Represents unstructured data; Then, entity linking and semantic enhancement are driven by business knowledge graphs, specifically: First, a business knowledge graph is constructed using a large model. B-KG ; Then deconstruct the data set Entity and Business Knowledge Graph B-KG Linking entities in the destructuring process involves extracting the entities to be linked from the destructuring results to form a set of entities to be linked. The large model considers each entity to be linked. e data,m Perform preliminary semantic matching and filter business knowledge graphs. B-KG Entities with a string similarity of not less than 0.5 or a preliminary semantic similarity of not less than 0.4 are selected as the candidate set. ; Then, by leveraging the text embedding capabilities of the large model, the entities to be linked are calculated. e data,m With candidate entities e hx,n Semantic similarity: ； In the formula: LLM-Emb() This represents a large model text embedding function used to convert input text into a high-dimensional semantic vector; Preset similarity threshold Sim X If the semantic similarity is not less than the similarity threshold, it is considered a valid match, and the corresponding candidate entity is the link result. Finally, the business knowledge graph B-KG The entity attributes linked in the middle are directly mapped to the entities in the deconstructed data set, outputting an enhanced data set with the desired business semantic attributes: ,in, s i Represents a set of business semantic attributes; Finally, a unified semantic representation encoding across modalities is achieved, specifically as follows: By using cross-modal unified semantic representation, augmented data from different modalities are transformed into unified high-dimensional semantic vectors: ； In the formula: These represent the weight coefficients of the large model embedding and the weight coefficients of the context embedding, respectively. e i express Link to business knowledge graph B-KG The entity in KG-Emb() Representing business knowledge graphs B-KG Entity embedding function; c i express The business context in which it is located Context-Emb() This indicates a context embedding function that outputs a context vector based on the business scenario in which the data resides. Noise filtering is used to remove noisy data that does not match the semantics of the business or the scenario. ； Preset noise threshold ,like If the expression is 0, it is removed; otherwise, the corresponding unified semantic vector is retained. v i ; Finally, a unified semantic representation vector set across modalities is obtained: ； Step S2, Feature Extraction: This includes semantic rule mining and feature pattern extraction, automatic rule generation and priority ranking, and dynamic binding of rules and data features; Semantic rule mining and feature pattern extraction include business document rule mining and historical data feature pattern extraction. Specifically, business document rule mining involves: first, inputting enterprise business documents and breaking them down into semantically coherent business topic blocks; then, constructing a rule structure that includes "business topic block + rule type + output format requirements". Prompt and will Prompt The data is fed into a large model to obtain preliminary rule candidates; then, the semantic parsing capabilities of the large model are used to map natural language constraints into structured expressions. ； In the formula: LLM-Parse() This represents a semantic parsing function for large models, which converts natural language into structured conditions; Constraint n Represents the natural language constraints of the large model output; Entity type Indicates the type of business entity; Calculate the confidence level of the rule: ； In the formula: Represents the weight coefficients output by the large model; This indicates the probability that the large model outputs this rule; This indicates that the rules are consistent with industry or business common sense; Preset rule confidence threshold Conf thrt ,reserve The rules are used to filter out low-confidence rules; Resolve conflicts arising from extraction rules across different documents or topics by using conflict resolution rules: ； In the formula: ConfSet A set of rules representing conflicts; This indicates the authority of the source document for the rules; Finally, the filtered and resolved rules are organized into a set of rule templates: ;in, rt f-j Includes rule type, business entity, and constraints; Historical feature data feature pattern extraction: Input high-quality historical dataset D hist Compared with historical quality problem datasets D error Completed through pre-training HGNN Extract feature pattern sets across fields / tables ; Automatic rule generation and priority sorting specifically involve: obtaining rule templates... RT With feature pattern set FP Input a large model, and the large model calculates the probability of rule generation: ； In the formula: LLM-Predict() This represents the prediction score of the large model for the combination of rule templates and feature pattern sets. Arrange the rules in descending order of their generation probability and select the rules with the highest generation probability. L Combining these elements generates a set of structured rules: ; Structured rules are evaluated based on business impact, data coverage, and confidence level. r i Weighted priority calculations are performed to ensure that high-value rules are executed first. ； In the formula: Imp(r i ) This indicates the degree of business impact of the rule; Cover(r i ) This indicates the data coverage of the rule; Confi(r i ) Indicates confidence in the rule; These represent the corresponding weight coefficients; According to the rules P(r i ) Sort from highest to lowest, and rules with higher priority are executed first in data quality inspection; The dynamic binding of rules and data features specifically involves: first, generating the rules... r i With cross-modal unified semantic representation vector v i Bind: ； Preset binding threshold Match th ,like Then the corresponding rule will be categorized into the applicable set of data; Real-time monitoring of data distribution and business document changes; dynamic updating, elimination, and addition of rules: Calculate the current data distribution P sent With high-quality data distribution Q ant KL divergence: ； Preset KL divergence threshold KL th If greater than KL th This proves that the data distribution changes significantly; Calculate the current document Doc new Compared with the original document Doc old Semantic similarity between them: ； Preset document similarity threshold Sim Dth If less than Sim Dth This proves that the business documents have been significantly updated; Evolutionary rule setting: If the statistical pattern corresponding to the rule exceeds KL th Furthermore, the KL divergence of adjacent windows in subsequent consecutive windows all exceeded [a certain value]. KL th If the corresponding rule is removed, then remove the rule; if the statistical pattern corresponding to the rule exceeds [a certain threshold], then remove the rule. KL th Furthermore, the KL divergence of adjacent windows in subsequent consecutive windows does not exceed KL th If the document similarity corresponding to the rule is less than 1, then adjust the constraints of the rule; if the document similarity corresponding to the rule is less than 1. Sim Dth Then, a new template generation rule is established based on the newly added constraints in the business document; The evolved rules are recalculated for priority and dynamically bound to data features again. Step S3, Data Quality Inspection: First, perform joint encoding of multimodal data quality features, then perform context-aware quality anomaly detection; specifically: Joint encoding of multimodal data quality features: First, the cross-modal unified semantic representation vector v i Mapped to corresponding text description v text,i , set of rules R i Convert to text description R text,i It employs a semantic encoding function of a large model to output semantic feature vectors. f s,i : ； In the formula: LLM-Encode() A text embedding function representing a large model; For structured data, construct a data relationship graph. G s,i =( J , ),in, J The union of the table node feature set and the field node feature set. To encode the graph structure using the GNN encoding method, we use the union of three types of edges: containing edges, associated edges, and computed edges. ； In the formula: GNN-Sencode() Representing heterogeneous graphs GNN The structure encoding function; For unstructured data, the inherent structure of the content is encoded using the structure encoding function of a large model: ； In the formula: LLM-Sencode() The structural encoding function represents the large model; Uns c,i Represents the content structure of unstructured data; semantic feature vector f s,i With structural feature vectors f t,i By concatenating the features, we obtain the joint feature vector: ； Context-aware quality anomaly detection: Use context embedding functions to embed business scenarios c i Encoded as scene vectors c emb,i and joint features f i Concatenate the data, input it into the large model context inference module, and calculate the anomaly probability: ； In the formula: LLM-Infer() Represents the classification inference function for a large model; Preset anomaly detection threshold Determine the abnormal state based on the abnormal probability and output the abnormal confidence level. Conf(o i ) : ， ； Output quality anomaly detection set And the corresponding anomaly confidence level; Step S4, Root Cause Analysis and Remediation: This includes the fusion of multi-source detection results, deep root cause analysis, interpretability report generation, adaptive remediation execution, and closed-loop verification. The fusion of multi-source detection results is specifically as follows: For each data v i Check each one to see if it meets the requirements. R i If each rule constraint is satisfied, then an abnormal rule matching result will occur. r m,i =0, if the data v i Not satisfied R i The constraints of any rule, then r m,i =1, generate a set of rule matching exceptions. ; Obtain data v i In the rules r j Single rule anomaly confidence Conf j,i : ； In the formula: Confidence(r j ) Indicates the confidence level of the rule itself; Vio j,i Representing data v i Regarding the rules r j Severity of violation: ； For data { v i The set of applicable rules for} R i All rules in the algorithm are merged to obtain the rule matching anomaly confidence score: ； Based on the quality anomaly detection set, the rule-matched anomaly set, and the corresponding confidence scores, the fusion anomaly score is calculated: ； In the formula: w oi , w ri These represent the corresponding weight coefficients; Preset score threshold : ； The final set of quality anomaly determination results is obtained as follows: ; Deep root cause analysis specifically includes: Based on the business knowledge graph B-KG and historical quality problem data, a causal graph containing "node-causal edge-causal probability" is constructed. Nodes include data entities, rules, business processes, system components, and abnormal events; causal edges are directed edges. Causal probability P ( J v | J u ) represents a node J u When it happens, node J v Conditional probability of occurrence: ； In the formula: Indicating historical quality issues J u and J v The number of times they occur simultaneously; Indicating historical data J u The number of times it occurs; LLM-Causal(J u ,J v ) Represents the reasoning from the large model J u arrive J v The causal strength; Represents the weighting coefficients, used to balance data statistics and large model inference; Based on the constructed causal graph, the final abnormal results are analyzed. O final Root cause localization: This involves identifying the final abnormal results. O final Anomalous events are mapped to target nodes in the causal graph. J mv Searching for target nodes in the causal graph J mv Obtain the set of all directed paths to the destination. P(J e ) And for each path p lj ∈ P(J e ) Obtain the product of the probabilities of all causal edges on the path: ； In the formula: Indicates traversing the causal path p lj All causal edges ( J u , J v ), and perform a multiplication operation on the causal probability of each edge; The optimal root cause path is selected by using the product of maximum probabilities. ； Finally, the rightmost root cause path PathM(p e ) The corresponding node sequence is transformed into a root cause chain described in natural language. The process of generating an interpretability report involves collecting data including anomalous data, root cause analysis, the impact of anomalous business operations obtained from a business knowledge graph, and the applicable rule-level content that triggered the anomalous operation, and then generating an interpretability report. The adaptive repair execution and closed-loop verification are as follows: Based on the root cause analysis type and business knowledge graph, the large model extracts the appropriate repair strategy, which includes automatic correction and manual review; the repaired data is then re-entered into the data preprocessing stage for full-process quality inspection to verify the repair results; if the anomaly rate is still higher than the threshold, rule updates or root cause analysis are triggered for repeated iterative repair until the requirements are met.

2. The data quality detection and management method based on a large model according to claim 1, characterized in that: The large model uses either GPT-4V or Llama 3-Vision.

3. The data quality detection and management method based on a large model according to claim 1, characterized in that: The raw data includes structured tables, semi-structured JSON / XML, unstructured text, or images.

Images