A data quality control compliance system fused with AI technology

Abstract

The application discloses a data quality control compliance system fusing AI technology, relates to the technical field of data processing, and solves the technical problems that efficient cataloging and sharing of data are difficult to realize, comprehensive understanding and adaptability to data regulations are lacked, changing compliance requirements are difficult to meet, and potential value of data assets is limited; through a full-chain AI driving engine: a full-link AI engine of "data exploration-data directory identification-data intelligent cataloging-data quality detection-data quality inspection report generation-directory scene construction" is innovatively constructed, the limitation of traditional single rule or AI technology is broken through, automation of a closed loop from semantic analysis to cataloging generation and quality detection is realized, and adaptability of the system to a complex data environment is enhanced. The AI+cataloging module adopts a triple matching strategy, and cataloging accuracy and efficiency are greatly improved. Through automatic quality detection and report generation, quality problems in data can be quickly identified.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data management, and particularly relates to a data quality control compliance system fusing AI technology. BACKGROUND

[0002] Data quality control compliance is a cross field of data governance and compliance management, and its core lies in ensuring that data meets certain requirements such as accuracy, completeness, consistency, and timeliness through technical means and management processes in the whole life cycle. For example, if there is a negative number or an abnormal value (such as 200 years old) in the patient age field of medical data, it violates the quality rules. At the same time, the data needs to comply with laws and regulations.

[0003] In the current field of data management and quality control, although the existing solutions have achieved automation to some extent, there are obvious deficiencies in intelligence and adaptability. These systems often lack comprehensive management of the whole life cycle of data, especially in terms of dynamic changes and security compliance of data. In addition, existing data management systems often show performance bottlenecks when dealing with large-scale data sets, making it difficult to adapt to changing data environments and business requirements, not only increasing maintenance costs, but also reducing the flexibility of the system. In the aspect of data cataloging, traditional methods rely on manual operation, which not only consumes time and effort, but also is prone to errors, affecting the accuracy and timeliness of the data catalog.

[0004] In addition, with the growth of data volume and the diversification of data sources, traditional data management systems are not up to the task of integrating and utilizing data, making it difficult to achieve efficient cataloging and sharing of data. In terms of compliance, existing data management systems often lack a comprehensive understanding and adaptability to data regulations, making it difficult to meet changing compliance requirements and limiting the potential value of data assets. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes a data quality control compliance system fusing AI technology, which is used to solve the technical problems of being difficult to achieve efficient cataloging and sharing of data, lacking a comprehensive understanding and adaptability to data regulations, being difficult to meet changing compliance requirements, and limiting the potential value of data assets, and the present application solves the above problems.

[0006] To solve the above problems, the first aspect of the present application provides a data quality control compliance system fusing AI technology, comprising:

[0007] Data exploration module: for scanning and identifying multi-source heterogeneous data, extracting the field name, data type, field length and data key information of each piece of data, and automatically tracking data blood relationship;

[0008] AI+ cataloging module: for using artificial intelligence technology to construct the extracted data into a data table to be cataloged, performing multi-dimensional comparison of the data table to be cataloged through basic matching and structural matching, extracting key information in the data in combination with a catalog element system, unifying various data formats, and generating a standardized data catalog;

[0009] Data classification module: clustering data tables according to business types, access frequencies and extracted data key information, establishing associations in combination with the results of data clustering and the similarity between data, constructing a data association topology graph, analyzing the association coefficient of data, and classifying data according to associations;

[0010] Quality detection module: for using a quality detection model and a pre-set rule engine to construct a screening strategy for different association classification data, processing data streams in combination with an AI+ data rule engine and a Spark big data analysis engine, starting corresponding verification tasks for structured data and unstructured data;

[0011] Report set module: for automatically detecting data of different quality attributes and security attributes for compliance through AI technology, and generating a quality evaluation report and a rectification list.

[0012] Optionally, in one example of the above aspect, the data exploration module comprises:

[0013] Data source access unit: constructing a multi-protocol adaptation interface to support access of structured, semi-structured, unstructured and streaming data;

[0014] Data field information extraction unit: directly reading field name, data type and field length information in metadata for structured data, using regular matching and machine learning classification for semi-structured data, data type Type(F) = Argmax(P(Typek|F_content, F_context)), wherein Typek is a candidate data type, F_content is a field content feature, F_context is a context semantic feature, field name and field length information are extracted; and extracting data key information based on a general corpus entity recognition model;

[0015] Blood lineage tracking unit: for ETL process, parse FROM / JOIN / INSERT INTO keywords in SQL script, extract field mapping relationship to track data lineage; for code processing process: through static analysis to identify variable assignment link, calculate the blood correlation degree Blood(X, Y) = ∑(W(Op_i) x Sim(X, Y, Op_i)) between data X and Y, wherein W(Op_i) is the weight of operation Op_i, Sim(X, Y, Op_i) is the feature similarity of data X, data Y and operation Op_i, and the data with blood correlation degree greater than the threshold is extracted to track the data lineage.

[0016] Optionally, in one example of the above aspect, the AI+cataloging module comprises:

[0017] Intelligent cataloging unit: for automatically analyzing and extracting the field name, data type, field length and data key information by using artificial intelligence technology, and generating a standardized data catalog;

[0018] Intelligent completion unit: for intelligently judging the type and content of missing information by semantic analysis and association mining of cataloged data, combining with the standard model in general corpus, automatically identifying the missing information in the catalog, and intelligently inferring and completing the missing entries by using AI algorithm;

[0019] Cataloging comparison unit: for comparing the cataloged data with the probe data in real time;

[0020] For the data in the data table, the basic attribute comparison is performed, the cosine similarity between the field name, data type, field length and data key information of the data is respectively calculated, and the weighted average of the results is obtained to obtain the basic attribute comparison coefficient of the data in the data table;

[0021] For the data table structure, the table structure similarity is compared by using the graph theory method:

[0022] ; wherein E1 and E2 are the field relationship edge sets of data table T1 and data table T2 respectively, the implicit relationship is identified by a deep learning model, and StructMatch(T1, T2) is the table structure similarity coefficient of data table T1 and data table T2;

[0023] Data catalog generation unit: combining the catalog element system to extract the key information in the data, unifying various data formats, and generating a standardized data catalog.

[0024] Optionally, in one example of the above aspect, the data tables are clustered according to the business type, access frequency and extracted data key information, the correlation is established combining the data clustering results and the similarity between the data, and the data correlation topology graph is constructed, including the following steps:

[0025] The data in the statistical data table is accessed in a preset time period, and the data is clustered and analyzed in combination with the extracted business type data and the data key information.

[0026] Each data in the data table is set as an atomic node, K-means algorithm is used for clustering analysis, the error sum of squares of the K value in the K-means algorithm is calculated by the elbow method, the K value corresponding to the sharp drop point of the error sum of squares is selected, clustering is performed, and K atomic node clustering clusters are obtained.

[0027] The atomic node attributes include: a unique identifier ID, a belonging clustering label, and a core field feature vector.

[0028] Each data table is set as an aggregated node, a data table level node, and the attributes include a table structure feature matrix, an access frequency sequence, and a data table business label vector.

[0029] According to the clustering results of the atomic nodes and the aggregated nodes, respectively, the association edges are established, and a data association topology graph is constructed.

[0030] Optionally, in one example of the above aspect, according to the clustering results of the atomic nodes and the aggregated nodes, respectively, the association edges are established, and a data association topology graph is constructed, including the following steps:

[0031] Any two atomic nodes in the same cluster automatically establish a basic association edge, and a data association topology graph is constructed, the edge weight of the basic association edge is a basic value, the edge weight basic value Ybase is set as 1 / (1+Distance(Node_i,Node_j)), wherein Distance(Node_i,Node_j) is the Euclidean distance of the feature vectors of the nodes Node_i and Node_j.

[0032] The K-means algorithm is used for clustering analysis of the aggregated nodes, and aggregated node clustering clusters are obtained, any two aggregated nodes in the same cluster automatically establish a basic association edge, and the edge weight basic value of the aggregated node is Wbase=S(Node_x,Node_y) / (1+Smax), wherein S(Node_x,Node_y) is the table structure similarity coefficient between the nodes Node_x and Node_y, and Smax is the maximum table structure similarity coefficient between the aggregated nodes.

[0033] Optionally, in one example of the above aspect, the association coefficient of the data is analyzed, and the data is classified according to the association, including the following steps:

[0034] According to the constructed data association topology graph and the edge weight basic value of the nodes, the degree centrality values of each atomic node and aggregated node are calculated, respectively.

[0035] The dynamic attenuation coefficient of the introduced edge and the node attribute weight are combined with the data basic attribute comparison coefficient and the table structure similarity coefficient to analyze the correlation coefficient of the data, and the formula is as follows:

[0036]

[0037] W(Node) is the correlation coefficient of the data, Base(Node_i) and Base(Node_x) are the degree centrality values of the atomic node and the aggregation node respectively, Attr(Node_i) and Attr(Node_x) are the attribute scores of the atomic node and the aggregation node respectively, and De is the attenuation factor;

[0038] The attenuation factor De is e^(-ΔT / T0), wherein ΔT is the number of days from the last update to the current time, and T0 is the attenuation period;

[0039] According to the correlation coefficient of the data from high to low, the data correlation is classified, the ranking [0%, 20%] interval is high correlation data, the ranking located in (20%, 60%] interval is medium correlation data, and the ranking located in (60%, 100%] interval is low correlation data.

[0040] Optionally, in one example of the above aspect, the atomic node attribute score of the atomic node and the aggregation node is calculated by the following formula:

[0041] The atomic node attribute score is:

[0042] wherein M_miss is the amount of defective data of the atomic node, M_total is the total amount of corresponding data of the atomic node, and Ts is the ratio of the time difference from the last update of the corresponding data of the atomic node to the current time to the update period;

[0043] The aggregation node attribute score is:

[0044] ; wherein S_freq is the access frequency normalized value, and V_biz is the data table data integrity.

[0045] Optionally, in one example of the above aspect, the quality detection module comprises:

[0046] The screening unit is configured to construct a screening strategy of different correlation classification data according to the quality detection model and the preset rule engine, scan the data table of different correlation data, screen the data according to the preset index and high-frequency problems of the data, and generate a quality warning report.

[0047] The screening strategy of different correlation classification data is: when the high correlation data screening appears quality warning, the other node data of the high correlation data associated edge is screened again; when the medium correlation data appears quality warning, the other node data of the medium correlation data associated edge and the edge weight basic value greater than the preset value is screened; when the low correlation data appears quality warning, the other node data is not screened;

[0048] The comprehensive detection unit is used for constructing a data stream hierarchical processing architecture, and comprises:

[0049] The first layer is an AI+ data rule engine, which performs data stream preprocessing, and performs abnormal data screening according to historical verification results;

[0050] The second layer is a Spark big data analysis layer, which processes TB-level data through a Spark big data analysis engine, identifies potential quality problems, and generates a quality detection report;

[0051] The Spark big data analysis engine is used to start corresponding verification tasks for structured data and unstructured data.

[0052] For structured data, a rule script is executed through SparkSQL to verify consistency and accuracy.

[0053] For unstructured data, an NLP model is called through SparkMLlib to verify standardization and sensitive information compliance.

[0054] Optionally, in one example of the above aspect, the report set module comprises:

[0055] The quality inspection report generation unit learns normal access patterns and abnormal behavior patterns through an LSTM model, marks abnormal behaviors of different correlation data, and performs quality attribute and security attribute detection on high correlation data and other node data associated with the high correlation data if abnormal behaviors of the high correlation data occur, to generate a quality evaluation report and a rectification list.

[0056] The cataloging report unit is used to display data table cataloging progress, cataloged data quantity and excluded cataloging data list information, and present field matching conditions and cataloging difference details by comparing cataloged data with original data.

[0057] The exploration report unit is used to record quality inspection report conditions of multi-source heterogeneous data in the data access process, and presents data distribution characteristics and data quantity statistics in the form of visual charts, and quality evaluation reports and rectification lists generated by the quality inspection report generation unit.

[0058] Compared with the prior art, the present application has the following advantages:

[0059] The application breaks through the limitations of traditional single rule or AI technology by constructing an AI engine for the whole chain of "data exploration-intelligent cataloging-quality detection-report generation", realizing the automation of the closed loop from semantic analysis to cataloging generation and quality detection. The efficiency of data processing is greatly improved, and the adaptability of the system to complex data environment is enhanced.

[0060] The application realizes accurate closed loop of AI cataloging: AI+cataloging module adopts "basic matching-structure matching-semantic matching" triple strategy, and the cataloging accuracy and efficiency are greatly improved. The migration learning and knowledge graph technology are deeply integrated to accurately analyze the semantic of complex data, forming a "intelligent comparison-semantic calibration-unified cataloging" closed loop, which is unique in the existing data management system.

[0061] The application realizes automatic quality detection and report generation: through the intelligent quality detection model, the quality problems in the data can be quickly identified, and detailed reports containing problems and improvement suggestions are automatically generated. Through the automatic closed loop mechanism of "detection-analysis-report-suggestion", the efficiency and accuracy of data quality management are greatly improved, compared with the traditional manual detection and report generation mode, the labor cost and time loss are significantly reduced, and the data quality management level is promoted to a new height. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0063] Figure 1 is one of the system framework schematic diagram of the present application;

[0064] Figure 2 is the second system framework schematic diagram of the present application;

[0065] Figure 3 is the data exploration module framework schematic diagram of the present application;

[0066] Figure 4 is the AI+cataloging module framework schematic diagram of the present application. DETAILED DESCRIPTION

[0067] The technical solutions of the present application will be described below in conjunction with the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0068] Referring to Figure 1 , the first aspect of the present application provides a data quality control compliance system fused with AI technology, comprising:

[0069] Data exploration module: used for scanning and identifying multi-source heterogeneous data, extracting the field name, data type, field length and data key information of each piece of data, and automatically tracking data blood relationship;

[0070] AI+cataloging module: used for constructing a to-be-cataloged data table by using artificial intelligence technology, performing multi-dimensional comparison of the to-be-cataloged data table through basic matching and structural matching, extracting key information in the data in combination with a catalog element system, unifying various data formats, and generating a standardized data catalog;

[0071] Data classification module: clustering data tables according to business types, access frequencies and extracted data key information, establishing association in combination with the clustering results of the data and the similarity between the data, constructing a data association topology graph, analyzing the association coefficient of the data, and classifying the data according to association; setting an association edge between data nodes corresponding to data in the same cluster, and analyzing the similarity between the data to establish the association edge;

[0072] Quality detection module: used for constructing screening strategies of different association classification data by using a quality detection model and a preset rule engine, processing data streams in combination with an AI+data rule engine and a Spark big data analysis engine, starting corresponding verification tasks on structured data and unstructured data;

[0073] Report set module: used for automatically detecting compliance of data with different quality attributes and safety attributes by using AI technology, and generating a quality evaluation report and a rectification list.

[0074] Specifically, in one implementation of the present embodiment, as shown in Figure 2 , the following can also be provided:

[0075] Data catalog identification module: intelligently scanning original data sources, screening high-value governance objects and classifying them, and outputting a suggested cataloged data resource list.

[0076] General corpus module: used for gathering governance rules, models and knowledge, containing a standard library, an index library, a model library, a rule library and a label library, and providing AI-driven data support for data quality inspection, intelligent cataloging and data source detection functions;

[0077] Catalog scenario construction module: taking data as raw materials and demand as the guide, locking field pain points (such as difficult risk control and low efficiency), screening multi-source data, converting them into scenario functions, and forming a “demand→data→scenario→value” closed loop. Realize the landing and measurable conversion of data value.

[0078] According to the scene requirements, functions and values, the data is screened and adapted, so that the data is converted from "scattered resources" to "scene productivity", a "demand-data-value" closed loop is constructed, and each piece of data can accurately serve the actual application and value creation of the scene.

[0079] Meanwhile, in the embodiment, by scanning and identifying multi-source heterogeneous data through the data exploration module, the field name, data type, field length and data key information of each piece of data are extracted, and the following can be set:

[0080] The data asset identification unit uses Al intelligent filtering to screen high-value governance objects, determines the data asset qualification, generates a data asset list, and adds data key information, which is then cataloged by the AI+ cataloging module to deeply mine the value potential of the data asset.

[0081] The data intellectual property identification unit analyzes the data intellectual property registration feasibility from the aspects of clear ownership and data value, and outputs a list of data intellectual property with registration potential.

[0082] Please refer to Figure 3 In one embodiment of the present application, the data exploration module comprises:

[0083] Data source access unit: build multi-protocol adaptation interface, support structured (relational database, Excel), semi-structured (JSON, XML), unstructured (log, document) and streaming data (Kafka, MQ) access;

[0084] Data field information extraction unit: for structured data, directly read the field name, data type and field length information in the metadata, for semi-structured data, use regular matching and machine learning classification, data type Type(F)=Argmax(P(Typek|F_content,F_context)), where Typek is the candidate data type, F_content is the field content feature, F_context is the context semantic feature, and the field name and field length information are extracted; based on the entity recognition model of general corpus to extract the key information of data; P(Typek|F_content,F_context) is the conditional probability, which represents the probability that the known feature belongs to the candidate data type when the content feature (F_content) and the context feature (F_context) are known.

[0085] Blood lineage tracking unit: for ETL process, parse FROM / JOIN / INSERT INTO keywords in SQL script, extract field mapping relationship to track data lineage; for code processing process: through static analysis to identify variable assignment link, calculate the blood correlation degree Blood(X, Y) = ∑(W(Op_i) x Sim(X, Y, Op_i)), wherein, W(Op_i) is the weight of operation Op_i, Sim(X, Y, Op_i) is the feature similarity of data X, data Y and operation Op_i, through extracting data with blood correlation degree greater than threshold, tracking data lineage.

[0086] Lineage visualization uses force-directed graph to show lineage relationship, node size is positively correlated with data volume, and edge thickness is positively correlated with Blood(X, Y) value.

[0087] Please refer to Figure 4 In one embodiment of the present application, the AI+ cataloging module comprises:

[0088] Intelligent cataloging unit: for automatically analyzing and extracting field name, data type, field length and data key information by using artificial intelligence technology, and generating standardized data directory;

[0089] Intelligent completion unit: for intelligently judging the type and content of missing information by semantic analysis and association mining of cataloged data, combining standard models in general corpus, automatically identifying missing information in the directory, and intelligently inferring and completing missing entries by using AI algorithm;

[0090] Cataloging comparison unit: for comparing cataloged data with probe data in real time;

[0091] For the data in the data table, the basic attribute comparison is carried out, the cosine similarity between the field name, data type, field length and data key information of the data is respectively calculated, and the weighted average of the results is carried out to obtain the basic attribute comparison coefficient of the data in the data table;

[0092] For data table structure, graph theory method is used to compare table structure similarity:

[0093] ; wherein, E1 and E2 are field relationship edge sets (such as primary key foreign key association) of data table T1 and data table T2 respectively, the implicit relationship is identified by a deep learning model, and StructMatch(T1, T2) is the table structure similarity coefficient of data table T1 and data table T2;

[0094] Data directory generation unit: combine directory element system to extract key information in data, unify various data formats, and generate standardized data directory. For example, table 1 shows the data format corresponding to the field name:

[0095] Table 1

[0096] Field Chinese Name Field English Name Field Type Belonging Catalog List ID inventory_id Character Type Information Resource Name info_resource_name Text Type Information Resource Summary info_resource_summary Character Type Data Source Unit data_source_dept_name Character Type Organizational Code data_source_dept_code Character Type Information Resource Format Classification info_resource_format_type Character Type Information Resource Format info_resource_format Character Type Important Area Classification important_area_type Character Type Important Area Classification Self-Filling Item important_area_type_other Character Type Belonging System Name system_name Character Type Related IRS Filing Application relation_app Character Type Unrelated System Reason not_relation_reason Text Type Collection Update Frequency update_frequency Character Type Specific Update Time concrete_update_time Character Type Creation Time create_time Date and Time Type Update Time update_time Date and Time Type Deletion Flag (0-Not Deleted, 1-Deleted) del_flag Character Type Cataloging Status 1-Cataloged, 0-Not Cataloged, 2-Cataloging catalog_status Character Type Imported Data JSON Object for Comparison with AI Suggested Cataloging old_data Text Type Original Information Resource Name origin_name Character Type Cataloging Generation Time ai_time Timestamp Type AI-Generated Data first_ai_data Text Type AI Cataloging Version ai_version Digital Type Original Information Item Quantity old_item_count Digital Type Current Information Item Quantity now_item_count Digital Type

[0097] In one embodiment of the present application, the data table is clustered according to the service type, access frequency and extracted data key information, the association is established combining the clustering result of the data and the similarity between the data, the data association topology graph is constructed, including the following steps:

[0098] The access frequency of the data in the data table in a preset time period is counted, the clustering analysis is performed combining the extracted service type data and the data key information;

[0099] Each data in the data table is set as an atomic node, the K-means algorithm is used for clustering analysis, the error sum of squares of the K value in the K-means algorithm is calculated through the elbow method, the K value corresponding to the sharp drop point of the error sum of squares is selected, the clustering is performed, and K atomic node clustering clusters are obtained;

[0100] The atomic node attribute includes a unique identifier ID, a belonging clustering label and a core field feature vector;

[0101] Each data table is set as an aggregated node, a data table level node, and the attribute includes a table structure feature matrix (M_schema, row = field, column = field name / type / constraint), an access frequency sequence and a data table service label vector;

[0102] The association edge is established according to the clustering result of the atomic node and the aggregated node respectively, and the data association topology graph is constructed.

[0103] In one embodiment of the present application, the association edge is established according to the clustering result of the atomic node and the aggregated node respectively, and the data association topology graph is constructed, including the following steps:

[0104] The basic association edge is automatically established between any two atomic nodes in the same cluster, the data association topology graph is constructed, the edge weight of the basic association edge is a basic value, the edge weight basic value Ybase is set as 1 / (1+Distance(Node_i,Node_j)), wherein Distance(Node_i,Node_j) is the Euclidean distance of the feature vectors of the node Node_i and the node Node_j;

[0105] The K-means algorithm is used for clustering analysis of the aggregation nodes to obtain an aggregation node cluster, and any two aggregation nodes in the same cluster automatically establish a basic association edge, and the edge weight basic value of the aggregation node is Wbase=S(Node_x,Node_y) / (1+Smax), wherein S(Node_x,Node_y) is a table structure similarity coefficient between the node Node_x and the node Node_y, and Smax is the maximum table structure similarity coefficient between the aggregation nodes.

[0106] In this example, the K-means is used for clustering of the core field feature vectors of a single data, and the implicit association at the data content level, such as similar user behaviors and transaction modes, can be recognized, thereby breaking through the limitation of the traditional explicit association based on the field name. Based on the table structure feature matrix (field type and index relationship) and the data table business label vector, the table level structure similarity and the business semantic association can be found, such as that the "user information table" and the "risk control scoring table" both belong to the user portrait module. By constructing the topology structure of "atomic node-aggregation node-association edge", the full-link association analysis from a single data to a data table is realized.

[0107] Through the complex relationship analysis driven by the topology graph, the degree centrality of the atomic nodes (such as high-frequency access transaction records) and the aggregation nodes (such as core business tables) is calculated, and the degree centrality analysis is performed, so that the key nodes in the data network can be quickly located, thereby providing a basis for data governance. Based on the clustering results and the association edge weight, the community structure in the data network can be recognized, thereby assisting the business module division or the security isolation.

[0108] In one of the embodiments of the present application, the association coefficient of the data is analyzed, and the data is classified according to the association, including the following steps:

[0109] According to the constructed data association topology graph and the edge weight basic value of the node, the degree centrality values of the atomic nodes and the aggregation nodes are respectively calculated;

[0110] The dynamic attenuation coefficient of the edge and the node attribute weight are introduced, and the association coefficient of the data is analyzed in combination with the basic attribute comparison coefficient of the data and the table structure similarity coefficient, and the formula is as follows:

[0111]

[0112] W(Node) is the association coefficient of the data, Base(Node_i) and Base(Node_x) are respectively the degree centrality values of the atomic nodes and the aggregation nodes, Attr(Node_i) and Attr(Node_x) are respectively the attribute scores of the atomic nodes and the aggregation nodes, and De is the attenuation factor;

[0113] The attenuation factor De is e^(-ΔT / T0), wherein ΔT is the number of days since the last update of the node to the present, and T0 is an attenuation period;

[0114] According to the ranking from high to low of the correlation coefficient of data, the data correlation is classified, the ranking [0%, 20%] interval is high correlation data, the ranking is located in (20%, 60%] interval is medium correlation data, and the ranking is located in (60%, 100%] interval is low correlation data.

[0115] In the embodiment, the correlation coefficient calculation is combined with the dynamic attenuation coefficient of the edge (reflecting the influence of time or access frequency on the correlation strength), the node attribute weight, and the basic attribute comparison coefficient (such as data value similarity) and table structure similarity coefficient (such as field type matching rate), so that the data correlation strength can be dynamically quantified.

[0116] In the embodiment, the weight is set as: alpha=0.7, beta=0.3, atomic node T0=7 days, and aggregated node T0=30 days.

[0117] Through the edge weight and node attribute in the topological graph, the propagation path of abnormal data (such as abnormal data flow from a log table to a core business table) can be traced back, and problem positioning is accelerated. By introducing the time attenuation factor, the correlation weight of historical data can be automatically reduced (such as the influence of old logs on current business is weakened), the static correlation model is avoided to be out of date, dynamic adaptability and scalability are achieved, and data dynamic changes are adapted.

[0118] In one of the embodiments of the application, the atomic node attribute score of the atomic node and the aggregated node is calculated through the following formula:

[0119] The atomic node attribute score is:

[0120] Wherein, M miss is the defect data amount of the atomic node, M total is the total amount of corresponding data of the atomic node, and Ts is the ratio of the time difference from the last update of the corresponding data of the atomic node to the present time to the update period.

[0121] The aggregated node attribute score is:

[0122] Wherein, S freq is the access frequency normalized value, and V biz is the data table data integrity.

[0123] In the embodiment, the atomic node T0=7 days, and the aggregated node T0=30 days.

[0124] In one of the embodiments of the application, the quality detection module comprises:

[0125] The screening unit is used for constructing a screening strategy of different correlation classification data according to a quality detection model and a preset rule engine, scanning a data table of different correlation data, screening data according to a preset index and a high-frequency problem of the data, and generating a quality early warning report;

[0126] The screening strategy of different correlation classification data is that, when quality early warning occurs in high correlation data screening, other node data of the high correlation data associated with an edge are screened again; when quality early warning occurs in medium correlation data, other node data of the medium correlation data associated with an edge and having an edge weight basic value greater than a preset value are screened; and when quality early warning occurs in low correlation data, other node data are not screened.

[0127] The comprehensive detection unit is used for constructing a data stream hierarchical processing architecture, including:

[0128] The first layer is an AI+ data rule engine, which performs data stream preprocessing and performs abnormal data screening according to historical verification results;

[0129] The second layer is a Spark big data analysis layer, which processes TB-level data through a Spark big data analysis engine, identifies potential quality problems, and generates a quality detection report;

[0130] The Spark big data analysis engine is used to start corresponding verification tasks for structured data and unstructured data.

[0131] For structured data, a rule script is executed through SparkSQL to verify consistency and accuracy.

[0132] For unstructured data, an NLP model is called through SparkMLlib to verify standardization and sensitive information compliance.

[0133] In one embodiment of the present application, the report set module includes:

[0134] The quality inspection report generation unit learns normal access patterns and abnormal behavior patterns through an LSTM model, labels abnormal behaviors of different correlation data, and performs quality attribute and security attribute detection on high correlation data and other node data associated with the high correlation data when abnormal behaviors occur in the high correlation data, to generate a quality evaluation report and a rectification list.

[0135] The cataloging report unit is used to display data table cataloging progress, the number of cataloged data, and excluded cataloging data list information, and presents field matching conditions and cataloging difference details by comparing cataloged data with original data.

[0136] Exploration report unit: used for recording the quality inspection report of multi-source heterogeneous data in data access process, presenting the data distribution characteristics and data volume statistics in the form of visual charts, and generating the quality evaluation report and rectification list generated by the quality inspection report generation unit.

[0137] Quality attribute detection: BI tools are used to show the real-time trends of data quality indicators: integrity rate, accuracy rate and delay rate; the compliance score of integrity rate, accuracy rate and delay rate data is calculated based on a weighted scoring model, and a red-yellow-green three-color early warning is generated. The Do-Calculus causal inference algorithm is used to locate the problem source, such as data missing due to ETL script errors. Simple repairs are performed through RPA robot process automation, and complex problems are pushed to the manual processing queue to generate a rectification list.

[0138] Security attribute detection:

[0139] Sensitive information compliance: using a pre-trained sensitive word library + string matching algorithm to identify hidden sensitive information in low correlation data, and determining as abnormal if the recognition rate is < 90% or the desensitization rate is > 10%.

[0140] Permission adaptation: based on the ACL simple permission list, checking whether the access subject has the "read-only" permission (high correlation data is only open to read-only permission by default), and determining as abnormal if the unauthorized access rate is > 3%.

[0141] Based on the sensitive information compliance detection results and the permission adaptation results, a quality evaluation report is generated.

[0142] The above examples are only used to illustrate the technical method of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A data quality control compliance system integrated with AI technology, characterized in that, Comprise: Data exploration module: for scanning and identifying multi-source heterogeneous data, extracting the field name, data type, field length and data key information of each piece of data, and automatically tracking data blood relationship; wherein, the key information includes: the directory to which it belongs, the list id, the information resource name, the information resource abstract, the data source unit, the organization code, the information resource format, the classification information resource format, the key field classification, the key field classification, the self-filling item belongs to the system name, the associated IRS filing application, the unassociated system reason, the collection update frequency, the specific update time, the creation time, the update time deletion mark, the cataloging state, the imported data json object, the original information resource name, the cataloging generation time, the Al generated data, the Al catalog version, the original information item quantity, the present information item quantity; AI+cataloging module: for using artificial intelligence technology to construct the extracted data into a data table to be cataloged, comparing the data table to be cataloged in multiple dimensions through basic matching and structural matching, extracting key information in the data combined with the directory element system, unifying various data formats, and generating a standardized data directory; Data classification module: clustering the data table according to business type, access frequency and extracted data key information, establishing association and constructing data association topology graph combined with the results of data clustering and the similarity between data, analyzing the association coefficient of data, and classifying data according to association; Quality detection module: for using quality detection model and pre-set rule engine to construct screening strategies for different association classification data, combining AI+data rule engine and Spark big data analysis engine to process data flow, starting corresponding verification tasks for structured data and unstructured data; Report set module: for automatically detecting the compliance of data with different quality attributes and security attributes through AI technology, generating quality evaluation report and rectification list; Wherein, analyzing the association coefficient of data and classifying data according to association, comprising the following steps: According to the constructed data association topology graph and the edge weight basic value of the node, the degree centrality value of each atomic node and aggregation node is calculated respectively; Introducing the dynamic attenuation coefficient of the edge and the node attribute weight, combining the basic attribute comparison coefficient of the data and the table structure similarity coefficient, analyzing the association coefficient of the data, the formula is as follows: W(Node) is the association coefficient of the data, Base(Node_i) and Base(Node_x) are the degree centrality values of the atomic node and the aggregation node respectively, Attr(Node_i) and Attr(Node_x) are the atomic node attribute scores of the atomic node and the aggregation node respectively, De is the attenuation factor; Ybase is the weight basic value of the basic association edge between atomic nodes, Wbase is the weight basic value of the basic association edge between aggregation nodes; The attenuation factor De = e^(-ΔT / T0), wherein, ΔT: the number of days from the last update of the node to the present, T0: attenuation period; According to the correlation coefficient of the data, ranking from high to low, the data correlation classification, ranking [0%, 20%] interval for high correlation data, ranking in (20%, 60%] interval for medium correlation data, ranking in (60%, 100%] interval for low correlation data; The atomic node attribute score of the atomic node and the atomic node attribute score of the aggregation node are calculated by the following formula: Atomic node attribute score: Wherein, M miss is the defect data amount of the atomic node, M total is the total amount of corresponding data of the atomic node, and Ts is the ratio of the time difference from the last update of the corresponding data of the atomic node to the current time to the update period. Aggregation node attribute score: ; where S_freq is the access frequency normalized value, and V_biz is the data table data completeness.

2. The data quality control compliance system fused with AI technology according to claim 1, characterized in that, The data exploration module comprises: The data source access unit: constructs a multi-protocol adaptation interface, supports the access of structured, semi-structured, unstructured and streaming data; The data field information extraction unit: directly reads the field name, data type and field length information in the metadata for structured data, uses regular matching and machine learning classification for semi-structured data, data type Type(F)=Argmax(P(Typek|F_content,F_context)), wherein Typek is a candidate data type, F_content is a field content feature, F_context is a context semantic feature, and the field name and field length information are extracted; the entity recognition model based on a general corpus extracts the key information of the data; P(Typek|F_content,F_context) is a conditional probability, which represents the probability that the known feature belongs to the candidate data type when the content feature (F_content) and the context feature (F_context) are known. The bloodline tracking unit: for the ETL process, parse the FROM / JOIN / INSERTINTO keywords in the SQL script to extract the field mapping relationship and track the data bloodline; for the code processing process: identify the variable assignment link through static analysis, calculate the bloodline correlation degree Blood(X,Y)=∑(W(Op_i)×Sim(X,Y,Op_i)) between data X and Y, wherein W(Op_i) is the weight of operation Op_i, Sim(X,Y,Op_i) is the feature similarity of data X, data Y and operation Op_i, and the data with a bloodline correlation degree greater than a threshold value is extracted to track the data bloodline.

3. The data quality control compliance system fused with AI technology according to claim 1, characterized in that, The AI+cataloging module comprises: The intelligent cataloging unit: used for automatically analyzing and extracting the field name, data type, field length and data key information using artificial intelligence technology, and generating a standardized data catalog; The intelligent completion unit: used for intelligently judging the type and content of missing information by semantic analysis and association mining of the cataloged data, automatically identifying the missing information in the catalog, and intelligently inferring and completing the missing entries using AI algorithms; The cataloging comparison unit: used for comparing the cataloged data with the exploration data in real time; For the data in the data table, the basic attribute comparison is performed, the cosine similarity between the field name, data type, field length and data key information of the data is respectively calculated, and the weighted average of the results is obtained to obtain the basic attribute comparison coefficient of the data in the data table; For the data table structure, the graph theory method is used to compare the table structure similarity: ; wherein E1 and E2 are field relation edge sets of the data table T1 and the data table T2 respectively, the implicit relation is identified through a deep learning model, and StructMatch(T1, T2) is a table structure similarity coefficient of the data table T1 and the data table T2. Data catalog generation unit: combine the catalog element system to extract the key information in the data, unify various data formats, and generate a standardized data catalog.

4. The data quality control compliance system fused with AI technology according to claim 1, characterized in that, The data table is clustered according to the business type, access frequency and extracted data key information, and the data association topology graph is constructed by combining the data clustering results and the similarity between data, including the following steps: Statistical data table data access frequency in a preset time period, combined with the extracted business type data and data key information, clustering analysis; Each data in the data table is set as an atomic node, and the K-means algorithm is used for clustering analysis. The elbow method is used to calculate the error sum of squares of K in the K-means algorithm, and the K value corresponding to the sharp drop point is selected for clustering to obtain K atomic node clustering clusters; Atomic node attributes include: unique identifier ID, cluster label and core field feature vector; Each data table is set as an aggregated node, and the data table level node has attributes including table structure feature matrix, access frequency sequence and data table business label vector; According to the clustering results of the atomic nodes and the aggregated nodes, the association edges are established, and the data association topology graph is constructed.

5. The data quality compliance system fused with AI technology according to claim 4, characterized in that, According to the clustering results of the atomic nodes and the aggregated nodes, the association edges are established, and the data association topology graph is constructed, including the following steps: Any two atomic nodes in the same cluster automatically establish a basic association edge to construct a data association topology graph. The edge weight of the basic association edge is a basic value, and the edge weight basic value Ybase is set to 1 / (1+Distance(Node_i,Node_j)). Wherein, Distance(Node_i,Node_j) is the Euclidean distance of the feature vectors of nodes Node_i and Node_j. The K-means algorithm is used for clustering analysis of the aggregated nodes to obtain aggregated node clustering clusters. Any two aggregated nodes in the same cluster automatically establish a basic association edge, and the edge weight basic value of the aggregated node is Wbase=S(Node_x,Node_y) / (1+Smax), wherein S(Node_x,Node_y) is the table structure similarity coefficient between nodes Node_x and Node_y, and Smax is the maximum table structure similarity coefficient between the aggregated nodes.

6. The data quality and compliance system fused with AI technology according to claim 1, characterized in that, The quality detection module comprises: The screening unit is used to construct a screening strategy for different association classification data according to the quality detection model and the preset rule engine, scan the data tables of different association data, screen the data according to the preset indicators and high-frequency problems, and generate a quality warning report. The screening strategy for different association classification data is: when high association data screening occurs quality warning, screen other node data of high association data again; when medium association data occurs quality warning, screen other node data of medium association data which establishes association edge and edge weight basic value greater than preset value; when low association data occurs quality warning, do not screen other node data; The comprehensive detection unit is used to construct a data flow hierarchical processing architecture, including: The first layer is an AI+ data rule engine, which performs data stream preprocessing, screens abnormal data according to historical verification results, and performs data stream preprocessing; The second layer is a Spark big data analysis layer, which processes TB-level data through a Spark big data analysis engine, identifies potential quality problems, and generates a quality detection report; Among them, through the Spark big data analysis engine, for structured data and unstructured data, the corresponding verification task is started: For structured data, execute rule scripts through SparkSQL to verify consistency and accuracy; For unstructured data, call NLP models through SparkMLlib to verify standardization and sensitive information compliance.

7. The data quality and compliance system fused with AI technology according to claim 1, characterized in that, The report set module includes: Quality inspection report generation unit, learn normal access mode and abnormal behavior mode through LSTM model, mark abnormal behavior of different correlation data; if abnormal behavior occurs in high correlation data, quality attribute and security attribute detection are performed on high correlation data and other node data associated with it, and quality evaluation report and rectification list are generated; Cataloging report unit: used to show data table cataloging progress, cataloged data quantity and excluded cataloging data list information, compare cataloged data with original data, present field matching situation and cataloging difference details; Investigation report unit: used to record the quality inspection report of multi-source heterogeneous data in the data access process, present the data distribution characteristics and data quantity statistics in the form of visual charts, and the quality evaluation report and rectification list generated by the quality inspection report generation unit.

Images