Semantic-based AI data governance method, system and equipment and storage medium

Abstract

The invention provides a semantic-based AI data governance method, system and device and a storage medium, and relates to the field of three-dimensional modeling. The method comprises the steps of obtaining to-be-treated enterprise data containing structured and unstructured data, and performing preprocessing and standardization processing respectively; and then semantic annotation is performed on the structured data by using a knowledge graph model, and semantic understanding is performed on the unstructured data by using a semantic understanding model. And performing classification and grading on the two types of labeled data based on the data governance rule base, and performing verification and optimization through the data governance knowledge base to form a to-be-governed data list. And treating the enterprise data according to the list to obtain preliminarily treated data. And finally, detecting by using a quality detection model, generating a problem data list, repairing according to the problem data list, and finally obtaining treated enterprise data. According to the scheme, the data management effect is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional modeling, and specifically to a semantic-based AI data governance method, system, device and storage medium. Background Art

[0002] At present, enterprise data is rapidly expanding in scale and increasingly diverse in types. Data governance has become an important part of enterprise digital transformation. Enterprise data usually includes structured data and unstructured data. Structured data mainly includes table data in business systems, and unstructured data mainly includes multimedia data such as text, pictures, audio and video. Existing data governance methods usually process data in a rule-matching manner, that is, a series of data processing rules are defined in advance, and then the data is quality tested and processed based on these rules. This method can achieve good results when processing structured data, but it is often difficult to accurately understand the semantic content of the data when processing unstructured data, resulting in poor governance results. Especially when processing unstructured data with complex business meanings, due to the lack of in-depth understanding of the data semantics, it is easy to cause problems such as inaccurate data classification and grading and unclear governance requirements, which affects the overall effect of data governance. Summary of the invention

[0003] The present application provides a semantic-based AI data governance method, system, device and storage medium, which improves the effect of data governance through the above-mentioned solution.

[0004] In a first aspect, the present application provides a semantic-based AI data governance method, the method comprising: Acquire enterprise data to be governed, the enterprise data to be governed includes structured data and unstructured data, preprocess the structured data to obtain preprocessed structured data, and standardize the unstructured data to obtain standardized unstructured data; Using a preset knowledge graph model to semantically annotate the preprocessed structured data to obtain semantically annotated structured data; Using a preset semantic understanding model to perform semantic understanding on the standardized unstructured data to obtain semantically annotated unstructured data; Based on a preset data governance rule library, the semantically annotated structured data and the semantically annotated unstructured data are classified and graded to obtain a data classification result; Based on a preset data governance knowledge base, the data classification results are verified and optimized to obtain a list of data to be governed; Performing data governance on the enterprise data to be governed based on the list of data to be governed to obtain enterprise data after preliminary governance; The data after preliminary governance is quality-checked based on a preset quality-check model to obtain a list of problematic data, and the data after preliminary governance is repaired based on the list of problematic data to obtain final governed enterprise data.

[0005] By adopting the above technical solution, the structured data is preprocessed and the unstructured data is standardized, laying the foundation for subsequent semantic understanding. On this basis, the preprocessed structured data is semantically annotated using the preset knowledge graph model, and the standardized unstructured data is semantically understood using the preset semantic understanding model, realizing the semantic level analysis of different types of data. By introducing the preset data governance rule library, the semantically annotated structured data and the semantically annotated unstructured data are classified and graded, making the data governance process more targeted. Furthermore, the data classification results are verified and optimized based on the preset data governance knowledge base to ensure the rationality and feasibility of the governance plan. Finally, after completing the preliminary governance based on the list of data to be governed, the quality inspection is carried out through the preset quality inspection model and the problem data is repaired to form a complete governance closed loop. This multi-level governance method based on semantic understanding not only improves the accuracy of data classification and grading, but also enhances the controllability of the governance process and the reliability of the governance results, thereby realizing the unified and efficient governance of structured data and unstructured data.

[0006] In a second aspect of the present application, a semantic-based AI data governance method system is provided, comprising: A data acquisition module, used to acquire enterprise data to be managed, wherein the enterprise data to be managed includes structured data and unstructured data, and preprocess the structured data to obtain preprocessed structured data, and standardize the unstructured data to obtain standardized unstructured data; A structured data annotation module is used to perform semantic annotation on the preprocessed structured data using a preset knowledge graph model to obtain semantically annotated structured data; An unstructured data annotation module is used to perform semantic understanding on the standardized unstructured data using a preset semantic understanding model to obtain semantically annotated unstructured data; A classification and grading module is used to classify and grade the semantically annotated structured data and the semantically annotated unstructured data based on a preset data governance rule library to obtain a data grading result; A verification module, used to verify and optimize the data classification results based on a preset data management knowledge base to obtain a list of data to be managed; A governance module, used for performing data governance on the enterprise data to be governed based on the list of data to be governed, to obtain the enterprise data after preliminary governance; The repair module is used to perform quality inspection on the data after preliminary management based on a preset quality inspection model to obtain a list of problem data, and to repair the data after preliminary management based on the list of problem data to obtain the final managed enterprise data.

[0007] In a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above method steps.

[0008] In the fourth aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the above method.

[0009] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. This application lays the foundation for subsequent semantic understanding by preprocessing structured data and standardizing unstructured data. On this basis, the preprocessed structured data is semantically annotated using a preset knowledge graph model, and the standardized unstructured data is semantically understood using a preset semantic understanding model, thus achieving semantic level analysis of different types of data.

[0010] 2. This application introduces a preset data governance rule base to classify and grade semantically annotated structured data and semantically annotated unstructured data, making the data governance process more targeted. Furthermore, the data classification results are verified and optimized based on the preset data governance knowledge base to ensure the rationality and feasibility of the governance solution.

[0011] 3. After completing the initial governance based on the list of data to be governed, this application conducts quality inspection and repairs the problematic data through the preset quality inspection model to form a complete governance closed loop. This multi-level governance method based on semantic understanding not only improves the accuracy of data classification and grading, but also enhances the controllability of the governance process and the reliability of the governance results, thereby achieving unified and efficient governance of structured and unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a semantic-based AI data governance method provided in an embodiment of the present application; Figure 2An architecture diagram of a semantic-based AI data governance system provided for an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0013] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0014] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0015] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0016] In order to facilitate understanding of the method and system provided by the embodiments of the present application, before introducing the embodiments of the present application, the background of the embodiments of the present application is first introduced.

[0017] With the deepening of digital transformation, enterprise data has shown the characteristics of rapid expansion in scale and increasingly diverse types. Data governance has become a key link for enterprises to achieve digital transformation. In the enterprise data system, data types mainly include structured data and unstructured data. Structured data mainly comes from table data in various business systems of the enterprise, while unstructured data includes multimedia data such as text documents, pictures, audio and video. At present, the mainstream data governance methods mostly adopt the technical route of rule matching, that is, by pre-defining a series of data processing rules, and then performing quality inspection and processing on the data based on these rules. Although this method performs well when processing structured data with clear format specifications, it shows obvious limitations when facing unstructured data. Especially when processing unstructured data containing complex business semantics, due to the inability to deeply understand the semantic content of the data, it is easy to cause problems such as inaccurate data classification and grading and unclear governance requirements, which seriously affects the overall effect and quality of enterprise data governance.

[0018] After the background introduction of the above content, those skilled in the art can understand the problems existing in the prior art. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0019] Based on the above background technology, please refer to Figure 1 , Figure 1 A flowchart of a semantic-based AI data governance method provided in an embodiment of the present application. The system can be implemented by a computer program or run as an independent tool application. Specifically, in an embodiment of the present application, the method can be applied on a server, but can also be applied to electronic devices such as servers. A semantic-based AI data governance method includes the following steps: S101, obtaining enterprise data to be governed, the enterprise data to be governed including structured data and unstructured data, preprocessing the structured data to obtain preprocessed structured data, and standardizing the unstructured data to obtain standardized unstructured data; Specifically, the enterprise data to be governed is obtained from various business systems of the enterprise through the data acquisition interface, including structured data such as table data in the business system, and unstructured data such as documents, pictures, audio and video. For the acquired structured data, the data from different sources is converted into a unified format specification through data format unification processing, and then the data is missing value detected and filled by the data cleaning algorithm, and the abnormal data is identified and removed by the outlier detection algorithm, so as to obtain the preprocessed structured data. For unstructured data, text standardization is used to convert text data into a standardized format, image standardization is used to unify the pictures into a specified resolution and color space, and audio and video encoding and decoding technology is used to convert audio and video data into a standard format, and finally the standardized unstructured data is obtained. This classification preprocessing method ensures the standardization and consistency of different types of data in subsequent processing, and provides a good data foundation for subsequent semantic analysis and governance. Through the above processing, the problems of diverse enterprise data sources, different formats, and uneven quality are solved, and the accuracy and efficiency of data governance are improved.

[0020] Based on the above embodiment, as an optional embodiment, the preprocessing the structured data to obtain the preprocessed structured data includes: S201, performing data format unification processing on the structured data to obtain first structured data; Specifically, the first step in preprocessing structured data is to unify the data format. Due to the diverse sources of enterprise data, the data formats used by different systems and departments are different, so unified processing is required to ensure data consistency. In the specific implementation process, the system first identifies the field types of structured data, including numeric, character, date, etc.; then, according to the preset unified format standard, various types of data are converted: for numeric data, the decimal places and measurement units are unified; for character data, the character encoding and case format are unified; for date data, the date representation format and time zone information are unified. During the processing, the system performs format conversion operations through the data conversion component, and verifies the validity of the conversion results to ensure that the converted data meets the preset standards. Through this format unification processing, the system obtains the first structured data with standardized format, which lays the foundation for subsequent data processing and analysis, and improves the comparability and availability of data.

[0021] S202, performing missing value detection on the first structured data to obtain missing data, and filling the first structured data based on the missing data to obtain second structured data; Specifically, after obtaining the first structured data with a unified format, in order to ensure the integrity of the data, the system needs to perform missing value detection and filling processing on the data. In the missing value detection stage, the system first scans each field in the first structured data for empty values, identifies records with empty data, null data, or special characters; then determines which empty values ​​are truly missing according to business rules, and generates a missing data list containing missing locations, missing types, and missing reasons. In the data filling stage, the system selects the corresponding filling strategy according to the characteristics of the missing data: for missing time series data, linear interpolation or time series prediction methods are used for filling; for missing categorical data, mode filling or inference filling based on similar records is used; for missing numerical data, average filling or regression model prediction filling is used. During the filling process, the system will consider the business meaning and relevance of the data to ensure the rationality of the filled value. Through this missing value detection and filling processing, the system generates second structured data with complete data, improves the quality and reliability of the data, and provides better support for subsequent data analysis and application.

[0022] S203: Perform outlier detection on the second structured data to determine abnormal data, and perform removal processing on the second structured data based on the abnormal data to obtain preprocessed structured data.

[0023] Specifically, after completing the missing value processing, the system needs to identify and process the outliers in the second structured data to ensure the accuracy of the data. In the outlier detection stage, the system uses a variety of detection methods: for numerical data, statistical methods such as the 3σ principle or the box plot method are used to identify values ​​that significantly deviate from the normal distribution; for categorical data, values ​​with abnormally low frequencies are found through frequency analysis; for time series data, mutation points and abnormal trends are found through time series analysis. The system organizes these detected abnormal records into an abnormal data list, which contains the location, type and degree information of the outliers. In the removal processing stage, the system adopts corresponding processing strategies according to the characteristics of the abnormal data: for obviously erroneous data, it is directly deleted; for suspicious outliers, it is decided whether to delete them after verification through business rules; for boundary value anomalies, it is handled by truncation or reassignment. The system will retain the processing records when performing the removal operation, which is convenient for subsequent tracing and verification. Through this outlier detection and removal processing, the system finally obtains clean and reliable pre-processed structured data, which significantly improves the quality and credibility of the data and provides an accurate data basis for subsequent data analysis and decision-making.

[0024] S102, using a preset knowledge graph model to semantically annotate the preprocessed structured data to obtain semantically annotated structured data; Specifically, after completing the preprocessing of structured data, in order to accurately understand the business semantics contained in the data, the system calls the preset knowledge graph model to semantically annotate the preprocessed structured data. In the specific implementation, the preprocessed structured data is first input into the knowledge graph model, which contains a pre-built enterprise domain knowledge system, covering multi-dimensional information such as business entities and attribute relationships. The knowledge graph model identifies business entities in the data, such as customer information, product information, transaction information, etc., through an entity recognition algorithm, and extracts attribute data of these entities, such as the customer's age, address, product specifications, price, etc. After identifying the business entities and attribute data, the knowledge graph model establishes a mapping relationship between the business entities and attribute data and the data fields based on the preset semantic rules to form semantic annotation information. In this way, the system obtains structured data after semantic annotation, so that the data not only has the original structured features, but also contains clear business semantic information. This semantic annotation method based on knowledge graph improves the system's ability to understand the business meaning of structured data and provides semantic support for subsequent data classification, grading and governance.

[0025] Based on the above embodiment, as an optional embodiment, the method of using a preset knowledge graph model to semantically annotate the preprocessed structured data to obtain the semantically annotated structured data includes: S301, inputting the preprocessed structured data into a preset knowledge graph model, and identifying business entities and attribute data in the preprocessed structured data; Specifically, in order to achieve semantic management of data, the system first needs to identify the business entities and attribute information in the data. In the specific implementation process, the system inputs the pre-processed structured data into the preset knowledge graph model for processing. The preset knowledge graph model contains the ontology knowledge base in the enterprise field, which stores the definitions, attribute characteristics and relationship rules of various business entities. The system maps the data fields with the concepts in the knowledge graph through the entity matching algorithm, and identifies the entity types in the data, such as customers, products, orders, etc.; at the same time, the attribute recognition algorithm is used to extract the attribute data related to each entity, such as the customer's contact information, product specifications, order transaction information, etc. In the recognition process, the system uses natural language processing technology to perform semantic analysis on the field name and data content, and combines contextual information to improve the accuracy of recognition. Through this knowledge graph-based entity and attribute recognition, the system successfully maps the original data to the semantic level, laying the foundation for subsequent semantic association analysis, and improving the semantic expression ability and machine understandability of the data.

[0026] S302: The preprocessed structured data is processed based on the business entity and the attribute data to obtain the semantically annotated structured data.

[0027] Specifically, after completing the identification of business entities and attribute data, the system needs to semantically annotate the preprocessed structured data to establish semantic associations between data. In the semantic annotation process, the system first creates a unique identifier for the identified business entity and associates the entity with the corresponding concept node in the knowledge graph; then the attribute data of each entity is standardized and the attribute value is mapped to the predefined attribute range in the knowledge graph. The system uses RDF triples to semantically describe the data, constructing entities, attributes and attribute values ​​into a subject-predicate-object structure, and adding semantic tags in the ontology vocabulary. In the annotation process, the system also discovers implicit relationships between entities through inference rules and annotates these relationships as additional semantic information. Through this semantic annotation process, the system generates semantically annotated structured data, which gives the data clear semantic meanings and associations, realizes the semantic interoperability of the data, and provides a semantic data foundation for subsequent knowledge reasoning and intelligent applications.

[0028] S103, using a preset semantic understanding model to perform semantic understanding on the standardized unstructured data to obtain semantically annotated unstructured data; Specifically, for standardized unstructured data, in order to deeply understand the semantic content contained therein, the system uses a preset semantic understanding model for processing. The semantic understanding model is constructed using a deep learning framework and includes a text semantic understanding module, an image content understanding module, and an audio and video content understanding module. For text data, the semantic understanding model first extracts key information from the text through natural language processing technology, including subject content, sentiment, entity relationships, etc., and semantically vectorizes the text based on a pre-trained language model. For image data, the model uses computer vision algorithms to identify target objects, scene features, text information, and other content in the image, and converts the recognition results into standardized semantic descriptions. For audio and video data, the model converts audio content into text through speech recognition technology, and extracts video key frames for scene analysis, and fuses them to obtain a complete semantic expression. After completing the above processing, the semantic understanding model integrates all parsing results to form unified semantic annotation information, and finally obtains semantically annotated unstructured data. This semantic understanding method based on deep learning achieves accurate interpretation of unstructured data content, overcomes the limitation of traditional rule matching methods that are difficult to understand complex semantics, and lays the foundation for subsequent data governance work.

[0029] Based on the above embodiment, as an optional embodiment, the method of using a preset semantic understanding model to perform semantic understanding on the standardized unstructured data to obtain semantically annotated unstructured data includes: S401, inputting the standardized unstructured data into a preset semantic understanding model to extract features, thereby obtaining key information and semantic content in the standardized unstructured data; Specifically, for standardized unstructured data, it is necessary to extract the key information and semantic content contained therein through semantic understanding. During the implementation process, the system inputs the standardized unstructured data into the preset semantic understanding model, which adopts a deep learning architecture and includes multiple functional modules such as text analysis, semantic parsing and relationship extraction. The system first identifies entity words, action words and attribute words in the text through word segmentation and part-of-speech tagging; then uses the semantic parsing module to analyze the sentence structure and identify the subject-verb-object relationship and semantic role; then uses the named entity recognition technology to extract proprietary entities such as names of people, places, and institutions in the text; finally, the relationship extraction module identifies the semantic relationship between entities. In the feature extraction process, the system also uses the attention mechanism to highlight important information and capture long-distance dependencies through context analysis. Through this feature extraction process, the system successfully extracts key information and semantic content from unstructured data, including core entities, event descriptions, attribute features and relationship information, providing basic data support for subsequent semantic annotation work.

[0030] S402: semantically annotate the standardized unstructured data based on the key information and the semantic content to obtain the semantically annotated unstructured data.

[0031] Specifically, after obtaining key information and semantic content, the system needs to semantically annotate the standardized unstructured data to establish a semantic representation of the text content. In the semantic annotation process, the system first builds a semantic framework to organize the extracted key information and semantic content according to the predefined semantic pattern; then adds entity type tags to the identified entities, attribute category tags to the attribute information, and event type tags to the event description. The system uses XML markup language to hierarchically annotate the text, embedding semantic tags and relationship tags while retaining the original text structure. In the annotation process, the system also processes the referential relationship in the text through the coreference resolution technology to ensure the coherence of the semantic annotation. For the implicit semantics in the text, the system supplements the annotation through semantic reasoning rules. Through this semantic annotation processing, the system generates semantically annotated unstructured data, which makes the text content have clear semantic hierarchy and association relationships, improves the machine understandability and semantic retrieval ability of the text, and provides semantically rich text data for subsequent knowledge fusion and intelligent applications.

[0032] S104, classifying and grading the semantically annotated structured data and the semantically annotated unstructured data based on a preset data governance rule library to obtain a data grading result; Specifically, after obtaining structured data after semantic annotation and unstructured data after semantic annotation, in order to achieve accurate classification and hierarchical management of data, the system performs data classification and grading based on the preset data governance rule base. The preset data governance rule base contains data classification rule sets and data grading rule sets, where the data classification rule set defines the data type classification standards under different business scenarios, and the data grading rule set specifies the evaluation standards for data sensitivity and importance. In the specific implementation process, the system first classifies the data according to business attributes based on the semantic annotation information, such as classifying customer information, transaction records, product data, etc. into corresponding business categories. Subsequently, the system applies data grading rules to evaluate each type of data, and comprehensively considers factors such as data sensitivity, business value, and frequency of use to divide the data into different security levels and importance. Through rule matching and scoring calculation, the system generates classification and grading identification for each piece of data to form a complete data grading result. This classification and grading method based on the rule base makes full use of the results of the previous semantic annotation, realizes the unified hierarchical management of structured data and unstructured data, and provides a clear basis for the subsequent formulation of differentiated governance strategies.

[0033] Based on the above embodiment, as an optional embodiment, the semantically annotated structured data and the semantically annotated unstructured data are classified and graded based on a preset data governance rule library to obtain a data classification result, including: S501, performing content classification on the semantically annotated structured data and the semantically annotated unstructured data according to the data classification rules in the data governance rule library to obtain a data content classification result; Specifically, in order to achieve standardized management and effective use of data, the system needs to classify the content of structured data and unstructured data after semantic annotation. In the specific implementation process, the system first reads the data classification rules in the data governance rule library, which define the characteristics, scope and classification standards of different types of data. According to the data classification rules, the system conducts multi-dimensional analysis of the data: divide the business categories to which the data belongs from the business field dimension, such as financial data, personnel data, production data, etc.; distinguish the data presentation form from the data form dimension, such as text, pictures, audio and video, etc.; identify the application scenarios of the data from the data use dimension, such as decision support, business operations, customer service, etc. In the classification process, the system compares the data with the classification rules through the rule matching algorithm, and accurately judges the category attributes of the data in combination with the semantic annotation information of the data. For cross-category data, the system processes it through a multi-label classification method. Through this data classification processing, the system generates data content classification results, realizes the orderly organization and classification management of data, and provides basic support for subsequent hierarchical management.

[0034] S502, performing a hierarchical evaluation on the data content classification result according to the data classification rules in the data governance rule base to obtain a hierarchical evaluation result; Specifically, in order to ensure the security and compliance of data, the system needs to conduct a hierarchical assessment of the data content classification results. During the hierarchical assessment process, the system reads the data classification rules in the data governance rule library, which define the assessment criteria for data sensitivity, importance and security level. The system first conducts a sensitivity assessment on the data according to the data classification rules to identify whether the data contains sensitive information such as personal privacy, commercial secrets, intellectual property rights, etc.; then evaluates the importance of the data from the perspective of data value, including factors such as the business value, frequency of use, and scope of influence of the data; and then evaluates the risk level of the data from a security perspective, including data leakage risk, data abuse risk, compliance risk, etc. During the assessment process, the system uses a multidimensional scoring model to weight the scores of different assessment dimensions to form a comprehensive assessment score. For cross-category composite data, the system takes the highest security level as the final assessment result. Through this hierarchical assessment process, the system generates a hierarchical assessment result, which provides a scientific basis for hierarchical management and access control of data.

[0035] S503: classify the semantically annotated structured data and the semantically annotated unstructured data based on the classification evaluation result to obtain the data classification result.

[0036] Specifically, in order to achieve refined management and differentiated protection of data, the system needs to perform hierarchical processing on semantically annotated structured data and semantically annotated unstructured data based on the hierarchical evaluation results. During the grading process, the system divides the data into different security levels according to the hierarchical evaluation results, generally including core confidentiality, highly sensitive, generally sensitive, and public. The system first adds a security level identifier to each piece of data to mark the confidentiality level and access rights of the data; then sets the corresponding access control policy according to the data security level, including data access rights, scope of use, transmission requirements, and storage requirements; then takes differentiated security protection measures for data of different levels, such as data encryption, desensitization, access audit, etc. During the hierarchical processing, the system records the hierarchical information of the data through metadata management to ensure the consistency and traceability of the hierarchical information. In the case of changes in data levels, the system updates the grading results in a timely manner through a dynamic adjustment mechanism. Through this data grading processing, the system generates data grading results, realizes hierarchical management and precise protection of data, effectively reduces data security risks, and improves the standardization of data management.

[0037] S105, based on a preset data governance knowledge base, verifying and optimizing the data classification result to obtain a list of data to be governed; Specifically, for the data classification results, in order to ensure the accuracy and rationality of classification and grading, the system needs to verify and optimize based on the preset data governance knowledge base. The preset data governance knowledge base contains multi-dimensional knowledge such as historical governance experience, industry best practices and expert rules. During the verification process, the system first compares the data classification results with the standard cases in the data governance knowledge base, and identifies abnormal or unreasonable data items through similarity calculation. For the identified abnormal data items, the system makes classification adjustments according to the optimization rules in the knowledge base, such as improving the security level of data containing sensitive information, adjusting the importance of repeated data, etc. At the same time, the system performs classification consistency checks on data items with strong correlations to ensure that the classification results of related data are coordinated with each other. After completing the verification and optimization, the system integrates the adjusted classification results with the basic information and semantic annotation information of the data to generate a list of data to be governed containing complete governance information. This knowledge base-based verification and optimization method improves the accuracy and reliability of data classification and provides a reliable basis for the subsequent formulation of accurate governance plans.

[0038] Based on the above embodiment, as an optional embodiment, the data classification result is verified and optimized based on a preset data governance knowledge base to obtain a list of data to be governed, including: S601, based on the data quality rules in the data governance knowledge base, perform quality assessment on the data grading result to obtain a quality assessment report; Specifically, in order to ensure the accuracy of data classification results and the reliability of data quality, the system needs to conduct a quality assessment of the data classification results. During the quality assessment process, the system first reads the data quality rules in the data governance knowledge base, which contain evaluation criteria for multiple dimensions such as data integrity, consistency, accuracy, and timeliness. The system uses the data quality detection engine to comprehensively scan the data to check whether there are quality problems such as null values, duplicate values, and outliers in the data; then the format standardization of the data is verified to ensure that the data format meets the predefined standards; then the timeliness of the data is evaluated to check whether the data is updated in a timely manner and meets business needs. During the assessment process, the system uses a quality scoring model to quantify and calculate various quality indicators to obtain a data quality score. For the quality problems found, the system records the problem type, scope of impact, and severity. Through this quality assessment process, the system generates a quality assessment report containing a list of quality problems, evaluation indicators, and improvement suggestions, providing a clear direction for subsequent data optimization.

[0039] S602, based on the data standard specification in the data governance knowledge base, perform a compliance check on the data classification result to obtain a compliance check report; Specifically, in order to ensure that the data classification results comply with relevant laws, regulations and industry standards, the system needs to conduct compliance checks on the data classification results. During the compliance check process, the system first reads the data standard specifications in the data governance knowledge base, which include compliance elements such as data privacy protection requirements, industry regulatory regulations, and data security standards. The system scans the data through the compliance check module to verify whether the data classification meets the privacy protection requirements; then checks whether the data classification and classification meet the industry regulatory regulations, including compliance requirements in terms of data storage location, transmission method, access control, etc.; then evaluates whether the data processing process meets the security standards, including the implementation of security measures such as data encryption, desensitization, and audit tracking. During the inspection process, the system uses a compliance assessment matrix to compare data processing activities with compliance requirements, identify non-compliant items and assess risk levels. For compliance risks found, the system records specific violations, impact levels and rectification suggestions. Through this compliance check process, the system generates a detailed compliance check report, which provides compliance basis and improvement direction for data governance.

[0040] S603, based on the historical governance experience in the data governance knowledge base, analyzing the quality assessment report and the compliance inspection report to determine the priority and governance plan to be governed; Specifically, in order to ensure the effectiveness of data governance and the rationality of resource utilization, the system needs to conduct in-depth analysis of quality assessment reports and compliance inspection reports. During the analysis process, the system first reads the historical governance experience in the data governance knowledge base, which includes past successful governance solutions, solutions and best practices. The system uses the problem analysis engine to classify and sort the problems found in the quality assessment report, and prioritizes them according to the severity, scope of impact and difficulty of resolution; then evaluates the risk items in the compliance inspection report, and determines the processing order according to the compliance risk level and the urgency of rectification; then the system matches the current problem with historical governance cases to extract solutions and lessons learned for similar problems. During the analysis process, the system adopts a decision support model, comprehensively considers factors such as business impact, resource investment and implementation difficulty, and formulates specific governance plans for each type of problem. For problems that require collaborative processing by multiple departments, the system formulates detailed collaboration plans and time nodes. Through this analysis and processing, the system forms a list of data to be governed that includes priority ranking and specific governance solutions, providing a clear execution path for subsequent governance work.

[0041] S604: Use the quality assessment report, the compliance check report, the priority of the data to be governed, and the governance plan as the list of data to be governed.

[0042] Specifically, in order to form a complete basic document for data governance work, the system needs to integrate the quality assessment report, compliance inspection report, priority to be governed and governance plan into a list of data to be governed. During the integration process, the system first imports the data quality problems, problem severity, impact scope and other contents in the quality assessment report into the list; then enters the violations, risk levels, rectification requirements and other contents in the compliance inspection report into the list; then adds the determined priority information to be governed to the list to clarify the processing order of each problem; finally, the specific governance plan content is added to the list, including processing measures, execution steps and completion time limit. During the integration process, the system organizes this information in a structured format to ensure the orderliness and readability of the list. Through this information integration process, the system forms a complete list of data to be governed that includes quality problems, compliance risks, processing priorities and solutions, providing a clear guidance document for the subsequent execution of governance work.

[0043] S106, performing data governance on the enterprise data to be governed based on the list of data to be governed, to obtain enterprise data after preliminary governance; Specifically, after obtaining the list of data to be governed, the system conducts targeted governance on the enterprise data to be governed according to the data classification and grading information and governance requirements in the list. In specific implementation, the system first extracts the enterprise data to be governed one by one according to the data items in the list of data to be governed, and selects the corresponding governance strategy according to the data grading results. For high-level sensitive data, the system uses data desensitization technology to encrypt personal privacy information; for important business data, the system uses data verification algorithms to check integrity and consistency, and repairs existing data quality problems; for ordinary data, the system performs basic data normalization processing, including format unification, encoding conversion and other operations. During the processing process, the system uses database operations to batch process structured data, and uses special content processing components to govern unstructured data. At the same time, the system records the governance process and results of each data item to form a governance log. Through this hierarchical and classified governance method, the system finally obtains the enterprise data after preliminary governance, realizes differentiated governance of data of different types and different importance, and improves the overall quality and security of the data.

[0044] Based on the above embodiment, as an optional embodiment, the step of performing data governance on the enterprise data to be governed based on the list of data to be governed to obtain the enterprise data after preliminary governance includes: S701, performing quality management on the enterprise data to be managed according to the management requirements in the list of data to be managed, to obtain first enterprise data; Specifically, in order to improve the quality level of enterprise data, the system needs to perform quality governance on the enterprise data to be governed according to the governance requirements in the list of data to be governed. In the process of quality governance, the system first performs data cleaning tasks item by item according to the priority order in the list of data to be governed; for data integrity issues, the system fills in missing values ​​through data completion algorithms and uses business rules to verify the rationality of supplementary data; for data accuracy issues, the system uses outlier detection methods to identify and process erroneous values ​​in the data, and corrects non-standard data through data verification rules; for data consistency issues, the system uses data standardization processing to convert data from different sources into a unified format to ensure data consistency; for data timeliness issues, the system replaces expired data through a data update mechanism to ensure data timeliness. In the governance process, the system records the execution status and processing results of each governance operation to form a governance log. Through this quality governance process, the system generates the first enterprise data with significantly improved quality levels, providing a reliable data foundation for subsequent data applications.

[0045] S702, performing standardized governance on the first enterprise data according to the governance solution in the list of data to be governed, to obtain second enterprise data; Specifically, in order to achieve standardized management of enterprise data, the system needs to standardize the first enterprise data according to the governance plan in the list of data to be governed. In the process of standardized governance, the system first unifies the naming and formatting of data fields according to the data standard specifications in the governance plan; for data coding issues, the system uses standardized coding rules to convert non-standard coding to ensure the consistency and recognizability of coding; for data format issues, the system uses format conversion tools to unify data in different formats into standard formats, including date formats, numerical formats, text formats, etc.; for data measurement unit issues, the system uses unit conversion modules to convert data in different units into unified standard units; for data classification issues, the system reclassifies data according to the standard classification system to ensure the standardization of classification. In the governance process, the system verifies the standardized processing results of each data item to ensure that the processed data meets the standard requirements. Through this standardized governance process, the system generates second enterprise data that meets unified standard specifications, providing a standardized basis for data interoperability and sharing.

[0046] S703: Perform security management on the second enterprise data according to the data classification result in the list of data to be managed, to obtain the enterprise data after preliminary management.

[0047] Specifically, in order to ensure the security and compliance of enterprise data, the system needs to perform security governance on the second enterprise data according to the data classification results in the list of data to be governed. In the security governance process, the system first determines the security level and protection requirements of different data according to the data classification results; for confidential data, the system uses data desensitization technology to process sensitive information, including encryption or masking of personal information such as ID card number and mobile phone number; for high-risk data, the system sets data access rights through access control mechanism to restrict access by unauthorized users; for important business data, the system uses data backup mechanism to create data copies to ensure the safe storage of data; for shared data, the system uses data watermark technology to mark the data source and usage restrictions. In the governance process, the system records each security processing operation to form a security audit log. Through this security governance process, the system generates enterprise data after initial governance with significantly improved security, which provides a guarantee for the safe use of enterprise data.

[0048] S107, performing quality inspection on the data after preliminary management based on a preset quality inspection model to obtain a list of problematic data, and repairing the data after preliminary management based on the list of problematic data to obtain final managed enterprise data.

[0049] Specifically, in order to ensure the effectiveness of data governance, the system uses a preset quality detection model to conduct a comprehensive quality detection on the data after preliminary governance. The preset quality detection model includes multiple detection dimensions such as data integrity detection, accuracy detection, consistency detection and timeliness detection. During the detection process, the model first samples and analyzes the data, identifies outliers and missing values ​​through statistical methods; then checks the standardization of the data format and the correctness of the business logic through rule matching; then verifies the data consistency between different data sources through association analysis; and finally evaluates whether the timeliness of the data meets business requirements. The detection results are organized into a list of problem data, which records in detail the location, type and severity of each problem data. Based on the list of problem data, the system starts the data repair process and adopts corresponding repair strategies for different types of problems: for data missing problems, the system supplements them through data backfilling or estimation; for data inconsistency problems, the system determines the correct value through source data comparison; for format errors, the system performs standardized conversion; for expired data, the system updates or marks them. Through this detection and repair mechanism, the system finally obtains the final governance enterprise data with qualified quality, realizes the closed-loop management of data governance, and ensures the high quality and availability of enterprise data.

[0050] See also Figure 2 , Figure 2An architecture diagram of a semantics-based AI data governance system provided in an embodiment of the present application, the semantics-based AI data governance system may include: The data acquisition module 1 is used to acquire the enterprise data to be managed, wherein the enterprise data to be managed includes structured data and unstructured data, and preprocess the structured data to obtain preprocessed structured data, and standardize the unstructured data to obtain standardized unstructured data; The structured data annotation module 2 is used to perform semantic annotation on the preprocessed structured data using a preset knowledge graph model to obtain semantically annotated structured data; The unstructured data annotation module 3 is used to perform semantic understanding on the standardized unstructured data using a preset semantic understanding model to obtain semantically annotated unstructured data; The classification and grading module 4 is used to classify and grade the semantically annotated structured data and the semantically annotated unstructured data based on a preset data governance rule library to obtain a data grading result; Verification module 5, used to verify and optimize the data classification results based on a preset data management knowledge base to obtain a list of data to be managed; A governance module 6 is used to perform data governance on the enterprise data to be governed based on the list of data to be governed, and obtain the enterprise data after preliminary governance; The repair module 7 is used to perform quality inspection on the data after preliminary management based on a preset quality inspection model to obtain a list of problematic data, and to repair the data after preliminary management based on the list of problematic data to obtain the final managed enterprise data.

[0051] It should be noted that: when the system provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0052] Please refer to Figure 3 The application also discloses an electronic device. Figure 3 The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302 or end-to-end wireless communication.

[0053] The communication bus 302 is used to realize the connection and communication between these components.

[0054] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0055] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0056] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field~Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0057] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read~Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally be at least one storage system located away from the aforementioned processor 301. Refer to Figure 3, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a semantic-based AI data governance method.

[0058] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program storing the semantic-based AI data governance method in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application. In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0059] In the several implementations provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, such as the division of modules, which is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, indirect coupling or communication connection of systems or modules, which can be electrical or other forms.

[0060] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figure 1 The semantic-based AI data governance method of the embodiment shown in the figure can be found in the specific execution process. Figure 1 The specific description of the illustrated embodiment will not be repeated here.

[0062] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0063] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0064] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0065] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A semantic-based AI data governance method, characterized in that: The method comprises: Acquire enterprise data to be governed, the enterprise data to be governed includes structured data and unstructured data, preprocess the structured data to obtain preprocessed structured data, and standardize the unstructured data to obtain standardized unstructured data; Using a preset knowledge graph model to semantically annotate the preprocessed structured data to obtain semantically annotated structured data; Using a preset semantic understanding model to perform semantic understanding on the standardized unstructured data to obtain semantically annotated unstructured data; Based on a preset data governance rule library, the semantically annotated structured data and the semantically annotated unstructured data are classified and graded to obtain a data classification result; Based on a preset data governance knowledge base, the data classification results are verified and optimized to obtain a list of data to be governed; Performing data governance on the enterprise data to be governed based on the list of data to be governed to obtain enterprise data after preliminary governance; The data after preliminary governance is quality-checked based on a preset quality-check model to obtain a list of problematic data, and the data after preliminary governance is repaired based on the list of problematic data to obtain final governed enterprise data.

2. The method according to claim 1, characterized in that The preprocessing of the structured data to obtain the preprocessed structured data includes: Performing data format unification processing on the structured data to obtain first structured data; Performing missing value detection on the first structured data to obtain missing data, and filling the first structured data based on the missing data to obtain second structured data; An outlier detection is performed on the second structured data to determine the abnormal data, and a removal process is performed on the second structured data based on the abnormal data to obtain preprocessed structured data.

3. The method according to claim 1, characterized in that The method of using a preset knowledge graph model to semantically annotate the preprocessed structured data to obtain the semantically annotated structured data includes: Inputting the preprocessed structured data into a preset knowledge graph model to identify business entities and attribute data in the preprocessed structured data; The preprocessed structured data is processed based on the business entity and the attribute data to obtain the semantically annotated structured data.

4. The method according to claim 1, characterized in that: The using of a preset semantic understanding model to perform semantic understanding on the standardized unstructured data to obtain semantically annotated unstructured data includes: Inputting the standardized unstructured data into a preset semantic understanding model for feature extraction to obtain key information and semantic content in the standardized unstructured data; The standardized unstructured data is semantically annotated based on the key information and the semantic content to obtain the semantically annotated unstructured data.

5. The method according to claim 1, characterized in that The method of classifying and grading the semantically annotated structured data and the semantically annotated unstructured data based on the preset data governance rule library to obtain a data grading result includes: According to the data classification rules in the data governance rule library, content classification is performed on the semantically annotated structured data and the semantically annotated unstructured data to obtain a data content classification result; According to the data classification rules in the data governance rule base, the data content classification results are graded and evaluated to obtain a graded evaluation result; The semantically annotated structured data and the semantically annotated unstructured data are graded based on the graded evaluation result to obtain the data graded result.

6. The method according to claim 1, characterized in that Based on the preset data governance knowledge base, the data classification result is verified and optimized to obtain a list of data to be governed, including: Based on the data quality rules in the data governance knowledge base, perform quality assessment on the data grading results to obtain a quality assessment report; Based on the data standard specifications in the data governance knowledge base, a compliance check is performed on the data classification result to obtain a compliance check report; Based on the historical governance experience in the data governance knowledge base, the quality assessment report and the compliance inspection report are analyzed to determine the priority and governance plan for governance; The quality assessment report, the compliance check report, the priorities to be governed and the governance plan are used as the list of data to be governed.

7. The method according to claim 1, characterized in that The step of performing data governance on the enterprise data to be governed based on the list of data to be governed to obtain the enterprise data after preliminary governance includes: According to the governance requirements in the list of data to be governed, quality management is performed on the enterprise data to be governed to obtain first enterprise data; According to the governance plan in the list of data to be governed, standardize and govern the first enterprise data to obtain second enterprise data; According to the data classification result in the list of data to be managed, the second enterprise data is security managed to obtain the enterprise data after preliminary management.

8. A semantic-based AI data governance system, characterized in that: The system comprises: A data acquisition module is used to acquire enterprise data to be managed, wherein the enterprise data to be managed includes structured data and unstructured data, and preprocess the structured data to obtain preprocessed structured data, and standardize the unstructured data to obtain standardized unstructured data; A structured data annotation module is used to perform semantic annotation on the preprocessed structured data using a preset knowledge graph model to obtain semantically annotated structured data; An unstructured data annotation module is used to perform semantic understanding on the standardized unstructured data using a preset semantic understanding model to obtain semantically annotated unstructured data; A classification and grading module is used to classify and grade the semantically annotated structured data and the semantically annotated unstructured data based on a preset data governance rule library to obtain a data grading result; A verification module, used to verify and optimize the data classification results based on a preset data management knowledge base to obtain a list of data to be managed; A governance module, used for performing data governance on the enterprise data to be governed based on the list of data to be governed, to obtain the enterprise data after preliminary governance; The repair module is used to perform quality inspection on the data after preliminary management based on a preset quality inspection model to obtain a list of problem data, and to repair the data after preliminary management based on the list of problem data to obtain the final managed enterprise data.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor, a memory and a transceiver, the memory is used to store instructions, the transceiver is used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

Images