Method for automatically constructing domain model based on metadata

Abstract

The invention relates to a method for automatically constructing a domain model based on metadata, and belongs to the technical field of data management. The method comprises the following steps: collecting and standardizing metadata of a multi-source service system; synonymous fields, master data and metadata rules are identified through semantic analysis and feature analysis; identifying a core service entity based on a rule; in a scene without physical foreign keys, intelligently deducing a relationship between entities through core field association, semantic verification and data distribution verification; and integrating the local models of the plurality of systems to form an enterprise-level unified domain model and outputting the enterprise-level unified domain model. According to the method, the problem of service association relationship breakage caused by lack of foreign keys and nonstandard metadata is effectively solved, automation and intelligence of domain model construction are realized, and the data governance efficiency and the data capitalization level are greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of data governance and information technology, and in particular, it is a method for automatically constructing domain models based on metadata. Background Technology

[0002] In the process of enterprise digital transformation, early information systems, after long-term operation and iteration, have gradually exposed many pain points in metadata management. These systems often suffer from missing information and untimely updates in their metadata documents, relying solely on database extraction of basic metadata information. Furthermore, in actual system design, for performance optimization and flexible expansion considerations, most systems have not created foreign key relationships, leading to broken metadata relationships and an inability to obtain complete business-related information.

[0003] Meanwhile, frequent staff turnover within enterprises often leads to insufficient and incomplete knowledge transfer related to metadata. New employees struggle to quickly and accurately understand the metadata design logic and business model implications of the original system. A large amount of outdated metadata remains idle, unable to be transformed into effective resources to support business decisions and system optimization, thus failing to realize its due data value.

[0004] In today's increasingly data-governed world, incomplete and non-standardized metadata has become a major obstacle to data standardization, data quality improvement, and business process optimization, severely impacting the efficiency and accuracy of cross-system data integration and data analysis applications. Therefore, systematically organizing existing metadata, uncovering its potential relationships, and constructing standardized domain models have become essential tasks for enterprises undergoing digital transformation. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a method for automatically constructing domain models based on metadata. This method can solve the industry problem of missing metadata relationships in environments without foreign keys. Through automated technology, it can intelligently mine business patterns, identify core entities, and deduce entity relationships from scattered and non-standard raw metadata. Ultimately, it can automatically construct a unified, standardized, and understandable enterprise-level domain model, thereby significantly improving data governance efficiency and activating the value of data assets.

[0006] The technical problem solved by this invention is achieved through the following technical solution: A method for automatically constructing a domain model based on metadata includes the following steps: Step 1: Collect raw metadata from one or more business systems and perform standardized preprocessing to form a standard metadata set; Step 2: Perform semantic parsing and feature analysis on the fields in the standard metadata set, identify synonymous fields, filter candidate fields in the master data, and extract feature patterns from the metadata; Step 3: Based on the extracted feature patterns, perform business clustering on the data table, identify core business entities, and classify the business. Step 4: In the absence of physical foreign key constraints, deduce the business relationships between main entities based on the association of candidate fields in the master data, field semantic consistency verification, and data distribution feature verification. Step 5: When there are multiple business systems, integrate the main entities and relationships of each system, unify the semantics and encoding, and supplement the inter-system relationships based on cross-system business processes to generate an enterprise-level unified domain model. Step 6: Output and visualize the domain model.

[0007] Furthermore, the raw data mentioned in step 1 includes the data table structure, field names, field types, field comments, length, precision, NOT NULL constraints, primary key identifiers, indexes and encrypted fields, as well as business-related information such as the data update frequency and storage volume of the data table. Standardized preprocessing includes data cleaning, format unification and missing value handling.

[0008] Furthermore, the specific implementation method for semantic parsing of fields in step 2 is as follows: using natural language processing technology to segment and abbreviate field names, and classifying and mapping fields with different names but the same semantics based on a preset synonym mapping rule library; the specific implementation method for extracting feature patterns from metadata is as follows: identifying and summarizing at least one pattern among primary key naming patterns, business attribute field identifiers, encrypted field features, and non-null constraint business meanings through rule matching or machine learning models.

[0009] Furthermore, the specific implementation method for identifying the main entity in step 3 is as follows: calculate the correlation between the data table and the candidate fields of the main data, and the coverage index in the business process, and select the data table representing the core business object as the main entity based on the index.

[0010] Furthermore, in step 4, the association based on the candidate fields of master data is as follows: using the candidate fields of master data that appear multiple times in different data tables and have consistent semantics as the association link, an association hypothesis is established between the data tables that carry the field; the data distribution feature verification is as follows: sampling the data values ​​of the inferred associated fields, and verifying the rationality of the association relationship by analyzing the overlap of their value range distribution and format consistency.

[0011] Furthermore, step 5, which involves supplementing inter-system connections based on cross-system business processes, involves establishing business logic connections between main entities that belong to different systems but participate in the same business process, according to the end-to-end business activity sequence of the enterprise.

[0012] The advantages and positive effects of this invention are: 1. Overcoming technical limitations and reducing management costs: This invention innovatively adopts a multi-dimensional field analysis and semantic mining strategy to efficiently solve the industry pain point of missing metadata relationships in environments without foreign keys, realize the automated derivation of entity relationships, eliminate the need for extensive manual intervention in metadata organization, and significantly reduce labor and time costs (tested to reduce manual organization workload by more than 80% and improve model building efficiency by 5-10 times).

[0013] 2. Activate data value and promote asset management: This invention fully activates the enterprise's outdated metadata resources through automated pattern mining and business semantic parsing, transforming scattered and disordered metadata into standardized and normalized domain models, significantly improving metadata utilization, providing solid model support for data governance, and accelerating the process of data asset management.

[0014] 3. Eliminate data silos and improve data application efficiency: The company-level domain model constructed by this invention realizes cross-system data semantic unification, breaks down model barriers and data silos between different systems, improves the accuracy (data matching accuracy rate is improved to over 95%) and efficiency of cross-system data interaction, data integration and data analysis, and provides reliable data support for enterprise business process optimization and strategic decision-making.

[0015] 4. Accumulate knowledge assets and ensure business continuity: This invention forms a standardized and systematic domain model knowledge accumulation, reducing the impact of personnel iteration on metadata understanding and business inheritance. New employees can quickly become familiar with the enterprise's business architecture and data logic through the domain model, improve the overall data management level and business collaboration efficiency of the enterprise, and ensure business continuity.

[0016] 5. High scalability and adaptability: This invention supports metadata access for multiple types of databases and multiple business systems. It can customize the regularity identification rules and business domain division standards according to the characteristics of enterprise business, adapt to the digital transformation needs of different industries (such as finance, manufacturing, and retail), and has a wide range of application scenarios and promotional value. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the accompanying drawings.

[0019] A method for automatically constructing domain models based on metadata, such as Figure 1 As shown, it includes the following steps: Step 1: Collect raw metadata from one or more business systems and perform standardized preprocessing to form a standard metadata set.

[0020] Multi-source metadata batch collection: Employing connection tools compatible with mainstream databases, and utilizing industry-standard interfaces such as JDBC and ODBC, it supports adaptation to various data sources, including relational databases (such as MySQL, Oracle, SQL Server, GoldenDB) and non-relational databases (such as MongoDB, HBase), to batch collect metadata from target system databases. The collection scope includes basic attributes such as table structure, field names, field types, field comments, length, precision, NOT NULL constraints, primary key identifiers, indexes, and encrypted fields, as well as business-related information such as data update frequency and storage capacity. After collection, the metadata is uniformly stored in a metadata collection repository, forming a centralized metadata resource pool.

[0021] Standardization preprocessing: Cleaning and standardizing the collected raw metadata. Data cleaning: Remove duplicate metadata records using deduplication algorithms, and identify and remove invalid data based on preset rules (such as table names containing "test" or "demo", fields having no actual business meaning and being completely empty). Standardize the format: Remove special characters (such as @, #, $) and irregular separators from field names and table names, and standardize the uppercase and lowercase naming format (lowercase naming convention is used by default); Missing value handling: For metadata with missing key attributes (such as unclear field comments), supplementary values ​​are made based on industry default rules and similar metadata characteristics to ensure the completeness of subsequent analysis.

[0022] Step 2: Perform semantic parsing and feature analysis on the fields in the standard metadata set, identify synonymous fields, filter candidate fields of master data, and extract feature patterns from the metadata.

[0023] Field semantic parsing and normalization: Utilize word segmentation algorithms in Natural Language Processing (NLP) (such as jieba segmentation and HanLP) to perform fine-grained word segmentation on field names and extract core business terms; combine with commonly used industry abbreviation dictionaries (such as preset mapping relationships such as "ID=Identifier", "CD=Code", "NM=Name", etc.) to establish an extensible field abbreviation mapping table, and realize the automatic matching and classification of synonymous fields with different names (such as "cust_no" and "customer_number", "ord_id" and "order_id").

[0024] Core field screening and master data candidate identification: Based on statistical analysis methods, the frequency of occurrence of each field in different data tables, the number of related data tables, the proportion of non-null data, and other indicators are calculated. Through a weighted scoring model, fields that appear frequently, run through multiple business scenarios, and have the function of identifying core business data are screened out and initially determined to be master data candidate sets (such as customer number, product code, order number, etc.).

[0025] Metadata Feature Learning and Pattern Repository Construction: Using machine learning models (such as random forests and logistic regression), feature learning is performed on entity names, primary keys, encrypted fields, and non-empty fields in metadata to extract typical features of various data types. Primary key characteristics: fixed length, prefix identifier (such as "PK_"), auto-incrementing sequence, etc. Encrypted field characteristics: specific prefix (such as "ENC_"), fixed field length with no semantic meaning, associated with encryption algorithm configuration records, etc. Characteristics of non-empty fields: required fields for business operations (such as customer name, transaction amount), strongly associated with core business processes, etc. Based on the extracted features, a rule base for recognizing various patterns is established to achieve automatic pattern matching of newly collected metadata.

[0026] Field semantic restoration optimization: For complex scenarios with "synonymous but different names" fields, NLP technology is used to deeply analyze the field name structure, removing irregular prefixes, suffixes, or separators (e.g., removing "sys_" from "sys_ord_no" and "_info" from "order_info_id"), restoring the core business meaning of the fields. Simultaneously, a dynamically updated field vocabulary and abbreviation mapping rules are built, supporting the addition of custom industry-specific terms and abbreviation mapping relationships, improving the accuracy of semantic parsing.

[0027] Step 3: Based on the extracted feature patterns, perform business clustering on the data table, identify core business entities, and classify the business.

[0028] Multi-dimensional pattern mining: Utilizing big data analytics to comprehensively uncover potential patterns within metadata, including: Entity naming conventions: such as a unified prefix ("pd_" represents product-related tables) and semantic classification (such as tables containing "user" and "customer" are classified as customer entities); Field name business attribute mapping rules: For example, fields containing "price" are mostly monetary attributes, and fields containing "date" are mostly time attributes; Primary key format and allocation rules: The primary key prefix should use the "PK + table name" combination format; Encrypted field characteristics: such as specific algorithm identifier suffix, field value encrypted storage format (such as Base64 encoding, MD5 hash value). Business constraints for non-null fields: For example, the transaction amount and transaction time fields in the core transaction table must be non-null.

[0029] Main entity identification: Based on the various patterns extracted above, the data table is classified according to business attributes. By calculating the business coreness of the data table (number of associated fields, matching degree with the candidate set of master data, and business process coverage), the main entities with core business value (such as customer entities, product entities, order entities, and transaction entities) are identified.

[0030] Entity Relationship Modeling: Based on the candidate set of master data and the results of field pattern identification, using the core fields of the main entity (such as customer ID and order ID) as the relationship hub, association analysis algorithms (including association rule mining based on field matching and similarity calculation) are used to associate other related data tables to construct a preliminary entity relationship network. On this basis, the business logic of dependencies, associations, and inclusions between entities is clarified, and relationship types such as one-to-one (e.g., customer entity and customer details entity), one-to-many (e.g., customer entity and order entity), and many-to-many (e.g., product entity and order entity) are defined. Core attributes such as association fields and association strength are defined simultaneously, ultimately forming a preliminary entity relationship model.

[0031] Main entity business classification and semantic optimization: Based on the enterprise's core business domains (such as customer domain, product domain, marketing domain, order domain, etc.), the main entities are classified, and metadata such as business description information (such as entity definition, business scope, core attribute description) and responsible person information are added to the entity. The business semantic expression of the entity relationship model is optimized to ensure that the model is understandable and reusable.

[0032] Step 4: In the absence of physical foreign key constraints, deduce the business relationships between main entities based on the association of candidate fields in the master data, field semantic consistency verification, and data distribution feature verification.

[0033] To address the common practice in most systems of employing a no-foreign-key design to avoid deadlocks, we now employ a multi-dimensional field analysis strategy to accurately deduce entity relationships, specifically including: Core field full-domain analysis: Based on indicators such as the frequency of occurrence, association frequency, and value distribution similarity of fields in different data tables, core fields (i.e. master data) that appear repeatedly and run through multiple business scenarios are selected, and implicit relationships between data tables are established with master data as the link. Business semantic consistency verification: By matching fields with the same semantics in different data tables (such as "cust_id" and "customer_id") through the field semantic normalization results, the relationship between entities is deduced in combination with the business process logic; Data distribution characteristics assist in verification: Analyze the data distribution characteristics of related fields (such as value range and format consistency) to verify the rationality of the derived entity relationship and eliminate false associations (such as associations with similar field names but no overlap in data distribution).

[0034] Step 5: When there are multiple business systems, integrate the main entities and relationships of each system, unify the semantics and encoding, and supplement the inter-system relationships based on cross-system business processes to generate an enterprise-level unified domain model.

[0035] Multi-system metadata standardization processing: Repeat steps (i) to (v) above to complete the metadata processing and system-level model construction of multiple related business systems of the enterprise (such as CRM system, resource system, order management system); establish semantic mapping dictionaries for each system model, unify the naming standards and encoding rules for cross-system entities and fields, and realize the unique identification of cross-system entities (such as assigning a globally unified code "CUST_ENTITY" to the customer entity).

[0036] Cross-system model integration: Based on the enterprise's overall business architecture, organizational structure, and core business processes, perform correlation analysis on various system-level models. Entity integration: Merge entities with the same business attributes (such as the "customer" entity in the CRM system and the "customer" entity in the marketing system) and unify core attributes and relationships; Redundancy removal: Delete duplicate or invalid associations (such as temporary table associations that are irrelevant to the core business process). Related Supplements: Based on cross-system business flow logic (such as order creation → inventory deduction → commission settlement), supplement the business relationships between systems to form a preliminary company-level domain model.

[0037] Model Review and Optimization: A review panel composed of business and technical experts will review the initial domain model from dimensions such as business accuracy, logical completeness, and scalability. Business accuracy: Verify whether the model conforms to actual business scenarios and process specifications; Logical integrity: Check for issues such as missing entities or broken relationships; Scalability: Assess whether the model can adapt to future business expansion and system iteration needs; Based on the review comments, the model structure was adjusted and business rules (such as data validation rules and access control rules) were added to ultimately form a formal company-level domain model.

[0038] Model Output and Visualization: Provides diverse model export functions, supporting export to standard format files (such as JSON, XML, SQL scripts), and is compatible with mainstream data governance platforms and modeling tools; builds a visual display interface, which intuitively presents the entity structure, relationships, and business domain division of the domain model through graphs, hierarchical structure diagrams, etc., and supports online querying, modification, and version management of the model, providing convenient support for subsequent data applications.

[0039] Step 6: Output and visualize the domain model.

[0040] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A method for automatically constructing a domain model based on metadata, characterized in that, Includes the following steps: Step 1: Collect raw metadata from one or more business systems and perform standardized preprocessing to form a standard metadata set; Step 2: Perform semantic parsing and feature analysis on the fields in the standard metadata set, identify synonymous fields, filter candidate fields in the master data, and extract feature patterns from the metadata; Step 3: Based on the extracted feature patterns, perform business clustering on the data table, identify core business entities, and classify the business. Step 4: In the absence of physical foreign key constraints, deduce the business relationships between main entities based on the association of candidate fields in the master data, field semantic consistency verification, and data distribution feature verification. Step 5: When there are multiple business systems, integrate the main entities and relationships of each system, unify the semantics and encoding, and supplement the inter-system relationships based on cross-system business processes to generate an enterprise-level unified domain model. Step 6: Output and visualize the domain model.

2. The method for automatically constructing a domain model based on metadata according to claim 1, characterized in that, The raw data mentioned in step 1 includes the data table structure, field names, field types, field comments, length, precision, NOT NULL constraints, primary key identifiers, indexes and encrypted fields, as well as business-related information such as the data update frequency and storage volume of the data table. Standardized preprocessing includes data cleaning, format unification and missing value handling.

3. The method for automatically constructing a domain model based on metadata according to claim 1, characterized in that, The specific implementation method for semantic parsing of fields in step 2 is as follows: natural language processing technology is used to segment and abbreviate field names, and based on a preset synonym mapping rule library, fields with different names but the same semantics are classified and mapped; the specific implementation method for extracting feature patterns from metadata is as follows: through rule matching or machine learning models, at least one pattern is identified and summarized regarding primary key naming patterns, business attribute field identifiers, encrypted field features, and non-null constraint business meanings.

4. The method for automatically constructing a domain model based on metadata according to claim 1, characterized in that, The specific implementation method for identifying the main entity in step 3 is as follows: calculate the correlation between the data table and the candidate fields of the main data, and the coverage index in the business process, and select the data table representing the core business object as the main entity based on the index.

5. The method for automatically constructing a domain model based on metadata according to claim 1, characterized in that, In step 4, the association based on the candidate fields of master data is as follows: using the candidate fields of master data that appear multiple times in different data tables and have consistent semantics as the association link, an association hypothesis is established between the data tables that carry the field; the data distribution feature verification is as follows: the data values ​​of the inferred associated fields are sampled, and the rationality of the association relationship is verified by analyzing the overlap of their value range distribution and format consistency.

6. The method for automatically constructing a domain model based on metadata according to claim 1, characterized in that, Step 5, which involves supplementing inter-system connections based on cross-system business processes, involves establishing business logic connections between main entities that belong to different systems but participate in the same business process, according to the end-to-end business activity sequence of the enterprise.

Images