Data warehouse construction method, device, equipment and medium based on Data Vault model

Abstract

The present application relates to the technical field of data warehouse construction, and in particular to a method, device, equipment and medium for constructing a data warehouse based on a Data Vault model. The present application first obtains metadata information of the source system, extracts business subject identifiers through semantic analysis; then performs intelligent matching based on a preset template library to determine the corresponding Hub entity model; then performs in-depth analysis of the data content to identify business keys and associations; finally, forms a complete Data Vault model structure through association mapping; standardization and automation of model construction are achieved through the template library and mapping mechanism, significantly improving the quality and efficiency of data warehouse modeling.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouse construction, and in particular to a data warehouse construction method, device, equipment and medium based on a Data Vault model. Background Art

[0002] As enterprises’ digital transformation deepens, data warehouses, as the core platform for enterprise data integration and analysis, are becoming increasingly important. Faced with the rapid growth and frequent changes of multi-source heterogeneous data, traditional data warehouse modeling methods are difficult to adapt to flexible and changing business needs, and a more flexible and scalable data warehouse construction solution is urgently needed.

[0003] Currently, automated modeling tools are mainly used to build data warehouses. These tools automatically generate target data models by analyzing the data structure and business rules of the source system. Tools usually include components such as model template libraries, mapping rule engines, and code generators, which can quickly complete the conversion from the source system to the target model.

[0004] However, when modeling data warehouses based on source systems, due to the complex structure of the source systems and the lack of clear business semantic annotations, automated modeling tools can often only perform simple syntax analysis based on table structures and field names, making it difficult to accurately identify the true meaning of business entities, resulting in a large deviation between the generated data model structure and the actual business scenario. At the same time, when dealing with data associations between multiple source systems, problems such as missing or redundant relationships are prone to occur, affecting the practicality of the data warehouse, and this situation needs to be further improved. Summary of the invention

[0005] In order to solve the problem that the data model structure generated by existing data warehouse modeling has a large deviation from the actual business scenario and has low practicality, this application provides a data warehouse construction method, device, equipment and medium based on the Data Vault model, adopting the following technical solutions:

[0006] In a first aspect, the present application provides a method for constructing a data warehouse based on a Data Vault model, comprising the following steps:

[0007] Obtain metadata information and data content from the data source system;

[0008] Extracting a business subject identifier according to the metadata information, wherein the business subject identifier is used to determine the business field classification of the data;

[0009] According to the business subject identifier, matching is performed in the Data Vault model template library to determine the Hub entity model corresponding to the business subject identifier;

[0010] Extracting business keys and analyzing relationships on the data content to obtain inter-entity association information;

[0011] According to the association information, a structured Data Vault model definition is generated, and according to the Hub entity model, the related Link entities and Satellite entities are associated and mapped to obtain a complete Data Vault model structure.

[0012] By adopting the above technical solution, since there is a problem of mismatch between model structure and business semantics in the existing data modeling process, in order to improve the accuracy and efficiency of data warehouse model construction; this application first obtains the metadata information of the source system and extracts the business subject identifier through semantic analysis; then performs intelligent matching based on the preset template library to determine the corresponding Hub entity model; then conducts in-depth analysis of the data content to identify business keys and associations; finally, forms a complete Data Vault model structure through association mapping; the standardization and automation of model construction are achieved through the template library and mapping mechanism, which significantly improves the quality and efficiency of data warehouse modeling.

[0013] Optionally, extracting the business subject identifier according to the metadata information specifically includes the following steps:

[0014] Obtaining table structure definition and field description in the metadata information;

[0015] Parsing the table structure definition and field description to obtain a candidate set of business entities;

[0016] The business subject identifier is extracted from the business entity candidate set according to a preset business subject identification rule, wherein the business subject identification rule includes:

[0017] Whether there is a Hub entity feature in the business entity candidate set is determined in turn according to the preset priority order. The Hub entity feature includes a business primary key feature and a core business object feature. The business primary key feature and the core business object feature can uniquely determine the Hub entity type in the Data Vault model.

[0018] By adopting the above technical solution, in order to improve the accuracy of business subject identification extraction, since it is often difficult to effectively identify and distinguish core business entities in the traditional data model construction process, for example, when processing the enterprise human resources system, there are multiple similar table structures such as "employee information table", "personnel basic information table" and "employee file table", it is difficult to accurately determine which one is the real core business entity; this application first extracts the table structure definition and field description from the metadata, and then generates a business entity candidate set through semantic analysis, and finally determines whether the candidate entity has the Hub entity characteristics based on the preset priority rules, so as to accurately identify the real business subject; through feature recognition and priority mechanism, a more reliable basis for business subject judgment is provided, which lays a solid foundation for the subsequent Data Vault model construction.

[0019] Optionally, business key extraction and relationship analysis are performed on the data content to obtain association information between entities, which specifically includes the following steps:

[0020] Identify a business identification field and an association relationship field from the data content, wherein the business identification field is used to construct a business key of a Hub entity, and the association relationship field is used to construct an association relationship of a Link entity;

[0021] For the business identification field, analyze it according to the preset Hub entity construction rules to obtain the Hub entity definition, including the business key, loading time and data source;

[0022] For the association relationship field, analyze it according to the preset Link entity construction rules to obtain the Link entity definition, including the foreign key reference, relationship type and timestamp of the associated Hub entity;

[0023] Context attribute information and historical change information are extracted from the data content, and grouped according to attribute characteristics to obtain Satellite entity definition, including descriptive attributes, effective time, and expiration time.

[0024] By adopting the above technical solution, in order to achieve the accurate construction of entity relationships in the Data Vault model, since traditional data modeling methods are difficult to accurately identify and maintain dynamic associations between entities when dealing with complex business scenarios, for example, when dealing with sales systems, in addition to the basic subordinate relationship between customers and orders, there are also temporal change information such as order status changes and payment method adjustments. These complex associations are often not fully reflected in the model; this application first identifies the business identification field and the association relationship field, which are used to construct the business key of the Hub entity and the association relationship of the Link entity respectively; then, according to the preset construction rules, the Hub entity definition and Link entity definition containing the complete time dimension are generated respectively; finally, by extracting context attributes and historical change information, a Satellite entity definition that can track attribute evolution is constructed; by introducing the time dimension and attribute grouping mechanism, the tracking of the complete life cycle of the business entity is realized.

[0025] Optionally, after obtaining the complete Data Vault model structure, the method further includes the following steps:

[0026] Acquire source system data associated with the Hub entity model, wherein the source system data includes a historical status record of a business object;

[0027] Parsing the source system data to extract data change information, wherein the data change information includes business key changes, relationship changes, attribute changes, and timestamps;

[0028] According to the data change information, construct a Data Vault loading rule, wherein the loading rule includes a business key loading strategy of a Hub entity, a relationship loading strategy of a Link entity, and a history loading strategy of a Satellite entity;

[0029] Based on the loading rules, an ETL mapping solution is generated, and the ETL mapping solution is provided to a data engineer for data loading implementation.

[0030] By adopting the above technical solution, this application first obtains the historical status records in the source system, and then extracts multi-dimensional change information such as business key changes, relationship changes, attribute changes, etc. through analysis, and then builds differentiated loading rules for Hub, Link and Satellite entities based on these change information, and finally generates an implementable ETL mapping solution; by establishing a complete data change tracking mechanism, accurate recording of the entire life cycle of business objects is achieved.

[0031] Optionally, after generating an ETL mapping solution based on the loading rule, the method further includes the following steps:

[0032] Perform load analysis on the Hub entity, Link entity and Satellite entity, and divide the entities into different loading modes according to entity characteristic parameters, wherein the entity characteristic parameters include business key distribution characteristics, association complexity and historical change frequency;

[0033] According to different loading modes, the corresponding data loading indicators are counted respectively to obtain the loading performance data of each entity type;

[0034] Perform multi-dimensional evaluation on the loading performance data of each entity type to obtain the entity loading score;

[0035] Calculate the ratio of the loading score of each entity type to the overall score, and use the ratio as the model optimization weight of the corresponding entity type;

[0036] Analyze each entity type according to the model optimization weights to identify key entity patterns that need to be optimized;

[0037] According to the characteristic analysis results of the key entity pattern, an optimization scheme of the Data Vault model is generated, wherein the optimization scheme includes a business key index optimization strategy of the Hub entity, a partition loading strategy of the Link entity, and a historical data storage strategy of the Satellite entity.

[0038] By adopting the above technical solution, the present application first performs load analysis and classification on different types of entities, then scores them by counting the loading performance indicators of various entities, and then determines the optimization weight based on the score ratio, thereby identifying the key entity patterns that need to be optimized, and finally generates a complete optimization plan based on the feature analysis results, including strategies such as index optimization, partition loading, and historical data storage; thereby improving the performance of the model.

[0039] Optionally, before analyzing each entity type according to the model optimization weight, the method further includes the following steps:

[0040] The model optimization weight of each entity type is double-weighted by the amount of historical data and the frequency of change to obtain the adjusted optimization weight;

[0041] The calculation formula for the adjusted optimization weight is: AWi,t=Wi,t×(1+α×log(Vi,t / V0))×(1+β×(Ci,t / C0)), wherein AWi,t represents the adjusted optimization weight, Wi,t represents the model optimization weight of the i-th entity type in the t-th time period, Vi,t represents the historical data volume of the i-th entity type in the t-th time period, V0 represents the baseline historical data volume, Ci,t represents the change frequency of the i-th entity type in the t-th time period, C0 represents the baseline change frequency, α represents the data volume adjustment factor and 0<α<1, β represents the change frequency adjustment factor and 0<β<1.

[0042] By adopting the above technical solution, the traditional weight calculation method often only considers the evaluation indicators of a single dimension, ignoring the interactive impact between data volume growth and change frequency. For example, in a banking transaction system, although the initial weight of the account entity is high, its historical data volume grows slowly and the change frequency is low, while the transaction flow entity has a lower initial weight, but due to its extremely high data growth rate and change frequency, the actual impact on system performance may be greater; the present application performs a compound operation on the basic weight with the logarithmic growth of the historical data volume and the linear growth of the change frequency, where α and β are respectively used as adjustable data volume and change frequency influencing factors, thereby realizing dynamic and accurate calculation of the weight; by introducing a nonlinear weighting mechanism, the actual impact of entity characteristics on system performance is more accurately reflected.

[0043] In a second aspect, the present application provides a data warehouse construction device based on a Data Vault model, comprising:

[0044] Data acquisition module, used to obtain metadata information and data content of the data source system;

[0045] A business subject identifier extraction module, used to extract a business subject identifier according to the metadata information, wherein the business subject identifier is used to determine the business field classification of the data;

[0046] A Hub entity model determination module, used to match the business subject identifier in the Data Vault model template library to determine the Hub entity model corresponding to the business subject identifier;

[0047] An entity association information acquisition module is used to extract business keys and analyze relationships of the data content to obtain entity association information;

[0048] The Data Vault model acquisition module is used to generate a structured Data Vault model definition according to the association information, and to associate and map related Link entities and Satellite entities according to the Hub entity model to obtain a complete Data Vault model structure.

[0049] Optionally, the business subject identifier extraction module includes:

[0050] A table structure acquisition unit, used to acquire the table structure definition and field description in the metadata information;

[0051] A business entity parsing unit, used to parse the table structure definition and field description to obtain a business entity candidate set;

[0052] The subject identification rule unit is used to extract the business subject identifier from the business entity candidate set according to a preset business subject identification rule, wherein the business subject identification rule includes:

[0053] The Hub feature judgment unit is used to judge whether there are Hub entity features in the business entity candidate set in accordance with a preset priority order. The Hub entity features include business primary key features and core business object features. The business primary key features and the core business object features can uniquely determine the Hub entity type in the Data Vault model.

[0054] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for constructing a data warehouse based on the DataVault model when executing the computer program.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned method for constructing a data warehouse based on the Data Vault model are implemented.

[0056] In summary, the present application includes at least one of the following beneficial technical effects:

[0057] Since there is a problem of mismatch between model structure and business semantics in the existing data modeling process, in order to improve the accuracy and efficiency of data warehouse model construction; this application first obtains the metadata information of the source system, extracts the business subject identifier through semantic analysis; then performs intelligent matching based on the preset template library to determine the corresponding Hub entity model; then conducts in-depth analysis of the data content to identify business keys and associations; finally, forms a complete DataVault model structure through association mapping; standardization and automation of model construction are achieved through the template library and mapping mechanism, which significantly improves the quality and efficiency of data warehouse modeling;

[0058] In order to improve the accuracy of business subject identification extraction, it is often difficult to effectively identify and distinguish core business entities in the traditional data model construction process. For example, when processing the enterprise human resources system, there are multiple similar table structures such as "employee information table", "personnel basic information table" and "employee archive table", making it difficult to accurately determine which is the real core business entity; this application first extracts the table structure definition and field description from the metadata, then generates a business entity candidate set through semantic analysis, and finally determines whether the candidate entity has the Hub entity characteristics based on the preset priority rules, so as to accurately identify the real business subject; through feature identification and priority mechanism, a more reliable basis for business subject judgment is provided, laying a solid foundation for the subsequent Data Vault model construction;

[0059] This application first identifies the business identification field and the association relationship field, which are used to construct the business key of the Hub entity and the association relationship of the Link entity respectively; then, according to the preset construction rules, the Hub entity definition and Link entity definition containing the complete time dimension are generated respectively; finally, by extracting context attributes and historical change information, a Satellite entity definition capable of tracking attribute evolution is constructed; by introducing the time dimension and attribute grouping mechanism, the tracking of the complete life cycle of the business entity is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of a method for constructing a data warehouse based on a Data Vault model according to an embodiment of the present application;

[0061] Figure 2 This is an example diagram of the Data Vault model structure in the data warehouse construction method based on the Data Vault model in the embodiment of the present application;

[0062] Figure 3 It is a flowchart of step S200 in the method for building a data warehouse based on the Data Vault model in an embodiment of the present application;

[0063] Figure 4It is a flowchart of step S400 in the method for building a data warehouse based on the Data Vault model in an embodiment of the present application;

[0064] Figure 5 It is a flowchart of generating an ETL mapping solution in a data warehouse construction method based on a Data Vault model in an embodiment of the present application;

[0065] Figure 6 It is a flowchart of generating a model optimization solution in a data warehouse construction method based on a Data Vault model in an embodiment of the present application;

[0066] Figure 7 It is a weight adjustment change trend diagram in the data warehouse construction method based on the Data Vault model in the embodiment of the present application;

[0067] Figure 8 It is a module schematic diagram of a data warehouse construction device based on a Data Vault model in an embodiment of the present application;

[0068] Fig. 9 It is a diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0070] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0071] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0072] In the first aspect, the present application provides a method for constructing a data warehouse based on the Data Vault model, referring to Figure 1 , including the following steps:

[0073] S100: Obtain metadata information and data content of a data source system.

[0074] Among them, the data source system usually includes multiple business systems, such as sales system, customer relationship management system, human resources system, etc. Each business system contains a large amount of structured and semi-structured data information, which needs to be integrated into a unified data warehouse. Metadata information mainly includes database object description information such as table structure definition, field attributes, primary key constraints, foreign key relationships, etc., while data content includes the actual stored business record data.

[0075] Specifically, the system accesses the data dictionary of the source system through the database connection adapter to extract the table structure definition and relationship constraint information. For example, in an enterprise ERP system, by querying information_schema, the table structure definition related to "sales order" is obtained, including the order master table, order detail table, order status table, etc.

[0076] S200: extracting a business subject identifier according to the metadata information, wherein the business subject identifier is used to determine a business field classification of the data.

[0077] Among them, the business subject identifier reflects the core entity type and domain attributes of the business data. By analyzing the naming rules, field characteristics and data relationships in the metadata information, the core business subjects in the system are identified. For example, in the retail industry, common business subjects include "customers", "goods", "orders", "stores", etc. These subject identifiers directly determine the overall architecture of the subsequent data warehouse model.

[0078] Specifically, heuristic rules are used to extract business topics: first, analyze the table name and primary key naming pattern to extract keywords with business meaning; then check the field description and annotation information to summarize the scope of business topics; finally, combine the association between database tables to determine the hierarchical relationship between primary and secondary business topics. For example, when analyzing the sales system, the "order" topic can be extracted through the table name "t_sales_order", the "customer" topic can be identified through the primary key "customer_id", and the subordinate relationship between orders and customers can be determined through foreign key relationships.

[0079] S300: According to the business subject identifier, matching is performed in the Data Vault model template library to determine the Hub entity model corresponding to the business subject identifier.

[0080] The Data Vault model template library stores common standard entity model definitions in various industries. Each template contains recommended business key structures, typical association patterns, and attribute grouping schemes. By matching business subject identifiers in the template library, the corresponding Hub entity model can be quickly determined to avoid repeated design work.

[0081] Specifically, for the identified business theme identifier, first perform an exact match search in the template library. If a completely matching template is found, the template definition is directly adopted; if an exact match cannot be found, the text similarity with the existing template is calculated, and the template with the highest similarity that exceeds the preset threshold is selected as the base template. For example, for the "membership" theme, the "customer" template may be matched. The system will use the Hub entity structure of the template as the basis and make appropriate adjustments based on actual business needs.

[0082] S400: extract business keys and perform relationship analysis on the data content to obtain inter-entity association information.

[0083] Among them, the business key is the attribute combination used to uniquely identify the business entity, and the association relationship between entities reflects the logical connection between business objects. By analyzing the field characteristics and value distribution in the data content, the attribute combination suitable as the business key can be identified. At the same time, by analyzing the foreign key constraints and associated fields between tables, the relationship network between entities can be sorted out.

[0084] Specifically, first analyze the field characteristics of the data content and identify field combinations with uniqueness, stability and business meaning as business key candidates. For example, when analyzing customer data, you may find that the combination of "document type + document number" has good uniqueness and stability, and is suitable as the business key of the customer entity. Then, by analyzing the foreign key constraints and associated query statements between tables, identify the association relationship between entities. For example, through the "customer_id" field in the order table, it can be determined that there is an association relationship between the order entity and the customer entity.

[0085] S500: Generate a structured Data Vault model definition according to the association information, and associate and map related Link entities and Satellite entities according to the Hub entity model to obtain a complete Data Vault model structure.

[0086] The structured Data Vault model definition includes three basic components: Hub entity, Link entity and Satellite entity. Hub entity is used to store the core identification information of business objects, Link entity is used to record the relationship between business objects, and Satellite entity is used to store the descriptive attributes and historical change records of business objects.

[0087] Specifically, first generate a Data Vault model definition document in a standard format based on the extracted business keys and associated information, including the structural specifications, naming rules, and loading strategies for each entity. Then, based on the determined Hub entity model, map and associate the relevant Link entities and Satellite entities. For example, for the customer topic, in addition to creating a Customer Hub entity to store customer identification information, it is also necessary to create a Customer Satellite entity to store basic customer attributes, and create corresponding Link entities to associate the relationship between customers and other business entities (such as orders, accounts, etc.). Ultimately, a complete and traceable data warehouse model structure is formed.

[0088] like Figure 2 As shown in the Data Vault model structure example diagram, the data warehouse constructed by this method includes three types of Hub entities, Link entities and Satellite entities. Among them, the Hub entity is used to store the identification information of the core business objects, such as the Customer Hub stores the customer hash key, customer business key, loading time and data source; the Link entity is used to record the relationship between business objects, such as the Order Customer Link stores the correspondence between the order hash key and the customer hash key; the Satellite entity is used to store descriptive attributes, such as the Customer Satellite stores the customer name, contact address, contact phone number and other attribute information. The association relationship between the entities is established through the hash key to ensure the consistency and integrity of the data.

[0089] For the sales order scenario, the Hub entity includes Customer Hub, Order Hub, and Product Hub, which store the corresponding business object identifiers respectively; the Order Customer Link entity records the correspondence between orders and customers; Customer Satellite and Order Satellite store customer information and order details respectively. This model structure clearly shows the relationship between business objects and can effectively manage historical changes.

[0090] In one embodiment, referring to Figure 3 In step S200, extracting the business subject identifier according to the metadata information specifically includes the following steps:

[0091] S210: Obtain table structure definition and field description in the metadata information.

[0092] The table structure definition includes basic structural information such as the name of the data table, field list, primary key settings and index information, and the field description includes attribute definitions such as the field name, data type, length limit, and whether it is allowed to be empty.

[0093] Specifically, the table structure definition in the source system is obtained through database connection. For example, the structure definition of the customer table (T_CUSTOMER) is obtained from a sales system, including detailed descriptions of fields such as customer number, customer name, and contact information. At the same time, the field description information in the business dictionary is extracted to form a complete metadata information set.

[0094] S220: Parse the table structure definition and field description to obtain a candidate set of business entities.

[0095] Among them, by parsing the table name, field name and description text in the metadata information, entity concepts with business meanings are identified, and these entity concepts are summarized to form a business entity candidate set.

[0096] S230: extracting the business subject identifier from the business entity candidate set according to a preset business subject identification rule.

[0097] Among them, the business subject identification rules include: judging whether there are Hub entity features in the business entity candidate set in a preset priority order, and the Hub entity features include business primary key features and core business object features. The business primary key features and the core business object features can uniquely determine the Hub entity type in the Data Vault model.

[0098] Specifically, the business topic identification rules define the standards and methods for identifying core business topics from candidate entities. The rules include analyzing the core nouns in the table names, identifying the business meaning of the primary key fields, and counting the referenced relationships. For example, tables containing keywords such as "master" and "base" often represent master data and have a higher priority for topic identification; tables referenced by multiple other tables are usually also important business topics.

[0099] In one embodiment, referring to Figure 4 In step S400, business key extraction and relationship analysis are performed on the data content to obtain the association information between entities, which specifically includes the following steps:

[0100] S410: Identify a business identification field and an association relationship field from the data content, wherein the business identification field is used to construct a business key of a Hub entity, and the association relationship field is used to construct an association relationship of a Link entity.

[0101] In this embodiment, a field feature identification table is established to assist the identification process. The table includes two types: business identification features and association relationship features.

[0102] Specifically, first traverse all field definitions in the data table, and calculate the feature score of each field according to the field feature identification table. When the business identification feature score of the field exceeds the preset threshold, it is determined as a business identification field; when the association feature score of the field exceeds the preset threshold, it is determined as an association field. For example, check whether the field name contains a feature suffix, whether it is defined as a primary key or foreign key constraint, whether it has a uniqueness indicator, etc., and determine the field type after comprehensive evaluation.

[0103] S420: Analyze the business identification field according to the preset Hub entity construction rules to obtain the Hub entity definition, including the business key, loading time and data source.

[0104] In this embodiment, a Hub entity rule mapping table is pre-established to define the Hub entity construction rules in different scenarios. The mapping table contains configuration items such as field combination mode, naming conventions, data type conversion rules, etc. At the same time, a business key generation strategy is set, including three types: single-field business key, combined field business key, and derived business key, and corresponding processing logic is configured for each type.

[0105] Specifically, first, according to the characteristics of the business identification field, query the Hub entity rule mapping table to obtain the applicable construction rules. Then extract the business key according to the rules. For single fields, directly use the original value. For combined fields, use the predefined connector splicing. For cases that require conversion, execute the corresponding conversion logic. Finally, through a unified entity definition template, generate the Hub entity structure definition including business keys, loading time and data source.

[0106] S430: For the association relationship field, analyze it according to the preset Link entity construction rule to obtain the Link entity definition, including the foreign key reference, relationship type and timestamp of the associated Hub entity.

[0107] In this embodiment, a Link entity pattern library is established to store common association relationship patterns and corresponding Link entity construction rules. The pattern library defines basic relationship types such as one-to-one, one-to-many, and many-to-many, as well as standard construction templates for each type. At the same time, relationship validity verification rules are configured to ensure that the generated Link entity meets data integrity requirements.

[0108] S440: extract context attribute information and historical change information from the data content, and group them according to attribute characteristics to obtain Satellite entity definition, including descriptive attributes, effective time, and expiration time.

[0109] In this embodiment, an attribute feature analysis matrix is ​​constructed to evaluate and classify each attribute field. The matrix contains evaluation criteria for dimensions such as change frequency, business importance, and data type, and defines grouping strategies under different combinations. At the same time, a historical tracking rule table is established to configure historical record retention requirements for different types of attributes.

[0110] In one embodiment, referring to Figure 5 , step S500, after obtaining the complete Data Vault model structure, the method further includes the following steps:

[0111] S600: Acquire source system data associated with the Hub entity model, wherein the source system data includes historical status records of business objects.

[0112] In this embodiment, a source system data acquisition configuration table is established, which defines the connection parameters, data extraction rules and historical status identification methods of different source systems. The configuration table contains information such as data source type (such as relational database, log file, etc.), access credentials, historical status identification field, etc., and configures the corresponding data extraction strategy according to the characteristics of the source system.

[0113] Specifically, first query the configuration table according to the Hub entity identifier to obtain the corresponding data source configuration, establish a data connection and perform data extraction. For different types of source systems, use corresponding extraction methods, such as using incremental timestamps for relational databases and file scanning for log files to ensure that the source system data including historical status is fully obtained.

[0114] S700: Parse the source system data and extract data change information, where the data change information includes business key change, relationship change, attribute change and timestamp.

[0115] In this embodiment, a change type identification matrix is ​​constructed to systematically identify different types of data changes. The matrix defines the change feature pattern, including field value comparison rules, change type determination conditions, and timestamp extraction methods. At the same time, a change priority rule table is established to handle multiple changes.

[0116] Specifically, the source system data is scanned using the change type identification matrix, and the change type is identified by comparing the field values ​​of adjacent records. When a field value difference is detected, it is determined whether it is a business key change, a relationship change, or an attribute change based on the change feature pattern, and the corresponding timestamp information is extracted.

[0117] S800: Constructing Data Vault loading rules according to the data change information, wherein the loading rules include a business key loading strategy of a Hub entity, a relationship loading strategy of a Link entity, and a history loading strategy of a Satellite entity.

[0118] In this embodiment, a loading strategy template library is pre-established to store standard loading strategies in different scenarios. The template library includes business key processing templates, relationship loading templates, and history processing templates. Each template defines a specific data processing flow and quality control requirements. At the same time, a strategy selection rule is configured to select an appropriate loading strategy based on the change characteristics.

[0119] S900 . Generate an ETL mapping solution based on the loading rules, and provide the ETL mapping solution to a data engineer for data loading implementation.

[0120] In this embodiment, an ETL mode conversion rule library is established to define a standard method for converting loading rules into specific ETL tasks. The rule library contains field mapping templates, conversion function libraries, and data flow templates, and configures adaptation rules for different ETL tools to ensure that the generated mapping solution can be executed in the target environment.

[0121] Specifically, according to the determined loading rules, query the ETL mode conversion rule library to obtain the corresponding conversion template. Convert the processing logic in the loading rules into a standard ETL task definition, including the specific steps of data extraction, conversion and loading. The generated ETL mapping solution contains detailed field mapping relationships, conversion rules and execution order, which is convenient for data engineers to implement.

[0122] In one embodiment, referring to Figure 6 , step S900, after generating an ETL mapping solution based on the loading rule, the method further comprises the following steps:

[0123] S910: Perform load analysis on the Hub entity, Link entity and Satellite entity, and divide the entities into different loading modes according to entity characteristic parameters.

[0124] The entity characteristic parameters include business key distribution characteristics, association complexity and historical change frequency.

[0125] In this embodiment, an entity feature evaluation framework is established, which includes a feature calculation rule table and a loading mode classification matrix. The feature calculation rule table defines the specific calculation methods of business key distribution characteristics (such as discreteness, repetition rate), association complexity (such as association depth, number of references) and historical change frequency (such as average daily change volume, peak change rate). The loading mode classification matrix determines the applicable loading mode according to different combinations of feature parameters.

[0126] Specifically, we first perform feature parameter calculations on each entity to obtain quantitative indicators of its business key distribution, association complexity, and change frequency. Then, we bring these indicators into the classification matrix and classify the entities into different modes such as real-time loading, batch loading, and incremental loading according to preset thresholds and rules.

[0127] S920. For different loading modes, corresponding data loading indicators are counted respectively to obtain loading performance data of each entity type.

[0128] In this embodiment, a loading performance index library is constructed, which includes dimensions such as throughput index, resource consumption index and quality index, and sets corresponding benchmark values ​​for different loading modes.

[0129] S930. Perform multi-dimensional evaluation on the loading performance data of each entity type to obtain an entity loading score.

[0130] In this embodiment, a performance scoring model library is established to store scoring standards and weight configurations in different application scenarios. The scoring model includes a performance dimension weight table, indicator normalization rules and scoring calculation formulas, and sets weight coefficients of different dimensions according to business importance.

[0131] Specifically, the collected performance data is processed according to the scoring model, first normalizing each indicator, and then calculating the weighted score according to the weight configuration. Finally, a standardized score reflecting the overall performance level of entity loading is obtained.

[0132] S940. Calculate the ratio of the loading score of each entity type to the overall score, and use the ratio as the model optimization weight of the corresponding entity type.

[0133] In this embodiment, a weight calculation rule set is designed to define a calculation method for the scoring ratio and a normalization processing rule.

[0134] Specifically, based on the score data of each entity type, its percentage of the overall score is calculated, and smoothing is applied to avoid extreme values. Finally, a standardized weight value reflecting the relative importance of each entity type is obtained.

[0135] S950: Analyze each entity type according to the model optimization weight, and identify key entity patterns that need to be optimized.

[0136] In this embodiment, an optimization target identification matrix is ​​constructed, the matrix defines weight-based priority judgment rules, performance bottleneck identification standards and optimization benefit evaluation methods, and sets corresponding identification thresholds.

[0137] Specifically, the optimization weight of the entity is compared with the preset threshold to identify the entity type with a higher weight. Then the performance characteristics and constraints of these entities are analyzed to determine the specific optimization goals and form a list of key entity patterns to be optimized.

[0138] S960: Generate an optimization solution for the Data Vault model according to the feature analysis results of the key entity pattern.

[0139] The optimization scheme includes the business key index optimization strategy of the Hub entity, the partition loading strategy of the Link entity and the historical data storage strategy of the Satellite entity.

[0140] In one embodiment, before analyzing each entity type according to the model optimization weight, the method also includes doubly weighting the model optimization weight of each entity type by the amount of historical data and the frequency of change to obtain an adjusted optimization weight; the calculation formula for the adjusted optimization weight is: AWi,t=Wi,t×(1+α×log(Vi,t / V0))×(1+β×(Ci,t / C0)), wherein Wi,t represents the model optimization weight of the i-th entity type in the t-th time period, Vi,t represents the amount of historical data of the i-th entity type in the t-th time period, V0 represents the baseline historical data amount, Ci,t represents the change frequency of the i-th entity type in the t-th time period, C0 represents the baseline change frequency, α represents the data amount adjustment factor and 0<α<1, β represents the change frequency adjustment factor and 0<β<1.

[0141] In this embodiment, a dynamic weight adjustment framework is established, which includes a benchmark parameter configuration table and an adjustment factor management mechanism. The benchmark parameter configuration table stores the benchmark historical data volume V0 and benchmark change frequency C0 of each entity type. These benchmark values ​​are determined by statistical analysis of historical operation data. The adjustment factor management mechanism is responsible for dynamically adjusting the α and β values ​​according to the characteristics of the business scenario to ensure the rationality of the weight calculation.

[0142] Specifically, first obtain the actual historical data volume Vi,t and change frequency Ci,t of each entity type from the performance monitoring system. Then, combine the pre-configured benchmark parameters V0 and C0, and the currently set adjustment factors α and β, and substitute them into the formula for calculation. Among them, the historical data volume is smoothed in logarithmic form to avoid weight imbalance caused by excessive data volume differences; the change frequency is linearly adjusted to reflect real-time requirements.

[0143] Reference Figure 7, when the amount of historical data (Vi,t / V0) increases, the weight growth curve shows obvious nonlinear characteristics due to the logarithmic smoothing process. The growth rate gradually slows down with the increase of data volume, effectively avoiding the weight imbalance problem caused by excessive data volume differences; at the same time, the three curves corresponding to different change frequencies (Ci,t / C0) show a hierarchical distribution in equal proportion, reflecting the influence of change frequency on weight through linear adjustment. For example, when the data volume ratio is 5, the weight in the high change frequency (4×C0) scenario is about twice that in the low change frequency (1×C0) scenario. This hierarchical feature ensures that entities with higher real-time requirements can obtain correspondingly higher optimization priorities.

[0144] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0145] In a second aspect, the present application provides a data warehouse construction device based on a Data Vault model. The data warehouse construction device based on a Data Vault model of the present application is described below in conjunction with the above-mentioned data warehouse construction method based on a Data Vault model.

[0146] Reference Figure 8 , a data warehouse construction device based on the Data Vault model, comprising:

[0147] Data acquisition module, used to obtain metadata information and data content of the data source system;

[0148] A business subject identifier extraction module, used to extract a business subject identifier according to the metadata information, wherein the business subject identifier is used to determine the business field classification of the data;

[0149] A Hub entity model determination module, used to match the business subject identifier in the Data Vault model template library to determine the Hub entity model corresponding to the business subject identifier;

[0150] An entity association information acquisition module is used to extract business keys and analyze relationships of the data content to obtain entity association information;

[0151] The Data Vault model acquisition module is used to generate a structured Data Vault model definition according to the association information, and to associate and map related Link entities and Satellite entities according to the Hub entity model to obtain a complete Data Vault model structure.

[0152] In one embodiment, the business subject identifier extraction module includes:

[0153] A table structure acquisition unit, used to acquire the table structure definition and field description in the metadata information;

[0154] A business entity parsing unit, used to parse the table structure definition and field description to obtain a business entity candidate set;

[0155] The subject identification rule unit is used to extract the business subject identifier from the business entity candidate set according to a preset business subject identification rule, wherein the business subject identification rule includes:

[0156] The Hub feature judgment unit is used to judge whether there are Hub entity features in the business entity candidate set in accordance with a preset priority order. The Hub entity features include business primary key features and core business object features. The business primary key features and the core business object features can uniquely determine the Hub entity type in the Data Vault model.

[0157] In one embodiment, the entity association information acquisition module includes:

[0158] A field identification unit, used to identify a business identification field and an association relationship field from the data content, wherein the business identification field is used to construct a business key of a Hub entity, and the association relationship field is used to construct an association relationship of a Link entity;

[0159] A Hub entity analysis unit, used to analyze the business identification field according to a preset Hub entity construction rule to obtain a Hub entity definition, including a business key, a loading time, and a data source;

[0160] A Link entity analysis unit, used to analyze the association relationship field according to a preset Link entity construction rule to obtain a Link entity definition, including a foreign key reference, a relationship type, and a timestamp of an associated Hub entity;

[0161] The Satellite entity analysis unit is used to extract context attribute information and historical change information from the data content, and group them according to attribute characteristics to obtain Satellite entity definition, including descriptive attributes, effective time and expiration time.

[0162] In one embodiment, it further includes:

[0163] A source data acquisition module, used to acquire source system data associated with the Hub entity model, wherein the source system data includes a historical status record of a business object;

[0164] A data change analysis module, used to parse the source system data and extract data change information, wherein the data change information includes business key changes, relationship changes, attribute changes and timestamps;

[0165] A loading rule construction module, used to construct a Data Vault loading rule according to the data change information, wherein the loading rule includes a business key loading strategy of a Hub entity, a relationship loading strategy of a Link entity, and a history loading strategy of a Satellite entity;

[0166] The ETL mapping generation module is used to generate an ETL mapping solution based on the loading rules, and provide the ETL mapping solution to a data engineer for data loading implementation.

[0167] In one embodiment, it further includes:

[0168] A load analysis module, used to perform load analysis on the Hub entity, Link entity and Satellite entity, and divide the entities into different loading modes according to entity characteristic parameters, wherein the entity characteristic parameters include business key distribution characteristics, association complexity and historical change frequency;

[0169] The performance statistics module is used to count the corresponding data loading indicators for different loading modes and obtain the loading performance data of each entity type;

[0170] The scoring calculation module is used to perform multi-dimensional evaluation on the loading performance data of each entity type to obtain the entity loading score;

[0171] A weight allocation module is used to calculate the proportion of the loading score of each entity type to the overall score, and use the proportion as the model optimization weight of the corresponding entity type;

[0172] A key entity identification module is used to analyze each entity type according to the model optimization weights and identify key entity patterns that need to be optimized;

[0173] The optimization scheme generation module is used to generate an optimization scheme for the DataVault model according to the characteristic analysis results of the key entity pattern. The optimization scheme includes the business key index optimization strategy of the Hub entity, the partition loading strategy of the Link entity and the historical data storage strategy of the Satellite entity.

[0174] In one embodiment, it further includes:

[0175] The weight adjustment module is used to perform double weighting of the historical data volume and change frequency on the model optimization weight of each entity type to obtain the adjusted optimization weight;

[0176] Among them, the adjusted optimization weight calculation formula is: AWi,t=Wi,t×(1+α×log(Vi,t / V0))×(1+β×(Ci,t / C0)), wherein Wi,t represents the model optimization weight of the i-th entity type in the t-th time period, Vi,t represents the historical data volume of the i-th entity type in the t-th time period, V0 represents the baseline historical data volume, Ci,t represents the change frequency of the i-th entity type in the t-th time period, C0 represents the baseline change frequency, α represents the data volume adjustment factor and 0<α<1, β represents the change frequency adjustment factor and 0<β<1.

[0177] In one embodiment, the present application provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data warehouse construction method based on the Data Vault model is implemented.

[0178] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0180] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0181] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a data warehouse based on a Data Vault model, characterized in that: The steps include: Obtain metadata information and data content from the data source system; Extracting a business subject identifier according to the metadata information, wherein the business subject identifier is used to determine the business field classification of the data; According to the business subject identifier, matching is performed in the Data Vault model template library to determine the Hub entity model corresponding to the business subject identifier; Extracting business keys and analyzing relationships on the data content to obtain inter-entity association information; Generate a structured Data Vault model definition based on the association information, and associate and map related Link entities and Satellite entities based on the Hub entity model to obtain a complete Data Vault model structure; Extracting the business subject identifier according to the metadata information specifically includes the following steps: Obtaining table structure definition and field description in the metadata information; Parsing the table structure definition and field description to obtain a candidate set of business entities; The business subject identifier is extracted from the business entity candidate set according to a preset business subject identification rule, wherein the business subject identification rule includes: Determine in sequence whether there is a Hub entity feature in the business entity candidate set according to a preset priority order, wherein the Hub entity feature includes a business primary key feature and a core business object feature, and the business primary key feature and the core business object feature can uniquely determine the Hub entity type in the Data Vault model; Business key extraction and relationship analysis are performed on the data content to obtain the association information between entities, which specifically includes the following steps: Identify a business identification field and an association relationship field from the data content, wherein the business identification field is used to construct a business key of a Hub entity, and the association relationship field is used to construct an association relationship of a Link entity; For the business identification field, analyze it according to the preset Hub entity construction rules to obtain the Hub entity definition, including the business key, loading time and data source; For the association relationship field, analyze it according to the preset Link entity construction rules to obtain the Link entity definition, including the foreign key reference, relationship type and timestamp of the associated Hub entity; Context attribute information and historical change information are extracted from the data content, and grouped according to attribute characteristics to obtain Satellite entity definition, including descriptive attributes, effective time, and expiration time.

2. The method for constructing a data warehouse based on the Data Vault model according to claim 1, characterized in that: After obtaining the complete Data Vault model structure, the method further includes the following steps: Acquire source system data associated with the Hub entity model, wherein the source system data includes a historical status record of a business object; Parsing the source system data to extract data change information, wherein the data change information includes business key changes, relationship changes, attribute changes, and timestamps; According to the data change information, construct a Data Vault loading rule, wherein the loading rule includes a business key loading strategy of a Hub entity, a relationship loading strategy of a Link entity, and a history loading strategy of a Satellite entity; Based on the loading rules, an ETL mapping solution is generated, and the ETL mapping solution is provided to a data engineer for data loading implementation.

3. The method for constructing a data warehouse based on the Data Vault model according to claim 2, characterized in that: After generating the ETL mapping solution based on the loading rules, the method further comprises the following steps: Perform load analysis on the Hub entity, Link entity and Satellite entity, and divide the entities into different loading modes according to entity characteristic parameters, wherein the entity characteristic parameters include business key distribution characteristics, association complexity and historical change frequency; According to different loading modes, the corresponding data loading indicators are counted respectively to obtain the loading performance data of each entity type; Perform multi-dimensional evaluation on the loading performance data of each entity type to obtain the entity loading score; Calculate the ratio of the loading score of each entity type to the overall score, and use the ratio as the model optimization weight of the corresponding entity type; Analyze each entity type according to the model optimization weights to identify key entity patterns that need to be optimized; According to the characteristic analysis results of the key entity pattern, an optimization scheme of the Data Vault model is generated, wherein the optimization scheme includes a business key index optimization strategy of the Hub entity, a partition loading strategy of the Link entity, and a historical data storage strategy of the Satellite entity.

4. The method for constructing a data warehouse based on the Data Vault model according to claim 3, characterized in that: Before analyzing each entity type according to the model optimization weight, the method further includes the following steps: The model optimization weight of each entity type is double-weighted by the amount of historical data and the frequency of change to obtain the adjusted optimization weight; The calculation formula for the adjusted optimization weight is: AWi,t=Wi,t×(1+α×log(Vi,t / V0))×(1+β×(Ci,t / C0)), wherein AWi,t represents the adjusted optimization weight, Wi,t represents the model optimization weight of the i-th entity type in the t-th time period, Vi,t represents the historical data volume of the i-th entity type in the t-th time period, V0 represents the baseline historical data volume, Ci,t represents the change frequency of the i-th entity type in the t-th time period, C0 represents the baseline change frequency, α represents the data volume adjustment factor and 0<α<1, β represents the change frequency adjustment factor and 0<β<1.

5. A data warehouse construction device based on the Data Vault model, characterized in that: include: Data acquisition module, used to obtain metadata information and data content of the data source system; A business subject identifier extraction module, used to extract a business subject identifier according to the metadata information, wherein the business subject identifier is used to determine the business field classification of the data; A Hub entity model determination module, used to match the business subject identifier in the Data Vault model template library to determine the Hub entity model corresponding to the business subject identifier; An entity association information acquisition module is used to extract business keys and analyze relationships of the data content to obtain entity association information; A Data Vault model acquisition module is used to generate a structured Data Vault model definition according to the association information, and to associate and map related Link entities and Satellite entities according to the Hub entity model to obtain a complete Data Vault model structure; The business subject identification extraction module includes: A table structure acquisition unit, used to acquire the table structure definition and field description in the metadata information; A business entity parsing unit, used to parse the table structure definition and field description to obtain a business entity candidate set; The subject identification rule unit is used to extract the business subject identifier from the business entity candidate set according to a preset business subject identification rule, wherein the business subject identification rule includes: A Hub feature judgment unit, used to judge whether there are Hub entity features in the business entity candidate set in sequence according to a preset priority order, wherein the Hub entity features include business primary key features and core business object features, and the business primary key features and the core business object features can uniquely determine the Hub entity type in the Data Vault model; The entity association information acquisition module includes: A field identification unit, used to identify a business identification field and an association relationship field from the data content, wherein the business identification field is used to construct a business key of a Hub entity, and the association relationship field is used to construct an association relationship of a Link entity; A Hub entity analysis unit, used to analyze the business identification field according to a preset Hub entity construction rule to obtain a Hub entity definition, including a business key, a loading time, and a data source; A Link entity analysis unit, used to analyze the association relationship field according to a preset Link entity construction rule to obtain a Link entity definition, including a foreign key reference, a relationship type, and a timestamp of an associated Hub entity; The Satellite entity analysis unit is used to extract context attribute information and historical change information from the data content, and group them according to attribute characteristics to obtain Satellite entity definition, including descriptive attributes, effective time and expiration time.

6. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for constructing a data warehouse based on a Data Vault model described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a data warehouse based on a Data Vault model according to any one of claims 1 to 4 are implemented.

Images