Implementation Method of Building Heterogeneous Data Warehouse

Abstract

The present application proposes an implementation method for building a heterogeneous data warehouse, comprising the following steps: step 1: identifying the types of heterogeneous data sources to obtain data asset import dimension descriptions and measurement system descriptions through data model mapping; step 2: based on the set heterogeneous data integration strategy library, according to the data asset import dimension descriptions and measurement system descriptions, converting the data business transaction dynamic process into a dynamic fact table; step 3: based on the constructed heterogeneous data ETL template, generating parallel incremental ETL tasks according to the dynamic fact table; step 4: according to the parallel incremental ETL tasks, accessing the heterogeneous data sources through the constructed heterogeneous data access interface to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an implementation method for building a heterogeneous data warehouse. Background Art

[0002] In the current information age, the data environment of enterprises and organizations is becoming increasingly complex, and the data sources are diversified, covering multiple data sources such as relational databases, NoSQL databases, text files, logs, API interfaces, etc., forming the so-called "heterogeneous data" pattern. These data sources not only differ in data format (such as structured, semi-structured, and unstructured data), but also in data models, storage methods, and access protocols, which brings severe challenges to data integration and analysis.

[0003] Traditional data warehouse construction methods often focus on processing single or similar data sources and are difficult to directly apply to heterogeneous data environments. For example, a simple ETL (Extract, Transform, Load) process usually requires customization of independent data extraction and transformation logic for each data source, which not only has a long development cycle and high maintenance costs, but also has difficulty coping with dynamic changes in data sources. In addition, with the explosive growth of data volume, traditional methods are inefficient in processing large-scale data, especially for real-time or near real-time data analysis needs.

[0004] Especially when processing dynamic business transaction data, due to the complexity and changeability of business logic, traditional static data models are difficult to adapt flexibly, resulting in delayed updates and poor data timeliness, which affects the accuracy and effectiveness of the decision support system. Summary of the invention

[0005] The purpose of this application is to propose an implementation method for building a heterogeneous data warehouse to solve or alleviate the technical problems existing in the prior art.

[0006] The technical solutions provided by the embodiments of this application are as follows:

[0007] A method for building a heterogeneous data warehouse, comprising:

[0008] Step 1: Identify the types of heterogeneous data sources to obtain the data asset import dimension description and measurement system description through data model mapping;

[0009] Step 2: Based on the set heterogeneous data integration strategy library, according to the data asset import dimension description and measurement system description, the data business transaction dynamic process is converted into a dynamic fact table;

[0010] Step 3: Based on the constructed heterogeneous data ETL template and the dynamic fact table, generate parallel incremental ETL tasks;

[0011] Step 4: According to the parallel incremental ETL task, the heterogeneous data source is accessed through the constructed heterogeneous data access interface to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse.

[0012] Optionally, the step 1 includes:

[0013] Traversing the data source connectors to establish a continuous connection relationship between the heterogeneous data sources and the data source connectors;

[0014] Based on the connection relationship, start a metadata extraction script to extract metadata information from the heterogeneous data source, wherein the metadata information includes at least one of the following: table structure, field name, data type, primary key, foreign key, index;

[0015] According to the data model mapping method, the metadata information is analyzed in terms of dimensions and metrics to obtain a data asset import dimension description and a metric system description.

[0016] Optionally, starting a metadata extraction script based on the connection relationship to extract metadata information from the heterogeneous data source includes:

[0017] Based on the connection relationship, calling the constructed heterogeneous data source dependency library to load the metadata extraction script;

[0018] Based on the loaded metadata extraction script, a metadata query template is called to extract metadata information from the heterogeneous data source.

[0019] Optionally, calling a constructed heterogeneous data source dependency library based on the connection relationship to load the metadata extraction script includes:

[0020] Adding the connection relationship to an abstract data access layer, so as to configure corresponding adapter modes for different heterogeneous data sources in the database access layer;

[0021] Based on the corresponding adapter modes configured for different heterogeneous data sources in the data access layer, the constructed heterogeneous data source dependency library is called to load the metadata extraction script.

[0022] Optionally, the extracting metadata information from the heterogeneous data source based on the loaded metadata extraction script, calling a metadata query template, includes:

[0023] Based on the loaded metadata extraction script, reading connection information, authentication credentials and query template paths of different heterogeneous data sources to dynamically load corresponding metadata query templates;

[0024] According to the loaded metadata query template, a full-page scan is performed on the heterogeneous data source to extract metadata information from the heterogeneous data source.

[0025] Optionally, performing dimension and metric parsing on the metadata information according to the data model mapping method to obtain a data asset import dimension description and a metric system description includes:

[0026] According to the data schema mapping method, contextually analyzing the metadata information to extract dimension fields and metric fields therefrom;

[0027] Based on the extracted dimension fields and measure fields, construct the dimension hierarchy and measure field statistics respectively;

[0028] According to the constructed dimension hierarchy, dimension association keys are defined to generate the data asset import dimension description;

[0029] According to the metric field statistics, metric identification and classification are performed to generate the metric system description.

[0030] Optionally, the contextual analysis of the metadata information to extract dimension fields and metric fields therefrom according to the data schema mapping method includes:

[0031] According to the data model mapping method, construct a field relationship diagram and business parsing logic;

[0032] Exploring the dependency and hierarchical relationships between fields in the metadata information according to the field relationship diagram;

[0033] Based on the dependency and hierarchical relationship between the fields, the metadata information is contextually analyzed to extract dimension fields and metric fields therefrom.

[0034] Optionally, constructing a field relationship diagram and business parsing logic according to the data model mapping method includes:

[0035] Projecting the data mode mapping method onto a graph data structure to create field mapping nodes and business mapping nodes, wherein the relationship between the nodes is represented by edges between the field mapping nodes and the business mapping nodes;

[0036] Based on the created field mapping nodes and business mapping nodes, build the field relationship diagram and business parsing logic.

[0037] Optionally, exploring the dependency relationship and hierarchical relationship between fields in the metadata information based on the field relationship graph and the constructed regular expression includes:

[0038] Constructing a potential role dependency tree according to the field relationship diagram;

[0039] Based on the constructed regular expression, community detection is performed on the metadata information according to the potential role dependency tree to explore the dependency and hierarchical relationship between fields in the metadata information.

[0040] Optionally, based on the dependencies and hierarchical relationships among the fields, contextually analyzing the metadata information to extract dimension fields and metric fields therefrom includes: based on the dependencies and hierarchical relationships among the fields, according to the acquired field names, data types and descriptions, using regular expressions and keyword matching to contextually analyze the metadata information to extract dimension fields and metric fields therefrom.

[0041] Optionally, constructing a dimension hierarchy and metric field statistics respectively according to the extracted dimension fields and metric fields includes:

[0042] Performing foreign key constraints on the extracted dimension fields and measure fields, and performing primary key matching based on the foreign key constraints to determine the dimensional hierarchical relationship between the fields;

[0043] Based on the set aggregation function, a gradient aggregation operation is performed on the dimension hierarchy relationship, and a hash table is used to store the aggregation operation results. The dimension hierarchy structure and the measurement field statistics are constructed by nesting the hash table.

[0044] Optionally, defining dimension association keys according to the constructed dimension hierarchy to generate the data asset import dimension description includes:

[0045] Performing a hierarchical depth analysis on the constructed dimension hierarchy structure to determine the dimension association hub key in each level;

[0046] Based on the dimension association hub key in each level, cross-level association is performed to determine the cross-level association relationship between different levels;

[0047] Generate composite keys based on cross-level association relationships between different levels to create dimension association key indexes;

[0048] Based on the dimension association key index, a dimension association key is defined, and the dimension association key is structured to generate the data asset import dimension description.

[0049] Optionally, the performing metric identification and classification according to the metric field statistics to generate the metric system description includes:

[0050] Performing statistics on the metric fields to generate a metric field distribution map;

[0051] Extracting the characteristics of the metric field distribution map to obtain the metric field property characteristics for metric identification and classification;

[0052] Business logic labels are assigned to the results of the metric identification and classification to generate the metric system description.

[0053] Optionally, the step 2 includes:

[0054] Parse the set heterogeneous data integration strategy library to obtain the strategy configuration for data extraction, conversion, and loading;

[0055] Based on the configuration script between the initiated strategy and description, according to the data extraction, transformation, and loading (ETL) strategy configuration, linear programming matching is performed with the data asset import dimension description and the measurement system description to transform the dynamic process of data business transactions into a dynamic fact table.

[0056] Optionally, the step 3 includes:

[0057] The constructed heterogeneous data ETL template is parsed to obtain a framework of pre-configured steps, transformation rules, and parameter settings that can be called when acquiring heterogeneous data assets from multiple heterogeneous data, and the framework of pre-configured steps, transformation rules, and parameter settings that can be called is mapped to the dynamic fact table to allocate an anchor point for incremental extraction and a data partition matching the anchor point to generate a parallel incremental ETL task.

[0058] Optionally, step 4 includes: extracting the incrementally extracted anchor point and the data partition matching the anchor point according to the parallel incremental ETL task, extracting the framework of pre-configured steps, conversion rules and parameter settings when obtaining heterogeneous data assets from multiple heterogeneous data based on the incrementally extracted anchor point and the data partition matching the anchor point, accessing the heterogeneous data sources based on the framework of pre-configured steps, conversion rules and parameter settings to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse.

[0059] The implementation method for building a heterogeneous data warehouse proposed in this application, through the type identification and data model mapping in step 1, the scheme can intelligently identify data from different sources and formats (such as structured, semi-structured, and unstructured data), and generate a unified data asset import dimension description and measurement system description according to the characteristics of these data. In this way, there is no need to customize the ETL logic for each data source separately, which significantly reduces the development complexity and maintenance cost, and improves the compatibility with heterogeneous data sources. Through step 2, based on the generation of dynamic fact tables, the dynamic change process of business transactions can be flexibly reflected, overcoming the limitations of traditional static data models. This dynamic modeling method ensures that the data warehouse can capture and reflect the latest business conditions in a timely manner, improves the timeliness of data and the accuracy of decision support systems. Through the parallel incremental ETL tasks generated based on the dynamic fact table in step 3, the parallel processing capabilities of modern computing resources can be effectively utilized to accelerate the data extraction, conversion and loading process. Compared with traditional methods, this greatly shortens the data processing cycle, especially when processing large-scale data sets, significantly improves efficiency, and better meets the needs of real-time or near real-time data analysis. The heterogeneous data access interface built in step 4 provides a unified access layer to connect various heterogeneous data sources, simplifying the data integration process and reducing the complexity of data access. This not only makes the construction of the data warehouse more standardized, but also ensures the efficiency and stability of the data integration process, which is conducive to long-term operation and maintenance and system scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0061] Figure 1 A flowchart of an implementation method for building a heterogeneous data warehouse according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0063] Figure 1 The following is a flow chart of a method for building a heterogeneous data warehouse according to an embodiment of the present application. A method for building a heterogeneous data warehouse includes:

[0064] Step 1: Identify the types of heterogeneous data sources to obtain the data asset import dimension description and measurement system description through data model mapping;

[0065] Step 2: Based on the set heterogeneous data integration strategy library, according to the data asset import dimension description and measurement system description, the data business transaction dynamic process is converted into a dynamic fact table;

[0066] Step 3: Based on the constructed heterogeneous data ETL template and the dynamic fact table, generate parallel incremental ETL tasks;

[0067] Step 4: According to the parallel incremental ETL task, the heterogeneous data source is accessed through the constructed heterogeneous data access interface to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse.

[0068] The implementation method for building a heterogeneous data warehouse proposed in this application, through the type identification and data model mapping in step 1, the scheme can intelligently identify data from different sources and formats (such as structured, semi-structured, and unstructured data), and generate a unified data asset import dimension description and measurement system description according to the characteristics of these data. In this way, there is no need to customize the ETL logic for each data source separately, which significantly reduces the development complexity and maintenance cost, and improves the compatibility with heterogeneous data sources. Through step 2, based on the generation of dynamic fact tables, the dynamic change process of business transactions can be flexibly reflected, overcoming the limitations of traditional static data models. This dynamic modeling method ensures that the data warehouse can capture and reflect the latest business conditions in a timely manner, improves the timeliness of data and the accuracy of decision support systems. Through the parallel incremental ETL tasks generated based on the dynamic fact table in step 3, the parallel processing capabilities of modern computing resources can be effectively utilized to accelerate the data extraction, conversion and loading process. Compared with traditional methods, this greatly shortens the data processing cycle, especially when processing large-scale data sets, significantly improves efficiency, and better meets the needs of real-time or near real-time data analysis. The heterogeneous data access interface built in step 4 provides a unified access layer to connect various heterogeneous data sources, simplifying the data integration process and reducing the complexity of data access. This not only makes the construction of the data warehouse more standardized, but also ensures the efficiency and stability of the data integration process, which is conducive to long-term operation and maintenance and system scalability.

[0069] Optionally, the step 1 includes:

[0070] Traversing the data source connectors to establish a continuous connection relationship between the heterogeneous data sources and the data source connectors;

[0071] Based on the connection relationship, start a metadata extraction script to extract metadata information from the heterogeneous data source, wherein the metadata information includes at least one of the following: table structure, field name, data type, primary key, foreign key, index;

[0072] According to the data model mapping method, the metadata information is analyzed in terms of dimensions and metrics to obtain a data asset import dimension description and a metric system description.

[0073] To this end, the above specific scheme in step 1 has the following technical advantages:

[0074] 1. By traversing the data source connectors and automatically establishing connection relationships, the solution achieves seamless access to diverse data sources, improving the usability and scalability of the system. Metadata extraction not only includes basic table structure information, but also includes in-depth details such as field names, data types, primary keys, foreign keys, indexes, etc., providing a comprehensive and accurate information foundation for subsequent data processing and analysis.

[0075] 2. Based on metadata information, the solution constructs the import dimension description and measurement system description of data assets through the dimension and measurement analysis of elements such as table structure. This process is essentially a logical unification and standardization of heterogeneous data, ensuring that data from different sources can be understood and operated under the same framework, solving the integration problem caused by differences in format and structure of heterogeneous data.

[0076] 3. The metadata information extraction and analysis process helps improve the transparency and traceability of data, and facilitates data managers and analysts to deeply understand the characteristics and quality of data. For example, by identifying primary keys and foreign keys, we can better understand the relationship between data, and the clarification of data types provides accurate guidance for data cleaning and conversion, reducing errors caused by data type mismatch.

[0077] 4. The metadata processing process greatly shortens the time from data source discovery to data warehouse model design, making the construction and adjustment of data warehouses faster. Especially in scenarios where new data sources need to be added frequently or data models need to be adjusted, this mechanism can quickly respond to business changes, speed up project iteration, and improve overall business agility.

[0078] To this end, an exemplary implementation code is provided during product implementation:

[0079]

[0080]

[0081] The above code shows how to traverse different data source connectors, establish connections, extract metadata, and perform simple parsing to distinguish dimensions and measures. In actual applications, the metadata_extractor module and the data_source_connectors module can be implemented according to the specific data source type and API.

[0082] Optionally, starting a metadata extraction script based on the connection relationship to extract metadata information from the heterogeneous data source includes:

[0083] Based on the connection relationship, calling the constructed heterogeneous data source dependency library to load the metadata extraction script;

[0084] Based on the loaded metadata extraction script, a metadata query template is called to extract metadata information from the heterogeneous data source.

[0085] To this end, building a heterogeneous data source dependency library and metadata query template has the following technical benefits:

[0086] 1. By calling heterogeneous data source dependency libraries, abstract encapsulation of different data source APIs and protocols is achieved, so that the metadata extraction process does not depend on specific data source technical details. This not only simplifies the integration work with new data sources, but also ensures that the system can adapt quickly when the data source changes or new data sources are added, greatly enhancing the system's compatibility and future scalability.

[0087] 2. Using metadata query templates, the system can automatically perform metadata extraction according to predefined rules and patterns, avoiding manual writing of query statements for each data source, reducing human errors and improving work efficiency. The templated approach also promotes the standardization of the metadata extraction process, ensuring that the data structure of each extraction is consistent, which is conducive to subsequent data processing and analysis.

[0088] 3. Through the centralized management and maintenance of metadata extraction scripts and query templates, repetitive coding work is reduced and the system maintenance cost is reduced. Developers only need to focus on the optimization and upgrade of templates and dependent libraries to meet the extraction needs of various data sources, which improves the work efficiency and response speed of the development team.

[0089] 4. The standardized metadata extraction process helps to form a unified data view and provides a solid foundation for data governance. The accuracy and completeness of metadata are the prerequisites for data quality management. This solution provides accurate metadata information for subsequent data cleaning, conversion and verification through automated and standardized extraction, supporting comprehensive data quality monitoring and improvement.

[0090] To this end, the following provides an exemplary implementation code. First, a MetadataExtractionManager class is defined to manage the metadata extraction process of heterogeneous data sources. DataSourceLibrary is used to load metadata extraction scripts specific to the data source, and the MetadataExtractor class is used to perform the actual metadata extraction operation. The query template dictionary query_templates is used to select the appropriate query template according to the type of data source.

[0091]

[0092]

[0093] Optionally, calling a constructed heterogeneous data source dependency library based on the connection relationship to load the metadata extraction script includes:

[0094] Adding the connection relationship to an abstract data access layer, so as to configure corresponding adapter modes for different heterogeneous data sources in the database access layer;

[0095] Based on the corresponding adapter modes configured for different heterogeneous data sources in the data access layer, the constructed heterogeneous data source dependency library is called to load the metadata extraction script.

[0096] To this end, by configuring the adapter mode in the abstract data access layer to call the heterogeneous data source dependency library to load the metadata extraction script, it has the following technical benefits:

[0097] 1. The adapter pattern allows the design of independent interfaces and implementations for different data sources, which means that the integration of new data sources only requires the development of corresponding adapters without modifying the core logic of the data warehouse or other existing adapters, greatly reducing the complexity and cost of system expansion.

[0098] 2. By encapsulating the access logic of the data source in the adapter, high cohesion and low coupling of the code are achieved. The adapters are independent of each other, and each adapter can be regarded as an independent module, which is easy to test and maintain, and also convenient for reusing adapter code in different projects.

[0099] 3. In the data access layer, the adapter pattern provides a unified interface, allowing upper-level applications to interact with different data sources in the same way. Regardless of how the characteristics and interfaces of the underlying data sources change, the loading process of the data extraction script can be standardized and consistent, thereby improving the stability and reliability of the system.

[0100] 4. When a data source has a problem or needs to be upgraded, you only need to focus on the adapter of that data source without affecting the operation of the entire system or other data sources. This significantly simplifies maintenance work and speeds up fault location and repair processes.

[0101] 5. Using the adapter pattern, developers can focus on business logic without having to delve into the specific details of each data source. The unified data access interface reduces development complexity, reduces programming errors caused by differences in data sources, and improves development efficiency and code quality.

[0102] 6. In the design of the data access layer, the adapter pattern naturally supports asynchronous calls and parallel processing mechanisms, which can further improve the efficiency of data extraction script loading and data processing. Especially when processing a large number of heterogeneous data sources, it can effectively utilize system resources and speed up the construction of data warehouses.

[0103] The following is a sample code that defines an abstract base class DataSourceAdapter and two specific adapters, MySQLAdapter and MongoDBAdapter, corresponding to MySQL and MongoDB data sources respectively. The DataAccessLayer class is responsible for managing the adapters and provides an interface for loading all data source metadata extraction scripts. In this way, you can flexibly add adapters for any newly added heterogeneous data sources, maintaining the openness and scalability of the system.

[0104]

[0105]

[0106] Optionally, the extracting metadata information from the heterogeneous data source based on the loaded metadata extraction script, calling a metadata query template, includes:

[0107] Based on the loaded metadata extraction script, reading connection information, authentication credentials and query template paths of different heterogeneous data sources to dynamically load corresponding metadata query templates;

[0108] According to the loaded metadata query template, a full-page scan is performed on the heterogeneous data source to extract metadata information from the heterogeneous data source.

[0109] To this end, the above dynamic loading and full-page scanning technology can achieve the following technical benefits:

[0110] 1. Dynamically read connection information, authentication credentials, and query template paths, so that the system can select the most appropriate query logic based on different data source characteristics at runtime, without the need to pre-configure all possible data sources and templates, greatly improving the adaptability and flexibility of the system. This means that even in the face of new or unknown data source types, the system can quickly adjust and integrate effectively.

[0111] 2. Separate the reading of authentication credentials from the execution of metadata extraction scripts and dynamically load them at runtime, reducing the risk of credential leakage. Avoid hard-coding credentials in scripts or configuration files, improve the security of data source access, and comply with data protection best practices.

[0112] 3. The full-page scanning strategy ensures in-depth exploration of data sources and comprehensive extraction of metadata information from each data source, including but not limited to table structure, field attributes, relationships, etc. This provides a detailed data model foundation for subsequent data warehouse construction and analysis, and ensures accurate data mapping and conversion.

[0113] 4. By dynamically matching the most suitable query template, the data reading process can be optimized in a targeted manner, the retrieval of invalid data can be reduced, and the efficiency of data extraction can be improved. In addition, the on-demand loading of templates reduces memory usage and optimizes system resource utilization.

[0114] 5. Encapsulate the query logic in a dynamically loadable template so that changes or optimizations to specific data sources can be made without changing the core extraction logic, simplifying system maintenance and future feature upgrades and reducing long-term operation and maintenance costs.

[0115] The following is an exemplary code, in which the MetadataExtractor class is responsible for reading the connection information and template path in the configuration file, dynamically loading the corresponding metadata query template (such as MySQL or MongoDB), and then using the template to perform a full-page scan to extract metadata from the data source.

[0116]

[0117]

[0118] Optionally, performing dimension and metric parsing on the metadata information according to the data model mapping method to obtain a data asset import dimension description and a metric system description includes:

[0119] According to the data schema mapping method, contextually analyzing the metadata information to extract dimension fields and metric fields therefrom;

[0120] Based on the extracted dimension fields and measure fields, construct the dimension hierarchy and measure field statistics respectively;

[0121] According to the constructed dimension hierarchy, dimension association keys are defined to generate the data asset import dimension description;

[0122] According to the metric field statistics, metric identification and classification are performed to generate the metric system description.

[0123] To this end, by analyzing metadata information and mapping it according to data models, we can achieve a deep understanding and structured organization of data assets, which has the following technical benefits:

[0124] 1. Through the careful analysis of metadata information, the system can automatically identify the dimension fields and measurement fields in the data set, which greatly enhances the ability to understand complex data sets. Users can quickly understand the composition of the data set without manually browsing each field, which promotes efficient discovery and use of data.

[0125] 2. Building a dimension hierarchy is a highlight of this solution. By identifying dimension fields and establishing hierarchical relationships between them, the system can form a clear data navigation map, making it easier for users to access and query data according to the logical structure. This structuring not only improves query efficiency, but also makes the data analysis process more intuitive and easy to understand.

[0126] 3. Defining dimension association keys is a key step to ensure consistency and integrity of data assets during the import process. It helps the system establish associations between different data sources, ensures the accuracy of cross-dataset analysis, and reduces the risk of wrong decisions caused by inconsistent data.

[0127] 4. Statistics and classification of metric fields form a description of the metric system. This process not only helps to quickly identify key performance indicators (KPIs), but also supports the standardization and normalization of metrics. This means that analysts can focus on core business indicators more quickly, conduct more accurate data analysis and prediction, and improve the quality of decision support.

[0128] 5. Automating the parsing process of dimensions and metrics helps build a unified data dictionary and metadata management system, which is essential for meeting data governance requirements. Good data governance can ensure the compliance of data use, support data lifecycle management, and meet increasingly stringent privacy protection and data security regulations.

[0129] The following code defines some auxiliary functions to complete the processing of metadata information, including extracting dimension and measure fields, building dimension hierarchies, defining dimension association keys, and classifying measure fields.

[0130]

[0131]

[0132] Optionally, the contextual analysis of the metadata information to extract dimension fields and metric fields therefrom according to the data schema mapping method includes:

[0133] According to the data model mapping method, construct a field relationship diagram and business parsing logic;

[0134] Exploring the dependency and hierarchical relationships between fields in the metadata information according to the field relationship diagram;

[0135] Based on the dependency and hierarchical relationship between the fields, the metadata information is contextually analyzed to extract dimension fields and metric fields therefrom.

[0136] To this end, through data model mapping and context analysis, the following technical benefits are achieved:

[0137] 1. Constructing a field relationship diagram not only depicts the static structural connection between data tables, but also deeply explores the dynamic business relevance and dependency between fields through business analysis logic. This helps to more accurately identify which fields play the role of dimensions that describe the characteristics of things, and which fields are the key to quantitative analysis - measurement fields. For example, through the analysis of order data, it is possible to clearly distinguish customer ID and product category as dimensions, and order amount and sales volume as metrics.

[0138] 2. Traditional metadata management often relies on manual review and annotation, which is time-consuming and error-prone. This solution automatically explores the dependencies and hierarchical relationships between fields, realizes the automatic extraction of dimension and metric fields, greatly reduces labor costs, improves processing speed, and makes the data preparation process more efficient and accurate.

[0139] 3. In a complex data ecosystem, a unified data model and dimensional standards are essential. This technical solution can ensure that the same dimensional fields in different data sources are consistently identified and processed, which helps to build a unified data dictionary and dimensional model, improve data consistency and comparability, and lay a solid foundation for subsequent data integration, analysis reports and decision support.

[0140] 4. By accurately identifying dimensions and measures, data cubes or data marts can be quickly built, supporting users to slice and dice data from multiple dimensions, and perform flexible data exploration and in-depth analysis. This is crucial for discovering patterns, trends, and correlations hidden behind the data, and enhances the flexibility and insight of data analysis.

[0141] 5. This technology not only serves the single data import process, but also provides strong technical support for continuous data governance. It makes data quality monitoring, metadata change management, data lineage tracking and other governance activities more systematic and automated, effectively improving the maturity of enterprise data management and the value of data assets.

[0142] The following code snippet first defines a simple list of metadata information, then builds an undirected graph based on the given field dependencies. Next, it defines the logic to distinguish between dimension fields and metric fields. Finally, it traverses the nodes in the graph, determines the role of each field based on the previously defined logic, and prints out the results of the dimension fields and metric fields.

[0143] import networkx as nx

[0144] #Metadata information, each element represents a field and its basic information

[0145]

[0146] Optionally, constructing a field relationship diagram and business parsing logic according to the data model mapping method includes:

[0147] Projecting the data mode mapping method onto a graph data structure to create field mapping nodes and business mapping nodes, wherein the relationship between the nodes is represented by edges between the field mapping nodes and the business mapping nodes;

[0148] Based on the created field mapping nodes and business mapping nodes, build the field relationship diagram and business parsing logic.

[0149] To this end, this method transforms the data model mapping method into a graph data structure, and builds a field relationship graph and business parsing logic on this basis, which has the following technical benefits:

[0150] 1. The graph data structure is easy to add, delete, modify and query. When the data source changes or business needs are adjusted, the nodes and edges can be quickly adjusted to reflect the latest data relationships, maintain the timeliness and accuracy of the data model, reduce maintenance costs, and improve the flexibility and scalability of the system.

[0151] 2. Based on the graph structure, graph algorithms can be applied to automatically discover patterns and optimize data flows, such as identifying data islands, circular dependencies, and other problems, automatically adjusting the order of data extraction, optimizing the data processing process, and improving the efficiency and quality of data processing.

[0152] 3. The graph structure is conducive to tracing data lineage, that is, the entire process from data source to final consumption, which facilitates data quality monitoring, data security auditing and compliance inspection. It is the basis for realizing the data governance system and ensures the integrity and credibility of the data.

[0153] 4. Through business mapping nodes, business rules and logic are encoded into the graph, so that the system can intelligently parse and classify data according to these logics. For example, it can automatically distinguish which fields are used as dimensions to describe business entities and which fields are used as metrics to reflect business indicators. This greatly improves the level of intelligent data processing.

[0154] 5. The constructed field relationship diagram and business analysis logic can be reused across projects and systems as a knowledge asset, reducing duplication of work and promoting the standardization and normalization of enterprise-level data architecture.

[0155] The following sample code shows how to use the NetworkX library to build a graph data structure based on field mapping and business mapping to represent field relationships and business logic.

[0156]

[0157]

[0158] This code first defines a `map_data_schema` function, which receives a set of field information and creates a graph data structure based on this information. In the graph, each node represents a field, and the node attributes record the field type, business meaning, and role (dimension or measure). Edges are used to represent business dependencies or other logical relationships between fields. Finally, the example shows how to query the information of nodes and edges in the graph, and how to filter out dimension and measure fields based on node attributes.

[0159] Optionally, exploring the dependency relationship and hierarchical relationship between fields in the metadata information based on the field relationship graph and the constructed regular expression includes:

[0160] Constructing a potential role dependency tree according to the field relationship diagram;

[0161] Based on the constructed regular expression, community detection is performed on the metadata information according to the potential role dependency tree to explore the dependency and hierarchical relationship between fields in the metadata information.

[0162] To this end, by constructing a potential role dependency tree and combining it with regular expressions for community detection, we aim to deeply explore the dependencies and hierarchical structures of fields in metadata information, which has the following technical benefits:

[0163] 1. By building a potential role dependency tree, the relationship network between fields can be intuitively displayed, helping analysts quickly grasp the core structure and logic of the data model. This not only improves the understanding of the deep-level associations of data, but also provides clear guidance for subsequent data integration, cleaning and analysis.

[0164] 2. Community detection using regular expressions is an intelligent method that can automatically identify groups of fields with similar characteristics or close connections, which often represent different dimensions or measurement levels of the data. This process automates the identification of complex data relationships, reduces the need for manual intervention, and improves processing efficiency.

[0165] 3. Regular expressions accurately capture specific dependency patterns between fields based on preset rules. This not only helps to accurately divide dimensions and measures, but also identifies implicit business logic and data constraints, improving the accuracy and reliability of the data model.

[0166] 4. By revealing the dependencies and hierarchical relationships between fields, this method helps to discover data redundancy, inconsistency and other issues, providing an important information basis for data governance. This is conducive to the continuous monitoring and improvement of data quality, ensuring the high quality and availability of data assets.

[0167] 5. This method is particularly effective when processing multi-source, heterogeneous data. It can adapt to the unique structures and patterns of different data sources, effectively integrate and parse complex data models through flexible regular expression matching, and enhance the adaptability and scalability of the system.

[0168] To this end, the following sample code shows how to use Python's networkx library to build a dependency tree and try to use regular expressions for simple community detection.

[0169]

[0170] In the above code, the simple_community_detection function is classified through regular expressions. Optionally, based on the dependencies and hierarchical relationships between the fields, the metadata information is contextually analyzed to extract dimension fields and metric fields therefrom, including: based on the dependencies and hierarchical relationships between the fields, according to the acquired field names, data types and descriptions, regular expressions and keyword matching are used to contextually analyze the metadata information to extract dimension fields and metric fields therefrom.

[0171] Optionally, constructing a dimension hierarchy and metric field statistics respectively according to the extracted dimension fields and metric fields includes:

[0172] Performing foreign key constraints on the extracted dimension fields and measure fields, and performing primary key matching based on the foreign key constraints to determine the dimensional hierarchical relationship between the fields;

[0173] Based on the set aggregation function, a gradient aggregation operation is performed on the dimension hierarchy relationship, and a hash table is used to store the aggregation operation results. The dimension hierarchy structure and the measurement field statistics are constructed by nesting the hash table.

[0174] To this end, the above solution has the following technical benefits through the application of foreign key constraints, determination of dimension levels, use of aggregate functions, and flexible operation of hash tables:

[0175] 1. By applying foreign key constraints to dimension fields and matching primary keys, the solution can identify and build clear dimensional hierarchical relationships. This process ensures the logic of the data model, so that each layer of dimensions can correctly reflect its context and granularity, and improves the accuracy and depth of multi-dimensional analysis in the data warehouse.

[0176] 2. Through gradient aggregation operations, the system can pre-calculate and store aggregate values ​​at different dimensional levels, reducing the amount of calculation required for real-time queries and significantly improving query response speed. At the same time, using hash tables to store these aggregate results, with the help of the fast search characteristics of hash tables, the efficiency of data access is further enhanced.

[0177] 3. The set aggregation function can be flexibly adjusted according to business needs, whether it is sum, average, maximum, minimum or other complex calculations, it can adapt to different analysis scenarios. In addition, with the expansion of the data model, new dimensions or measure fields can be easily integrated into the existing dimension hierarchy and statistical system, enhancing the scalability of the system.

[0178] 4. The constructed dimension hierarchy and hash table nesting operations provide a structured perspective for data management and maintenance. Data administrators can more easily track data quality, monitor data changes, and perform necessary data cleansing and update operations, reducing management complexity.

[0179] 5. The clear division of the dimensional hierarchy and the detailed statistics of the measurement fields provide data analysts with a rich and orderly data view. This not only supports advanced analysis operations such as multi-dimensional drilling and slicing, but also facilitates the discovery of hidden relationships and trends between data, promoting the improvement of data insight.

[0180] The following is a sample code that shows how to build a dimension hierarchy and measure field statistics based on the extracted dimension fields and measure fields.

[0181]

[0182]

[0183] Optionally, defining dimension association keys according to the constructed dimension hierarchy to generate the data asset import dimension description includes:

[0184] Performing a hierarchical depth analysis on the constructed dimension hierarchy structure to determine the dimension association hub key in each level;

[0185] Based on the dimension association hub key in each level, cross-level association is performed to determine the cross-level association relationship between different levels;

[0186] Generate composite keys based on cross-level association relationships between different levels to create dimension association key indexes;

[0187] Based on the dimension association key index, a dimension association key is defined, and the dimension association key is structured to generate the data asset import dimension description.

[0188] To this end, the dimensional model in the above data warehouse has the following technical benefits:

[0189] 1. Through the in-depth analysis of the dimension hierarchy, the dimension-related hub keys at each level are identified. This process helps to clearly define the logical relationship and dependency between different dimensions. This not only makes it easier for data analysts and business users to better understand and navigate complex multidimensional data, but also promotes data transparency and traceability.

[0190] 2. By determining cross-level associations and generating composite keys as dimension association key indexes, the speed and efficiency of data queries are significantly improved. Composite keys combine information from multiple dimensions, allowing the system to quickly locate specific data subsets, reducing the time cost of retrieving required information in a big data environment, especially when performing complex queries involving multi-dimensional analysis.

[0191] 3. The generated data asset import dimension description provides a standardized framework for data integration and ETL (extraction, transformation, loading) processes. The structured dimension association key ensures that the data extracted from different source data can be accurately mapped to the corresponding dimension level, reducing the complexity and error rate of data integration and accelerating the data preparation process.

[0192] 4. The definition and structuring of dimension association keys implements a mechanism to ensure data consistency across data tables and business areas. Maintaining this consistency is crucial to avoiding ambiguity and errors in data analysis and helps improve the reliability of decision support systems.

[0193] 5. This solution allows flexible expansion and adjustment of the dimensional model to adapt to changing business needs. As new data sources are added or analysis requirements change, the dimension association key system can be quickly adjusted to ensure that the data warehouse can continue to meet diverse analysis requests, enhancing the flexibility and adaptability of the system.

[0194] The following is a code example to illustrate how to implement the above process:

[0195]

[0196]

[0197] Optionally, the performing metric identification and classification according to the metric field statistics to generate the metric system description includes:

[0198] Performing statistics on the metric fields to generate a metric field distribution map;

[0199] Extracting the characteristics of the metric field distribution map to obtain the metric field property characteristics for metric identification and classification;

[0200] Business logic labels are assigned to the results of the metric identification and classification to generate the metric system description.

[0201] To this end, the solution is described by constructing a measurement system, which has the following technical benefits:

[0202] 1. Generate a distribution chart through statistical analysis of the metric field, which intuitively shows the frequency distribution of each metric value, helping data analysts quickly understand the characteristics and potential patterns of the data set. This step improves the understanding of the global characteristics of the data set, and is conducive to discovering outliers, skewed distributions or other atypical characteristics, thus laying the foundation for subsequent analysis.

[0203] 2. By extracting the characteristics of the distribution graph, the system can automatically identify the nature of the measurement (such as continuous, discrete, normal distribution, skewed distribution, etc.) and classify it accordingly. This automated process reduces the workload of manual classification, improves the efficiency and consistency when processing large-scale data sets, and avoids subjective errors that may occur in human classification.

[0204] 3. Based on the identification and classification of metrics, the abstract metric data is combined with specific business scenarios through the assignment of business logic labels. This step ensures that the results of data analysis can directly serve business decisions, such as marking "sales" as a revenue indicator and classifying "customer satisfaction score" as a service quality indicator. This not only enhances the practicality and explanatory power of the analysis results, but also promotes the implementation of data-driven decision-making.

[0205] 4. The constructed measurement system description provides a unified standard for data management within the organization, which facilitates data communication and integration across departments. The standardized measurement system reduces the problem of inconsistent data interpretation caused by differences in personal understanding and improves the consistency and reliability of data sharing and analysis.

[0206] 5. A clear description of the measurement system provides a solid foundation for the application of advanced analytical techniques (such as machine learning models and predictive analysis). Through a clearly classified and labeled measurement system, it is possible to more accurately select applicable algorithm models, improve prediction accuracy and insight, and further drive business optimization and strategy formulation.

[0207] To this end, the above process is implemented through the following sample code framework:

[0208]

[0209] Optionally, the step 2 includes:

[0210] Parse the set heterogeneous data integration strategy library to obtain the strategy configuration for data extraction, conversion, and loading;

[0211] Based on the configuration script between the initiated strategy and description, according to the data extraction, transformation, and loading (ETL) strategy configuration, linear programming matching is performed with the data asset import dimension description and the measurement system description to transform the dynamic process of data business transactions into a dynamic fact table.

[0212] To this end, in step 2, by parsing the heterogeneous data integration strategy library and implementing sophisticated ETL strategy configuration, combined with the dimensional description of data assets and the measurement system description, the transformation of the dynamic process of data business transactions into a dynamic fact table is achieved, which has the following technical benefits:

[0213] 1. The introduction of the heterogeneous data integration strategy library allows the system to adopt the best extraction, conversion and loading strategies for data from different sources and structures. This means that whether it is a structured database, semi-structured log or unstructured text, it can be efficiently integrated through pre-set strategies, reducing the workload of manual processing and speeding up the data preparation process.

[0214] 2. Through linear programming matching, it ensures that the data extraction and conversion process strictly follows business rules and data quality requirements, effectively solving data inconsistency and redundancy problems. This mechanism can automatically detect and correct data deviations, ensuring the consistency and accuracy of the data imported into the dynamic fact table in terms of dimensions and measurements.

[0215] 3. The design of dynamic fact tables enables the data model to reflect changes in business transactions in real time, providing an instant and comprehensive data view for the decision support system. As business transactions are dynamically updated, fact tables can also be adjusted accordingly, ensuring that the analysis results are closely synchronized with the current business status, improving the timeliness and accuracy of decision-making.

[0216] 4. Support rapid response to complex business scenarios: Combining the measurement system description with the data asset dimension, it can flexibly respond to changing business needs and analysis scenarios, quickly build or adjust data models, support the rapid launch and iteration of new business functions, and enhance the market adaptability and competitiveness of the enterprise.

[0217] The following provides a sample code framework for implementing heterogeneous data integration, policy analysis, ETL configuration application, and the process of building a dynamic fact table.

[0218]

[0219]

[0220] Optionally, the step 3 includes:

[0221] The constructed heterogeneous data ETL template is parsed to obtain a framework of pre-configured steps, transformation rules, and parameter settings that can be called when acquiring heterogeneous data assets from multiple heterogeneous data, and the framework of pre-configured steps, transformation rules, and parameter settings that can be called is mapped to the dynamic fact table to allocate an anchor point for incremental extraction and a data partition matching the anchor point to generate a parallel incremental ETL task.

[0222] To this end, step 3 has the following technical benefits when processing large-scale heterogeneous data sets:

[0223] 1. By parsing pre-configured ETL templates, the system can dynamically adjust the data extraction, conversion, and loading processes according to different data source characteristics and business needs. This design allows data engineers to quickly adapt to changes in data structure or new data sources without having to manually modify the code each time, greatly improving the maintainability and adaptability of the system.

[0224] 2. Map preconfigured steps to dynamic fact tables and assign anchor points for incremental extraction. This process automatically identifies key points of data changes, thereby efficiently performing incremental updates. Compared with full data processing, incremental ETL significantly reduces the time and resource consumption of data processing, and is particularly suitable for processing real-time or near real-time data streams. At the same time, by matching data partitions and generating parallel tasks, multi-core CPUs or distributed computing resources can be fully utilized to further accelerate processing speed.

[0225] 3. Preconfigured conversion rules with clear parameter settings help maintain consistency and accuracy in data processing. In complex data conversion processes, such as type conversion, data cleaning, data standardization, etc., these preset rules can effectively avoid human errors and ensure data quality.

[0226] 4. For scenarios involving multiple heterogeneous data sources, managing the ETL process in a templated and framework-based manner can greatly simplify the complexity of data integration. System administrators or developers only need to focus on higher-level policy configurations without having to delve into the specific details of each data source, which reduces the error rate and improves development efficiency.

[0227] 5. Pre-configured ETL templates and transformation rule frameworks promote the reuse of codes and strategies, making it possible to share best practices between different projects or departments. This not only reduces duplication of work, but also promotes the formation of data processing standards, ensuring that data processing methods throughout the organization remain consistent and professional.

[0228] To this end, the following example code is created to illustrate how to implement step 3:

[0229]

[0230]

[0231] The above code shows how to read the ETL template file, parse the steps, transformation rules and parameters, and then dynamically create parallel incremental ETL tasks in the Airflow DAG based on this information.

[0232] Optionally, step 4 includes: extracting the incrementally extracted anchor point and the data partition matching the anchor point according to the parallel incremental ETL task, extracting the framework of pre-configured steps, conversion rules and parameter settings when obtaining heterogeneous data assets from multiple heterogeneous data based on the incrementally extracted anchor point and the data partition matching the anchor point, accessing the heterogeneous data sources based on the framework of pre-configured steps, conversion rules and parameter settings to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse.

[0233] To this end, the solution based on the above step 4 has the following technical effects:

[0234] 1. By executing incremental ETL tasks in parallel, the efficiency of data extraction, conversion and loading is significantly improved. Incremental data from different data sources can be processed simultaneously, reducing the total time of data processing. Especially when facing large-scale data sets, this parallel processing mechanism is crucial to shortening the data warehouse update cycle.

[0235] 2. The design based on anchor points and data partitions enables the system to flexibly respond to changes in the data source structure. When a new data source is added or the data structure changes, only the corresponding anchor point definition and data partition strategy need to be adjusted, without large-scale modification of the entire ETL process, which improves the scalability and maintenance convenience of the system.

[0236] 3. By accurately matching the anchor points and data partitions of incremental extraction, it is possible to accurately identify and extract data changes since the last extraction, avoiding the waste of resources in full extraction. At the same time, it ensures the consistency and accuracy between the data in the data warehouse and the source system, which is of great significance for decision analysis and business operations.

[0237] 4. Preconfigured steps, conversion rules, and parameter setting frameworks allow reuse of proven processing logic, reducing repetitive development work. Combined with the strategy of dynamically allocating tasks, computing resources can be used more efficiently, especially in cloud environments or distributed systems, which can be scaled on demand and effectively control costs.

[0238] 5. Parallel processing frameworks are usually accompanied by error handling and retry mechanisms. Even if a part of the task fails, it will not affect the entire ETL process, enhancing the stability and fault tolerance of the system. At the same time, by monitoring the execution status of each task, problems can be discovered and solved in a timely manner to ensure the continuous availability of the data warehouse.

[0239] The following sample code shows how to extract and process incremental data from heterogeneous data sources based on the conceptual logic of parallel incremental ETL tasks.

[0240]

[0241]

[0242] The present application embodiment provides a device for building a heterogeneous data warehouse, which includes:

[0243] A first processing unit is used to identify the type of heterogeneous data sources, so as to obtain a data asset import dimension description and a measurement system description by means of data model mapping;

[0244] The second processing unit is used to convert the data business transaction dynamic process into a dynamic fact table based on the set heterogeneous data integration strategy library and the data asset import dimension description and measurement system description;

[0245] A third processing unit is used to generate a parallel incremental ETL task based on the constructed heterogeneous data ETL template and the dynamic fact table;

[0246] The fourth processing unit is used to access the heterogeneous data source through the constructed heterogeneous data access interface according to the parallel incremental ETL task to obtain the heterogeneous data assets to be processed for processing so as to build a heterogeneous data warehouse.

[0247] For an exemplary description of each unit in the above device, please refer to the above Figure 1 Records of.

[0248] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer executable program, and the processor is used to run the computer executable program to implement the method described in any embodiment of the present application.

[0249] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for building a heterogeneous data warehouse, characterized in that: include: Step 1: Identify the types of heterogeneous data sources to obtain the data asset import dimension description and measurement system description through data model mapping; Step 2: Based on the set heterogeneous data integration strategy library, according to the data asset import dimension description and measurement system description, the data business transaction dynamic process is converted into a dynamic fact table; Step 3: Based on the constructed heterogeneous data ETL template and the dynamic fact table, generate parallel incremental ETL tasks; Step 4: According to the parallel incremental ETL task, the heterogeneous data source is accessed through the constructed heterogeneous data access interface to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse; wherein, the step 1 includes: Traversing data source connectors to establish a connection relationship between heterogeneous data sources and the data source connectors; Based on the connection relationship, start a metadata extraction script to extract metadata information from the heterogeneous data source, wherein the metadata information includes at least one of the following: table structure, field name, data type, primary key, foreign key, index; According to the data model mapping method, the metadata information is analyzed in terms of dimensions and metrics to obtain a description of the data asset import dimensions and a description of the metric system; The step of starting a metadata extraction script based on the connection relationship to extract metadata information from the heterogeneous data source includes: Based on the connection relationship, calling the constructed heterogeneous data source dependency library to load the metadata extraction script; Based on the loaded metadata extraction script, a metadata query template is called to extract metadata information from the heterogeneous data source; The step of performing dimension and metric analysis on the metadata information according to the data model mapping method to obtain a data asset import dimension description and a metric system description includes: According to the data schema mapping method, contextually analyzing the metadata information to extract dimension fields and metric fields therefrom; Based on the extracted dimension fields and measure fields, construct the dimension hierarchy and measure field statistics respectively; According to the constructed dimension hierarchy, dimension association keys are defined to generate the data asset import dimension description; According to the metric field statistics, metric identification and classification are performed to generate the metric system description; The step of contextually analyzing the metadata information to extract dimension fields and metric fields therefrom according to the data schema mapping method includes: According to the data model mapping method, construct a field relationship diagram and business parsing logic; Exploring the dependency and hierarchical relationships between fields in the metadata information according to the field relationship diagram; Based on the dependency and hierarchical relationship between the fields, contextually analyzing the metadata information to extract dimension fields and metric fields therefrom; The step of constructing a dimension hierarchy and metric field statistics based on the extracted dimension fields and metric fields includes: Performing foreign key constraints on the extracted dimension fields and measure fields, and performing primary key matching based on the foreign key constraints to determine the dimensional hierarchical relationship between the fields; Based on the set aggregation function, a gradient aggregation operation is performed on the dimension hierarchy relationship, and a hash table is used to store the aggregation operation results. The dimension hierarchy structure and the measurement field statistics are constructed by nesting the hash table.

2. A method for building a heterogeneous data warehouse according to claim 1, characterized in that: Based on the connection relationship, calling the constructed heterogeneous data source dependency library to load the metadata extraction script includes: Adding the connection relationship to an abstract data access layer, so as to configure corresponding adapter modes for different heterogeneous data sources in the data access layer; Based on the corresponding adapter modes configured for different heterogeneous data sources in the data access layer, the constructed heterogeneous data source dependency library is called to load the metadata extraction script.

3. The method for building a heterogeneous data warehouse according to claim 1, characterized in that: The step of calling a metadata query template based on the loaded metadata extraction script to extract metadata information from the heterogeneous data source includes: Based on the loaded metadata extraction script, reading connection information, authentication credentials and query template paths of different heterogeneous data sources to dynamically load corresponding metadata query templates; According to the loaded metadata query template, a full-page scan is performed on the heterogeneous data source to extract metadata information from the heterogeneous data source.

4. The method for implementing a heterogeneous data warehouse according to claim 1, characterized in that: The method of constructing a field relationship diagram and business parsing logic according to the data model mapping method includes: Projecting the data mode mapping method onto a graph data structure to create field mapping nodes and business mapping nodes, wherein the relationship between the nodes is represented by edges between the field mapping nodes and the business mapping nodes; Build field relationship diagram and business parsing logic based on the created field mapping nodes and business mapping nodes; The step of exploring the dependency and hierarchical relationships between fields in the metadata information based on the field relationship graph and the constructed regular expression includes: Constructing a potential role dependency tree according to the field relationship diagram; Based on the constructed regular expression, according to the potential role dependency tree, community detection is performed on the metadata information to explore the dependency and hierarchical relationship between fields in the metadata information; The method of contextually analyzing the metadata information based on the dependencies and hierarchical relationships between the fields to extract dimension fields and metric fields therefrom includes: based on the dependencies and hierarchical relationships between the fields, according to the acquired field names, data types and descriptions, using regular expressions and keyword matching to contextually analyze the metadata information to extract dimension fields and metric fields therefrom.

5. The method for building a heterogeneous data warehouse according to claim 2, characterized in that: Defining dimension association keys according to the constructed dimension hierarchy to generate the data asset import dimension description includes: Performing a hierarchical depth analysis on the constructed dimension hierarchy structure to determine the dimension association hub key in each level; Based on the dimension association hub key in each level, cross-level association is performed to determine the cross-level association relationship between different levels; Generate composite keys based on cross-level association relationships between different levels to create dimension association key indexes; Based on the dimension association key index, a dimension association key is defined, and the dimension association key is structured to generate the data asset import dimension description.

6. The method for building a heterogeneous data warehouse according to claim 2, characterized in that: The step of performing metric identification and classification based on the metric field statistics to generate the metric system description includes: Performing statistics on the metric fields to generate a metric field distribution map; Extracting the characteristics of the metric field distribution map to obtain the metric field property characteristics for metric identification and classification; Business logic labels are assigned to the results of the metric identification and classification to generate the metric system description.

7. The method for building a heterogeneous data warehouse according to claim 1, characterized in that: The step 2 comprises: Parse the set heterogeneous data integration strategy library to obtain the strategy configuration for data extraction, conversion, and loading; Based on the configuration script between the initiated strategy and description, according to the strategy configuration of data extraction, conversion and loading, linear programming matching is performed with the data asset import dimension description and the measurement system description to transform the dynamic process of data business transactions into a dynamic fact table.

8. The method for building a heterogeneous data warehouse according to claim 1, characterized in that: The step 3 comprises: Parsing the constructed heterogeneous data ETL template to obtain a framework of pre-configured steps, transformation rules, and parameter settings that can be called when acquiring heterogeneous data assets from multiple heterogeneous data, and mapping the framework of pre-configured steps, transformation rules, and parameter settings that can be called to the dynamic fact table to allocate an anchor point for incremental extraction and a data partition matching the anchor point to generate a parallel incremental ETL task; Among them, the step 4 includes: extracting the incrementally extracted anchor point and the data partition matching the anchor point according to the parallel incremental ETL task, extracting the framework of pre-configured steps, conversion rules and parameter settings when obtaining heterogeneous data assets from multiple heterogeneous data based on the incrementally extracted anchor point and the data partition matching the anchor point, accessing the heterogeneous data source based on the framework of pre-configured steps, conversion rules and parameter settings to obtain the heterogeneous data assets to be processed for processing, so as to build a heterogeneous data warehouse.

Images