Data warehouse construction method and device, equipment, storage medium and program product

Abstract

The invention relates to a data warehouse construction method and device, equipment, a storage medium and a program product. Comprising the following steps: accessing various data sources through a big data processing and distribution system, configuring a data processing component for preprocessing, and writing a data stream into a distributed file system in a structured format to form data of a data preparation area; forming data of a dimension layer based on index information set by a data modeling function of the big data platform and dimension information of each service; cleaning the data in the data preparation area according to the SQL script corresponding to each piece of index information to form data of a data detail layer; generating a summary wide table according to the data of the data detail layer and the dimension layer, and forming the data of the data service layer; and according to the current data demand, extracting the data application table from the data of the data service layer to form the data of the data application layer, and providing services for the outside. By adopting the method, the data warehouse is automatically constructed according to the business requirements, and the technical requirements on users are greatly reduced.

Description

Technical Field

[0001] This application relates to the field of data warehouse technology, and in particular to a data warehouse construction method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Current data warehouse construction involves a series of steps, such as business modeling, domain modeling, logical design, and physical design, and is achieved using specific reporting tools or programs. However, this approach suffers from long lead times, difficulty in meeting increasing real-time requirements, and limited application scenarios. Furthermore, these reporting tools have limited functionality in supporting data access and governance, and often require business and technical personnel to possess advanced SQL (Structured Query Language) skills, thus creating a high technical barrier. Summary of the Invention

[0003] Therefore, it is necessary to provide a data warehouse construction method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems. This method can automatically build a data warehouse according to business needs, significantly reducing the technical requirements for users.

[0004] In a first aspect, this application provides a method for constructing a data warehouse, the data warehouse comprising a data preparation area, a data detail layer, a data service layer, a data application layer, and a dimension layer; the method includes:

[0005] By connecting to multiple data sources through a big data processing and distribution system, configuring data processing components to perform preprocessing operations on multiple data sources, and writing the preprocessed data stream into a distributed file system for storage in a predefined structured data format, the data in the data preparation area is formed.

[0006] Obtain preset indicator information and determine the dimension information corresponding to each business. Configure dimension tables based on indicator information and dimension information to form dimension layer data. The indicator information is set based on the data modeling function of the big data platform.

[0007] Based on the SQL script corresponding to each indicator, the data in the data preparation area is cleaned and processed to form the data detail layer.

[0008] Based on the data from the detail layer and the dimension layer, a summary wide table is generated and output to form the data service layer.

[0009] Based on current data requirements, the corresponding data application tables are extracted from the data service layer to form the data application layer, thereby providing services to the outside world.

[0010] In one embodiment, the method further includes:

[0011] Store the metadata of the data warehouse in a memory-based search engine;

[0012] In response to a metadata viewing command, the system searches for the target metadata corresponding to the command using a search engine and generates view content based on the target metadata for display.

[0013] In one embodiment, the method further includes:

[0014] Receive change instructions for the data warehouse through a visual interface;

[0015] In response to a change order, update the data table structure or data in the data layer pointed to by the change order in the data warehouse.

[0016] In one embodiment, updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse includes:

[0017] Generate data processing code based on the change instructions;

[0018] Execute data processing code to update the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0019] In one embodiment, the method further includes:

[0020] When data is processed and loaded into any target data layer of the data warehouse, extract data samples and corresponding data quality metrics.

[0021] Data samples and data quality metrics are input into a pre-trained anomaly detection model to identify anomaly patterns and degrees of anomaly.

[0022] Determine the corresponding exception handling strategy based on the exception pattern and the degree of exception;

[0023] Implement exception handling strategies.

[0024] In one embodiment, the indicator information includes atomic indicators, derived indicators, and composite indicators.

[0025] Secondly, this application also provides a data warehouse construction apparatus, wherein the data warehouse includes a data preparation area, a data detail layer, a data service layer, a data application layer, and a dimension layer; the apparatus includes:

[0026] The preparation area construction module is used to access multiple data sources through the big data processing and distribution system, configure data processing components to perform preprocessing operations on multiple data sources, and write the preprocessed data stream into a distributed file system for storage in a predefined structured data format to form the data in the data preparation area.

[0027] The dimension layer construction module is used to acquire preset indicator information, determine the dimension information corresponding to each business, configure the dimension table according to the indicator information and dimension information, and form the data of the dimension layer; the indicator information is set based on the data modeling function of the big data platform.

[0028] The detail layer construction module is used to clean and process the data in the data preparation area according to the SQL script corresponding to each indicator information, forming the data detail layer;

[0029] The service layer construction module is used to generate a summary wide table for output based on the data from the data detail layer and the data from the dimension layer, forming the data service layer data;

[0030] The application layer construction module is used to extract the corresponding data application table from the data service layer according to the current data requirements, forming the data application layer data, thereby providing services to the outside world.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0034] The aforementioned data warehouse construction methods, devices, computer equipment, computer-readable storage media, and computer program products access multiple data sources through a big data processing and distribution system. Data processing components are configured to preprocess these data sources, and the preprocessed data streams are written to a distributed file system in a predefined structured data format for storage, forming the data preparation area. Pre-defined indicator information is acquired, and dimension information corresponding to each business is determined. Dimension tables are configured based on the indicator and dimension information, forming the dimension layer data. The indicator information is set based on the data modeling function of the big data platform. The data in the data preparation area is cleaned using SQL scripts corresponding to each indicator, forming the data detail layer data. A summary wide table is generated and output based on the data detail layer data and the dimension layer data, forming the data service layer data. Based on current data requirements, corresponding data application tables are extracted from the data service layer data, forming the data application layer data, thereby providing services externally. By combining dimensional modeling principles with the powerful modeling capabilities of big data platforms, and taking a business-oriented approach, a series of indicators that accurately reflect actual business scenarios are predefined. This helps to fundamentally solve problems such as ambiguous business definitions and inconsistent field standards, avoiding repeated ETL (Extract, Transform, Load) processing and thus improving development efficiency. It can automatically build data warehouses based on business needs, significantly reducing the technical requirements for users and enabling business personnel without technical backgrounds to easily participate in the construction and management of data warehouses. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a data warehouse construction method in one embodiment;

[0037] Figure 2 This is a flowchart illustrating a data warehouse construction method in another embodiment;

[0038] Figure 3 This is a flowchart illustrating the data association steps in one embodiment;

[0039] Figure 4 This is a structural block diagram of a data warehouse construction device in one embodiment;

[0040] Figure 5This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0043] In one exemplary embodiment, such as Figure 1 As shown, a data warehouse construction method is provided. Taking the application of this method to a server as an example, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The data warehouse includes a data preparation area, a data detail layer, a data service layer, a data application layer, and a dimension layer; the method includes:

[0044] Step 102: Access multiple data sources through the big data processing and distribution system, configure the data processing component to perform preprocessing operations on the multiple data sources, and write the preprocessed data stream into the distributed file system for storage in a predefined structured data format to form the data in the data preparation area.

[0045] The data warehouse consists of several layers: Operation Data Store (ODS), also known as the source layer, receives data from various data sources after extraction, cleaning, and transmission. Data Warehouse Details (DWD) acts as an isolation layer between the business data layer and the raw data layer. It typically maintains the same data granularity as the ODS layer and primarily performs cleaning and normalization operations on the data in the ODS layer, such as removing empty, dirty, and outlier data. Data Warehouse Service (DWS) is based on the foundational data in the DWD layer and, according to specific business needs (e.g., users, orders), generates wide tables with numerous fields and a clear business focus. These tables are used to provide subsequent business queries, such as OLAP (Online Analytical Processing) analysis and data distribution. The Application Data Service (ADS) layer primarily provides data for data products and data analysis services. It can be stored in high-real-time systems such as Elasticsearch and Redis for online use, or in Hive for data analysis and data mining services. The Dimension Layer (DIM) typically configures data tables, such as user profiles and product profiles, to describe specific transactions based on business needs and stores them in this layer.

[0046] Optionally, the Apache NIFI big data processing and distribution system technology can be used to access multiple data sources, and components and information flows can be configured to realize basic data cleaning and access functions. The data can be accessed and stored in a structured manner in the distributed file system (HDFS) to realize the ODS layer in the data warehouse.

[0047] Step 104: Obtain the preset indicator information and determine the dimension information corresponding to each business. Configure the dimension table according to the indicator information and dimension information to form the data of the dimension layer. The indicator information is set based on the data modeling function of the big data platform.

[0048] The data modeling function based on the big data platform allows business personnel to configure relevant indicator information based on specific business processes. This information describes the attributes of the business scenario and clarifies business definitions and field standards. Based on the indicator information in the pre-set indicator system, specific dimensions are defined according to the business requirements, and dimension tables are configured to construct the DIM layer.

[0049] In one exemplary embodiment, the indicator information includes atomic indicators, derived indicators, and composite indicators.

[0050] Atomic metrics are indivisible measures with clear business meaning based on a specific business process; they can be understood as the smallest unit for measuring that process. Atomic metrics are obtained through rule-based analysis and used to measure specified targets. They can also be used for aggregation operations. Derived metrics are instantiations of atomic metrics in specific business scenarios, measuring business activities and providing statistical analysis of certain states of entity objects. Composite metrics are derived metrics based on atomic and derived metrics, and can be composed of multiple atomic or derived metrics combined through specific operational rules. This approach, combining dimensional modeling principles with the powerful modeling capabilities of big data platforms, predefines a series of attributes that accurately reflect actual business scenarios, including atomic, derived, and composite metrics, guided by specific business needs. These metrics represent a high degree of abstraction of statistical logic and algorithms, helping to fundamentally solve problems such as ambiguous business definitions and inconsistent field standards.

[0051] Step 106: Clean the data in the data preparation area according to the SQL script corresponding to each indicator information to form the data detail layer.

[0052] Based on the existing data specifications implemented by atomic indicators, derived indicators, and composite indicators, SQL scripts are written to perform data cleaning and processing on the data in the ODS layer, generating DWD layer data.

[0053] Step 108: Based on the data from the data detail layer and the data from the dimension layer, generate a summary wide table for output, forming the data for the data service layer.

[0054] Specifically, based on the DWD layer data and DIM layer data, a summary wide table is generated and output to form the DWS layer data.

[0055] Step 110: Based on the current data requirements, extract the corresponding data application table from the data service layer to form the data application layer, thereby providing services to the outside world.

[0056] Specifically, based on the specific data analysis and data mining needs, specific data application tables are extracted from the DWS layer data to form ADS layer data for external service.

[0057] In the aforementioned data warehouse construction method, multiple data sources are accessed through a big data processing and distribution system. Data processing components are configured to preprocess these data sources, and the preprocessed data streams are written to a distributed file system in a predefined structured data format for storage, forming the data preparation area. Pre-defined indicator information is acquired, and the dimension information corresponding to each business is determined. Dimension tables are configured based on the indicator and dimension information to form the dimension layer data. The indicator information is set based on the data modeling function of the big data platform. The data in the data preparation area is cleaned using SQL scripts corresponding to each indicator, forming the data detail layer data. A summary wide table is generated and output based on the data detail layer and dimension layer data, forming the data service layer data. Based on current data requirements, the corresponding data application table is extracted from the data service layer data, forming the data application layer data, thus providing services externally. By combining the principles of dimensional modeling with the powerful modeling capabilities of the big data platform, and guided by specific business needs, a series of indicator information that accurately reflects actual business scenarios is predefined. This helps to fundamentally solve problems such as ambiguous business definitions and inconsistent field standards, avoiding repeated ETL processing and thus improving development efficiency. It can automatically build data warehouses according to business needs, which greatly reduces the technical requirements for users and enables business personnel without technical backgrounds to easily participate in the construction and management of data warehouses.

[0058] In an exemplary embodiment, the method further includes: storing metadata of the data warehouse in a memory-based search engine; responding to a metadata viewing instruction, searching for the target metadata corresponding to the metadata viewing instruction through the search engine, and generating view content based on the target metadata for display.

[0059] Understandably, when building a data warehouse, if there is a lack of effective management of business metadata and its association with dimensional data, additional metadata organization will be required in subsequent data analysis and mining work. This not only increases the workload but also leads to redundancy in the process.

[0060] To address this issue, this embodiment associates and stores the generated indicator system with the metadata of the data warehouse table, providing a quick view for subsequent business operations and increasing the robustness of the data.

[0061] Metadata viewing commands can be understood as operation commands or query statements used to search for relevant metadata from a search engine. In specific implementations, the specific metadata of the data warehouse is stored in a memory-based search engine (such as Elasticsearch) for users to search and view, providing a quick way to view metadata. View content refers to the metadata found through visual interfaces. Optionally, in response to metadata viewing commands, the data mapping system retrieves the corresponding metadata from the search engine and displays it in the form of a data map. Optionally, the data reporting system retrieves the corresponding metadata from the search engine and displays it in the form of data reports.

[0062] In an exemplary embodiment, the method further includes: receiving a change instruction for a data warehouse through a visual interface; and updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse in response to the change instruction.

[0063] Understandably, after the data warehouse is built, business users can leverage the dimensional modeling capabilities provided by the big data platform to modify the table structure or data at each layer of the data warehouse through a user-friendly visual interface, without needing to write complex SQL scripts. On the page, business users can define table information, including the data warehouse layer information to which the table belongs and business classification information. On the field page, users can associate indicator systems with fields according to business needs, achieving standardized field constraints. This not only promotes collaboration between the product and development teams and reduces communication costs, but also simplifies the data warehouse maintenance process and lowers the overall implementation cost.

[0064] In scenarios involving business adjustments, business personnel can use this function to input change instructions, and the computer equipment can quickly generate standardized tables based on the change instructions and enter them into the specified data warehouse layer. This achieves decoupling of business and technology and increases the flexibility of the data warehouse construction process.

[0065] In an exemplary embodiment, updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse includes: generating data processing code according to the change instruction; and executing the data processing code to complete the update of the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0066] The change instructions include table structure change instructions and data change instructions, which are used to modify the structure or data of the specified data layer's table, including adding, modifying, or deleting data. In response to the change instructions, the computer automatically generates corresponding data definition language and data processing logic code, and executes the generated code to automatically update the structure and data of the corresponding data layer's table in the data warehouse.

[0067] In an exemplary embodiment, the method further includes: extracting data samples and corresponding data quality indicators when data is processed and loaded into any target data layer of the data warehouse; inputting the data samples and data quality indicators into a pre-trained anomaly detection model to identify anomaly patterns and anomaly severity; determining corresponding anomaly handling strategies based on the anomaly patterns and anomaly severity; and executing the anomaly handling strategies.

[0068] Specifically, data flow is monitored, and data samples and corresponding data quality indicators are extracted in real-time or near real-time as data is processed and loaded into any target data layer of the data warehouse. These data samples and indicators are then input into a pre-trained anomaly detection model to automatically identify anomaly patterns and their severity. Anomaly patterns include, but are not limited to, outliers, format errors, out-of-bounds enumeration values, and logical contradictions. Dynamic responses are then implemented based on the anomaly patterns and severity, triggering predefined anomaly handling strategies. These strategies include, but are not limited to, sending alerts to data managers, logging the abnormal data, or executing pre-configured automated repair scripts to correct the data.

[0069] In one exemplary embodiment, refer to Figure 2 The data warehouse construction method includes the following steps:

[0070] 1. For user business data, use Apache NIFI technology to access multiple data sources, and configure components and information flows to realize basic data cleaning and access functions, and access the data in HDFS in a structured way to generate ODS layer data.

[0071] 2. Based on the data modeling function of the big data platform, business personnel can configure relevant atomic indicators, derived indicators and composite indicators based on specific business processes to form an indicator system.

[0072] 3. Based on the existing indicator system, define specific dimensions according to business requirements, configure dimension tables, and construct the DIM layer. Taking the user dimension as an example, a user dimension table can contain dimension information describing a user, such as name, gender, phone number, and associated tenant information, and use the indicator system to standardize these dimensions.

[0073] 4. Based on the existing data specifications implemented by atomic indicators, derived indicators, and composite indicators, write SQL scripts to clean the data in the ODS layer, generate DWD layer data, and realize the modeling of the DWD layer.

[0074] 5. Based on the DWD layer data and DIM layer data, a summary wide table is generated for output, forming the DWS layer data and realizing the construction of the DWS layer.

[0075] 6. Based on specific data analysis and data mining needs, extract specific data application tables from the DWS layer data to form ADS layer data for external service.

[0076] In one exemplary embodiment, refer to Figure 3 Based on the metadata of the data warehouse and the metrics system, a connection is established. This involves associating the metrics system with the table metadata of the data warehouse, and with the field metadata of the data warehouse, to obtain comprehensive metadata. This comprehensive metadata is then stored in an in-memory Elasticsearch instance, providing users with a quick view for their business needs. For example, table and field metadata information is associated with the information table stored in the metrics system via SQL scripts, using the unique fields of atomic or derived metrics, to generate a comprehensive metadata information table containing table metadata information, field metadata information, and metric information.

[0077] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0078] Based on the same inventive concept, this application also provides a data warehouse construction apparatus for implementing the data warehouse construction method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data warehouse construction apparatus embodiments provided below can be found in the limitations of the data warehouse construction method described above, and will not be repeated here.

[0079] In one exemplary embodiment, such as Figure 4 As shown, a data warehouse construction apparatus is provided. The data warehouse includes a data preparation area, a data detail layer, a data service layer, a data application layer, and a dimension layer; the apparatus includes:

[0080] The preparation area construction module 402 is used to access multiple data sources through the big data processing and distribution system, configure data processing components to perform preprocessing operations on multiple data sources, and write the preprocessed data stream into a distributed file system for storage in a predefined structured data format to form the data in the data preparation area.

[0081] The dimension layer construction module 404 is used to obtain preset indicator information, determine the dimension information corresponding to each business, configure the dimension table according to the indicator information and dimension information, and form the data of the dimension layer; the indicator information is set based on the data modeling function of the big data platform.

[0082] The detail layer construction module 406 is used to clean and process the data in the data preparation area according to the SQL script corresponding to each indicator information, so as to form the data detail layer.

[0083] The service layer construction module 408 is used to generate a summary wide table based on the data from the data detail layer and the data from the dimension layer, and output it to form the data service layer.

[0084] The application layer construction module 410 is used to extract the corresponding data application table from the data service layer according to the current data requirements, form the data application layer data, and thus provide services to the outside world.

[0085] The aforementioned data warehouse construction device combines dimensional modeling principles with the powerful modeling capabilities of a big data platform. Guided by specific business needs, it pre-defines a series of metrics that accurately reflect actual business scenarios. This helps fundamentally solve problems such as ambiguous business definitions and inconsistent field standards, avoiding redundant ETL processing and thus improving development efficiency. It can automatically build data warehouses based on business requirements, significantly reducing the technical requirements for users and enabling business personnel without technical backgrounds to easily participate in the construction and management of data warehouses.

[0086] In an exemplary embodiment, the apparatus further includes a metadata processing module, which is used to store the metadata of the data warehouse into a memory-based search engine; in response to a metadata viewing instruction, search for the target metadata corresponding to the metadata viewing instruction through the search engine, and generate view content based on the target metadata for display.

[0087] In an exemplary embodiment, the apparatus further includes an update module, which is configured to receive a change instruction for the data warehouse through a visual interface; and in response to the change instruction, update the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0088] In an exemplary embodiment, the update module is further configured to generate data processing code according to the change instruction; and execute the data processing code to complete the update of the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0089] In an exemplary embodiment, the apparatus further includes an anomaly handling module, which is configured to extract data samples and corresponding data quality indicators when data is processed and loaded into any target data layer of the data warehouse; input the data samples and data quality indicators into a pre-trained anomaly detection model to identify anomaly patterns and anomaly severity; determine corresponding anomaly handling strategies based on anomaly patterns and anomaly severity; and execute the anomaly handling strategies.

[0090] In one exemplary embodiment, the indicator information includes atomic indicators, derived indicators, and composite indicators.

[0091] Each module in the aforementioned data warehouse construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0092] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data warehouse construction method.

[0093] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0094] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: accessing multiple data sources through a big data processing and distribution system; configuring a data processing component to preprocess the multiple data sources; and writing the preprocessed data stream into a distributed file system for storage in a predefined structured data format to form data in a data preparation area; acquiring preset indicator information and determining the dimension information corresponding to each business; configuring a dimension table based on the indicator information and dimension information to form data in the dimension layer; the indicator information is set based on the data modeling function of a big data platform; cleaning the data in the data preparation area according to the SQL script corresponding to each indicator information to form data in the data detail layer; generating a summary wide table for output based on the data detail layer data and the dimension layer data to form data in the data service layer; and extracting the corresponding data application table from the data in the data service layer according to the current data requirements to form data in the data application layer, thereby providing services externally.

[0095] In one embodiment, when the processor executes the computer program, it further performs the following steps: storing metadata of the data warehouse into a memory-based search engine; responding to a metadata viewing instruction, searching for the target metadata corresponding to the metadata viewing instruction through the search engine, and generating and displaying view content based on the target metadata.

[0096] In one embodiment, when the processor executes the computer program, it further performs the following steps: receiving a change instruction for the data warehouse through a visual interface; and updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse in response to the change instruction.

[0097] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating data processing code according to the change instruction; and executing the data processing code to update the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0098] In one embodiment, when the processor executes the computer program, it further performs the following steps: when data is processed and loaded into any target data layer of the data warehouse, extracting data samples and corresponding data quality indicators; inputting the data samples and data quality indicators into a pre-trained anomaly detection model to identify anomaly patterns and anomaly severity; determining corresponding anomaly handling strategies based on anomaly patterns and anomaly severity; and executing the anomaly handling strategies.

[0099] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: accessing multiple data sources through a big data processing and distribution system; configuring data processing components to preprocess the multiple data sources; and writing the preprocessed data stream into a distributed file system for storage in a predefined structured data format to form data in the data preparation area; acquiring preset indicator information and determining the dimension information corresponding to each business; configuring dimension tables based on the indicator information and dimension information to form dimension layer data; the indicator information is set based on the data modeling function of the big data platform; cleaning the data in the data preparation area according to the SQL script corresponding to each indicator information to form data in the data detail layer; generating a summary wide table for output based on the data detail layer data and the dimension layer data to form data in the data service layer; and extracting the corresponding data application table from the data service layer data according to the current data requirements to form data in the data application layer, thereby providing services externally.

[0100] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: storing the metadata of the data warehouse into a memory-based search engine; in response to a metadata viewing instruction, searching for the target metadata corresponding to the metadata viewing instruction through the search engine, and generating and displaying view content based on the target metadata.

[0101] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving a change instruction for the data warehouse through a visual interface; and updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse in response to the change instruction.

[0102] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating data processing code according to the change instruction; and executing the data processing code to update the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0103] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when data is processed and loaded into any target data layer of the data warehouse, extracting data samples and corresponding data quality indicators; inputting the data samples and data quality indicators into a pre-trained anomaly detection model to identify anomaly patterns and anomaly severity; determining corresponding anomaly handling strategies based on anomaly patterns and anomaly severity; and executing the anomaly handling strategies.

[0104] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: accessing multiple data sources through a big data processing and distribution system; configuring data processing components to preprocess the multiple data sources; and writing the preprocessed data stream into a distributed file system for storage in a predefined structured data format to form data in a data preparation area; acquiring preset indicator information and determining the dimension information corresponding to each business; configuring dimension tables based on the indicator information and dimension information to form dimension layer data; the indicator information is set based on the data modeling function of the big data platform; cleaning the data in the data preparation area according to the SQL script corresponding to each indicator information to form data in a data detail layer; generating a summary wide table for output based on the data detail layer data and the dimension layer data to form data in a data service layer; and extracting the corresponding data application table from the data service layer data according to current data requirements to form data in a data application layer, thereby providing services externally.

[0105] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: storing the metadata of the data warehouse into a memory-based search engine; in response to a metadata viewing instruction, searching for the target metadata corresponding to the metadata viewing instruction through the search engine, and generating and displaying view content based on the target metadata.

[0106] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving a change instruction for the data warehouse through a visual interface; and updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse in response to the change instruction.

[0107] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating data processing code according to the change instruction; and executing the data processing code to update the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

[0108] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when data is processed and loaded into any target data layer of the data warehouse, extracting data samples and corresponding data quality indicators; inputting the data samples and data quality indicators into a pre-trained anomaly detection model to identify anomaly patterns and anomaly severity; determining corresponding anomaly handling strategies based on anomaly patterns and anomaly severity; and executing the anomaly handling strategies.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of data warehouse construction, characterized by, The data warehouse comprises a data preparation area, a data detail layer, a data service layer, a data application layer and a dimension layer; the method comprises: Accessing multiple data sources through a big data processing and distribution system, configuring a data processing component to perform preprocessing operations on the multiple data sources, and writing the preprocessed data stream in a predefined structured data format to a distributed file system for storage, forming data of the data preparation area; Obtaining preset index information and determining dimension information corresponding to each business, configuring a dimension table according to the index information and the dimension information, and forming data of the dimension layer; the index information is set based on a data modeling function of a big data platform; According to the SQL script corresponding to each index information, the data of the data preparation area is cleaned and processed to form the data of the data detail layer; According to the data of the data detail layer and the data of the dimension layer, a summary wide table is generated for output, forming data of the data service layer; According to the current data requirement, the corresponding data application table is extracted from the data of the data service layer to form the data of the data application layer, thereby providing services externally.

2. The method of claim 1, wherein, The method further comprises: Storing metadata of the data warehouse into a memory-based search engine; In response to a metadata viewing instruction, the target metadata corresponding to the metadata viewing instruction is found through the search engine, and view content is generated according to the target metadata for display.

3. The method of claim 1, wherein, The method further comprises: Receiving a change instruction for the data warehouse through a visual interface; In response to the change instruction, updating the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

4. The method of claim 3, wherein, The updating of the data table structure or data of the data layer pointed to by the change instruction in the data warehouse comprises: Generating a data processing code according to the change instruction; Executing the data processing code to complete the updating of the data table structure or data of the data layer pointed to by the change instruction in the data warehouse.

5. The method of claim 1, wherein, The method further comprises: Extracting a data sample and a corresponding data quality index when data is processed and loaded into any target data layer of the data warehouse; Inputting the data sample and the data quality index into a pre-trained anomaly detection model to identify an abnormal pattern and an abnormal degree; According to the abnormal pattern and the abnormal degree, determining a corresponding abnormal processing strategy; Executing the abnormal processing strategy.

6. The method according to any one of claims 1 to 5, characterized in that, The index information comprises atomic indexes, derived indexes and composite indexes.

7. A data warehouse construction apparatus characterized by comprising: The data warehouse comprises a data preparation area, a data detail layer, a data service layer, a data application layer and a dimension layer; the apparatus comprises: A preparation area construction module configured to access multiple data sources through a big data processing and distribution system, configure a data processing component to perform preprocessing operations on the multiple data sources, and write the preprocessed data stream in a predefined structured data format to a distributed file system for storage, forming data of the data preparation area; A dimension layer construction module is configured to acquire preset index information, determine dimension information corresponding to each service, configure a dimension table according to the index information and the dimension information, and form data of the dimension layer; A detail layer construction module is configured to perform cleaning processing on data in the data preparation area according to a SQL script corresponding to each index information, and form data of the data detail layer; A service layer construction module is configured to generate a summary wide table according to data of the data detail layer and data of the dimension layer, output the summary wide table, and form data of the data service layer; An application layer construction module is configured to extract a corresponding data application table from the data of the data service layer according to a current data requirement, form data of the data application layer, and provide services to the outside.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images