A method and system for capturing and processing change data in a multi-layered data warehouse

By adopting a change data capture method for multi-layered data warehouses based on dynamic log streams and a distributed processing framework, the problems of insufficient data capture efficiency and accuracy in multi-layered data warehouses are solved. This method achieves efficient data synchronization and parallel processing of computing tasks, thereby improving the versatility and accuracy of data capture.

CN117493469BActive Publication Date: 2026-11-10CHENGDU UNION BIG DATA TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311455328.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-03
Publication Date
2026-11-10
Estimated Expiration
2043-11-03

AI Technical Summary

Technical Problem

Existing data change capture technologies are insufficient to meet the business scenario requirements of data capture efficiency and accuracy in multi-tiered data warehouses, especially in the field of layered computing and interdependent data storage. Simple full or incremental backup strategies are insufficient to achieve complete data capture.

Method used

By employing dynamic log stream-based data change capture technology and combining it with a distributed processing framework, the data warehouse is structurally divided and a basic layer is set. Preprocessing operators are constructed, and a parallel computing approach is used to capture and process change data, thereby achieving data synchronization between the source and target ends and parallel computing of computational tasks.

Benefits of technology

It improves the efficiency of data change capture tasks in multi-layered data warehouses, reduces computing costs, enhances the versatility and accuracy of data capture, and meets the needs of data analysis scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117493469B_ABST
    Figure CN117493469B_ABST
Patent Text Reader

Abstract

This invention provides a method and system for capturing and processing change data in a multi-layered data warehouse, relating to the field of change data capture technology. The method flow is as follows: Change data from the source end is synchronized to the target end's data warehouse based on a dynamic log stream from the source end; the target end's data warehouse is structurally partitioned and a basic layer is set based on business requirements; after structural partitioning and basic layer setting, a distributed processing framework is used to construct preprocessing operators for the basic layer; after constructing the preprocessing operators, parallel computing is used to capture and process the change data synchronized to the data warehouse. This invention, based on log stream data change capture technology and the integrated stream-batch computing capability of a distributed processing framework, achieves synchronization of source end data and layered data in the data warehouse, computation of the data warehouse's basic layer data, and parallel computation of computational tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of change data capture and processing technology, and more specifically, to a method and system for capturing and processing change data in a multi-layered data warehouse. Background Technology

[0002] With the continuous development and upgrading of big data technology, data warehouses used in data analysis scenarios have also adopted more detailed and complete layered designs. How to collect various source data into the data warehouse and capture data changes in real time has always been a crucial issue of concern in the industry. Currently, mainstream data change capture (CDC) technologies mainly include two categories: query-based and log-based, both playing an important role in handling data collection tasks for various relational and non-relational databases. However, for data warehouses, which have multiple layers involving layered computation and interdependencies, simple full or incremental backup CDC strategies are insufficient to provide complete support, making the data capture efficiency and accuracy unable to meet the needs of data analysis scenarios. Summary of the Invention

[0003] This invention provides a method and system for capturing and processing change data in a multi-layered data warehouse, which solves the problem that the data capture efficiency and accuracy of existing data change capture technologies cannot meet the needs of data analysis business scenarios.

[0004] In a first aspect, embodiments of the present invention provide a method for capturing and processing change data in a multi-layered data warehouse, the method flow of which is as follows:

[0005] Dynamic log streams from the source end synchronize changed data from the source end to the data warehouse at the target end.

[0006] Based on business requirements, the data warehouse of the target end is structurally divided and the basic layer is set;

[0007] After structural division and basic layer setting, a distributed processing framework is used to construct preprocessing operators for the basic layer;

[0008] After the preprocessing operators are constructed, parallel computing is used to capture and process the changed data synchronized to the data warehouse.

[0009] In the above embodiments, based on the log stream data change capture technology and the integrated stream-batch computing capability of the distributed processing framework, the synchronization of source data and data warehouse layered data, the computation of basic data in the data warehouse, and the parallel computation of computing tasks are realized.

[0010] As one of the optional implementations of this application, before synchronizing the changed data from the source end to the data warehouse of the target end based on the dynamic log stream from the source end, the source end log status needs to be enabled.

[0011] In the above embodiments, when the source log status is enabled, the source dynamic log stream can be obtained normally. If the log status is abnormal, it will directly affect the data synchronization result.

[0012] As some optional implementations of this application, the process of synchronizing changed data from the source end to the data warehouse of the target end based on the dynamic log stream of the source end is as follows:

[0013] Based on the dynamic log stream at the source, data change capture technology is used to obtain the changed data at the source and the structural information of the changed data at the source.

[0014] Based on the structural information of the source change data, a preliminary data mapping structure between the source and target data warehouses is constructed, and the source change data is used as the basic data of the target data warehouse.

[0015] The data processing mode is selected based on the data type of the basic data in the target data warehouse. The data processing modes include streaming synchronization mode, incremental synchronization mode and full synchronization mode.

[0016] Basic data synchronization is performed based on the data mapping structure and data processing mode of the source and target data warehouses.

[0017] In the above embodiments, by establishing a mapping relationship between the various layers of the source and target data warehouses, data synchronization between the source and target can be achieved; at the same time, the synchronization mode can be switched based on the data type, thereby enabling the reading of basic data.

[0018] As some optional implementation methods of this application, the process of structurally dividing the target data warehouse and setting the basic layer based on business requirements is as follows:

[0019] Based on business needs, the structure of the target data warehouse is divided into the source layer, data detail layer, data service layer, data dimension layer, and data application layer.

[0020] Select several layers as the base layers from the source layer, data detail layer, data service layer, data dimension layer, and data application layer;

[0021] Configure the storage structure of the base layer, and ultimately construct the data mapping structure of the target data warehouse based on the base layer.

[0022] In the above embodiments, considering the specific logical complexity of the specific business domains in the data warehouse layering, the basic layer with relatively low business logic complexity in the data warehouse layering was selected to extend the data change capture function, which increases versatility and reduces the additional computational cost introduced in data change capture.

[0023] As some optional implementations of this application, the process of constructing preprocessing operators for the base layer using a distributed processing framework is as follows:

[0024] A distributed processing framework is used to construct preprocessing operators for the base layer;

[0025] The preprocessing operators are stored through the storage structure of the base layer, which includes a source layer, a data detail layer, and a data dimension layer.

[0026] As some optional implementations of this application, the data detail layer and the data dimension layer are respectively based on several preprocessing operators to generate a data cleaning calculation module and a data dimension classification module.

[0027] In the above embodiments, based on the traditional source-layer data change capture, a distributed computing module is introduced for different layers of the data warehouse, which expands the functional scope of data warehouse data change capture and processing.

[0028] As some optional implementations of this application, the process of capturing and processing changed data synchronized to the data warehouse using parallel computing is as follows:

[0029] The source layer is used to synchronize the changed data from the source end and to persistently output the changed data.

[0030] The data cleaning and calculation module of the data detail layer is used to clean and calculate the changed data at the source end, and the cleaned and calculated changed data is persistently output.

[0031] The data dimension classification module of the data dimension layer performs statistical calculations on the source change data and persists the calculated change data for output.

[0032] Perform comprehensive validation on the persistent output data of the source layer, data detail layer, and data dimension layer.

[0033] In the above embodiments, the parallel processing structure for data change capture tasks at each layer and the introduction of computing modules at each layer have obvious decomposability. With the powerful parallel scheduling capabilities of the current mainstream distributed processing framework, data change capture tasks of multi-layer data warehouses can be processed synchronously, improving the efficiency of multi-branch data change capture tasks.

[0034] In a second aspect, the present invention provides a multi-layer data warehouse change data capture and processing system, the system comprising:

[0035] A data synchronization unit that synchronizes changed data from the source end to the data warehouse of the target end based on the dynamic log stream from the source end;

[0036] The structure setting unit divides the target data warehouse into structural parts and sets the basic layer based on business requirements.

[0037] The operator construction unit, after structural division and basic layer setting, uses a distributed processing framework to preprocess the basic layer operators for construction.

[0038] The parallel computing unit, after constructing the preprocessing operator, uses parallel computing to capture and process the changed data synchronized to the data warehouse.

[0039] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned multi-layer data warehouse change data capture and processing method.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-layer data warehouse change data capture and processing method.

[0041] The beneficial effects of this invention are as follows:

[0042] (1) This invention realizes data collection and change data synchronization for multiple basic layers of data warehouse: Traditional data warehouse data change capture and collection schemes only perform full data synchronization and change capture for the post source layer, and also require the introduction of message middleware to assist data distribution, resulting in large architectural overhead; This invention, based on the distributed processing framework, designs and introduces distributed computing modules for different layers of data warehouse, expanding the functional scope of data warehouse data change capture and processing, based on the traditional post source layer data change capture. At the same time, considering the specific business domain logic complexity of the data warehouse layer, this invention selects the basic layer with relatively low business logic complexity in the data warehouse layer for data change capture function extension, increasing versatility and reducing the additional computing cost introduced in data change capture.

[0043] (2) The present invention adopts a parallel processing structure for the data change capture task at each layer and the introduction of each layer computing module has obvious decomposability. With the help of the powerful parallel scheduling processing capability of the current mainstream distributed processing framework, the data change capture task of multi-layer data warehouse can be processed synchronously, which improves the efficiency of multi-branch data change capture task. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment described in the embodiments of the present invention;

[0046] Figure 2 This is a flowchart illustrating the steps of the multi-layer data warehouse change data capture and processing method described in this embodiment of the invention.

[0047] Figure 3 This is a flowchart of the multi-layer data warehouse change data capture and processing method according to an embodiment of the present invention;

[0048] Figure 4 This is a structural block diagram of the multi-layer data warehouse change data capture and processing system described in an embodiment of the present invention. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0050] To address the issue that existing data change capture technologies cannot meet the needs of data analysis business scenarios in terms of data capture efficiency and accuracy, this application provides a method and system for capturing and processing change data in a multi-layered data warehouse. Before introducing the specific technical solution of this application, the hardware operating environment involved in the embodiments of this application will be described first.

[0051] Please see Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application.

[0052] like Figure 1As shown, the computer device may include: a processor, such as a central processing unit (CPU), a communication bus, a user interface, a network interface, and memory. The communication bus is used to enable communication between these components. The user interface may include a display screen and an input unit such as a keyboard; optionally, the user interface may also include a standard wired interface or a wireless interface. The network interface may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive; optionally, the memory may also be a storage device independent of the aforementioned processor.

[0053] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] like Figure 1 As shown, a memory, as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a software program storage module.

[0055] exist Figure 1 In the computer device shown, the network interface is mainly used for data communication with the network server; the user interface is mainly used for data interaction with the user; the processor and memory in the computer device of this application can be set in the computer device, and the computer device calls the multi-level data warehouse change data capture and processing system stored in the memory through the processor, and executes the multi-level data warehouse change data capture and processing method provided in the embodiment of this application.

[0056] Based on the hardware environment of the foregoing embodiments, embodiments of this application provide a multi-layered data warehouse change data capture and processing method. Please refer to [link to relevant documentation]. Figure 2 , Figure 3 , Figure 2 This is a flowchart illustrating the steps of the multi-layered data warehouse change data capture and processing method. Figure 3 The flowchart below shows the method for capturing and processing change data in a multi-layered data warehouse.

[0057] (1) Based on the dynamic log stream of the source end, the changed data of the source end is synchronized to the data warehouse of the target end.

[0058] Specifically, Data Change Capture (CDC) techniques typically fall into two main categories: those based on query files and those based on log files. Since log files provide detailed historical changes to all source data, ensuring data consistency and offering advantages such as streaming consumption, this invention employs a dynamic log stream-based CDC technique, using the distributed processing framework (FLINK) as its technological foundation.

[0059] In this embodiment of the invention, before synchronizing the changed data from the source end to the data warehouse of the target end based on the dynamic log stream from the source end, it is necessary to enable the source end log status. When the source end log status is enabled, the dynamic log stream from the source end can be obtained normally. If the log status is abnormal, it will directly affect the result of data synchronization.

[0060] Specifically, the process of synchronizing changed data from the source end to the target data warehouse based on the dynamic log stream from the source end is as follows:

[0061] (1.1) Based on the dynamic log stream of the source end, data change capture technology is used to obtain the source end changed data and the structure information of the source end changed data.

[0062] (1.2) Based on the structural information of the source change data, a preliminary data mapping structure between the source and target data warehouses is constructed, and the source change data is used as the basic data of the target data warehouse.

[0063] (1.3) Select the data processing mode based on the data type of the basic data of the target data warehouse. The data processing mode includes streaming synchronization mode, incremental synchronization mode and full synchronization mode.

[0064] Specifically, for new historical batch data, the EARLYEST OFFSET mode can be selected; for resuming historical batch data, the LATEST OFFSET mode can be selected; and for real-time processed data, the INITIAL streaming mode can be selected, which includes both full and incremental synchronization. Taking real-time processed data as an example, when performing full and incremental synchronization, the log file is first obtained and a full query is performed to obtain all the data in the data table. Then, the log change information stream is obtained in real time, and this log change information stream is used as the basis for materializing the changed data.

[0065] (1.4) Perform basic data synchronization based on the data mapping structure and data processing mode of the source and target data warehouses.

[0066] Specifically, based on the initially constructed data mapping structure between the source and target ends and the log stream, the full query results are used as the basic data for synchronization on the target end based on the distributed processing framework, and incremental change operations are performed. At the same time, since the distributed processing framework has a built-in incremental data synchronization tool, the message middleware can be skipped, and the data can be directly synchronized and captured on the target end.

[0067] (2) Based on business needs, the data warehouse of the target end is structurally divided and the basic layer is set. Compared with ordinary databases, data warehouses are usually not a single data storage structure. A complete data warehouse usually divides the data in it into multiple layers according to different purpose requirements.

[0068] Specifically, the process of structurally partitioning and defining the basic layer of the target data warehouse based on business needs is as follows:

[0069] (2.1) There are various schemes for the layering of data warehouses. Depending on business needs, a more complete five-layer structure or a three-layer structure with larger granularity can be chosen. Preferably, this invention divides the data warehouse into a five-layer structure, including the source layer, data detail layer, data service layer, data dimension layer, and data application layer. ① The main function of the source layer is to back up the basic data captured in the data warehouse. This also means that the basic data in this layer should maintain the same structure as the original data as much as possible without additional calculation processing. ② The main function of the data detail layer is to preprocess the basic data in the source layer, such as completing or cleaning missing values. ③ The main function of the data service layer is to perform operations such as aggregation on the data after preprocessing in the data detail layer. ④ The main function of the data dimension layer is to divide the data into high-dimensional data (such as business data tables with tens of millions or hundreds of millions of data) and low-dimensional data (such as configuration tables with relatively small data) according to the data indicators and dimensions. ⑤ The main function of the data application layer is to combine specific business logic and specific data analysis objectives, and combine all the previous data warehouse layers to realize data analysis and application.

[0070] (2.2) Since each layer of the data warehouse has computational requirements, the data change capture task of the data warehouse cannot simply complete the data change synchronization; it also needs to introduce computational processing capabilities. Introducing complex computational logic after data synchronization would make the entire data synchronization process bloated, violating the original business logic; therefore, it is necessary to determine the basic layer with relatively simple computational logic in the data warehouse as the target; preferably, the basic layer includes the source layer, the data detail layer, and the data dimension layer.

[0071] (2.3) Configure the storage structure of the base layer, and finally construct the data mapping structure between the source and target data warehouses based on the base layer.

[0072] Specifically, the storage structure of each layer is determined based on the specific data volume and business model. For example, the storage of source layer business data can use a traditional relational database, while some other more complex storage can choose a non-relational database. Only after the specific storage structure of each layer in the data warehouse is determined can the data mapping structure between the source and target ends of the data change capture task be finally determined, thereby delineating the specific collection boundaries.

[0073] (3) After the structural division and basic layer setting, the distributed processing framework is used to construct the preprocessing operator for the basic layer. One reason for using the distributed processing framework as the basis of the data change capture technology is that the distributed processing framework allows for custom data processing operators. After clarifying the data mapping structure between the source and target ends and the specific collection boundaries, this invention designs preprocessing operators for different levels of data warehouse business needs in order to complete the processing tasks in each layer of the data warehouse.

[0074] Specifically, the process of constructing preprocessing operators for the base layer using a distributed processing framework is as follows:

[0075] (3.1) A distributed processing framework is used to construct preprocessing operators for the base layer.

[0076] (3.2) The preprocessing operators are stored through the storage structure of the base layer, wherein the base layer includes a source layer, a data detail layer and a data dimension layer.

[0077] Specifically, the data detail layer and the data dimension layer generate a data cleaning calculation module and a data dimension classification module based on several preprocessing operators, respectively.

[0078] The setup steps for the data cleaning and calculation module are as follows:

[0079] ① Construction of basic operators: Basic operators are mainly used for numerical calculations, logical comparisons, concatenation, and truncation of field types. That is, for numeric fields, numerical calculations and logical comparisons can be performed directly; while for other types of fields, such as string types, additional characters need to be introduced for character concatenation and character truncation.

[0080] ② Construction of general cleaning strategy operators. The general cleaning strategy mainly involves processing missing values ​​such as removal, completion, and interpolation.

[0081] ③ Construction of targeted cleaning strategy operators: The targeted cleaning strategy mainly determines the specific outliers and critical values ​​based on business needs. After the initial screening by the general cleaning strategy, missing value data will be cleaned. Outlier data needs to be determined according to the specific situation in targeted cleaning. For example, the age limit for people should be 1-150. Any value outside this range is an outlier and should be deleted.

[0082] Specifically, unlike the data detail layer, the data dimension layer primarily plays a role in dimension classification and processing within the data warehouse. Therefore, the main tasks of the data dimension classification module in the data dimension layer are statistical classification and post-classification persistence. The specific setup steps for the data dimension classification module are as follows:

[0083] ① Determine the dimensional classification boundaries, that is, determine the data volume classification standards and dimensional classification standards based on the specific business dimension volume, such as taking 100 million as the critical point for high-dimensional data volume classification.

[0084] ② Data dimension layer persistence container selection, that is, according to the upstream classification boundary, different dimension data are classified and stored. Different storage schemes are selected according to the high-dimensional and low-dimensional data after classification. In this invention, low-dimensional data is finally persisted using relational databases, and high-dimensional data is finally persisted using non-relational databases.

[0085] ③ Construction of statistical operators and output operators: For the division of data dimensions, this invention uses the statistical operators of the distributed processing framework to perform classification processing according to the set critical points of the dimensions; then, according to the determined persistent storage structure after classification, the output operators are used to output the statistically classified data to the storage structure of the final data dimension layer.

[0086] (4) After the preprocessing operator is constructed, the change data captured by the data warehouse is processed by parallel computing.

[0087] Specifically, the process of capturing and processing changed data synchronized to the data warehouse using parallel computing is as follows:

[0088] (4.1) The source layer is used to synchronize the changed data at the source end and to persistently output the changed data;

[0089] (4.2) The data cleaning and calculation module of the data detail layer is used to clean and calculate the change data at the source end, and the cleaned and calculated change data is persistently output.

[0090] (4.3) The data dimension classification module of the data dimension layer is used to perform statistical calculations on the source change data, and the statistically calculated change data is persistently output.

[0091] (4.4) Perform comprehensive verification on the persistent output data of the source layer, data detail layer and data dimension layer.

[0092] Specifically, after clarifying the hierarchical structure of the data warehouse and designing the preprocessing operators between each layer, this invention adopts a parallel processing approach to initiate the entire data change capture and processing task. The specific steps are as follows:

[0093] ① Determine the specific parallel branches. For example, if the specific business only needs the source layer, data detail layer, and data dimension layer, then the number of parallel branches is set to 3. In actual processing, it is only necessary to simultaneously enable the full data change capture of the source layer, the data synchronization and data cleaning calculation module of the data detail layer, and the data synchronization and data dimension classification module of the data dimension layer.

[0094] ② Submit a data change capture task based on the distributed processing framework and view the results. After a specific synchronous processing task is submitted, the processing log can be viewed in real time through the distributed processing framework, and timely responses can be made to any abnormal situations during the process.

[0095] ③ After the data change capture task is completed, you can check in the data warehouse whether the data change capture results of each layer can meet the specific requirements. The specific general requirements are as follows:

[0096] Source layer: Data should be collected and synchronized in its entirety. The amount of data collected from the source and target ends should be consistent, and there should be no significant loss of longitude or change in content between the collected fields.

[0097] Data detail layer: First, the amount of data collected should be consistent; second, the general cleaning results should ensure that no missing values ​​are generated (or that missing values ​​are appropriately processed according to preset completion / interpolation rules); finally, ensure that there are no obvious outliers in the data.

[0098] Data Dimension Layer: First, the total number of high-dimensional and low-dimensional data after layering should be consistent with the total number of data in the source. At the same time, the data volume of high-dimensional and low-dimensional data should conform to the initially defined filtering rules, and there should be no data volume exceeding the limit.

[0099] In summary, this invention achieves data acquisition and change data synchronization across multiple foundational layers of a data warehouse. Traditional data warehouse change capture schemes only perform full data synchronization and change capture at the source layer, and also require the introduction of message middleware to assist data distribution, resulting in significant architectural overhead. This invention, based on a distributed processing framework, builds upon traditional source layer data change capture by designing and introducing distributed computing modules for different layers of the data warehouse, thus expanding the functional scope of data change capture processing. Furthermore, considering the specific logical complexity of the business domains within the data warehouse layers, this invention selects the foundational layers with relatively low business logic complexity for data change capture function extension, increasing versatility and reducing additional computational costs in data change capture. Because this invention employs a parallel processing structure for each layer's data change capture tasks and introduces computing modules at each layer, it exhibits significant decomposability. Leveraging the powerful parallel scheduling capabilities of current mainstream distributed processing frameworks, it can simultaneously process data change capture tasks across multiple data warehouse layers, improving the efficiency of multi-branch data change capture tasks.

[0100] Furthermore, in one embodiment, based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a multi-layer data warehouse change data capture and processing system, which corresponds one-to-one with the method described in Embodiment 1. Please refer to [link / reference]. Figure 4 , Figure 4 This is a structural block diagram of the multi-layered data warehouse change data capture system, the system comprising:

[0101] A data synchronization unit that synchronizes changed data from the source end to the data warehouse of the target end based on the dynamic log stream from the source end.

[0102] The structure setting unit divides the data warehouse of the target end into structural parts and sets the basic layer based on business requirements.

[0103] The operator construction unit, after structural division and basic layer setting, uses a distributed processing framework to preprocess the basic layer operators.

[0104] The parallel computing unit, after constructing the preprocessing operator, uses parallel computing to capture and process the changed data synchronized to the data warehouse.

[0105] It should be noted that each unit in the multi-layer data warehouse change data capture and processing system in this embodiment corresponds one-to-one with each step in the multi-layer data warehouse change data capture and processing method in the aforementioned embodiment. Therefore, the specific implementation method and the technical effects achieved in this embodiment can be referred to the implementation method of the aforementioned multi-layer data warehouse change data capture and processing method, and will not be repeated here.

[0106] Furthermore, in one embodiment, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the methods in the foregoing embodiments.

[0107] In addition, in one embodiment, this application also provides a computer storage medium storing a computer program that is executed by a processor to implement the methods described in the foregoing embodiments.

[0108] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.

[0109] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0110] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0111] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0112] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0113] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0115] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for capturing and processing change data in a multi-layered data warehouse, characterized in that, The method flow is as follows: Dynamic log streams from the source end synchronize changed data from the source end to the data warehouse at the target end. Based on business requirements, the data warehouse at the target end is structurally divided and its basic layer is defined, including: Based on business needs, the structure of the target data warehouse is divided into the source layer, data detail layer, data service layer, data dimension layer, and data application layer. Select several layers as the base layers from the source layer, data detail layer, data service layer, data dimension layer, and data application layer; Configure the storage structure of the base layer, and ultimately construct the data mapping structure between the source and target data warehouses based on the base layer; After structural partitioning and basic layer definition, a distributed processing framework is used to construct preprocessing operators for the basic layer, including: A distributed processing framework is used to construct preprocessing operators for the base layer; The preprocessing operators are stored through the storage structure of the base layer, which includes a source layer, a data detail layer, and a data dimension layer. The data detail layer and the data dimension layer are respectively based on several preprocessing operators to generate a data cleaning calculation module and a data dimension classification module; After the preprocessing operators are constructed, parallel computing is used to capture and process the changed data synchronized to the data warehouse, including: The source layer is used to synchronize the changed data from the source end and to persistently output the changed data. The data cleaning and calculation module of the data detail layer is used to clean and calculate the changed data at the source end, and the cleaned and calculated changed data is persistently output. The data dimension classification module of the data dimension layer performs statistical calculations on the source change data and persists the calculated change data for output. Perform comprehensive validation on the persistent output data of the source layer, data detail layer, and data dimension layer.

2. The method for capturing and processing change data in a multi-layer data warehouse according to claim 1, characterized in that, Before synchronizing changed data from the source end to the target data warehouse using a dynamic log stream based on the source end, the source end log status needs to be enabled.

3. The method for capturing and processing change data in a multi-layer data warehouse according to claim 1, characterized in that, The process of synchronizing changed data from the source to the target data warehouse based on the source's dynamic log stream is as follows: Based on the dynamic log stream at the source, data change capture technology is used to obtain the changed data at the source and the structural information of the changed data at the source. Based on the structural information of the source change data, a preliminary data mapping structure between the source and target data warehouses is constructed, and the source change data is used as the basic data of the target data warehouse. The data processing mode is selected based on the data type of the basic data in the target data warehouse. The data processing modes include streaming synchronization mode, incremental synchronization mode and full synchronization mode. Basic data synchronization is performed based on the data mapping structure and data processing mode of the source and target data warehouses.

4. A multi-layer data warehouse change data capture and processing system for implementing the method of claim 1, characterized in that, The system includes: A data synchronization unit that synchronizes changed data from the source end to the data warehouse of the target end based on the dynamic log stream from the source end; The structure setting unit divides the target data warehouse into structural parts and sets the basic layer based on business requirements. The operator construction unit, after structural division and basic layer setting, uses a distributed processing framework to preprocess the basic layer operators for construction. The parallel computing unit, after the preprocessing operator is constructed, uses parallel computing to capture and process the changed data synchronized to the data warehouse.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes a computer program, it implements the multi-layer data warehouse change data capture and processing method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-layer data warehouse change data capture and processing method according to any one of claims 1-3.

Citation Information

Patent Citations

  • Multi-warehouse construction method and system based on streaming processing

    CN116795816A