Real-time Data Warehouse Construction Method, Query Method and System Based on Asynchronous Materialized Views

Abstract

The present invention provides a method, query method and system for constructing a real-time data warehouse based on asynchronous materialized views, including: obtaining database logs, using data import technology to import the database logs into a preset message queue, and then importing them into a Doris database and storing them in the raw data layer to generate raw data tables; using Structured Query Language to create a first materialized view, storing the calculation results obtained by querying the first target Structured Query Language from the raw data tables into the first materialized view to obtain a primary materialized view, serving as the detailed data layer, and generating detailed data tables; using Structured Query Language to create a second materialized view, storing the calculation results obtained by querying the second target Structured Query Language from the detailed data tables into the second materialized view to obtain a secondary materialized view, serving as the data service layer, and generating data service tables. The method for constructing a real-time data warehouse based on asynchronous materialized views provided by the present invention reduces the development cost and maintenance cost.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time data warehouses, and in particular to a method, a query method, and a system for constructing a real-time data warehouse based on an asynchronous materialized view. Background Technique

[0002] In the field of big data, a real-time data warehouse (RTDW) is an important data warehouse technology that focuses on ingesting, processing, and analyzing data streams immediately or nearly in real time to provide timely business insights and decision support. It can continuously receive real-time data streams from various sources, such as database transactions, log files, sensors, API calls, etc., and process and analyze them within a short time after the data is generated.

[0003] At present, most real-time data warehouses in the industry use the Lambda architecture or the Kappa architecture, or a special architecture that combines the two. For example, Figure 1 As shown, in the Lambda architecture, some services with high real-time requirements are processed in the form of stream computing, while the offline computing of these services still exists. The goal is to run real-time and batch operations in parallel. Finally, the unified data service layer merges the results and gives them to the front end. Generally, the batch processing results are used as the standard, and the real-time results are mainly for quick response. The emergence of the Kappa architecture is to solve various problems of the Lambda architecture, such as the need to maintain two sets of logics, which results in a relatively high maintenance cost, and the resource usage is also much higher than that of a single set of logic. As Figure 2 shown, in the Kappa architecture, when the data needs to be reprocessed or the data changes, the historical data can be replayed for recalculation. However, this solution also has limitations. The speed of stream computing is not as high as the throughput of batch offline computing. Although it can be alleviated by increasing resources, there is still a long data calculation cycle.

[0004] At present, whether it is the Kappa architecture or the Lambda architecture, there are problems of excessively high development costs in supporting real-time data warehouses. The Lambda architecture needs to maintain two sets of logics. In addition to generating one set of logic for offline reports, it also needs to use a streaming computing engine (such as Flink) to maintain another set for real-time computing tasks. The Kappa architecture uses a streaming computing engine for computing. By ingesting streaming data in full volume, the Kafka middleware stores intermediate data, and then the streaming computing engine is used for computing. The main disadvantage is that the iteration speed of new requirements is slow. If the table structure or logic is frequently changed, full-volume computing needs to be performed frequently, resulting in a high time cost. Moreover, the resource requirements for full-volume computing are relatively higher than those for offline computing, especially for the CPU, memory, and hard disk. If it encounters a business scenario of multi-table association, the requirement for the hard disk needs to be upgraded from a mechanical hard disk (HDD) to a solid-state drive (SSD). Because in the scenario of multi-stream association, the size of the state is constantly expanding. Against the background of reducing the state size, the use of incremental state checkpoints must use RocksDB (an embedded database) as the database for state storage, and the performance of RocksDB depends to a large extent on the storage medium. In addition, the processing of streaming data based on a streaming computing engine (FlinkSQL or DataStream) is naturally more difficult to code than offline development or ordinary Structured Query Language (SQL). Although the current support for streaming computing by FlinkSQL is already very perfect, for data accuracy, a large number of retraction streams under regular joins also need to be processed. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, a query method, and a system for constructing a real-time data warehouse based on an asynchronous materialized view to eliminate or improve one or more defects existing in the prior art.

[0006] On the one hand, the present invention provides a method for constructing a real-time data warehouse based on an asynchronous materialized view, and the method includes the following steps:

[0007] Obtain database logs; use data import technology to import the database logs into a preset message queue;

[0008] Import the data in the message queue into the Doris database and store it in the raw data layer to generate a raw data table;

[0009] Use Structured Query Language to create a first materialized view, and store the calculation result obtained by querying the first target Structured Query Language from the raw data table into the first materialized view to obtain a primary materialized view, and use the primary materialized view as the detailed data layer to generate a detailed data table;

[0010] Create a second materialized view using the structured query language, and store the calculation results obtained by querying the second target structured query language from the detail data table into the second materialized view to obtain a secondary materialized view. Use the secondary materialized view as the data service layer to generate a data service table; the data service table is provided for the business side to query.

[0011] In some embodiments of the present invention, obtaining database logs includes:

[0012] Obtain binary logs or redo logs in a relational database to capture change events in the relational database, where the change events at least include insertions, updates, and deletions.

[0013] In some embodiments of the present invention, using data import technology to import the database logs into a preset message queue includes:

[0014] Use real-time data capture technology implemented based on a preset streaming computing engine to stream the captured database logs to a Kafka message queue.

[0015] In some embodiments of the present invention, importing the data in the message queue into a Doris database includes:

[0016] Adopt a stream loading method to stream the data in the message queue into the Doris database in single or batch form;

[0017] Or adopt a timed batch loading method to import the data in the message queue into the Doris database in a batch loading manner.

[0018] In some embodiments of the present invention, after generating the data service table, it further includes:

[0019] Write the data service table into an OLAP engine for service calls by the backend interface.

[0020] In some embodiments of the present invention, when creating the primary materialized view and the secondary materialized view, the method further includes:

[0021] Perform parameter settings on the primary materialized view and the secondary materialized view respectively to control the scheduling method, refresh frequency, and resource consumption of the primary materialized view and the secondary materialized view.

[0022] On the other hand, the present invention also provides a real-time data warehouse query method based on an asynchronous materialized view, and the method includes the following steps:

[0023] In a strong real-time scenario, directly execute a target Structured Query Language (SQL) query to query data in Doris;

[0024] In a quasi-real-time scenario, query the data service table generated by the real-time data warehouse construction method based on the asynchronous materialized view described above in Doris.

[0025] In some embodiments of the present invention, the method further includes:

[0026] Import the output of Doris into a preset unified interface service platform, and the preset unified interface service platform uses QuickService; add the data service table to the unified interface service platform to provide data services externally.

[0027] On the other hand, the present invention also provides a real-time data warehouse query system based on an asynchronous materialized view, including a processor, a memory, and a computer program / instructions stored on the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any one of the methods mentioned above.

[0028] On the other hand, the present invention also provides a computer-readable storage medium, on which computer program / instructions are stored. When the computer program / instructions are executed by a processor, the steps of any one of the methods mentioned above are implemented.

[0029] The present invention provides a real-time data warehouse construction method, query method, and system based on an asynchronous materialized view, including: obtaining database logs, using data import technology to import the database logs into a preset message queue, and then importing them into the Doris database and storing them in the raw data layer to generate a raw data table; using Structured Query Language (SQL) to create a first materialized view, storing the calculation result obtained by querying the first target Structured Query Language from the raw data table into the first materialized view to obtain a primary materialized view, serving as the detailed data layer, and generating a detailed data table; using Structured Query Language (SQL) to create a second materialized view, storing the calculation result obtained by querying the second target Structured Query Language from the detailed data table into the second materialized view to obtain a secondary materialized view, serving as the data service layer, and generating a data service table. The construction method provided by the present invention uses fewer components compared with the prior art. It only needs to write SQL statements in Doris to implement the development of (quasi-)real-time tasks, without additional Flink components, middleware, HDFS environment, and professional real-time development platforms; compared with the existing Flink solutions, there is no need to pay attention to the characteristics of data flow, no various complex backflow streams and multiple association methods. Only developers need to understand basic SQL syntax. At the same time, only the SQL logic of the maintenance points needs to be concerned, without paying attention to the cumbersome backtracking, dynamic tables, multi-stream associations, etc. in stream computing, which greatly reduces the development cost and maintenance cost.

[0030] Additional advantages, objects, and features of the present invention will be partly described below, and will partly become apparent to those of ordinary skill in the art after studying the following, or can be learned from the practice of the present invention. The objects and other advantages of the present invention can be realized and obtained by the structure specifically pointed out in the specification and the drawings.

[0031] Those skilled in the art will understand that the objects and advantages that can be achieved by the present invention are not limited to the above specifically described, and the above and other objects that the present invention can achieve will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention. In the drawings:

[0033] Figure 1 is a schematic diagram of the construction process of a real-time data warehouse based on the Lambda architecture in an embodiment of the present invention.

[0034] Figure 2 is a schematic diagram of the construction process of a real-time data warehouse based on the Kappa architecture in an embodiment of the present invention.

[0035] Figure 3 is a schematic diagram of the steps of a method for constructing a real-time data warehouse based on an asynchronous materialized view in an embodiment of the present invention.

[0036] Figure 4 is a schematic diagram of the process of a method for constructing a real-time data warehouse based on an asynchronous materialized view in an embodiment of the present invention.

[0037] Figure 5 is a schematic diagram of the process of a method for querying a real-time data warehouse based on an asynchronous materialized view in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] To make the objects, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0039] Here, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution of the present invention are shown in the drawings, and other details less related to the present invention are omitted.

[0040] It should be emphasized that when the term "comprising / including" is used herein, it refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0041] Here, it should also be noted that if not otherwise specified, the term "connection" in this text can refer not only to direct connection, but also to indirect connection with intermediaries.

[0042] In the following, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0043] To solve the problem that existing real-time data warehouse architectures, such as the Kappa architecture, Lambda architecture, etc., all have too high development costs for supporting real-time data warehouses, the present invention provides a method for constructing a real-time data warehouse based on asynchronous materialized views, as Figure 3 shown, the present invention includes the following steps S101 to S104:

[0044] Step S101: Obtain database logs; use data import technology to import the database logs into a preset message queue.

[0045] Step S102: Import the data in the message queue into the Doris database and store it in the raw data layer to generate a raw data table.

[0046] Step S103: Use Structured Query Language to create a first materialized view, and store the calculation results obtained by querying the first target Structured Query Language from the raw data table into the first materialized view to obtain a primary materialized view, and use the primary materialized view as the detailed data layer to generate a detailed data table.

[0047] Step S104: Use Structured Query Language to create a second materialized view, and store the calculation results obtained by querying the second target Structured Query Language from the detailed data table into the second materialized view to obtain a secondary materialized view, and use the secondary materialized view as the data service layer to generate a data service table. Among them, the data service table is provided for the business side to query.

[0048] As Figure 4 shown, it is a flow schematic diagram of a method for constructing a real-time data warehouse based on asynchronous materialized views.

[0049] For ease of understanding, the materialized view will be described first. The materialized view includes a synchronous materialized view and an asynchronous materialized view.

[0050] Synchronous Materialized View: A materialized view stores a pre-computed dataset in a special table in the database. The data stored in the table is the result set after the user-defined Structured Query Language (SQL). The database updates or deletes the data in the table according to certain fixed policies. The tables in the materialized view are always strongly consistent with the original table in real time, and no additional manual maintenance cost is required. The purpose of creating a materialized view is that when querying, the database engine will automatically match the optimal materialized view and obtain the real data from it. However, its disadvantage is that the base table for creating a materialized view can only use a single table, resulting in lower flexibility during use.

[0051] Asynchronous materialized views are similar to synchronous materialized views, except that the data timeliness of asynchronous materialized views is lower than that of synchronous ones, but their flexibility is much higher than that of synchronous materialized views. In terms of data consistency, asynchronous materialized views maintain eventual consistency with the original table data and also support the construction of multi-table associations. However, they are usually used in scenarios where timeliness requirements are not high and perform well in quasi-real-time and offline scenarios.

[0052] The core technology of the present invention lies in the "asynchronous materialized view function" of Doris. Its basic principle is mainly to materialize streaming data into a physical table in the OLAP engine and perform materialized view modeling at the table level using standard SQL statements. The upper-layer materialized view table is written into a new materialized view through SQL statements, and the new materialized view can still be modeled using standard SQL, thus completing the establishment of multi-level nested materialized views. Specifically:

[0053] In step S101, obtain the database log; use data import technology to import the database log into a preset message queue.

[0054] In some embodiments, in a relational database such as MySQL or Oracle, binary logs (MySQL) or Log logs (Oracle) can capture database change events such as insertions, updates, deletions, etc. These logs record data changes and provide a basis for real-time data synchronization.

[0055] In some embodiments, use real-time data capture technology implemented based on a preset streaming computing engine, such as FlinkCDC. Flink CDC can read the binary logs of the database, capture data changes in real time, and process them through the Flink stream processing framework, that is, stream the captured database logs to the Kafka message queue through Flink CDC.

[0056] In step S102, import the data in the message queue into the Doris database and store it in the raw data layer to generate a raw data table.

[0057] In some embodiments, the method of Stream Load is adopted to stream write the data in the message queue into the Doris database in single or batch form; or the method of Routine Load is adopted to import the data in the message queue into the Doris database in a batch loading manner.

[0058] Among them, Stream Load and Routine Load are two import methods of the Doris database. Stream Load is a streaming data loading method, usually used in scenarios of large-scale real-time data import, especially for import operations that require low latency. Its characteristic is that data is streamed and written into the target table in single or batch form, and it is suitable for scenarios with high requirements for real-time performance. Routine Load is another common data loading method. It usually imports data from an external system into the target data warehouse in a batch loading manner. Different from Stream Load, Routine Load emphasizes periodic and batch processing more, and is usually used in scenarios where real-time performance is not required. The Doris database (formerly known as Palo) is a high-performance distributed analytical database designed for large-scale data analysis and real-time query.

[0059] The data imported into the Doris database is stored in the Operational Data Store (ODS), generating an original data table. Among them, the ODS layer is used to store original data, which is usually the unprocessed original data extracted from source systems (such as business systems, log systems, etc.). It is usually the first layer in the data warehouse, and the data is updated frequently. Correspondingly, the data warehouse architecture usually also includes the Data Warehouse Detail (DWD) layer and the DataWarehouse Summary (DWS) layer. The DWD layer is the detailed layer after cleaning, transformation, and processing of the ODS layer data, containing detailed data in the business domain, which has been standardized and normalized to a certain extent. The data at this level is usually used for further analysis, but it is not yet suitable for direct summarization or large-scale queries. The DWS layer is the aggregation layer of the DWD layer data, usually performing summarization, calculation, and preprocessing to improve query performance. The data at this layer is usually aggregated by dimension and is suitable for reports and large-scale analysis.

[0060] In step S103, a first materialized view is created using the Structured Query Language, and the calculation result obtained by querying the first target Structured Query Language from the original data table is stored in the first materialized view to obtain a primary materialized view. The primary materialized view is used as the Data Warehouse Detail layer to generate a detailed data table.

[0061] In some embodiments, the first materialized view is created using the statement

CREATE MATERIALIZEDVIEW DWD_{table name}……

[0062] Among them, the first target structured query language refers to the SQL query used when creating the first materialized view. These queries define the way to obtain data from the original data tables in the original data layer (ODS layer), and the data is calculated or transformed in a certain way (such as the above-mentioned aggregation, filtering, etc.) and then stored in the first materialized view.

[0063] Based on the above operations, a primary materialized view is obtained, and the primary materialized view is used as the detailed data layer (DWD layer) to generate a detailed data table.

[0064] In step S104, the second materialized view is created again using the structured query language, and the calculation results obtained by querying the second target structured query language from the detailed data table are stored in the second materialized view to obtain a secondary materialized view. The secondary materialized view is used as the data service layer to generate a data service table.

[0065] Similarly, in some embodiments, the second materialized view is created using the statement

CREATE MATERIALIZEDVIEW DWD_{table name}……

[0066] Among them, the second target structured query language refers to the SQL query used when creating the second materialized view. These queries define the way to obtain data from the detailed data tables in the detailed data layer (DWD layer), and the data is calculated or transformed in a certain way (such as the above-mentioned aggregation, filtering, etc.) and then stored in the second materialized view.

[0067] Based on the above operations, a secondary materialized view is obtained, and the secondary materialized view is used as the data service layer (DWS layer) to generate a data service table.

[0068] As described above, the DWS layer is the aggregation layer of the DWD layer data. Usually, summarization, calculation, and preprocessing are performed to improve query performance. The data at this layer is usually aggregated by dimension and is suitable for reports and large-scale analysis. Therefore, the obtained data service table can be queried by the business side.

[0069] In some embodiments, after generating the data service table, the data service table is written into an OLAP engine, such as databases like Doris, ClickHouse, StarRocks, etc., for service calls by the backend interface.

[0070] In some embodiments, regarding the timeliness of data calculation between model levels, it depends on the scheduling of each materialized view and can also be automatically scheduled according to whether the data of the previous materialized view has changed. Therefore, parameter settings are performed when creating the primary materialized view and the secondary materialized view to control the scheduling method, refresh frequency, resource consumption, etc. of the primary materialized view and the secondary materialized view.

[0071] It should be noted that in order to ensure that the nested modeling method of the materialized view provided by the present invention can provide the best performance, it is not recommended to adjust the refresh frequency to the second level. Because a large amount of data refreshing will cause excessive resource consumption and lead to scheduling failures.

[0072] The ways of refreshing the materialized view include full refresh and partition incremental refresh:

[0073] Full refresh: Calculate all the data of the materialized view definition SQL.

[0074] Partition incremental refresh: When the partition data of the materialized view base table changes, identify the corresponding changed partitions of the materialized view and only refresh these partitions, thereby achieving partition incremental refresh without refreshing the entire materialized view.

[0075] According to the above rules, it is recommended to use the real-time (associated) query of Doris raw data in the case of high timeliness requirements and small data volume. In the case of high timeliness requirements and when the business table allows partitioning, nested materialized views with a higher scheduling frequency can be used, and the scheduling frequency of the materialized view can be appropriately increased, such as once every 5 minutes. In this way, the purpose of realizing the real-time data warehouse function without using the Flink streaming computing engine can be achieved.

[0076] Thus, the present invention also provides a real-time data warehouse query method based on an asynchronous materialized view, as Figure 5 In the dashed box, the method includes the following steps:

[0077] In a strong real-time scenario, it is carried out in the form of direct query, and the target structured query language is executed in Doris to query data. Among them, the SQL query supports multi-table join queries, which is the same as the usage method of ordinary MySQL.

[0078] In a quasi-real-time scenario, the data service table generated by the above real-time data warehouse construction method based on asynchronous materialized views is queried in Doris. This data service table is obtained through multi-level nested modeling of asynchronous materialized views and can provide customized data services for multiple business demanders.

[0079] In some embodiments, the output of Doris is imported into a preset unified interface service platform. Exemplarily, the preset unified interface service platform can adopt QuickService. By adding the data service table to the unified interface service platform, data services can be provided externally.

[0080] To facilitate comparison and reflect the benefits of the present invention, such as Figure 5As shown above the dashed box, the technical solutions of the Lambda architecture and the Kappa architecture described in the background technology section are presented. Before the data stream enters the streaming computing engine Flink, it is necessary to use a data synchronization tool to flatten the database change log into ChangeLog streaming data (or other types of streaming data), and write this data into the Kafka message queue. In the initial Kafka middleware, the data stream is ingested using Flink (such as FlinkSQL), and the data is normalized after ingestion. This process is rather cumbersome. If there are problems with the upstream middleware or for other reasons data replay is required, window deduplication needs to be performed in Flink. In regular business, the situation of multi-table (multi-stream) association inevitably occurs. For such an association, for each additional stream associated, it is necessary to perform pre-deduplication on the stream, and the logic is complex. Additionally, in the multi-stream association scenario, in order not to make the Flink state size expand infinitely, it is necessary to pay attention to the entire business process because the business process determines the lifecycle of the Flink state. Between the streaming computing levels (such as DWD => DWS), Kafka is generally used for message storage, and there are often additional messages (withdrawal streams and update streams) between the levels. At this time, it is necessary to properly handle the redundant withdrawal streams through the data stream pass-through of Flink, such as merging or discarding. Then, the data in the DWS layer often still exists in the Kafka message queue, and through an additional OLAP engine, such as ElasticSearch, ClickHouse, Kudu, etc., the data stream in Kafka is persisted. However, because the past mainstream OLAP engines generally have limited support for data updates and deletions, if there are updates in the upstream Kafka data, special processing needs to be performed according to different OLAP engines, such as marking a specific data entry as to be deleted through an additional field. Finally, the service layer can adopt the QuickService unified interface platform to add the tables in the OLAP engine into the platform and provide real-time data services externally.

[0081] Correspondingly to the above method, the present invention also provides a real-time data warehouse query system based on an asynchronous materialized view. The system includes a computer device, the computer device includes a processor and a memory, computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0082] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the foregoing edge computing server deployment method are implemented. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0083] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link.

[0084] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0085] In the present invention, features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0086] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for building a real-time data warehouse based on asynchronous materialized views, characterized in that: The method comprises the following steps: Obtain database logs; use data import technology to import the database logs into a preset message queue; Import the data in the message queue into the Doris database and store it in the original data layer to generate an original data table; Using a structured query language to create a first materialized view, and storing the calculation results obtained by querying the first target structured query language from the original data table into the first materialized view to obtain a primary materialized view, and using the primary materialized view as a detailed data layer to generate a detailed data table; A second materialized view is created using the structured query language, and a calculation result obtained by querying the detail data table using the second target structured query language is stored in the second materialized view to obtain a secondary materialized view. The secondary materialized view is used as a data service layer to generate a data service table; the data service table is provided for query by the business party.

2. The method for constructing a real-time data warehouse based on asynchronous materialized views according to claim 1, characterized in that: Get database logs, including: Obtain a binary log or a redo log in a relational database to capture change events of the relational database, where the change events at least include insert, update, and delete.

3. The method for constructing a real-time data warehouse based on asynchronous materialized views according to claim 1, characterized in that: Using data import technology, the database log is imported into a preset message queue, including: Using real-time data capture technology based on a preset streaming computing engine, the captured database logs are streamed to the Kafka message queue.

4. The method for constructing a real-time data warehouse based on asynchronous materialized views according to claim 1, characterized in that: Importing the data in the message queue into the Doris database includes: Using the streaming loading method, the data in the message queue is streamed into the Doris database in the form of a single entry or batches; Alternatively, a timed batch loading method is adopted to import the data in the message queue into the Doris database in a batch loading manner.

5. The method for constructing a real-time data warehouse based on asynchronous materialized views according to claim 1, characterized in that: After generating the data service table, it also includes: The data service table is written into the OLAP engine for the back-end interface to perform service calls.

6. The method for constructing a real-time data warehouse based on asynchronous materialized views according to claim 1, characterized in that: When creating the primary materialized view and the secondary materialized view, the method further includes: Parameters are set for the primary materialized view and the secondary materialized view respectively to control the scheduling mode, refresh frequency and resource consumption of the primary materialized view and the secondary materialized view.

7. A real-time data warehouse query method based on asynchronous materialized views, characterized in that: The method comprises the following steps: In a strong real-time scenario, directly execute the target structured query language to query data in Doris; In a quasi-real-time scenario, a data service table generated by the real-time data warehouse construction method based on asynchronous materialized views according to any one of claims 1 to 6 is queried in Doris.

8. The real-time data warehouse query method based on asynchronous materialized views according to claim 7 is characterized in that: The method further comprises: The output of Doris is imported into a preset unified interface service platform, and the preset unified interface service platform adopts QuickService; the data service table is added to the unified interface service platform to provide data services externally.

9. A real-time data warehouse query system based on asynchronous materialized views, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images