Data materialization method and device, computer equipment and storage medium

Abstract

The invention provides a data materialization method and device, computer equipment and a storage medium, and the method comprises the steps: creating a source pasting virtual table according to a data theme and data element information of a streaming data source, and creating a service virtual table for the source pasting virtual table; the source pasting virtual table is used for indexing streaming data in the streaming data source; the business virtual table comprises a data cleaning rule for data element information in the source pasting virtual table; according to a data cleaning rule and an index relationship in the source pasting virtual table, performing real-time cleaning on streaming data meeting the data cleaning rule in the streaming data source to obtain a target data table; obtaining materialization index information and materialization time granularity related to the target data table; the physicochemical index information at least comprises index measurement, index dimensions and index increment columns; and performing incremental materialization processing on the detail data in the target data table according to the materialization index information and the materialization time granularity to obtain a materialization data table.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a data materialization method, apparatus, computer equipment, and storage medium. Background Art

[0002] In real-time data processing scenarios, to better meet user query requirements, pre-materialization of real-time data is often employed. Traditional materialization methods either employ a full materialization mechanism (that is, materializing all current real-time data upon receiving a user's materialization request) or employ an architecture that combines the distributed data stream engine Flink with an external system.

[0003] However, full materialization can lead to storage redundancy and computing resource overload, while the Flink + external system architecture is complex, challenging to maintain, and difficult for users to use. Therefore, the current materialization methods have significant drawbacks and urgently need to be optimized. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a data materialization method, apparatus, computer device, and storage medium to avoid storage redundancy and computing resource overload problems, reduce the difficulty and complexity of data materialization, and improve data query efficiency.

[0005] In a first aspect, an embodiment of the present disclosure provides a data materialization method, comprising: Creating a source virtual table based on the data subject and data metadata of the streaming data source, and creating a business virtual table for the source virtual table; the source virtual table is used to index the streaming data in the streaming data source; the business virtual table includes data cleaning rules for the data metadata in the source virtual table; According to the data cleaning rules and the index relationship in the source virtual table, the streaming data in the streaming data source that meets the data cleaning rules is cleaned in real time to obtain a target data table; Obtaining materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information includes at least indicator measurement, indicator dimension, and indicator increment column; According to the materialization index information and the materialization time granularity, incremental materialization processing is performed on the detailed data in the target data table to obtain a materialized data table; the materialized data table includes materialized window data corresponding to each materialization processing and the materialization time range of the materialized window data.

[0006] In a possible implementation manner, after obtaining the physical and chemical data table, the method further includes: Obtaining a user's data query request and determining the query indicator information and query time range corresponding to the data query request; According to the query index information and the query time range, a query result matching the data query request is determined from the materialized window data and / or the target data table.

[0007] In a possible implementation, determining a query result matching the data query request from the materialized window data and the target data table according to the query indicator information and the query time range includes: Splitting the query index information and the query time range according to the materialized time range and the materialized index information to obtain first sub-query index information and a first sub-query time range for which matching target window data exists, and second sub-query index information and a second sub-query time range for which matching materialized window data does not exist; performing calculation processing on the detailed data in the target data table that matches the second subquery time range according to the second subquery indicator information to obtain calculated data; A query result matching the data query request is determined based on the calculated data and the target window data in the materialized data table that matches the first sub-query indicator information and the first sub-query time range.

[0008] In a possible implementation, creating a source virtual table matching the data subject based on the data subject and data metadata of the streaming data source, and creating a business virtual table for the source virtual table, includes: Determine the table structure and time column of the source virtual table that matches each data subject in the streaming data source; the time column is used to indicate the time information of the streaming data; Determine data meta information associated with a source virtual table according to the data meta information of the stream data under the data subject in the streaming data source; Creating a post source virtual table for each of the data topics according to the table structure, the time column and the data meta information; For each of the source virtual tables, data cleaning rules for the data metadata in the source virtual table are determined according to data processing requirements related to the data subject, and a business virtual table corresponding to the source virtual table is created according to the data cleaning rules.

[0009] In a possible implementation, the step of performing real-time cleaning on the streaming data in the streaming data source that meets the data cleaning rules according to the data cleaning rules and the index relationship in the source virtual table to obtain a target data table includes: Continuously acquiring target stream data that needs to be cleaned from the stream data mapped by the source virtual table in the streaming data source according to the data meta-information targeted by the data cleansing rule, the index relationship corresponding to the data meta-information in the source virtual table, and the index relationship corresponding to the data subject matched by the source virtual table; Clean the target stream data according to the data cleaning rules to obtain cleaned detailed data; Determine a target table structure of a target data table according to the data cleaning rule, and create a target data table according to the target table structure; The detailed data is stored in the target data table using the primary key specified for the target data table.

[0010] In a possible implementation, the indicator metric is used to indicate an aggregate calculation operation on the first target column in the target data table; The indicator dimension is used to indicate a grouping operation on multiple second target columns in the target data table; The indicator increment column is used to indicate the time column of the detailed data in the target data table and the time format information under the time column.

[0011] In a possible implementation, performing incremental materialization processing on the detailed data in the target data table according to the materialization index information and the materialization time granularity to obtain the materialized data table includes: According to the materialization time granularity, the materialization processing task is triggered regularly; Using the materialized processing task, determining each target incremental data in the target data table that matches the materialized time range corresponding to the materialized time granularity according to the time information of the detailed data in the target data table under the indicator increment column; Incrementally materialize the target incremental data according to the indicator metric and the indicator dimension to obtain materialized window data that matches the materialization time range, and store the materialized window data in a materialized data table according to the materialization time range.

[0012] In a second aspect, an embodiment of the present disclosure further provides a data materialization device, including: a creation module, configured to create a source virtual table based on the data subject and data metadata of the streaming data source, and to create a business virtual table for the source virtual table; the source virtual table is configured to index the streaming data in the streaming data source; the business virtual table includes data cleaning rules for the data metadata in the source virtual table; a cleaning module, configured to clean the streaming data in the streaming data source that meets the data cleaning rules in real time according to the data cleaning rules and the index relationship in the source virtual table, to obtain a target data table; An acquisition module, configured to acquire materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information includes at least indicator metrics, indicator dimensions, and indicator increment columns; The materialization module is used to perform incremental materialization processing on the detailed data in the target data table according to the materialization indicator information and the materialization time granularity to obtain a materialized data table; the materialized data table includes the materialization window data corresponding to each materialization processing and the materialization time range of the materialized window data.

[0013] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0014] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, including a computer program, which, when executed, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0015] The data materialization method, apparatus, computer equipment and storage medium provided by the embodiments of the present disclosure can realize data cleaning before data materialization by actively constructing a source virtual table related to the streaming data source, and then creating a business virtual table based on the source virtual table, thereby reducing the complexity and difficulty of data cleaning. Based on the acquired materialization index information, the data in the target data table obtained after cleaning can be automatically incrementally materialized according to the materialization time granularity, thereby reducing storage overhead, improving computing resource utilization, and avoiding computing resource overload. Moreover, without the need for additional docking with an external system, the embodiments of the present disclosure only need to utilize a pre-developed virtualization engine, and through the actively created source virtual table and business virtual table as well as the materialization index information and materialization time granularity, it can realize automatic incremental materialization processing of real-time data, which not only reduces the complexity of the architecture and the complexity of operation and maintenance, but also reduces the difficulty of user use.

[0016] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0018] Figure 1 A flow chart of a data materialization method provided by an embodiment of the present disclosure is shown; Figure 2 A schematic diagram of a data cleaning process provided by an embodiment of the present disclosure is shown; Figure 3 A process diagram of an indicator incremental materialization provided by an embodiment of the present disclosure is shown; Figure 4 A schematic diagram of a data query process provided by an embodiment of the present disclosure is shown; Figure 5 A schematic diagram of a data materialization device provided by an embodiment of the present disclosure is shown; Figure 6 A schematic structural diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0020] In addition, the terms "first," "second," and the like in the description and claims of the embodiments of the present disclosure and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments described herein can be practiced in an order other than that shown or described herein.

[0021] In this document, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0022] Research has found that to better meet user query needs, existing Flink + external system architectures often include the following two options: one is the Flink + MySQL direct query model, which first processes real-time data through Flink, cleans the real-time data to obtain detailed data, and stores it in a MySQL detailed data table with a data timestamp. When users query, they directly query the detailed data table according to the user-specified time window. The other is the Flink + materialization scheduling system + query system model, which processes real-time data through Flink, cleans the real-time data to obtain detailed data, and stores it in a MySQL detailed data table. Users use an external scheduling system to trigger the early materialization of the detailed data and store the materialization results in MySQL. When users query, they query the detailed data table and the materialization results by calling the external query system to obtain the query results. However, the Flink + MySQL direct query model requires maintaining a dual Flink + MySQL cluster, which has high operation and maintenance costs and a complex architecture. In addition, the user has a high barrier to entry for using Flink, which increases the user experience. The Flink + materialized scheduling system + query system approach requires three systems, with higher operation and maintenance costs and architectural complexity. There is still a barrier to entry, making it difficult for users to use.

[0023] Based on the above research, the present disclosure provides a data materialization method, apparatus, computer equipment and storage medium. By actively constructing a source virtual table related to a streaming data source, and then creating a business virtual table based on the source virtual table, data cleaning before data materialization can be achieved, reducing the complexity and difficulty of data cleaning. Based on the acquired materialization index information, the data in the target data table obtained after cleaning can be automatically incrementally materialized according to the materialization time granularity, reducing storage overhead, improving computing resource utilization, and avoiding computing resource overload. Moreover, the embodiment of the present disclosure only needs to utilize a pre-developed virtualization engine without the need for additional docking with an external system. By actively creating a source virtual table and a business virtual table as well as materialization index information and materialization time granularity, it can achieve automatic incremental materialization processing of real-time data, which not only reduces the complexity of the architecture and the complexity of operation and maintenance, but also reduces the difficulty of user use.

[0024] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this disclosure for the above problems below should be the contributions made by the inventors to this disclosure during the disclosure process.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0026] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include: Kafka: An open source stream processing platform that serves as a high-throughput distributed publish-subscribe messaging system that handles all the stream data consumers are streaming across websites. JavaScript Object Notation (JSON) is an open standard file format and data interchange format that is easy for humans to read and write, and easy for machines to parse and generate. Flink: An open source stream processing framework with a distributed data flow engine at its core. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute both batch and stream processing programs.

[0028] To facilitate understanding of this embodiment, a data materialization method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the data materialization method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with a certain computing capability, where the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, etc.; in some possible implementations, the data materialization method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0029] The data materialization method provided by the embodiment of the present disclosure is described below by taking the execution subject as a virtualization engine as an example.

[0030] like Figure 1FIG. 1 is a flowchart of a data materialization method provided by an embodiment of the present disclosure, which may include the following steps: S101: Create a source virtual table based on the data subject and data metadata of the streaming data source, and create a business virtual table for the source virtual table; the source virtual table is used to index the streaming data in the streaming data source; the business virtual table includes data cleaning rules for the data metadata in the source virtual table.

[0031] First of all, it should be noted that the present disclosure belongs to a virtualization engine product. The developed virtualization engine can realize the materialization processing of real-time data and support fast query of any time window. It is suitable for real-time data analysis, monitoring systems and distributed stream processing scenarios.

[0032] A streaming data source contains a streaming data table that includes real-time streaming data. In this disclosure, as long as the data can be transmitted through a message-based middleware, it can be defined as streaming data, and the source of the data can be defined as a streaming data source. For example, a streaming data source can be a Kafka data source, which acts as a message-based middleware for transmitting streaming data.

[0033] A streaming data source can have multiple data topics. A single topic can contain multiple pieces of real-time streaming data, with different streams of data in different topics. Data metadata indicates the data attributes of the streaming data, such as the data field, data format, field type, and data example. Data formats can be, for example, JSON or comma-separated values (CSV).

[0034] The source virtual table can be defined based on the Structured Query Language (SQL), and is a mapping of the data metadata of the streaming data source (for example, it can be a description of the data metadata of the streaming data under each topic of the streaming data source), and is an abstraction made to facilitate the subsequent indexing of external streaming data. The source virtual table has a specified time column and table structure, and may not include any specific streaming data, but includes data attribute information corresponding to the streaming data in the streaming data source and a topic associated with the streaming data source. By creating a source virtual table, the streaming data table in the external streaming data source can be mapped to a virtualization system, and the index reference to the data table in the external streaming data source can be realized, completing the transformation from the physical table to the virtualization.

[0035] A source virtual table can be used to locate streaming data under a topic in a streaming data source. A streaming data source can correspond to multiple source virtual tables.

[0036] A business virtual table can also be a SQL-based virtual table that includes at least one data cleansing rule for cleaning at least some of the data metadata in the source virtual table. A source virtual table can correspond to at least one business virtual table, and different business virtual tables can include data cleansing rules specific to different data metadata.

[0037] During implementation, various streaming data sources requiring data cleansing can be pre-connected. For example, if a Kafka streaming data source is used, the Kafka streaming data source can be connected based on the user-provided Kafka host (Broker) address, username, and password. After connecting to the streaming data source, the user can use SQL statements to initiate a request to create a source virtual table for the streaming data source. This request can include information such as the data topic, table structure, and time column associated with the source virtual table. In response to the request, the virtualization engine can generate a source virtual table associated with the streaming data source based on the user-specified data topic associated with the source virtual table and the data metadata of the streaming data under that topic. After the source virtual table is created, the engine can retrieve data cleansing rules defined by the user based on SQL statements for certain data metadata in the source virtual table. For example, the data cleansing rules can be user-defined SQL logic for certain data metadata in the source virtual table, such as SQL logic for data metadata such as age, height, order quantity, and order type. The virtualization engine can generate a business virtual table that matches the data cleaning rules and associate it with the source virtual table, and then launch the business virtual table to trigger the data cleaning task.

[0038] In one embodiment, the above S101 can be implemented according to the following steps: S101 - 1 : Determine the table structure and time column of the source virtual table that matches each data subject in the streaming data source; the time column is used to indicate the time information of the streaming data.

[0039] Here, the table structure is used to indicate the structure of the source virtual table. For example, the table structure can be what kind of data each column of the source virtual table contains. The time column is a column of the source virtual table, which can be used to store the time information of each flow data under the data topic associated with the source virtual table.

[0040] In practice, since streaming source virtual tables are defined based on SQL and require the specification of the table's associated topic, data format, time column, and table structure, when creating a source virtual table for a streaming data source, the user can specify the topic and time column that each source virtual table should be associated with in the streaming data source. The virtualization engine can then retrieve the table structure and time column of the source virtual table that matches each topic in the streaming data source.

[0041] S101 - 2 : Determine data meta information associated with the source virtual table based on data meta information of the streaming data under the data subject in the streaming data source.

[0042] In specific implementations, for any data theme's source virtual table, the virtualization engine can retrieve the data metadata of the streaming data under that data theme from the streaming data source and use that data metadata as the data metadata associated with the source virtual table corresponding to that data theme. If the user indicates a data format, the data format in the data metadata is updated using the user's specified format; if the user does not indicate a data format, the data format indicated by the data metadata is used.

[0043] S101 - 3 : Create a source virtual table for each data topic based on the table structure, time column, and data metadata.

[0044] During specific implementation, for each topic, the virtualization engine can create a source virtual table including the data metadata information, the mapping relationship between the data metadata information and the topic, and the time column according to the table structure defined for the source virtual table corresponding to the topic and the data metadata associated with the source virtual table corresponding to the topic, as the source virtual table of the topic, and can update the information under the time column in the source virtual table according to the time information of the stream data under the topic.

[0045] S101-4: For each source virtual table, determine data cleaning rules for the data metadata in the source virtual table according to data processing requirements related to the data subject, and create a business virtual table corresponding to the source virtual table according to the data cleaning rules.

[0046] Here, the data processing requirement is used to indicate the need for data cleaning of the data metadata in the source virtual table. Different topics can correspond to different data processing requirements. The specific data processing requirements can be related to the specific application scenarios corresponding to the topic, and the embodiments of the present disclosure do not specifically limit them. For example, the data processing requirement under a certain data topic can be to filter the streaming data that does not meet a certain condition in fields A and B. There can be multiple data processing requirements under a topic, and therefore, there can also be multiple data cleaning rules under a topic. The number of business virtual tables corresponding to a source virtual table can be determined based on the materialized indicator information, the specific application scenarios of the streaming data, and the cleaning requirements. The materialized indicator information is used to indicate how to materialize the detailed data in the target data table.

[0047] In specific implementations, for each source virtual table, the user-defined data processing requirements for each business virtual table, using SQL (or Flink SQL syntax), are obtained. The virtualization engine then obtains these data processing requirements. Based on these data processing requirements, the virtualization engine determines the corresponding data cleansing rules for each business virtual table. Data cleansing rules are used to extract, transform, and load data (ETL), specifically the entire process of extracting data from a streaming data source, transforming it, and loading it into the virtualization engine. The virtualization engine then creates business virtual tables corresponding to the source virtual table based on the data cleansing rules and associates the source virtual table with the business virtual table.

[0048] S102: According to the data cleaning rules and the index relationship in the source virtual table, the streaming data in the streaming data source that meets the data cleaning rules is cleaned in real time to obtain a target data table.

[0049] Here, since the streaming data in the streaming data source arrives in real time, after the business virtual table goes online, the virtual table engine will continuously consume the streaming data in the corresponding streaming data source and continuously perform real-time data cleansing according to the data cleansing rules in the business virtual table.

[0050] When creating a business virtual table, the user needs to specify the output table name (i.e., the target data table name) where the cleaned detailed data will be stored. After the data is cleaned and detailed data is obtained, the detailed data can be stored in the target data table with the target data table name.

[0051] In specific implementation, after creating a business virtual table, you can launch it online. Based on the data cleansing rules in the launched business virtual table, the virtualization engine can trigger a background data cleansing task. This data cleansing task determines the target data metadata that needs to be processed in the source virtual table. Then, based on the topic associated with the source virtual table and the index relationship between the target data metadata in the source virtual table and the streaming data source, it locates each specific stream data item in the streaming data source that matches the target data metadata under that topic. ETL processing is then performed on this stream data item according to the data cleansing rules to obtain the detailed data corresponding to this stream data item, which is then stored in the target data table.

[0052] For example, the business virtual table is associated with the source virtual table. Based on the source virtual table, the topic, Kafka address, data metadata and other information of the stream data to be processed can be determined. Based on this information, the stream data in the streaming data source is cleaned using data cleaning rules to obtain detailed data.

[0053] In one embodiment, the above S102 may be implemented as follows: S102-1: Based on the data meta-information targeted by the data cleaning rules, the index relationship corresponding to the data meta-information in the source virtual table, and the index relationship corresponding to the data subject matched by the source virtual table, the target stream data that needs to be cleaned is continuously obtained from the stream data mapped by the source virtual table in the streaming data source.

[0054] During implementation, the virtualization engine can determine the data metadata that needs to be cleaned based on the data cleansing rules in the business virtual table. It then determines the target data metadata that matches the data metadata that needs to be cleaned from the source virtual table, obtains the index relationship between the target data metadata and the streaming data source from the source virtual table, and obtains the index relationship corresponding to the topic that matches the source virtual table from the source virtual table. Based on these index relationships, it locates each target stream data item under the corresponding topic in the streaming data source and continuously obtains this target stream data item, using this obtained target stream data item as the target stream data item that needs to be cleaned.

[0055] S102-2: Clean the target stream data according to the data cleaning rules to obtain cleaned detailed data.

[0056] Here, the cleaned stream data can be defined as detailed data.

[0057] For example, the virtualization engine may perform ETL processing on the target stream data continuously acquired from the streaming data source according to various data cleaning rules in the business virtual table to obtain various pieces of cleaned detailed data.

[0058] S102-3: Determine the target table structure of the target data table according to the data cleaning rules, and create the target data table according to the target table structure.

[0059] During implementation, the virtualization engine can determine the required data columns in the target data table based on the data cleansing rules in the business virtual table, thereby obtaining the target table structure. The virtualization engine can then create a target data table with the target table structure and name it the user-specified target data table name.

[0060] In this way, the created target data table can be used as a table in the virtualization engine to store the detailed data obtained after cleaning.

[0061] S102-4: Using the primary key specified for the target data table, store the detail data in the target data table.

[0062] When launching a business virtual table, users need to specify its primary key. This primary key is used to deduplicate detail data stored in the target data table. For example, if two detail data items share the same primary key, they are considered duplicates.

[0063] In specific implementation, the virtualization engine can use the primary key to determine the detailed data with the same data under the primary key, deduplicate the detailed data, and then store the deduplicated detailed data and the non-duplicate detailed data in the target data table.

[0064] like Figure 2 FIG. 1 is a schematic diagram of a data cleaning process provided by an embodiment of the present disclosure, which may include the following steps: The user accesses the streaming data source in the virtualization engine. The virtualization engine sends an access feedback response to the user. The user instructs to create a source virtual table for the streaming data source, and the virtualization engine creates the corresponding source virtual table and feeds it back to the user. The user instructs the data cleansing rules for the source virtual table to create a business virtual table. The virtualization engine creates the corresponding business virtual table based on the data cleansing rules and feeds it back to the user. The user chooses to go online with the business virtual table to instruct the virtualization engine to perform data cleansing operations based on the data cleansing rules in the business virtual table. The virtualization engine derives the table structure of the target data table based on the data cleansing rules and creates the target data table, while triggering the background data cleansing task. The data cleansing task will continuously clean the streaming data in the streaming data source in real time and write the cleaning results to the target data table. The virtualization engine can feed back the target data table including detailed data to the user.

[0065] In this way, this application achieves access to streaming data sources through source virtual tables. Using business virtual table technology, data cleansing rules for streaming data are defined. Based on the definition of business virtual tables, the table structure of detailed data (i.e., data cleansing results) is derived and detailed tables are automatically created. Using the online technology of business virtual tables, data cleansing operations and storage of streaming data are automatically completed.

[0066] S103: Acquire materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information at least includes indicator measurement, indicator dimension, and indicator increment column.

[0067] Here, materialized indicator information is used to materialize the detailed data in the target data table. The materialization time granularity indicates the time range of the data to be materialized. This disclosure supports materializing data at any granularity, so the materialization time granularity can be arbitrary. For example, the materialization time granularity can be year, month, day, hour, or a specific time range (such as every 12 hours), thereby enabling materialization of detailed data within each year, month, day, hour, or specific time range.

[0068] The materialized indicator information may specifically include indicator metrics, indicator dimensions, and indicator increment columns. The indicator metrics are used to indicate the aggregation calculation operations for the first target column in the target data table; the indicator dimensions are used to indicate the grouping operations for multiple second target columns in the target data table; and the indicator increment columns are used to indicate the time column of the detailed data in the target data table and the time format information under the time column.

[0069] Here, the first target column is the data column in the target data table that the indicator measurement targets. Typically, one indicator measurement corresponds to one target column. The aggregation calculation operation is used to aggregate the detailed data in the first target column. The aggregation calculation operation can be indicated by an aggregation function. This application does not specifically limit the aggregation function. For example, it can be a sum function SUM, a calculation function COUNT, an average function AVG, etc.

[0070] The second target columns are the multiple data columns in the target data table targeted by the indicator dimension. The indicator dimension can be used to group multiple second target columns in the target data table during materialization, such as in a group by operation in SQL. The second target columns may or may not include the first target column. If the second target column includes the first target column, during materialization, the second target column can be grouped based on the indicator dimension first, and then the grouped results in the first target column can be aggregated based on the indicator metric. If the second target column does not include the first target column, the grouping and aggregation processes can be performed simultaneously during materialization to obtain both grouped and aggregated results.

[0071] When using materialized indicator information for incremental materialization, the time information in the indicator increment column is needed to determine whether the detailed data is incremental. The detailed data in the indicator increment column is generally formatted as a timestamp or date. Therefore, the indicator increment column can be used to indicate the time column of the detailed data in the target data table. The time column can include the time information of the detailed data in the corresponding data format.

[0072] The materialized time granularity must match the time format information indicated by the metric increment column. For example, if the time format information is a timestamp, the materialized time granularity can be year / month / day / hour / specific time range; if the time format information is year-month-day, the materialized time granularity can be year / month / day.

[0073] In practice, users can model metrics for the target data table, specifying the materialized metric information and materialized time granularity associated with the target data table and publishing it to the virtualization engine. The materialized time granularity must be supported by the metric's increment column. For example, if the increment column is a date string in the yyyy-MM-dd format, the materialized time granularity must be greater than or equal to days. This allows the virtualization engine to retrieve the materialized metric information and materialized time granularity.

[0074] S104: performing incremental materialization processing on the detailed data in the target data table according to the materialization index information and the materialization time granularity to obtain a materialized data table; the materialized data table includes the materialized window data corresponding to each materialization processing and the materialization time range of the materialized window data.

[0075] Incremental materialization is used to materialize detailed data within the materialization time range corresponding to the materialization time granularity. A materialization time granularity can include multiple materialization time ranges. For example, if the materialization time granularity is daily, the materialization time range is the time range corresponding to each day; if the materialization time granularity is monthly, the materialization time range is the time range corresponding to each month; and if the materialization time granularity is hourly, the materialization time range is the time range corresponding to each hour.

[0076] A materialized data table can include multiple partitioned tables, with each partitioned table storing materialized window data within a materialization time range. A materialized window data item at a materialization time granularity can include multiple items, each with a materialized time range that matches the materialization granularity. For example, if the materialization time granularity is hourly, the materialized time range can include hourly items, and the materialized window data can include materialized data corresponding to each hour. For example, if the materialization time granularity is daily, the materialized data table can include partitioned tables corresponding to each day, each partitioned table storing materialized window data for each day. Materialized window data is the pre-computed data obtained after materializing the detailed data within the materialization time range. The materialized window data obtained from each materialization process is stored in the partitioned table corresponding to the materialization time range. For example, if the materialization time granularity is hourly, the materialized window data can be the materialized data corresponding to the detailed data for each hour, and the materialized window data is stored in the partitioned table corresponding to the hour.

[0077] In specific implementations, the virtualization engine can trigger a materialization task. This task uses the materialization time granularity and the indicator increment column to identify the data partitions from the target data table that match the materialization time granularity. The data in these partitions is used as incremental materialized data. The incremental materialized data is then materialized using the indicator metrics and indicator dimensions to generate materialized window data. This materialized window data is then stored in the partition table corresponding to the materialization time range. Based on each partition table, a materialized data table is generated.

[0078] In one embodiment, the above S104 can be implemented according to the following steps: S104-1: Trigger the materialization processing task regularly according to the materialization time granularity.

[0079] Here, due to data arrival delays in the target data table, for example, detailed data for the past day may not arrive until a certain time in the early morning of the next day. To ensure that materialization processing is performed immediately upon the arrival of detailed data within a certain materialization time range, the materialization processing task can be periodically triggered using the materialization time granularity. The triggering interval of the materialization processing task can be related to the materialization time granularity and be less than the materialization time range corresponding to the materialization time granularity. For example, if the materialization time granularity is hourly, the materialization processing task can be triggered every 10 minutes / 20 minutes to perform materialization processing immediately upon the arrival of detailed data for the past hour. For another example, if the materialization time granularity is daily, the materialization processing task can be triggered every 1 hour / every 2 hours / every 6 hours to perform materialization processing immediately upon the arrival of detailed data for the past day.

[0080] In specific implementation, after publishing the materialized indicator information and materialized time granularity, you can first create a materialized view related to the materialized indicator information. Then the materialized view will periodically trigger the materialized processing task according to the materialized time granularity.

[0081] S104-2: Using the materialized processing task, according to the time information of the detailed data in the target data table under the indicator increment column, determine each target incremental data in the target data table that matches the materialized time range corresponding to the materialized time granularity.

[0082] During specific implementation, the virtualization engine can use the materialized processing task to find a processable partition from the target data table that matches the materialized time range corresponding to the materialized time granularity based on the time information of the detailed data in the target data table under the indicator increment column, and use the detailed data within the processable partition as the target incremental data. For example, if the materialized granularity is hourly, the materialized processing task can be used to find the processable partition corresponding to the previous hour from the target data table based on the time information of the detailed data in the target data table under the indicator increment column, and use the detailed data within the partition as the target incremental data for the previous hour.

[0083] S104-3: Perform incremental materialization processing on the target incremental data according to the indicator measurement and indicator dimension to obtain materialized window data that matches the materialization time range, and store the materialized window data in the materialized data table according to the materialization time range.

[0084] During specific implementation, the virtualization engine can perform aggregation calculation processing on the target incremental data under the first target column indicated by the indicator measurement, and group processing on the target incremental data under each second target column indicated by the indicator dimension. Based on the aggregation calculation processing results and the group processing results, the materialized window data that matches the materialized time range is determined, and the materialized window data is stored in the materialized data table according to the materialized time range.

[0085] For example, when the materialization granularity is hourly, the target incremental data of the first target column in the past hour can be aggregated and calculated, and the target incremental data of each second target column in the past hour can be grouped and processed. Based on the aggregation calculation results and the grouping processing results, the materialized window data corresponding to the past hour is obtained, and the materialized window data is stored in the partition table corresponding to the past hour.

[0086] In this way, by materializing the incremental columns of indicator information, it is possible to process only the newly added data without the need to materialize the full amount of detailed data, thus avoiding storage redundancy and computing resource overload problems.

[0087] like Figure 3 As shown, a process diagram of incremental materialization of indicators provided by an embodiment of the present disclosure may include a user publishing materialized indicator information and materialized time granularity in a virtualization engine based on a target data table. The virtualization engine may create a materialized view corresponding to the materialized indicator information and asynchronously trigger a materialized processing task. The materialized processing task is used to automatically determine and process the detailed data within the corresponding time window (i.e., the materialized time range) based on the incremental column and materialized time granularity of the target data table, and write the processing result (i.e., the materialized window data) into the materialized data table corresponding to the materialized view. At the same time, the materialized processing task will also be triggered to execute the processing task periodically to realize the materialized processing of the detailed data within each materialized time range corresponding to the materialized time granularity.

[0088] In this way, this application proposes an incremental materialization method for streaming detailed data / indicators. Based on the incremental columns of streaming detailed data, it can automatically determine the processable conditions of the indicator window and automatically complete the indicator calculation, thereby realizing the incremental materialization of streaming detailed data / indicators and completing data materialization at hourly / dayly / monthly / yearly granularity.

[0089] In one embodiment, after obtaining the materialized data table, data query of any time window based on the materialized data table is performed. Specifically, the data query can be implemented according to the following steps T1 to T2: T1: Obtain the user's data query request and determine the query indicator information and query time range corresponding to the data query request.

[0090] Here, a data query request is used to query streaming data in a streaming data source. The data query request may include query metric information and a query time range. The query metric information may include the query metric metric and query metric dimension specified by the user. The query time range indicates the time range within which the streaming data to be queried falls.

[0091] During specific implementation, the virtualization engine may obtain a data query request initiated by a user, parse the data query request, and determine the query indicator information and query time range included in the data query request.

[0092] T2: Based on the query metric information and query time range, determine the query results that match the data query request from the materialized window data and / or target data table.

[0093] In specific implementation, based on the query index information and the query time range, a query can be first performed on the materialized data table to determine whether there is target materialized window data that completely matches the data query request. If so, the target materialized window data can be directly used as the query result. If not, if there is target materialized window data in the materialized data table that partially matches the data query request, an index calculation can be performed on the detailed data corresponding to the unmatched portion in the detailed data table to obtain a calculation result, and the partially matched target materialized window data and the calculation result can be merged into the query result. If there is no materialized window data in the materialized data table that matches the data query request, an index calculation can be performed on the detailed data in the target data table that meets the query index information and the query time range to obtain a calculation result, and the calculation result can be used as the query result.

[0094] In one embodiment, the above T2 can be implemented according to the following steps: T2-1: Split the query indicator information and query time range based on the materialized time range and materialized indicator information to obtain the first sub-query indicator information and first sub-query time range for which matching target window data exists, and the second sub-query indicator information and second sub-query time range for which matching materialized window data does not exist.

[0095] Here, the materialized time range refers to the time range corresponding to the materialized window data, and the materialized indicator information refers to the materialized indicator information used for the materialized window data. The target window data refers to the target within the materialized window data whose materialized time range falls within the query time range and whose materialized indicator information matches the query indicator information. Specifically, the matching of the materialized indicator information and the query indicator information can include the consistency of the materialized indicator metric in the materialized indicator information and the consistency of the materialized indicator dimension in the materialized indicator information and the query indicator dimension in the query indicator information.

[0096] The first subquery indicator information can be indicator information within the query indicator information that matches the materialized indicator information corresponding to the target window data. The first subquery range can be a time range within the query time range that overlaps with the materialized time range corresponding to the target window data. For example, the first subquery indicator information and the first subquery range can be the materialized indicator information and materialized time range corresponding to the target window data. The second subquery indicator information is the indicator information within the query indicator information excluding the first subquery indicator information; the second subquery time range is the time range within the query indicator information excluding the first subquery time range. If the second subquery indicator information and the second subquery time range do not have matching window data in the materialized data table, indicator calculation must be performed in the detailed data table to obtain the query result. The first subquery indicator information and the first subquery time range are used to query the target data table to obtain the target window data; the second subquery indicator information and the second subquery time range are used to query the target data table to obtain real-time calculated data. That is, by using the first subquery metric information, the first subquery time range, the second subquery metric information, and the second subquery time range, a data query request can be split into a materialized view subquery targeting the materialized data table and a detail data subquery targeting the target data table. Here, the materialized view can be understood as the materialized window data obtained after materialization.

[0097] For example, materialized matching can be performed based on the materialized indicator information corresponding to each materialized window data in the materialized data table. Candidate window data whose materialized indicator information matches the query indicator information can be determined from each materialized window data. Then, based on the materialized time range corresponding to the candidate window data, target window data whose query time range includes the corresponding materialized time range can be determined from the candidate window data. The materialized indicator information and materialized time range corresponding to the target window data are used as the first sub-query indicator information and the first sub-query range. The query indicator information other than the first sub-query indicator information is used as the second sub-query indicator range, and the query time range other than the first sub-query range is used as the second sub-query time range.

[0098] Alternatively, the query time range can be first split according to the materialized time granularity into a third sub-query time range that matches the materialized time granularity and a fourth sub-query time range that does not match the materialized time granularity. For example, if the user's query time range is 2023-10-01 08:00 to 2023-10-02 12:00, and the materialized time granularity is daily, the third sub-query time range can be the entire day of 2023-10-01, and the fourth sub-query time range can be 2023-10-02 00:00 to 12:00. Then, it can be determined whether the query metric information contains sub-query indicator information that matches the materialized indicator information of the materialized window data within the third sub-query time range. If so, the materialized window data is used as the target window data, the sub-query indicator information is used as the first sub-query indicator information, and the third sub-query time range is used as the first sub-query time range. The query index information except the first sub-query index information is used as the second sub-query index range, and the query time range except the first sub-query range is used as the second sub-query time range.

[0099] If not, it can be determined that the target window data does not exist, and the query index information and the query time range are used as the second sub-query index range and the second sub-query time range respectively.

[0100] T2-2: Based on the second sub-query indicator information, the detailed data in the target data table that matches the second sub-query time range is calculated and processed to obtain calculated data.

[0101] In specific implementation, a virtualization engine can be used to perform indicator calculation on the detailed data in the target data table within the second sub-query time range according to the query indicator measurement and query indicator dimension indicated by the second sub-query indicator information to obtain calculated data.

[0102] T2-3: Determine the query result that matches the data query request based on the calculated data and the target window data in the materialized data table that matches the first sub-query indicator information and the first sub-query time range.

[0103] In specific implementation, the first sub-query indicator information can be extracted from the materialized data table to perform incremental indicator calculation, and the target window data whose materialized time range is the first sub-query time range can be integrated with the calculated data to obtain the query results and feedback to the user.

[0104] For example, you can extract the target window data hit by the query request from the materialized data table, perform real-time calculations on the detailed data within the second subquery time range, and obtain the calculation results. The target window data and the calculation results are then combined to obtain the query result.

[0105] like Figure 4 The figure shows a process diagram of a data query provided by an embodiment of the present disclosure, wherein the virtualization engine can receive a data query request initiated by a user and perform materialized matching on the data query request (performing the above-mentioned step T2-1), determining the target materialized view hit by the user (i.e., target window data) and the second sub-query index information and second sub-query time range that do not need to hit any materialized view. In this way, the data query request can be split into a materialized view sub-query for the materialized data table and a detailed data table sub-query for the target data table. The target materialized view (i.e., the materialized view sub-query result) is queried from the materialized data table, and based on the second sub-query index information and the second sub-query time range, the calculated data (i.e., the detailed data table sub-query result) is determined from the target data table. The target materialized view and the calculated data are merged to obtain the query result and fed back to the user.

[0106] In this way, this application can automatically split the query request into materialized view subqueries and detailed data subqueries according to the query time range, and then automatically merge the materialized view subquery results and detailed data subquery results to obtain the query results. This can realize the use of automatically materialized materialized window data, improve data query efficiency, and significantly improve query performance.

[0107] Overall, this application significantly improves query performance through dynamic query optimization based on metrics, splitting query requests into materialized view subqueries and detailed data subqueries. The incremental materialization mechanism, which incrementally materializes cleaned detailed data in a timely manner, reduces storage overhead and improves computing resource utilization. Data materialization and querying are implemented using a virtualization engine, eliminating the need for deploying additional external components and reducing operational complexity.

[0108] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0109] Based on the same inventive concept, the embodiment of the present disclosure also provides a data materialization device corresponding to the data materialization method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data materialization method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0110] like Figure 5 FIG. 1 is a schematic diagram of a data materialization device provided by an embodiment of the present disclosure, comprising: A creation module 501 is configured to create a source virtual table based on the data subject and data metadata of a streaming data source, and to create a business virtual table for the source virtual table; the source virtual table is configured to index streaming data in the streaming data source; the business virtual table includes data cleansing rules for the data metadata in the source virtual table; A cleaning module 502 is configured to clean the streaming data in the streaming data source that meets the data cleaning rules in real time according to the data cleaning rules and the index relationship in the source virtual table to obtain a target data table; An acquisition module 503 is configured to acquire materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information includes at least indicator metrics, indicator dimensions, and indicator increment columns; The materialization module 504 is used to perform incremental materialization processing on the detailed data in the target data table according to the materialization indicator information and the materialization time granularity to obtain a materialized data table; the materialized data table includes the materialization window data corresponding to each materialization processing and the materialization time range of the materialized window data.

[0111] In a possible implementation, the apparatus further includes a query module 505, which, after obtaining the materialized data table, is configured to: Obtaining a user's data query request and determining the query indicator information and query time range corresponding to the data query request; According to the query index information and the query time range, a query result matching the data query request is determined from the materialized window data and / or the target data table.

[0112] In a possible implementation, the query module 505, when determining a query result matching the data query request from the materialized window data and the target data table based on the query indicator information and the query time range, is configured to: Splitting the query index information and the query time range according to the materialized time range and the materialized index information to obtain first sub-query index information and a first sub-query time range for which matching target window data exists, and second sub-query index information and a second sub-query time range for which matching materialized window data does not exist; performing calculation processing on the detailed data in the target data table that matches the second subquery time range according to the second subquery indicator information to obtain calculated data; A query result matching the data query request is determined based on the calculated data and the target window data in the materialized data table that matches the first sub-query indicator information and the first sub-query time range.

[0113] In a possible implementation, the creation module 501, when creating a source virtual table matching the data subject and data metadata of the streaming data source and creating a business virtual table for the source virtual table, is configured to: Determine the table structure and time column of the source virtual table that matches each data subject in the streaming data source; the time column is used to indicate the time information of the streaming data; Determine data meta information associated with a source virtual table according to the data meta information of the stream data under the data subject in the streaming data source; Creating a post source virtual table for each of the data topics according to the table structure, the time column and the data meta information; For each of the source virtual tables, data cleaning rules for the data metadata in the source virtual table are determined according to data processing requirements related to the data subject, and a business virtual table corresponding to the source virtual table is created according to the data cleaning rules.

[0114] In one possible implementation, the cleaning module 502, when performing real-time cleaning on the streaming data in the streaming data source that meets the data cleaning rules according to the data cleaning rules and the index relationship in the source virtual table to obtain the target data table, is configured to: Continuously acquiring target stream data that needs to be cleaned from the stream data mapped by the source virtual table in the streaming data source according to the data meta-information targeted by the data cleansing rule, the index relationship corresponding to the data meta-information in the source virtual table, and the index relationship corresponding to the data subject matched by the source virtual table; Clean the target stream data according to the data cleaning rules to obtain cleaned detailed data; Determine a target table structure of a target data table according to the data cleaning rule, and create a target data table according to the target table structure; The detailed data is stored in the target data table using the primary key specified for the target data table.

[0115] In a possible implementation, the indicator metric is used to indicate an aggregate calculation operation on the first target column in the target data table; The indicator dimension is used to indicate a grouping operation on multiple second target columns in the target data table; The indicator increment column is used to indicate the time column of the detailed data in the target data table and the time format information under the time column.

[0116] In a possible implementation, the materialization module 504, when performing incremental materialization processing on the detailed data in the target data table according to the materialization indicator information and the materialization time granularity to obtain the materialized data table, is configured to: According to the materialization time granularity, the materialization processing task is triggered regularly; Using the materialized processing task, determining each target incremental data in the target data table that matches the materialized time range corresponding to the materialized time granularity according to the time information of the detailed data in the target data table under the indicator increment column; Incrementally materialize the target incremental data according to the indicator metric and the indicator dimension to obtain materialized window data that matches the materialization time range, and store the materialized window data in a materialized data table according to the materialization time range.

[0117] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0118] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 6 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention, comprising: Processor 601, memory 602 and bus 603. Memory 602 stores machine-readable instructions executable by processor 601. Processor 601 is configured to execute the machine-readable instructions stored in memory 602. When the machine-readable instructions are executed by processor 601, processor 601 performs the following steps: S101: creating a source virtual table based on the data subject and data metadata of a streaming data source, and creating a business virtual table for the source virtual table; the source virtual table is used to index streaming data in a streaming data source; the business virtual table includes data cleaning rules for the data metadata in the source virtual table; S102: According to the rules and the index relationship in the source virtual table, the streaming data in the streaming data source that meets the data cleaning rules is cleaned in real time to obtain the target data table; S103: obtaining the materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information at least includes indicator measurement, indicator dimension and indicator increment column; and S104: according to the materialized indicator information and materialized time granularity, the detailed data in the target data table is incrementally materialized to obtain a materialized data table; the materialized data table includes the materialized window data corresponding to each materialization processing and the materialized time range of the materialized window data.

[0119] The above-mentioned memory 602 includes a memory 6021 and an external memory 6022; the memory 6021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 601, as well as the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021. When the computer device is running, the processor 601 and the memory 602 communicate through the bus 603, so that the processor 601 executes the execution instructions mentioned in the above method embodiment.

[0120] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the data materialization method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0121] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the software update method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0122] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0125] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0126] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0127] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0128] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A data materialization method, characterized in that: include: Creating a source virtual table based on the data subject and data metadata of the streaming data source, and creating a business virtual table for the source virtual table; the source virtual table is used to index the streaming data in the streaming data source; the business virtual table includes data cleaning rules for the data metadata in the source virtual table; According to the data cleaning rules and the index relationship in the source virtual table, the streaming data in the streaming data source that meets the data cleaning rules is cleaned in real time to obtain a target data table; Obtaining materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information includes at least indicator measurement, indicator dimension, and indicator increment column; According to the materialization index information and the materialization time granularity, incremental materialization processing is performed on the detailed data in the target data table to obtain a materialized data table; the materialized data table includes materialized window data corresponding to each materialization processing and the materialization time range of the materialized window data.

2. The method according to claim 1, characterized in that After obtaining the physical and chemical data table, the method further includes: Obtaining a user's data query request and determining the query indicator information and query time range corresponding to the data query request; According to the query index information and the query time range, a query result matching the data query request is determined from the materialized window data and / or the target data table.

3. The method according to claim 2, characterized in that Determining a query result matching the data query request from the materialized window data and the target data table according to the query indicator information and the query time range includes: Splitting the query index information and the query time range according to the materialized time range and the materialized index information to obtain first sub-query index information and a first sub-query time range for which matching target window data exists, and second sub-query index information and a second sub-query time range for which matching materialized window data does not exist; performing calculation processing on the detailed data in the target data table that matches the second subquery time range according to the second subquery indicator information to obtain calculated data; A query result matching the data query request is determined based on the calculated data and the target window data in the materialized data table that matches the first sub-query indicator information and the first sub-query time range.

4. The method according to claim 1, wherein The step of creating a source virtual table that matches the data subject and data meta information of the streaming data source, and creating a business virtual table for the source virtual table, includes: Determine the table structure and time column of the source virtual table that matches each data subject in the streaming data source; the time column is used to indicate the time information of the streaming data; Determining data meta information associated with a source virtual table according to the data meta information of the stream data under the data subject in the streaming data source; Creating a post source virtual table for each of the data topics according to the table structure, the time column and the data meta information; For each of the source virtual tables, data cleaning rules for the data metadata in the source virtual table are determined according to data processing requirements related to the data subject, and a business virtual table corresponding to the source virtual table is created according to the data cleaning rules.

5. The method according to claim 1, wherein The step of performing real-time cleaning on the streaming data in the streaming data source that meets the data cleaning rules according to the data cleaning rules and the index relationship in the source virtual table to obtain a target data table includes: Continuously acquiring target stream data that needs to be cleaned from the stream data mapped by the source virtual table in the streaming data source according to the data meta-information targeted by the data cleansing rule, the index relationship corresponding to the data meta-information in the source virtual table, and the index relationship corresponding to the data subject matched by the source virtual table; Clean the target stream data according to the data cleaning rules to obtain cleaned detailed data; Determine a target table structure of a target data table according to the data cleaning rule, and create a target data table according to the target table structure; The detailed data is stored in the target data table using the primary key specified for the target data table.

6. The method according to claim 1, characterized in that The indicator metric is used to indicate an aggregate calculation operation on the first target column in the target data table; The indicator dimension is used to indicate a grouping operation on multiple second target columns in the target data table; The indicator increment column is used to indicate the time column of the detailed data in the target data table and the time format information under the time column.

7. The method according to claim 1, characterized in that The incremental materialization processing is performed on the detailed data in the target data table according to the materialization index information and the materialization time granularity to obtain a materialized data table, including: According to the materialization time granularity, the materialization processing task is triggered regularly; Using the materialized processing task, determining each target incremental data in the target data table that matches the materialized time range corresponding to the materialized time granularity according to the time information of the detailed data in the target data table under the indicator increment column; Incrementally materialize the target incremental data according to the indicator metric and the indicator dimension to obtain materialized window data that matches the materialization time range, and store the materialized window data in a materialized data table according to the materialization time range.

8. A data materialization device, characterized in that: include: a creation module, configured to create a source virtual table based on the data subject and data metadata of the streaming data source, and to create a business virtual table for the source virtual table; the source virtual table is configured to index the streaming data in the streaming data source; the business virtual table includes data cleaning rules for the data metadata in the source virtual table; a cleaning module, configured to clean the streaming data in the streaming data source that meets the data cleaning rules in real time according to the data cleaning rules and the index relationship in the source virtual table, to obtain a target data table; An acquisition module, configured to acquire materialized indicator information and materialized time granularity related to the target data table; the materialized indicator information includes at least indicator metrics, indicator dimensions, and indicator increment columns; The materialization module is used to perform incremental materialization processing on the detailed data in the target data table according to the materialization indicator information and the materialization time granularity to obtain a materialized data table; the materialized data table includes the materialization window data corresponding to each materialization processing and the materialization time range of the materialized window data.

9. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the data materialization method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a computer device, the computer device executes the steps of the data materialization method according to any one of claims 1 to 7.

Images