Materialized view processing method, related device and cloud time series data storage system

Abstract

The embodiment of the invention discloses a materialized view processing method, a related device and a cloud time series data storage system. The method comprises the following steps: pulling a data stream from a source data fragment contained in a source database; according to a specified time interval, executing aggregation calculation on the pulled data of each time interval to obtain aggregated data corresponding to each time interval; storing the aggregated data corresponding to each time interval into materialized view data fragments contained in a materialized view database; different materialized view data fragments are responsible for data storage of different dimension combinations. According to the embodiment of the invention, the constructed materialized view database supports dimension aggregation and time aggregation at the same time, the data input magnitude of the materialized view database can be greatly reduced, and the resource consumption required for constructing the materialized view database is reduced; therefore, a materialized view database in an infinite time range can be constructed on the source database.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a materialized view processing method and related apparatus, and a cloud time-series data storage system. Background Technology

[0002] Time series data, also known as time sequence data, refers to data recorded in chronological order, with each data point corresponding to a specific point in time. In cloud time series database scenarios, the intervals at which data is written to the cloud time series database are typically low time intervals such as milliseconds, seconds, and minutes. Therefore, cloud time series databases need to handle massive amounts of data write requests and perform analysis and queries based on these massive amounts of data. This often results in data query implementation consuming a large amount of computing resources.

[0003] Therefore, how to provide a time-series data storage solution that can reduce the resources consumed by data query is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a materialized view processing method and apparatus, a cloud time-series data storage system, an electronic device, a computer-readable storage medium, and a computer program product. Embodiments of this application create materialized views based on a source database, thereby enabling time-series data queries based on the materialized views, thus reducing the computational resources consumed in time-series data queries.

[0005] One aspect of this application provides a materialized view processing method, which includes: pulling a data stream from source data shards contained in a source database; performing aggregation calculations on the data of each time interval according to a specified time interval to obtain aggregated data corresponding to each time interval; storing the aggregated data corresponding to each time interval into materialized view data shards contained in the materialized view database; and different materialized view data shards being responsible for storing data of different dimension combinations.

[0006] In another exemplary embodiment, pulling a data stream from a source data shard contained in a source database includes: determining a start time point and an end time point corresponding to the data stream to be pulled; pulling the write-ahead log of the source data shard, and obtaining the data stream recorded from the start time point from the write-ahead log; after the source data shard successfully writes data, it records the written data in the write-ahead log; when the end time point is reached, pulling the write-ahead log stops.

[0007] In another exemplary embodiment, retrieving the write-ahead log from the source data shard and obtaining the data stream recorded from the write-ahead log starting from the start time point includes: decoding the retrieved write-ahead log to obtain a decoded write-ahead log; filtering out non-write type data from the decoded write-ahead log to obtain a filtered write-ahead log; and parsing the filtered write-ahead log to obtain the data stream recorded from the start time point.

[0008] In another exemplary embodiment, the method further includes: obtaining the log sequence number corresponding to the data recorded in each time interval from the write-ahead log; after determining that the aggregate data corresponding to any time interval is stored in the materialized view data shard, submitting the corresponding log sequence number to the source data shard, so that the source data shard performs data deletion processing on the write-ahead log based on the submitted log sequence number.

[0009] In another exemplary embodiment, according to a specified time interval, aggregation calculations are performed on the data streams of each time interval to obtain the aggregated data corresponding to each time interval, including: each time interval is reached, aggregation calculations are performed on the data in the currently reached time interval to obtain the aggregated data corresponding to the current time interval;

[0010] The process of storing the aggregated data corresponding to each time interval into the materialized view data fragments contained in the materialized view database includes: storing the aggregated data corresponding to the currently arrived time interval into the materialized view data fragments contained in the materialized view database.

[0011] In another exemplary embodiment, storing the aggregated data corresponding to each time interval into a materialized view data shard contained in the materialized view database includes: obtaining a specified materialized view input format; the materialized view input format includes at least one of a field for marking the database table to which the aggregated data belongs, a field for characterizing the aggregated dimension, a field for characterizing the operator calculation value, a field for characterizing the operator type, and a field for characterizing the time interval to which the aggregated data belongs; and inputting the aggregated data corresponding to each time interval into the materialized view data shard for storage according to the materialized view input format.

[0012] In another exemplary embodiment, storing the aggregated data corresponding to each time interval into the materialized view data shards contained in the materialized view database includes: obtaining the materialized view routing rules sent by the scheduler; the materialized view routing rules are generated by the scheduler after querying the materialized view meta-database to obtain the dimension combinations corresponding to each materialized view data shard; and according to the materialized view routing rules, inputting the aggregated data corresponding to each time interval into the corresponding materialized view data shards for storage.

[0013] In another exemplary embodiment, the method of pulling a data stream from source data shards contained in a source database further includes: receiving an aggregation computation task sent by the scheduler; the scheduler, in response to a materialized view creation request, after determining the source data shard to which the data to be aggregated belongs, generating an aggregation computation task to instruct the pulling of a data stream from the corresponding source data shard, wherein different aggregation computation tasks correspond to different source data shards; and pulling a data stream from the source data shards contained in the source database based on the aggregation computation task.

[0014] In another aspect of this application, a materialized view processing apparatus is provided. The apparatus includes: a pull module configured to pull a data stream stored in memory for a specified duration from source data shards contained in a source database; an aggregation module configured to perform aggregation calculations on the data for each time interval according to a specified time interval, so as to obtain aggregated data corresponding to each time interval; and a storage module configured to store the aggregated data corresponding to each time interval into materialized view data shards contained in the materialized view database; different materialized view data shards are responsible for storing data of different dimension combinations.

[0015] Another aspect of this application provides a cloud time-series data storage system, comprising: a source database for storing time-series data, including multiple source nodes, each source node deploying at least one source data shard; a materialized view database, including multiple materialized view nodes, each materialized view node deploying at least one materialized view data shard, different materialized view data shards being responsible for data storage of different dimension combinations; and a compute node pool, including multiple compute nodes, each compute node being configured to pull data streams from at least one source node, and perform aggregation calculations on the data of each time interval according to a specified time interval to obtain aggregated data corresponding to each time interval, and input the aggregated data corresponding to each time interval into at least one materialized view data shard for storage.

[0016] Another aspect of this application provides an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the materialized view processing method as described above.

[0017] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the materialized view processing method as described above.

[0018] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor of an electronic device, implements the materialized view processing method as described above.

[0019] The technical solution provided by the embodiments of this application, on the one hand, constructs a materialized view database based on the source database. The materialized view database is composed of materialized view data shards, and different materialized view data shards are responsible for storing data of different dimension combinations. This makes the data stored in the materialized view database perform dimension aggregation on some dimensions of the data. Therefore, responding to data queries through the constructed materialized view database can save the online aggregation process, thereby saving the computing resources required for queries. On the other hand, the data written to the materialized view database is aggregated data obtained by performing aggregation calculations on the data in each time interval according to a specified time interval. This not only enables the constructed materialized view database to support both dimension aggregation and time aggregation, but also significantly reduces the data input volume of the materialized view database and reduces the resource consumption required to construct the materialized view database. Thus, it is possible to construct a materialized view database with an unlimited time range on the source database.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the implementation environment involved in this application;

[0022] Figure 2 This is a schematic diagram illustrating a cloud time-series data storage system, as shown in an exemplary embodiment of this application;

[0023] Figure 3 Is Figure 2 A schematic diagram of another cloud time-series data storage system proposed based on the previous embodiment;

[0024] Figure 4 This is an exemplary flowchart illustrating the creation operation performed on materialized view metadata;

[0025] Figure 5 This is an exemplary distribution diagram of aggregate computing tasks and access tasks;

[0026] Figure 6 This is a schematic diagram illustrating an exemplary aggregate computing task scheduling process;

[0027] Figure 7 This is a schematic diagram illustrating the process of a computing node performing an aggregate computing task.

[0028] Figure 8This is a flowchart illustrating a materialized view processing method in an exemplary embodiment of this application;

[0029] Figure 9 Is Figure 8 A flowchart of another materialized view processing method technically proposed in the illustrated embodiment;

[0030] Figure 10 This is a block diagram illustrating a materialized view processing apparatus according to an exemplary embodiment of this application;

[0031] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0036] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0037] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] Cloud time-series databases are characterized by the need to handle massive write requests and perform analytical queries based on massive amounts of data, which often results in data queries on cloud time-series databases consuming significant computing resources. Therefore, it is necessary to propose a time-series data storage solution that can reduce the resource consumption of data queries.

[0039] To address the aforementioned technical challenges, one approach is to pre-aggregate certain dimensions of the data. This eliminates the need for online aggregation during queries, conserving computational resources. A materialized view is a special database object that contains pre-computed data representing the results of a query and stores it in the database to improve query performance. Therefore, using materialized views allows for pre-aggregation of data dimensions, thereby accelerating queries.

[0040] In existing technical implementations, materialized view engines are single-machine engines, which are not suitable for distributed application scenarios such as cloud time-series databases. For example, if there are multiple single-machine materialized view engines in a distributed system, since each single-machine engine performs dimensional aggregation independently, the data for a single dimension is actually distributed across multiple nodes, and secondary aggregation is still required to achieve analysis and querying.

[0041] Therefore, embodiments of this application propose a cloud time-series data storage system based on materialized views, as well as a materialized view processing method, a materialized view processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. These embodiments will be described in detail below.

[0042] Please see first. Figure 1 , Figure 1 This is a schematic diagram of the implementation environment involved in this application. The implementation environment includes a terminal 110 and a server 120, which communicate with each other via wired or wireless means.

[0043] Terminal 110 is a device for information interaction with the user. It receives user input requests to create, delete, modify, and query materialized views, and returns the operation results to the user. For example, terminal 110 is configured with a graphical user interface (GUI), through which the user can interact with terminal 110. For instance, the user triggers the GUI to input request content, and the user can also obtain the request result from the GUI. Alternatively, terminal 110 may also be configured with input devices such as a keyboard and mouse to facilitate information interaction with the user. Terminal 210 can be a computer, smartphone, tablet, laptop, smart home appliance, etc., without limitation.

[0044] Server 120 provides data services for information interaction between terminal 110 and the user. Of course, if terminal 110 does not require data services, it may not interact with server 120. In the embodiments of this application, server 120 is a server cluster or distributed system composed of multiple physical servers, specifically implemented as a cloud time-series database, also known as a cloud time-series data storage system. It is used to store massive amounts of time-series data, and to perform analysis and queries based on this massive data, and supports the construction of materialized views. Detailed materialized view construction schemes will be described in subsequent embodiments and will not be elaborated here.

[0045] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram illustrating a cloud time-series data storage system as an exemplary embodiment of this application. This cloud time-series data storage system can be applied to... Figure 1 The implementation environment shown, for example, the cloud time-series data storage system can be deployed as server 120.

[0046] like Figure 2 As shown, the cloud time-series data storage system includes a source database 210, a materialized view database 220, and a computing node pool 230.

[0047] Source database 210 is used to store time-series data; it can also be understood as providing source data to materialized view database 220. Therefore, the data stored in source database 210 is massive. Source database 210 is a distributed database, including multiple source nodes, with at least one source data shard deployed on each source node. This embodiment does not limit the amount of source node data contained in source database 210, nor does it limit the number of source data shards deployed on each source node. Generally, one source data shard is deployed on each source node.

[0048] The materialized view database 220 is used to perform dimensional aggregation on the time-series data stored in the source database 210. Since the materialized view database 220 stores data obtained by pre-aggregating some dimensions of the time-series data, data queries based on the materialized view database 220 can eliminate the need for subsequent aggregation, thus saving computational resources. To adapt to distributed application scenarios, the materialized view database 220 is also a distributed database, including multiple materialized view nodes. Each materialized view node deploys at least one materialized view data shard, and different materialized view data shards are responsible for storing data of different dimensional combinations. In other words, based on the distributed configuration of the materialized view database 220, different materialized view data shards are used to implement data aggregation of different dimensional combinations. Unlike materialized view implementations based on a single-machine engine, where data for a single dimension is actually distributed across multiple materialized view nodes, the materialized view database 220 can directly provide data queries to users without requiring secondary dimensional aggregation. It should be noted that the embodiments of this application do not limit the dimensional data contained in the dimensional combination corresponding to each materialized view data piece, which can be set based on actual application requirements.

[0049] The compute node pool 230 provides multiple compute nodes, each of which executes stream computation logic. Each compute node executing stream computation logic means that it pulls a data stream from at least one source node, performs aggregation calculations on the data for each specified time interval, obtains aggregated data for each time interval, and then inputs the aggregated data for each time interval into at least one materialized view data shard for storage, thereby realizing the construction of the materialized view.

[0050] The following is through Figure 3-7 The content shown here provides a detailed introduction to the stream computing logic executed by the computing node.

[0051] First of all, it should be noted that Figure 3 Is Figure 2A schematic diagram of another cloud time-series data storage system proposed based on the previous embodiment. Figure 4 This is an exemplary flowchart illustrating the creation operation performed on the materialized view metadata. Figure 5 This is an exemplary distribution diagram of aggregation computing tasks and access tasks. Figure 6 This is a schematic diagram illustrating an exemplary aggregate computing task scheduling process. Figure 7 This is a schematic diagram illustrating an exemplary computing node performing an aggregate computing task.

[0052] like Figure 3 As shown, the cloud time-series data storage system further includes a materialized view metadata database 240 and a scheduler 250.

[0053] The materialized view metadata database 240 is used to persist the created materialized view metadata objects. It can also be understood as storing the metadata corresponding to the materialized view database 220, such as the dimension combination information corresponding to each materialized view data shard. The materialized view metadata database 240 can be a distributed database or a regular database; no restriction is placed here.

[0054] Scheduler 250 periodically retrieves metadata from the materialized view metadata database 240, performs scheduling calculations in response to input requests, generates corresponding scheduling tasks, and assigns these tasks. Generally, the scheduling calculations performed by scheduler 250 are based on materialized view creation requests. Based on the materialized view creation requests and the metadata retrieved from the materialized view metadata database 240, scheduler 250 generates aggregation calculation tasks and assigns these tasks to at least one computing node in the computing node pool 230.

[0055] like Figure 4 For example, after receiving a materialized view creation request initiated by a user terminal, the compute node generates a corresponding metadata object operation. The materialized view creation request contains a materialized view statement, which describes the materialized view rules. Therefore, the metadata object operation instructs the creation of the materialized view database 220 corresponding to the materialized view rules and calls the management interface of the materialized view metadata database 240 to persistently store the metadata of the created materialized view database 220. The scheduler 250, by executing a scheduling process, retrieves metadata from the materialized view metadata database 240 and generates and allocates corresponding aggregation computing tasks.

[0056] For example, the scheduler 250 determines the source data shard to which the data to be aggregated belongs, generates a corresponding number of aggregation computing tasks based on the number of source data shards, and different aggregation computing tasks indicate that data should be pulled from different source data shards and aggregation computing should be performed. Then, the scheduler determines the computing nodes to which the aggregation computing tasks are to be assigned from the computing node pool 230, and assigns the aggregation computing tasks to the determined computing nodes. Figure 6 As an example, the data to be aggregated belongs to four source data shards. Therefore, the scheduler 250 generates four aggregation computing tasks accordingly, and assigns aggregation computing task 1 and aggregation computing task 2 to computing node 1, and aggregation computing task 3 and aggregation computing task 4 to computing node 2.

[0057] The embodiments of this application do not limit the specific method by which the scheduler 250 determines the computing nodes to be assigned to the aggregate computing tasks. For example, as an exemplary implementation, the scheduler 250 determines the computing nodes to be assigned to the aggregate computing tasks based on the load data of each computing node in the computing node pool 230. Alternatively, the scheduler 250 may also determine the computing nodes to be assigned to the aggregate computing tasks in other ways, such as determining which computing nodes or nodes will execute the aggregate computing tasks according to preset allocation rules.

[0058] In another exemplary embodiment, the aggregation computation task generated by scheduler 250 also includes materialized view routing rules. These materialized view routing rules instruct compute nodes to input the aggregated data obtained from performing the aggregation computation into specific materialized view data shards within the materialized view database 220. For example, assuming the materialized view database 220 contains materialized view data shard 1 and materialized view data shard 2, where materialized view data shard 1 is responsible for storing data related to dimension combination 1 and materialized view data shard 2 is responsible for storing data related to dimension combination 2, the aggregation computation task generated by scheduler 250 includes materialized view routing rules instructing compute nodes to route data related to dimension combination 1 to materialized view data shard 1 and data related to dimension combination 2 to materialized view data shard 2.

[0059] Figure 3 The diagram also illustrates that the applications deployed on the compute nodes in compute node pool 230 include access response programs and stream computing programs. The stream computing program is the stream computing logic executed by the compute node. The access response program is responsible for responding to materialized view operations initiated by user terminals, such as creating, deleting, modifying, and querying materialized views. Querying materialized views can be further divided into querying materialized view metadata and querying materialized view status. The access response program can also handle input read / write operations in source database 210.

[0060] In some exemplary embodiments, to save costs, the access response program and the stream computing program run on the same computing node. In this case, a flow control policy can be added to the aggregate computing task. The flow control policy indicates that when the computing node's computing resource margin is lower than a preset limit, the access response program is triggered to perform flow limiting, but this does not affect the access response program's normal read / write operations. The computing resources of the computing node, such as CPU (Central Processing Unit) or memory, are not limited here.

[0061] In some exemplary embodiments, to achieve isolation between aggregation computing tasks and access tasks, the stream computing program and the access response program can run on different computing nodes, which can improve the overall performance of the cloud time-series data storage system. For example Figure 5 As shown, aggregation computing tasks 1 and 2 are executed by computing node 1, while the access task is executed by computing node 2, to achieve data read / write to the source database 210. It should be noted that the access task mentioned in this embodiment corresponds to the access response program; therefore, the specific content of the access task is related to the specific input request. Figure 5 The content shown does not imply that the access task is limited to data read / write tasks targeting the source database 210.

[0062] Please continue reading. Figure 7 The process of a compute node executing an aggregation computation task can include four stages: pulling data from the source data shards, performing aggregation computation based on a specified time interval, uploading the aggregated data corresponding to each time interval to the materialized view database 220, and submitting the log sequence number to the source data shards. Each stage will be described in detail below.

[0063] (1) The stage of fetching data from source data fragments

[0064] The compute node determines the start and end times of the data stream to be pulled based on the aggregation compute task. If the aggregation compute task indicates that a data stream written within a specified duration should be pulled, then the corresponding start and end times can be determined. If the aggregation compute task indicates that a data stream written within an unlimited duration should be pulled, then the specific start time can be determined, and the end time can be a null value or other preset value, such as infinity.

[0065] After successfully writing data, the source data shard records the written data in the Write-Ahead Logging (WAL). Therefore, compute nodes can retrieve the data stream from the source data shard by pulling the WAL from the source data shard and obtaining the data stream recorded from the start time point. When the end time point is reached, pulling the WAL stops, thus enabling the retrieval of the data stream from the source data shard.

[0066] In some exemplary embodiments, the compute node decodes the pulled write-ahead log to obtain the decoded write-ahead log, filters out non-write type data from the decoded write-ahead log to obtain the filtered write-ahead log, and then parses the data recorded from the start time point from the filtered write-ahead log.

[0067] (2) The stage of performing aggregate calculations based on a specified time interval

[0068] The computing nodes perform aggregation calculations on the data retrieved for each time interval according to the specified time interval, so as to obtain the aggregated data corresponding to each time interval. The specified time interval can be specified by the scheduler 250, and this embodiment does not limit the specific way the scheduler 250 specifies the time interval.

[0069] In some exemplary embodiments, whenever a time interval is reached, the compute node performs aggregation calculations on the data pulled within the currently reached time interval to obtain the aggregated data corresponding to the current time interval. For example, assuming a time interval is 20 minutes, whenever the compute node pulls a data stream within a 20-minute period, it performs aggregation calculations on this data stream to obtain the aggregated data corresponding to those 20 minutes.

[0070] Therefore, this embodiment performs aggregation calculations based on a specified time interval. This can be understood as performing data aggregation at the time granularity of the specified time interval within the computing node, and then storing the aggregated data in the materialized view database 220. This significantly reduces the amount of data input to the materialized view database 220, thereby reducing the write performance consumption on the materialized view side and improving the overall performance of the cloud time-series data storage system. Furthermore, it can be understood that the construction of the materialized view database 220 achieves dimensional aggregation of data. Therefore, the technical solution of this embodiment implements two aggregations: time aggregation and dimensional aggregation.

[0071] Furthermore, if a compute node experiences a crash or other anomaly and then restarts, it does not need to fetch all the data again, thereby reducing the amount of computing resources required when the compute node restarts. This also improves the overall performance of the cloud time-series data storage system to some extent.

[0072] In addition, this embodiment can achieve near real-time data persistence in real-time scenarios, instead of persisting the data to the materialized view database 220 only after the full amount of data has been retrieved and the aggregation calculation has been completed, which further improves the overall performance of the cloud time-series data storage system.

[0073] (3) Upload the aggregated data corresponding to each time interval to the materialized view database 220.

[0074] After obtaining the aggregated data corresponding to a time interval, the computing node can upload the aggregated data corresponding to this time interval to the materialized view database 220.

[0075] In some exemplary embodiments, considering that different operators in the materialized view statement contained in the materialized view creation request may require different data formats, a materialized view input format is specified.

[0076] The operators in a materialized view statement can be understood as the computational logic that needs to be performed on the data to build the materialized view database 220. For example, for the min / max / sum / count operators, only the values ​​are needed for dimensional aggregation in the materialized view database 220. However, for the mean operator, the values ​​of both the count and sum operators are required for dimensional aggregation. If each materialized view data shard contains only the value of one of the operators, dimensional aggregation cannot be achieved. The first / last operators require an additional time field. If only the values ​​are included, each materialized view data shard cannot obtain the actual result by performing dimensional aggregation. Based on these circumstances, the materialized view input format specified in this embodiment includes at least one of the following: a field for identifying the database table to which the aggregated data belongs, a field for representing the aggregated dimension, a field for representing the operator's calculated value, a field for representing the operator type, and a field for representing the time interval to which the aggregated data belongs.

[0077] For example, the specified materialized view input format is represented as follows:

[0078] measurement tags fieldname=value, type=xxx,extend ＿ field = time;

[0079] Here, `measurement` is a field used to identify the database table to which the aggregated data belongs. `tags` is a field used to characterize the aggregation dimensions, identifying which data needs to be aggregated together; it can also be called dimension combination. `fieldname=value` is a field used to characterize the calculated value of the operator (field). `type=xxx` is a field used to characterize the operator type, indicating the type of calculation required by the operator. Different operators in the materialized view have different aggregation methods, therefore, it is necessary to indicate the calculation type of the operator. ＿ field=time is a field used to characterize the time interval to which the aggregated data belongs. ＿ The field is an additional data field used for auxiliary calculations in the materialized view, such as for the first / last operator, extend.＿ field represents a specific time point; for the mean operator, extend ＿ The field represents the values ​​of the `count` and `sum` operators. It's understandable that `extend`... ＿ The fields are also different, but the corresponding extends of the same operator type ＿ The fields should be consistent. Time is used to represent the time interval to which the input aggregate data belongs. For example, if the time interval is 20 minutes, the start time of the data stream is 12:00, and the end time is 13:00, then the Time corresponding to each write of aggregate data is 12:00, 12:20, 12:40, and 13:00 respectively.

[0080] Therefore, this embodiment ensures the implementation of dimensional aggregation in the materialized view database 220 by setting a specified materialized view input format, thereby ensuring the implementation of the two aggregation mechanisms, namely time aggregation and dimensional aggregation.

[0081] In some exemplary embodiments, a materialized view engine is deployed on each materialized view node, which is used to store aggregated data in the materialized view data shards.

[0082] For example, the process of materializing view engines storing aggregated data may include the following steps:

[0083] S1, after receiving the aggregated data corresponding to any time interval, determine whether there is historically stored aggregated data in the materialized view data shard;

[0084] S2, perform aggregation calculations on the historically stored aggregated data and the aggregated data corresponding to the currently received time interval, and update and store the obtained aggregated data on the materialized view data shards;

[0085] S3 directly stores the aggregated data corresponding to the currently received time interval in the materialized view data fragment.

[0086] It is important to understand that in S1, it is determined whether there is historically stored aggregate data in the materialized view data shard. If not, it means that the aggregate data of the currently received time interval is the first time that the materialized view engine has received aggregate data of the time interval. Therefore, S3 is executed to directly store the aggregate data corresponding to the currently received time interval in the materialized view data shard.

[0087] If yes, it means that the aggregated data for the currently received time interval is not the first time the materialized view engine has received aggregated data for that time interval. Historical aggregated data exists in the materialized view data shards. Therefore, S2 is executed, performing aggregation calculations on this historically stored aggregated data and the aggregated data corresponding to the currently received time interval, and then updating and storing the calculated aggregated data on the view data shards. It can be understood that updating and storing the calculated aggregated data on the materialized view data shards means deleting the historically stored aggregated data from the materialized view data shards and then storing the calculated aggregated data on the materialized view data shards.

[0088] Therefore, it can be concluded that the materialized view engine also adopts an aggregation mechanism to store the aggregated data corresponding to each time interval on the materialized view data shards. Based on this aggregation mechanism, the materialized view data shards do not store the aggregated data corresponding to each time interval separately, but perform aggregated storage. This can significantly reduce the data storage volume of the materialized view database and reduce the storage resource consumption of the materialized view database, thereby further improving the overall performance of the cloud time-series data storage system.

[0089] For example, the materialized view engine may implement the storage of aggregated data into materialized view data shards by including a first process of writing aggregated data into a memory table and a second process of sinking the data in the memory table to disk.

[0090] In the first process, the values ​​corresponding to the same key in the aggregated data are aggregated to obtain the aggregated data, which is then written to an in-memory table. This in-memory table, such as a memtable, is primarily used to store and manage data to support fast data read and write operations. The in-memory format can be organized as a dictionary, storing data in key-value pairs. The key can be a fieldkey, and the value can be a schema (a table describing the database structure and organization). This can be represented in the following organizational format:

[0091] fieldkey: measurement+value_filed_name+tags+type

[0092] values: value+extend_vector+time

[0093] It can be seen that the organization format of the fieldkey is the same for different operator types, but the table structure of the values ​​will vary depending on the different extends in the aggregated data. ＿The field may vary. Therefore, in this embodiment, after performing aggregation calculations on values ​​with the same fieldkey, the aggregated data is then written to an in-memory table, which can reduce the computational resource consumption of the materialized view engine.

[0094] For example, performing aggregation calculations on values ​​with the same fieldkey can include several methods. For instance, the min / max / sum / count operators may not have an extend function. ＿ For the field, simply aggregate a column of data and select the first "time" column. For the first / last operators, you need to select "extend". ＿ For additional time fields in the `field` operator, `first` selects the value corresponding to the additional time field with the smallest value and retains the original time field; `last` does the same. The `mean` operator needs to first aggregate and calculate `extend`. ＿ The count / sum in the field is calculated, and the mean is calculated based on the aggregated count / sum.

[0095] In the second process, the fields representing the aggregation dimensions in the keys of the memory table are replaced with time series identifiers, and then the replaced data is dumped to disk. To reduce storage pressure on the disk, this embodiment sets the disk organization format to be different from the memory format. The specific disk organization format is as follows:

[0096] fieldkey:measurement+tsid+value field+type

[0097] value_vector:value+extend_vector+time

[0098] As can be seen, the most significant difference between the disk organization format and the memory format is that the `tags` field has been replaced with the `tsid` (Time Series Identifiers) field. The time series identifiers are used to uniquely identify each time series in the source database 210, and a unique time series identifier is assigned to each time series in the source database 210. Therefore, based on the time series identifiers, the amount of data stored on the disk can be further reduced, which can further improve the overall performance of the cloud time series data storage system.

[0099] (4) The stage of submitting log sequence numbers to source data shards

[0100] The compute node also retrieves the Log Sequence Number (LSN) corresponding to the data recorded within each time interval from the write-ahead log. The LSN is used to identify the position of the data recorded in the log file. After successfully uploading the corresponding materialized view storage shard for the aggregate data of any time interval, the compute node also submits the corresponding LSN to the source data shard, so that the source data shard can perform data deletion processing on the write-ahead log based on the submitted LSN.

[0101] In some exemplary embodiments, the log sequence number submitted by the compute node to the source data shard can be a log sequence number range corresponding to this time interval. Based on the submitted log sequence number range, the source data shard deletes the data in the write-ahead log that corresponds to this log sequence number range.

[0102] For example Figure 7 As shown, after the compute node uploads the aggregated data 1 corresponding to the first time interval to the materialized view database 220, it submits a log sequence number range of 100-200 to the source data shard 1. The source data shard 1 then deletes the data corresponding to log sequence number range 100-200 from the write-ahead log. After the compute node uploads the aggregated data 2 corresponding to the second time interval to the materialized view database 220, it submits a log sequence number range of 200-300 to the source data shard 1. The source data shard 1 then deletes the data corresponding to log sequence number range 200-300 from the write-ahead log.

[0103] In some other exemplary embodiments, the log sequence number submitted by the compute node to the source data shard can be the log sequence number corresponding to the last data in this time interval. Based on the submitted log sequence number, the source data shard deletes the data in the write-ahead log that is located before this log sequence number.

[0104] Still Figure 7 As shown, after the compute node uploads the aggregated data 1 corresponding to the first time interval to the materialized view database 220, it submits a log sequence number of 200 to the source data shard 1. The source data shard 1 then deletes the data before log sequence number 200 from the write-ahead log, which means deleting the data corresponding to log sequence number 100-200 from the write-ahead log. After the compute node uploads the aggregated data 1 corresponding to the second time interval to the materialized view database 220, it submits a log sequence number of 300 to the source data shard 1. The source data shard 1 then deletes the data before log sequence number 300 from the write-ahead log, which means deleting the data corresponding to log sequence number 200-300 from the write-ahead log.

[0105] As can be seen from the above example, this embodiment can reduce the resource consumption required for the write-ahead log by deleting the data in the write-ahead log in a timely manner, thereby improving the overall performance of the cloud time-series data storage system to a certain extent.

[0106] Therefore, it can be concluded that, based on Figure 2-7 The example cloud time-series data storage system enables the construction of materialized views on top of a distributed time-series database (i.e., source database 210), allowing for the specification of an unlimited time range and supporting both dimensional and temporal aggregation. Furthermore, since temporal aggregation is performed on the compute nodes before dimensional aggregation in the materialized view database 220, the data input volume of the materialized view can be significantly reduced, thereby reducing the resource consumption of the materialized view.

[0107] based on Figure 2-7 For an example of a cloud time-series data storage system, please refer to [link / reference]. Figure 8 The embodiments of this application also propose a materialized view processing method. This materialized view processing method can be... Figure 2 The computational nodes in the cloud time-series data storage system shown are executed, but this embodiment does not limit the materialized view processing method to be performed only by... Figure 2 The computing nodes in the cloud time-series data storage system shown are executing.

[0108] like Figure 8 As shown, in an exemplary embodiment, the materialized view processing method includes S810-S830, which are described in detail below:

[0109] S810 pulls a data stream from source data shards contained in the source database.

[0110] The source database is used to store time-series data; it can also be understood as providing source data to the materialized view database. The data stored in the source database is massive. The source database is a distributed database, comprising multiple source nodes, with at least one source data shard deployed on each source node. This embodiment does not limit the number of source nodes in the source database, nor does it limit the number of source data shards deployed on each source node. Generally, one source data shard is deployed on each source node.

[0111] The compute node pulls a data stream from the source data shards contained in the source database. This can be a data stream written within a specified time period from the source data shards, or a data stream written within an unlimited time period from the source data shards. This embodiment does not limit the time range to which the compute node pulls the data stream.

[0112] In some exemplary embodiments, the compute node determines the start time and end time corresponding to the data stream to be pulled, then pulls the write-ahead log of the source data shard, obtains the data stream recorded from the start time from the write-ahead log, and stops pulling the write-ahead log when the end time is reached.

[0113] It should be noted that if the data stream written within a specified duration is pulled from the source data shard, the corresponding start and end times can be determined. If the data stream written over an unlimited duration is pulled from the source data shard, the specific start time can be determined, while the end time can be an empty value or a preset value.

[0114] After successfully writing data, the source data shard records the written data in the write-ahead log. Therefore, compute nodes can retrieve the corresponding data stream from the source data shard by pulling the write-ahead log from the source data shard and obtaining the data stream recorded from the start time point. When the end time point is reached, pulling the write-ahead log stops, thus enabling the retrieval of the corresponding data stream from the source data shard.

[0115] In some exemplary embodiments, the compute node decodes the pulled write-ahead log to obtain the decoded write-ahead log, filters out non-write type data from the decoded write-ahead log to obtain the filtered write-ahead log, and then parses the data stream recorded from the start time point from the filtered write-ahead log.

[0116] S820 performs aggregation calculations on the data retrieved for each time interval according to the specified time interval, so as to obtain the aggregated data corresponding to each time interval.

[0117] The compute nodes perform aggregation calculations on the data retrieved for each time interval according to the specified time interval, so as to obtain the aggregated data corresponding to each time interval. The specified time interval can be a unit duration preset by the compute node itself, or a unit duration specified by the scheduler or other devices. This embodiment does not limit the duration range of the time interval, nor does it limit the specific method of specifying the time interval.

[0118] In some exemplary embodiments, whenever a time interval is reached, the compute node performs aggregation calculations on the data pulled within the currently reached time interval to obtain the aggregated data corresponding to the current time interval, and stores the aggregated data corresponding to the current time interval in the materialized view data shards contained in the materialized view database. For example, assuming a time interval of 20 minutes, the compute node will perform aggregation calculations on this data stream every time it pulls a data stream within 20 minutes to obtain the aggregated data corresponding to those 20 minutes.

[0119] S830 stores the aggregated data corresponding to each time interval into the materialized view data shards contained in the materialized view database; different materialized view data shards are responsible for storing data of different dimension combinations.

[0120] The materialized view database is used to perform dimensional aggregation on time-series data stored in the source database. Since the materialized view database stores data obtained by performing aggregation on a portion of the data dimensions in advance, data queries based on the materialized view database can eliminate the need for the aggregation process. Therefore, in this embodiment, the aggregated data corresponding to each time interval is stored in the materialized view data shards contained in the materialized view database through computing nodes. This construction of the materialized view database can save the computing resources required for data queries, thereby accelerating the query process.

[0121] For example, the materialized view database can also be constructed as a distributed database. The materialized view database includes multiple materialized view nodes, each of which deploys at least one materialized view data shard, and different materialized view data shards are responsible for storing data of different dimensional combinations, thereby enabling the materialized view database constructed in this embodiment to adapt to distributed application scenarios.

[0122] Therefore, in the technical solution provided in this embodiment, the computing node performs aggregation calculations on the data stream pulled from the source data shards based on a specified time interval. This can be understood as performing a data aggregation based on the time granularity of the specified time interval. The resulting aggregated data is then input into the materialized view database for storage, which can greatly reduce the amount of data input to the materialized view database, thereby reducing the write performance consumption on the materialized view side and thus improving the overall system performance.

[0123] It is understandable that the construction of the materialized view database itself achieves dimensional aggregation of data. Combined with the temporal aggregation of data further implemented in this embodiment, the technical solution provided in this embodiment essentially achieves two aggregations: temporal aggregation plus dimensional aggregation. Based on this two-stage data aggregation mechanism, the amount of data input to the materialized view database can be significantly reduced.

[0124] Furthermore, based on the distributed configuration of the materialized view database, different materialized view data shards are used to achieve data aggregation of different dimension combinations. Unlike materialized view implementation schemes built on a single-machine engine, there is no problem that data of a single dimension is actually distributed across multiple materialized view nodes. Therefore, the materialized view database can be directly provided to users for data querying without the need to perform secondary aggregation on the dimensions.

[0125] If a compute node experiences a crash or other anomaly and then restarts, it does not need to fetch the entire dataset again. This can be understood as the compute node restarting only from the time interval where the anomaly occurred, rather than fetching the entire data stream over the entire time period. Therefore, it significantly reduces the computational resources required for a compute node restart, which in turn improves overall system performance.

[0126] In addition, this embodiment can achieve near real-time data persistence to disk in real-time scenarios, instead of persisting the materialized view database to disk only after the full amount of data has been retrieved and the aggregation calculation has been completed, which further improves the overall performance of the system.

[0127] In another exemplary embodiment, considering that the data formats required by different operators in the materialized view statement included in the materialized view creation request may be inconsistent, the compute node obtains the specified materialized view input format and inputs the aggregated data corresponding to each time interval into the materialized view data shard for storage according to the materialized view input format. The materialized view input format includes at least one of the following: a field for marking the database table to which the aggregated data belongs, a field for representing the aggregated dimension, a field for representing the operator's calculated value, a field for representing the operator type, and a field for representing the time interval to which the aggregated data belongs. The field for representing the time interval to which the aggregated data belongs may include additional data fields and time interval fields. For a detailed description of the materialized view input format, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here. This embodiment ensures the implementation of dimension aggregation in the materialized view database by setting a specified materialized view input format, thereby ensuring the implementation of the two aggregation mechanisms, namely time aggregation and dimension aggregation.

[0128] In another exemplary embodiment, the compute node receives an aggregate computing task sent by the scheduler and, based on the received aggregate computing task, pulls a data stream from the source data shards contained in the source database.

[0129] The materialized view metadata database is a materialized view metadata object created through persistence. It can also be understood as a way to store the metadata corresponding to the materialized view database, such as the dimension combination information corresponding to each materialized view data shard.

[0130] The scheduler periodically retrieves metadata from the materialized view metadata database, performs scheduling calculations in response to input requests, generates corresponding scheduling tasks, and assigns these tasks. Generally, the scheduling calculations performed by the scheduler are based on materialized view creation requests. Based on the materialized view creation request and the metadata retrieved from the materialized view metadata database, the scheduler generates aggregation calculation tasks and assigns these tasks to at least one compute node in the compute node pool. The compute nodes respond to the aggregation calculation tasks by executing the processes shown in S810-S830.

[0131] The aggregation computation task generated by the scheduler may contain materialized view routing rules. The compute nodes obtain the materialized view routing rules sent by the scheduler, and then input the aggregated data corresponding to each time interval into the corresponding materialized view data fragments for storage according to the materialized view routing rules.

[0132] Materialized view routing rules instruct compute nodes to input aggregated data obtained from performing aggregation calculations into specific materialized view data shards within the materialized view database. Therefore, materialized view routing rules are generated by the scheduler after querying the materialized view metadata database to obtain the dimension combinations corresponding to each materialized view data shard. For example, suppose the materialized view database contains materialized view data shard 1 and materialized view data shard 2. Materialized view data shard 1 is responsible for storing data related to dimension combination 1, and materialized view data shard 2 is responsible for storing data related to dimension combination 2. Therefore, the materialized view routing rules instruct compute nodes to route data related to dimension combination 1 to materialized view data shard 1, and data related to dimension combination 2 to materialized view data shard 2.

[0133] Please continue reading. Figure 9 , Figure 9 Is Figure 8 A flowchart of another materialized view processing method technically proposed in the illustrated embodiment. (See flowchart for example.) Figure 9 As shown, in another exemplary embodiment, the materialized view processing method further includes S910-S920, which are described in detail below:

[0134] S910 retrieves the log sequence number corresponding to the data recorded in each time interval from the write-ahead log.

[0135] Log sequence numbers are used to identify the position of data recorded in the log file. Compute nodes retrieve the log sequence numbers corresponding to the data recorded within each time interval from the write-ahead log, which means retrieving the position of the data recorded within each time interval in the write-ahead log.

[0136] S920: After determining that the aggregated data corresponding to any time interval is stored in the materialized view data shard, the corresponding log sequence number is submitted to the source data shard so that the source data shard performs data deletion processing on the write-ahead log based on the submitted log sequence number.

[0137] After determining that the aggregated data corresponding to any time interval is stored in the materialized view data shard, the compute node submits the corresponding log sequence number to the source data shard so that the source data shard can perform data deletion processing on the write-ahead log based on the submitted log sequence number.

[0138] As an example implementation, the log sequence number submitted by the compute node to the source data shard can be a log sequence number range corresponding to this time interval. Based on the submitted log sequence number range, the source data shard deletes the data in the write-ahead log that corresponds to this log sequence number range.

[0139] For example Figure 7 As shown, after the compute node uploads the aggregated data 1 corresponding to the first time interval to the materialized view database 220, it submits a log sequence number range of 100-200 to the source data shard 1. The source data shard 1 then deletes the data corresponding to log sequence number range 100-200 from the write-ahead log. After the compute node uploads the aggregated data 2 corresponding to the second time interval to the materialized view database 220, it submits a log sequence number range of 200-300 to the source data shard 1. The source data shard 1 then deletes the data corresponding to log sequence number range 200-300 from the write-ahead log.

[0140] As another exemplary implementation, the log sequence number submitted by the compute node to the source data shard can be the log sequence number corresponding to the last data in this time interval. Based on the submitted log sequence number, the source data shard deletes the data in the write-ahead log that is before this log sequence number.

[0141] Still Figure 7 As shown, after the compute node uploads the aggregated data 1 corresponding to the first time interval to the materialized view database 220, it submits a log sequence number of 200 to the source data shard 1. The source data shard 1 then deletes the data before log sequence number 200 from the write-ahead log, which means deleting the data corresponding to log sequence number 100-200 from the write-ahead log. After the compute node uploads the aggregated data 1 corresponding to the second time interval to the materialized view database 220, it submits a log sequence number of 300 to the source data shard 1. The source data shard 1 then deletes the data before log sequence number 300 from the write-ahead log, which means deleting the data corresponding to log sequence number 200-300 from the write-ahead log.

[0142] As can be seen from the above example, in the technical solution provided in this embodiment, the computing node can promptly delete the data in the write-ahead log by assigning a commit log sequence number to the source data, thereby reducing the resource consumption required for the write-ahead log and improving the overall system performance to a certain extent.

[0143] Figure 10 This is a block diagram illustrating a materialized view processing apparatus according to an exemplary embodiment of this application. The materialized view processing apparatus can be configured in... Figure 2 On the computing nodes of the cloud time-series data storage system shown. For example... Figure 10 As shown, the exemplary materialized view processing apparatus includes:

[0144] The pull module 1010 is configured to pull the data stream that will be stored in memory within a specified time period from the source data shards contained in the source database;

[0145] The aggregation module 1020 is configured to perform aggregation calculations on the data pulled from each time interval according to the specified time interval, so as to obtain the aggregated data corresponding to each time interval.

[0146] Storage module 1030 is configured to store aggregated data corresponding to each time interval into materialized view data shards contained in the materialized view database; different materialized view data shards are responsible for storing data of different dimension combinations.

[0147] In another exemplary embodiment, the pull module 1010 is further configured as follows:

[0148] Determine the start and end times for the data stream to be retrieved;

[0149] Pull the write-ahead log of the source data shard and obtain the data stream recorded from the start time point from the write-ahead log; after successfully writing data, the source data shard will record the written data in the write-ahead log;

[0150] When the end time is reached, the fetching of the write-ahead log will stop.

[0151] In another exemplary embodiment, the pull module 1010 is further configured as follows:

[0152] Decode the retrieved write-ahead log to obtain the decoded write-ahead log;

[0153] Non-write type data is filtered out from the decoded write-ahead log to obtain the filtered write-ahead log;

[0154] The data stream, starting from the beginning time point, is obtained by parsing the filtered write-ahead log.

[0155] In another exemplary embodiment, the pull module 1010 is further configured as follows:

[0156] Receive the aggregation calculation task sent by the scheduler; in response to the materialized view creation request, after determining the source data shard to which the data to be aggregated belongs, the scheduler generates an aggregation calculation task to instruct the data stream to be pulled from the corresponding source data shard, and different aggregation calculation tasks correspond to different source data shards.

[0157] Based on the aggregation computing task, a data stream is pulled from the source data shards contained in the source database.

[0158] In another exemplary embodiment, the materialized view processing apparatus further includes a submission module configured to:

[0159] Obtain the log sequence number corresponding to the data recorded in each time interval from the write-ahead log;

[0160] After determining that the aggregated data corresponding to any time interval is stored in the materialized view data shard, the corresponding log sequence number is submitted to the source data shard so that the source data shard can perform data deletion processing on the write-ahead log based on the submitted log sequence number.

[0161] In another exemplary embodiment, the aggregation module 1020 is further configured as follows:

[0162] For each time interval reached, an aggregation calculation is performed on the data retrieved within the current time interval to obtain the aggregated data corresponding to the current time interval;

[0163] Furthermore, the storage module 1030 is further configured as follows:

[0164] The aggregated data corresponding to the current arrival time interval is stored in the materialized view data shards contained in the materialized view database.

[0165] In another exemplary embodiment, the storage module 1030 is further configured as follows:

[0166] Obtain the specified materialized view input format; the materialized view input format includes at least one of the following: a field for identifying the database table to which the aggregated data belongs, a field for representing the aggregated dimension, a field for representing the operator's calculated value, a field for representing the operator type, and a field for representing the time interval to which the aggregated data belongs;

[0167] According to the materialized view input format, the aggregated data corresponding to each time interval is input into the materialized view data fragment for storage.

[0168] In another exemplary embodiment, the storage module 1030 is further configured as follows:

[0169] Obtain the materialized view routing rules sent by the scheduler; the materialized view routing rules are generated by the scheduler after querying the materialized view meta-database to obtain the dimension combinations corresponding to each materialized view data shard;

[0170] Based on the materialized view routing rules, the aggregated data corresponding to each time interval is input into the corresponding materialized view data shards for storage.

[0171] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the materialized view processing apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0172] Embodiments of this application also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the materialized view processing method provided in the various embodiments above.

[0173] Figure 11 A schematic diagram of a computer system suitable for implementing an electronic device according to embodiments of this application is shown. It should be noted that the electronic device can be... Figure 2 This refers to any computing node in the cloud time-series data storage system shown. It should also be noted that... Figure 11 The computer system 1100 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0174] like Figure 11 As shown, the computer system 1100 includes a Central Processing Unit (CPU) 1101, which can perform various appropriate actions and processes based on a computer program stored in Read-Only Memory (ROM) 1102 or a computer program loaded from storage portion 1108 into Random Access Memory (RAM) 1103, such as performing the methods described in the above embodiments. Various computer programs and data required for system operation are also stored in RAM 1103. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An Input / Output (I / O) interface 1105 is also connected to bus 1104.

[0175] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0176] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs various functions defined in the system of this application.

[0177] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0179] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0180] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, implements the materialized view processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0181] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the materialized view processing method provided in the various embodiments described above.

[0182] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

[0183] It is understood that in the specific implementation of this application, data such as time-series data, data streams, and write-ahead logs are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for processing materialized views, characterized in that, The method includes: Pull a data stream from the source data shards contained in the source database; According to the specified time interval, perform aggregation calculations on the data of each time interval to obtain the aggregated data corresponding to each time interval; The aggregated data corresponding to each time interval is stored in the materialized view data shards contained in the materialized view database; different materialized view data shards are responsible for storing data of different dimension combinations.

2. The method according to claim 1, characterized in that, The data stream is pulled from the source data shards contained in the source database, including: Determine the start and end times for the data stream to be retrieved; The write-ahead log of the source data shard is retrieved, and the data stream recorded from the start time point is obtained from the write-ahead log; after the source data shard successfully writes data, it records the written data in the write-ahead log; When the specified end time is reached, the fetching of the pre-written log will stop.

3. The method according to claim 2, characterized in that, Retrieve the write-ahead log from the source data shard and obtain the data stream recorded from the write-ahead log starting from the start time point, including: Decode the retrieved write-ahead log to obtain the decoded write-ahead log; Non-write type data is filtered out from the decoded write-ahead log to obtain the filtered write-ahead log; The data stream recorded from the start time point is obtained by parsing the filtered write-ahead log.

4. The method according to claim 2, characterized in that, The method further includes: Obtain the log sequence number corresponding to the data recorded in each time interval from the pre-written log; After determining that the aggregated data corresponding to any time interval is stored in the materialized view data shard, the corresponding log sequence number is submitted to the source data shard so that the source data shard performs data deletion processing on the write-ahead log based on the submitted log sequence number.

5. The method according to claim 1, characterized in that, According to the specified time interval, perform aggregation calculations on the data streams for each time interval to obtain the aggregated data corresponding to each time interval, including: Upon reaching a time interval, an aggregation calculation is performed on the data retrieved within the currently reached time interval to obtain the aggregated data corresponding to the current time interval; The aggregated data corresponding to each time interval is stored in the materialized view data shards contained in the materialized view database, including: The aggregated data corresponding to the currently arrived time interval is stored in the materialized view data fragments contained in the materialized view database.

6. The method according to claim 1, characterized in that, The aggregated data corresponding to each time interval is stored in the materialized view data shards contained in the materialized view database, including: Obtain the specified materialized view input format; the materialized view input format includes at least one of the following: a field for marking the database table to which the aggregated data belongs, a field for representing the aggregated dimension, a field for representing the operator calculation value, a field for representing the operator type, and a field for representing the time interval to which the aggregated data belongs; According to the materialized view input format, the aggregated data corresponding to each time interval is input into the materialized view data fragment for storage.

7. The method according to claim 1, characterized in that, The aggregated data corresponding to each time interval is stored in the materialized view data shards contained in the materialized view database, including: Obtain the materialized view routing rules sent by the scheduler; the materialized view routing rules are generated by the scheduler after querying the materialized view meta-database to obtain the dimension combinations corresponding to each materialized view data shard; According to the materialized view routing rules, the aggregated data corresponding to each time interval is input into the corresponding materialized view data fragment for storage.

8. The method according to claim 7, characterized in that, The method further includes: pulling data streams from source data shards contained in the source database; The scheduler receives the aggregation calculation task sent by the scheduler; in response to the materialized view creation request, the scheduler generates an aggregation calculation task to indicate the data stream to be pulled from the corresponding source data shard after determining the source data shard to which the data to be aggregated belongs, and different aggregation calculation tasks correspond to different source data shards. Based on the aggregation computing task, a data stream is pulled from the source data shards contained in the source database.

9. A cloud time-series data storage system, characterized in that, include: The source database is used to store time-series data, including multiple source nodes, with at least one source data shard deployed on each source node; The materialized view database consists of multiple materialized view nodes. Each materialized view node deploys at least one materialized view data shard, and different materialized view data shards are responsible for storing data of different combinations of dimensions. The compute node pool includes multiple compute nodes. Each compute node is configured to pull a data stream from at least one source node and perform aggregation calculations on the data for each time interval according to a specified time interval to obtain aggregated data corresponding to each time interval. The aggregated data corresponding to each time interval is then input into at least one materialized view data shard for storage.

10. The cloud time-series data storage system according to claim 9, characterized in that, Also includes: Materialized view metadata database, used to store metadata corresponding to the materialized view database; A scheduler is configured to retrieve the metadata from the materialized view metadata database, generate an aggregate computing task based on the metadata, and allocate the aggregate computing task to computing nodes in the computing node pool. Different aggregation computation tasks indicate that data should be pulled from different source data shards and aggregation computations should be performed.

11. The cloud time-series data storage system according to claim 9, characterized in that: A stream computing program and an access response program run together on the same computing node; the access response program is used to respond to materialized view operations initiated by user terminals, and the materialized view operations include at least one of creating a materialized view, deleting a materialized view, modifying a materialized view, and querying a materialized view; or, The stream computing program and the access response program run on different computing nodes.

12. The cloud time-series data storage system according to claim 9, characterized in that: Each materialized view node is equipped with a materialized view engine, which is used to store aggregated data in the materialized view data shards. The materialized view engine implements the storage of aggregated data into materialized view data shards, including a first process of writing aggregated data into a memory table, and a second process of sinking the data in the memory table to the disk. In the first process, after performing aggregation calculations on the values ​​corresponding to the same keys in the aggregated data, the resulting aggregated data is written into the memory table. The memory table stores data in key-value pairs. And / or, in the second process, after replacing the fields in the key of the memory table that represent the aggregation dimension with time series identifiers, the replaced data is then sunk to disk.

13. The cloud time-series data storage system according to claim 9, characterized in that: Each materialized view node is equipped with a materialized view engine, which is used to store the aggregated data corresponding to each time interval in the materialized view data shards. After receiving aggregated data corresponding to any time interval, the materialized view engine determines whether there is historically stored aggregated data in the materialized view data shard. If so, perform aggregation calculation on the historically stored aggregated data and the aggregated data corresponding to the currently received time interval, and update and store the materialized view data fragments with the calculated aggregated data; If not, the aggregated data corresponding to the currently received time interval will be directly stored in the materialized view data fragment.

14. A materialized view processing apparatus, characterized in that, The device includes: The pull module is configured to pull a data stream that will be stored in memory within a specified time period from the source data shards contained in the source database. The aggregation module is configured to perform aggregation calculations on the data retrieved for each time interval according to the specified time interval, so as to obtain the aggregated data corresponding to each time interval. The storage module is configured to store the aggregated data corresponding to each time interval into the materialized view data shards contained in the materialized view database; different materialized view data shards are responsible for storing data of different dimension combinations.

15. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the electronic device to implement the materialized view processing method as described in any one of claims 1-8.

16. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the materialized view processing method according to any one of claims 1-8.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor of an electronic device, it implements the materialized view processing method as described in any one of claims 1-8.

Images