Data processing methods, data query methods and devices

Abstract

This specification provides one or more embodiments of a data processing method, a data query method, and an apparatus, relating to the field of database technology. The method is applied to a database based on a log-structured LSM tree, where the database includes an ordered string table (SSTable) storing aggregated data. The solution provided in this specification can, in response to a merge operation targeting target data, merge the target data into the SSTable; and update the aggregated data stored in the SSTable based on the target data merged into the SSTable. The solution provided in this specification enables the database to utilize the multi-version data storage capability of the LSM tree, updating the aggregated data stored in the SSTable during data merging without incurring additional resource overhead; and automatically sensing the validity of the aggregated data during data querying, thereby accelerating the data query process and outputting correct query results.

Description

Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data processing method, a data query method, and an apparatus. Background Technology

[0002] Aggregated data refers to a single result generated by performing calculations or statistical operations on massive amounts of data in a database. For example, the maximum value of a column in a database can be calculated and pre-stored as aggregated data. During data querying, by calling the pre-stored aggregated data, the amount of data that needs to be queried can be reduced, thereby improving the database's query performance and response speed.

[0003] In related technologies, as data manipulation language (DML) statements are executed in the database, the pre-built aggregated data in the database needs to be updated frequently, which consumes additional system resources during the update process, and the validity of the aggregated data cannot be automatically determined. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a data processing method, a data query method, and an apparatus.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data processing method is proposed, applied to a database based on a log structure merging LSM tree, the database including an ordered string table SSTable, the SSTable storing aggregated data, the method comprising:

[0007] In response to a merge operation targeting the target data, merge the target data into the SSTable;

[0008] Update the aggregated data stored in the SSTable based on the target data to be merged into the SSTable.

[0009] According to a second aspect of one or more embodiments of this specification, a data query method is proposed, applied to an LSM-based database, the database including an SSTable storing aggregated data, the method comprising:

[0010] In response to a query command targeting specific data, validate the validity of the aggregated data;

[0011] Query specific data based on valid aggregated data from the aggregated data.

[0012] According to a third aspect of one or more embodiments of this specification, a data processing apparatus is provided, applied to an LSM-based database, the database including an SSTable storing aggregated data, including:

[0013] The merge module is used to merge the target data into the SSTable in response to a merge operation targeting the target data.

[0014] The update module is used to update the aggregated data stored in the SSTable based on the target data to be merged into the SSTable.

[0015] According to a fourth aspect of one or more embodiments of this specification, a data query apparatus is provided, applied to an LSM-based database, the database including an SSTable storing aggregated data, including:

[0016] The validation module is used to validate the validity of aggregated data in response to query commands targeting specified data.

[0017] The query module is used to query specified data based on valid aggregated data in the aggregated data.

[0018] According to a fifth aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0019] processor;

[0020] Memory used to store processor-executable instructions;

[0021] The processor implements the method as described in the first aspect and / or the method as described in the second aspect by executing executable instructions.

[0022] According to a sixth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method of the first aspect and / or the steps of the method of the second aspect.

[0023] The method provided in this specification can be applied to LSM tree-based databases. By storing aggregated data in an SSTable within the LSM tree, the database can leverage the multi-version data storage capability of the LSM tree. When data merging occurs in the LSM tree, the aggregated data stored in the SSTable is updated without incurring additional resource overhead. Furthermore, when data queries occur in the LSM tree, the validity of the aggregated data is automatically detected, thereby accelerating the data query process and outputting correct query results. Attached Figure Description

[0024] Figure 1This is a schematic diagram of a database structure based on an LSM tree, provided as an exemplary embodiment.

[0025] Figure 2 This is a schematic flowchart of a data processing method provided in an exemplary embodiment.

[0026] Figure 3 This is a schematic diagram of a tree-like storage structure provided in an exemplary embodiment.

[0027] Figure 4 This is a flowchart illustrating a data query method provided in an exemplary embodiment.

[0028] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0029] Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment.

[0030] Figure 7 This is a schematic diagram of the structure of a data query device provided in an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0032] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0033] Aggregated data refers to a single result generated by performing calculations or statistical operations on massive amounts of data in a database. For example, the maximum value of a column in a database can be calculated and pre-stored as aggregated data. During data querying, by calling the pre-stored aggregated data, the amount of data that needs to be queried can be reduced, thereby improving the database's query performance and response speed.

[0034] In related technologies, materialized views can be used to pre-store aggregated data as a view table to accelerate database queries. However, configuring materialized views is complex, requiring extensive adaptation to the database computing engine and interaction with the view table through corresponding database interfaces. Furthermore, materialized views consume significant storage space during storage and require substantial system resources for each update. In addition, during querying, materialized views can only serve specific queries and cannot be adapted to ad-hoc queries. They also cannot automatically determine the validity of aggregated data during the query process, potentially leading to incorrect database query output results.

[0035] In view of this, the embodiments of this specification, by storing aggregated data in an SSTable within an LSM tree, enable the database to quickly update aggregated data based on the multi-version data storage capability of the LSM tree, and automatically determine the validity of aggregated data during the query process.

[0036] Specifically, this disclosure first provides a data processing method that, in response to a merge operation on target data, merges the target data into an SSTable; and updates the aggregated data stored in the SSTable based on the merged target data. Since the update of the aggregated data in the SSTable occurs when data merging occurs in the LSM tree, it does not trigger an aggregated data update with every data change in the database, thereby improving the database's response speed and avoiding additional resource overhead for executing DML statements such as data update and data insert commands.

[0037] Furthermore, this disclosure also provides a data query method that can verify the validity of aggregated data in response to a query command for specified data; thereby querying the specified data based on the valid aggregated data. Since the aggregated data is stored in an SSTable within an LSM tree, the data query method provided by this disclosure can automatically detect the validity of the aggregated data, thereby accelerating the data query process and outputting correct query results.

[0038] To facilitate understanding, the following will combine... Figure 1 First, the application scenarios of the embodiments in this specification will be introduced.

[0039] The solution provided in this manual is applicable to databases based on Log Structured Merge (LSM) trees.

[0040] Figure 1 This is a schematic diagram of a database structure based on an LSM tree, provided as an exemplary embodiment. Figure 1As shown, an LSM tree can include a memory table (Memtable) and a sorted string table (SSTable).

[0041] The Memtable is an in-memory data structure within the LSM tree. Because the Memtable is stored in memory, data read and write operations within it can be performed efficiently. When a write operation occurs in the database, the data to be written is first written to the Memtable to maintain fast write performance. However, due to limited memory capacity, when the size of the Memtable reaches a certain threshold, the data stored in the Memtable will be compacted into an SSTable on disk to free up memory storage space for new data writes.

[0042] SSTable is a disk data structure in an LSM tree used for data persistence. For example, SSTables can be divided into dump SSTables and baseline SSTables. For example, when data in a Memtable is merged to disk, the SSTable in which this data is written is called a dump SSTable. When multiple dump SSTables accumulate to a certain number or size, they can be merged into a larger SSTable file to reduce disk space usage and optimize read performance. Similarly, the baseline SSTable is the lowest-level SSTable file generated after the merge operation, containing the data updated during the merge process as well as all previously stored data in the database.

[0043] In some embodiments, when the amount of data in the database is small, the dump SSTable and the baseline SSTable can be the same SSTable. That is, when the data in the Memtable is merged to the disk, the SSTable used to receive this data is the lowest-level SSTable, i.e., the baseline SSTable.

[0044] It should be understood that, despite Figure 1 Although not shown in the specification, the number of dump SSTables can be one or more. Multiple dump SSTables can also form a multi-level storage structure. For example, dump SSTables can be divided into first-level dump SSTables and second-level dump SSTables. Data in multiple first-level dump SSTables can be merged into one second-level dump SSTable, and data in multiple second-level dump SSTables can be merged into one baseline SSTable. This specification does not limit the embodiments in this way.

[0045] Next, the exemplary implementation methods of this specification will be described in detail with reference to the above application scenarios.

[0046] First, this specification provides a data processing method that can be executed by any electronic device.

[0047] Figure 2 This is a schematic flowchart of a data processing method provided in an exemplary embodiment. The data processing method is applied to an LSM-based database, which includes an SSTable. The SSTable stores aggregated data.

[0048] like Figure 2 As shown in the embodiments of this specification, the data processing method includes the following steps.

[0049] S201, in response to a merge operation on the target data, merges the target data into the SSTable.

[0050] It should be noted that, as described above regarding the application scenarios, data merging operations may occur between the Memtable and the dump SSTable, between upper and lower dump SSTables, or between the dump SSTable and the baseline SSTable.

[0051] In other words, the target data in this embodiment can be the basic data stored in the Memtable or the basic data stored in the dump SSTable. The SSTable in this embodiment can be the dump SSTable or the baseline SSTable.

[0052] It is understood that the target data in the embodiments of this specification is the basic data stored in the table being merged, and the SSTable is the SSTable used to receive the merged basic data. For example, when the database merges the basic data stored in the Memtable into the dump SSTable, the target data is the basic data stored in the Memtable, and the SSTable is the dump SSTable. This specification will not elaborate further on this aspect in the embodiments.

[0053] Unless otherwise specified, the basic data in this specification refers to unaggregated data stored in the database. Aggregated data can be obtained by aggregating the large amount of basic data stored in the database.

[0054] The aggregated data in this specification can be of any type, such as the number of rows of the basic data (COUNT), the sum of the basic data (SUM), the average value of the basic data (AVG), the maximum value of the basic data (MAX), the minimum value of the basic data (MIN), etc.

[0055] It is understood that, depending on the actual needs, the embodiments of this specification may configure aggregated data for one or several columns of basic data in the database table, or may configure aggregated data for all columns of basic data in the database table separately. The embodiments of this disclosure do not limit this.

[0056] Furthermore, for each column of basic data, one type of aggregated data can be configured, or multiple types of aggregated data can be configured. To facilitate the retrieval of aggregated data, when multiple types of aggregated data are configured, type tags can be added to the aggregated data during storage, enabling the database to quickly and accurately retrieve the corresponding type of aggregated data.

[0057] For example, for a product price list in a database, the basic data may be several product names and prices stored in the data, while the aggregated data may be the product name with the highest price and its corresponding price, or it may be the total number of products in the product price list.

[0058] In some embodiments, the SSTable is configured with a tree-like storage structure (e.g., a B+ tree structure), which includes leaf nodes and non-leaf nodes. Leaf nodes store basic data, while non-leaf nodes store aggregated data. The aggregated data stored in a non-leaf node can represent the aggregated result of the basic data stored in its descendant leaf nodes.

[0059] In some embodiments, non-leaf nodes include a first non-leaf node and a second non-leaf node, wherein the first non-leaf node is the parent node of a leaf node, and the second non-leaf node is the ancestor node of the first non-leaf node.

[0060] The aggregated data stored in each first non-leaf node can be obtained by aggregating the basic data stored in the child nodes of that first non-leaf node; the aggregated data stored in each second non-leaf node can be obtained by aggregating the aggregated data stored in the child nodes of that second non-leaf node.

[0061] It should be understood that in a tree-structured storage, the number of first non-leaf nodes and second non-leaf nodes can be one or more. The second non-leaf node may be the parent node of the first non-leaf node, or it may be the parent node of the first non-leaf node's parent. In other words, the second non-leaf node can be any ancestor node of the first non-leaf node.

[0062] It is worth noting that when calculating the data stored in each non-leaf node, only the data stored in the child nodes of that non-leaf node needs to be input, without considering all descendant nodes of that non-leaf node. This improves the computational efficiency of aggregated data.

[0063] In some embodiments, there may be multiple leaf nodes, each leaf node is used to store basic data of different primary key ranges, and the basic data stored in each leaf node is arranged in order according to its primary key so as to quickly determine the validity of the aggregated data when calling the aggregated data.

[0064] Accordingly, when merging target data into an SSTable, the target data can be merged into the corresponding leaf node according to the primary key of the target data, so that the target data and the basic data stored in the leaf node are arranged in order according to the primary key.

[0065] It should be noted that the amount of basic data stored in each leaf node can be determined based on the actual application scenario. The amount of basic data stored in each leaf node can be the same or different, and this specification does not limit this. For example, for basic data that is frequently updated, the basic data capacity of the leaf nodes storing this basic data can be appropriately reduced, thereby avoiding frequent updates to the corresponding aggregate data and reducing the failure range when calling the aggregate data. Conversely, for relatively fixed basic data, the basic data capacity of the leaf nodes storing this basic data can be appropriately increased, so that an aggregate data can represent as many features of the basic data as possible, reducing the amount of data that needs to be traversed during querying.

[0066] Furthermore, based on the aforementioned application scenarios, it can be seen that SSTable can be divided into dump SSTable and baseline SSTable. In the embodiments of this specification, both dump SSTable and baseline SSTable can be configured with the above-mentioned tree-like storage structure.

[0067] For example, Figure 3 This is a schematic diagram of a tree-like storage structure provided in an exemplary embodiment. For example... Figure 3 As shown, the tree-like storage structure includes leaf nodes C1, C2, C3, and C4, first non-leaf nodes B1 and B2, and second non-leaf node A1. Among them, Figure 3 The connecting lines in the diagram illustrate the parent-child relationships between these nodes. For example, the first non-leaf node B1 is the common parent of leaf nodes C1 and C2, the first non-leaf node B2 is the common parent of leaf nodes C3 and C4, and the second non-leaf node A1 is the common parent of the first non-leaf nodes B1 and B2.

[0068] exist Figure 3 In the leaf nodes shown, each leaf node stores basic data from the database. For example, leaf node C1 can store basic data with a primary key range of 1 to 400. Leaf node C2 can store basic data with a primary key range of 401 to 800; this will not be elaborated further in the embodiments of this specification.

[0069] exist Figure 3 In the first non-leaf nodes shown, the aggregated data stored in each first non-leaf node is obtained by aggregating the basic data stored in its child nodes. For example, the aggregated data stored in the first non-leaf node B1 is the aggregation result of the basic data stored in leaf nodes C1 and C2. The aggregated data stored in the first non-leaf node B2 is the aggregation result of the basic data stored in leaf nodes C3 and C4.

[0070] And for Figure 3 The second non-leaf node A1 shown is obtained by aggregating the aggregated data stored in its child nodes. In other words, the aggregated data stored in the second non-leaf node A1 is the aggregation result of the aggregated data stored in the first non-leaf nodes B1 and B2.

[0071] For example, suppose the underlying data stored in the SSTable is product price data. This product price data can be allocated to the corresponding leaf nodes for storage based on the primary key of the data. In some application scenarios, it is necessary to frequently query the average price of these products to monitor price changes.

[0072] Please continue to refer to Figure 3 At this point, the aggregated data represents the average price of the goods. Leaf nodes C1, C2, C3, and C4 store the goods price data, respectively. The first non-leaf node B1 stores the average of the goods price data stored in leaf nodes C1 and C2 (i.e., the aggregated data), for example, this average is 110. The first non-leaf node B2 stores the average of the goods price data stored in leaf nodes C3 and C4, for example, this average is 130. The second non-leaf node A1 stores the average of the aggregated data stored in the first non-leaf node B1 and the first non-leaf node B2, which is (110 + 130) ÷ 2 = 120.

[0073] Therefore, when a user wants to query the average price of goods stored in leaf nodes C1 and C2, they can directly access the aggregated data stored in the first non-leaf node B1. Similarly, when a user wants to query the average price of goods stored in leaf nodes C1, C2, C3, and C4, they can directly access the aggregated data stored in the second non-leaf node A1. This allows the solution provided in this specification to meet different user query needs by maintaining aggregated data, thereby improving the database's response speed to user queries.

[0074] S202, Update the aggregated data stored in SSTable based on the target data to be merged into SSTable.

[0075] For example, the following processing can be performed on each leaf node in the SSTable that contains the target data:

[0076] Update the aggregated data stored in the ancestor nodes of the leaf node based on the target data to be merged into the leaf node.

[0077] It is understandable that in the tree-structured storage of SSTable, since the aggregation result of the next-level node is actually the input of the previous-level node, when calculating the aggregated data stored in the second non-leaf node of each level, it is only necessary to call the aggregated data stored in its next-level node, without traversing the basic data and aggregated data stored in all its descendant nodes. Therefore, the embodiments in this specification can quickly update the aggregated data stored in each node of the tree-structured storage without consuming a large amount of memory and without affecting the read and write performance of the database.

[0078] Furthermore, in the embodiments of this specification, since the update process of aggregated data can occur during the merging operation in the LSM tree background, the batch processing method during the merging operation also makes the calculation of aggregated data more efficient. Moreover, based on the relationship between nodes in the tree-like storage structure, the aggregated data stored in leaf nodes that have not been merged with target data can be reused. Simultaneously, if the aggregated data stored in all child nodes of a non-leaf node has not changed, the aggregated data stored in that non-leaf node can also be reused. Therefore, the embodiments of this disclosure can significantly reduce the computational load of aggregated data and maintain aggregated data more efficiently.

[0079] Furthermore, since aggregated data is stored only in SSTable, and considering the aforementioned application scenarios, when data is written to the LSM tree, the data is directly written to the Memtable. Only when a data merge operation is performed will the basic data stored in the Memtable be merged into the SSTable. Therefore, updating or inserting data does not lead to frequent updates to the aggregated data, and thus will not affect the execution efficiency of DML statements in the database.

[0080] Based on the same inventive concept, this disclosure also provides a data query method, as shown in the following embodiments. Since the principle by which this method solves the problem is similar to that of the data processing method described above, the implementation of this method can refer to the implementation of the above method embodiments, and repeated details will not be described again.

[0081] It should be noted that, similar to the data processing method embodiments described above, this data query method is also applied to an LSM tree-based database, which includes an SSTable. The SSTable stores aggregated data.

[0082] The SSTable in this embodiment also has a tree-like storage structure, which includes leaf nodes and non-leaf nodes. Leaf nodes store basic data, while non-leaf nodes store aggregated data. The aggregated data stored in a non-leaf node can represent the aggregation result of the basic data stored in its descendant leaf nodes.

[0083] In some embodiments, non-leaf nodes include a first non-leaf node and a second non-leaf node, wherein the first non-leaf node is the parent node of a leaf node, and the second non-leaf node is the ancestor node of the first non-leaf node.

[0084] The aggregated data stored in each first non-leaf node can be obtained by aggregating the basic data stored in the child nodes of that first non-leaf node; the aggregated data stored in each second non-leaf node can be obtained by aggregating the aggregated data stored in the child nodes of that second non-leaf node.

[0085] In some embodiments, there may be multiple leaf nodes, each leaf node is used to store basic data of different primary key ranges, and the basic data stored in each leaf node is arranged in order according to its primary key, so as to quickly determine the validity of the aggregated data when querying aggregated data.

[0086] Specifically, since the tree-like storage structure configured in the SSTable in this specification embodiment is similar to that in the above data processing method embodiment, the description of the tree-like storage structure can be referred to the description in the above data processing method embodiment, and will not be repeated in this specification embodiment.

[0087] Figure 4 This diagram illustrates a data query method according to an embodiment of the present disclosure. This method can be executed by any electronic device. Figure 4 As shown, the data query method provided in this embodiment includes the following steps.

[0088] S401, in response to a query command for specified data, verifies the validity of the aggregated data.

[0089] It should be noted that after receiving a query command for specified data, the LSM tree can first merge the underlying data stored in the SSTable (including the dump SSTable and the baseline SSTable) and the Memtable through an iterator.

[0090] For example, you can use multiple iterators to iterate through the underlying data stored in both SSTable and Memtable, following the primary key in ascending order. If there is underlying data with the same primary key, you need to merge them based on the age of the data.

[0091] For the baseline SSTable, assuming that the primary key of the current baseline SSTable iteration is RowKey1, and the smallest primary key that all other iterators (e.g., the iterator of the dump SSTable, the iterator of the Memtable) are about to iterate to is RowKey2, then all the aggregated data corresponding to the basic data within the primary key range [RowKey1, RowKey2) are valid aggregated data and can be directly called when querying data.

[0092] In other words, if the first leaf node stores basic data with the same primary key as the first target primary key and / or the second target primary key, then the aggregated data stored in the ancestor nodes of that first leaf node is invalid. Here, the first leaf node is the leaf node in the baseline SSTable, the first target primary key is the primary key of the basic data stored in the Memtable, and the second target primary key is the primary key of the basic data stored in the dumped SSTable.

[0093] For dumping an SSTable, if the second leaf node stores basic data with the same primary key as the first target primary key, then the aggregated data stored in the ancestor nodes of that second leaf node is invalid. Here, the second leaf node is the leaf node in the dumped SSTable, and the first target primary key is the primary key of the basic data stored in the Memtable.

[0094] Furthermore, if a second leaf node stores basic data whose primary key falls within the target primary key range, then the aggregated data stored in the ancestor nodes of that second leaf node is invalid. The target primary key range is the range of primary keys for the basic data stored in the leaf nodes of the baseline SSTable.

[0095] In other words, for a dumped SSTable, the aggregated data is only valid if all the aggregated objects stored on it are data that does not exist in the baseline SSTable.

[0096] S402: Query the specified data based on the valid aggregated data in the aggregated data.

[0097] It should be noted that after validating the aggregated data, the valid aggregated data can be iterated through in the SSTable using an iterator. Then, by combining the valid aggregated data with the merged base data, the query for the specified data can be completed.

[0098] For example, in a product price list, if the merged base data includes price data for 1000 products numbered 1 to 1000, and the price data for products numbered 1 to 800 remained unchanged during the merge process, then the aggregated data for these products is valid aggregated data. In this case, if one wishes to query the highest price of the 1000 products numbered 1 to 1000, one can directly retrieve the aggregated data of the maximum price for products numbered 1 to 800 and compare it with that of products numbered 801 to 1000, thus significantly reducing the amount of data that needs to be traversed during the query process.

[0099] The data query method provided in the embodiments of this specification can automatically determine the validity of aggregated data during the query process and complete the query of specified data by combining valid aggregated data, thereby improving the data query efficiency while ensuring the accuracy of the query.

[0100] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0101] Please refer to Figure 6 , Figure 6 A data processing device 600 is provided, which can be applied to, for example Figure 5 The device shown is used to implement the technical solution of this specification. Exemplarily, the data processing apparatus 600 can be applied to an LSM tree-based database, which includes an SSTable storing aggregated data. The data processing apparatus 600 may include:

[0102] Merge module 601 is used to merge target data into SSTable in response to a merge operation on target data.

[0103] Update module 602 is used to update the aggregated data stored in SSTable based on the target data to be merged into SSTable.

[0104] In some embodiments, the SSTable is configured with a tree-like storage structure, which includes leaf nodes and non-leaf nodes. Leaf nodes store basic data, while non-leaf nodes store aggregated data, which displays the aggregated results of the basic data.

[0105] In some embodiments, there are multiple leaf nodes, and each leaf node is used to store basic data for different primary key ranges.

[0106] In some embodiments, non-leaf nodes include a first non-leaf node and a second non-leaf node, wherein the first non-leaf node is the parent node of a leaf node, and the second non-leaf node is the ancestor node of the first non-leaf node. The aggregated data stored in each first non-leaf node is obtained by aggregating the basic data stored in the child nodes of that first non-leaf node, and the aggregated data stored in each second non-leaf node is obtained by aggregating the aggregated data stored in the child nodes of that second non-leaf node.

[0107] In some embodiments, the merging module 601 is specifically used to merge the target data into the corresponding leaf node according to the primary key of the target data, so that the target data and the basic data stored in the leaf node are arranged in order according to the primary key.

[0108] In some embodiments, the update module 602 is specifically used to perform the following processing for each leaf node that has merged target data: update the aggregated data stored in the ancestor node of the leaf node according to the target data merged into the leaf node.

[0109] In some embodiments, the database also includes a memory table, Memtable. SSTable is a dump SSTable or a baseline SSTable, and the target data is the base data stored in the Memtable or the base data stored in the dump SSTable.

[0110] Please refer to Figure 7 , Figure 7 A data query device 700 is provided, which can be applied to, for example Figure 5 The device shown implements the technical solution of this specification. Exemplarily, the data query device 700 can be applied to an LSM-based database, which includes an SSTable storing aggregated data. The data query device 700 may include:

[0111] The verification module 701 is used to verify the validity of aggregated data in response to a query command for specified data.

[0112] The query module 702 is used to query specified data based on valid aggregated data in the aggregated data.

[0113] In some embodiments, the SSTable is configured with a tree-like storage structure, which includes leaf nodes and non-leaf nodes. The leaf nodes store basic data, and the non-leaf nodes store aggregated data, which displays the aggregation results of the basic data.

[0114] In some embodiments, there are multiple leaf nodes, and each leaf node is used to store basic data for different primary key ranges.

[0115] In some embodiments, non-leaf nodes include a first non-leaf node and a second non-leaf node, wherein the first non-leaf node is the parent node of a leaf node, and the second non-leaf node is the ancestor node of the first non-leaf node. The aggregated data stored in each first non-leaf node is obtained by aggregating the basic data stored in the child nodes of that first non-leaf node, and the aggregated data stored in each second non-leaf node is obtained by aggregating the aggregated data stored in the child nodes of that second non-leaf node.

[0116] In some embodiments, the SSTable includes a dump SSTable and a baseline SSTable, and the database further includes a Memtable. The verification module 701 is specifically configured to: if a first leaf node stores basic data with a primary key identical to the first target primary key and / or the second target primary key, then the aggregated data stored in the ancestor node of that first leaf node is invalid. Here, the first leaf node is a leaf node in the baseline SSTable, the first target primary key is the primary key of the basic data stored in the Memtable, and the second target primary key is the primary key of the basic data stored in the dump SSTable.

[0117] In some embodiments, the SSTable includes a dumped SSTable, and the database further includes a Memtable. The verification module 701 is specifically configured to: if the second leaf node stores basic data with the same primary key as the first target primary key, then the aggregated data stored in the ancestor node of the second leaf node is invalid. Here, the second leaf node is a leaf node in the dumped SSTable, and the first target primary key is the primary key of the basic data stored in the Memtable.

[0118] In some embodiments, the SSTable includes a dump SSTable and a baseline SSTable. The verification module 701 is specifically configured to: if the second leaf node stores basic data whose primary key falls within the target primary key range, then the aggregated data stored in the ancestor node of the second leaf node is invalid. Here, the second leaf node is a leaf node in the dump SSTable, and the target primary key range is the primary key range of the basic data stored in the leaf nodes of the baseline SSTable.

[0119] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0120] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0121] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0122] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0125] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0126] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0127] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data processing method applied to a database based on a log-structured merge (LSM) tree, the database comprising a sorted string table (SSTable), the SSTable storing aggregated data, the method comprising: in response to a merge operation on target data, merging the target data into the SSTable; updating the aggregated data stored in the SSTable according to the target data merged into the SSTable; the SSTable being configured with a tree-shaped storage structure, the tree-shaped storage structure comprising leaf nodes and non-leaf nodes; wherein the leaf nodes are configured to store base data, and the non-leaf nodes are configured to store aggregated data, the aggregated data being configured to show aggregation results of the base data. 2.The method of claim 1, wherein a number of the leaf nodes is plural, and each leaf node is configured to store base data of a different primary key range.

3. The method of claim 1, the non-leaf nodes comprising a first non-leaf node and a second non-leaf node, wherein, a first non-leaf node is a parent node of a leaf node, and a second non-leaf node is an ancestor node of the first non-leaf node; aggregated data stored in each first non-leaf node is obtained by aggregating base data stored in child nodes of the first non-leaf node; and aggregated data stored in each second non-leaf node is obtained by aggregating aggregated data stored in child nodes of the second non-leaf node. 4.The method of claim 2, wherein the merging of the target data into the SSTable comprises: according to a primary key of the target data, merging the target data into a corresponding leaf node, so that the target data and base data stored in the leaf node are arranged in order according to the primary key. 5.The method of claim 4, wherein the updating of the aggregated data stored in the SSTable according to the target data merged into the SSTable comprises: for each leaf node having target data merged therein, the following processing is performed respectively: updating aggregated data stored in an ancestor node of the leaf node according to target data merged into the leaf node. 6.The method of claim 1, wherein the database further comprises a memory table (Memtable) ; the SSTable is a dump SSTable or a baseline SSTable, and the target data is base data stored in the Memtable or base data stored in the dump SSTable. 7.A data query method applied to a database based on an LSM tree, the database comprising an SSTable, the SSTable storing aggregated data, the method comprising: in response to a query instruction on specified data, checking validity of the aggregated data; querying the specified data according to valid aggregated data in the aggregated data; the SSTable being configured with a tree-shaped storage structure, the tree-shaped storage structure comprising leaf nodes and non-leaf nodes; wherein the leaf nodes are configured to store base data, and the non-leaf nodes are configured to store aggregated data, the aggregated data being configured to show aggregation results of the base data. 8.The method of claim 7, wherein the leaf nodes are a plurality of leaf nodes, each of the leaf nodes storing base data of a different primary key range.

9. The method of claim 7, the non-leaf nodes comprising a first non-leaf node and a second non-leaf node, wherein, The first non-leaf node is a parent node of a leaf node, and the second non-leaf node is an ancestor node of the first non-leaf node; The aggregated data stored in each first non-leaf node is obtained by aggregating the base data stored in the child nodes of the first non-leaf node; The aggregated data stored in each second non-leaf node is obtained by aggregating the aggregated data stored in the child nodes of the second non-leaf node. 10.The method of claim 7, wherein the SSTables include a dump SSTable and a baseline SSTable, and the database further includes a Memtable; The verifying the validity of the aggregated data comprises: If the first leaf node stores base data with a primary key identical to the first target primary key and / or the second target primary key, the aggregated data stored in the ancestor node of the first leaf node is invalid. The first leaf node is a leaf node in the baseline SSTable, the first target primary key is a primary key of base data stored in the Memtable, and the second target primary key is a primary key of base data stored in the dump SSTable. 11.The method of claim 7, wherein the SSTables include a dump SSTable, and the database further includes a Memtable; The verifying the validity of the aggregated data comprises: If the second leaf node stores base data with a primary key identical to the first target primary key, the aggregated data stored in the ancestor node of the second leaf node is invalid. The second leaf node is a leaf node in the dump SSTable, and the first target primary key is a primary key of base data stored in the Memtable. 12.The method of claim 8, wherein the SSTables include a dump SSTable and a baseline SSTable; The verifying the validity of the aggregated data comprises: If the second leaf node stores base data with a primary key falling within the target primary key range, the aggregated data stored in the ancestor node of the second leaf node is invalid. The second leaf node is a leaf node in the dump SSTable, and the target primary key range is a primary key range of base data stored in the leaf nodes of the baseline SSTable. 13.A data processing apparatus applied to a database based on a LSM tree, the database including SSTables storing aggregated data, the apparatus being configured to implement the method of any one of claims 1 to 6, and comprising: a merging module configured to merge target data to the SSTables in response to a merge operation on the target data; an updating module configured to update the aggregated data stored in the SSTables according to the target data merged to the SSTables.

14. A data query apparatus applied to a LSM tree-based database, the database comprising SSTables storing aggregated data, the apparatus being configured to implement the method of any one of claims 7 to 12, comprising: a verification module configured to verify validity of the aggregated data in response to a query instruction for specified data; and a query module configured to query the specified data according to valid aggregated data in the aggregated data.

15. An electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of any one of claims 1 to 6 and / or the method of any one of claims 7 to 12 by executing the executable instructions.

16. A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 6 and / or the steps of the method of any one of claims 7 to 12.

Images