Data Processing Method and Device for Cassandra Key-Value Storage System Based on Interval Tree Hierarchy

Abstract

The present invention discloses a data processing method and device for a Cassandra key-value storage system based on interval tree layering, which relates to the field of data storage. The method hierarchically designs the external storage layer and the interval tree. According to the hierarchical characteristics of the LSM tree, the external storage layer is divided into a low-level external storage layer and a high-level external storage layer, and the interval tree is correspondingly divided into a low-level interval tree and a high-level interval tree. For the frequently changing data in the low-level external storage layer, a low-level interval tree is constructed to achieve fast response; for the relatively stable data in the high-level external storage layer, a high-level interval tree is constructed to achieve efficient data management and query. An adaptive construction strategy is adopted. When writing data each time, the original interval tree is preferentially incrementally updated, and reconstruction is only triggered when the interval tree is in an unbalanced state, reducing unnecessary computational overhead. The present invention solves the problems of high performance loss caused by frequent reconstruction of the interval tree and low overall reading and storage efficiency in the high-frequency read-write scenario of Cassandra.

Description

Technical Field

[0001] The present invention relates to the field of data storage, and particularly to a data processing method and device for a Cassandra key-value storage system based on interval tree layering. Background Art

[0002] Cassandra is an open-source distributed NoSQL database, widely used in scenarios requiring high availability, scalability, and fault tolerance, especially in applications with large-scale data storage and high concurrent access. Its design concept is based on the Log-Structured Merge-Tree (LSM tree) data structure, providing efficient data writing, reading, and updating capabilities. Cassandra utilizes a distributed architecture and, through partitioning, load balancing, and replication mechanisms, ensures the uniform distribution of data among multiple nodes, enhancing the system's scalability and fault tolerance.

[0003] The LSM tree is the core data structure of Cassandra, and its greatest advantage lies in optimizing write performance. Data is first written into the in-memory mutable table (MemTable). When the mutable table reaches its capacity limit, it is converted into a read-only immutable table and asynchronously flushed to disk.

[0004] On disk, data is initially stored in the L0 layer of the LSM tree in the form of Sorted String Table (SSTable) files. As the number of files in the L0 layer continues to grow, the system will periodically trigger an inter-layer compaction operation to migrate the data orderly to higher levels. During the compaction process, the system selects an SSTable file from layer i and finds all the file sets in layer i + 1 whose key ranges overlap with it. Subsequently, these files are loaded into memory for unified merging and sorting. During this process, the system validates the key-value pairs, eliminates duplicate or expired data, and finally generates a new SSTable file, which is written back to layer i + 1.

[0005] Through this hierarchical and progressive data structure that continuously organizes data, the LSM tree not only achieves efficient batch writing but also ensures the orderliness and efficient reading performance of data on disk, making Cassandra particularly suitable for write-intensive application scenarios. Moreover, the distributed architecture enables Cassandra to support large-scale and geographically distributed clusters, evenly distributing data to multiple nodes through the consistent hashing algorithm. Cassandra's partitioning and replication mechanisms ensure high availability and fault tolerance of data. When a node fails, the system can automatically initiate a data transfer request and ensure data is not lost, still being able to provide continuous services. Nevertheless, as the data volume increases, the scalability and performance of the system will also face bottlenecks.

[0006] Although Cassandra has powerful distributed storage capabilities, there are still some deficiencies. To optimize the read performance, Cassandra introduces an interval tree structure in metadata management to improve the search efficiency. However, the improvement space of the interval tree is limited, and its frequent construction and update consume system resources, thus affecting the overall read and write performance. Therefore, how to improve the overall performance of the system by improving the interval tree structure of Cassandra has become an important research direction currently. Summary of the Invention

[0007] The purpose of this application is to propose a data processing method and device for a Cassandra key-value storage system based on interval tree layering for the above-mentioned technical problems.

[0008] In the first aspect, the present invention provides a data processing method for a Cassandra key-value storage system based on interval tree layering. The external storage layer of the LSM tree in each storage node in the Cassandra key-value storage system is divided into a low-level external storage layer and a high-level external storage layer, and the corresponding interval tree is divided into a low-level interval tree and a high-level interval tree, which are respectively used to record and manage the key ranges of each ordered string table file in the low-level external storage layer and the key ranges of each ordered string table file in the high-level external storage layer. The data processing method includes a data writing process, and its steps include:

[0009] Obtain a data writing request and get the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written to obtain the hash value corresponding to the data to be written, and determine the target storage node where the data to be written is to be stored according to the hash value corresponding to the data to be written;

[0010] If an immutable memory table is generated during the process of writing data to be written into a target storage node, and the immutable memory table is written into the low-level external storage layer of the LSM tree of the target storage node and an ordered string table file of the low-level external storage layer is generated, then it is determined whether an inter-layer merge operation of the low-level external storage layer in the LSM tree will be triggered. If not, it is checked whether the left subtree and the right subtree of each node of the low-level interval tree are balanced. If not, a reconstruction operation of the low-level interval tree is triggered; if so, after the inter-layer merge operation of the low-level external storage layer in the LSM tree is completed, a reconstruction operation of the low-level interval tree is triggered, and it is determined whether the ordered string table file of the low-level external storage layer needs to be written into the high-level external storage layer of the LSM tree and an ordered string table file of the high-level external storage layer is generated. If so, it is determined whether an inter-layer merge operation of the high-level external storage layer in the LSM tree will be triggered after the ordered string table file of the low-level external storage layer is written into the high-level external storage layer of the LSM tree. If so, after the inter-layer merge operation of the high-level external storage layer in the LSM tree is completed, a reconstruction operation of the high-level interval tree is triggered; otherwise, it is checked whether the left subtree and the right subtree of each node of the high-level interval tree are balanced. If not, a reconstruction operation of the high-level interval tree is triggered.

[0011] Preferably, it further includes:

[0012] In response to determining that the left subtree and the right subtree of each node of the low-level interval tree are balanced, the left subtree and the right subtree of each node of the high-level interval tree are balanced, and / or the ordered string table file of the low-level external storage layer does not need to be written into the high-level external storage layer of the LSM tree, the existing structure is maintained.

[0013] Preferably, it further includes:

[0014] After the reconstruction operation of the high-level interval tree is completed, the existing structure is maintained.

[0015] Preferably, in the target storage node, first it is determined whether the data to be written can be written into the mutable memory table of the memory layer of the LSM tree of the target storage node. If so, the data to be written is written into the mutable memory table of the memory layer of the LSM tree of the target storage node, and the existing structure is maintained; otherwise, the mutable memory table is converted into an immutable memory table, a new mutable memory table is created, the data to be written is written into the new mutable memory table, and the immutable memory table is written into the external storage layer of the LSM tree of the target storage node.

[0016] Preferably, the external storage layer of the LSM tree has N layers, and the corresponding levels increase gradually from top to bottom, where the first several levels at the top are set as the low-level external storage layer, and the remaining levels are the high-level external storage layer.

[0017] Preferably, it further includes a data reading process, and its steps include:

[0018] Obtain a data reading request, parse to obtain the primary key of the data to be read, calculate the corresponding hash value, and determine the target storage node where the data to be read is located according to the hash value;

[0019] First, search for the data to be read in the memory layer of the target storage node according to the primary key of the data to be read. If the data to be read cannot be found in the memory layer, search for the ordered string table file where the data to be read is located in the low-level interval tree; if the ordered string table file where the data to be read is located is found in the low-level interval tree, obtain the reading result through the ordered string table file where the data to be read is located; if the ordered string table file where the data to be read is located cannot be found in the low-level interval tree, search for the ordered string table file where the data to be read is located in the high-level interval tree; if the ordered string table file where the data to be read is located is found in the high-level interval tree, obtain the reading result through the ordered string table file where the data to be read is located, and if the ordered string table file where the data to be read is located cannot be found in the high-level interval tree, return a reading failure.

[0020] In a second aspect, the present invention provides a data processing device for a Cassandra key-value storage system based on interval tree layering. The external memory layer of the LSM tree of each storage node in the Cassandra key-value storage system is divided into a low-level external memory layer and a high-level external memory layer, and the corresponding interval tree is divided into a low-level interval tree and a high-level interval tree, which are respectively used to record and manage the key ranges of each ordered string table file in the low-level external memory layer and the key ranges of each ordered string table file in the high-level external memory layer. The data processing device includes a data writing module, including:

[0021] A node determination module, configured to obtain a data writing request and obtain the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written to obtain the hash value corresponding to the data to be written, and determine the target storage node where the data to be written is to be stored according to the hash value corresponding to the data to be written;

[0022] A writing module, configured to, if an immutable memory table is generated during the process of writing data to be written to a target storage node, write the immutable memory table to a low-level external storage layer in the LSM tree of the target storage node and generate an ordered string table file for the low-level external storage layer, then determine whether an inter-layer merge operation of the low-level external storage layer in the LSM tree will be triggered. If not, check whether the left subtree and the right subtree of each node in the low-level interval tree are balanced. If not, trigger a reconstruction operation for the low-level interval tree. If so, after completing the inter-layer merge operation of the low-level external storage layer in the LSM tree, trigger a reconstruction operation for the low-level interval tree, and determine whether the ordered string table file of the low-level external storage layer needs to be written to a high-level external storage layer in the LSM tree and generate an ordered string table file for the high-level external storage layer. If so, determine whether an inter-layer merge operation of the high-level external storage layer in the LSM tree will be triggered after the ordered string table file of the low-level external storage layer is written to the high-level external storage layer in the LSM tree. If so, after completing the inter-layer merge operation of the high-level external storage layer in the LSM tree, trigger a reconstruction operation for the high-level interval tree. Otherwise, check whether the left subtree and the right subtree of each node in the high-level interval tree are balanced. If not, trigger a reconstruction operation for the high-level interval tree.

[0023] In a third aspect, the present invention provides an electronic device, including one or more processors; a storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation manner of the first aspect.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium, having a computer program stored thereon, which when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0025] In a fifth aspect, the present invention provides a computer program product, including a computer program, which when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) The data processing method of the Cassandra key-value storage system based on interval tree layering proposed by the present invention reduces the waste of system resources by reducing the need for frequent construction of interval trees during the data writing process. Traditional methods usually need to reconstruct the entire interval tree each time, while the present invention, through the interval tree layering design, usually only needs to reconstruct the low-level interval tree each time during reconstruction, and the high-level interval tree is reconstructed less frequently, thus effectively reducing the reconstruction overhead and improving the writing performance.

[0028] (2) The data processing method of the Cassandra key-value storage system based on interval tree layering proposed by the present invention can flexibly select the corresponding level of interval tree to be accessed during data reading by dividing the interval tree into a low-level interval tree and a high-level interval tree. In some cases, only querying the low-level interval tree can meet the requirements, thus reducing the computational burden and memory consumption during the query process and improving the query efficiency.

[0029] (3) The data processing method of the Cassandra key-value storage system based on interval tree layering proposed by the present invention can reduce the resource consumption caused by interval tree construction, utilize the characteristics of interval tree layering to greatly improve the read and write performance of Cassandra; through more precise interval tree management, it ensures the efficiency during data writing and reading processes. Especially when dealing with large-scale data, it can better balance performance and resource utilization. This optimization not only improves the system's response speed but also enhances its adaptability in high-concurrency and high-throughput scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0031] Figure 1 It is a flowchart of the data processing method of the Cassandra key-value storage system based on interval tree layering for the embodiments of this application;

[0032] Figure 2 It is a structural diagram of the external storage layer and the interval tree of the data processing method of the Cassandra key-value storage system based on interval tree layering for the embodiments of this application;

[0033] Figure 3 It is a schematic diagram of the data processing device of the Cassandra key-value storage system based on interval tree layering for the embodiments of this application;

[0034] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0036] Figure 1 A data processing method for a Cassandra key-value storage system based on interval tree layering provided by an embodiment of the present application is shown. The external memory layer of the LSM tree in each storage node of the Cassandra key-value storage system is divided into a low-level external memory layer and a high-level external memory layer, and the corresponding interval tree is divided into a low-level interval tree and a high-level interval tree, which are respectively used to record and manage the key ranges of each ordered string table file in the low-level external memory layer and the key ranges of each ordered string table file in the high-level external memory layer. The data processing method includes a data writing process, and its steps include:

[0037] S1. Obtain a data writing request and get the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written to obtain the hash value corresponding to the data to be written, and determine the target storage node where the data to be written is to be stored according to the hash value corresponding to the data to be written.

[0038] In a specific embodiment, the external memory layer of the LSM tree has N layers, and the corresponding levels increase gradually from top to bottom, where the first layers at the top are set as the low-level external memory layer, and the remaining layers are the high-level external memory layer.

[0039] Specifically, the overall structure of the LSM tree (Log-Structured Merge-Tree) is jointly composed of an internal memory layer and an external memory layer. Among them, the internal memory layer includes a variable internal memory table and an immutable internal memory table. When the variable internal memory table is full, it can be converted into an immutable internal memory table and prepared to be flushed to the external memory layer. The external memory layer consists of multiple levels, and each level contains several sorted string table files as the persistent form of data on the disk.

[0040] An interval tree is a metadata information structure used to record and manage the key ranges (i.e., data intervals) of each ordered string table file in the external storage layer. Each ordered string table file stores a segment of ordered key ranges on disk, and the interval tree constructs a tree-like structure through these key ranges for fast positioning and retrieval. Each node of the interval tree usually contains information such as the minimum key (Low), maximum key (High), and central key (Center) corresponding to the ordered string table, which is used for efficient interval search and determination of interval overlap. The interval tree is stored in memory and synchronously updates the key ranges in the interval tree according to the key ranges stored in the external storage layer.

[0041] The construction of the interval tree follows the attribution rule based on the central key: using the central key as the dividing point, all intervals covering this central key are uniformly placed in this node; while the intervals not covering this central key are recursively placed in the left subtree or right subtree of this node according to their positions. Refer to Figure 2 , in one embodiment, assume that the key interval ranges of two ordered string table files are 6 - 12 and 9 - 15, and the central key of the current node is 12. Since both of these intervals cover the central key 12, that is, their interval ranges contain 12, so they will be simultaneously mounted on the current node with the central key of 12. While the interval ranges less than 12 or greater than 12 will continue to be recursively placed in the left subtree or right subtree corresponding to the current node. When the key range of the written data gradually becomes larger, the left subtree and / or right subtree of the current node can be further extended as the next node and generate the left subtree and / or right subtree of the next node.

[0042] The embodiments of this application layer the external storage layer into a low-level external storage layer and a high-level external storage layer. Refer to Figure 2 , in one embodiment, there are a total of 7 layers in the external storage layer. Among them, layers L0 to L3 are the low-level external storage layer, and a low-level interval tree is constructed for it to quickly respond to the frequent changes of data; while layers L4 to L6 are the high-level external storage layer, and its data is relatively stable, then another high-level interval tree is constructed separately to achieve efficient management and query. Therefore, two interval trees, namely the low-level interval tree and the high-level interval tree, are respectively used to record and manage the key ranges of the ordered string table files in the low-level external storage layer and the high-level external storage layer. In a preferred embodiment, the top layers of the external storage layer are used as the low-level external storage layer, and the remaining layers at the bottom are used as the high-level external storage layer. In other embodiments, other layers can also be selected as the boundary between the low-level external storage layer and the high-level external storage layer.

[0043] During the data writing process, first parse the obtained data writing request to get the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written, obtain the corresponding hash value according to the calculation, and determine the target storage node where the data to be written will be stored according to the hash value.

[0044] In a specific embodiment, in the target storage node, first determine whether the data to be written can be written into the mutable memory table in the memory layer of the LSM tree of the target storage node. If so, write the data to be written into the mutable memory table in the memory layer of the LSM tree of the target storage node and maintain the existing structure; otherwise, convert the mutable memory table into an immutable memory table, create a new mutable memory table, write the data to be written into the new mutable memory table, and write the immutable memory table into the external storage layer of the LSM tree of the target storage node.

[0045] Specifically, on the target storage node, determine whether the data to be written can be written into the mutable memory table. If the data volume of the data to be written is less than or equal to the remaining space of the mutable memory table, the data to be written can be written into the mutable memory table and the existing structure is maintained. If the data volume of the data to be written is greater than the remaining space of the mutable memory table, the data to be written cannot be written into the mutable memory table. It is necessary to convert the mutable memory table into an immutable memory table, create a new mutable memory table, and write the data to be written into the new mutable memory table. The immutable memory table is then written into the external storage layer in the LSM tree of the target storage node and an ordered string table file is generated.

[0046] S2. If an immutable memory table is generated during the process of writing the data to be written into the target storage node, and the immutable memory table is written into the lower-level external storage layer in the LSM tree of the target storage node and an ordered string table file of the lower-level external storage layer is generated, then determine whether it will trigger an inter-layer merge operation of the lower-level external storage layer in the LSM tree. If it will not trigger, check whether the left subtree and the right subtree of each node in the lower-level interval tree are balanced. If they are not balanced, trigger the reconstruction operation of the lower-level interval tree; if it will trigger, after completing the inter-layer merge operation of the lower-level external storage layer in the LSM tree, trigger the reconstruction operation of the lower-level interval tree, and determine whether the ordered string table file of the lower-level external storage layer needs to be written into the higher-level external storage layer in the LSM tree and an ordered string table file of the higher-level external storage layer is generated. If so, determine whether it will trigger an inter-layer merge operation of the higher-level external storage layer in the LSM tree after the ordered string table file of the lower-level external storage layer is written into the higher-level external storage layer in the LSM tree. If so, after completing the inter-layer merge operation of the higher-level external storage layer in the LSM tree, trigger the reconstruction operation of the higher-level interval tree; otherwise, check whether the left subtree and the right subtree of each node in the higher-level interval tree are balanced. If they are not balanced, trigger the reconstruction operation of the higher-level interval tree.

[0047] In a specific embodiment, it further includes:

[0048] In response to determining that the left and right subtrees of each node in the low-level interval tree are balanced, the left and right subtrees of each node in the high-level interval tree are balanced, and / or the ordered string table file in the low-level external storage layer does not need to be written to the high-level external storage layer in the LSM tree, the existing structure is maintained.

[0049] In a specific embodiment, it further includes:

[0050] After completing the reconstruction operation of the high-level interval tree, the existing structure is maintained.

[0051] Specifically, when the immutable memory table is written to the low-level external storage layer in the LSM tree of the target storage node, first write the immutable memory table to the L0 layer of the low-level external storage layer, and determine whether the layer merging operation between the low-level external storage layers in the LSM tree will be triggered after the immutable memory table is written to the L0 layer of the low-level external storage layer, that is, whether the layer merging operation will occur between the L0-L3 layers. If it will not be triggered, check whether the left and right subtrees of each node in the low-level interval tree are balanced. If they are balanced, continue to maintain the existing structure. If they are not balanced, trigger the reconstruction operation of the low-level interval tree. When the low-level interval tree has been balanced, maintain the existing structure. If it will be triggered, after completing the layer merging operation between the low-level external storage layers, trigger the reconstruction operation of the low-level interval tree to restore its balance, and further determine whether the ordered string table file in the low-level external storage layer needs to be written to the high-level external storage layer and generate the ordered string table file in the high-level external storage layer. If not, continue to maintain the existing structure; if needed, after the ordered string table file in the low-level external storage layer is written to the high-level external storage layer, determine whether the layer merging operation between the high-level external storage layers will be triggered, that is, whether the layer merging operation will occur between the L4-L6 layers. If it will be triggered, reconstruct the high-level interval tree, complete the layer merging operation between the high-level external storage layers to adapt to the new data distribution, and then maintain the existing structure; if it will not be triggered, check whether the left and right subtrees of each node in the high-level interval tree are balanced. If they are balanced, continue to maintain the existing structure without any modification; if they are not balanced, trigger the reconstruction operation of the high-level interval tree, and then maintain the existing structure.

[0052] Furthermore, the layer merging operation is to write the ordered string table file in the external storage layer of the upper level to the external storage layer of the lower level. Whether the layer merging operation will be triggered depends on whether the data volume in the L0 layer exceeds the data volume threshold in the L0 layer after the data to be written is written to the L0 layer. If so, write the ordered string table file in the L0 layer to the L1 layer, and so on.

[0053] Further, each interval tree in the embodiments of the present application adopts an adaptive construction strategy, and preferentially performs incremental updates when writing data to be written, that is, only adjusts the local interval structure for the newly added data to reduce the calculation overhead and improve the writing efficiency. In one embodiment, it can be set that when the left subtree and the right subtree of each node of the interval tree differ by five or more levels, it indicates that the left subtree and the right subtree of each node of the interval tree are unbalanced, and this is used as a basis for checking whether the left subtree and the right subtree of each node of the high-level interval tree are balanced. Only when the interval tree reaches an unbalanced state, such as when the left and right subtrees of any subtree differ by five or more levels, a reconstruction is triggered, and the interval tree is reconstructed according to the latest data distribution to make the interval tree regain balance, thereby effectively avoiding the performance loss caused by frequent reconstructions.

[0054] When writing data, first write the data to be written into the memory layer. When the memory layer is full, the data in the memory layer will be flushed to the ordered string table file in the disk. In the traditional method, each time a new ordered string table file is written, it will cause the reconstruction of the interval tree. After adopting the interval tree hierarchical design of the embodiments of the present application, the reconstruction operation of the interval tree mostly targets the ordered string table file in the low-level external storage layer, reducing the resource overhead required in the reconstruction process. In addition, with the help of the adaptive construction method, there is no need to reconstruct the interval tree every time, and reconstruction is only performed when the interval tree reaches a very unbalanced state, thereby avoiding the performance degradation of the system caused by frequent reconstruction.

[0055] In a specific embodiment, it further includes a data reading process, and its steps include:

[0056] Obtain a data reading request and parse to obtain the primary key of the data to be read and calculate the corresponding hash value, and determine the target storage node where the data to be read is located according to the hash value;

[0057] First, look for the data to be read in the memory layer of the target storage node according to the primary key of the data to be read. If the data to be read cannot be found in the memory layer, look for the ordered string table file where the data to be read is located in the low-level interval tree; if the ordered string table file where the data to be read is located is found in the low-level interval tree, obtain the reading result through the ordered string table file where the data to be read is located; if the ordered string table file where the data to be read is located cannot be found in the low-level interval tree, look for the ordered string table file where the data to be read is located in the high-level interval tree; if the ordered string table file where the data to be read is located is found in the high-level interval tree, obtain the reading result through the ordered string table file where the data to be read is located, and if the ordered string table file where the data to be read is located cannot be found in the high-level interval tree, return a reading failure.

[0058] Specifically, an embodiment of the present application can also propose a data reading method for a Cassandra key-value storage system based on interval tree layering, and the steps are as follows: First, obtain a data reading request, parse the data reading request to obtain the primary key of the data to be read and calculate the corresponding hash value, and the coordinating node determines the target storage node where the data to be read is located according to the hash value corresponding to the primary key of the data to be read. The coordinating node sends the data reading request to the corresponding target storage node. After receiving the data reading request, the target storage node first searches for the data to be read in the memory layer. If the data to be read is found in the memory layer, the reading result is returned to the coordinating node, and finally the user's reading request is completed. If the data to be read cannot be found in the memory layer, the target storage node first searches for the ordered string table file where the data to be read is located in the low-level interval tree according to the primary key of the data to be read. If the ordered string table file where the data to be read is located can be found in the low-level interval tree, the target storage node reads the data in the ordered string table file where the data to be read is located to obtain the reading result, and returns the reading result to the coordinating node, and finally the user's reading request is completed; if the ordered string table file where the data to be read is located cannot be found in the low-level interval tree, the target storage node searches for the ordered string table file where the data to be read is located from the high-level interval tree according to the primary key of the data to be read. If the ordered string table file where the data to be read is located can be found in the high-level interval tree, the target storage node reads the data in the ordered string table file where the data to be read is located to obtain the reading result, and returns the reading result to the coordinating node, and finally the user's reading request is completed; if the ordered string table file where the data to be read is located cannot be found in the high-level interval tree, a reading failure result is returned.

[0059] During the data search process, the required data is first searched in the memory layer. If not found, the corresponding ordered string table file is located through the interval tree. Since most reading operations are concentrated in the low-level external storage layer of the LSM tree, only the low-level interval tree needs to be searched, and it is not necessary to traverse all levels of the interval tree, thus greatly reducing the overhead during the search process and improving the reading efficiency.

[0060] Further referring to Figure 3 , as an implementation of the methods shown in the above figures, an embodiment of a data processing device for a Cassandra key-value storage system based on interval tree layering is provided in the present application. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0061] An embodiment of the present application provides a data processing device for a Cassandra key-value storage system based on interval tree layering. The external memory layer of the LSM tree in each storage node in the Cassandra key-value storage system is divided into a low-level external memory layer and a high-level external memory layer, and the corresponding interval tree is divided into a low-level interval tree and a high-level interval tree, which are respectively used to record and manage the key ranges of each ordered string table file in the low-level external memory layer and the key ranges of each ordered string table file in the high-level external memory layer. The data processing device includes a data writing module, including:

[0062] A node determination module 1, configured to obtain a data writing request and obtain the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written to obtain the hash value corresponding to the data to be written, and determine the target storage node where the data to be written is to be stored according to the hash value corresponding to the data to be written;

[0063] A writing module 2, configured to generate an immutable memory table during the process of writing the data to be written to the target storage node, write the immutable memory table to the low-level external memory layer in the LSM tree of the target storage node and generate an ordered string table file for the low-level external memory layer, then determine whether an inter-layer merge operation of the low-level external memory layer in the LSM tree will be triggered. If not, check whether the left subtree and the right subtree of each node in the low-level interval tree are balanced. If not, trigger the reconstruction operation of the low-level interval tree; if so, after completing the inter-layer merge operation of the low-level external memory layer in the LSM tree, trigger the reconstruction operation of the low-level interval tree, and determine whether the ordered string table file of the low-level external memory layer needs to be written to the high-level external memory layer in the LSM tree and generate an ordered string table file for the high-level external memory layer. If so, determine whether an inter-layer merge operation of the high-level external memory layer in the LSM tree will be triggered after the ordered string table file of the low-level external memory layer is written to the high-level external memory layer in the LSM tree. If so, after completing the inter-layer merge operation of the high-level external memory layer in the LSM tree, trigger the reconstruction operation of the high-level interval tree, otherwise check whether the left subtree and the right subtree of each node in the high-level interval tree are balanced. If not, trigger the reconstruction operation of the high-level interval tree.

[0064] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. As Figure 4 shown, the electronic device in this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer execution instructions; the processor 401 is used to execute the computer execution instructions stored in the memory to implement each step executed by the electronic device in the above embodiment. Specifically, reference can be made to the relevant descriptions in the foregoing method embodiments.

[0065] Optionally, the memory 402 can be either independent or integrated with the processor 401.

[0066] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401.

[0067] An embodiment of the present invention further provides a computer storage medium, in which computer-executable instructions are stored. When the processor 401 executes the computer-executable instructions, the above method is implemented.

[0068] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0069] In the embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be in electrical, mechanical or other forms.

[0070] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to implement the solution of this embodiment.

[0071] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in one unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of a hardware plus software functional unit.

[0072] The above integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules are stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or the processor 401 to execute some steps of the methods in various embodiments of the present application.

[0073] It should be understood that the above-mentioned processor 401 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor may be a microprocessor or the processor 401 may also be any conventional processor 401, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by the hardware processor 401, or by a combination of hardware and software modules in the processor 401.

[0074] The memory 402 may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc.

[0075] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the bus 403 in the drawings of this application is not limited to only one bus 403 or one type of bus 403.

[0076] The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disks, or optical discs. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0077] An exemplary storage medium is coupled to the processor 401, enabling the processor 401 to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a master device.

[0078] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method for a Cassandra key-value storage system based on interval tree layering, characterized in that In the Cassandra key-value storage system, the external storage layer of the LSM tree of each storage node is divided into a low-level external storage layer and a high-level external storage layer, and the corresponding interval tree is divided into a low-level interval tree and a high-level interval tree, which are respectively used to record and manage the key ranges of each ordered string table file in the low-level external storage layer and the key ranges of each ordered string table file in the high-level external storage layer. The data processing method includes a data writing process, and its steps include: Obtain a data writing request and get the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written to obtain the hash value corresponding to the data to be written, and determine the target storage node where the data to be written is to be stored according to the hash value corresponding to the data to be written; If an immutable memory table is generated during the process of writing the data to be written to the target storage node, and the immutable memory table is written to the low-level external storage layer in the LSM tree of the target storage node and an ordered string table file in the low-level external storage layer is generated, then determine whether an inter-layer merge operation of the low-level external storage layer in the LSM tree will be triggered. If not, check whether the left subtree and the right subtree of each node in the low-level interval tree are balanced. If not, trigger a reconstruction operation of the low-level interval tree; if so, after completing the inter-layer merge operation of the low-level external storage layer in the LSM tree, trigger a reconstruction operation of the low-level interval tree, and determine whether the ordered string table file in the low-level external storage layer needs to be written to the high-level external storage layer in the LSM tree and an ordered string table file in the high-level external storage layer is generated. If so, determine whether an inter-layer merge operation of the high-level external storage layer in the LSM tree will be triggered after the ordered string table file in the low-level external storage layer is written to the high-level external storage layer in the LSM tree. If so, after completing the inter-layer merge operation of the high-level external storage layer in the LSM tree, trigger a reconstruction operation of the high-level interval tree; otherwise, check whether the left subtree and the right subtree of each node in the high-level interval tree are balanced. If not, trigger a reconstruction operation of the high-level interval tree.

2. The data processing method of the Cassandra key-value storage system based on interval tree layering according to claim 1, wherein, It also includes: In response to determining that the left subtree and the right subtree of each node in the low-level interval tree are balanced, the left subtree and the right subtree of each node in the high-level interval tree are balanced, and / or the ordered string table file in the low-level external storage layer does not need to be written to the high-level external storage layer in the LSM tree, maintain the existing structure.

3. The data processing method of the Cassandra key-value storage system based on interval tree layering according to claim 1, characterized in that, It also includes: After completing the reconstruction operation of the high-level interval tree, maintain the existing structure.

4. The data processing method of the Cassandra key-value storage system based on interval tree layering according to claim 1, characterized in that, In the target storage node, first determine whether the data to be written can be written into the mutable memory table in the memory layer of the LSM tree of the target storage node. If so, write the data to be written into the mutable memory table in the memory layer of the LSM tree of the target storage node and maintain the existing structure; otherwise, convert the mutable memory table into an immutable memory table, create a new mutable memory table, write the data to be written into the new mutable memory table, and write the immutable memory table into the external storage layer of the LSM tree of the target storage node.

5. The data processing method of the Cassandra key-value storage system based on interval tree layering according to claim 1, wherein The external storage layer of the LSM tree has N layers, and the corresponding levels increase gradually from top to bottom. Among them, the first several levels at the top are set as the low-level external storage layer, and the remaining levels are the high-level external storage layer.

6. The data processing method of the Cassandra key-value storage system based on interval tree layering according to claim 1, characterized in that, It also includes a data reading process, and its steps include: Obtain a data read request, parse it to get the primary key of the data to be read, calculate the corresponding hash value, and determine the target storage node where the data to be read is located according to the hash value; First, search for the data to be read in the memory layer of the target storage node according to the primary key of the data to be read. If the data to be read cannot be found in the memory layer, search for the ordered string table file where the data to be read is located in the low-level interval tree; if the ordered string table file where the data to be read is located is found in the low-level interval tree, obtain the read result through the ordered string table file where the data to be read is located; if the ordered string table file where the data to be read is located cannot be found in the low-level interval tree, search for the ordered string table file where the data to be read is located in the high-level interval tree; if the ordered string table file where the data to be read is located is found in the high-level interval tree, obtain the read result through the ordered string table file where the data to be read is located. If the ordered string table file where the data to be read is located cannot be found in the high-level interval tree, return a read failure.

7. A data processing device for a Cassandra key-value storage system based on interval tree layering, characterized in that, The external storage layer of the LSM tree of each storage node in the Cassandra key-value storage system is divided into a low-level external storage layer and a high-level external storage layer, and the corresponding interval tree is divided into a low-level interval tree and a high-level interval tree, which are respectively used to record and manage the key ranges of each ordered string table file in the low-level external storage layer and the key ranges of each ordered string table file in the high-level external storage layer. The data processing device includes a data writing module, including: A node determination module, configured to obtain a data write request, obtain the primary key of the data to be written, perform a hash calculation on the primary key of the data to be written to obtain the hash value corresponding to the data to be written, and determine the target storage node where the data to be written is to be stored according to the hash value corresponding to the data to be written; A writing module, configured to determine whether an inter - level merge operation of a low - level external storage layer in the LSM tree will be triggered if an immutable memory table is generated during the process of writing the data to be written into the target storage node, and the immutable memory table is written into the low - level external storage layer of the LSM tree of the target storage node and an ordered string table file of the low - level external storage layer is generated. If the inter - level merge operation will not be triggered, check whether the left subtree and the right subtree of each node of the low - level interval tree are balanced. If they are not balanced, trigger a reconstruction operation of the low - level interval tree; if the inter - level merge operation will be triggered, after the inter - level merge operation of the low - level external storage layer in the LSM tree is completed, trigger a reconstruction operation of the low - level interval tree, and determine whether the ordered string table file of the low - level external storage layer needs to be written into the high - level external storage layer in the LSM tree and an ordered string table file of the high - level external storage layer is generated. If so, determine whether an inter - level merge operation of the high - level external storage layer in the LSM tree will be triggered after the ordered string table file of the low - level external storage layer is written into the high - level external storage layer in the LSM tree. If so, after the inter - level merge operation of the high - level external storage layer in the LSM tree is completed, trigger a reconstruction operation of the high - level interval tree; otherwise, check whether the left subtree and the right subtree of each node of the high - level interval tree are balanced. If they are not balanced, trigger a reconstruction operation of the high - level interval tree.

8. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 - 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 - 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 - 6.

Images