Data processing method and apparatus, electronic device and storage medium

Abstract

A data processing method, which is executed by an electronic device. The method comprises: in response to a write operation for target data in a log-structured merge tree, determining expired data generated by the write operation, wherein the log-structured merge tree comprises at least two data layers, each data layer comprises a preset quantity of disk files, each data layer corresponds to a cache structure, and each cache structure is used for storing meta-information of the expired data in the corresponding data layer (S201); determining a keyword of the target data as a target keyword of the expired data (S203); on the basis of the target keyword, searching the preset quantity of disk files comprised in each data layer for a target disk file in which the expired data is located (S204); writing the target keyword into the cache structure corresponding to the data layer in which the target disk file is located, wherein the target keyword is used for identifying the meta-information of the expired data in the target disk file (S205); and in response to the quantity of the keyword in the cache structure corresponding to at least one data layer satisfying a preset quantity condition, by means of the meta-information identified by the keyword in the cache structure respectively corresponding to the at least one data layer, positioning the expired data in the disk files comprised in the at least one data layer, and carrying out recovery processing on the positioned expired data (S207).

Description

Data processing method, device, electronic device and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023112795390, filed on October 7, 2023, entitled “Data processing method, device, electronic device and storage medium,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application belongs to the field of computer technology, and specifically relates to a data processing method, device, electronic device and storage medium. Background Art

[0004] In a storage system based on a Log-Structured Merge Tree (LSM tree), compaction is often required to eliminate expired data. Compaction refers to eliminating expired data in the storage engine according to a specific strategy to free up disk space.

[0005] Related technologies usually consider a record as expired data and delete it only when a record with the same keyword (Key) and a larger version is encountered during the compression process. However, the timing of encountering two identical records during the compression process is uncontrollable. An expired record may be compressed multiple times (write amplification) and continue to exist in the system, wasting the system's computing resources and disk input / output (I / O) resources. Secondly, a scan query will also read the expired data from the disk into memory and then discard it, resulting in high read latency (read amplification). Finally, expired data exists in the database system for a long time, resulting in ineffective utilization of disk space (space amplification).

[0006] It can be seen that the expired data recovery method in the related art is a passive recovery mechanism, and system resources cannot be effectively utilized.

[0007] Summary of the Invention

[0008] The present application provides a data processing method, device, electronic device and storage medium.

[0009] In one aspect, the present application proposes a data processing method, comprising:

[0010] In response to a write operation on target data in a log-structured merge tree, determining expired data generated by the write operation; the log-structured merge tree includes at least two data layers, each data layer includes a preset number of disk files, and each data layer corresponds to a cache structure, each cache structure is used to store metadata of expired data in the corresponding data layer;

[0011] determining the keyword of the target data as the target keyword of the expired data;

[0012] Based on the target keyword, searching for the target disk file where the expired data is located from a preset number of disk files included in each data layer;

[0013] Writing the target keyword into a cache structure corresponding to the data layer where the target disk file is located; the target keyword is used to identify the meta information of expired data in the target disk file; and

[0014] In response to the number of keywords in the cache structure corresponding to at least one data layer meeting a preset quantity condition, the metadata identified by the keywords in the cache structure corresponding to the at least one data layer is used to locate expired data in the disk files included in the at least one data layer, and the located expired data is recycled.

[0015] On the other hand, the present application provides a data processing device, comprising:

[0016] an expired data acquisition module, configured to determine, in response to a write operation on target data in a log-structured merge tree, expired data generated by the write operation; the log-structured merge tree comprising at least two data layers, each of which comprising a preset number of disk files, and each of which correspondingly comprises a cache structure, each cache structure being configured to store metadata of expired data in the corresponding data layer;

[0017] a target disk file search module, configured to determine the keyword of the target data as the target keyword of the expired data; and based on the target keyword, search for the target disk file where the expired data is located from a preset number of disk files included in each data layer;

[0018] A keyword writing module, configured to write the target keyword into a cache structure corresponding to the data layer where the target disk file is located; the target keyword is used to identify meta information of expired data in the target disk file;

[0019] A recycling module is used to respond to the number of keywords in the cache structure corresponding to at least one data layer meeting a preset number condition, use the metadata identified by the keywords in the cache structure corresponding to the at least one data layer, locate expired data in the disk files included in the at least one data layer, and recycle the located expired data.

[0020] On the other hand, the present application proposes an electronic device for data processing, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the data processing method as described above.

[0021] On the other hand, the present application proposes a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the data processing method as described above.

[0022] On the other hand, the present application proposes a computer program product, comprising a computer program, which implements the data processing method as described above when executed by a processor.

[0023] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0025] Fig. 1 is a schematic diagram showing an implementation environment of a data processing method according to an exemplary embodiment.

[0026] Fig. 2 is a schematic flow chart showing a data processing method according to an exemplary embodiment.

[0027] Fig. 3 is a schematic diagram showing a cache structure according to an exemplary embodiment.

[0028] FIG4 is a flowchart illustrating steps of searching for a target disk file containing expired data from a preset number of disk files included in each data layer based on a target keyword according to an exemplary embodiment.

[0029] FIG5 is a flowchart illustrating the steps of searching for a target disk file containing expired data from a preset number of disk files included in each data layer based on a target keyword according to another exemplary embodiment.

[0030] Fig. 6 is a schematic flow chart showing a data processing method according to an exemplary embodiment.

[0031] FIG7 is an overall framework diagram of a TDSQL distributed database system according to an exemplary embodiment.

[0032] Fig. 8 is a diagram showing a storage framework of a single storage node according to an exemplary embodiment.

[0033] Fig. 9 is a block diagram showing a data processing device according to an exemplary embodiment.

[0034] FIG10 is a hardware structure block diagram of a server provided according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] In order to better illustrate the present application, the technical terms used in the embodiments of the present application are explained below:

[0037] LSM trees: A data structure that supports add, delete, read, modify, and sequential scan operations, and avoids random disk writes through batch storage technology. The core feature of LSM trees is the use of sequential writes to improve write performance. However, the layered design (layering here refers to the division into memory and files) slightly reduces read performance. However, by sacrificing a small amount of read performance in exchange for high write performance, LSM trees have become a very popular storage structure. Databases based on LSM trees include, but are not limited to, LevelDB and HBase. LevelDB is an efficient key-value database. HBase is a distributed, column-oriented, open source database.

[0038] The following is an introduction to the structure of the LSM tree:

[0039] The LSM tree consists of the following three important components:

[0040] MemTable (memory table): A memtable is a data structure in memory that stores recently updated data and organizes this data in order by key. The LSM tree does not have a clear data structure definition for how to organize data in order. For example, HBase uses a skip table to ensure the order of keys in memory.

[0041] Because data is temporarily stored in memory, which is not reliable storage and will be lost if there is a power outage, write-ahead logging (WAL) is usually used to ensure data reliability.

[0042] Immutable MemTable: When a memtable reaches a certain size, it is converted to an immutable memtable. An immutable memtable is an intermediate state that transforms a memtable into an ordered set of key-value pairs. Write operations are handled by the new memtable, and data updates are not blocked during the transfer process.

[0043] SSTable (Sorted String Table): An ordered set of key-value pairs, representing the on-disk data structure of an LSM tree group. To speed up reading of ordered key-value pairs, you can create key indexes and Bloom filters to accelerate key lookups.

[0044] Compaction: In a storage engine based on an LSM tree, the storage engine periodically merges data. This process can eliminate dirty data, and this action of eliminating dirty data is called a compression action.

[0045] The LSM tree stores all data insertion, modification, and deletion operations in memory. When a certain amount of data has been processed, the data is written sequentially to disk in batches. When the in-memory table reaches a certain size, the data is flushed ("flush") to persistent storage, creating an ordered set of key-value pairs. Records with the same key may exist in different ordered sets of key-value pairs; the most recent record is the correct one. The flush operation forcibly writes cached data to main memory or disk to ensure data consistency and reliability.

[0046] In some data index structures, a hierarchical mechanism is often used to store metadata. Metadata is data that describes the actual data and is used to describe the attributes (features) of the actual data. For example, metadata can be the file name of the actual data or a pointer to the storage address of the actual data. Metadata can also have a corresponding identifier to identify the metadata. Metadata and its corresponding identifier can form a key-value pair. Each key-value pair can include a keyword (Key) and the value (Value) corresponding to the Key. The Value is the metadata itself, and the Key is used to identify the Value of the metadata.

[0047] The LSM tree can be divided into N+1 layers (N is a positive integer), namely L0, L1, ..., LN. The data in the L0 layer can be stored in the memory of the storage device. Since the memory space is usually small, the data scale of the L0 layer is usually small. The metadata in the L1 to LN layers of the LSM tree can be stored on the disk, and the storage space of the disk is usually large. Therefore, the L1 to LN layers can have a larger data scale. Each layer from L0 to LN can have one or more LSM tree subtrees.

[0048] When writing new data to the LSM tree, the new metadata and its corresponding keywords can be written into the subtree of the L0 layer in the form of key-value pairs. When the amount of data in the L0 layer exceeds a certain threshold, the data in the L0 layer can be merged, and the merged data can be written to the subtree in the L1 layer. In this way, the memory can have more free memory space to store other newly written data. Similarly, when the data scale in the L1 layer exceeds the preset threshold, the data in the L1 layer can be merged, and the merged data can be written to the subtree in the L2 layer, and so on. In the process of continuous merging and writing, the data scale of the next layer of subtree usually becomes larger and larger.

[0049] During the merging process of data in each layer, the Values ​​corresponding to the same Key can be merged into a new Value, and a new key-value pair can be constructed based on the new Value and the original Key. The Key of the new key-value pair remains unchanged, and the Value of the new key-value pair is the new Value obtained after the merger. In this way, redundant and invalid information in the LSM tree can be eliminated to achieve the purpose of reducing storage overhead.

[0050] Read amplification: When reading data, the actual amount of data read is greater than the actual amount of data. For example, in an LSM tree, you must first check whether the current key exists in the memory table. If not, you continue to search in the ordered key-value pair set.

[0051] Write amplification: When writing data, the actual amount of data written is greater than the actual amount of data. For example, when writing to an LSM tree, a compact operation may be triggered, resulting in the actual amount of data written being much greater than the data size of the key.

[0052] Space amplification: The actual disk space occupied by the data is greater than the actual size of the data. For a key, only the latest record is valid, and the previous records can be cleaned up and recycled.

[0053] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] FIG1 is a schematic diagram illustrating an implementation environment of a data processing method according to an exemplary embodiment. As shown in FIG1 , the implementation environment may include at least a terminal 01 and a server 02. The terminal 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present application.

[0056] Specifically, the server 02 can be used to quickly locate and identify expired data and recycle expired data. Optionally, the server 02 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0057] Specifically, the terminal 01 can be used to collect expired data generated by write operations and send the expired data to the server 02. The terminal 01 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, intelligent voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions here.

[0058] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0059] It should be noted that FIG1 is only an example. In other scenarios, other implementation environments may also be included.

[0060] FIG2 is a flow chart of a data processing method according to an exemplary embodiment. The method can be used in the implementation environment of FIG1 . This specification provides the method operation steps as described above in the embodiment or flowchart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, as shown in FIG2 , the method may include:

[0061] S201, in response to a write operation on target data in a log-structured merge tree, determining expired data generated by the write operation; the log-structured merge tree includes at least two data layers, each data layer includes a preset number of disk files, and each data layer corresponds to a cache structure, each cache structure is used to store metadata of expired data in the corresponding data layer.

[0062] Optionally, the target data is data already stored in an LSM tree, which is the target of the write operation. An LSM tree comprises at least two data layers, each of which comprises a preset number of disk files. The data structure of the LSM tree on disk is an ordered set of key-value pairs. That is, the disk files store metadata about the data and the keywords corresponding to the metadata. The metadata and the keywords are written into the LSM tree as key-value pairs.

[0063] Optionally, the write operation is a write operation that can generate expired data. Since the write operation that can generate expired data will eventually generate an additional expired data when it falls on the data layer, in order to quickly and accurately locate the disk file where the expired data is located, it can be determined based on the front-end operation which write loads that fall in the database system will generate expired data. Furthermore, in the database system, except for the Insert write operation, which will not generate expired data, other write operations will generate an expired data after being successfully written. Therefore, the write operation that can generate expired data can be a write operation other than the Insert write operation, such as update, delete, and other operations of the application load.

[0064] Exemplarily, the expired data may include, but is not limited to, invalid data, data with a lower version, dirty data, etc. Dirty data refers to data that is not within a given range or is meaningless to actual business, or data in an illegal format, or data with non-standard encoding, or data with unclear business logic.

[0065] It should be noted that in addition to the two-part structure of memory and disk file in the related technology, the LSM tree in the embodiment of the present application also maintains a cache structure (memory cache Buffer) for each data layer. Each cache structure is used to store the metadata of expired data in the corresponding data layer. A simple jump table is used inside each memory cache Buffer to maintain the order of the key.

[0066] Figure 3 is a schematic diagram of a cache structure according to an exemplary embodiment. As shown in Figure 3, assuming that the LSM tree includes 6 data layers (L0, L1, L2, L3, L4, L5), a memory cache Buffer0 can be maintained for L0, a memory cache Buffer1 can be maintained for L1, a memory cache Buffer2 can be maintained for L2, a memory cache Buffer3 can be maintained for L3, a memory cache Buffer4 can be maintained for L4, and a memory cache Buffer5 can be maintained for L5. The numbers in each data layer refer to the file identifier (SSTable ID) of the disk file, for example, "34" in the L3 layer. The numbers below the file identifier refer to the range of the maximum key and minimum key corresponding to the disk file. For example, "151" below "34" in the L3 layer refers to the minimum key, and "200" refers to the "maximum key". The embodiment of the present application stores the metadata of expired data through the cache structure, and the metadata of the expired data is identified by the corresponding key.

[0067] In this embodiment, since the data written into the LSM tree exists in the form of a key-value pair consisting of its metadata and the key corresponding to the metadata, when the terminal object writes the metadata of the target data stored in the LSM tree, expired data will be generated. The terminal can send the expired data to the server, or the server can directly obtain the expired data from the terminal.

[0068] It should be noted that the same terminal object can write different data in the LSM at the same time, thereby generating multiple expired data. Different terminal objects can also write different data or the same data in the LSM at the same time, thereby generating multiple expired data. This application does not make specific restrictions on this.

[0069] S203: Determine the keyword of the target data as the target keyword of the expired data.

[0070] S204 , based on the target keyword, searching for the target disk file where the expired data is located from a preset number of disk files included in each data layer.

[0071] Since the LSM tree itself stores the metadata of the data and the key-value pairs consisting of the keys corresponding to the metadata, each key-value pair can include a keyword (Key) and the value (Value) corresponding to the Key. The Value is the metadata itself, and the Key is used to identify the metadata as the Value. During the write operation on the target data, the corresponding Key does not change, so the server can directly use the keyword of the target data stored in the LSM tree as the target keyword of the expired data. And through the target keyword, the server can actively, accurately and quickly search for the target disk file where the expired data is located from the preset number of disk files included in each data layer, that is, actively, accurately and quickly identify which target disk file in which data layer in the LSM tree the expired data is located.

[0072] S205 , writing the target keyword into the cache structure corresponding to the data layer where the target disk file is located; the target keyword is used to identify the meta information of expired data in the target disk file.

[0073] In this embodiment, after the server locates the target disk file where the expired data is located, since a corresponding cache structure is maintained for each disk file, the server can directly write the target keyword of the expired data into the cache structure corresponding to the data layer where the target disk file is located, so as to identify the metadata of the expired data in the target disk file through the target keyword.

[0074] S207, in response to the number of keywords in the cache structure corresponding to at least one data layer meeting a preset quantity condition, using the metadata identified by the keywords in the cache structure corresponding to at least one data layer, locate the expired data in the disk files included in at least one data layer, and recycle the located expired data.

[0075] In this embodiment, when the server locates expired data through a cache structure designed for expired data and an algorithm for quickly locating expired data, the server can actively identify the amount of expired data in each disk file, and based on this information, can trigger a recycling program for expired data in a variety of ways.

[0076] The preset quantity condition may be the relationship between the number of keywords and a threshold value. The threshold value may be preset as needed or dynamically set based on historical recycling records. The preset quantity condition may be that the relative relationship between the number of keywords and other quantities conforms to a preset relationship. Other quantities may be, for example, a preset number or the total number of keywords in the cache structure corresponding to at least one data layer. The relative relationship may be, for example, a ratio or other functional relationship. Recycling processing may be a process of releasing storage space occupied by expired data. Furthermore, recycling processing may be a process of deleting expired data and reclaiming the storage space occupied by the deleted expired data.

[0077] The embodiment of the present application, through a cache structure designed for expired data and an algorithm for quickly locating expired data, can convert passive identification of expired data into active identification, thereby quickly locating the specific disk file in which the expired data is located, and writing the corresponding keyword into the corresponding cache structure. The number of keywords stored in the cache structure can accurately estimate the amount of expired data in each disk file, and then based on the number of expired data in each disk file, the expired data in each disk file can be accurately and promptly recovered, thereby improving the storage efficiency, system performance and query efficiency of the distributed database and reducing the waste of space and resources. Specific beneficial effects can be:

[0078] Timely release of storage space: By proactively identifying and recycling expired data, storage space can be released promptly, preventing data from occupying excessive storage resources. This helps improve storage efficiency and reduce space waste.

[0079] Improve system performance: By proactively identifying and recycling expired data, unnecessary data access and processing operations can be reduced, thereby improving the system's read and write performance. Reducing the presence of expired data can reduce the complexity and redundancy of data access.

[0080] Optimize query efficiency: Proactively identifying expired data can reduce the amount of data that needs to be processed during query operations, thereby improving query efficiency. By excluding expired data, you can locate and retrieve the required valid data more quickly.

[0081] Improve data consistency and reliability: Proactively identifying expired data can reduce its impact on the system and improve data consistency and reliability. Timely clearing of expired data can prevent its negative impact on system functionality and performance.

[0082] It should be noted that the process of searching for the target disk file containing expired data from a preset number of disk files included in each data layer based on the target keyword in step S204 can be implemented in a variety of ways, which are not specifically limited.

[0083] In one embodiment, a preset number of disk files included in each data layer can be read to find the target disk file where the expired data is located. In another embodiment, the target disk file where the expired data is located can be determined by combining binary search and Bloom filter.

[0084] It should be noted that the combination of binary search and Bloom filter can be implemented in a variety of ways. In one approach, binary search and Bloom filter operations can be performed sequentially on each data layer in the LSM tree. In another approach, binary search and Bloom filter operations can be performed on each data layer in the LSM tree in parallel, or in a random order.

[0085] The following describes step S204 by taking the example of performing a binary search and a Bloom filter operation on each data layer in the LSM tree in parallel, or performing a binary search and a Bloom filter operation on each data layer in the LSM tree in a random order. FIG4 is a flow diagram of the steps of searching for a target disk file containing expired data from a preset number of disk files included in each data layer based on a target keyword, which may include:

[0086] S2041 , determining preliminary disk files corresponding to expired data based on the target keyword and sorting position information of a preset number of disk files included in each data layer in each data layer.

[0087] S2043 : When it is determined based on the Bloom filter information of the preliminary selected disk file that expired data exists in the preliminary selected disk file, determine the preliminary selected disk file as the target disk file.

[0088] In this embodiment, a binary search can be performed based on the target keyword and the sorting position information of a preset number of disk files included in each data layer to determine the primary disk file corresponding to the expired data, that is, to determine the range of disk files within which the expired data may be located. Bloom filter information of the primary disk file is then obtained from memory, and the server determines whether the expired data exists in the primary disk file using the Bloom filter information. If so, the server determines that the primary disk file is the target disk file; if not, the server determines that the primary disk file is not the target disk file.

[0089] It should be noted that corresponding Bloom filter information can be generated in advance for each disk file in the LSM tree. The generation process of the Bloom filter information may include:

[0090] A Bloom filter is a data structure consisting of a bit array of a preset length and a preset number of independent hash functions. The bit array is initialized to 0, so all hash functions can hash the input data as evenly as possible. When the key corresponding to the data in each disk file is inserted into the Bloom filter, the key is calculated through a preset number of hash functions to generate a preset number of hash values. The key is then mapped to a position in the bit array using the preset number of hash values. The element in the bit array at the mapped position becomes 1, and one element can be mapped to the preset number of positions.

[0091] Therefore, it is possible to first determine the range of the disk file in which the expired data may be located through a binary search. Since the binary search has fewer comparisons, faster search speed, and better average performance, it is possible to quickly and accurately determine the range of the disk file in which the expired data may be located while consuming fewer resources, that is, to quickly and accurately determine the preliminary disk file in which the expired data may be located. On this basis, the Bloom filter information of the preliminary disk file is used to determine whether the expired data is actually located in the preliminary disk file. Since the Bloom filter information uses a bit array and several hash functions to determine whether a key exists in a certain disk file, the space utilization rate is high, and only the bit array needs to be queried, without the need for actual query data, the query efficiency is high, thereby improving the efficiency and accuracy of determining the target disk file and reducing the consumption of system resources in the process of determining the target disk file.

[0092] In an optional embodiment, a preset number of disk files in each candidate data layer of at least two data layers are arranged in order, and the method of determining the preliminary disk files corresponding to the expired data based on the target keyword and the sorting position information of the preset number of disk files included in each data layer in each data layer may include:

[0093] The target keyword is compared with the first candidate keyword of the disk file in the middle position in each candidate data layer.

[0094] When the target keyword is equal to the first candidate keyword, the primary selection disk file is determined to be the disk file in the middle position in each candidate data layer.

[0095] In the case that the target keyword is smaller than the first candidate keyword, a disk file having the same keyword as the target keyword is searched from the disk files located before the disk file in the middle position to obtain a preliminary disk file.

[0096] In the case that the target keyword is greater than the first candidate keyword, a disk file having the same keyword as the target keyword is searched from the disk files after the disk file in the middle position to obtain a preliminary disk file.

[0097] In this embodiment, a preset number of disk files in each candidate data layer are arranged in order, and the candidate data layer is a data layer other than the first data layer in at least two data layers. For binary search, the server can compare the target keyword with the first candidate keyword of the disk file in the middle position in each candidate data layer to determine whether the target keyword is equal to the first candidate keyword. If so, the server determines that the starting disk file is the disk file in the middle position in each candidate data layer. If not, and the target keyword is smaller than the first candidate keyword, the server searches for a disk file with the same keyword as the target keyword from the disk files before the disk file in the middle position to obtain a preliminary disk file. If not, and the target keyword is larger than the first candidate keyword, the server searches for a disk file with the same keyword as the target keyword from the disk files after the disk file in the middle position to obtain a preliminary disk file.

[0098] In other implementations, if the disk file where the expired data may fall is not found in the disk files in the candidate data layers in the above manner, it can be considered that the expired data falls into the first data layer.

[0099] In other implementations, “the target keyword is equal to the first candidate keyword” may also be replaced by “the difference between the target keyword and the first candidate keyword is less than a preset difference threshold”.

[0100] Therefore, the first candidate keyword of the disk file in the middle position in each data layer can be used as the dividing line of binary search. With fewer comparisons and less resource consumption, it is possible to quickly and accurately determine the range of the disk file in which the expired data may be located, that is, to quickly and accurately determine the preliminary disk file in which the expired data may be located.

[0101] In an optional embodiment, in step S2043, when it is determined based on the Bloom filter information of the preliminary selected disk file that expired data exists in the preliminary selected disk file, determining the preliminary selected disk file as the target disk file may include:

[0102] The target keyword is hashed using a preset number of hash functions to obtain a preset number of hash values.

[0103] The target keyword is mapped to a bit array through a preset number of hash values ​​to obtain a preset number of mapping results.

[0104] When the preset number of mapping results are all 1, it is determined that the expired data exists in the preliminary selected disk file, and the preliminary selected disk file is determined as the target disk file.

[0105] In this embodiment, corresponding Bloom filter information is generated in advance for each disk file in the LSM tree. The Bloom filter information is a data structure consisting of a bit array with a preset length of bits and a preset number of independent hash functions. The bit array is initialized to 0, and the elements at the positions where the bit array is mapped become 1.

[0106] Based on this, the server can perform hash processing on the target keyword through the preset number of hash functions and also obtain a preset number of hash values. The server maps the preset number of hash values ​​to a bit array to obtain a preset number of mapping results. If one of the preset number of mapping results is 0, that is, one is not 1, it indicates that the target keyword is definitely not in the preliminary disk file, and it is determined that the preliminary disk file is not the target disk file. If the preset number of mapping results are all 1, it indicates that the target keyword may exist in the preliminary disk file, and it is determined that the preliminary disk file is the target disk file.

[0107] Since the Bloom filter information uses a bit array and several hash functions to determine whether a key exists in a disk file, if one of the preset number of mapping results is 0, it means that it is definitely not in the disk file. If the preset number of mapping results are all 1, it indicates that the target keyword may exist in the preliminary disk file. The preliminary disk file is then used as the target disk file, thereby avoiding the resource consumption of reading the disk file in the data layer to view the target disk file where the target keyword is located, thereby improving the efficiency and accuracy of determining the target disk file and reducing the consumption of system resources in the target disk file determination process.

[0108] The following describes step S204 by taking the example of performing binary search and Bloom filter operations sequentially according to the order of each data layer in the LSM tree. FIG5 is a flowchart illustrating the steps of searching for a target disk file containing expired data from a preset number of disk files included in each data layer based on a target keyword, which may include:

[0109] S2042, determining that the last layer of the log structure merge tree is the current data layer; and arranging a preset number of disk files in the current data layer in order.

[0110] S2044 , determining the primary disk files corresponding to the expired data in the current data layer based on the target keyword and sorting position information of a preset number of disk files in the current data layer.

[0111] S2046 , when it is determined based on the Bloom filter information of the corresponding preliminary disk file that the expired data does not exist in the corresponding preliminary disk file, re-determine the upper data layer of the current data layer as the current data layer.

[0112] S2048, repeating the operation of re-determining the previous data layer of the current data layer as the current data layer based on the target keyword and the sorting position information of a preset number of disk files in the current data layer, until the current data layer is the first data layer of the log structure merge tree.

[0113] S20410: Determine that the first data layer is the target disk file.

[0114] Accordingly, the above method may further include:

[0115] When it is determined based on the Bloom filter information of the corresponding preliminary disk file that expired data exists in the corresponding preliminary disk file, the corresponding preliminary disk file is determined as the target disk file.

[0116] In this embodiment, since the files in the L0 layer (first layer) of the LSM tree are intertwined, and the disk files from L1 to the last layer are all sorted in order, if the target disk file can be found when traversing from the last data layer, there is no need to traverse the data layers before the last data layer, which can reduce the number of traversals and reduce the consumption of system resources in the process of determining the target disk file. Therefore, the server can first traverse the last data layer, take the last data layer as the current data layer, and first determine the preliminary disk file corresponding to the expired data in the current data layer through a binary search based on the target keyword and the sorting position information of a preset number of disk files in the current data layer. Then, based on the Bloom filter information of the preliminary disk file, determine whether the expired data exists in the preliminary disk file corresponding to the current data layer. If so, it indicates that the expired data does exist in the preliminary disk file, and the preliminary disk file corresponding to the current data layer is determined as the target disk file.

[0117] If not, the server will re-traverse the data layer above the current data layer, and use the data layer above the current data layer as the current data layer again, repeating the sorting position information of the preset number of disk files in the current data layer based on the target keyword and the current data layer, and determining the preliminary disk file corresponding to the expired data in the current data layer through binary search, and then determining whether the expired data exists in the preliminary disk file corresponding to the current data layer based on the Bloom filter information of the preliminary disk file. If so, it indicates that the expired data does exist in the preliminary disk file, and the preliminary disk file corresponding to the current data layer is determined to be the target disk file.

[0118] If not, continue to traverse the data layer above the current data layer, and use the data layer above the current data layer as the current data layer, and repeat the above operation until traversing to the first data layer. If after traversing to the first data layer (that is, the current data layer is the first data layer), the target disk file where the expired data is located is still not found, then the expired data is most likely located in the first data layer, and the server can directly determine the disk file in the first data layer as the target disk file.

[0119] The following example illustrates the process of determining the target disk file where the expired data is located:

[0120] Assume that the log structure merge tree includes 5 data layers, namely L0 data layer, L1 data layer, L2 data layer, L3 data layer, and L4 data layer. The server first uses the L4 data layer as the current data layer, and determines the preliminary disk file corresponding to the expired data in the L4 data layer through the above-mentioned binary search method based on the target keyword and the sorting position information of the preset number of disk files in the L4 data layer, that is, determines which disk file in the L4 data layer the expired data may fall into. The server determines whether the expired data actually exists in the disk file based on the Bloom filter information of the preliminary disk file. If so, the disk file in the L4 data layer is used as the target disk file, so that the key of the expired data can be written into the cache structure corresponding to the L4 data layer later. If not, the server uses the L3 data layer as the current data layer again.

[0121] The server determines the preliminary disk file corresponding to the expired data in the L3 data layer based on the target keyword and the sorting position information of a preset number of disk files in the L3 data layer through the above-mentioned binary search method, that is, determines which disk file in the L3 data layer the expired data may fall into. The server determines whether the expired data actually exists in the disk file based on the Bloom filter information of the preliminary disk file. If so, the disk file in the L3 data layer is used as the target disk file so that the expired file can be subsequently written into the cache structure corresponding to the L3 data layer. If not, the server re-uses the L2 data layer as the current data layer.

[0122] Similarly, if the target disk file where the expired data is located is not found in the L1 data layer, the server traverses the L0 data layer and uses the disk file in the L0 data layer as the target disk file so that the key of the expired data can be written into the cache structure corresponding to the L0 data layer.

[0123] In the embodiment of the present application, the method of performing binary search and Bloom filter operations in sequence according to the order of each data layer in the LSM tree does not traverse each data layer. Instead, the previous data layer will be traversed only when the target disk file where the key of the expired data is located is not found in the next data layer. When the target disk file is found in the next data layer, there is no need to traverse the previous data layer, which can reduce the number of traversals and thus reduce the consumption of system resources in the process of determining the target disk file.

[0124] In an optional embodiment, a corresponding counting device may be provided for each disk file in each data layer; then, after writing the target keyword into the cache structure corresponding to the data layer where the target disk file is located, the method may further include:

[0125] The number of counting devices corresponding to the target disk file is increased by 1.

[0126] Accordingly, in step S207, in response to the number of keywords in the cache structure corresponding to at least one data layer meeting a preset number condition, locating expired data in the disk files included in the at least one data layer using the meta information identified by the keywords in the cache structure corresponding to each of the at least one data layer, and recycling the located expired data may include:

[0127] According to the number of counting devices corresponding to the disk files included in each data layer, the expired data in the disk files included in each data layer is recycled.

[0128] In this embodiment, a counting device, such as a counter, can also be maintained for each disk file included in each data layer of the LSM tree. The number in the counting device represents the number of expired data existing in the corresponding disk file. After a keyword is stored in a corresponding cache structure, the number of the counting device corresponding to the disk file where the keyword is located can be increased by 1. After the server writes the target keyword into the cache structure corresponding to the data layer where the target disk file is located, the server can increase the number of the counting device corresponding to the target disk file by 1. The number of the counting device corresponding to the target disk file is used to represent the number of expired data existing in the target disk file.

[0129] It should be noted that when the server writes the target keyword into the cache structure corresponding to the data layer where the target disk file is located and adds 1 to the number of counting devices corresponding to the target disk file, there may also be an operation of writing other keywords into the cache structure corresponding to the data layer where other disk files are located and adding 1 to the counting devices corresponding to other disk files. That is, each layer of data layer includes a corresponding number of counting devices for the disk files. The server can recycle expired data in the disk files included in each data layer according to the number of counting devices corresponding to the disk files included in each data layer. Since the number of maintained counting devices can accurately express the information of expired data in each disk file, the number in the maintained counting device is used as the basis for triggering data recovery, so that expired data can be recycled in a timely and accurate manner.

[0130] It should be noted that the above-mentioned recycling of expired data in the disk files included in each data layer according to the number of counting devices corresponding to the disk files included in each data layer can be achieved in various ways, that is, the recycling program can be triggered in a variety of ways, and there is no specific limitation on this.

[0131] In one embodiment, the decision to initiate a data recovery process for expired data in a disk file can be made based on whether the expired data in the disk file meets a preset quantity requirement. In another embodiment, the data recovery process can be integrated into the compaction process, adding an additional mechanism to trigger the compaction process to achieve the purpose of recovering expired data.

[0132] The following describes the process of recycling expired data in a disk file, taking "actively deciding whether to start a recycling program for expired data in a disk file based on whether the expired data in the disk file meets a preset number condition" as an example:

[0133] Accordingly, the above-mentioned recycling of expired data in the disk files included in each data layer according to the number of counting devices corresponding to the disk files included in each data layer may include:

[0134] When the number of counting devices corresponding to the preset disk file meets the preset number condition, the expired data in the preset disk file is located by using the meta information identified by the keyword in the cache structure corresponding to the data layer where the preset disk file is located, the located expired data is deleted, and the preset disk file with the expired data deleted is rewritten to the disk used to store the preset disk file; or

[0135] When the number of counting devices corresponding to the preset disk file and the number of counting devices corresponding to the adjacent disk files both meet the preset number conditions, the metadata identified by the keyword in the cache structure corresponding to the data layer where the preset disk file is located is used to locate the expired data in the preset disk file, delete the located expired data and the expired data in the adjacent disk files, rewrite the preset disk file from which the expired data has been deleted to the disk used to store the preset disk file, and rewrite the adjacent disk file from which the expired data has been deleted to the disk used to store the adjacent disk files.

[0136] Optionally, the preset disk file may be any disk file among the disk files included in each data layer, and the adjacent disk file may be a disk file that is located in the same data layer as the preset disk file and is adjacent to the preset disk file.

[0137] Optionally, the “number of counting devices corresponding to the preset disk file satisfies the preset quantity condition” may mean that the ratio between the number of counting devices corresponding to the preset disk file (i.e., the number of expired data present in the preset disk file) and the total number of all data included in the preset disk file is greater than a preset ratio threshold value, which may be set according to actual business needs and is not specifically limited thereto. Alternatively, the “number of counting devices corresponding to the preset disk file satisfies the preset quantity condition” may mean that the number of counting devices corresponding to the preset disk file (i.e., the number of expired data present in the preset disk file) is greater than a certain preset quantity threshold value, which may be set according to actual business needs and is not specifically limited thereto.

[0138] The server can traverse the counting devices maintained for each disk file in real time or periodically. If the server finds that the number of counting devices corresponding to a preset disk file meets a preset number condition, it will actively trigger a recycling process for the preset disk file. In one embodiment, the preset disk file can be rewritten separately to delete the expired data in the preset disk file. For example, the expired data in the preset disk file can be deleted and the preset disk file with the expired data deleted can be rewritten to the disk.

[0139] In another embodiment, several adjacent disk files can be jointly rewritten to improve the efficiency of expired data recovery. For example, the joint rewriting process can be as follows: the server can traverse the counting devices maintained for each disk file in real time or periodically. If the server finds that the number of counting devices corresponding to a preset disk file meets the preset number condition, and at the same time finds that the number in the counting devices corresponding to the adjacent disk files meets the preset number condition, then the preset disk file and the adjacent disk files are jointly rewritten. For example, the expired data in the preset disk file and the expired data in the adjacent disk files are deleted at the same time, and the preset disk file after the expired data is deleted and the adjacent disk file after the expired data is deleted are rewritten to the disk at the same time.

[0140] In this embodiment, since the number of maintained counting devices can accurately express the information of expired data in each disk file, whether the number in the counting device corresponding to the disk file meets the preset quantity condition is used as the basis for triggering data recovery. The recovery of expired data does not depend on the compression program, but the expired data is actively recovered, so that the recovery of expired data is within a controllable range, thereby improving the efficiency and accuracy of expired data recovery and reducing the cost of expired data recovery.

[0141] The following describes the process of recycling expired data in disk files, taking "attaching the recycling program to the compression program and adding an additional compression triggering mechanism to achieve the purpose of recycling expired data" as an example:

[0142] Accordingly, the above-mentioned recycling of expired data in the disk files included in each data layer according to the number of counting devices corresponding to the disk files included in each data layer may include:

[0143] In response to the merge instruction, it is determined whether the number of expired data in the disk files within a first preset range is greater than a first preset number threshold based on the number of counting devices corresponding to the disk files included in each data layer; the first preset range is the data range in any one data layer of at least two data layers.

[0144] If it is determined that the amount of expired data in the disk files within a first preset range is greater than a first preset number threshold, and the total amount of data in the disk files within the first preset range is less than a second preset number threshold, the data within the first preset range and the data within the second preset range are sorted to recycle the expired data in the disk files within the first preset range. The data layer in which the second preset range is located is a data layer below the data layer in which the first preset layer is located, and the data within the first preset range and the data within the second preset range have an intersection.

[0145] It should be noted that the merge instruction can be triggered by the terminal object or automatically triggered periodically by the server.

[0146] When a compression is triggered, the server can select data within a first preset range in an adjacent layer for organization, and the data within the first preset range satisfies that the total amount of data involved does not exceed the second preset quantity threshold, and the proportion of expired data involved is greater than the first preset quantity threshold. Optionally, the first preset range can be the entire data range in any data layer, or it can be a partial data range in any data layer. The first preset quantity threshold and the second preset quantity threshold can be set according to actual business needs and are not specifically limited to this. For example, the proportion of expired data is greater than the first preset quantity threshold, which may mean that the proportion of expired data in the first preset range is the highest.

[0147] In response to the merge instruction, the server can obtain the number of counting devices corresponding to the disk files included in each data layer. Since the maintained number of counting devices can accurately express the information of expired data in each disk file in each data layer, it is possible to determine whether the number of expired data in disk files within a first preset range in any data layer is greater than the first preset number threshold based on the number of counting devices corresponding to the disk files included in each data layer.

[0148] If the server determines that the amount of expired data in the disk files within the first preset range is greater than a first preset threshold, the server may further determine whether the total amount of data in the disk files within the first preset range is less than a second preset threshold. If so, the server may organize data within a certain range in an adjacent data layer. Specifically, the server may sort the data within the first preset range and the data within the second preset range to recycle the expired data in the disk files within the first preset range.

[0149] It should be noted that, since data within a certain range in adjacent data layers are organized, the data layer where the second preset range is located may be the data layer below the data layer where the first preset layer is located, and the data within the first preset range has an intersection with the data within the second preset range.

[0150] Therefore, data within a certain range in adjacent layers can be organized through an additional triggered compression mechanism. Since the number of expired data in disk files within the first preset range is greater than the first preset number threshold, and the total number of data in disk files within the first preset range is less than the second preset number threshold, as much expired data as possible can be recovered, thereby improving the efficiency of expired data recovery.

[0151] In other embodiments, it is also possible not to maintain a counting device for each disk file, and directly read the number of expired data in each disk file from the cache structure corresponding to each data layer, and recycle the expired data in each disk file based on the number of expired data in each disk file read. In one embodiment, the server can directly read the number of expired data in each disk file from the cache structure corresponding to each data layer, and when it is determined that the ratio of expired data in a certain disk file to the total amount of data included in the disk file reaches a certain ratio, rewrite the disk file separately to delete the expired data; or rewrite several adjacent disk files to delete the expired data. In another embodiment, an additional mechanism for triggering compression can also be added. When compression is triggered, data within a certain range in adjacent layers is selected for organization to recycle as much expired data as possible, thereby improving the efficiency of recycling expired data.

[0152] The following is an overall description of the data processing method in the embodiment of the present invention:

[0153] FIG6 is a flow chart of a data processing method according to an exemplary embodiment. As shown in FIG6 :

[0154] Assume that the LSM tree includes 8 data layers, namely L0, L1, L2, L3, L4, L5, L6, and L7.

[0155] Assume that write operations are performed on the target data corresponding to Key1 and the target data corresponding to Key2 stored in the LSM data at the same time, then the expired data is the target data corresponding to Key1 and the target data corresponding to Key2, and the target keywords are Key1 and Key2.

[0156] Based on Key1 and Key2, the server obtains the file identification information (SST ID) of the preset number of disk files included in each data layer. The server obtains the bloom filter information corresponding to the disk file according to the SST ID, so as to find out which disk file on which data layer the target data corresponding to Key1 and the target data corresponding to Key2 are located in according to the bloom filter information. As shown in Figure 6, assuming that the target data corresponding to Key1 and the target data corresponding to Key2 are located in disk file 1 of layer L6

[0157] During the process of recovering expired data, an additional mechanism for triggering compression can be added to organize the data within a certain range in the adjacent layers. Specifically, it can be: assuming that the number of expired data in the disk files within the first preset range (the range corresponding to disk file 1) in the L6 layer is greater than the first preset number threshold, and the total number of data in the disk files within the first preset range is less than the second preset number threshold, and the data in the L6 layer within the first preset range and the data in the L7 layer within the second preset range have an intersection, then when an additional compression mechanism is triggered, the server can sort the data in the L6 layer within the first preset range and the data in the L7 layer within the second preset range to recover the expired data in the disk files (i.e., disk file 1) in the L6 layer within the first preset range to obtain the target disk file.

[0158] The embodiments of the present application can be applied to distributed database management systems (for example, the distributed database TDSQL system). The TDSQL distributed database system adopts a storage-computing separation architecture and supports distributed transactions and distributed storage. The recycling mechanism for expired data in the LSM tree in the embodiments of the present application can be used to recycle expired data in a single storage node in distributed storage. Because the recycling mechanism can actively identify expired data and is more convenient and flexible in cleaning up expired data, it can enable the TDSQL database system to have better improvements in disk space amplification and read-write amplification.

[0159] FIG7 is a diagram showing an overall framework of a TDSQL distributed database system according to an exemplary embodiment. As shown in FIG7 , the TDSQL distributed database system may include:

[0160] Computing module (SQLEngine): adopts multi-master mode, and each SQLEngine is readable and writable; the stateless design allows the flexible addition or removal of any number of computing nodes at any time according to business traffic needs.

[0161] Storage module (TDStore): Based on the business data storage requirements, distributed storage nodes (TDStore nodes) can be added or removed, and elastic scaling of capacity can be achieved through automatic data migration without the business layer being aware of it.

[0162] Management and control module (TDMetaCluster): Responsible for scheduling the splitting, merging, migration, and master node switching of data partitions; scheduling the expansion and contraction of the storage layer; implementing load balancing scheduling at the storage layer; and providing abnormal event alerts in various dimensions.

[0163] Figure 8 is a diagram of the storage framework of a single storage node, according to an exemplary embodiment. As shown in Figure 8, TDSQL, as a distributed database management system, stores data from several data partitions on each storage node. Data from different data partitions can be scheduled on each storage node. Specifically for a single storage node, data from all data partitions is stored in a single LSM tree. The LSM tree uses a compression program to continuously organize data at different layers to achieve balanced read and write performance for stored data.

[0164] FIG9 is a block diagram of a data processing device according to an exemplary embodiment. As shown in FIG9 , the data processing device includes:

[0165] The expired data acquisition module 901 is used to determine the expired data generated by the write operation in response to the write operation on the target data in the log structure merge tree; the log structure merge tree includes at least two data layers, each data layer includes a preset number of disk files, and each data layer corresponds to a cache structure, and each cache structure is used to store the metadata of the expired data in the corresponding data layer.

[0166] The target disk file search module 903 is configured to determine the keyword of the target data as the target keyword of the expired data, and search for the target disk file where the expired data is located from a preset number of disk files included in each data layer based on the target keyword.

[0167] The keyword writing module 905 is used to write the target keyword into the cache structure corresponding to the data layer where the target disk file is located; the target keyword is used to identify the meta information of expired data in the target disk file.

[0168] The recycling module 907 is used to locate expired data in the disk files included in at least one data layer in response to the number of keywords in the cache structure corresponding to at least one data layer meeting a preset number condition, using the metadata identified by the keywords in the cache structure corresponding to at least one data layer, and recycle the located expired data.

[0169] In an optional embodiment, the target disk file search module includes:

[0170] The preliminary disk file determination unit is configured to determine the preliminary disk file corresponding to the expired data based on the target keyword and the sorting position information of a preset number of disk files included in each data layer in each data layer.

[0171] The target disk file determining unit is configured to determine the preliminary disk file as the target disk file when it is determined based on the bloom filter information of the preliminary disk file that expired data exists in the preliminary disk file.

[0172] In an optional embodiment, a preset number of disk files in each candidate data layer of at least two data layers are arranged in order, and the candidate data layer is a data layer other than the first data layer of the at least two data layers; the preliminary disk file determination unit includes:

[0173] The comparison subunit is used to compare the target keyword with the first candidate keyword of the disk file in the middle position in each candidate data layer.

[0174] The intermediate disk file determination subunit is used to determine the primary disk file as the disk file in the middle position in each candidate data layer when the target keyword is equal to the first candidate keyword.

[0175] The first search subunit is used to search for a disk file with the same keyword as the target keyword from disk files before the disk file in the middle position when the target keyword is smaller than the first candidate keyword, so as to obtain a preliminary disk file.

[0176] The second search subunit is used to search for a disk file with the same keyword as the target keyword from disk files after the disk file in the middle position when the target keyword is greater than the first candidate keyword, so as to obtain a preliminary disk file.

[0177] In an optional embodiment, the Bloom filter information of the preliminary selected disk file is a data structure consisting of a bit array with a preset bit length and a preset number of hash functions; the target disk file determination unit includes:

[0178] A hash processing subunit, configured to perform hash processing on the target keyword using a preset number of hash functions to obtain a preset number of hash values;

[0179] A mapping subunit, configured to map a target keyword to a bit array through a preset number of hash values ​​to obtain a preset number of mapping results;

[0180] The target disk file generating subunit is used to determine that the expired data exists in the preliminary disk file when a preset number of mapping results are all 1, and determine the preliminary disk file as the target disk file.

[0181] In an optional embodiment, the apparatus further comprises:

[0182] The non-target disk file determination module is used to determine that the expired data does not exist in the preliminary disk file when any mapping result among a preset number of mapping results is 0, and determine that the preliminary disk file is not the target disk file.

[0183] In an optional embodiment, the target disk file search module includes:

[0184] The current data layer determining unit is used to determine the last layer of the log structure merge tree as the current data layer; a preset number of disk files in the current data layer are arranged in order;

[0185] a current preliminary disk file determination unit, configured to determine the preliminary disk file corresponding to the expired data in the current data layer based on the target keyword and sort position information of a preset number of disk files in the current data layer;

[0186] a redetermining unit, configured to redetermine a data layer above the current data layer as the current data layer when the corresponding preliminary disk file is determined based on the bloom filter information of the preliminary disk file;

[0187] A repeating unit is used to repeat the operation of re-determining the previous data layer of the current data layer as the current data layer based on the target keyword and the sorting position information of a preset number of disk files in the current data layer, until the current data layer becomes the first data layer of the log structure merge tree;

[0188] The target disk file determination subunit is used to determine the disk file in the first data layer as the target disk file.

[0189] In an optional embodiment, the apparatus further comprises:

[0190] The file existence determination module is used to determine the corresponding preliminary disk file as the target disk file when it is determined that expired data exists in the corresponding preliminary disk file based on the Bloom filter information of the preliminary disk file.

[0191] In an optional embodiment, the apparatus further comprises:

[0192] The counting adjustment module is used to increase the number of counting devices corresponding to the target disk file by 1.

[0193] Recycling module, including:

[0194] The recycling processing unit is used to recycle expired data in the disk files included in each data layer according to the number of counting devices corresponding to the disk files included in each data layer.

[0195] In an optional embodiment, the recycling unit includes:

[0196] A first deleting subunit is configured to locate expired data in a preset disk file using metadata identified by a keyword in a cache structure corresponding to a data layer where the preset disk file is located, delete the located expired data, and rewrite the preset disk file from which the expired data has been deleted to a disk for storing the preset disk file, when the number of counting devices corresponding to the preset disk file meets a preset number condition; or

[0197] The second deleting subunit locates expired data in the preset disk file using the meta information identified by the keyword in the cache structure corresponding to the data layer where the preset disk file is located, deletes the located expired data and the expired data in the adjacent disk files, rewrites the preset disk file from which the expired data has been deleted to the disk for storing the preset disk file, and rewrites the adjacent disk file from which the expired data has been deleted to the disk for storing the adjacent disk files;

[0198] The preset disk file is any disk file among the disk files included in each data layer, and the adjacent disk file is a disk file adjacent to the preset disk file.

[0199] In an optional embodiment, the recycling unit:

[0200] a response subunit, configured to, in response to the merge instruction, determine whether the number of expired data in the disk files within a first preset range is greater than a first preset number threshold value based on the number of counting devices corresponding to the disk files included in each data layer; the first preset range being a range of data in any one of the at least two data layers;

[0201] a sorting processing subunit, configured to, upon determining that the amount of expired data in the disk files within the first preset range is greater than a first preset amount threshold and the total amount of data in the disk files within the first preset range is less than a second preset amount threshold, sort the data within the first preset range and the data within the second preset range, so as to recycle the expired data in the disk files within the first preset range;

[0202] The data layer where the second preset range is located is a data layer below the data layer where the first preset range is located, and the data within the first preset range and the data within the second preset range have an intersection.

[0203] It should be noted that the device embodiment provided in the embodiments of the present application and the above-mentioned method embodiment are based on the same inventive concept.

[0204] An embodiment of the present application also provides an electronic device for data processing, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement a data processing method provided in any of the above embodiments.

[0205] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method provided in the above method embodiment.

[0206] Optionally, in the embodiments of this specification, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program code, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0207] The memory of the embodiment of this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, applications required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0208] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the data processing method provided in the above method embodiment.

[0209] The embodiments of the data processing methods provided in the embodiments of the present application can be executed on an electronic device, such as a terminal, a computer terminal, an electronic device or a similar computing device. Electronic device Figure 10 is a hardware structure block diagram of an electronic device provided according to an exemplary embodiment. As shown in Figure 10, the electronic device 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1010 (the central processing unit 1010 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1030 for storing data, and one or more storage media 1020 (such as one or more mass storage devices) for storing application programs 1023 or data 1022. Among them, the memory 1030 and the storage medium 1020 can be temporary storage or permanent storage. The program stored in the storage medium 1020 may include one or more modules, each module may include a series of instruction operations on the electronic device. Furthermore, the central processing unit 1010 may be configured to communicate with the storage medium 1020 and execute a series of instruction operations in the storage medium 1020 on the electronic device 1000. The electronic device 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1040, and / or one or more operating systems 1021, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0210] The input / output interface 1040 can be used to receive or send data via a network. The specific example of the above-mentioned network may include a wireless network provided by the communication provider of the electronic device 1000. In one example, the input / output interface 1040 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 1040 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly. It will be understood by those skilled in the art that the structure shown in Figure 10 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the electronic device 1000 may also include more or fewer components than those shown in Figure 10, or have a configuration different from that shown in Figure 10.

[0211] It should be noted that the order of the embodiments of the present application described above is for descriptive purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0212] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and electronic device embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0213] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data processing method, performed by an electronic device, the method comprising: In response to a write operation on target data in a log structure merge tree, determining expired data generated by the write operation; The log structure merge tree includes at least two data layers, each data layer includes a preset number of disk files, and each data layer corresponds to a cache structure, each cache structure is used to store metadata of expired data in the corresponding data layer; determining the keyword of the target data as the target keyword of the expired data; Based on the target keyword, searching for the target disk file where the expired data is located from a preset number of disk files included in each data layer; Writing the target keyword into a cache structure corresponding to the data layer where the target disk file is located; The target keyword is used to identify the meta information of expired data in the target disk file; and In response to the number of keywords in the cache structure corresponding to at least one data layer satisfying a preset number condition, expired data in the disk files included in the at least one data layer is located using the meta-information identified by the keywords in the cache structure corresponding to each of the at least one data layer, and the located expired data is recycled.

2. The data processing method according to claim 1, wherein based on the target keyword, searching for the target disk file where the expired data is located from a preset number of disk files included in each data layer comprises: Determine the preliminary disk file corresponding to the expired data based on the target keyword and the sorting position information of the preset number of disk files included in each data layer in each data layer; In a case where it is determined based on the Bloom filter information of the preliminary selected disk file that the expired data exists in the preliminary selected disk file, the preliminary selected disk file is determined to be the target disk file.

3. The data processing method according to claim 2, wherein a preset number of disk files in each candidate data layer of the at least two data layers are arranged in order, and the candidate data layer is a data layer other than the first data layer of the at least two data layers; The determining the preliminary disk file corresponding to the expired data based on the target keyword and the sorting position information of the preset number of disk files included in each data layer in each data layer includes: Comparing the target keyword with the first candidate keyword of the disk file in the middle position in each layer of the candidate data layer; In the case that the target keyword is equal to the first candidate keyword, the primary selection disk file is determined to be a disk file in the middle position in each candidate data layer.

4. The data processing method according to claim 3, wherein the determining the preliminary disk files corresponding to the expired data based on the target keyword and the sorting position information of the preset number of disk files included in each data layer in each data layer further comprises: In the case that the target keyword is smaller than the first candidate keyword, a disk file having the same keyword as the target keyword is searched from disk files located before the disk file in the middle position to obtain the preliminary disk file.

5. The data processing method according to claim 3 or 4, wherein the determining the preliminary disk files corresponding to the expired data based on the target keyword and the sorting position information of the preset number of disk files included in each data layer in each data layer further comprises: In the case that the target keyword is greater than the first candidate keyword, a disk file having the same keyword as the target keyword is searched from the disk files located after the disk file in the middle position to obtain the preliminary disk file.

6. The data processing method according to any one of claims 2 to 5, wherein the Bloom filter information of the preliminary disk file is a data structure composed of a bit array with a length of preset bits and a preset number of hash functions; and when it is determined based on the Bloom filter information of the preliminary disk file that the expired data exists in the preliminary disk file, determining the preliminary disk file as the target disk file comprises: The target keyword is hashed by using the preset number of hash functions to obtain a preset number of hash value; Mapping the target keyword to the bit array through the preset number of hash values ​​to obtain a preset number of mapping results; When the preset number of mapping results are all 1, it is determined that the expired data exists in the preliminary selected disk file, and the preliminary selected disk file is determined to be the target disk file.

7. The data processing method according to claim 6, further comprising: When any one of the preset number of mapping results is 0, it is determined that the expired data does not exist in the preliminary selected disk file, and it is determined that the preliminary selected disk file is not the target disk file.

8. The data processing method according to any one of claims 1 to 7, wherein based on the target keyword, searching for the target disk file where the expired data is located from a preset number of disk files included in each data layer comprises: Determine that the last layer of the log structure merge tree is the current data layer; The preset number of disk files in the current data layer are arranged in order; Determine the primary disk file corresponding to the expired data in the current data layer based on the target keyword and the sorting position information of a preset number of disk files in the current data layer; When it is determined based on the Bloom filter information of the corresponding preliminary disk file that the expired data does not exist in the corresponding preliminary disk file, re-determine the previous data layer of the current data layer as the current data layer; Repeating the operation of re-determining the previous data layer of the current data layer as the current data layer based on the target keyword and the sorting position information of the preset number of disk files in the current data layer, until the current data layer is the first data layer of the log structure merge tree; The disk file in the first data layer is determined as the target disk file.

9. The data processing method according to claim 8, further comprising: When it is determined that the expired data exists in the corresponding preliminary selected disk file based on the Bloom filter information of the corresponding preliminary selected disk file, the corresponding preliminary selected disk file is determined to be the target disk file.

10. The data processing method according to any one of claims 1 to 9, wherein a corresponding counting device is provided for each of the disk files in each of the data layers; the method further comprises: After writing the target keyword into the cache structure corresponding to the data layer where the target disk file is located, adding 1 to the number of counting devices corresponding to the target disk file; In response to the number of keywords in the cache structure corresponding to at least one data layer satisfying a preset number condition, using the meta information identified by the keywords in the cache structure corresponding to each of the at least one data layer, locating expired data in the disk file included in the at least one data layer, and recycling the located expired data, including: In response to the number of counting devices corresponding to the disk files included in at least one data layer satisfying a preset quantity condition, expired data in the disk files included in the at least one data layer is located using the meta-information identified by the keyword in the cache structure corresponding to each of the at least one data layer, and the located expired data is recycled.

11. The data processing method according to claim 10, in response to the number of counting devices corresponding to the disk files included in at least one data layer satisfying a preset number condition, locating expired data in the disk files included in the at least one data layer using meta information identified by a keyword in a cache structure corresponding to each of the at least one data layer, and recycling the located expired data, comprising: When the number of counting devices corresponding to the preset disk file meets the preset number condition, the metadata identified by the keyword in the cache structure corresponding to the data layer where the preset disk file is located is used to locate the expired data in the preset disk file, delete the located expired data, and rewrite the preset disk file from which the expired data has been deleted into the disk used to store the preset disk file.

12. The data processing method according to claim 10, in response to the number of counting devices corresponding to the disk files included in at least one data layer satisfying a preset number condition, locating expired data in the disk files included in the at least one data layer using meta information identified by a keyword in a cache structure corresponding to each of the at least one data layer, and recycling the located expired data, comprising: In the case where the number of counting devices corresponding to the preset disk file and the number of counting devices corresponding to the adjacent disk files both meet the preset number conditions, using the meta information identified by the keyword in the cache structure corresponding to the data layer where the preset disk file is located, locate the expired data in the preset disk file, delete the located expired data and the expired data in the adjacent disk file, rewrite the preset disk file from which the expired data has been deleted to the disk used to store the preset disk file, and rewrite the adjacent disk file from which the expired data has been deleted to the disk used to store the adjacent disk file; The preset disk file is any disk file among the disk files included in each data layer, and the adjacent disk file is a disk file adjacent to the preset disk file.

13. The data processing method according to claim 10, wherein in response to the number of counting devices corresponding to the disk files included in at least one data layer satisfying a preset number condition, using the meta information identified by the keyword in the cache structure corresponding to each of the at least one data layer, locating the expired data in the disk files included in the at least one data layer, and recycling the located expired data, comprising: In response to the merge instruction, according to the number of counting devices corresponding to the disk files included in each data layer, using the meta information identified by the keyword in the cache structure corresponding to the data layer where the disk files in the first preset range are located, determine the number of expired data in the disk files in the first preset range, and judge whether the number of expired data in the disk files in the first preset range is greater than a first preset number threshold; the first preset range is the data range in any one of the at least two data layers; When it is determined that the amount of expired data in the disk files within the first preset range is greater than the first preset amount threshold, and the total amount of data in the disk files within the first preset range is less than the second preset amount threshold, the data within the first preset range and the data within the second preset range are sorted to recycle the expired data in the disk files within the first preset range; The data layer where the second preset range is located is a data layer below the data layer where the first preset range is located, and the data within the first preset range and the data within the second preset range have an intersection.

14. A data processing device, comprising: An expired data acquisition module, configured to determine, in response to a write operation on target data in a log structure merge tree, expired data generated by the write operation; The log structure merge tree includes at least two data layers, each data layer includes a preset number of disk files, and each data layer corresponds to a cache structure, each cache structure is used to store metadata of expired data in the corresponding data layer; a target disk file search module, used to determine the keyword of the target data as the target keyword of the expired data, and based on the target keyword, search for the target disk file where the expired data is located from a preset number of disk files included in each data layer; A keyword writing module, used for writing the target keyword into a cache structure corresponding to the data layer where the target disk file is located; The target keyword is used to identify the meta information of expired data in the target disk file; A recycling module is used to respond to the number of keywords in the cache structure corresponding to at least one data layer satisfying a preset number condition, using the metadata identified by the keywords in the cache structure corresponding to the at least one data layer, locate the expired data in the disk file included in the at least one data layer, and recycle the located expired data.

15. An electronic device for data processing, the electronic device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded by the processor and executing the data processing method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 10.

17. A computer program product, comprising at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 13.

Images