Data processing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data storage, in particular to a data processing method and device, equipment and a storage medium, and is used for improving the deletion efficiency of different keys in an LSM number. The method comprises the following steps: for any hierarchy of a log structure merging LSM tree, obtaining each to-be-processed file, namely a read-only memory table or a source SST associated with the hierarchy, sorting and traversing a first type of keys in the files, and determining whether the keys are invalid or not according to a preset judgment condition; wherein the judgment condition comprises at least one of the following conditions: the key itself or the value corresponding to the key itself meets a first value taking condition, and the key itself or the value corresponding to the key itself and the preorder key itself or the value corresponding to the key itself meets a second value taking condition; the first type of keys are different keys in the to-be-processed files; and for each to-be-processed file, generating a target SST through the key value pair data of the effective key.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In storage systems, Log-Structured Merge (LSM) trees and their variants are widely used to store metadata information. The characteristic of LSM is that it converts random key-value (KV) write operations into sequential write-ahead log (WAL) writes, thereby significantly improving write performance. LSM forms a memtable / immutable memory table through a skip table or other extended structures, and uses an ordered sorted string table (SST) organization on the disk to provide high-speed KV search performance.

[0003] When using LSM to store KV, if the same key needs to be written multiple times, it will be written in append mode, so there may be multiple records with the same key in LSM at the same time, of which only one record is valid (visible by query). When immTable is sunk to SST, or SSTs are merged, the last written same key and its corresponding value will be selected and written to the target SST, and the source SST will be deleted, thus ensuring that there is only one record for the same key in LSM.

[0004] In the related technology, only identical keys can be naturally deleted when SST is merged, while natural deletion of different keys cannot be realized. However, in actual use, the need to delete non-identical keys will appear during data cleaning, business logic changes, and performance optimization. Although the tombstone method can specify a deletion range, the keys in the range will be deleted when the tombstone is merged to the bottom layer of the LSM. However, if the keys to be deleted are not continuous intervals, the tombstone merge operation may cause significant performance degradation, especially when the deletion range is not continuous, which increases the calculation and I / O overhead of the merge.

[0005] In summary, how to effectively solve the deletion of different keys to achieve faster processing speed, higher efficiency and more accurate results is an urgent problem to be solved. Summary of the invention

[0006] Embodiments of the present application provide a data processing method, device, electronic device, and storage medium to improve the efficiency of effective deletion of different keys.

[0007] A data processing method provided in an embodiment of the present application includes:

[0008] For any level of the LSM tree, obtain each to-be-processed file associated with the level; wherein each of the to-be-processed files is a read-only memory table or a source SST;

[0009] The first-class keys in each of the to-be-processed files are sorted and traversed according to the size of the keys. During the traversal process, whether the first-class keys are invalid is determined according to a preset judgment condition; wherein the judgment condition includes at least one of the following: the target parameter associated with the first-class key satisfies a first value condition, and the target parameter associated with the first-class key and its predecessor first-class key respectively satisfies a second value condition; the first-class key is a different key in each of the to-be-processed files; the target parameter associated with each first-class key includes at least one of the following: the first-class key itself, and the value corresponding to the first-class key;

[0010] For each of the files to be processed, a target SST is generated according to the key-value pair data of the determined valid key.

[0011] Compared with the tombstone method, this data processing method can more effectively handle the key deletion problem in non-continuous intervals, reduce the storage space occupied by invalid data, and further improve the overall performance of the storage system. At the same time, invalid items in the LSM tree can be cleaned up without waiting for merging to the bottom layer. If a key is judged to be valid, its associated key-value pair data is retained and merged into the target SST.

[0012] Optionally, the target parameter includes the first type of key itself, and the first value condition includes at least one of the following: the size of the key is within the first preset range of the hierarchical association, and the key converted by the first preset rule is the same as the first specified parameter of the hierarchical association;

[0013] The second value condition includes: the current key and its predecessor key satisfy the first preset relationship of the hierarchical association;

[0014] The target parameter includes the value corresponding to the first type of key, and the first value condition includes at least one of the following: the size of the value is within the second preset range of the hierarchical association, and the value converted by the second preset rule is the same as the second specified parameter of the hierarchical association;

[0015] The second value-taking condition includes: the value corresponding to the current key and the value corresponding to its preceding key satisfy the second preset relationship of the hierarchical association.

[0016] By performing double judgment on the first type of key and its value, the flexibility and accuracy of data judgment are improved. The first value condition supports a wider range of application scenarios by limiting the range or conversion rules of key and value. The second value condition ensures the relevance and continuity of data by checking the relationship between the current key and the previous key, or the current value and the previous value. It can flexibly handle the data requirements of different application scenarios through multiple judgment conditions.

[0017] Optionally, each of the to-be-processed files is: at least one read-only memory table to be sunk generated in real time in the level, or multiple source SSTs to be merged generated in real time in the level and its adjacent levels, or multiple source SSTs to be merged generated in real time in the level;

[0018] Then, for each of the to-be-processed files, a target SST is generated according to the key-value pair data of the determined valid key, including:

[0019] Whenever a first-category key is determined to be an invalid key, the key-value pair data corresponding to the invalid key is deleted;

[0020] Whenever a first-category key is determined to be a valid key, the key-value pair data corresponding to the valid key is added to the target SST.

[0021] In the above process, through real-time processing, the data can be quickly screened and optimized when the read-only memory table is sunk to generate SST or the source SST is merged, thereby improving the efficiency of data storage. Each time the key is determined to be valid, the corresponding operation is performed immediately, which not only simplifies the data processing process, but also ensures that the data in the target SST is valid. This real-time processing strategy can significantly reduce data redundancy and optimize storage resources in large-scale data systems.

[0022] Optionally, each of the to-be-processed files is at least one read-only memory table obtained by performing a background scan on the level, or multiple source SSTs obtained by performing a background scan on the level and its adjacent levels, or multiple source SSTs obtained by performing a background scan on the level;

[0023] Then, for each of the to-be-processed files, a target SST is generated according to the key-value pair data of the determined valid key, including:

[0024] Count the proportion of invalid keys in each of the files to be processed;

[0025] According to the proportion of invalid keys corresponding to each of the to-be-processed files and the order between each of the to-be-processed files, the to-be-processed files are divided into at least one file group;

[0026] For each file group obtained by the division, the key-value pair data corresponding to the invalid key in the file group is deleted, and the key-value pair data corresponding to the valid key in the file group is added to the target SST corresponding to the file group.

[0027] During the above background scanning process, the key-value pairs in the existing SST are judged to confirm the proportion of valid keys. If the proportion of valid keys in a certain SST is very low, a merge operation is triggered to reduce space occupancy; if all the items in the SST are invalid, the SST can be deleted directly. Background scanning can naturally recycle invalid items in each layer of the LSM tree without explicitly deleting these items. This processing method can also clean up invalid data in the bottom-level SST in advance without sinking upper-level data.

[0028] Optionally, the dividing the files to be processed into at least one file group according to the proportion of invalid keys corresponding to each of the files to be processed and the order between each of the files to be processed includes:

[0029] Determine each target file in which the invalid key ratio is higher than a preset threshold among the files to be processed;

[0030] Divide each n adjacent target files into a file group; where n is a positive integer greater than or equal to 2, and the order between the target files is determined according to the size of the key.

[0031] In the above process, we first filter out target files with a high percentage of invalid keys, reducing the number of files that need to be processed and ensuring that resources are focused on invalid files that need to be deleted. We group target files by order and quantity, so that each operation processes multiple files instead of processing them individually, which helps improve the speed and efficiency of data processing.

[0032] Optionally, a scanning frequency of background scanning of the LSM tree is positively correlated with a speed of writing data in the LSM tree.

[0033] In the above process, dynamically adjusting the scanning frequency can detect and process invalid data more timely, reduce data redundancy in storage, keep data up to date, improve query performance and storage efficiency, reduce the pressure of real-time processing, and avoid data problems accumulated over a long period of time.

[0034] Optionally, before determining whether the first type of key is invalid according to a preset determination condition, the method further includes:

[0035] For the second type of keys in each of the files to be processed, only the key-value pair data corresponding to the second type of keys written last time is retained, wherein the second type of keys are the same keys in each of the files to be processed.

[0036] In the embodiment of the present application, since the same key may be written multiple times at different times, only the key-value pair data corresponding to the latest version of the key needs to be retained. Therefore, before sorting and traversing the first type of keys, the second type of keys will be merged and data will be cleaned up, all historical versions of the same key will be deleted, and only the last written version will be retained, thereby simplifying the subsequent operation process.

[0037] An embodiment of the present application provides a data processing device, including:

[0038] An acquisition unit, configured to acquire, for any level of the LSM tree, each to-be-processed file in the level; wherein each of the to-be-processed files is a read-only memory table or a source SST;

[0039] A determination unit is used to sort and traverse the first-category keys in each of the to-be-processed files according to the size of the keys, and during the traversal process, determine whether the first-category keys are invalid according to a preset determination condition; wherein the determination condition includes at least one of the following: the target parameter associated with the first-category key satisfies a first value condition, and the target parameter associated with the first-category key and its predecessor first-category key respectively satisfies a second value condition; the first-category key is a different key in each of the to-be-processed files; the target parameter associated with each first-category key includes at least one of the following: the first-category key itself, and the value corresponding to the first-category key;

[0040] The generating unit is used to generate a target SST for each of the to-be-processed files according to the key-value pair data of the determined valid key.

[0041] Optionally, the target parameter includes the first type of key itself, and the first value condition includes at least one of the following: the size of the key is within the first preset range of the hierarchical association, and the key converted by the first preset rule is the same as the first specified parameter of the hierarchical association;

[0042] The second value condition includes: the current key and its predecessor key satisfy the first preset relationship of the hierarchical association;

[0043] The target parameter includes the value corresponding to the first type of key, and the first value condition includes at least one of the following: the size of the value is within the second preset range of the hierarchical association, and the value converted by the second preset rule is the same as the second specified parameter of the hierarchical association;

[0044] The second value-taking condition includes: the value corresponding to the current key and the value corresponding to its preceding key satisfy the second preset relationship of the hierarchical association.

[0045] Optionally, each of the to-be-processed files is: at least one read-only memory table to be sunk generated in real time in the level, or multiple source SSTs to be merged generated in real time in the level and its adjacent levels, or multiple source SSTs to be merged generated in real time in the level;

[0046] The generating unit is specifically used for:

[0047] Whenever a first-category key is determined to be an invalid key, the key-value pair data corresponding to the invalid key is deleted;

[0048] Whenever a first-category key is determined to be a valid key, the key-value pair data corresponding to the valid key is added to the target SST.

[0049] Optionally, each of the to-be-processed files is at least one read-only memory table obtained by performing a background scan on the level, or multiple source SSTs obtained by performing a background scan on the level and its adjacent levels, or multiple source SSTs obtained by performing a background scan on the level, then the generating unit is specifically used for:

[0050] Count the proportion of invalid keys in each of the files to be processed;

[0051] According to the proportion of invalid keys corresponding to each of the to-be-processed files and the order between each of the to-be-processed files, the to-be-processed files are divided into at least one file group;

[0052] For each file group obtained by the division, the key-value pair data corresponding to the invalid key in the file group is deleted, and the key-value pair data corresponding to the valid key in the file group is added to the target SST corresponding to the file group.

[0053] Optionally, the generating unit is specifically used for:

[0054] Determine each target file in which the invalid key ratio is higher than a preset threshold among the files to be processed;

[0055] Divide each n adjacent target files into a file group; where n is a positive integer greater than or equal to 2, and the order between the target files is determined according to the size of the key.

[0056] Optionally, a scanning frequency of background scanning of the LSM tree is positively correlated with a speed of writing data in the LSM tree.

[0057] Optionally, the device further comprises:

[0058] A deduplication unit is used to retain only the key-value pair data corresponding to the second-category key written last time in each of the files to be processed before the determination unit determines whether the first-category key is invalid according to a preset determination condition, wherein the second-category key is the same key in each of the files to be processed.

[0059] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned data processing methods.

[0060] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned data processing methods.

[0061] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned data processing methods.

[0062] The beneficial effects of this application are as follows:

[0063] The embodiments of the present application provide a data processing method, device, electronic device and storage medium. Specifically, for any level of the LSM tree, the present application can process the read-only memory table associated with the level or the key in the source SST, and the to-be-processed files associated with different levels can reflect the data in different storage stages. When processing different keys (i.e., first-class keys) in the to-be-processed files, first sort them by size. By sorting the first-class keys in each to-be-processed file by size, the orderliness of the keys during the traversal process can be ensured, thereby avoiding repeated comparisons and invalid judgment operations. Then traverse in order, and during the traversal process, analyze whether the key is invalid according to the preset judgment conditions. In this process, the preset judgment conditions are based on a multi-dimensional judgment method, which can flexibly respond to the invalid key judgment requirements in different application scenarios, and improve the efficiency and intelligence of data processing. Finally, for each to-be-processed file, the target SST is generated according to the key-value pair data of the determined valid key. In this process, if a key is judged to be invalid, its associated key-value pair data can be deleted to achieve natural cleaning of different keys. Compared with the tombstone method, it can more effectively handle the key deletion problem of non-continuous intervals, reduce the storage space occupied by invalid data, and further improve the overall performance of the storage system. At the same time, invalid items in the LSM tree can be cleaned up without waiting to be merged to the bottom layer. If a key is judged to be valid, its associated key-value pair data will be retained and merged into the target SST.

[0064] In general, this application improves the efficiency of deleting different keys through a flexible invalid key determination mechanism and efficient sorting processing, while also improving storage space utilization and query performance, thereby achieving more efficient storage management and data processing capabilities.

[0065] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0067] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0068] Figure 2 An LSM tree structure diagram provided in an embodiment of the present application;

[0069] Figure 3 A flowchart of the implementation of the background processing phase after the LSM tree is constructed provided in an embodiment of the present application;

[0070] Figure 4 A specific flow chart of a method for executing data processing provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the structure of a data processing device in an embodiment of the present application;

[0072] Figure 6 The present invention is a schematic diagram of the hardware structure of an electronic device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0074] The following is an introduction to some concepts involved in the embodiments of the present application.

[0075] 1. LSM tree: It is a storage structure used to handle high write loads. It reduces disk write operations by storing data in layers, thereby improving write efficiency. The main layers of the LSM tree include the memory layer and the disk layer. Among them, the memory layer includes memory tables and read-only memory tables, which are used to process newly written key-value pairs; the disk layer can be divided into several SST layers. SST is a persistent storage structure in the disk, organized in layers, each layer contains several SSTs, and each SST file stores ordered key-value pairs.

[0076] 2. Memory table and read-only memory table: The memory table is a memory data structure that stores newly written key-value pairs in the LSM tree. It is a mutable data structure used to temporarily save write operations. The read-only memory table is a data structure converted when the memory table reaches capacity limit. Specifically, when the memory table is full, its content will be converted to a read-only state and become a read-only memory table. After the conversion, the read-only memory table is no longer allowed to be modified, ensuring the consistency and stability of the data.

[0077] 3. SST: It is a disk data structure used for persistent storage of key-value data. It is organized in layers, and each layer contains merged and optimized data (specifically key-value data). Specifically, data is first written to the SST of Level 0 from a read-only memory table, and then gradually merged. The SSTs of Level 1 and above contain merged and optimized data. Each level of SST is larger than the previous one, and stores more key-value data.

[0078] 4. Sinking: refers to the process of writing the data in the read-only memory table to the disk to form an SST. This process persists the data in the memory to the disk for subsequent persistent storage and reading operations. Sinking usually occurs after the memory table reaches a certain capacity and is converted into a read-only memory table. Then the background asynchronously sinks the data in the read-only memory table to the disk.

[0079] 5. Merge: It refers to selecting SSTs with overlapping key ranges from adjacent levels, merging and sorting all key-value pairs in the SST, and rewriting them into new SSTs in the lower level, so that all SSTs in each level and the key-value pairs in each SST are arranged in order.

[0080] 6. Source SST and target SST: Source SST refers to the original SST before merging. Target SST refers to the new SST generated after data processing (such as deleting invalid data, merging identical data, etc.) on the source SST. Or it is the new SST generated after sinking the read-only memory table.

[0081] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0082] The data processing method in the embodiment of the present application can be executed by an electronic device, which can be a terminal device or a server, that is, the method can be executed by the terminal device or the server alone, or by the terminal device and the server together. Among them, the terminal device includes but is not limited to mobile phones, tablet computers, laptops, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a client related to data processing can be installed on the terminal device, which can be software (such as a browser, data processing software, etc.), or a web page, a small program, etc. The server is a background server corresponding to the software or web page, small program, etc., or a server specifically used for data processing, which is not specifically limited in this application.

[0083] The embodiments of the present application can be applied to various scenarios, including not only database storage and file system management scenarios, but also but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other fields. These scenarios require efficient data storage, management and processing capabilities. LSM trees, through their unique hierarchical storage mechanism and optimized writing method, can well meet the needs of these applications for large-scale data processing and real-time performance.

[0084] The following describes the data processing method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0085] like Figure 1 As shown, an implementation flow chart of a data processing method provided in an embodiment of the present application is applied to an electronic device. The specific implementation flow of the method is as follows S11 to S13:

[0086] S11: For any level of the LSM tree, obtain each to-be-processed file in the level; wherein each to-be-processed file is a read-only memory table or a source SST.

[0087] Among them, the read-only memory table refers to a sorted data structure stored in memory, which is used to save data dumped from the memory table. It cannot be modified. Once created, it can only be read, and the data will be written to disk during the sinking operation.

[0088] In the embodiment of the present application, the LSM tree uses a multi-layer storage structure to store data. When a write request comes, it is first cached in the memory table. When the capacity of the memory table exceeds the threshold, it will be converted to a read-only memory table, and then the data in the read-only memory table will be sunk to the 0th level of the disk, and then the data will be moved down layer by layer through the merge operation.

[0089] like Figure 2 As shown, an LSM tree structure diagram provided by an embodiment of the present application. Figure 2 In the LSM tree, the LSM tree mainly includes the memory layer and the disk layer. The memory layer can be recorded as the -1th level in the LSM tree. Assuming that there are n+1 SST layers in the disk layer, they are recorded as the 0th to n+1th levels in the LSM tree.

[0090] For example, at the -1 level, the files to be processed may be one or more read-only memory tables in the memory layer. The sinking operation sinks the data of the read-only memory table to the SST. Usually, one read-only memory table is sunk to the SST on the disk at a time. Of course, the data of multiple read-only memory tables can also be sunk to the SST simultaneously, which is not specifically limited in this article.

[0091] At level 0 to level n, the files to be processed can be the SSTs contained in these levels and their adjacent next levels. For example, at level 1, the files to be processed may be SST11, SST12, SST21, SST22, etc. The merged SSTs will form a new SST at level 2. The SSTs of these adjacent layers will be merged to optimize storage and query performance and integrate data to a higher level.

[0092] At the last level, level n+1, there is no adjacent next level, so the files to be processed can be SSTs in this level, such as SST(n+1)1, SST(n+1)2, SST(n+1)3, etc. At this level, the merge operation occurs within the layer, that is, SSTs at the same level are merged to maintain the orderliness of the data and optimize space utilization.

[0093] After obtaining the files to be processed, the following will analyze in detail the processing process of the files to be processed at any level:

[0094] S12: Sort and traverse the first-category keys in each file to be processed according to the size of the key. During the traversal process, determine whether the first-category keys are invalid according to preset judgment conditions; wherein the judgment conditions include at least one of the following: the target parameter associated with the first-category key satisfies the first value condition, and the target parameter associated with the first-category key and its predecessor first-category key respectively satisfies the second value condition; the first-category key is a different key in each file to be processed; the target parameter associated with each first-category key includes at least one of the following: the first-category key itself, and the value corresponding to the first-category key.

[0095] In actual applications, due to frequent write, update and delete operations, the LSM tree may contain the same or different keys. Different keys are because each key represents a different data item, and the same key is because the same key is repeatedly written or updated in multiple operations. Among them, the first type of key refers to the different keys in each to-be-processed file, and the second type of key in this application refers to the same key in each to-be-processed file.

[0096] For the same key (i.e., the second type of key), during the read-only memory table sinking or SST merging process, the processing method is to retain the latest key-value pair and delete the old invalid version, reducing redundant data and saving storage space. For different keys (i.e., the first type of key), during the read-only memory table sinking or SST merging process, these key-value pairs are arranged in order. When processing different keys, the key issue is how to determine whether a key and its corresponding key-value pair should continue to be retained or deleted.

[0097] Therefore, the present application proposes a determination condition for the first type of key, based on which it can be determined whether a key is invalid, and the invalid key should be deleted. The determination condition is specifically based on the determination of the target parameter associated with the first type of key.

[0098] For a key data, the target parameter may specifically refer to the attribute of the key data itself, or may refer to the value data associated with the key data, or may include both the attribute of the key data itself and the value data associated with the key data.

[0099] Among them, the attribute of the key data itself refers to the value of the key itself; the value data associated with the key data refers to the value corresponding to the key.

[0100] For example, for a certain key, the target parameter may be the value of the key itself, or the value corresponding to the key, or the target parameter may contain both the value of the key itself and the value corresponding to the key.

[0101] In the embodiment of the present application, each to-be-processed file associated with a certain level contains at least one first-class key. To process these keys, they must first be sorted according to their size (which can be the key value or alphabetical order), and after the sorting is completed, these keys are traversed in sequence.

[0102] In an optional implementation, the size of the key may be a numerical value of the key or may be an alphabetical order, which is not limited herein.

[0103] For example, if the key is numeric, you can sort it according to the value of the key, such as sorting from small to large (for example, the values ​​of the three keys are 5, 15, and 10, respectively, and they will be sorted as 5, 10, and 15 according to the value size).

[0104] For example, if the key is a string type, it can be sorted in alphabetical order (for example, the three key strings are "apple", "cherry", and "banana", and they will be sorted in alphabetical order as "apple", "banana", and "cherry").

[0105] After sorting the keys by size, the keys are traversed one by one, and the keys can be analyzed one by one based on the preset judgment conditions to see whether they are invalid.

[0106] In an optional implementation, the traversal can be performed one by one according to a single key, or can be performed in groups according to multiple keys. The present application does not make any specific limitation on the number of keys traversed each time during the traversal process.

[0107] In an optional implementation, in order to reduce storage space and improve data processing efficiency, before determining whether the first type of key is invalid according to a preset judgment condition, it also includes:

[0108] For the second-category keys in each file to be processed, only the key-value pair data corresponding to the second-category key written last time is retained, where the second-category keys are the same keys in each file to be processed.

[0109] For example, if there are three identical second-category keys, such as key2, in one or some of the files to be processed, corresponding to three different time points: key2_1 (corresponding to time T1), key2_2 (corresponding to time T2), and key2_3 (corresponding to time T3). Before performing the traversal operation, the system will check these three keys and only retain key2_3 and its corresponding value at time T3 based on the timestamp, and the other two versions of the key-value pair data will be deleted.

[0110] It should be noted that the above is just a simple example. In actual applications, there may be more identical keys, and version merging and data cleaning can all be performed in the above manner, which will not be described one by one here.

[0111] After this operation is completed, the first-class keys will be judged for these files that have been merged and cleaned up by the second-class key versions to determine which keys are valid and which are invalid.

[0112] In practical applications, the judgment conditions proposed in this application are dynamic, and each level has its own associated judgment conditions. These judgment conditions may vary from level to level, and the judgment conditions between different levels may be different. This is because the data characteristics and processing requirements of different levels may be different, so specific judgment conditions need to be formulated to ensure the validity and consistency of the data. Of course, the judgment conditions between some levels may also be the same, specifically they may be completely the same or partially the same.

[0113] In an optional implementation, for any level, the determination condition includes but is not limited to the following one or a combination thereof:

[0114] Judgment condition one: the size of the first type of key itself is within the first preset range associated with the level.

[0115] In the embodiment of the present application, the judgment condition 1 is applicable to continuous data, such as value range judgment, and its validity is determined by judging whether the value of the key is within a predetermined range.

[0116] For example, assuming that the first preset range of the hierarchical association is [150, 250], then the first-category key is valid only if the value is between 150 and 250. For example, a first-category key of 160 will be considered valid, while a first-category key of 100 will be considered invalid.

[0117] Judgment condition two: the first type of key itself satisfies the first preset rule that the key converted is the same as the first specified parameter associated with the level.

[0118] For example, the conversion rule is to calculate the sum of each bit in the value of key, and the first specified parameter associated with this level is 7. If the value of key is 123, then 1+2+3=6, which does not meet the condition. If the value of key is 25, then 2+5=7, this key is considered valid because the sum of its bits is the same as the specified parameter.

[0119] Judgment condition three: the first type of key itself and its predecessor first type of key itself satisfy the first preset relationship of the hierarchical association.

[0120] In an embodiment of the present application, judgment condition three is applicable to continuous data (such as increasing values) and discrete data (such as serialized identifiers), and is used to check the relationship between the current key and its previous key or previous keys, such as an increasing relationship, a decreasing relationship, etc.

[0121] For example, the preset relationship is that the difference between the last few digits of the current key and the previous key is the same. If the current key is ABCD1234, the last 4 digits are 1234; the previous key is ABCD5678, the last 4 digits are 5678, the calculated difference of the last 4 digits is 5678-1234=4444. The specified difference is 4444, so the condition is met. If the current key is ABCD6789, the last 4 digits are 6789, the calculated difference is 6789-1234=5555, which does not match the specified difference of 4444, so the condition is not met.

[0122] Judgment condition four: the size of the value corresponding to the first type of key is within the second preset range associated with the level.

[0123] In the embodiment of the present application, determination condition four is applicable to continuous data (such as a numerical range) and is used to determine whether the value corresponding to the key is within the specified range to ensure its validity.

[0124] For example, if the value of the key is 600, 800, 1200, and the second preset range associated with the hierarchy is [500, 1000], 600 and 800 are considered valid, while 1200 is considered invalid.

[0125] Judgment condition five: the size of the value corresponding to the first type of key after conversion by the second preset rule is the same as the second specified parameter associated with the level.

[0126] For example, if the transformation rule is to square value, the transformed value needs to match the specified parameter. If value is squared, and the second specified parameter associated with the level is 144, value 12 (12*12=144) meets the condition, but value 10 does not. It can be applied to continuous data and discrete data, depending on the form of the transformation rule.

[0127] Judgment condition six: the value corresponding to the first type of key and the value corresponding to its predecessor key satisfy the second preset relationship of hierarchical association.

[0128] In an optional embodiment, the preset relationship is "the value difference does not exceed 10", and the value of the current key is 50, and the value of the previous key is 40, then the condition is met. If the value of the current key is 55, and the value of the previous key is 40, then the condition is not met. Applicable to continuous data (such as numerical differences) and discrete data (such as differences in serialized data).

[0129] It should be noted that the above determination conditions are only examples, and in fact, the specific content and form of the determination conditions are not limited. In the embodiments of the present application, the determination conditions can be adjusted and set according to the needs and specific circumstances of the actual application, and the specific determination conditions are not limited.

[0130] For each file to be processed, when determining the validity of each first-class key in real time, the corresponding operation is performed, which will be described in detail below.

[0131] S13: For each file to be processed, a target SST is generated according to the key-value pair data of the determined valid key.

[0132] In an embodiment of the present application, the key-value pair data corresponding to the valid key is determined and added to the target SST. The key-value pair data of the invalid key can be deleted or not processed. The present application does not make specific restrictions on the specific processing method of the key-value pair data of the invalid key. The following takes the deletion of the key-value pair data corresponding to the determined invalid key as an example.

[0133] In the embodiment of the present application, for each to-be-processed file, the key-value pair data corresponding to the determined invalid key is deleted, and the process of generating the target SST according to the key-value pair data corresponding to the determined valid key can be applied to the real-time construction stage of the LSM tree, and can also be applied to the background processing stage after the LSM tree is built. The following will be introduced respectively:

[0134] (I) Real-time construction phase:

[0135] During the real-time construction of the LSM tree, data will continue to be added to the system as write operations are performed. During the real-time processing, the data structure in the LSM tree needs to be continuously maintained and updated. The data processing method in the embodiment of the present application can realize the processing of different keys through the following two sub-cases:

[0136] Subcase 1: In this case, the first type of key (different keys) and the second type of key (same key) are processed synchronously. When new data is written to the LSM tree, all keys are checked and judged immediately. For the same key, the versions are merged, the latest key value is retained, and the old version of the data is deleted. For different keys, it is determined whether it is an invalid key based on the preset judgment conditions. If invalid, it is deleted immediately. If it is a valid key, its corresponding key-value pair data is synchronously written to the target SST. This ensures that all keys can be processed synchronously when writing, avoiding data redundancy and accumulation of invalid data.

[0137] In the embodiment of the present application, for the first type of key in each file to be processed, it is determined whether the first type of key is invalid according to a preset judgment condition.

[0138] If the first-category key is invalid, then each time a first-category key is determined to be an invalid key, at least one key-value pair data corresponding to the invalid key is deleted.

[0139] In the embodiment of the present application, whether the first-class key is invalid is identified by a preset judgment condition. The specific preset judgment condition is described above, and the present application will not repeat it here. If the judgment result shows that a first-class key is invalid, then the key and its associated key-value pair data will be deleted.

[0140] If the first-category key is valid, then each time a first-category key is determined to be a valid key, at least one key-value pair data corresponding to the valid key is added to the target SST.

[0141] Generally, multiple source SSTs correspond to one target SST. For read-only memory tables and target SSTs, the relationship between them can be one-to-one or many-to-one, that is, one read-only memory table is sunk to one target SST, or multiple read-only memory tables are sunk to the same target SST.

[0142] In an optional implementation, the traversal can be performed one by one according to a single key, or in groups according to multiple keys. When traversing multiple keys, the processing method can be one by one, that is, the result of each key is processed immediately after it is obtained; or after the results of multiple keys are processed, a unified operation is performed, such as deleting invalid keys and adding valid keys. This application does not specifically limit the number of keys processed each time during the traversal process and the specific operation steps.

[0143] Sub-case 2: In this case, the second type of key (the same key) will be merged immediately when it is written in real time. That is, when the same key is written repeatedly, the system will immediately retain the key value of the latest version and delete the old version. In contrast, the processing of the first type of key (different keys) may not be real-time, but triggered by regular or irregular background scanning.

[0144] In an embodiment of the present application, for the processing process of sub-case two, each file to be processed can be: at least one read-only memory table obtained by performing a background scan on a certain level, or, multiple source SSTs obtained by performing a background scan on a certain level and its adjacent levels, or, multiple source SSTs obtained by performing a background scan on a certain level.

[0145] In an embodiment of the present application, for different keys, the system will perform batch scans on these keys at a specific time point or according to certain trigger conditions to determine the validity of the keys. Valid keys will be retained and written to the target SST, while invalid keys will be deleted in batches, which is basically the same as the specific implementation of the above-mentioned sub-case one, except that the processing of different keys and the processing of the same key are not performed together, but are processed through regular or irregular scanning. The specific process of processing the second type of keys will not be repeated here.

[0146] The above is an introduction to the processing flow of the real-time construction phase of the LSM tree. The following is an introduction to the processing flow of the background processing phase after the LSM tree is built.

[0147] (ii) Background processing phase after construction is completed.

[0148] Background processing refers to a batch scanning operation performed when the storage load is low. It is used to scan files at one or more levels, delete invalid keys, and organize and merge valid data to further optimize the storage structure and performance.

[0149] In an embodiment of the present application, for the background scanning processing process, each file to be processed can be at least one read-only memory table obtained by performing a background scan on a certain level, or, multiple source SSTs obtained by performing a background scan on a certain level and its adjacent levels, or, multiple source SSTs obtained by performing a background scan on a certain level.

[0150] like Figure 3 As shown, the implementation flow chart of the background processing phase after the LSM tree is built provided by the embodiment of the present application includes the following steps S31-S35:

[0151] S31: For any level of the LSM tree, obtain each to-be-processed file associated with the level.

[0152] In the embodiment of the present application, when multiple source SSTs are obtained by background scanning of a certain level and its adjacent levels, there are two situations for the files to be processed:

[0153] The first case is multiple source SSTs for the level and its adjacent next level, which means that not only the current level is scanned, but also the files of the adjacent levels are involved to ensure data optimization across levels.

[0154] For example, Figure 2 For example, for the first level, the files to be processed may be SST11, SST12, SST13, ..., SST1n, SST21, SST22, SST23, ..., SST2n, which not only scans the current first level, but also involves files in the adjacent next level, the second level.

[0155] The second case is multiple source SSTs within the level, which means that the target of the background scanning process can also be files within the current level.

[0156] For example, still Figure 2 For example, for the n+1th level, the files to be processed can be SST(n+1)1, SST(n+1)2, SST(n+1)3, ..., SST(n+1)n. The files to be processed by the background scanning are the files in the n+1th level of the current level, and the generated target SST is also the newly generated SST of the n+1th level.

[0157] The files to be processed are obtained through the above process. The background processing process will be described in detail below.

[0158] S32: sort and traverse the first category of keys in each to-be-processed file according to the size of the key, and count the proportion of invalid keys in each to-be-processed file according to the preset judgment conditions.

[0159] It should be noted here that the second type of keys have been deduplicated during the real-time construction phase of the LSM tree, so there is no need to further process the same keys during background processing.

[0160] For the determination of invalid keys, refer to the determination process of the above real-time processing, and then count the proportion of invalid keys in each file to be processed.

[0161] S33: Divide each to-be-processed file into at least one file group according to the invalid key ratio corresponding to each to-be-processed file and the order between each to-be-processed file.

[0162] Among them, the file group refers to a collection of files to be processed that are grouped according to the invalid key ratio and the sorting rules between the files to be processed. Each file group contains one or more files to be processed, which are treated as a whole during the processing process to facilitate centralized management and optimization of data.

[0163] Specifically, the steps of dividing each to-be-processed file into at least one file group are as follows S3301-S3302:

[0164] S3301: Determine, among the files to be processed, target files whose invalid key ratio is higher than a preset threshold;

[0165] In the embodiment of the present application, firstly, it is necessary to identify the to-be-processed files whose invalid key ratio is higher than a preset threshold. The purpose of this step is to filter out the to-be-processed files that need to be focused on, that is, the to-be-processed files that contain a high proportion of invalid data.

[0166] In an optional embodiment, the preset threshold can be adjusted according to the specific application scenario. For example, the preset threshold can be set to 60%, 70% or other suitable ratios. When the proportion of invalid keys in a file exceeds the threshold, the file will be identified as a target file.

[0167] S3302: Divide each n adjacent target files into a file group; wherein n is a positive integer greater than or equal to 2, and the order between the target files is determined according to the size of the key.

[0168] In the embodiment of the present application, after the target files are confirmed, the target files are further grouped. The specific method is to divide n adjacent target files (n is a positive integer greater than or equal to 2) into a file group. The adjacent relationship of the target files is determined according to the order of the size of the key.

[0169] For example, suppose there are three target files, each containing the following key ranges: file A: key1 to key100; file B: key201 to key300; file C: key401 to key500. If n = 2, these files are divided into the following groups according to the order of key size: file A and file B form a file group; file C forms a file group alone. In this way, the division of the target file groups is based on the order of keys in each file, which helps to effectively process the keys later.

[0170] After each file group is divided, the file group can be processed, which will be described in detail below.

[0171] S34: For each file group obtained by the division, the key-value pair data corresponding to the invalid key in the file group is deleted, and the key-value pair data corresponding to the valid key in the file group is added to the target SST corresponding to the file group.

[0172] In an embodiment of the present application, in each file group, all key-value pairs corresponding to invalid keys are first deleted. These invalid data have been identified as no longer needed, so they are removed from the file group to reduce storage space and optimize data quality. Adding the key-value pairs corresponding to valid keys in the file group to the target SST corresponding to the file group can ensure that these data are efficiently stored in the level where they are located, thereby improving data retrieval efficiency and the overall performance of database storage.

[0173] In an optional embodiment, the scanning frequency of the background scanning of the LSM tree is positively correlated with the speed of writing data in the LSM tree.

[0174] For example, in a system with high write load, the number of key-value pairs written per second reaches tens of thousands. By increasing the scanning frequency, invalid data can be quickly cleared to maintain the compactness and efficiency of data storage. Conversely, when the writing speed slows down, the frequency of background scanning can be reduced to avoid unnecessary waste of resources.

[0175] In a specific implementation, the process of a data processing method provided in an embodiment of the present application can be as follows: Figure 4 As shown, the following steps are included:

[0176] S41: For any level of the LSM tree, obtain each to-be-processed file associated with the level.

[0177] In an embodiment of the present application, each file to be processed may be: at least one read-only memory table to be sunk generated in real time in the level, or multiple source SSTs to be merged generated in real time in the level and its adjacent levels, or multiple source SSTs to be merged generated in real time in the level.

[0178] In an embodiment of the present application, for the -1th level, the file to be processed is a read-only memory table to be sunk. At levels 0 to n, the file to be processed can be the SST contained in these levels and the next adjacent levels. At the last level (n+1), the file to be processed is the SST in this level.

[0179] Optionally, if the level is level 0, the number of each to-be-processed file associated with the level is generally a read-only memory table, and in order to improve storage efficiency, it can also be multiple read-only memory tables.

[0180] S42: sort the keys of the files to be processed (including the first-category keys and the second-category keys) according to the sizes of the keys and traverse them in sequence.

[0181] In the embodiment of the present application, the process of sorting and traversing is the same as the above part and will not be repeated here.

[0182] S44: For the second-category key in each file to be processed, only the key-value pair corresponding to the second-category key written last time is retained.

[0183] In the embodiment of the present application, the second type key processing process in each file to be processed is the same as the above part and will not be repeated here.

[0184] S44: For the first type of key in each file to be processed, determine whether the first type of key is invalid according to a preset judgment condition. If so, execute S44; otherwise, execute S46.

[0185] S45: Whenever a first-category key is determined to be an invalid key, the key-value pair data corresponding to the invalid key is deleted.

[0186] In the embodiment of the present application, whether the first-class key is invalid is identified by a preset judgment condition. The specific preset judgment condition is described above, and the present application will not repeat it here. If the judgment result shows that a first-class key is invalid, then the key and its associated key-value pair data will be deleted.

[0187] S46: Whenever a first-category key is determined to be a valid key, the key-value pair data corresponding to the valid key is added to the target SST.

[0188] In the embodiment of the present application, for the first-category key and its key-value pair data that are determined to be valid, these data are written into the target SST, and the target SST level is a processed storage structure.

[0189] Based on the same inventive concept, the present application also provides a data processing device. Figure 5As shown, it is a schematic diagram of the structure of the data processing device 500, which may include:

[0190] An acquisition unit 501 is used to acquire, for any level of the log structure merge LSM tree, each to-be-processed file in the level; wherein each of the to-be-processed files is a read-only memory table or a source SST;

[0191] The determination unit 502 is used to sort and traverse the first-category keys in the files to be processed according to the size of the keys, and during the traversal process, determine whether the first-category keys are invalid according to a preset determination condition; wherein the determination condition includes at least one of the following: the target parameter associated with the first-category key satisfies a first value condition, and the target parameter associated with the first-category key and its predecessor first-category key respectively satisfies a second value condition; the first-category key is a different key in the files to be processed; the target parameter associated with each first-category key includes at least one of the following: the first-category key itself, and the value corresponding to the first-category key;

[0192] The generating unit 503 is used to generate a target SST for each of the to-be-processed files according to the key-value pair data of the determined valid key.

[0193] Optionally, the target parameter includes the first type of key itself, and the first value condition includes at least one of the following: the size of the key is within the first preset range of the hierarchical association, and the key converted by the first preset rule is the same as the first specified parameter of the hierarchical association;

[0194] The second value condition includes: the current key and its predecessor key satisfy the first preset relationship of the hierarchical association;

[0195] The target parameter includes the value corresponding to the first type of key, and the first value condition includes at least one of the following: the size of the value is within the second preset range of the hierarchical association, and the value converted by the second preset rule is the same as the second specified parameter of the hierarchical association;

[0196] The second value-taking condition includes: the value corresponding to the current key and the value corresponding to its preceding key satisfy the second preset relationship of the hierarchical association.

[0197] Optionally, each of the to-be-processed files is: at least one read-only memory table to be sunk generated in real time in the level, or multiple source SSTs to be merged generated in real time in the level and its adjacent levels, or multiple source SSTs to be merged generated in real time in the level;

[0198] The generating unit 503 is specifically used for:

[0199] Whenever a first-category key is determined to be an invalid key, the key-value pair data corresponding to the invalid key is deleted;

[0200] Whenever a first-category key is determined to be a valid key, the key-value pair data corresponding to the valid key is added to the target SST.

[0201] Optionally, each of the to-be-processed files is at least one read-only memory table obtained by performing a background scan on the level, or multiple source SSTs obtained by performing a background scan on the level and its adjacent levels, or multiple source SSTs obtained by performing a background scan on the level, then the generating unit 503 is specifically used for:

[0202] Count the proportion of invalid keys in each of the files to be processed;

[0203] According to the proportion of invalid keys corresponding to each of the to-be-processed files and the order between each of the to-be-processed files, the to-be-processed files are divided into at least one file group;

[0204] For each file group obtained by the division, the key-value pair data corresponding to the invalid key in the file group is deleted, and the key-value pair data corresponding to the valid key in the file group is added to the target SST corresponding to the file group.

[0205] Optionally, the generating unit 503 is specifically used for:

[0206] Determine each target file in which the invalid key ratio is higher than a preset threshold among the files to be processed;

[0207] Divide each n adjacent target files into a file group; where n is a positive integer greater than or equal to 2, and the order between the target files is determined according to the size of the key.

[0208] Optionally, a scanning frequency of background scanning of the LSM tree is positively correlated with a speed of writing data in the LSM tree.

[0209] Optionally, the device 500 further includes:

[0210] The deduplication unit 504 is used to retain only the key-value pair data corresponding to the second-category key written last time in each of the files to be processed before the determination unit determines whether the first-category key is invalid according to a preset determination condition, wherein the second-category key is the same key in each of the files to be processed.

[0211] The data processing device provided in the above-mentioned embodiment of the present application and the data processing method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects.

[0212] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0213] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0214] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device can be a server, or a terminal device such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the above data processing method.

[0215] like Figure 6 As shown, it is a schematic diagram of an electronic device provided in some embodiments of the present application, including a memory 601, a communication module 603 and one or more processors 602.

[0216] The memory 601 is used to store computer programs executed by the processor 602. The memory 601 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0217] The memory 601 may be a volatile memory, such as a random-access memory (RAM); the memory 601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 601 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 601 may be a combination of the above memories.

[0218] The processor 602 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 602 is used to implement the above data processing method when calling the computer program stored in the memory 601 .

[0219] The communication module 603 is used to communicate with terminal devices and other servers.

[0220] The specific connection medium between the memory 601, the communication module 603 and the processor 602 is not limited in the embodiment of the present application. Figure 6 In the embodiment, the memory 601 and the processor 602 are connected via a bus 604. The bus 604 is connected to the processor 602 via a bus 604. Figure 6 The connections between the other components are only for illustration and are not intended to be limiting. The bus 604 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 6 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0221] The memory 601 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the data processing method of the embodiment of the present application. The processor 602 is used to execute the above-mentioned data processing method, such as Figure 1 shown.

[0222] In some possible implementations, various aspects of the data processing method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the data processing method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 1 Follow the steps shown in .

[0223] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0224] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0225] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0226] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0227] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0228] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0229] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: For any level of the log structure merge LSM tree, obtain each to-be-processed file associated with the level; wherein each of the to-be-processed files is a read-only memory table or a source sort string table SST; The first-class keys in each of the to-be-processed files are sorted and traversed according to the size of the keys. During the traversal process, whether the first-class keys are invalid is determined according to a preset judgment condition; wherein the judgment condition includes at least one of the following: the target parameter associated with the first-class key satisfies a first value condition, and the target parameter associated with the first-class key and its predecessor first-class key respectively satisfies a second value condition; the first-class key is a different key in each of the to-be-processed files; the target parameter associated with each first-class key includes at least one of the following: the first-class key itself, and the value corresponding to the first-class key; For each of the files to be processed, a target SST is generated according to the key-value pair data of the determined valid key.

2. The method according to claim 1, characterized in that If the target parameter includes the first type of key itself, the first value condition includes at least one of the following: the size of the key is within the first preset range of the hierarchical association, and the key converted by the first preset rule is the same as the first specified parameter of the hierarchical association; The second value condition includes: the current key and its predecessor key satisfy the first preset relationship of the hierarchical association; The target parameter includes the value corresponding to the first type of key, and the first value condition includes at least one of the following: the size of the value is within the second preset range of the hierarchical association, and the value converted by the second preset rule is the same as the second specified parameter of the hierarchical association; The second value-taking condition includes: the value corresponding to the current key and the value corresponding to its preceding key satisfy the second preset relationship of the hierarchical association.

3. The method according to claim 1, characterized in that The files to be processed are: at least one read-only memory table to be sunk generated in real time in the level, or multiple source SSTs to be merged generated in real time in the level and its adjacent levels, or multiple source SSTs to be merged generated in real time in the level; Then, for each of the to-be-processed files, a target SST is generated according to the key-value pair data of the determined valid key, including: Whenever a first-category key is determined to be an invalid key, the key-value pair data corresponding to the invalid key is deleted; Whenever a first-category key is determined to be a valid key, the key-value pair data corresponding to the valid key is added to the target SST.

4. The method according to claim 1, characterized in that Each of the to-be-processed files is at least one read-only memory table obtained by performing background scanning on the level, or multiple source SSTs obtained by performing background scanning on the level and its adjacent levels, or multiple source SSTs obtained by performing background scanning on the level; Then, for each of the to-be-processed files, a target SST is generated according to the key-value pair data of the determined valid key, including: Count the proportion of invalid keys in each of the files to be processed; According to the proportion of invalid keys corresponding to each of the to-be-processed files and the order between each of the to-be-processed files, the to-be-processed files are divided into at least one file group; For each file group obtained by the division, the key-value pair data corresponding to the invalid key in the file group is deleted, and the key-value pair data corresponding to the valid key in the file group is added to the target SST corresponding to the file group.

5. The method according to claim 4, characterized in that The step of dividing the files to be processed into at least one file group according to the invalid key ratio corresponding to each file to be processed and the order between the files to be processed comprises: Determine each target file in which the invalid key ratio is higher than a preset threshold among the files to be processed; Divide each n adjacent target files into a file group; where n is a positive integer greater than or equal to 2, and the order between the target files is determined according to the size of the key.

6. The method according to claim 4, characterized in that The scanning frequency of the background scanning of the LSM tree is positively correlated with the speed of writing data in the LSM tree.

7. The method according to any one of claims 1 to 6, characterized in that: Before determining whether the first type of key is invalid according to the preset judgment condition, the method further includes: For the second type of keys in each of the files to be processed, only the key-value pair data corresponding to the second type of keys written last time is retained, wherein the second type of keys are the same keys in each of the files to be processed.

8. A data processing device, characterized in that: include: An acquisition unit, configured to acquire, for any level of the LSM tree, each to-be-processed file in the level; wherein each of the to-be-processed files is a read-only memory table or a source SST; A determination unit is used to sort and traverse the first-category keys in each of the to-be-processed files according to the size of the keys, and during the traversal process, determine whether the first-category keys are invalid according to a preset determination condition; wherein the determination condition includes at least one of the following: the target parameter associated with the first-category key satisfies a first value condition, and the target parameter associated with the first-category key and its predecessor first-category key respectively satisfies a second value condition; the first-category key is a different key in each of the to-be-processed files; the target parameter associated with each first-category key includes at least one of the following: the first-category key itself, and the value corresponding to the first-category key; The generating unit is used to generate a target SST for each of the to-be-processed files according to the key-value pair data of the determined valid key.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 7.

Images