Information processing method and device, equipment and storage medium

Abstract

The method comprises: selecting a target storage partition with the highest invalid information rate from a plurality of storage partitions included in a storage unit according to LSMT information in a key partition, wherein the invalid information rate is used to represent the proportion of invalid key value information in total key value information in the storage partition; detecting validity information corresponding to each key value information in the target storage partition, and screening valid key value information from a plurality of key value information stored in the target storage partition according to the validity information corresponding to each key value information, wherein the validity information comprises valid or invalid; transferring the valid key value information to a first storage partition other than the target storage partition, and erasing the plurality of key value information stored in the target storage partition. The application can reduce the increase of writing and improve the processing efficiency of data.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to an information processing method, apparatus, device, and storage medium. Background Technology

[0002] Currently, key-value stores based on Log-Structured Merge Tree (LSMT) can receive key-value records corresponding to write operations through a memory table (MemTable). When the size of the MemTable reaches a certain threshold, it stores the key-value records in the storage unit of the key-value store system (e.g., hard disk drive or solid-state drive) in the format of a Sorted String Table (SST). However, as key-value records are updated, the number of invalid key-value records accumulated in the SST file increases, therefore, garbage collection of invalid key-value records in the SST file is required.

[0003] However, the existing processing methods for garbage collecting SST files increase the write operations of key-value records, resulting in write amplification of the key-value storage system. Therefore, the efficiency of the above information processing methods is low. Summary of the Invention

[0004] This disclosure provides an information processing method, apparatus, device, and storage medium that can improve the efficiency of information processing methods.

[0005] In a first aspect, embodiments of this disclosure provide an information processing method applied to a key-value storage system with key-value separation. The storage unit in the key-value storage system includes a key partition and multiple storage partitions, wherein the key partition is used to store Log Structure Merge Tree (LSMT) information, and the storage partitions are used to store key-value information; the method includes:

[0006] Based on the LSMT information in the key partition, the target storage partition with the highest invalid information rate is selected from multiple storage partitions included in the storage unit, wherein the invalid information rate is used to represent the proportion of invalid key value information in the storage partition in the total key value information;

[0007] The validity information corresponding to each key-value information in the target storage partition is detected, and based on the validity information corresponding to each key-value information, valid key-value information is filtered out from multiple key-value information stored in the target storage partition, wherein the validity information includes valid or invalid.

[0008] copy the valid key-value information to a first storage partition other than the target storage partition, and erase the plurality of key-value information stored in the target storage partition.

[0009] In a second aspect, the embodiments of the present disclosure provide an information processing device, applied to a key-value separation key-value storage system, a storage unit in the key-value storage system comprising a key partition and a plurality of storage partitions, wherein the key partition is used to store log-structured merge tree (LSMT) information, and the storage partitions are used to store key-value information, and the device comprises:

[0010] a selection module configured to select, according to the LSMT information in the key partition, a target storage partition with the highest invalid information rate from the plurality of storage partitions comprised in the storage unit, wherein the invalid information rate is used to represent the proportion of invalid key-value information in the total key-value information in the storage partition;

[0011] a screening module configured to detect validity information corresponding to each key-value information in the target storage partition, and screen, according to the validity information corresponding to each key-value information, valid key-value information from the plurality of key-value information stored in the target storage partition, wherein the validity information comprises valid or invalid;

[0012] a processing module configured to copy the valid key-value information to a first storage partition other than the target storage partition, and erase the plurality of key-value information stored in the target storage partition.

[0013] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising:

[0014] a processor, and a memory connected to the processor in communication;

[0015] the memory stores computer-executable instructions;

[0016] the processor executes the computer-executable instructions stored in the memory, to implement the information processing method of the first aspect.

[0017] In a fourth aspect, the embodiments of the present disclosure provide a computer-readable storage medium, which stores computer-executable instructions, and when a processor executes the computer-executable instructions, the information processing method of the first aspect is implemented.

[0018] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, which, when executed by a processor, implements the information processing method of the first aspect.

[0019] The information processing method, device, equipment and storage medium provided by the embodiment include: selecting a target storage partition with the highest invalid information rate from a plurality of storage partitions included in a storage unit according to LSMT information in a key partition, wherein the invalid information rate is used to represent the proportion of invalid key value information in total key value information in the storage partition; detecting validity information corresponding to each key value information in the target storage partition, and selecting valid key value information from a plurality of key value information stored in the target storage partition according to the validity information corresponding to each key value information, wherein the validity information includes valid or invalid; storing the valid key value information into a first storage partition except the target storage partition, and erasing the plurality of key value information stored in the target storage partition. In the embodiment, the target storage partition with the highest invalid information rate is selected from the plurality of storage partitions included in the storage unit according to the LSMT information, and the key value information in the target storage partition is processed, so that the LSMT information and the garbage collection granularity of the storage partition are aligned, the garbage collection of the LSMT information is performed at the same time, and the garbage collection of the storage partition is also performed, compared with the separate recovery method, the number of garbage collection is saved, the write amplification of the key value information is reduced, and therefore the data processing efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A structural schematic diagram of a key-value separation key-value storage system provided by an embodiment of the present disclosure;

[0022] Figure 2 A flowchart of an information processing method provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of an information processing method provided by an embodiment of the present disclosure;

[0024] Figure 4 A flowchart of another information processing method provided by an embodiment of the present disclosure;

[0025] Figure 5 A structural block diagram of an information processing device provided by an embodiment of the present disclosure;

[0026] Figure 6 A hardware structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] To make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the drawings in the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present disclosure.

[0028] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] At present, a key-value storage system based on a Log-Structured Merge Tree (LSMT) can receive a key-value record corresponding to a write operation through a Memory Table (MemTable). The key-value record written by a user is organized in the MemTable according to the order of the key, and when the size of the key-value record in the MemTable reaches a certain threshold, it will store the key-value record in the format of a Sorted String Table (SST) into a storage unit of the key-value storage system. Optionally, the storage unit is a persistent storage medium (for example, a hard disk drive or a solid state disk), and the SST file can be flushed to the underlying persistent storage medium through a flush thread.

[0030] However, with the update of the key-value record in the LSMT, the number of invalid key-value records accumulated in the SST file is increasing, and therefore, it is necessary to perform garbage collection on the invalid key-value records in the SST file.

[0031] In the prior art, the invalid key-value records in the SST file are garbage collected by compressing the SST file. The specific steps are as follows: determining the valid key-value records and the invalid key-value records in the SST file; deleting the invalid key-value records in the SST file and transferring the valid key-value records in the SST file to other SST files. However, when the invalid key-value records in the SST file are garbage collected, the valid key-value records in the SST file need to be transferred to other SST files, which increases the write operation of the key-value record and causes the write amplification of the key-value storage system, and therefore the processing efficiency of the above information is low.

[0032] Further, the LSM-tree key-value storage system built on the SSD (Solid State Disk) has the problem of Two-Level GC (two-layer garbage collection). Among them, the first layer: the LSM-tree key-value storage system itself needs to perform garbage collection on expired key-value records. The second layer: the underlying SSD device has a device garbage collection mechanism transparent to the user (that is, the user can trigger the garbage collection of the SSD device). Since the two layers of garbage collection are separated from each other, this further exacerbates the problem of write amplification.

[0033] Therefore, how to reduce the write amplification of the key-value storage system to improve the processing efficiency of information is a problem to be solved at present.

[0034] The compression operation of the LSMT key-value storage system in the prior art will cause a serious write amplification problem, and the root cause is that the compression process will rewrite the value part of the key-value record. Therefore, in order to reduce the write operation of the key-value record, a key-value separated LSMT key-value storage system can be used, and the SST file in the LSMT only stores the key part information, and the value part information is stored separately in other partitions. In this way, when the SST file is compressed, only the key part of the key-value record needs to be rewritten, reducing the amount of data to be rewritten, thereby reducing the write amplification of the key-value storage system and improving the processing efficiency of information.

[0035] However, the key-value separated LSMT key-value storage system still has the problem of write amplification. For example, the key-value separated LSMT key-value storage system can be a ZNS SSD (Zoned Namespace SSD, Zoned Namespace SSD system), which includes a plurality of separately partitioned storage units. Among them, the storage unit can be a key partition or a storage partition. Alternatively, the ZNS SSD system includes a key partition and a plurality of storage partitions, wherein the key partition is used to store the LSMT information, and the storage partition is used to store the key-value information. However, since the storage partition and the key partition are separately garbage collected, the number of garbage collections of the key-value storage system is relatively large.

[0036] Therefore, in order to further reduce the write amplification of the key-value storage system, the embodiment provides the following technical concept: aligning the garbage collection granularity of the LSMT information and the garbage collection granularity of the storage partition, and performing garbage collection on the storage partition at the same time of performing garbage collection on the LSMT information. The specific processing method is as follows: first, according to the LSMT information in the key partition, selecting a target storage partition with the highest invalid information rate from a plurality of storage partitions included in the storage unit; then, detecting the validity information corresponding to each key-value information in the target storage partition, and according to the validity information corresponding to each key-value information, screening valid key-value information from a plurality of key-value information stored in the target storage partition; and finally, transferring the valid key-value information to a first storage partition other than the target storage partition, and erasing the plurality of key-value information stored in the target storage partition.

[0037] In this case, since the target storage partition with the highest invalid information rate is selected from a plurality of storage partitions included in the storage unit according to the LSMT information, and the key-value information in the target storage partition is processed, the garbage collection granularity of the LSMT information and the garbage collection granularity of the storage partition are aligned, the storage partition is garbage collected at the same time of garbage collecting the LSMT information, compared with the separate recovery method, the number of garbage collections is saved, the write amplification of the key-value information is reduced, and therefore the data processing efficiency is improved.

[0038] The application scenarios of the embodiments of the present disclosure are explained as follows:

[0039] The information processing method provided by the embodiments of the present disclosure can be applied to the scene of performing consistent processing on data. Figure 1 A structure diagram of a key-value separation key-value storage system provided by the embodiments of the present disclosure is shown in FIG. 1. Figure 1 As shown in the figure, the key-value storage system includes a memory 101 and a storage unit 102, and the storage unit 102 includes a key partition 1021 and a plurality of storage partitions 1022. The key-value storage system can receive a key-value record corresponding to a write operation through a MemTable, store the MemTable in the memory, and when the size of the key-value record in the MemTable reaches a certain threshold, store the key information in the key-value record and the position information of the key-value record to the key partition 1021 through an SST file, and store the key-value record to the storage partition 1022. When the key-value record needs to be called, the position information of the corresponding key-value record can be queried through the key information, and the key-value record can be called from the storage partition 1022 through the position information of the key-value record. Optionally, Figure 1The key-value separation key-value storage system in the method can be a ZNS solid state disk. Accordingly, the information processing method provided by the embodiments of the present disclosure can be applied to a scenario of garbage collection of the ZNS solid state disk. The information processing method provided by the embodiments of the present disclosure is described in detail below by means of detailed embodiments.

[0040] Figure 2 A flowchart of an information processing method provided by the embodiments of the present disclosure is shown in FIG. 1. The information processing method can be applied to a key-value separation key-value storage system. The storage unit in the key-value storage system includes a key partition and a plurality of storage partitions. The key partition is used to store log-structured merge tree (LSMT) information, and the storage partitions are used to store key-value information. As shown in FIG. 1, the method includes the following steps. Figure 2

[0041] S201, selecting, according to the LSMT information in the key partition, a target storage partition with the highest invalid information rate from the plurality of storage partitions included in the storage unit, wherein the invalid information rate is used to represent the proportion of invalid key-value information in the total key-value information in the storage partition.

[0042] In the embodiments of the present disclosure, the LSMT information includes a plurality of index information. The index information includes key information and storage location information of the key-value information corresponding to the key information. It should be noted that the key partition can include a plurality of SST files, and the plurality of index information can be stored in the SST files.

[0043] Optionally, the storage location information can include a file identifier of the key-value information. For example, the storage location information is file A. Optionally, the storage location information can include a file identifier of the key-value information and a storage partition identifier where the file identifier is located. For example, the storage location information is file A and storage partition 1. The key-value information can be a key-value record.

[0044] In the embodiments of the present disclosure, the storage partition can be represented as Zone. When the LSMT information is compressed, the deleted key information in the plurality of index information can be synchronized to each corresponding storage partition Zone. In this way, the storage partition can obtain the invalid information rate (i.e., the garbage rate) of the storage partition.

[0045] ​Accordingly, selecting, according to the LSMT information in the key partition, a target storage partition with the highest invalid information rate from a plurality of storage partitions included in the storage unit, comprises: in response to compression processing of a plurality of index information included in the LSMT information, recording deleted key information in the plurality of index information to obtain statistical information; for each storage partition in the storage unit, determining, according to the statistical information, the number of invalid key value information in the storage partition, wherein the invalid key value information is the key value information corresponding to the deleted key information; taking the ratio of the number of invalid key value information in the storage partition to the total number of key value information stored in the storage partition as the invalid information rate of the storage partition, and selecting the target storage partition with the highest invalid information rate from the plurality of storage partitions.

[0046] Optionally, the statistical information includes a plurality of deleted key information and a storage partition where key value information corresponding to each key information is stored. Accordingly, determining, according to the statistical information, the number of invalid key value information in the storage partition comprises: sequentially determining each deleted key information in the statistical information, and adding 1 to the number of invalid key value information in the storage partition corresponding to the key information.

[0047] For example, the number of invalid key value information in the storage partition can be represented by N stale , and the total number of key value information stored in the storage partition can be represented by N total . Accordingly, the invalid information rate can be represented as: N stale / N total .

[0048] It should be noted that when the storage amount of the storage unit meets a preset condition, the target storage partition with the highest invalid information rate is selected from the plurality of storage partitions for garbage collection processing.

[0049] In some embodiments, when the storage amount of the storage unit is greater than a preset value, a target storage partition with the highest invalid information rate is selected from a plurality of storage partitions included in the storage unit according to the LSMT information in the key partition. In this embodiment, the value of the preset value is not specifically limited. Optionally, the preset value can be a preset multiple of the total amount of the storage unit. For example, the preset value can be 0.8 times, 0.85 times, etc. of the total amount of the storage unit.

[0050] In other embodiments, when the interval between the last time the target storage partition is selected and the current time reaches a preset length of time, a target storage partition with the highest invalid information rate is selected from a plurality of storage partitions included in the storage unit according to the LSMT information in the key partition. In this embodiment, the value of the preset length of time is not specifically limited. Optionally, the preset length of time can be 1 day, 2 days, etc.

[0051] S202, detecting validity information corresponding to each key-value information in the target storage partition, and filtering valid key-value information from the plurality of key-value information stored in the target storage partition according to the validity information corresponding to each key-value information, wherein the validity information includes valid or invalid.

[0052] In the embodiments of the present disclosure, whether the key information in the key-value information is the latest key information can be determined according to the key information in the key-value information.

[0053] Optionally, the LSMT information includes a plurality of index information, and the index information includes key information and storage location information of the key-value information corresponding to the key information. Accordingly, whether the key information in the key-value information is the latest key information can be determined according to the key information in the index information. Specifically, for each key-value information, the key information corresponding to the key-value information is determined, the key information includes a key and a key identifier, and the key identifier is used to represent the version corresponding to the key; according to the key in the key information, the latest version key identifier corresponding to the key is determined from the plurality of index information, and it is judged whether the key identifier in the key information is the same as the latest version key identifier; if they are the same, it is determined that the validity information corresponding to the key-value information is valid, and if they are different, it is determined that the validity information corresponding to the key-value information is invalid.

[0054] For example, the key-value information 1 is key_A+seq1, value1. The key information corresponding to the key-value information 1 is key_A+seq1. The key in the key information is key_A, and the key identifier in the key information is seq1. The plurality of index information includes "key_A+seq1, file1", "key_A+seq2, file2", and "key_A+seq3, file3". According to the key (key_A) in the key information, the latest version key identifier corresponding to the key is seq3. It is determined that the key identifier (seq1) in the key information is different from the latest version key identifier (seq3), and it is determined that the validity information corresponding to the key-value information (key_A+seq1, value1) is invalid.

[0055] It should be noted that when the number of index information in the key partition LSM-tree is large, the time for determining the validity information corresponding to the key-value information is long. Therefore, on the basis of ensuring the detection accuracy, different detection methods can be used according to the type of the storage partition where the key-value information is located.

[0056] Optionally, the types of the storage partitions include a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein the hotness value of the key-value information stored in the first hotness level storage partition is greater than the hotness value of the key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than the hotness value of the key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent the update times of the key information corresponding to the key-value information. For example, the first hotness level storage partition can be denoted as a Hot partition; the second hotness level storage partition can be denoted as a Warm partition; and the third hotness level storage partition can be denoted as a Cold partition. In the embodiments of the present application, the number of the first hotness level storage partition, the second hotness level storage partition and the third hotness level storage partition is not limited.

[0057] In some embodiments, the type of the target storage partition can be determined first; if the type of the target storage partition is the first hotness level storage partition or the second hotness level storage partition, the validity of the key-value information is not determined according to the key information in the index information; if the type of the target storage partition is the third hotness level storage partition, the validity of the key-value information is determined according to the key information in the index information.

[0058] Optionally, before the step of determining the key identifier of the latest version corresponding to the key from the plurality of index information according to the key in the key information, the type of the target storage partition is determined; if the type of the target storage partition is the first hotness level storage partition or the second hotness level storage partition, it is determined whether the key identifier in the key information is the key identifier of the latest version in the target storage partition; if yes, it is determined that the validity information corresponding to the key-value information is valid; if no, it is determined that the validity information corresponding to the key-value information is invalid.

[0059] If the type of the target storage partition is the third hotness level storage partition, the step of determining the key identifier of the latest version corresponding to the key from the plurality of index information according to the key in the key information is performed.

[0060] Here, since the probability that a plurality of key-value information corresponds to the same key information is relatively large in the first hotness level storage partition or the second hotness level storage partition, the first hotness level storage partition or the second hotness level storage partition can be used to check the validity.

[0061] S203, the valid key-value information is transferred to the first storage partition other than the target storage partition, and the plurality of key-value information stored in the target storage partition is erased.

[0062] In the embodiments of the present disclosure, the first storage partition is of the same type as the target storage partition. Accordingly, the step is: determining the first storage partition of the same type as the target storage partition from the plurality of storage partitions except the target storage partition; and transferring the valid key-value information to the first storage partition.

[0063] For example, if the target storage partition is a Hot partition, the first storage partition is also a Hot partition.

[0064] Here, since the number of times of updating the key-value information in the storage partitions of the same type is similar, the number of valid key-value information in the storage partition can be reduced, and the transferring of the valid key-value information is reduced, thereby further improving the efficiency of information processing.

[0065] It should be noted that the number of valid key-value information can be zero, one or more. When the number of valid key-value information is zero, the plurality of key-value information stored in the target storage partition can be directly erased, and the step of transferring the valid key-value information is not performed.

[0066] The information processing method provided in the embodiments is: selecting a target storage partition with the highest invalid information rate from a plurality of storage partitions included in a storage unit according to LSMT information in a key partition, wherein the invalid information rate is used to represent the proportion of invalid key-value information in the total key-value information in the storage partition; detecting the validity information corresponding to each key-value information in the target storage partition, and selecting valid key-value information from the plurality of key-value information stored in the target storage partition according to the validity information corresponding to each key-value information, wherein the validity information includes valid or invalid; transferring the valid key-value information to a first storage partition except the target storage partition, and erasing the plurality of key-value information stored in the target storage partition. In the embodiments of the present disclosure, since the target storage partition with the highest invalid information rate is selected from the plurality of storage partitions included in the storage unit according to the LSMT information, and the key-value information in the target storage partition is processed, the LSMT information and the garbage collection granularity of the storage partition are aligned, the garbage collection of the LSMT information is performed at the same time, and the garbage collection of the storage partition is also performed. Compared with the separate recovery method, the number of garbage collection is saved, the write amplification of the key-value information is reduced, and therefore the processing efficiency of the data is improved.

[0067] It should be noted that in order to further reduce the write amplification of the key-value information, the index information corresponding to the transferred key-value information is not transferred, and only the index information is updated.

[0068] Correspondingly, after the valid key-value information is migrated to the first storage partition other than the target storage partition, the method further includes: for each valid key-value information, obtaining target index information corresponding to the valid key-value information from the plurality of index information; determining migration path information of the valid key-value information from the target storage partition to the first storage partition, and adding the migration path information to the target index information to obtain new index information corresponding to the valid key-value information.

[0069] When it is needed to query a certain key-value information, the key-value information is retrieved from the storage partition according to the target index information and the migration path information corresponding to the key-value information.

[0070] For example, as shown in Figure 3 The index information includes SST1, SST2, SST4, SST5, SST6 and SST7; wherein, the SST1 and the SST2 are located in the first level of the LSMT information; the key value interval of the key information stored in the SST1 is [1, 15], and the key value interval of the key information stored in the SST2 is [5, 20]. The SST4 and the SST5 are located in the second level of the LSMT information; the key value interval of the key information stored in the SST4 is [1, 20], and the key value interval of the key information stored in the SST5 is [7, 25]. The SST6 and the SST7 are located in the third level of the LSMT information; the key value interval of the key information stored in the SST6 is [1, 11], and the key value interval of the key information stored in the SST7 is [12, 25]. As shown in Figure 3 If the key information of the key-value information is 19, the target index information corresponding to the key information 19 is the SST2. The migration path information corresponding to the SST2 is SST2→SST4→SST7.

[0071] Figure 4 A flowchart of another information processing method provided by the embodiment of the present disclosure is provided. In the embodiment, the key-value storage system further includes a hot set information, and the hot set information includes the average update times of the key information and the update times of the key information corresponding to each of the plurality of index information. Optionally, when the key-value information in the MemTable is flushed to the storage unit, the update times of each key-value information can be determined first, and then the key-value information is stored in the storage partition of the corresponding type according to the update times. As shown in Figure 4 The method for writing the key-value information includes:

[0072] S401, in response to receiving the key-value information to be written, obtaining the update times of the key information in the key-value information, the minimum update times of the key information and the average update times of the key information from the hot set information.

[0073] In the embodiments of the present disclosure, the hotness set information maintains the update times of each key. After garbage collection is performed on the storage partition, the update times of each key information added in the storage partition are transmitted to the hotness set information, and the update times of the key information are added to the added update times. The hotness set information can be represented by Hotness Set.

[0074] The hotness set information further includes the average update times of the key information. The average update times of the key information are determined based on the update times of the deleted key information in the hotness set information and the update times of the deleted key information. It should be noted that the hotness set information cannot maintain the occurrence frequencies of all the keys, otherwise, the memory space will be too large. Alternatively, the number of keys maintained by the hotness set information can be limited to a preset number. In the embodiments, the value of the preset number is not limited. For example, the preset number can be 0.5% of the entire storage space of the memory unit. When the number of keys maintained by the hotness set information reaches the preset number, the key with the minimum update times in the hotness set information can be selected for replacement.

[0075] S402, determining the hotness value level of the key value information according to the update times, the average update times and the minimum update times, and determining the second storage partition matching the hotness value level from the plurality of storage partitions.

[0076] Alternatively, the types of the storage partitions include a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein the hotness value of the key value information stored in the first hotness level storage partition is greater than the hotness value of the key value information stored in the second hotness level storage partition, and the hotness value of the key value information stored in the second hotness level storage partition is greater than the hotness value of the key value information stored in the third hotness level storage partition, wherein the hotness value is used to represent the update times of the key information corresponding to the key value information. Accordingly, the step is: if the update times are greater than the average update times, determining that the hotness value level of the key value information is the first hotness level, and determining the first hotness level storage partition as the second storage partition from the plurality of storage partitions; if the update times are less than or equal to the average update times and greater than the minimum update times, determining that the hotness value level of the key value information is the second hotness level, and determining the second hotness level storage partition as the second storage partition from the plurality of storage partitions; and if the update times are less than or equal to the minimum update times, determining that the hotness value level of the key value information is the third hotness level, and determining the third hotness level storage partition as the second storage partition from the plurality of storage partitions.

[0077] S403, storing the key value information into the second storage partition.

[0078] In the embodiments of the present disclosure, since the update times of the key-value information are determined first when the key-value information is flushed to the storage unit, and then the key-value information is stored in the storage partition of the corresponding type according to the update times, the update times of the key-value information in the storage partition of the same type are similar, and when the storage partition is garbage collected, the number of valid key-value information in the storage partition can be reduced, and the valid key-value information can be stored, so that the efficiency of information processing is further improved.

[0079] Figure 5 A structural block diagram of an information processing device provided in the embodiments of the present disclosure is provided. The information processing device is applied to a key-value separation key-value storage system, and a storage unit in the key-value storage system includes a key partition and a plurality of storage partitions. The key partition is used to store log-structured merge tree (LSMT) information, and the storage partition is used to store key-value information. Referring to Figure 5 The device includes a selection module 501, a screening module 502, and a processing module 503.

[0080] The selection module 501 is configured to select a target storage partition with the highest invalid information rate from the plurality of storage partitions included in the storage unit according to the LSMT information in the key partition, where the invalid information rate is used to represent the proportion of invalid key-value information in the total key-value information in the storage partition.

[0081] The screening module 502 is configured to detect the validity information corresponding to each key-value information in the target storage partition, and screen the valid key-value information from the plurality of key-value information stored in the target storage partition according to the validity information corresponding to each key-value information, where the validity information includes valid or invalid.

[0082] The processing module 503 is configured to store the valid key-value information in a first storage partition other than the target storage partition, and erase the plurality of key-value information stored in the target storage partition.

[0083] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information, and the index information includes key information and storage location information of key value information corresponding to the key information; accordingly, the selecting module 501 selects a target storage partition with the highest invalid information rate from a plurality of storage partitions included in a storage unit according to the LSMT information in the key partition, and specifically includes: in response to compression processing of a plurality of index information included in the LSMT information, recording deleted key information in the plurality of index information, obtaining statistical information; for each storage partition in the storage unit, determining the number of invalid key value information in the storage partition according to the statistical information, wherein the invalid key value information is key value information corresponding to the deleted key information; taking the ratio of the number of invalid key value information in the storage partition to the total number of key value information stored in the storage partition as the invalid information rate of the storage partition, and selecting a target storage partition with the highest invalid information rate from the plurality of storage partitions.

[0084] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information, and the index information includes key information and storage location information of key value information corresponding to the key information; accordingly, the selecting module 501 selects a target storage partition with the highest invalid information rate from a plurality of storage partitions included in a storage unit according to the LSMT information in the key partition, and specifically includes: in response to compression processing of a plurality of index information included in the LSMT information, recording deleted key information in the plurality of index information, obtaining statistical information; for each storage partition in the storage unit, determining the number of invalid key value information in the storage partition according to the statistical information, wherein the invalid key value information is key value information corresponding to the deleted key information; taking the ratio of the number of invalid key value information in the storage partition to the total number of key value information stored in the storage partition as the invalid information rate of the storage partition, and selecting a target storage partition with the highest invalid information rate from the plurality of storage partitions.

[0085] According to one or more embodiments of the present disclosure, the type of the storage partition includes a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein the hotness value of the key value information stored in the first hotness level storage partition is greater than the hotness value of the key value information stored in the second hotness level storage partition, and the hotness value of the key value information stored in the second hotness level storage partition is greater than the hotness value of the key value information stored in the third hotness level storage partition, wherein the hotness value is used to represent the update times of the key information corresponding to the key value information;

[0086] Correspondingly, the apparatus further comprises a verification module; the verification module is configured to determine the type of the target storage partition; if the type of the target storage partition is the first hotness level storage partition or the second hotness level storage partition, determine whether the key identifier in the key information is the key identifier of the latest version in the target storage partition; if yes, determine that the validity information corresponding to the key-value information is valid; if not, determine that the validity information corresponding to the key-value information is invalid; if the type of the target storage partition is the third hotness level storage partition, determine the key identifier of the latest version corresponding to the key in the key information from the plurality of index information.

[0087] According to one or more embodiments of the present disclosure, the processing module 503 is configured to migrate the valid key-value information to a first storage partition other than the target storage partition, specifically comprising: determining a first storage partition having the same type as the target storage partition from a plurality of storage partitions other than the target storage partition; and migrating the valid key-value information to the first storage partition.

[0088] According to one or more embodiments of the present disclosure, the LSMT information comprises a plurality of index information, and the apparatus further comprises an updating module; the updating module is configured to, for each valid key-value information, acquire target index information corresponding to the valid key-value information from the plurality of index information; determine migration path information of the valid key-value information from the target storage partition to the first storage partition, and add the migration path information to the target index information to obtain new index information corresponding to the valid key-value information.

[0089] According to one or more embodiments of the present disclosure, the key-value storage system further comprises hotness set information, the hotness set information comprising an average update number of key information and an update number of key information corresponding to each of the plurality of index information; the apparatus further comprises a writing module; the writing module is configured to, in response to receiving key-value information to be written, acquire the update number of key information corresponding to the key-value information, the minimum update number of key information in the hotness set information, and the average update number of key information from the hotness set information; determine the hotness value level of the key-value information according to the update number, the average update number, and the minimum update number; determine a second storage partition matching the hotness value level from the plurality of storage partitions; and store the key-value information into the second storage partition.

[0090] According to one or more embodiments of the present disclosure, the types of the storage partitions include a first hotness level storage partition, a second hotness level storage partition, and a third hotness level storage partition, wherein a hotness value of key-value information stored in the first hotness level storage partition is greater than a hotness value of key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than a hotness value of key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent an update number of key information corresponding to the key-value information; accordingly, the write module determines a hotness value level of the key-value information according to the update number, the average update number, and the minimum update number, and determines a second storage partition matching the hotness value level from the plurality of storage partitions, and specifically includes: if the update number is greater than the average update number, determining that the hotness value level of the key-value information is a first hotness level, and determining the first hotness level storage partition as the second storage partition from the plurality of storage partitions; if the update number is less than or equal to the average update number and greater than the minimum update number, determining that the hotness value level of the key-value information is a second hotness level, and determining the second hotness level storage partition as the second storage partition from the plurality of storage partitions; and if the update number is less than or equal to the minimum update number, determining that the hotness value level of the key-value information is a third hotness level, and determining the third hotness level storage partition as the second storage partition from the plurality of storage partitions.

[0091] The selection module 501, the screening module 502, and the processing module 503 are sequentially connected. The information processing apparatus provided in this embodiment can execute the technical solutions of the method embodiments, and has similar implementation principles and technical effects. Details are not described herein.

[0092] Figure 6 A hardware structure schematic diagram of an electronic device is provided for the embodiments of the present disclosure. Referring to Figure 6 The electronic device 600 can be a terminal device or a server. The terminal device can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a portable media player (PMP), a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 6 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0093] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0094] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0095] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0096] It should be noted that the computer-readable medium in the above disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0097] The computer-readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0098] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0099] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0100] The flow diagrams and the block diagrams in the drawings are meant as possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0101] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0102] The functions described above in the specification of the present disclosure can be performed by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0103] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] In a first aspect, according to one or more embodiments of the present disclosure, an information processing method is provided, applied to a key-value separation key-value storage system, a storage unit in the key-value storage system includes a key partition and a plurality of storage partitions, wherein the key partition is used to store log-structured merge tree (LSMT) information, and the storage partitions are used to store key-value information; the method comprises:

[0105] According to the LSMT information in the key partition, a target storage partition with the highest invalid information rate is selected from the plurality of storage partitions included in the storage unit, wherein the invalid information rate is used to represent the proportion of invalid key-value information in the total key-value information in the storage partition;

[0106] Detect the validity information corresponding to each key-value information in the target storage partition, and according to the validity information corresponding to each key-value information, filter out valid key-value information from the plurality of key-value information stored in the target storage partition, wherein the validity information includes valid or invalid;

[0107] Copy the valid key-value information to a first storage partition other than the target storage partition, and erase the plurality of key-value information stored in the target storage partition.

[0108] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information, and the index information includes key information and storage location information of key value information corresponding to the key information; accordingly, the selecting, from a plurality of storage partitions included in a storage unit, a target storage partition with a highest invalid information rate according to the LSMT information in the key partition includes: in response to compression processing of the plurality of index information included in the LSMT information, recording deleted key information in the plurality of index information to obtain statistical information; for each storage partition in the storage unit, determining, according to the statistical information, a number of invalid key value information in the storage partition, wherein the invalid key value information is key value information corresponding to the deleted key information; taking a ratio of the number of invalid key value information in the storage partition to a total number of key value information stored in the storage partition as an invalid information rate of the storage partition, and selecting, from the plurality of storage partitions, a target storage partition with a highest invalid information rate.

[0109] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information, and the index information includes key information and storage location information of key value information corresponding to the key information; accordingly, the detecting, for each key value information in the target storage partition, validity information corresponding to the key value information includes: for each key value information, determining key information corresponding to the key value information, the key information including a key and a key identifier, the key identifier being used to represent a version corresponding to the key; determining, according to the key in the key information, a key identifier of a latest version corresponding to the key from the plurality of index information, and judging whether the key identifier in the key information is same as the key identifier of the latest version; if same, determining that the validity information corresponding to the key value information is valid, and if different, determining that the validity information corresponding to the key value information is invalid.

[0110] According to one or more embodiments of the present disclosure, the type of the storage partition includes a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein a hotness value of key value information stored in the first hotness level storage partition is greater than a hotness value of key value information stored in the second hotness level storage partition, and the hotness value of the key value information stored in the second hotness level storage partition is greater than a hotness value of key value information stored in the third hotness level storage partition, wherein the hotness value is used to represent a number of updates of key information corresponding to the key value information;

[0111] Correspondingly, before the determining the latest version key identifier corresponding to the key from the plurality of index information according to the key in the key information, the method further includes: determining the type of the target storage partition; if the type of the target storage partition is the first hotness level storage partition or the second hotness level storage partition, determining whether the key identifier in the key information is the latest version key identifier in the target storage partition; if yes, determining that the validity information corresponding to the key value information is valid; if not, determining that the validity information corresponding to the key value information is invalid; if the type of the target storage partition is the third hotness level storage partition, performing the step of determining the latest version key identifier corresponding to the key from the plurality of index information according to the key in the key information.

[0112] According to one or more embodiments of the present disclosure, the copying the valid key value information to the first storage partition other than the target storage partition includes: determining a first storage partition having the same type as the target storage partition from a plurality of storage partitions other than the target storage partition; and copying the valid key value information to the first storage partition.

[0113] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information; and after the copying the valid key value information to the first storage partition other than the target storage partition, the method further includes: for each valid key value information, obtaining target index information corresponding to the valid key value information from the plurality of index information; determining migration path information of the valid key value information from the target storage partition to the first storage partition, and adding the migration path information to the target index information to obtain new index information corresponding to the valid key value information.

[0114] According to one or more embodiments of the present disclosure, the key value storage system further includes hotness set information, the hotness set information including an average update number of key information and an update number of key information corresponding to each of the plurality of index information.

[0115] The method further includes: in response to receiving the key value information to be written, obtaining, from the hotness set information, an update number of key information in the key value information, a minimum update number of key information in the hotness set information, and an average update number of key information; determining a hotness value level of the key value information according to the update number, the average update number, and the minimum update number; determining a second storage partition matching the hotness value level from the plurality of storage partitions; and storing the key value information into the second storage partition.

[0116] According to one or more embodiments of the present disclosure, the types of the storage partitions include a first hotness level storage partition, a second hotness level storage partition, and a third hotness level storage partition, wherein a hotness value of key-value information stored in the first hotness level storage partition is greater than a hotness value of key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than a hotness value of key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent an update frequency of key information corresponding to the key-value information.

[0117] Accordingly, the determining, according to the update frequency, the average update frequency, and the minimum update frequency, of a hotness value level of the key-value information, and the determining, from the plurality of storage partitions, of a second storage partition matching the hotness value level, includes: if the update frequency is greater than the average update frequency, determining that the hotness value level of the key-value information is a first hotness level, and determining, from the plurality of storage partitions, a first hotness level storage partition as the second storage partition; if the update frequency is less than or equal to the average update frequency and greater than the minimum update frequency, determining that the hotness value level of the key-value information is a second hotness level, and determining, from the plurality of storage partitions, a second hotness level storage partition as the second storage partition; and if the update frequency is less than or equal to the minimum update frequency, determining that the hotness value level of the key-value information is a third hotness level, and determining, from the plurality of storage partitions, a third hotness level storage partition as the second storage partition.

[0118] In a second aspect, according to one or more embodiments of the present disclosure, an information processing apparatus is provided, which is applied to a key-value separation key-value storage system, a storage unit in the key-value storage system includes a key partition and a plurality of storage partitions, wherein the key partition is used to store log-structured merge tree (LSMT) information, and the storage partitions are used to store key-value information, and the apparatus includes:

[0119] a selection module configured to select, according to the LSMT information in the key partition, a target storage partition with a highest invalid information rate from the plurality of storage partitions included in the storage unit, wherein the invalid information rate is used to represent a proportion of invalid key-value information in total key-value information in the storage partition;

[0120] a screening module configured to detect validity information corresponding to each key-value information in the target storage partition, and screen valid key-value information from a plurality of key-value information stored in the target storage partition according to the validity information corresponding to each key-value information, wherein the validity information includes valid or invalid;

[0121] The processing module is configured to store the valid key-value information into a first storage partition other than the target storage partition, and erase the plurality of key-value information stored in the target storage partition.

[0122] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information, and each index information includes key information and storage location information of key-value information corresponding to the key information; accordingly, the selecting module selects the target storage partition with the highest invalid information rate from the plurality of storage partitions according to the LSMT information in the key partition, and specifically includes: in response to compression processing of the plurality of index information included in the LSMT information, recording the deleted key information in the plurality of index information to obtain statistical information; for each storage partition in the storage unit, determining the number of invalid key-value information in the storage partition according to the statistical information, wherein the invalid key-value information is the key-value information corresponding to the deleted key information; taking the ratio of the number of invalid key-value information in the storage partition to the total number of key-value information stored in the storage partition as the invalid information rate of the storage partition, and selecting the target storage partition with the highest invalid information rate from the plurality of storage partitions.

[0123] According to one or more embodiments of the present disclosure, the LSMT information includes a plurality of index information, and each index information includes key information and storage location information of key-value information corresponding to the key information; accordingly, the selecting module selects the target storage partition with the highest invalid information rate from the plurality of storage partitions according to the LSMT information in the key partition, and specifically includes: in response to compression processing of the plurality of index information included in the LSMT information, recording the deleted key information in the plurality of index information to obtain statistical information; for each storage partition in the storage unit, determining the number of invalid key-value information in the storage partition according to the statistical information, wherein the invalid key-value information is the key-value information corresponding to the deleted key information; taking the ratio of the number of invalid key-value information in the storage partition to the total number of key-value information stored in the storage partition as the invalid information rate of the storage partition, and selecting the target storage partition with the highest invalid information rate from the plurality of storage partitions.

[0124] According to one or more embodiments of the present disclosure, the type of the storage partition includes a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein the hotness value of the key-value information stored in the first hotness level storage partition is greater than the hotness value of the key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than the hotness value of the key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent the update times of the key information corresponding to the key-value information.

[0125] Correspondingly, the apparatus further comprises a verification module; the verification module is configured to determine the type of the target storage partition; if the type of the target storage partition is the first hotness level storage partition or the second hotness level storage partition, determine whether the key identifier in the key information is the key identifier of the latest version in the target storage partition; if yes, determine that the validity information corresponding to the key-value information is valid; if not, determine that the validity information corresponding to the key-value information is invalid; if the type of the target storage partition is the third hotness level storage partition, determine the key identifier of the latest version corresponding to the key in the key information from the plurality of index information.

[0126] According to one or more embodiments of the present disclosure, the processing module is configured to migrate the valid key-value information to a first storage partition other than the target storage partition, specifically comprising: determining a first storage partition having the same type as the target storage partition from a plurality of storage partitions other than the target storage partition; and migrating the valid key-value information to the first storage partition.

[0127] According to one or more embodiments of the present disclosure, the LSMT information comprises a plurality of index information; the apparatus further comprises an updating module; the updating module is configured to, for each valid key-value information, acquire target index information corresponding to the valid key-value information from the plurality of index information; determine migration path information of the valid key-value information from the target storage partition to the first storage partition, and add the migration path information to the target index information to obtain new index information corresponding to the valid key-value information.

[0128] According to one or more embodiments of the present disclosure, the key-value storage system further comprises hotness set information, the hotness set information comprising an average update number of key information and an update number of key information corresponding to each of the plurality of index information.

[0129] The apparatus further comprises a writing module; the writing module is configured to, in response to receiving key-value information to be written, acquire, from the hotness set information, an update number of key information in the key-value information, a minimum update number of key information in the hotness set information, and an average update number of key information; determine a hotness value level of the key-value information according to the update number, the average update number, and the minimum update number; determine a second storage partition matching the hotness value level from the plurality of storage partitions; and store the key-value information into the second storage partition.

[0130] According to one or more embodiments of the present disclosure, the types of the storage partitions include a first hotness level storage partition, a second hotness level storage partition, and a third hotness level storage partition, wherein a hotness value of key-value information stored in the first hotness level storage partition is greater than a hotness value of key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than a hotness value of key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent an update frequency of key information corresponding to the key-value information.

[0131] Accordingly, the writing module determines a hotness value level of the key-value information according to the update frequency, the average update frequency, and the minimum update frequency, and determines a second storage partition matching the hotness value level from the plurality of storage partitions, and specifically includes: if the update frequency is greater than the average update frequency, determining that the hotness value level of the key-value information is a first hotness level, and determining a first hotness level storage partition from the plurality of storage partitions as the second storage partition; if the update frequency is less than or equal to the average update frequency and greater than the minimum update frequency, determining that the hotness value level of the key-value information is a second hotness level, and determining a second hotness level storage partition from the plurality of storage partitions as the second storage partition; and if the update frequency is less than or equal to the minimum update frequency, determining that the hotness value level of the key-value information is a third hotness level, and determining a third hotness level storage partition from the plurality of storage partitions as the second storage partition.

[0132] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including: a processor, and a memory connected to the processor in communication;

[0133] The memory stores computer execution instructions;

[0134] The processor executes the computer execution instructions stored in the memory to implement the information processing method as described in the above first aspect and various possible designs of the first aspect.

[0135] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the information processing method as described in the above first aspect and various possible designs of the first aspect is implemented.

[0136] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, and when a processor executes the computer program, the information processing method as described in the above first aspect and various possible designs of the first aspect is implemented.

[0137] The above description is only preferred embodiments of the present disclosure and a description of the principles of the technology applied. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present disclosure (but not limited to) can be formed.

[0138] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination.

[0139] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. An information processing method characterized by comprising: The application is applied to a key-value storage system with key-value separation, a storage unit in the key-value storage system comprises a key partition and a plurality of storage partitions, wherein the key partition is used for storing log-structured merge tree (LSMT) information, and the storage partitions are used for storing key-value information; the method comprises the following steps: selecting a target storage partition with the highest invalid information rate from the plurality of storage partitions comprised in the storage unit according to the LSMT information in the key partition, wherein the invalid information rate is used for representing the proportion of invalid key-value information in total key-value information in the storage partition; detecting validity information corresponding to each key-value information in the target storage partition, and screening valid key-value information from the plurality of key-value information stored in the target storage partition according to the validity information corresponding to each key-value information, wherein the validity information comprises valid or invalid; storing the valid key-value information to a first storage partition other than the target storage partition, and erasing the plurality of key-value information stored in the target storage partition.

2. The method of claim 1, wherein, The LSMT information comprises a plurality of index information, and the index information comprises key information and storage location information of key-value information corresponding to the key information; Accordingly, the step of selecting a target storage partition with the highest invalid information rate from the plurality of storage partitions comprised in the storage unit according to the LSMT information in the key partition comprises the following steps: in response to compression processing of the plurality of index information comprised in the LSMT information, recording deleted key information in the plurality of index information to obtain statistical information; for each storage partition in the storage unit, determining the number of invalid key-value information in the storage partition according to the statistical information, wherein the invalid key-value information is key-value information corresponding to the deleted key information; taking the ratio of the number of invalid key-value information in the storage partition to the total number of key-value information stored in the storage partition as the invalid information rate of the storage partition, and selecting a target storage partition with the highest invalid information rate from the plurality of storage partitions.

3. The method of claim 1, wherein, The LSMT information comprises a plurality of index information, and the index information comprises key information and storage location information of key-value information corresponding to the key information; Accordingly, the step of detecting validity information corresponding to each key-value information in the target storage partition comprises the following steps: for each key-value information, determining key information corresponding to the key-value information, wherein the key information comprises a key and a key identifier, and the key identifier is used for representing the version corresponding to the key; determining the key identifier of the latest version of the key corresponding to the key in the key information from the plurality of index information, and judging whether the key identifier in the key information is same as the key identifier of the latest version; if same, determining that the validity information corresponding to the key-value information is valid, and if different, determining that the validity information corresponding to the key-value information is invalid.

4. The method of claim 3, wherein, The type of the storage partition includes a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein the hotness value of the key-value information stored in the first hotness level storage partition is greater than the hotness value of the key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than the hotness value of the key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent the update times of the key information corresponding to the key-value information; Correspondingly, before the step of determining the latest version of the key identifier corresponding to the key in the key information from the plurality of index information, the method further comprises: determining the type of the target storage partition; if the type of the target storage partition is the first hotness level storage partition or the second hotness level storage partition, determining whether the key identifier in the key information is the latest version of the key identifier in the target storage partition, if yes, determining that the validity information corresponding to the key-value information is valid, if not, determining that the validity information corresponding to the key-value information is invalid; if the type of the target storage partition is the third hotness level storage partition, performing the step of determining the latest version of the key identifier corresponding to the key in the key information from the plurality of index information.

5. The method of claim 4, wherein, The step of storing the valid key-value information into the first storage partition other than the target storage partition comprises: determining a first storage partition having the same type as the target storage partition from a plurality of storage partitions other than the target storage partition; storing the valid key-value information into the first storage partition.

6. The method of claim 1, wherein, The LSMT information includes a plurality of index information, and after the step of storing the valid key-value information into the first storage partition other than the target storage partition, the method further comprises: for each valid key-value information, obtaining target index information corresponding to the valid key-value information from the plurality of index information; determining migration path information of the valid key-value information from the target storage partition to the first storage partition, and adding the migration path information to the target index information to obtain new index information corresponding to the valid key-value information.

7. The method of claim 1, wherein, The key-value storage system further comprises hotness set information, the hotness set information including the average update times of the key information and the update times of the key information corresponding to each of the plurality of index information; and the method further comprises: in response to receiving the key-value information to be written, obtaining the update times of the key information corresponding to the key-value information, the minimum update times of the key information and the average update times of the key information from the hotness set information; determining the hotness value level of the key-value information according to the update times, the average update times and the minimum update times, and determining a second storage partition matching the hotness value level from the plurality of storage partitions; storing the key-value information into the second storage partition.

8. The method of claim 7, wherein, The types of the storage partitions include a first hotness level storage partition, a second hotness level storage partition and a third hotness level storage partition, wherein the hotness value of the key-value information stored in the first hotness level storage partition is greater than the hotness value of the key-value information stored in the second hotness level storage partition, and the hotness value of the key-value information stored in the second hotness level storage partition is greater than the hotness value of the key-value information stored in the third hotness level storage partition, wherein the hotness value is used to represent the update times of the key information corresponding to the key-value information; Accordingly, the determining the hotness value level of the key-value information according to the update times, the average update times and the minimum update times, and determining the second storage partition matching the hotness value level from the plurality of storage partitions, comprises: If the update times are greater than the average update times, it is determined that the hotness value level of the key-value information is the first hotness level, and the first hotness level storage partition is determined as the second storage partition from the plurality of storage partitions; If the update times are less than or equal to the average update times and greater than the minimum update times, it is determined that the hotness value level of the key-value information is the second hotness level, and the second hotness level storage partition is determined as the second storage partition from the plurality of storage partitions; If the update times are less than or equal to the minimum update times, it is determined that the hotness value level of the key-value information is the third hotness level, and the third hotness level storage partition is determined as the second storage partition from the plurality of storage partitions.

9. An information processing apparatus, comprising: The application is applied to a key-value separation key-value storage system, a storage unit in the key-value storage system comprises a key partition and a plurality of storage partitions, wherein the key partition is used to store log structure merge tree (LSMT) information, and the storage partitions are used to store key-value information, and the device comprises: A selection module is configured to select a target storage partition with the highest invalid information rate from the plurality of storage partitions included in the storage unit according to the LSMT information in the key partition, wherein the invalid information rate is used to represent the proportion of invalid key-value information in total key-value information in the storage partition; A screening module is configured to detect validity information corresponding to each key-value information in the target storage partition, and screen valid key-value information from the plurality of key-value information stored in the target storage partition according to the validity information corresponding to each key-value information, wherein the validity information comprises valid or invalid; A processing module is configured to transfer the valid key-value information to a first storage partition other than the target storage partition, and erase the plurality of key-value information stored in the target storage partition.

10. An electronic device, comprising: It comprises: A processor and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to realize the information processing method in any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the information processing method in any one of claims 1 to 8 is realized.

12. A computer program product, characterised in that, The computer program includes a computer program that, when executed by a processor, implements the information processing method according to any one of claims 1 to 8.

Images