Data processing apparatus and method, and device and computer-readable storage medium

Abstract

The present application belongs to the technical field of computers. Disclosed are a data processing apparatus and method, and a device and a computer-readable storage medium. The data processing apparatus comprises: a first module and a second module, wherein the first module is used for receiving an identification part, and the second module is used for querying, on the basis of the received identification part, compressed metadata fully stored in a memory, so as to obtain a data part corresponding to the received identification part, and returning the obtained data part. The compressed metadata is obtained by compressing a corresponding identification part and data part in the metadata. In the present application, by means of compressed metadata fully stored in a memory, the metadata can be read, so that a data part corresponding to a received identification part is obtained, which reduces the reading overheads of the metadata, and also improves the reading efficiency of the metadata.

Description

Data processing device, method, apparatus and computer-readable storage medium

[0001] This application claims priority to Chinese patent application No. 202311683843.1 filed on December 8, 2023, entitled “Device, method, apparatus and computer-readable storage medium for processing data,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a data processing device, method, equipment and computer-readable storage medium. Background Art

[0003] In the field of computer technology, various businesses exist, each generating a large amount of business data. Consequently, storage services are required for this data. These services involve writing business data to disk for storage and generating metadata corresponding to the business data. This metadata identifies the business data and facilitates subsequent access. Since metadata itself is a type of data, it also requires processing.

[0004] Summary of the Invention

[0005] The present application provides a data processing device, method, equipment and computer-readable storage medium to implement metadata processing. The technical solution provided by the present application includes the following aspects.

[0006] In a first aspect, a device for processing data is provided. The device includes a first module and a second module. The first module is configured to receive an identification portion. The second module is configured to query compressed metadata based on the received identification portion, obtain a data portion corresponding to the received identification portion, and return the obtained data portion. The compressed metadata is obtained by compressing the corresponding identification portion and data portion in the metadata, and the compressed metadata is fully stored in memory.

[0007] If metadata is not compressed, the space required for metadata may be larger than the space available in memory, making it impossible to write the entire metadata to memory. Therefore, this application compresses metadata to obtain compressed metadata, which includes a compressed identification portion and a compressed data portion. The space required for compressed metadata can be smaller than the space available in memory, allowing the entire compressed metadata to be stored in memory.

[0008] Therefore, when reading metadata, the application retrieves and returns the corresponding data by querying the fully stored compressed metadata in memory based on the received identifier. In other words, the application can read metadata simply by accessing memory, without having to access the disk and read the metadata from the disk one level at a time. This reduces metadata reading overhead and improves metadata reading efficiency, potentially increasing reading performance by more than double.

[0009] In one possible implementation, the compressed identification portion corresponds to a first pointer, which is used to point to the compressed data portion. A second module is configured to query the compressed identification portion based on the received identification portion to obtain a reference identification portion, and obtain the data portion corresponding to the received identification portion based on the first pointer corresponding to the reference identification portion, so that the reference identification portion matches the received identification portion.

[0010] Among them, after querying and obtaining the matching reference identification part based on the received identification part, the reference identification part belongs to the compressed identification part, and thus the reference identification part corresponds to a first pointer. Then, according to the indication of the first pointer, the compressed data part can be quickly obtained to obtain the data part corresponding to the received identification part, so that the efficiency of obtaining the data part is higher.

[0011] In one possible implementation, the apparatus further includes a third module configured to write the service data and a write-ahead log (WAL) to a disk and generate metadata indicating the service data. The WAL corresponds to the metadata, the first module and the second module are deployed on a file system, and the third module is deployed outside the file system.

[0012] Because the third module is deployed outside the file system, it does not need to write the business data and the WAL corresponding to the metadata to disk through the file system, thus avoiding the generation and writing of the file system WAL and the disk WAL. In other words, after the third module completes the writing of the business data and the WAL corresponding to the metadata, the metadata writing process can be directly executed, improving the efficiency of metadata writing and improving writing performance by more than 100%.

[0013] In one possible implementation, the device further includes: a fourth module, configured to write the metadata to the disk, obtain the metadata and WAL in the disk in parallel when the metadata in the memory is cleared, filter the metadata and WAL in the disk obtained in parallel according to the priorities of the metadata and WAL in the disk, and restore the metadata in the memory based on the filtering results.

[0014] The memory adopts a non-persistent storage mode, so the metadata in the memory may be cleared, and the metadata needs to be rebuilt in the memory. In this application, the metadata and WAL in the disk are obtained in parallel to rebuild the metadata in the memory. The parallel acquisition method is more efficient, which improves the efficiency of metadata reconstruction and can improve the reconstruction performance by more than 1 times. Among them, the metadata and WAL in the disk obtained in parallel may be repeated, that is, different metadata and WAL may be used to rebuild the same metadata in the memory (that is, metadata including the same identification part). Therefore, this application filters the metadata and WAL in the disk obtained in parallel in combination with priority to achieve metadata reconstruction in the memory according to the results obtained by filtering.

[0015] In one possible implementation, the metadata in the disk includes metadata of multiple partitions, and the metadata of each partition includes multiple groups of metadata. The priority of any group of metadata is determined according to the position of any group in the partition where any group is located and the position of the partition where any group is located in multiple partitions.

[0016] For different partitions, the later the partition is located, the later the metadata was written, and the newer the metadata is. For the same partition, the later the metadata is located, the later it was written, and the newer the metadata is. Newer metadata should have a higher priority, allowing it to replace lower-priority metadata and ensure the accuracy of subsequently reconstructed metadata.

[0017] In one possible implementation, the business data in the disk includes multiple data groups, the WAL includes multiple WALs corresponding to the multiple data groups, and the priority of any WAL is determined according to the position of the data group corresponding to any WAL in the multiple data groups.

[0018] For different data groups, the later the data group is located, the later it was written and the newer the data group is. Since WALs correspond one-to-one to data groups, the newer the data group, the newer the WAL. Furthermore, the newer the WAL, the higher its priority should be, so that it can replace lower-priority WALs and ensure the accuracy of the metadata subsequently rebuilt.

[0019] In one possible implementation, the metadata in the disk includes metadata of multiple partitions, and the multiple partitions include a reference partition. The device also includes: a fifth module, which is used to obtain the metadata corresponding to the reference partition from the memory according to the identifier of the reference partition when the data volume of the metadata of the reference partition exceeds a threshold, and write the obtained metadata to the reference partition to overwrite the metadata whose data volume exceeds the threshold.

[0020] This implementation eliminates the need to read metadata from disk, merge it, and then rewrite it. Instead, the metadata corresponding to the reference partition in memory is directly written to the reference partition on disk, ensuring that the amount of metadata for the reference partition on disk does not exceed the threshold. This reduces the overhead of the merge process and improves its efficiency, potentially increasing merge performance by more than double.

[0021] In one possible implementation, the compressed data portion includes multi-level characters, a second pointer corresponding to a lower-level character in two adjacent levels of characters, the second pointer being used to point to a higher-level character in the two adjacent levels, and the first pointer being used to point to a lowest-level character in the multi-level characters. A second module is configured to determine the lowest-level character pointed to by the first pointer corresponding to the reference identification portion, and to search upwards level by level based on the second pointer corresponding to the lowest-level character until a highest-level character in the multi-level characters is obtained, and to use the data portion consisting of the lowest-level character to the highest-level character as the data portion corresponding to the received identification portion.

[0022] The data portion may be a string of characters, and different strings may overlap. Therefore, the different strings may be comprehensively compressed so that the compressed data portion includes multiple levels of characters. After receiving the identification portion, a reference identification portion is obtained by querying the compressed identification portion based on the received identification portion. The lowest-level character in the multiple levels of characters is determined based on a first pointer corresponding to the reference identification portion. The highest-level character is obtained by querying upward based on a second pointer corresponding to the lowest-level character. The data portion is then composed of the characters from the lowest level to the highest level. The data portion thus composed is the data portion corresponding to the received identification portion.

[0023] In one possible implementation, the compressed data portion includes a data portion group obtained by compressing multiple data portions. The reference identification portion further corresponds to a data portion identifier, and the data portion identifier is used to indicate one of the multiple data portions. A second module is configured to determine the data portion group pointed to by the first pointer corresponding to the reference identification portion, decompress the data portion group to obtain multiple data portions, and select, from the multiple data portions, a data portion indicated by the data portion identifier corresponding to the reference identification portion as the data portion corresponding to the received identification portion.

[0024] After receiving the identification part, the reference identification part is obtained by querying in the compressed identification part according to the received identification part, the data part group is determined according to the first pointer corresponding to the reference identification part, and multiple data parts are obtained. Then, a data part is obtained from the multiple data parts according to the data part identifier corresponding to the reference identification part. The obtained data part is the data part corresponding to the received identification part.

[0025] In one possible implementation, the compressed data portion is index information, which is used to point to the data portion on the disk. The second module is configured to determine the index information pointed to by the first pointer corresponding to the reference identification portion, and use the data portion on the disk pointed to by the index information as the data portion corresponding to the received identification portion.

[0026] In this implementation method, after receiving the identification part, the reference identification part is obtained by querying in the compressed identification part according to the received identification part, the index part is determined according to the first pointer corresponding to the reference identification part, and the data part in the disk is obtained according to the index part. The obtained data part is the data part corresponding to the received identification part.

[0027] In one possible implementation, the metadata is hot metadata, which is metadata used to indicate hot business data, or the hot metadata is metadata located in a working volume, where a working volume is a volume on a disk whose access frequency exceeds a threshold.

[0028] That is, when metadata is hot, it is compressed and stored in full in memory. When metadata is cold, it is not compressed and is stored on disk instead of in memory. This approach further reduces the memory space occupied by compressed metadata.

[0029] In one possible implementation, the multi-level characters are located in a memory space in a memory, the memory space including at least one memory page, and the memory space having a starting address. A second pointer corresponding to a non-highest-level character in the multi-level characters includes an offset of a character in the previous level of the non-highest-level character relative to the starting address; and a second pointer corresponding to a lowest-level character in the multi-level characters also includes the starting address.

[0030] In this implementation, the second pointer is compressed, which reduces the memory space occupied by the second pointer, so that more space in the memory can be used to store the compressed metadata, ensuring the full storage of the compressed metadata.

[0031] In a second aspect, a method for processing data is provided. In this method, an identification portion is received. Compressed metadata is queried based on the received identification portion to obtain a data portion corresponding to the received identification portion. The obtained data portion is returned. The compressed metadata is obtained by compressing the corresponding identification portion and data portion in the metadata, and the compressed metadata is fully stored in memory.

[0032] In one possible implementation, the compressed identification portion corresponds to a first pointer, and the first pointer is used to point to the compressed data portion. Querying the compressed metadata based on the received identification portion to obtain the data portion corresponding to the received identification portion includes: querying the compressed identification portion based on the received identification portion to obtain a reference identification portion, where the reference identification portion matches the received identification portion; and obtaining the data portion corresponding to the received identification portion based on the first pointer corresponding to the reference identification portion.

[0033] In a possible implementation, the method further includes: writing the business data and the WAL to a disk, generating metadata for indicating the business data, and the WAL corresponds to the metadata.

[0034] In one possible implementation, the method further includes: writing the metadata to disk, and when the metadata in the memory is cleared, obtaining the metadata and WAL in the disk in parallel, filtering the metadata and WAL in the disk obtained in parallel according to the priorities of the metadata and WAL in the disk, and restoring the metadata in the memory based on the filtering results.

[0035] In one possible implementation, the metadata in the disk includes metadata of multiple partitions, and the metadata of each partition includes multiple groups of metadata. The priority of any group of metadata is determined according to the position of any group in the partition where any group is located and the position of the partition where any group is located in multiple partitions.

[0036] In one possible implementation, the business data in the disk includes multiple data groups, the WAL includes multiple WALs corresponding to the multiple data groups, and the priority of any WAL is determined according to the position of the data group corresponding to any WAL in the multiple data groups.

[0037] In one possible implementation, the metadata in the disk includes metadata of multiple partitions, and the multiple partitions include a reference partition. The method also includes: when the data volume of the metadata of the reference partition exceeds a threshold, obtaining the metadata corresponding to the reference partition from the memory according to the identifier of the reference partition, and writing the obtained metadata to the reference partition to overwrite the metadata whose data volume exceeds the threshold.

[0038] In one possible implementation, the compressed data portion includes multi-level characters, a second pointer corresponding to a lower-level character in two adjacent levels of characters, the second pointer being used to point to a higher-level character in the two adjacent levels of characters, and the first pointer being used to point to a lowest-level character in the multi-level characters. Obtaining the data portion corresponding to the received identification portion based on the first pointer corresponding to the reference identification portion includes: determining the lowest-level character pointed to by the first pointer corresponding to the reference identification portion, searching upwards level by level based on the second pointer corresponding to the lowest-level character until a highest-level character in the multi-level characters is obtained, and using the data portion consisting of the lowest-level character to the highest-level character as the data portion corresponding to the received identification portion.

[0039] In one possible implementation, the compressed data portion includes a data portion group obtained by compressing multiple data portions, and the reference identification portion further corresponds to a data portion identifier, the data portion identifier being used to indicate one of the multiple data portions. Obtaining the data portion corresponding to the received identification portion based on a first pointer corresponding to the reference identification portion includes: determining the data portion group pointed to by the first pointer corresponding to the reference identification portion, decompressing the data portion group to obtain multiple data portions, and selecting, from the multiple data portions, a data portion indicated by the data portion identifier corresponding to the reference identification portion as the data portion corresponding to the received identification portion.

[0040] In one possible implementation, the compressed data portion is index information, and the index information is used to point to a data portion on a disk. Obtaining the data portion corresponding to the received identification portion based on a first pointer corresponding to the reference identification portion includes: determining the index information pointed to by the first pointer corresponding to the reference identification portion, and using the data portion on the disk pointed to by the index information as the data portion corresponding to the received identification portion.

[0041] In one possible implementation, the metadata is hot metadata, which is metadata used to indicate hot business data, or the hot metadata is metadata located in a working volume, where a working volume is a volume on a disk whose access frequency exceeds a threshold.

[0042] In one possible implementation, the multi-level characters are located in a memory space in a memory, the memory space including at least one memory page, and the memory space having a starting address. A second pointer corresponding to a non-highest-level character in the multi-level characters includes an offset of a character in the previous level of the non-highest-level character relative to the starting address; and a second pointer corresponding to a lowest-level character in the multi-level characters also includes the starting address.

[0043] In a third aspect, a device for processing data is provided, which includes a memory and a processor; at least one computer instruction is stored in the memory, and the at least one computer instruction is loaded and executed by the processor, so that the device for processing data implements the method for processing data provided by the above-mentioned second aspect or any possible implementation method of the second aspect.

[0044] Optionally, there are one or more processors and one or more memories.

[0045] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0046] In a fourth aspect, a computer program or computer program product is provided, which includes: computer instructions, which, when executed by a computer, enable the computer to execute the method for processing data provided by the above-mentioned second aspect or any possible implementation of the second aspect.

[0047] In a fifth aspect, a computer-readable storage medium is provided, which stores computer instructions. When the computer instructions are run on a computer, the method for processing data provided by the above-mentioned second aspect or any possible implementation of the second aspect is executed by the computer.

[0048] In a sixth aspect, a chip is provided, comprising a processor for calling and executing instructions stored in a memory from a memory, so that a computer equipped with the chip executes the method for processing data provided in the second aspect or any possible implementation of the second aspect.

[0049] In the seventh aspect, another chip is provided, comprising: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, a computer equipped with the chip executes the method for processing data provided by the second aspect or any possible implementation of the second aspect.

[0050] Among them, the technical effects achieved by the technical solutions provided by the second to seventh aspects of this application and the corresponding possible implementation methods can be found in the above description of the technical effects achieved by the technical solutions provided by the first aspect and the corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG1 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0052] FIG2 is a schematic structural diagram of a compressed identification portion provided in an embodiment of the present application;

[0053] FIG3 is a schematic diagram of a metadata reading process provided in an embodiment of the present application;

[0054] FIG4 is a schematic diagram of another metadata reading process provided in an embodiment of the present application;

[0055] FIG5 is a schematic diagram of another metadata reading process provided in an embodiment of the present application;

[0056] FIG6 is a schematic diagram of a memory and a disk provided in an embodiment of the present application;

[0057] FIG7 is a schematic diagram of a metadata writing process provided in an embodiment of the present application;

[0058] FIG8 is a schematic diagram of a metadata reconstruction process provided in an embodiment of the present application;

[0059] FIG9 is a schematic diagram of a metadata merging process provided in an embodiment of the present application;

[0060] FIG10 is a flowchart of a method for processing data provided in an embodiment of the present application;

[0061] FIG11 is a schematic structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0063] With the rapid development of digital technology, the amount of business data generated by businesses has exploded. As the foundation of digital technology, storage services have garnered significant attention from internet and cloud vendors. Providing storage services for business data requires writing the data to disk for storage and generating metadata corresponding to the data. This metadata identifies the data and facilitates subsequent access to the stored data. Since metadata itself is a form of data, it also requires processing.

[0064] In the process of processing metadata, the related technology first writes the metadata into the memory. After the memory is full, the metadata is flushed from the memory to the disk. The disk is divided into multiple levels. After the metadata is flushed from the memory to the disk, the metadata is first stored in the first level of the disk. When the first level of the disk is full, the metadata is written from the first level of the disk to the second level of the disk, and so on. Accordingly, when it is necessary to read the stored metadata, the memory is first accessed. If the metadata to be read is read in the memory, the business data is obtained according to the indication of the metadata. If the metadata to be read is not read in the memory, the first level, second level, third level, and so on of the disk are accessed in sequence until the metadata to be read is read at a certain level of the disk, and the business data is obtained according to the indication of the metadata.

[0065] However, due to the limited storage space in memory, the amount of metadata that can be written to memory is also limited. This makes it unlikely that the required metadata will be found in memory. Consequently, it is often necessary to continue accessing the disk to read the metadata level by level, or even traverse all levels of the disk. This processing method is expensive and inefficient.

[0066] The present application provides a device for processing data. The device can be in software form. The software-based device can be embedded in other software with data processing requirements or applied to hardware products. The present application does not limit this. As shown in Figure 1, the device includes a first module 101 and a second module 102.

[0067] The first module 101 is configured to receive an identification portion. The second module 102 is configured to query the compressed metadata based on the received identification portion, obtain the data portion corresponding to the received identification portion, and return the obtained data portion. The compressed metadata is obtained by compressing the corresponding identification portion and data portion in the metadata, and the entire compressed metadata is stored in memory.

[0068] The metadata includes a corresponding identification part and a data part, the identification part is used to uniquely identify the data part, and the data part is used to indicate the storage location of the business data. Exemplarily, in the case of block storage, the business data is stored in the form of data blocks. For example, the business data can be a data block aligned in units of 4 kilobytes (KB). 4KB is only an example and is not limited to this in the embodiments of the present application. In addition, the identification part includes a key (key, K), and the data part includes a value (value, V). V is used to indicate the storage location of the data block, so the metadata is also called a KV pair. Accordingly, the device for processing data can be RocksDB, which is a database (data base, DB) for managing KV pairs. Alternatively, in the case of object storage, the business data is stored in the form of an object, the identification part includes an identification obtained by mapping the object, and the data part includes information for indicating the storage location of the object. Alternatively, in the case of file storage, the business data is stored in the form of a file, the identification part includes an identification obtained by mapping the file, and the data part includes information for indicating the storage location of the file. Exemplarily, the manner of mapping an object or a file includes but is not limited to memory mapping (MMAP). The embodiment of the present application does not limit the manner of mapping.

[0069] If metadata is not compressed, it will take up a lot of space, but the memory space is limited, making it difficult to write the entire metadata into the memory. Therefore, in the embodiments of the present application, the metadata is compressed to obtain compressed metadata, which includes a compressed identification portion and a compressed data portion. Compared to metadata, compressed metadata requires less space, and even if the memory space is limited, the compressed metadata can be written into the memory in full, so that the entire compressed metadata is stored in the memory.

[0070] Accordingly, when a user needs to obtain the data portion to determine the storage location of the business data, the user provides the identification portion they hold. The identification portion is then received, and the compressed metadata stored in the memory is queried based on the received identification portion. The data portion corresponding to the received identification portion is then returned to the user. This completes the metadata reading process. Since the data portion indicates the storage location of the business data, the user can determine the storage location of the business data based on the received data portion and obtain the business data from that storage location.

[0071] It is precisely because the full amount of compressed metadata is stored in the memory that the embodiment of the present application can access the memory to realize the reading of metadata, without having to continue to access the disk and read the metadata in the disk step by step. Therefore, not only the overhead required to read the metadata that needs to be read is reduced and the read amplification is reduced, but also the efficiency of reading metadata is improved and the rapid reading of metadata is achieved, which is conducive to the rapid query of business data. Among them, the read amplification is the ratio of the first data amount to the second data amount, the second data amount is the data amount of the metadata that needs to be read, and the first data amount is the total data amount that needs to be read in order to read the metadata that needs to be read. Since the embodiment of the present application does not need to read the metadata in the disk step by step, the first data amount is reduced, thereby reducing the read amplification.

[0072] In an exemplary embodiment, the compressed identification portion corresponds to a first pointer, which is used to point to the compressed data portion. Second module 102 is configured to query the compressed identification portion based on the received identification portion to obtain a reference identification portion, and obtain the data portion corresponding to the received identification portion based on the first pointer corresponding to the reference identification portion, so that the reference identification portion matches the received identification portion.

[0073] After querying the compressed identification part according to the received identification part, the reference identification part can be obtained, and the reference identification part is one of the compressed identification parts. Since the compressed identification part corresponds to the first pointer, the reference identification part also corresponds to the first pointer, so that the compressed data part can be obtained according to the indication of the first pointer. For example, the first pointer includes the starting address and length of the compressed data part, and the compressed data part can be obtained according to the starting address and length. The compressed data part corresponds to the reference identification part. Since the reference identification part matches the received identification part, the compressed data part also corresponds to the received identification part, and the data part can be obtained according to the compressed data part, and the obtained data part is returned as the data part corresponding to the received identification part.

[0074] In an exemplary embodiment, the compressed identification portion includes multiple levels of first characters. The first character at the higher level of two adjacent levels of first characters corresponds to a third pointer, which is used to point to the first character at the lower level of the two adjacent levels. For example, the third pointer corresponding to the higher level first character includes the starting address and length of the lower level first character. The first pointer corresponds to the lowest level first character of the multiple levels of first characters, and the first pointer is used to point to the compressed data portion.

[0075] Among them, the identification part is a character string, and the identification parts in different metadata (i.e., different character strings) may have a high degree of overlap. For example, referring to Figure 2, only the last 4 bits of the 8 identification parts are different, and the other parts are the same. Therefore, the identification parts in different metadata can be compressed so that the compressed identification parts include multiple levels of first characters. Exemplarily, the multiple levels of first characters can form a tree structure as shown in Figure 2, such as a prefix tree, a multi-granularity tree and other structures, or can form other structures, which are not limited in the embodiments of the present application. In Figure 2, in order to avoid redundancy, the third pointer corresponding to the character 89 and the first pointer corresponding to some of the first characters in the lowest-level first character are not shown. The partial first characters refer to 01, 02, 03 and 04 corresponding to 89, and 01, 02 and 03 corresponding to 90.

[0076] Based on the multiple levels of first characters included in the compressed identification portion, when searching the compressed identification portion based on the received identification portion, the first character at the highest level in the multiple levels of first characters is first obtained. Then, based on the third pointer corresponding to the highest level first character, the search is continued downwards level by level until the lowest level first character in the multiple levels of first characters is obtained. The reference identification portion is formed from the highest level first character to the lowest level first character. In other words, in the compressed identification portion, the search direction is from top to bottom, i.e., from the highest level first character to the lowest level first character.

[0077] For example, referring to Figure 2, taking the received identification part 90123456789abcdef012345679004 as an example, when searching in the multi-level first characters, 90123456789abcdef (that is, the first character of the highest level in the multi-level first characters) that matches the received identification part is first obtained, and according to the third pointer corresponding to 90123456789abcdef, 01 that matches the received identification part is obtained, and according to the third pointer corresponding to 01, 23 that matches the received identification part is obtained, and according to the third pointer corresponding to 23, 45 that matches the received identification part is obtained, and according to the third pointer corresponding to 45, 67 that matches the received identification part is obtained. The third pointer corresponding to 67 yields 89 and 90. Since 89 does not match the received identification portion, while 90 does, the third pointer corresponding to 90 yields 01, 02, 03, and 04, resulting in 04 (i.e., the lowest-level first character in the multi-level first character sequence) that matches the received identification portion. 90123456789abcdef through 04 form the reference identification portion, which matches the received identification portion. In other words, the reference identification portion and the received identification portion are identical.

[0078] For another example, referring to Figure 2 , taking the received identification part as 90123456789abcdef0123456790 as an example, after obtaining 01, 02, 03 and 04 according to the third pointer corresponding to 90, it is considered that these four characters all match the received identification part, and then 90123456789abcdef to 01, 02, 03 and 04 respectively constitute four reference identification parts, which match the received identification part, that is, the reference identification part includes the received identification part.

[0079] For example, before compressing the identification portion in different metadata, an embodiment of the present application pre-partitions the memory to obtain multiple partitions (zones). Each partition is used to store a group of multi-level first characters (for example, a group of multi-level first characters is a tree structure) and the compressed data portion corresponding to the group of multi-level first characters. If the memory is not pre-partitioned, the compressed identification portion only includes a group of multi-level first characters. Then, as the metadata continues to increase, the volume of the group of multi-level first characters will increase. For example, as the number of levels increases, the number of first characters included in each level also increases. When the volume exceeds a certain threshold, the efficiency of querying based on the group of multi-level first characters is very low, and the group of multi-level first characters needs to be split, which will cause additional overhead. By dividing the memory to obtain multiple partitions, the compressed identification portion can include multiple groups of multi-level first characters. Even if the metadata continues to increase, the volume of each group of multi-level first characters is unlikely to exceed the threshold, will not affect the query efficiency, and therefore, there is no need to split it, and no additional overhead will be caused.

[0080] Exemplarily, the multi-level first characters are located in a first memory space in a memory, the first memory space includes at least one memory page, and the first memory space has a first starting address; the third pointer corresponding to the first character of a level other than the lowest level in the multi-level first characters includes: an offset of the first character of the next level of the first character of the next level relative to the first starting address; and the first pointer corresponding to the first character of the highest level in the multi-level first characters also includes: the first starting address.

[0081] For example, in Figure 2, the non-lowest-level first characters include: 90123456789abcdef, 01, 23, 45, 47, 89 and 90, and 01 is between 90123456789abcdef and 23. The highest-level first characters include: 90123456789abcdef. In this embodiment, a global pointer is not used as the third pointer, but the first starting address and offset (or only the offset) are used as the third pointer. As a result, the space occupied by the third pointer can be reduced, which is equivalent to compressing the third pointer, further saving memory space. Of course, using a global pointer as the third pointer can also be used as an implementation method of the embodiment of the present application, and the embodiment of the present application is not limited to this.

[0082] The above example illustrates one possible implementation method for obtaining a reference identifier through querying. The methods for obtaining a reference identifier through querying in the embodiments of the present application are not limited to this method. Regardless of the method used to obtain the reference identifier, the reference identifier corresponds to a first pointer. Thus, based on the first pointer corresponding to the reference identifier, the data portion corresponding to the received identifier can be obtained. For example, the methods provided in the embodiments of the present application include, but are not limited to, the following three.

[0083] In the first acquisition method, the data part is a character string, and the data parts in different metadata may overlap. Therefore, the data parts in different metadata can be compressed so that the compressed data part includes multiple levels of characters. In order to distinguish it from the first character included in the compressed identification part above, the characters included in the compressed data part are referred to as second characters in the following text. Among them, the lower-level second character in the two adjacent levels of second characters corresponds to a second pointer, and the second pointer is used to point to the higher-level second character in the two adjacent levels of second characters. For example, the second pointer corresponding to the lower-level second character includes: the starting address and length of the higher-level second character. In addition, the first pointer is used to point to a lowest-level second character in the multi-level second characters. For example, the first pointer includes the starting address and length of the lowest-level second character.

[0084] The second module 102 is configured to determine the lowest-level second character pointed to by the first pointer corresponding to the reference identification portion, and to search upwards level by level based on the second pointer corresponding to the lowest-level second character until a highest-level second character among the multiple levels of second characters is obtained. The data portion consisting of the lowest-level second character to the highest-level second character is then used as the data portion corresponding to the received identification portion. In other words, in the compressed data portion, the search direction is from bottom to top, i.e., from the lowest-level second character to the highest-level second character.

[0085] For example, referring to FIG3 , taking the reference identification portion 90123456789abcdef012345679004 as an example, based on the first pointer corresponding to the reference identification portion, the lowest-level second character yy among the multi-level second characters can be determined. Based on the second pointer corresponding to yy, ff is obtained. Based on the second pointer corresponding to ff, ee is obtained. Based on the second pointer corresponding to ee, dd is obtained. Based on the second pointer corresponding to dd, cc is obtained. Based on the second pointer corresponding to cc, bb is obtained. Based on the second pointer corresponding to bb, aa is obtained. Since aa does not correspond to a second pointer, it can be determined that aa belongs to the highest-level second character among the multi-level second characters. The data portion from the lowest-level second character yy to the highest-level second character aa is a data portion, i.e., aabbccddeeffyy. This data portion can be used as the data portion corresponding to the received identification portion.

[0086] In an exemplary embodiment, the multi-level second characters are located in a second memory space in a memory, the memory space including at least one memory page and having a second starting address. A second pointer corresponding to a second character at a level other than the highest level in the multi-level second characters includes an offset of the second character at the previous level relative to the second starting address. A second pointer corresponding to a second character at the lowest level in the multi-level second characters also includes the second starting address.

[0087] Among them, in order to facilitate the distinction from the first memory space and the first starting address above, the naming of the second memory space and the second starting address is adopted. Taking Figure 3 as an example, the non-highest-level second characters include: bb, cc, dd, ee, ff, xx and yy, and the lowest-level second characters include: xx and yy. In this implementation, a global pointer is not used as the second pointer, but the second starting address and offset (or only the offset) are used as the second pointer. In this way, the space occupied by the second pointer can be reduced, which is equivalent to compressing the second pointer, further saving memory space. Of course, using a global pointer as the second pointer can also be used as an implementation method of an embodiment of the present application, and the embodiment of the present application is not limited to this.

[0088] In addition, in addition to the above-mentioned second pointer, a fourth pointer may also exist in the compressed data portion. According to the above description, the role of the second pointer includes: in the process of obtaining the data portion corresponding to the received identification portion according to the first pointer corresponding to the reference identification portion, implementing a bottom-up query in the compressed data portion. The role of the fourth pointer may include: in other processes, implementing a top-down query in the compressed data portion. The embodiments of the present application do not limit other processes, and other processes can be set according to actual needs. In other words, the role played by the fourth pointer in the compressed data portion is similar to the role played by the third pointer in the compressed identification portion.

[0089] Exemplarily, in the compressed data portion, the second character at the higher level in two adjacent levels of second characters corresponds to a fourth pointer, and the fourth pointer is used to point to the second character at the lower level in the two adjacent levels of characters. For example, the fourth pointer corresponding to the second character at the higher level includes: the starting address and length of the second character at the lower level. In an exemplary embodiment, the embodiment of the present application can also compress the fourth pointer. For example, since the multi-level second characters are located in the second memory space, the second memory space has a second starting address. Therefore, the fourth pointer corresponding to the non-lowest level second character in the multi-level second characters can include: the offset (offset) of the second character at the next level of the non-lowest level second character relative to the second starting address. The fourth pointer corresponding to the second character at the highest level in the multi-level second characters also includes: the second starting address.

[0090] The second obtaining method is that the compressed data part is a data part group, which is obtained by compressing multiple data parts. The reference identification part also corresponds to the data part identification, and the data part identification is used to indicate one data part among the multiple data parts.

[0091] The second module 102 is used to determine the data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain multiple data parts, and among the multiple data parts, use a data part indicated by the data part identifier corresponding to the reference identification part as the data part corresponding to the received identification part.

[0092] As mentioned above, the first pointer is used to point to the compressed data portion. Since the compressed data portion in this acquisition method is a data portion group, the first pointer is used to point to the data portion group. For example, the first pointer includes: the starting address and length of the data portion group. After determining the data portion group pointed to by the first pointer, the data portion group can be decompressed in the memory to obtain multiple data portions. Among them, although decompressing the data portion group in the memory will occupy additional space in the memory, the occupied space is small and will not affect the memory's full storage of compressed metadata. Optionally, after obtaining multiple data portions by decompression and obtaining the data portion corresponding to the received identification portion based on this, the multiple data portions obtained by decompression can be deleted to prevent these data portions from continuously occupying space in the memory.

[0093] For example, referring to Figure 4 , taking the reference identifier portion 90123456789abcdef012345679004 as an example, data portion group a can be determined based on the first pointer corresponding to the reference identifier portion. Decompressing data portion group a yields data portion 1, data portion 2, and data portion 3 within data portion group a. If the data portion identifier corresponding to the reference identifier portion indicates data portion 2, data portion 2 within data portion group a is used as the data portion corresponding to the received identifier portion.

[0094] In some embodiments, the multiple data portions obtained by decompression correspond one-to-one to the multiple data portion identifiers. When determining the data portion indicated by the data portion identifier corresponding to the reference identifier portion, the multiple data portion identifiers are queried based on the data portion identifier corresponding to the reference identifier portion. When a data portion identifier is found, the data portion corresponding to the found data portion identifier is used as the data portion indicated by the data portion identifier corresponding to the reference identifier portion.

[0095] For example, still using the situation shown in Figure 4 as an example, data portion 1 corresponds to data portion identifier 00, data portion 2 corresponds to data portion identifier 01, and data portion 3 corresponds to data portion identifier 10. If the data portion identifier corresponding to the reference identifier portion is 01, then data portion 2 corresponding to 01 is used as the data portion corresponding to the received identifier portion.

[0096] In some other embodiments, the multiple data portions obtained by decompression have the same length and occupy contiguous space, and the data portion identifier corresponding to the reference identifier portion represents the order of the data portion among the multiple data portions. When determining the data portion indicated by the data portion identifier corresponding to the reference identifier portion, based on the order represented by the data portion identifier corresponding to the reference identifier portion and the aforementioned identical length, a data portion is selected from the multiple data portions as the data portion indicated by the data portion identifier corresponding to the reference identifier portion.

[0097] For example, still taking the situation shown in FIG4 as an example, data part 1, data part 2, and data part 3 are all 16 bits long, occupying bits 0 to 47, i.e., data part 1 occupies bits 0 to 15, data part 2 occupies bits 16 to 31, and data part 3 occupies bits 32 to 47. If the data part identifier corresponding to the reference identification portion is 2, then it represents the second data part among the multiple data parts. Then, the second data part among data part 1, data part 2, and data part 3, i.e., data part 2 occupying bits 16 to 31, can be used as the data part corresponding to the received identification portion.

[0098] In the third method, the compressed data portion is index information, which is used to point to the data portion on the disk. For example, the index information is the starting address and length of the data portion on the disk.

[0099] The second module 102 is configured to determine the index information pointed to by the first pointer corresponding to the reference identification portion, and use the data portion in the disk pointed to by the index information as the data portion corresponding to the received identification portion.

[0100] In this acquisition method, the compressed identification portion is stored in memory, while the data portion is stored on disk, separating the identification portion from the data portion. In the case of block storage, since the identification portion is K and the data portion is V, this is also called KV separation. As mentioned above, the first pointer is used to point to the compressed data portion. Since the compressed data portion in this acquisition method is index information, the first pointer is used to point to the index information. For example, the first pointer includes the starting address and length of the index information.

[0101] For example, referring to FIG5 , taking the reference identification portion 90123456789abcdef012345679004 as an example, based on the first pointer corresponding to the reference identification portion, index information 1 can be determined. Based on index information 1, data portion 1 on the disk can also be determined. Therefore, data portion 1 is used as the data portion corresponding to the received identification portion. Although this acquisition method requires accessing the disk to obtain the data portion, this access is performed based on the instructions of the index information. This access is targeted and does not incur significant overhead or reduce metadata reading efficiency. This is completely different from the process of accessing the disk to read metadata step by step in the aforementioned related art.

[0102] In the above, three methods of obtaining the data portion corresponding to the received identification portion are illustrated with examples for obtaining the data portion corresponding to the received identification portion according to the first pointer corresponding to the reference identification portion. The methods of obtaining in the embodiments of the present application are not limited to this. For example, the compressed data portion can also be obtained by separately compressing the data portion in a metadata (different from the first method of obtaining described above). Accordingly, the compressed data portion can be obtained according to the first pointer corresponding to the reference identification portion, and the compressed data portion is decompressed in the memory, and the decompressed data portion is directly used as the data portion corresponding to the received identification portion. The embodiments of the present application can select, use in combination, or use in turn the various methods of obtaining described above according to actual needs.

[0103] In an exemplary embodiment, the metadata is hot metadata. That is, in this embodiment of the present application, only the hot metadata is compressed, the compressed metadata is written in full to memory, and the cold metadata is not compressed or written to memory, but is instead stored on disk. For example, in this embodiment of the present application, when memory space is insufficient, this scheme of compressing only the hot metadata can be employed. When memory space is sufficient, both the hot and cold metadata are compressed, and the compressed metadata is written in full to memory.

[0104] Among them, hot metadata is metadata used to indicate hot business data. Hot business data can be business data with a high reading frequency, or business data determined based on other indicators. Alternatively, hot metadata is metadata located in a volume in working state (also called a storage volume). A volume in working state is a volume in the disk whose access frequency exceeds a threshold. For example, if the access frequency of a volume exceeds 30 times every 60 seconds, the volume can be considered to be in working state. Correspondingly, cold metadata is metadata other than hot metadata in all metadata, or metadata with a low probability of being read. Although cold metadata is not compressed or written to memory, it will result in the need to access the disk when reading cold metadata, but since the possibility of reading the cold metadata is low, the possibility of accessing the disk is also low, and the impact on reading efficiency is acceptable.

[0105] For example, embodiments of the present application can periodically and dynamically determine whether a piece of metadata is hot metadata, thereby adjusting the compressed metadata stored in the memory based on the result of the dynamic determination. For example, if a piece of metadata is converted from hot metadata to cold metadata, the compressed metadata obtained by compressing the metadata is deleted from the memory. For another example, if a piece of metadata is converted from cold metadata to hot metadata, the metadata is compressed and the compressed metadata is written to the memory.

[0106] The above mainly describes the metadata reading process. Next, other metadata processing processes are described, including but not limited to metadata writing, metadata merging, and metadata reconstruction (also known as playback).

[0107] Regarding the metadata writing process, related technologies first generate a write-ahead log (WAL) corresponding to the metadata, and then write the metadata WAL to disk through the file system. After the metadata WAL is written, the metadata is written to memory through the file system. However, when the metadata WAL is written to disk through the file system, the file system generates a file system WAL and the disk also generates a disk WAL. Therefore, the metadata WAL, the file system WAL, and the disk WAL need to be written to disk together before the metadata writing process can be executed. This not only increases the overhead of the writing process but also reduces writing efficiency.

[0108] In an exemplary embodiment, the apparatus provided by the embodiments of the present application further includes a third module configured to write the business data and the WAL to disk and generate metadata indicating the business data. The WAL corresponds to the metadata, and a WAL corresponding to each metadata is generated based on the metadata. The first module 101 and the second module 102 are deployed on the file system, and the third module is deployed outside the file system.

[0109] Since the third module is deployed outside the file system, the third module does not need to write the business data and the WAL corresponding to the metadata to the disk through the file system, but can directly write the business data and the WAL corresponding to the metadata to the disk. As a result, the above-mentioned WAL of the file system and the WAL of the disk will not be generated, and there is no need to write the WAL of the file system and the WAL of the disk to the disk, thus avoiding the superposition of different WALs. Therefore, after completing the writing of the business data and the WAL corresponding to the metadata, the writing process of the metadata can be executed, which not only reduces the overhead of the writing process, reduces write amplification, but also improves the writing efficiency. Among them, write amplification is the ratio of the third data amount to the fourth data amount, the fourth data amount is the data amount of the metadata that needs to be written, and the third data amount is the total data amount that needs to be written in order to write the metadata that needs to be written. Since the embodiment of the present application does not need to write the WAL of the file system and the WAL of the disk, the third data amount is reduced, thereby reducing write amplification.

[0110] In some implementations, the WAL is appended to the disk after the business data is written to disk. That is, the WAL is written to disk along with the business data, or the WAL is added to the end of the business data. In other implementations, if the WAL cannot be appended to the disk, the WAL can be written separately after the business data is written to disk. In this implementation, the WAL is no longer located at the end of the business data, and the disk can store different WALs continuously, or the space occupied by each WAL on the disk is continuous.

[0111] After the business data and WAL are written to the disk, the metadata writing process can be executed. In addition to compressing the metadata and writing the compressed metadata in full to the memory as described above, the incremental metadata can also be written to the memory. For example, the memory is divided into a first part and a second part. The first part is used to store the compressed metadata written in full. The first part can be divided into multiple partitions as described above. The second part is used to store the incremental metadata. When the second part is full, the metadata is flushed from the second part to the disk. In this way, not only can the disk store the metadata persistently, but the second part of the memory can also be released to facilitate the continued writing of the incremental metadata to the second part of the memory.

[0112] As shown in Figure 6, the second part of memory includes memory objects (mem tables) and read-only memory objects (immutable mem tables). After metadata is incrementally written to the second part of memory, it is temporarily stored in the mem tables. When one mem table is full, it switches to another mem table, and the full mem table becomes the immutable mem table. When the immutable mem table is also full, the metadata is flushed from the immutable mem table to disk.

[0113] Continuing to refer to Figure 6, the disk includes multiple sets of metadata, one set of metadata is, for example, a sorted string table (SST), and SST is a block-based file format, which is not limited in this embodiment of the present application. In the case where the disk includes SST, the metadata is flushed to the disk, that is, the metadata is stored in the disk as SST. For example, the disk may include multiple partitions, each partition including multiple sets of metadata, such as multiple SSTs. Among them, different partitions can each independently correspond to different mem tables and immutable mem tables, or they can share the same mem table and immutable mem table. Among them, the multiple partitions in the disk can correspond one-to-one to the multiple partitions of the first part of the above-mentioned memory. For example, the number of multiple partitions can be determined according to the number of storage pools (i.e., the above-mentioned storage volumes), or the number of multiple partitions can be determined by other means, which is not limited in this embodiment of the present application.

[0114] As shown in Figure 7, Figure 7 shows an exemplary process of writing metadata. First, the business data and WAL are written to the disk (i.e., ① shown in Figure 7), and then metadata for indicating the business data is generated and written to the second part of the memory (i.e., ② shown in Figure 7), so that the metadata can be subsequently flushed from the second part of the memory to the disk. Then, the metadata is compressed and written to the first part of the memory (i.e., ③ shown in Figure 7) to ensure that the full amount of compressed metadata is stored.

[0115] As can be seen, each write adds a WAL to disk. For example, if 10 writes are performed, 10 WALs are generated: the first WAL corresponds to the first write, and the tenth WAL corresponds to the tenth write. Furthermore, each write adds metadata to the second portion of memory. However, at any given moment, only a portion of the added metadata may be flushed to disk. For example, if 10 writes are performed, 10 metadata are added to the second portion of memory. Of these 10 metadata, metadata 1 to 5 (corresponding to writes 1 to 5) have been flushed to disk, while metadata 6 to 10 (corresponding to writes 6 to 10) have not yet been flushed to disk and remain in the second portion of memory.

[0116] Based on this, an embodiment of the present application can add a reference pointer at the last metadata to be flushed to the disk. The reference pointer is used to point to the WAL corresponding to the first metadata that has not yet been flushed to the disk. The reference pointer shows the relationship between the metadata and the WAL on the disk. For example, in the example above, since the 1st to 5th metadata have been flushed to the disk, a reference pointer is added after the 5th metadata on the disk. The reference pointer is used to point to the 6th WAL corresponding to the 6th metadata. In an embodiment of the present application, this reference pointer can be used in the reconstruction process for the metadata, as described below.

[0117] Regarding the metadata reconstruction process, the device provided by the embodiment of the present application also includes: a fourth module, which is used to write the metadata to the disk, and when the metadata in the memory is cleared, obtain the metadata and WAL in the disk in parallel, filter the metadata and WAL in the disk obtained in parallel according to the priority of the metadata and WAL in the disk, and restore the metadata in the memory based on the results of the filtering.

[0118] Among them, the process of writing metadata to disk has been explained above and will not be repeated here. Since the memory is used for non-persistent storage, the metadata in the memory will be cleared after a power outage, that is, the metadata in the first part of the memory and the second part of the memory will be cleared. Among them, since the second part of the memory is used to incrementally store metadata, the metadata in the second part of the memory does not need to be rebuilt, and only the new metadata needs to be written to the second part of the memory when new metadata is generated due to the subsequent writing of new business data. However, since the first part of the memory is used to store the compressed metadata in full, the compressed metadata in the first part of the memory needs to be rebuilt.

[0119] As shown in Figure 8, the reconstruction process requires the use of metadata and WALs on disk. As explained above, WALs are generated based on metadata, and therefore can be used to reconstruct metadata. Figure 8 shows two types of WALs: the WAL at the end of the business data is the appended WAL described above, and the continuous WAL is the individually written WAL described above. At least one of these two types of WALs can exist on disk.

[0120] In embodiments of the present application, metadata and WALs from disks are acquired in parallel. Compared to serial acquisition, parallel acquisition can improve acquisition efficiency, thereby improving metadata reconstruction efficiency. For example, the number of tasks can be determined based on the number of partitions on the disk and the amount of business data. Each task is used to acquire metadata from one disk or one WAL. Multiple tasks can be executed simultaneously to achieve the aforementioned parallel acquisition. For example, embodiments of the present application can combine the reference pointers described above to determine the relationship between metadata and WALs from disks, thereby rationally acquiring metadata or WALs from disks. For example, using the reference pointer example above, if 10 tasks are established, 5 of them are used to acquire metadata from disks 1 to 5, while the other 5 tasks are used to acquire WALs 6 to 10 from disks. For example, in the case of block storage, the amount of business data is the number of data blocks. In the case of object storage, the amount of business data is the number of objects. In the case of file storage, the amount of business data is the number of files.

[0121] However, there may be duplication in the WALs and metadata obtained in parallel. For example, there may be duplication between the WALs and metadata, or duplication between different WALs, or duplication between different metadata. Taking WAL duplication as an example, a user writes the same business data twice. The value of the business data during the first write is m, and the value of the business data during the second write is n, which is equivalent to changing the value of the business data from m to n. Accordingly, two WALs will be generated, and there will be duplication between these two WALs because they are used to reconstruct the same metadata, that is, metadata with the same identification part. Of course, in order to ensure the accuracy of the data, the WAL corresponding to the business data written for the second time needs to be used to reconstruct the metadata, and the WAL corresponding to the business data written for the first time will not be used to reconstruct the metadata.

[0122] It can be seen that when there are duplications between WALs and metadata, the WALs and metadata need to be screened to ensure that the metadata reconstructed in memory based on the WALs and metadata is the latest metadata, and that the business data indicated by the latest metadata is the latest business data. To this end, the embodiments of the present application set priorities for the WALs and metadata to facilitate screening based on priority. For example, for duplicate WALs and metadata, the WALs with higher priorities are retained and the metadata is reconstructed, while the metadata with lower priorities is ignored. For another example, for different duplicate WALs, the WAL with the highest priority is retained and the metadata is reconstructed, while the WAL with a lower priority is ignored. For another example, for different duplicate metadata, the metadata with the highest priority is retained and the metadata is reconstructed in memory, while the metadata with a lower priority is ignored.

[0123] In an exemplary embodiment, the priority of WAL is higher than the priority of metadata on disk. The reason is that each time business data is written, WAL and new metadata are generated accordingly. According to the above description, WAL will be written directly to disk, while new metadata will be written to memory first, and will not be flushed to disk until the memory is full. Therefore, the WAL on disk is more comprehensive and can reflect the new metadata, but the metadata on disk may not be comprehensive enough and cannot reflect the new metadata because the new metadata may not have been flushed from memory to disk. Therefore, the embodiment of the present application makes the priority of WAL higher than the priority of metadata on disk.

[0124] Exemplarily, the business data on the disk includes multiple data groups. For example, in the case of block storage, a data group is a data block. In the case of object storage, a data group is an object. In the case of file storage, a data group is a file. The WAL includes multiple WALs corresponding to the multiple data groups. The priority of any WAL is determined by the position of the data group corresponding to any WAL within the multiple data groups. For example, a WAL is determined by a logical block address (LBA). A larger LBA indicates that the WAL was written to the disk later and the data group corresponding to the WAL is located later in the multiple data groups. Therefore, a larger LBA indicates a higher priority. In addition, different WALs can be located in chunks. A chunk is a continuous storage space on the disk. A larger chunk identification (ID) indicates that the WAL was written to the disk later and the data group corresponding to the WAL is located later in the multiple data groups. Therefore, a larger chunk ID indicates a higher priority.

[0125] Exemplarily, the metadata in the disk includes metadata of multiple partitions, and the metadata of each partition includes multiple groups of metadata, and a group of metadata is, for example, an SST. The priority of any group of metadata is determined according to the position of any group in the partition where any group is located and the position of the partition where any group is located in multiple partitions. For example, a group of metadata has a chunk ID. The larger the chunk ID, the later the group of metadata was written to the disk and the later the position of the group of metadata in the partition where it is located. Therefore, the larger the chunk ID, the higher the priority. In addition, a group of metadata also has a partition ID. The larger the partition ID, the later the group of metadata was written to the disk and the later the position of the partition where the group is located in multiple partitions. Therefore, the larger the partition ID, the higher the priority.

[0126] For example, assume there are 12 solid-state disks (SSDs), each with 86 partitions, for a total of 1024 partitions. Based on a single SSD bandwidth of 3 gigabytes per second (GB / s), 12 SSDs provide 36 GB / s of bandwidth. Rebuilding 150 gigabytes of metadata in memory takes less than 5 seconds, significantly less than the 10 seconds required in related technologies.

[0127] Regarding the metadata merging process, the related art merges the metadata stored in two adjacent levels based on the multiple levels included in the disk, which is also called data compression (compaction). For example, when it is necessary to merge the metadata stored in the first level and the second level, the metadata stored in the first level and the metadata stored in the second level are read respectively. After merging the read metadata, for example, after merging different metadata including the same identification part, the merged metadata is obtained, and the merged metadata is written to the second level, thereby releasing the first level to facilitate the subsequent flushing of the metadata in the memory to the first level. It is not difficult to see that this process requires both reading metadata from the disk and writing metadata to the disk after the merging, resulting in read amplification and write amplification, increasing the merging overhead, and reducing the merging efficiency.

[0128] Exemplarily, the apparatus provided in an embodiment of the present application further includes: a fifth module configured to, when the amount of metadata for a reference partition exceeds a threshold, retrieve metadata corresponding to the reference partition from memory based on an identifier of the reference partition, and write the retrieved metadata to the reference partition to overwrite the metadata whose amount exceeds the threshold. Because this merging process directly performs an overwrite write, eliminating the need to read metadata from disk and merge, read amplification and write amplification are avoided, resulting in lower overhead and higher efficiency for the merging process.

[0129] The metadata on the disk includes metadata for multiple partitions, including a reference partition. The embodiments of the present application divide multiple partitions in memory and disk, making it possible to merge metadata based on reference partitions. This merging method is a fine-grained, local merging method. In other words, when metadata in a reference partition needs to be merged, there is no need to wait for other partitions; the metadata can be directly merged according to demand, which provides greater flexibility in merging.

[0130] In an exemplary embodiment, the data volume of the metadata of the reference partition exceeds the threshold value, which may be that the data volume of the metadata in the reference partition itself exceeds the data volume threshold value, or the number of multiple groups of metadata in the reference partition exceeds a certain threshold value, for example, the number of SSTs exceeds the SST number threshold value. This embodiment of the present application does not limit this. As shown in Figure 9, in the case where the data volume of the metadata of the reference partition exceeds the threshold value, the identification of the reference partition can be obtained. Since the multiple partitions in the disk correspond one-to-one to the multiple partitions included in the first part of the memory, the reference partition can be determined from the multiple partitions included in the first part of the memory based on the identification of the reference partition, so that the metadata corresponding to the reference partition can be obtained from the first part of the memory. By writing the obtained metadata directly to the reference partition in the disk, the written metadata can cover the metadata in the reference partition whose data volume exceeds the threshold value, that is, the data volume of the metadata in the reference partition no longer exceeds the threshold value.

[0131] Among them, according to the above description, the first part of the memory stores the compressed metadata in full, and therefore the metadata corresponding to the reference partition obtained from the first part of the memory may refer to the compressed metadata. The embodiment of the present application may write the compressed metadata directly to the reference partition in the disk, or may restore the compressed metadata to metadata and then write it to the disk. When restoring the compressed identification part, taking the example of the compressed identification part including multiple levels of first characters, the multiple levels of first characters may be traversed to form the identification part. When restoring the compressed data part, you may refer to the three methods of obtaining it in the above description, which will not be elaborated here.

[0132] For example, in the process of overwriting metadata whose data volume exceeds a threshold, the writing of metadata to the first part of the memory is suspended, and metadata is only written to the second part of the memory. The reason for suspending the writing of data to the first part of the memory is that during the overwriting process, the metadata stored in the first part of the memory needs to be written to the reference partition on the disk. If new metadata continues to be written to the first part of the memory during this process, it may cause data asynchrony, affecting data consistency. After the overwriting process is completed, the metadata written to the second part of the memory during the overwriting process is synchronized to the first part of the memory. In this way, the first part of the memory is guaranteed to store the full amount of compressed metadata, and the metadata synchronized to the first part of the memory will be written to the partition on the disk during the subsequent merging process.

[0133] For example, during the process of overwriting metadata whose data volume exceeds a threshold, metadata is read from the second portion of the memory, and metadata reading from the first portion of the memory is suspended. This is because metadata writing to the first portion of the memory has been suspended during the overwriting process. If the metadata to be read is metadata written during the overwriting process, it cannot be read from the first portion of the memory, affecting reading efficiency. After the overwriting process is completed, access to the first portion of the memory is restored to read metadata from the first portion of the memory. This is because after the overwriting process is completed, the metadata written during the overwriting process is synchronized to the first portion of the memory, and the first portion of the memory has resumed full storage of the compressed metadata, making reading from the first portion of the memory more efficient.

[0134] The above describes the various processes for processing metadata. Each process involves the first module 101, the second module 102, the third module, the fourth module, and the fifth module. The division of these modules into their respective functions is merely an example and is not intended to limit the embodiments of this application. In other words, in actual applications, the aforementioned functions can be assigned to different functional modules as needed to complete all or part of the functions described above.

[0135] Exemplarily, for any module, the module can be split to obtain multiple sub-modules, and the multiple sub-modules jointly realize the function of the module. Alternatively, different modules can be combined, and the combined modules can realize the functions of different modules. Alternatively, the modules can be split and then combined, so that the functions that the modules can realize are more flexible. For example, Figures 7 to 9 show: the first part of the memory, the second part of the memory, the part of the disk for storing metadata, and the part of the disk for storing business data and WAL, a total of four parts. The embodiment of the present application can split the first module 101 to the fifth module and then combine them to obtain four modules corresponding to the four parts one by one, and each module is used to realize the relevant functions of a part corresponding to the module.

[0136] In summary, the embodiment of the present application stores the compressed metadata in full in the memory, so that when the compressed metadata is queried based on the received identification part, the data part corresponding to the received identification part can be obtained and returned without having to access the disk to query the metadata, thereby reducing read amplification and effectively improving the metadata reading performance.

[0137] In addition, by deploying a module outside the file system to write WAL and business data to the disk, there is no need to go through the file system, thus avoiding the superposition of different WALs, and the writing of WAL can be completed quickly and the metadata writing process can be entered, thereby reducing write amplification and effectively improving the writing efficiency of metadata. In the process of merging metadata, the merging efficiency of metadata is effectively improved by overwriting, reducing read amplification and write amplification. In the process of rebuilding metadata, the metadata and WAL in the disk are obtained in parallel, and are screened in combination with priority. This method can effectively improve the reconstruction efficiency of metadata. Among them, the improvement of the reading efficiency, writing efficiency, merging efficiency and reconstruction efficiency of metadata in the embodiments of the present application can reach more than 1 times.

[0138] The above describes the apparatus for processing data provided in the embodiments of the present application. Corresponding to the apparatus described above, the embodiments of the present application also provide a method for processing data. For example, this method can be applied to the apparatus for processing data shown in FIG1 , or to a computer device, and the embodiments of the present application do not limit this. As shown in FIG10 , the method for processing data provided in the embodiments of the present application includes the following steps 1001 and 1002.

[0139] Step 1001: Receive the identification portion.

[0140] Step 1002: query the compressed metadata based on the received identification part, obtain the data part corresponding to the received identification part, and return the obtained data part. The compressed metadata is obtained by compressing the corresponding identification part and data part in the metadata, and the compressed metadata is fully stored in the memory.

[0141] In an exemplary embodiment, the compressed identification portion corresponds to a first pointer, and the first pointer is used to point to the compressed data portion. Querying the compressed metadata based on the received identification portion to obtain the data portion corresponding to the received identification portion includes: querying the compressed identification portion based on the received identification portion to obtain a reference identification portion, wherein the reference identification portion matches the received identification portion; and obtaining the data portion corresponding to the received identification portion based on the first pointer corresponding to the reference identification portion.

[0142] In one embodiment, the compressed data portion includes multi-level characters, a second pointer corresponding to a lower-level character in two adjacent levels of characters, the second pointer being used to point to a higher-level character in the two adjacent levels of characters, and the first pointer being used to point to a lowest-level character in the multi-level characters. Obtaining the data portion corresponding to the received identification portion based on the first pointer corresponding to the reference identification portion includes: determining the lowest-level character pointed to by the first pointer corresponding to the reference identification portion, searching upwards level by level based on the second pointer corresponding to the lowest-level character until a highest-level character in the multi-level characters is obtained, and using the data portion consisting of the lowest-level character to the highest-level character as the data portion corresponding to the received identification portion.

[0143] In another embodiment, the compressed data portion includes a data portion group obtained by compressing multiple data portions, and the reference identification portion further corresponds to a data portion identifier, the data portion identifier being used to indicate one of the multiple data portions. Obtaining the data portion corresponding to the received identification portion based on a first pointer corresponding to the reference identification portion includes: determining the data portion group pointed to by the first pointer corresponding to the reference identification portion, decompressing the data portion group to obtain multiple data portions, and selecting, from the multiple data portions, a data portion indicated by the data portion identifier corresponding to the reference identification portion as the data portion corresponding to the received identification portion.

[0144] In yet another embodiment, the compressed data portion is index information, and the index information is used to point to a data portion on a disk. Obtaining the data portion corresponding to the received identification portion based on a first pointer corresponding to the reference identification portion includes: determining the index information pointed to by the first pointer corresponding to the reference identification portion, and using the data portion on the disk pointed to by the index information as the data portion corresponding to the received identification portion.

[0145] Exemplarily, the metadata is hot metadata, which is metadata indicating hot business data, or the hot metadata is metadata located in a working volume, where a working volume is a volume on a disk whose access frequency exceeds a threshold.

[0146] In an exemplary embodiment, the multi-level characters are located in a memory space in a memory, the memory space including at least one memory page, and the memory space having a starting address. A second pointer corresponding to a non-highest-level character in the multi-level characters includes an offset of a character in the previous level of the non-highest-level character relative to the starting address. The second pointer corresponding to a lowest-level character in the multi-level characters also includes the starting address.

[0147] Exemplarily, the embodiment of the present application also provides a metadata writing process, and the method further includes: writing the business data and WAL to the disk, generating metadata for indicating the business data, and the WAL corresponds to the metadata.

[0148] An embodiment of the present application also provides a metadata reconstruction process, and the method further includes: writing the metadata to disk, and when the metadata in the memory is cleared, obtaining the metadata and WAL in the disk in parallel, filtering the metadata and WAL in the disk obtained in parallel according to the priority of the metadata and WAL in the disk, and restoring the metadata in the memory based on the filtering results.

[0149] In an exemplary embodiment, the metadata in the disk includes metadata of multiple partitions, the metadata of each partition includes multiple groups of metadata, and the priority of any group of metadata is determined according to the position of any group in the partition where any group is located and the position of the partition where any group is located in multiple partitions.

[0150] In addition, the business data in the disk includes multiple data groups, and the WAL includes multiple WALs corresponding to the multiple data groups. The priority of any WAL is determined according to the position of the data group corresponding to any WAL in the multiple data groups.

[0151] In an exemplary embodiment, the embodiment of the present application also provides a metadata merging process, where the metadata in the disk includes metadata of multiple partitions, and the multiple partitions include a reference partition. The method also includes: when the data volume of the metadata of the reference partition exceeds a threshold, obtaining the metadata corresponding to the reference partition from the memory according to the identifier of the reference partition, and writing the obtained metadata to the reference partition to overwrite the metadata whose data volume exceeds the threshold.

[0152] It should be understood that the beneficial effects of the method shown in FIG10 are the same as those of the data processing device shown in FIG1 when implementing its functions. In addition, the method embodiment and the device provided in the above embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0153] In addition, an embodiment of the present application provides a device for processing data, which includes a memory and a processor; at least one computer instruction is stored in the memory, and at least one computer instruction is loaded and executed by the processor, so that the device for processing data implements the method for processing data shown in Figure 10.

[0154] Referring to FIG. 11 , FIG. 11 shows a schematic structural diagram of an exemplary data processing device 1100 of the present application. The data processing device 1100 includes at least one processor 1101 , a memory 1103 , and at least one network interface 1104 .

[0155] The processor 1101 is, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor or one or more integrated circuits or application-specific integrated circuits (ASICs) for implementing the solution of the present application, a programmable logic device (PLD), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the advanced reduced instruction set machine (ARM) architecture. It can implement or execute various logic blocks, modules and circuits described in conjunction with the disclosure of this application. A processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0156] Optionally, data processing device 1100 further includes a bus 1102. Bus 1102 is used to transmit information between the various components of data processing device 1100. Bus 1102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Bus 1102 may be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, FIG11 shows only one line, but this does not indicate that there is only one bus or only one type of bus.

[0157] The memory 1103 may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache.

[0158] By way of example and not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0159] The memory 1103 may also be other types of storage devices that can store static information and instructions. Or it may be other types of dynamic storage devices that can store information and instructions. Or it may be other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 1103 is, for example, independent and connected to the processor 1101 via the bus 1102. The memory 1103 may also be integrated with the processor 1101.

[0160] The network interface 1104 uses any transceiver-like device to communicate with other devices or communication networks. The communication network can be Ethernet, a radio access network (RAN), or a wireless local area network (WLAN). The network interface 1104 can include a wired network interface or a wireless network interface. Specifically, the network interface 1104 can be an Ethernet interface, such as a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a WLAN interface, a cellular network interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of the present application, the network interface 1104 can be used for the data processing device 1100 to communicate with other devices.

[0161] In a specific implementation, as some embodiments, the processor 1101 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG11 . Each of these processors may be a single-core processor or a multi-core processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0162] In a specific implementation, as some embodiments, the data processing device 1100 may include multiple processors, such as processor 1101 and processor 1105 shown in Figure 11. Each of these processors can be a single-core processor or a multi-core processor. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0163] In some embodiments, the memory 1103 is used to store program instructions 1110 for executing the solution of the present application, and the processor 1101 can execute the program instructions 1110 stored in the memory 1103. That is, the data processing device 1100 can implement the method provided in the method embodiment, i.e., the method shown in Figure 10, through the processor 1101 and the program instructions 1110 in the memory 1103. The program instructions 1110 may include one or more software modules. Optionally, the processor 1101 itself may also store program instructions for executing the solution of the present application.

[0164] During the specific implementation process, the data processing device 1100 of the present application may correspond to a computer device for executing the above method. The processor 1101 in the data processing device 1100 reads the instructions in the memory 1103, so that the data processing device 1100 shown in Figure 11 can execute all or part of the steps in the method embodiment.

[0165] The data processing device 1100 may also correspond to the apparatus shown in FIG1 , and each functional module in the apparatus shown in FIG1 is implemented using software of the data processing device 1100. In other words, the functional modules included in the apparatus shown in FIG1 are generated by the processor 1101 of the data processing device 1100 after reading the program instructions 1110 stored in the memory 1103.

[0166] Among them, each step of the method shown in Figure 10 is completed by the hardware integrated logic circuit or software instructions in the processor of the data processing device 1100. The steps of the method embodiment disclosed in this application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method embodiment in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0167] An embodiment of the present application provides a computer program or a computer program product, which includes: computer instructions, which, when executed by a computer, enable the computer to execute the method for processing data shown in FIG10 above.

[0168] Illustratively, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the method for processing data shown in FIG. 10 is executed by the computer.

[0169] An embodiment of the present application provides a chip, including a processor, for calling and executing instructions stored in a memory, so that a computer equipped with the chip executes the method for processing data shown in FIG10 above.

[0170] An embodiment of the present application also provides another chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the computer equipped with the chip executes the method for processing data shown in Figure 10 above.

[0171] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0172] In this application, the terms "first," "second," and the like are used to distinguish between identical or similar items with substantially the same purpose and function. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms. These terms are simply used to distinguish one element from another.

[0173] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0174] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of partitions means two or more partitions. The terms "system" and "network" are often used interchangeably herein.

[0175] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0176] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed items. The term "and / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this application generally indicates that the associated objects are in an "or" relationship.

[0177] It should also be understood that the terms “if” and “if” may be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting.” Similarly, the phrases “if it is determined that ” or “if [stated condition or event] is detected” may be interpreted to mean “upon determining ” or “in response to determining ” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event],” depending on the context.

[0178] The above description is merely an embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A device for processing data, characterized in that: The device comprises: A first module is used to receive an identification part; The second module is used to query the compressed metadata according to the received identification part, obtain the data part corresponding to the received identification part, and return the obtained data part. The compressed metadata is obtained by compressing the corresponding identification part and data part in the metadata, and the compressed metadata is fully stored in the memory.

2. The device according to claim 1, characterized in that The compressed identification part corresponds to a first pointer, and the first pointer is used to point to the compressed data part; The second module is used to query the compressed identification part according to the received identification part to obtain the reference identification part, and obtain the data part corresponding to the received identification part according to the first pointer corresponding to the reference identification part, and the reference identification part matches the received identification part.

3. The device according to claim 1 or 2, characterized in that: The device also includes: The third module is used to write business data and write-ahead log WAL to disk, generate metadata indicating the business data, the WAL corresponds to the metadata, the first module and the second module are deployed on the file system, and the third module is deployed outside the file system.

4. The device according to claim 3, characterized in that The device also includes: The fourth module is used to write the metadata to the disk, and when the metadata in the memory is cleared, obtain the metadata in the disk and the WAL in parallel, filter the metadata in the disk and the WAL obtained in parallel according to the priorities of the metadata in the disk and the WAL, and restore the metadata in the memory according to the filtering results.

5. The device according to claim 4, characterized in that The metadata in the disk includes metadata of multiple partitions, and the metadata of each partition includes multiple groups of metadata. The priority of any group of metadata is determined according to the position of the any group in the partition where the any group is located and the position of the partition where the any group is located in the multiple partitions.

6. The device according to claim 4 or 5, characterized in that The business data in the disk includes multiple data groups, the WAL includes multiple WALs corresponding to the multiple data groups one by one, and the priority of any WAL is determined according to the position of the data group corresponding to the any WAL in the multiple data groups.

7. The device according to any one of claims 1 to 6, characterized in that: The metadata in the disk includes metadata of a plurality of partitions, the plurality of partitions including a reference partition, and the apparatus further includes: The fifth module is used to obtain the metadata corresponding to the reference partition from the memory according to the identifier of the reference partition when the data volume of the metadata of the reference partition exceeds the threshold, and write the obtained metadata into the reference partition to cover the metadata whose data volume exceeds the threshold.

8. The device according to claim 2, characterized in that The compressed data portion includes multi-level characters, and the lower-level character in two adjacent levels of characters corresponds to a second pointer, the second pointer is used to point to the higher-level character in the two adjacent levels of characters, and the first pointer is used to point to a lowest-level character in the multi-level characters; The second module is used to determine the lowest-level character pointed to by the first pointer corresponding to the reference identification part, and to query upward step by step according to the second pointer corresponding to the lowest-level character until a highest-level character among the multi-level characters is obtained, and to use the data part composed of the lowest-level character to the highest-level character as the data part corresponding to the received identification part.

9. The device according to claim 2 or 8, characterized in that The compressed data portion includes a data portion group, the data portion group is obtained by compressing multiple data portions, the reference identification portion also corresponds to a data portion identification, and the data portion identification is used to indicate one of the multiple data portions; The second module is used to determine the data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain the multiple data parts, and among the multiple data parts, use a data part indicated by the data part identifier corresponding to the reference identification part as the data part corresponding to the received identification part.

10. The device according to any one of claims 2, 8 and 9, characterized in that: The compressed data portion is index information, and the index information is used to point to the data portion in the disk; The second module is used to determine the index information pointed to by the first pointer corresponding to the reference identification part, and use the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

11. The device according to any one of claims 1 to 10, characterized in that: The metadata is hot metadata; The hot metadata is metadata used to indicate hot business data, or the hot metadata is metadata located in a volume in a working state, and the volume in a working state is a volume in a disk whose access frequency exceeds a threshold.

12. The device according to claim 8, characterized in that The multi-level characters are located in a memory space in the memory, the memory space includes at least one memory page, and the memory space has a starting address; The second pointer corresponding to the non-highest level character in the multi-level character includes: the offset of the upper level character of the non-highest level character relative to the starting address; The second pointer corresponding to the lowest-level character in the multi-level characters also includes: the starting address.

13. A method for processing data, characterized in that: The method comprises: receiving identification part; The compressed metadata is queried according to the received identification part, the data part corresponding to the received identification part is obtained, and the obtained data part is returned. The compressed metadata is obtained by compressing the corresponding identification part and data part in the metadata, and the compressed metadata is fully stored in the memory.

14. The method according to claim 13, characterized in that The compressed identification part corresponds to a first pointer, and the first pointer is used to point to the compressed data part; The step of querying the compressed metadata according to the received identification part to obtain the data part corresponding to the received identification part includes: querying the compressed identification portion according to the received identification portion to obtain a reference identification portion, wherein the reference identification portion matches the received identification portion; According to the first pointer corresponding to the reference identification part, the data part corresponding to the received identification part is obtained.

15. The method according to claim 13 or 14, characterized in that The method further comprises: The business data and the write-ahead log WAL are written into a disk, and the metadata indicating the business data is generated, wherein the WAL corresponds to the metadata.

16. The method according to claim 15, characterized in that The method further comprises: The metadata is written to a disk, and when the metadata in the memory is cleared, the metadata in the disk and the WAL are obtained in parallel, and according to the priorities of the metadata in the disk and the WAL, the metadata in the disk and the WAL obtained in parallel are filtered, and the metadata in the memory is restored according to the filtering result.

17. The method according to claim 16, characterized in that The metadata in the disk includes metadata of multiple partitions, and the metadata of each partition includes multiple groups of metadata. The priority of any group of metadata is determined according to the position of the any group in the partition where the any group is located and the position of the partition where the any group is located in the multiple partitions.

18. The method according to claim 16 or 17, characterized in that The business data in the disk includes multiple data groups, the WAL includes multiple WALs corresponding to the multiple data groups one by one, and the priority of any WAL is determined according to the position of the data group corresponding to the any WAL in the multiple data groups.

19. The method according to any one of claims 13 to 18, characterized in that: The metadata in the disk includes metadata of a plurality of partitions, the plurality of partitions including a reference partition, and the method further includes: When the data volume of the metadata of the reference partition exceeds a threshold, the metadata corresponding to the reference partition is obtained from the memory according to the identifier of the reference partition, and the obtained metadata is written to the reference partition to cover the metadata whose data volume exceeds the threshold.

20. The method according to claim 14, characterized in that The compressed data portion includes multi-level characters, and the lower-level character in two adjacent levels of characters corresponds to a second pointer, the second pointer is used to point to the higher-level character in the two adjacent levels of characters, and the first pointer is used to point to a lowest-level character in the multi-level characters; The obtaining, according to the first pointer corresponding to the reference identification part, a data part corresponding to the received identification part, comprises: Determine the lowest-level character pointed to by the first pointer corresponding to the reference identification part, query upward step by step according to the second pointer corresponding to the lowest-level character until a highest-level character among the multi-level characters is obtained, and use the data part composed of the lowest-level character to the highest-level character as the data part corresponding to the received identification part.

21. The method according to claim 14 or 20, characterized in that The compressed data portion includes a data portion group, the data portion group is obtained by compressing multiple data portions, the reference identification portion also corresponds to a data portion identification, and the data portion identification is used to indicate one of the multiple data portions; The obtaining, according to the first pointer corresponding to the reference identification part, a data part corresponding to the received identification part, comprises: Determine the data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain the multiple data parts, and among the multiple data parts, use a data part indicated by the data part identifier corresponding to the reference identification part as the data part corresponding to the received identification part.

22. The method according to any one of claims 14, 20 and 21, characterized in that: The compressed data portion is index information, and the index information is used to point to the data portion in the disk; The obtaining, according to the first pointer corresponding to the reference identification part, a data part corresponding to the received identification part, comprises: Determine the index information pointed to by the first pointer corresponding to the reference identification part, and use the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

23. The method according to any one of claims 13 to 22, characterized in that: The metadata is hot metadata; The hot metadata is metadata used to indicate hot business data, or the hot metadata is metadata located in a volume in a working state, and the volume in a working state is a volume in a disk whose access frequency exceeds a threshold.

24. The method according to claim 20, characterized in that The multi-level characters are located in a memory space in the memory, the memory space includes at least one memory page, and the memory space has a starting address; The second pointer corresponding to the non-highest level character in the multi-level character includes: the offset of the upper level character of the non-highest level character relative to the starting address; The second pointer corresponding to the lowest-level character in the multi-level characters also includes: the starting address.

25. A device for processing data, characterized in that The device includes a memory and a processor; the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor, so that the data processing device implements the method for processing data described in any one of claims 13-24.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor to enable a computer to implement the method for processing data as described in any one of claims 13-24.

27. A computer program product, characterized in that The computer program product comprises computer instructions, and the computer instructions are executed by a processor to enable a computer to implement the method for processing data according to any one of claims 13 to 24.

Images