Apparatus, method, device and computer readable storage medium for processing data

Abstract

The application discloses a device, a method, equipment and a computer readable storage medium for processing data, and belongs to the technical field of computers. The device for processing data comprises a first module and a second module. The first module is used for receiving an identification part. The second module is used for querying compressed metadata stored in a memory in full according to the received identification part, obtaining a data part corresponding to the received identification part, and returning the obtained data part. The compressed metadata is obtained by compressing corresponding identification parts and data parts in metadata. The application can realize reading of metadata by using the compressed metadata stored in the memory in full, so that the data part corresponding to the received identification part is obtained. The reading cost of metadata is reduced, and the reading efficiency of metadata is improved.

Description

[0001] This application is a divisional application. The original application has the application number 202311683843.1 and the original application date is December 8, 2023. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to apparatus, methods, devices and computer-readable storage media for processing data. Background Technology

[0003] In the field of computer technology, there are various business operations with a large amount of business data, thus requiring storage services for this data. Providing storage services involves writing the business data to disk for storage and generating metadata corresponding to the business data. This metadata indicates the business data and facilitates subsequent retrieval. Since metadata itself is also a type of data, it also needs to be processed. Summary of the Invention

[0004] This application provides an apparatus, method, device, and computer-readable storage medium for processing data to realize the processing of metadata. The technical solutions provided by this application include the following aspects.

[0005] Firstly, an apparatus for processing data is provided. The apparatus includes a first module and a second module. The first module is used to receive an identifier portion. The second module is used to query compressed metadata based on the received identifier portion, obtain the data portion corresponding to the received identifier portion, and return the obtained data portion. The compressed metadata is obtained by compressing the corresponding identifier portion and data portion in the metadata, and the compressed metadata is stored entirely in memory.

[0006] Without compression, the metadata might require more space than memory can provide, making it impossible to write the entire metadata into memory. Therefore, this application compresses the metadata to obtain compressed metadata, which includes a compressed identifier portion and a compressed data portion. The compressed metadata requires less space than memory can provide, allowing the compressed metadata to be stored entirely in memory.

[0007] Therefore, during the metadata reading process, based on the received identifier, the entire compressed metadata stored in memory is queried, and the corresponding data portion can be obtained and returned. In other words, this application can read metadata simply by accessing memory, without needing to access the disk and read metadata from it step by step. This reduces the metadata reading overhead, improves metadata reading efficiency, and can more than double the read performance.

[0008] In one possible implementation, the compressed identifier portion corresponds to a first pointer, which is used to point to the compressed data portion. The second module is used to query the compressed identifier portion based on the received identifier portion to obtain a reference identifier portion, and then obtain the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion. The reference identifier portion matches the received identifier portion.

[0009] In this process, after querying the matching reference identifier based on the received identifier, the reference identifier belongs to the compressed identifier and therefore has a corresponding first pointer. The compressed data can be quickly obtained according to the indication of the first pointer, so as to obtain the data corresponding to the received identifier, which makes the data acquisition efficiency high.

[0010] In one possible implementation, the apparatus further includes a third module for writing business data and a write-ahead log (WAL) to disk, generating metadata to indicate the business data. The WAL corresponds to the metadata. The first and second modules are deployed on the file system, while the third module is deployed outside the file system.

[0011] Because this third module is deployed outside the file system, it does not need to write business data and the corresponding WAL (Write-Ahead Log) of metadata to disk through the file system, thus avoiding the generation and writing processes of both the file system's WAL and the disk's WAL. In other words, after the third module completes the writing of business data and the corresponding WAL of metadata, the metadata writing process can be executed directly, improving the metadata writing efficiency and achieving a performance improvement of more than 100%.

[0012] In one possible implementation, the apparatus further includes: a fourth module for writing metadata to disk, acquiring metadata and WAL in parallel when the metadata in memory is cleared, filtering the acquired metadata and WAL in parallel according to their priority, and restoring the metadata in memory based on the filtering results.

[0013] Since the memory uses a non-persistent storage model, the metadata in memory may be cleared, necessitating metadata reconstruction in memory. This application employs parallel acquisition of metadata and WAL (Write-Ahead Log) from the disk to reconstruct the metadata in memory. This parallel acquisition method is highly efficient, improving metadata reconstruction efficiency by more than double. However, the parallel acquisition of metadata and WAL from the disk may contain duplicates; different metadata and WAL sets may be used to reconstruct the same metadata (i.e., metadata including the same identifier portion) in memory. Therefore, this application prioritizes and filters the parallel acquisition of metadata and WAL from the disk to achieve metadata reconstruction in memory based on the filtering results.

[0014] In one possible implementation, the metadata in the disk includes metadata for multiple partitions, and the metadata for each partition includes multiple sets of metadata. The priority of any set of metadata is determined based on the position of any set within the partition in which it is located and the position of the partition in which it is located among multiple partitions.

[0015] For different partitions, the later the partition is located, the later the metadata was written, and the newer the metadata. Within the same partition, the later the metadata is located within that partition, the later the metadata was written, and the newer the metadata. Newer metadata should have higher priority to replace lower-priority metadata, ensuring the accuracy of subsequently reconstructed metadata.

[0016] In one possible implementation, the business data on the disk includes multiple data groups, and the WAL includes multiple WALs that correspond one-to-one with the multiple data groups. The priority of any WAL is determined according to the position of the data group corresponding to any WAL in the multiple data groups.

[0017] For different data groups, the later the data group is located in the database, the later its write time and the newer the data group. Since there is a one-to-one correspondence between the Write-Ahead Log (WAL) and the data group, a newer data group means a newer WAL. Newer WALs should have higher priority to replace lower-priority WALs and ensure the accuracy of subsequently reconstructed metadata.

[0018] In one possible implementation, the metadata in the disk includes metadata of multiple partitions, including a reference partition. The device further includes a fifth module, which is used to retrieve the metadata corresponding to the reference partition from memory according to the identifier of the reference partition when the amount of metadata of the reference partition exceeds a threshold, and write the retrieved metadata to the reference partition to overwrite the metadata whose amount of data exceeds the threshold.

[0019] This implementation eliminates the need to read metadata from disk, merge the metadata, and then rewrite it. Instead, it directly writes the metadata corresponding to the reference partition in memory to the reference partition on disk, ensuring that the amount of metadata in the reference partition on disk does not exceed the threshold. This reduces the overhead of the merging process, improves its efficiency, and can more than double the merging performance.

[0020] In one possible implementation, the compressed data portion includes multi-level characters. A second pointer corresponds to the lower-level character in two adjacent character levels, pointing to the higher-level character. A first pointer points to the lowest-level character in the multi-level character set. A second module determines the lowest-level character pointed to by the first pointer corresponding to the reference identifier portion. It then queries upwards level by level based on the second pointer corresponding to the lowest-level character until the highest-level character in the multi-level character set is obtained. The data portion consisting of the lowest-level character to the highest-level character is then used as the data portion corresponding to the received identifier portion.

[0021] The data portion can be a string, and different strings may overlap. Therefore, different strings can be compressed to produce a multi-level character set. After receiving the identifier portion, a reference identifier portion is retrieved from the compressed identifier portion based on the received identifier portion. The lowest-level character in the multi-level character set is determined using the first pointer corresponding to the reference identifier portion. The highest-level character is then retrieved by looking up from the second pointer corresponding to the lowest-level character. The data portion is composed of characters from the lowest level to the highest level, and this composed data portion is the data portion corresponding to the received identifier portion.

[0022] In one possible implementation, the compressed data portion includes a data portion group, which is obtained by compressing multiple data portions. A reference identifier portion also corresponds to a data portion identifier, which is used to indicate one of the multiple data portions. The second module is used to determine the data portion group pointed to by the first pointer corresponding to the reference identifier portion, decompress the data portion group to obtain multiple data portions, and select the data portion indicated by the data portion identifier corresponding to the reference identifier portion as the received data portion corresponding to the identifier portion.

[0023] After receiving the identifier portion, the reference identifier portion is obtained by querying the compressed identifier portion based on the received identifier portion. The data portion group is determined according to the first pointer corresponding to the reference identifier portion, resulting in multiple data portions. Then, a data portion is obtained from the multiple data portions according to the data portion identifier corresponding to the reference identifier portion. The obtained data portion is the data portion corresponding to the received identifier portion.

[0024] In one possible implementation, the compressed data portion is index information, which points to the data portion on the disk. The second module determines the index information pointed to by the first pointer corresponding to the reference identifier portion, and uses the data portion on the disk pointed to by the index information as the data portion corresponding to the received identifier portion.

[0025] In this implementation, after receiving the identifier portion, the reference identifier portion is obtained by querying the compressed identifier portion based on the received identifier portion. The index portion is determined based on the first pointer corresponding to the reference identifier portion. The data portion on the disk is obtained based on the index portion. The obtained data portion is the data portion corresponding to the received identifier portion.

[0026] In one possible implementation, the metadata is hot metadata. Hot metadata is metadata used to indicate hot business data, or hot metadata is metadata located in an active volume, which is a volume on disk whose access frequency exceeds a threshold.

[0027] In other words, when the metadata is "hot" metadata, it is compressed and stored entirely in memory. Conversely, when the metadata is "cold" metadata, compression is not required; the metadata is stored on disk instead of being written to memory. This approach further conserves memory space occupied by the compressed metadata.

[0028] In one possible implementation, the multilevel characters reside in a memory space comprising at least one memory page, and the memory space has a starting address. The second pointer corresponding to the non-highest-level character in the multilevel characters includes: the offset of the character above the non-highest-level character relative to the starting address; the second pointer corresponding to the lowest-level character in the multilevel characters also includes: the starting address.

[0029] In this implementation, the second pointer is compressed, reducing the memory space occupied by the second pointer and allowing more space in memory to be used to store the compressed metadata, thus ensuring the full storage of the compressed metadata.

[0030] Secondly, a method for processing data is provided. In this method, an identifier portion is received. Based on the received identifier portion, the compressed metadata is queried to obtain the data portion corresponding to the received identifier portion, and the obtained data portion is returned. The compressed metadata is obtained by compressing the corresponding identifier portion and data portion in the metadata, and the compressed metadata is stored entirely in memory.

[0031] In one possible implementation, the compressed identifier portion corresponds to a first pointer, which is used to point to the compressed data portion. The process of querying the compressed metadata based on the received identifier portion to obtain the data portion corresponding to the received identifier portion includes: querying the compressed identifier portion based on the received identifier portion to obtain a reference identifier portion, which matches the received identifier portion; and obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion.

[0032] In one possible implementation, the method further includes: writing business data and WAL to disk, generating metadata to indicate the business data, with the WAL corresponding to the metadata.

[0033] In one possible implementation, the method further includes: writing metadata to disk, retrieving metadata and WAL from disk in parallel when the metadata in memory is cleared, filtering the metadata and WAL from disk in parallel according to their priority, and restoring the metadata in memory based on the filtering results.

[0034] In one possible implementation, the metadata in the disk includes metadata for multiple partitions, and the metadata for each partition includes multiple sets of metadata. The priority of any set of metadata is determined based on the position of any set within the partition in which it is located and the position of the partition in which it is located among multiple partitions.

[0035] In one possible implementation, the business data on the disk includes multiple data groups, and the WAL includes multiple WALs that correspond one-to-one with the multiple data groups. The priority of any WAL is determined according to the position of the data group corresponding to any WAL in the multiple data groups.

[0036] In one possible implementation, the metadata in the disk includes metadata for multiple partitions, including a reference partition. The method further includes: if the amount of metadata for the reference partition exceeds a threshold, retrieving the metadata corresponding to the reference partition from memory based on the identifier of the reference partition, and writing the retrieved metadata to the reference partition to overwrite the metadata whose amount of data exceeds the threshold.

[0037] In one possible implementation, the compressed data portion includes multi-level characters. A second pointer corresponds to the lower-level character in two adjacent character levels, pointing to the higher-level character. A first pointer points to the lowest-level character in the multi-level characters. Obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion includes: determining the lowest-level character pointed to by the first pointer of the reference identifier portion; querying upwards level by level according to the second pointer corresponding to the lowest-level character until the highest-level character in the multi-level characters is obtained; and using the data portion consisting of the lowest-level character to the highest-level character as the data portion corresponding to the received identifier portion.

[0038] In one possible implementation, the compressed data portion includes a data portion group, which is obtained by compressing multiple data portions. A reference identifier portion also corresponds to a data portion identifier, which is used to indicate one of the multiple data portions. Obtaining the data portion corresponding to the received identifier portion based on a first pointer corresponding to the reference identifier portion includes: determining the data portion group pointed to by the first pointer corresponding to the reference identifier portion; decompressing the data portion group to obtain multiple data portions; and selecting the data portion indicated by the data portion identifier corresponding to the reference identifier portion as the received data portion corresponding to the identifier portion.

[0039] In one possible implementation, the compressed data portion is index information, which is used to point to the data portion on the disk. Obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion includes: determining the index information pointed to by the first pointer corresponding to the reference identifier portion, and using the data portion on the disk pointed to by the index information as the data portion corresponding to the received identifier portion.

[0040] In one possible implementation, the metadata is hot metadata. Hot metadata is metadata used to indicate hot business data, or hot metadata is metadata located in an active volume, which is a volume on disk whose access frequency exceeds a threshold.

[0041] In one possible implementation, the multilevel characters reside in a memory space comprising at least one memory page, and the memory space has a starting address. The second pointer corresponding to the non-highest-level character in the multilevel characters includes: the offset of the character above the non-highest-level character relative to the starting address; the second pointer corresponding to the lowest-level character in the multilevel characters also includes: the starting address.

[0042] Thirdly, a data processing apparatus is provided, the apparatus including a memory and a processor; the memory stores at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to cause the data processing apparatus to implement the data processing method provided in the second aspect or any possible implementation thereof.

[0043] Optionally, there may be one or more processors and one or more memories.

[0044] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0045] Fourthly, a computer program or computer program product is provided, comprising: computer instructions that, when executed by a computer, cause the computer to perform the method for processing data provided by the second aspect or any possible implementation thereof.

[0046] Fifthly, a computer-readable storage medium is provided that stores computer instructions, wherein when the computer instructions are executed on a computer, the method for processing data provided in the second aspect or any possible implementation thereof is executed by the computer.

[0047] In a sixth aspect, a chip is provided, including a processor for retrieving and executing instructions stored in a memory, causing a computer equipped with the chip to perform the method for processing data provided in the second aspect or any possible implementation thereof.

[0048] In a seventh aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, the processor is used to execute code in the memory, and when the code is executed, a computer with the chip installed performs the data processing method provided by the second aspect or any possible implementation thereof.

[0049] The technical effects achieved by the technical solutions provided by the second to seventh aspects of this application and their corresponding possible implementations can be found in the above description of the technical effects achieved by the technical solutions provided by the first aspect and their corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0051] Figure 2 This is a schematic diagram of the structure of a compressed identification portion provided in an embodiment of this application;

[0052] Figure 3 A schematic diagram illustrating a metadata reading process provided in an embodiment of this application;

[0053] Figure 4 A schematic diagram illustrating another metadata reading process provided in an embodiment of this application;

[0054] Figure 5 A schematic diagram illustrating yet another metadata reading process provided in an embodiment of this application;

[0055] Figure 6 A schematic diagram of a memory and a disk provided for an embodiment of this application;

[0056] Figure 7 A schematic diagram illustrating a metadata writing process provided in an embodiment of this application;

[0057] Figure 8 A schematic diagram illustrating a metadata reconstruction process provided in an embodiment of this application;

[0058] Figure 9 A schematic diagram illustrating a metadata merging process provided in an embodiment of this application;

[0059] Figure 10 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0060] Figure 11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0061] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0062] With the rapid development of digital technology, business data is experiencing explosive growth. Storage services, as the foundation of digital technology, have received significant attention from internet and cloud vendors. Providing storage services for business data requires writing the data to disk for storage and generating corresponding metadata. This metadata indicates the business data and facilitates subsequent retrieval of the stored data. Since metadata itself is also a type of data, it also requires processing.

[0063] In the process of processing metadata, related technologies first write the metadata into memory. Once memory is full, the metadata is flushed from memory to disk. The disk is divided into multiple levels. After the metadata is flushed from memory to disk, it is initially stored in the first level of the disk. When the first level of the disk is full, the metadata is written from the first level to the second level, and so on. Correspondingly, when stored metadata needs to be read, memory is accessed first. If the required metadata is found in memory, the business data is obtained according to the instructions in that metadata. If the required metadata is not found in memory, the first, second, third, and so on levels of the disk are accessed sequentially until the required metadata is found in a certain level on the disk, at which point the business data is obtained according to the instructions in that metadata.

[0064] However, due to the limited storage space in memory, the amount of metadata that can be written to memory is also limited. This makes it less likely that the required metadata can be read from memory, often necessitating further disk access to read the metadata level by level, and sometimes even requiring traversal of each level on the disk. This processing method is costly and inefficient.

[0065] This application provides a data processing apparatus, which can be in software form. The software apparatus can be embedded in other software with data processing needs, or it can be applied to hardware products. This application does not limit this. Figure 1 As shown, the device includes a first module 101 and a second module 102.

[0066] The first module 101 is used to receive the identifier portion. The second module 102 is used to query the compressed metadata based on the received identifier portion, obtain the data portion corresponding to the received identifier portion, and return the obtained data portion. The compressed metadata is obtained by compressing the corresponding identifier portion and data portion in the metadata, and the compressed metadata is stored in memory in its entirety.

[0067] Metadata includes a corresponding identifier and a data portion. The identifier uniquely identifies the data portion, and the data portion indicates the storage location of the business data. For example, in the case of block storage, business data is stored in the form of data blocks. For instance, the business data could be data blocks aligned to 4 kilobytes (KB). 4KB is merely an example, and this embodiment does not limit the scope. Furthermore, the identifier includes a key (K), and the data portion includes a value (V), where V indicates the storage location of the data block; therefore, the metadata is also called a KV pair. Accordingly, the data processing apparatus can be RocksDB, a database (DB) for managing KV pairs. Alternatively, in the case of object storage, business data is stored in the form of objects. The identifier includes an identifier mapped to the object, and the data portion includes information indicating the storage location of the object. Or, in the case of file storage, business data is stored in the form of files. The identifier includes an identifier mapped to the file, and the data portion includes information indicating the storage location of the file. For example, the methods of mapping objects or files include, but are not limited to, memory-mapped (MMAP) mapping. This application does not limit the methods of mapping.

[0068] Without compression, the metadata would require a significant amount of space, while memory has limited capacity, making it difficult to write the entire metadata into memory. Therefore, in this embodiment, the metadata is compressed to obtain compressed metadata, which includes a compressed identifier portion and a compressed data portion. Compared to raw metadata, compressed metadata requires less space, and even with limited memory capacity, the compressed metadata can be fully written into memory, ensuring its complete storage.

[0069] Accordingly, when a user needs to obtain a portion of the data to determine the storage location of the business data, the user provides the identifier portion they hold. The system then receives this identifier portion, queries the fully compressed metadata stored in memory based on the received identifier portion, and obtains the data portion corresponding to the received identifier portion. This data portion is then returned to the user. This completes the metadata retrieval. Since the data portion indicates the storage location of the business data, the user can determine the storage location of the business data based on the received data portion and obtain the business data from that location.

[0070] Because the compressed metadata is stored entirely in memory, this embodiment of the application can read the metadata simply by accessing memory, without needing to access the disk and read the metadata level by level. Therefore, it not only reduces the overhead of reading the required metadata and reduces read amplification, but also improves the efficiency of reading metadata and enables fast metadata reading, thereby facilitating rapid querying of business data. Here, read amplification is the ratio of a first data volume to a second data volume, where the second data volume is the amount of metadata to be read, and the first data volume is the total amount of data that needs to be read to access the required metadata. Since this embodiment of the application does not require reading the metadata level by level from the disk, it reduces the first data volume, thereby reducing read amplification.

[0071] In an exemplary embodiment, the compressed identifier portion corresponds to a first pointer, which is used to point to the compressed data portion. The second module 102 is used to query the compressed identifier portion based on the received identifier portion to obtain a reference identifier portion, and to obtain the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion, wherein the reference identifier portion matches the received identifier portion.

[0072] After querying the compressed identifier portion based on the received identifier portion, a reference identifier portion can be obtained, which is one of the compressed identifier portions. Since the compressed identifier portion corresponds to the first pointer, the reference identifier portion also corresponds to the first pointer. Therefore, the compressed data portion can be obtained based on the indication of the first pointer. For example, if the first pointer includes the start address and length of the compressed data portion, then the compressed data portion can be obtained based on the start address and length. This compressed data portion corresponds to the reference identifier portion. Because the reference identifier portion matches the received identifier portion, the compressed data portion also corresponds to the received identifier portion. Therefore, the data portion can be obtained based on the compressed data portion, and the obtained data portion is returned as the data portion corresponding to the received identifier portion.

[0073] In an exemplary embodiment, the compressed identifier portion includes multiple levels of first characters. The higher-level first character among two adjacent levels corresponds to a third pointer, which points to the lower-level first character. For example, the third pointer corresponding to the higher-level first character includes the starting address and length of the lower-level first character. The lowest-level first character among the multiple levels corresponds to the aforementioned first pointer, which points to the compressed data portion.

[0074] The identifier portion is a string, and the identifier portions (i.e., different strings) in different metadata may have a high degree of overlap. For example, see... Figure 2Of the eight identifier parts, only the last four characters differ, while the rest are identical. Therefore, the identifier parts in different metadata can be compressed so that the compressed identifier part includes a multi-level first character. For example, the multi-level first character can form... Figure 2 The tree-like structures shown, such as prefix trees and multi-granularity trees, can also form other structures, and this application does not limit them in this regard. Figure 2 To avoid redundancy, the third pointer corresponding to the character 89 and the first pointer corresponding to part of the first characters in the lowest level first character are not shown. This part of the first characters refers to 01, 02, 03 and 04 corresponding to 89, and 01, 02 and 03 corresponding to 90.

[0075] Based on the multi-level first characters included in the compressed identifier portion, when querying the compressed identifier portion according to the received identifier portion, the highest-level first character among the multi-level first characters is first obtained. Then, the query proceeds level by level downwards according to the third pointer corresponding to the highest-level first character until the lowest-level first character among the multi-level first characters is obtained. The reference identifier portion described above is composed of the highest-level first character to the lowest-level first character. In other words, in the compressed identifier portion, the query direction is from top to bottom, i.e., from the highest-level first character to the lowest-level first character.

[0076] For example, see Figure 2 Taking the received identifier 90123456789abcdef012345679004 as an example, when querying in the multi-level first character, first obtain 90123456789abcdef (i.e., the highest-level first character in the multi-level first character) that matches the received identifier. Then, based on the third pointer corresponding to 90123456789abcdef, obtain 01 that matches the received identifier. Based on the third pointer corresponding to 01, obtain 23 that matches the received identifier. Based on the third pointer corresponding to 23, obtain 45 that matches the received identifier. Based on the third pointer corresponding to 45, obtain 67 that matches the received identifier. Based on the third pointer corresponding to 67, obtain 89 and 90. Since 89 does not match the received identifier while 90 does, based on the third pointer corresponding to 90, obtain 01, 02, 03, and 04, and finally obtain 04 (i.e., the lowest-level first character in the multi-level first character) that matches the received identifier. The reference identifier portion consists of 90123456789abcdef to 04, and this reference identifier portion matches the received identifier portion, that is, the reference identifier portion is consistent with the received identifier portion.

[0077] For example, see also Figure 2Taking the received identifier as 90123456789abcdef0123456790 as an example, after obtaining 01, 02, 03 and 04 according to the third pointer corresponding to 90, it is assumed that these four characters all match the received identifier. Then, 90123456789abcdef to 01, 02, 03 and 04 respectively form four reference identifier parts. These reference identifier parts match the received identifier part. In other words, the reference identifier parts include the received identifier part.

[0078] For example, in this embodiment of the application, before compressing the identifier portion in different metadata, the memory is pre-divided into multiple zones. Each zone is used to store a set of multi-level first characters (a set of multi-level first characters is, for example, a tree structure), and the compressed data portion corresponding to that set of multi-level first characters. If the memory is not pre-divided, the compressed identifier portion only includes a set of multi-level first characters. As the metadata increases, the size of this set of multi-level first characters will become increasingly large. For example, the number of levels increases, and the number of first characters included in each level also increases. When the size exceeds a certain threshold, the efficiency of querying based on this set of multi-level first characters will be very low, thus requiring the splitting of this set of multi-level first characters, which will cause additional overhead. However, by dividing the memory into multiple zones, the compressed identifier portion can include multiple sets of multi-level first characters. Even if the metadata continues to increase, the size of each set of multi-level first characters is unlikely to exceed the threshold and will not affect the query efficiency. Therefore, there is no need to split and no additional overhead is incurred.

[0079] For example, the multi-level first character is located in a first memory space in memory, the first memory space includes at least one memory page, and the first memory space has a first starting address; the third pointer corresponding to the non-lowest-level first character in the multi-level first character includes: the offset of the next-level first character of the non-lowest-level first character relative to the first starting address. The first pointer corresponding to the highest-level first character in the multi-level first character also includes: the first starting address.

[0080] For example, in Figure 2In this implementation, the first characters that are not the lowest level include: 90123456789abcdef, 01, 23, 45, 47, 89, and 90, with 01 located between 90123456789abcdef and 23. The first character of the highest level includes: 90123456789abcdef. In this implementation, a global pointer is not used as the third pointer; instead, the first starting address and offset (or only the offset) are used as the third pointer. This reduces the space occupied by the third pointer, effectively compressing it and further saving memory space. Of course, using a global pointer as the third pointer can also be an implementation method of this application, and this application does not limit this implementation.

[0081] The above examples illustrate one possible implementation of obtaining the reference identifier portion through querying. However, the methods for obtaining the reference identifier portion in this application are not limited to this. Regardless of the method used to obtain the reference identifier portion, each reference identifier portion corresponds to a first pointer, thereby allowing the data portion corresponding to the received identifier portion to be obtained based on the first pointer corresponding to the reference identifier portion. For example, the methods provided in this application include, but are not limited to, the following three.

[0082] The first method involves a string as the data portion. Since data portions in different metadata may overlap, the data portions from different metadata can be compressed, resulting in a compressed data portion containing multiple levels of characters. To distinguish it from the first character included in the compressed identifier portion mentioned above, all characters included in the compressed data portion will be referred to as second characters in the following text. Each adjacent level of second characters corresponds to a second pointer, which points to the higher-level second character within that pair. For example, the second pointer corresponding to the lower-level second character includes the starting address and length of the higher-level second character. Additionally, the first pointer points to the lowest-level second character among the multiple levels of second characters. For example, the first pointer includes the starting address and length of that lowest-level second character.

[0083] The second module 102 is used to determine the lowest-level second character pointed to by the first pointer corresponding to the reference identifier portion, and to query upwards level by level according to the second pointer corresponding to the lowest-level second character until the highest-level second character among the multi-level second characters is obtained. The data portion composed of the lowest-level second character to the highest-level second character is used as the data portion corresponding to the received identifier portion. That is to say, in the compressed data portion, the query direction is from bottom to top, that is, from the lowest-level second character to the highest-level second character.

[0084] For example, see Figure 3Taking the reference identifier 90123456789abcdef012345679004 as an example, based on the first pointer corresponding to this reference identifier, we can determine the lowest-level second character 'yy' in the multi-level second character set. From the second pointer corresponding to 'yy', we get 'ff'; from the second pointer corresponding to 'ff', we get 'ee'; from the second pointer corresponding to 'ee', we get 'dd'; from the second pointer corresponding to 'dd', we get 'cc'; from the second pointer corresponding to 'cc', we get 'bb'; and from the second pointer corresponding to 'bb', we get 'aa'. Since 'aa' does not have a corresponding second pointer, we can determine that 'aa' belongs to the highest-level second character in the multi-level second character set. The data part, consisting of the lowest-level second character 'yy' to the highest-level second character 'aa', is a data part, namely aabbccddeeffyy. This data part can be used as the data part corresponding to the received identifier part.

[0085] In an exemplary embodiment, the multi-level second character is located in a second memory space in memory, the memory space including at least one memory page, and the memory space having a second starting address. The second pointer corresponding to the non-highest-level second character in the multi-level second character includes: the offset of the previous level second character of the non-highest-level second character relative to the second starting address. The second pointer corresponding to the lowest-level second character in the multi-level second character also includes: the second starting address.

[0086] To distinguish it from the first memory space and first starting address mentioned above, the names "second memory space" and "second starting address" are used. Figure 3 For example, the non-highest level second characters include: bb, cc, dd, ee, ff, xx, and yy, while the lowest level second characters include: xx and yy. In this implementation, a global pointer is not used as the second pointer; instead, a second starting address and offset (or only the offset) are used as the second pointer. This reduces the space occupied by the second pointer, effectively compressing it and further saving memory. Of course, using a global pointer as the second pointer can also be an implementation method of this application, and this application does not limit this approach.

[0087] In addition to the second pointer mentioned above, a fourth pointer may also exist in the compressed data portion. As explained above, the second pointer's function includes: performing a bottom-up query within the compressed data portion during the process of obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion. The fourth pointer's function may include: performing a top-down query within the compressed data portion in other processes. This application embodiment does not limit other processes; other processes can be set according to actual needs. In other words, the function of the fourth pointer in the compressed data portion is similar to the function of the third pointer mentioned above in the compressed identifier portion.

[0088] For example, in the compressed data portion, the higher-level second character among two adjacent levels of second characters corresponds to a fourth pointer, which is used to point to the lower-level second character among the two adjacent levels of characters. For example, the fourth pointer corresponding to the higher-level second character includes: the starting address and length of the lower-level second character. In an exemplary embodiment, the embodiments of this application can also compress the fourth pointer. For example, since the multi-level second characters are located in a second memory space, and the second memory space has a second starting address, the fourth pointer corresponding to the non-lowest-level second character among the multi-level second characters can include: the offset of the next-level second character of the non-lowest-level second character relative to the second starting address. The fourth pointer corresponding to the highest-level second character among the multi-level second characters also includes: the second starting address.

[0089] The second method of obtaining the data involves compressing the data portion into a data portion group, which is obtained by compressing multiple data portions. The reference identifier also corresponds to a data portion identifier, which is used to indicate one of the multiple data portions.

[0090] The second module 102 is used to determine the data part group pointed to by the first pointer corresponding to the reference identifier part, decompress the data part group to obtain multiple data parts, and among the multiple data parts, take the data part indicated by the data part identifier corresponding to the reference identifier part as the data part corresponding to the received identifier part.

[0091] As mentioned earlier, the first pointer is used to point to the compressed data portion. Since the compressed data portion in this acquisition method is a data portion group, the first pointer is used to point to the data portion group. For example, the first pointer includes: the starting address and length of the data portion group. After determining the data portion group pointed to by the first pointer, the data portion group can be decompressed in memory to obtain multiple data portions. Although decompressing the data portion group in memory will occupy additional memory space, the space occupied is small and will not affect the full storage of the compressed metadata in memory. Optionally, after obtaining multiple data portions through decompression and using them as the basis to obtain the data portion corresponding to the received identifier portion, the multiple data portions obtained through decompression can be deleted to prevent these data portions from continuously occupying memory space.

[0092] For example, see Figure 4 Taking the reference identifier 90123456789abcdef012345679004 as an example, data part group a can be determined based on the first pointer corresponding to this reference identifier. Decompressing data part group a yields data part 1, data part 2, and data part 3. If the data part identifier corresponding to the reference identifier indicates data part 2, then data part 2 in data part group a is taken as the data part corresponding to the received identifier.

[0093] In some implementations, the multiple data portions obtained by decompression are associated one-to-one with multiple data portion identifiers. When determining the data portion indicated by the data portion identifier corresponding to the reference identifier portion, multiple data portion identifiers are queried based on the data portion identifier corresponding to the reference identifier portion. After a data portion identifier is matched, the data portion corresponding to the matched data portion identifier is taken as the data portion indicated by the data portion identifier corresponding to the reference identifier portion.

[0094] For example, still using Figure 4 Taking the example shown, data part 1 corresponds to data part identifier 00, data part 2 corresponds to data part identifier 01, and data part 3 corresponds to data part identifier 10. If the data part identifier corresponding to the reference identifier part is 01, then data part 2 corresponding to 01 will be used as the data part corresponding to the received identifier part.

[0095] In other embodiments, the multiple data portions obtained by decompression have the same length and occupy contiguous space. The data portion identifier corresponding to the reference identifier represents the order of a data portion among the multiple data portions. When determining the data portion indicated by the data portion identifier corresponding to the reference identifier, based on the order represented by the data portion identifier corresponding to the reference identifier and the aforementioned same length, one data portion is selected from the multiple data portions as the data portion indicated by the data portion identifier corresponding to the reference identifier.

[0096] For example, still using Figure 4 Taking the example shown, data portion 1, data portion 2, and data portion 3 are all 16 bits long, occupying bits 0 to 47 in total. Specifically, data portion 1 occupies bits 0 to 15, data portion 2 occupies bits 16 to 31, and data portion 3 occupies bits 32 to 47. If the data portion corresponding to the reference identifier is identified as 2, then it represents the second data portion among multiple data portions. Therefore, the second data portion among data portions 1, 2, and 3, i.e., data portion 2 occupying bits 16 to 31, can be used as the data portion corresponding to the received identifier.

[0097] The third method involves obtaining the data by setting up an index, which points to the data portion on the disk. For example, this index might contain the starting address and length of the data portion on the disk.

[0098] The second module 102 is used to determine the index information pointed to by the first pointer corresponding to the reference identifier portion, and to use the data portion in the disk pointed to by the index information as the data portion corresponding to the received identifier portion.

[0099] In this acquisition method, the compressed identifier portion resides in memory, while the data portion resides on disk, thus separating the identifier and data portions. In the case of block storage, since the identifier portion is K and the data portion is V, this is also called KV separation. As mentioned earlier, the first pointer is used to point to the compressed data portion. Since the compressed data portion in this acquisition method is index information, the first pointer is used to point to the index information. For example, the first pointer includes: the starting address and length of the index information.

[0100] For example, see Figure 5Taking the reference identifier 90123456789abcdef012345679004 as an example, index information 1 can be determined based on the first pointer corresponding to this reference identifier. Based on index information 1, data portion 1 on the disk can then be determined. Therefore, data portion 1 is used as the data portion corresponding to the received identifier. Although this method requires accessing the disk to obtain the data portion, this access is based on the indication of the index information. This access is targeted, does not incur significant overhead, and does not reduce metadata reading efficiency. It is completely different from the process of accessing the disk to read metadata level by level in the related technologies mentioned above.

[0101] The above examples illustrate three methods for obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion. The methods described in this application are not limited to these. For example, the compressed data portion can also be obtained by separately compressing a data portion in metadata (different from the first method described above). Accordingly, the compressed data portion can be obtained based on the first pointer corresponding to the reference identifier portion. The compressed data portion is then decompressed in memory, and the decompressed data portion is directly used as the data portion corresponding to the received identifier portion. This application can, according to actual needs, selectively use, combine, or alternately use the various methods described above.

[0102] In the exemplary embodiment, the metadata is hot metadata. That is, this application embodiment only compresses the hot metadata, obtains the compressed metadata, and writes it entirely into memory. Cold metadata is not compressed or written into memory; instead, it is stored on disk. Exemplarily, this application embodiment can employ this scheme of compressing only hot metadata when memory space is insufficient. When memory space is sufficient, both hot and cold metadata are compressed, and the compressed metadata is written entirely into memory.

[0103] Hot metadata refers to metadata used to indicate frequently accessed business data. Hot business data can be frequently accessed business data or business data determined based on other metrics. Alternatively, hot metadata can be metadata located in an active volume (also called a storage volume). An active volume is defined as a volume whose disk access frequency exceeds a threshold. For example, if a volume is accessed more than 30 times every 60 seconds, it can be considered an active volume. Correspondingly, cold metadata refers to metadata other than hot metadata, or metadata with a low probability of being read. Although not compressing or writing cold metadata to memory will require disk access when reading it, the low probability of reading cold metadata means a low probability of disk access, and the impact on read efficiency is acceptable.

[0104] For example, embodiments of this application can periodically and dynamically determine whether a piece of metadata is hot metadata, and adjust the compressed metadata stored in memory based on the result of the dynamic determination. For instance, if a piece of metadata is converted from hot metadata to cold metadata, the compressed metadata obtained by compressing that metadata is deleted from memory. Or, if a piece of metadata is converted from cold metadata to hot metadata, the metadata is compressed, and the resulting compressed metadata is written into memory.

[0105] The above mainly describes the metadata reading process. Next, we will describe other metadata processing processes, including but not limited to the metadata writing process, the metadata merging process, and the metadata reconstruction (also known as playback) process.

[0106] Regarding the metadata writing process, relevant technologies first generate a write-ahead log (WAL) based on the metadata, and then write the WAL to disk via the file system. After the WAL is written, the metadata is then written to memory via the file system. However, when writing the WAL to disk via the file system, both the file system and the disk generate their own WALs. Therefore, the metadata WAL, the file system WAL, and the disk WAL must all be written to disk together before the metadata writing process can be executed. This not only increases the overhead of the writing process but also reduces writing efficiency.

[0107] In an exemplary embodiment, the apparatus provided in this application further includes: a third module, configured to write business data and WAL to a disk, generate metadata for indicating the business data, wherein the WAL corresponds to the metadata, and a WAL corresponding to a piece of metadata is generated based on the metadata. The first module 101 and the second module 102 are deployed on a file system, and the third module is deployed outside the file system.

[0108] Since the third module is deployed outside the file system, it does not need to write business data and the corresponding WAL (Write-Ahead Log) to disk through the file system. Instead, it can directly write the business data and the corresponding WAL to disk. Therefore, the aforementioned file system WAL and disk WAL are not generated, and there is no need to write the file system WAL and disk WAL to disk, avoiding the overlap of different WALs. Thus, after completing the writing of business data and the corresponding WAL to metadata, the metadata writing process can be performed, which not only reduces the overhead of the writing process and reduces write amplification, but also improves writing efficiency. Here, write amplification is the ratio of the third data volume to the fourth data volume. The fourth data volume is the amount of metadata to be written, while the third data volume is the total amount of data to be written in order to write the metadata. Since this embodiment does not require writing the file system WAL and disk WAL, the third data volume is reduced, thereby reducing write amplification.

[0109] In some implementations, the Write-Ahead Log (WAL) is appended to the disk after the business data is written to the disk; that is, the WAL is written to the disk along with the business data, or the WAL is appended to the end of the business data. In other implementations, if it is not possible to append the WAL to the disk, the WAL can be written separately after the business data is written to the disk. In this implementation, the WAL is no longer located at the end of the business data, and the disk can store different WALs contiguously, or in other words, the space occupied by each WAL on the disk is contiguous.

[0110] After writing business data and WAL to disk, the metadata writing process can be performed. This includes compressing the metadata as described above and writing the compressed metadata in full to memory, as well as writing metadata incrementally to memory. For example, memory can be divided into a first part and a second part. The first part stores the fully written compressed metadata, and this first part can be further divided into multiple partitions as described above. The second part stores the incrementally written metadata. When the second part is full, the metadata is flushed from the second part to disk. This not only enables persistent storage of metadata on disk but also frees up the second part of memory, allowing for continued incremental writing of metadata to the second part of memory.

[0111] like Figure 6 As shown, the second part of memory includes memory objects (mem table) and read-only memory objects (immutable mem table). After metadata is incrementally written to the second part of memory, the metadata is temporarily stored in the mem table. When one mem table is full, the system switches to another mem table, and the full mem table becomes an immutable mem table. When the immutable mem table is also full, the metadata is flushed from the immutable memtable to disk.

[0112] See also Figure 6 The disk contains multiple sets of metadata, one set of which may be, for example, a sorted string table (SST). SST is a block-based file format, which is not limited in this embodiment. When the disk includes SST, the metadata is flushed to the disk, that is, the metadata is stored as SST on the disk. For example, the disk may include multiple partitions, each partition including multiple sets of metadata, such as multiple SSTs. Different partitions may correspond independently to different mem tables and immutable mem tables, or they may share the same mem table and immutable mem table. The multiple partitions on the disk may correspond one-to-one with the multiple partitions in the first part of the memory mentioned above. For example, the number of partitions may be determined according to the number of storage pools (i.e., the aforementioned storage volumes), or it may be determined in other ways, which is not limited in this embodiment.

[0113] like Figure 7 As shown, Figure 7 An exemplary process for writing metadata is illustrated. This involves first writing business data and WAL to disk (i.e.,...). Figure 7 As shown in ①), metadata for indicating business data is then generated and written into the second part of memory (i.e., Figure 7 As shown in ②), this facilitates the subsequent flushing of metadata from the second part of memory to disk. Next, the metadata is compressed and written to the first part of memory (i.e.,...). Figure 7 As shown in ③), this ensures full storage of the compressed metadata.

[0114] Therefore, each write operation adds a Write-Ahead Log (WAL) to the disk. For example, 10 writes result in 10 WALs, with the first WAL corresponding to the first write operation and the tenth WAL corresponding to the tenth write operation. Furthermore, each write operation also adds metadata to the second part of memory. However, at any given moment, only a portion of the added metadata may be flushed to disk. For instance, after 10 writes, 10 metadata entries are added to the second part of memory; the first to fifth metadata entries (corresponding to the first to fifth write operations) have been flushed to disk, while the sixth to tenth metadata entries (corresponding to the sixth to tenth write operations) remain in the second part of memory.

[0115] Based on this, embodiments of this application can add a reference pointer at the last metadata that has been flushed to disk. This reference pointer points to the WAL corresponding to the first metadata that has not yet been flushed to disk, indicating the relationship between the metadata on disk and the WAL. For example, in the example above, since the first to fifth metadata have been flushed to disk, a reference pointer is added after the fifth metadata on disk. This reference pointer points to the sixth WAL corresponding to the sixth metadata. In embodiments of this application, this reference pointer can be used during the reconstruction of metadata, as detailed below.

[0116] Regarding the metadata reconstruction process, the apparatus provided in this application embodiment further includes: a fourth module, used to write metadata to disk, and in the case that the metadata in memory is cleared, to obtain the metadata and WAL in disk in parallel, to filter the metadata and WAL in disk obtained in parallel according to the priority of the metadata and WAL in disk, and to restore the metadata in memory according to the filtering result.

[0117] The process of writing metadata to disk has already been explained above and will not be repeated here. Since memory is used for non-persistent storage, the metadata in memory will be cleared after a power outage; that is, the metadata in both the first and second parts of memory will be cleared. Because the second part of memory is used for incremental metadata storage, the metadata in the second part of memory does not need to be rebuilt. Instead, new metadata only needs to be written to the second part of memory when new business data is subsequently written. However, because the first part of memory is used for full storage of compressed metadata, the compressed metadata in the first part of memory needs to be rebuilt.

[0118] like Figure 8As shown, the reconstruction process requires the use of metadata and WAL from the disk. As explained above, the WAL is generated based on the metadata; therefore, the WAL can be used to reconstruct the metadata. Among these... Figure 8 Two types of WALs are shown: the WAL at the end of the business data is the WAL appended as described above, and the consecutive WALs are the WALs written individually as described above. At least one of these two types of WALs can exist on the disk.

[0119] In this embodiment, metadata and WAL (Write-Ahead Log) from the disk are acquired in parallel. Compared to serial acquisition, parallel acquisition improves acquisition efficiency, thereby improving the efficiency of metadata reconstruction. For example, the number of tasks can be determined based on the number of partitions and the amount of business data on the disk. Each task is used to acquire metadata or a WAL from one disk, and multiple tasks are executed synchronously to achieve the aforementioned parallel acquisition. For example, this embodiment can combine the reference pointer described above to determine the relationship between metadata and WAL on the disk, thereby reasonably acquiring metadata or WAL from the disk. For instance, continuing with the example of the reference pointer above, if 10 tasks are established, 5 tasks are used to acquire the first to fifth metadata items from the disk, while the other 5 tasks are used to acquire the sixth to tenth WAL items from the disk. For example, in the case of block storage, the number of business data items is the same as the number of data blocks. In the case of object storage, the number of business data items is the same as the number of objects. In the case of file storage, the number of business data items is the same as the number of files.

[0120] However, duplicates may exist in the WAL and metadata acquired in parallel. For example, there may be duplicates between WALs and metadata, or between different WALs, or between different metadata sets. Taking WAL duplication as an example, a user might write the same business data twice. The first write uses the value 'm', and the second write uses the value 'n', effectively changing the value from 'm' to 'n'. Consequently, two WALs will be generated, which are duplicates because they are used to reconstruct the same metadata—metadata with the same identifier portion. Of course, to ensure data accuracy, the metadata should be reconstructed using the WAL corresponding to the second write, not the WAL corresponding to the first write.

[0121] It can be seen that when WAL and metadata are duplicated, they need to be filtered to ensure that the metadata reconstructed in memory based on the WAL and metadata is the latest metadata, and the business data indicated by the latest metadata is the latest business data. To this end, embodiments of this application set priorities for WAL and metadata to facilitate filtering based on priority. For example, for duplicate WAL and metadata, the higher-priority WAL is retained and its metadata is reconstructed, while the lower-priority metadata is ignored. As another example, for duplicate different WALs, the highest-priority WAL is retained and its metadata is reconstructed, while WALs with lower priorities are ignored. Yet another example, for duplicate different metadata, the highest-priority metadata is retained and its metadata is reconstructed in memory, while metadata with lower priorities is ignored.

[0122] In the exemplary embodiment, the Write-Ahead Log (WAL) has a higher priority than the metadata on the disk. This is because a WAL and new metadata are generated each time business data is written. As explained above, the WAL is written directly to the disk, while new metadata is first written to memory and then flushed to the disk only after memory is full. Therefore, the WAL on the disk is more comprehensive and can reflect the new metadata, but the metadata on the disk may not be comprehensive enough to reflect the new metadata because the new metadata may not have been flushed from memory to the disk yet. Therefore, this embodiment prioritizes the WAL over the metadata on the disk.

[0123] For example, business data on the disk comprises multiple data groups. For instance, in block storage, a data group represents a data block. In object storage, a data group represents an object. In file storage, a data group represents a file. WALs (Write-Ahead Allocation) include multiple WALs corresponding one-to-one with the data groups. The priority of any WAL is determined by the position of its corresponding data group within the multiple data groups. For example, a WAL is determined by a logical block address (LBA). A larger LBA indicates that the WAL was written to the disk later and that the corresponding data group is located further down the list of data groups; therefore, a larger LBA indicates a higher priority. Furthermore, different WALs can reside in chunks. A chunk is a contiguous block of storage space on the disk. A larger chunk identification (ID) indicates that the WAL was written to the disk later and that the corresponding data group is located further down the list of data groups; therefore, a larger chunk ID indicates a higher priority.

[0124] For example, the metadata in the disk includes metadata for multiple partitions, and each partition's metadata includes multiple sets of metadata, such as an SST. The priority of any set of metadata is determined by the position of that set within the partition in which it resides and the position of that partition within the multiple partitions. For instance, a set of metadata has a chunk ID; a larger chunk ID indicates that the metadata was written to the disk later and that it is located further down the partition, thus giving it higher priority. Additionally, a set of metadata also has a partition ID; a larger partition ID indicates that the metadata was written to the disk later and that it is located further down the partition, thus giving it higher priority.

[0125] For example, assuming there are 12 disks, such as solid-state drives (SSDs), each SSD includes 86 partitions, for a total of 1024 partitions. Based on a single SSD bandwidth of 3 gigabytes per second (GB / s), 12 SSDs provide 36 GB / s of bandwidth. The time required to rebuild 150 gigabytes (GB) of metadata in memory can be controlled within 5 seconds, which is less than the 10 seconds required for reconstruction in related technologies.

[0126] Regarding the metadata merging process, related technologies utilize multiple disk levels, merging metadata stored in adjacent levels, also known as data compaction. For example, when merging metadata stored in the first and second levels, the metadata stored in the first and second levels is read separately. After merging the read metadata (e.g., merging different metadata containing the same identifier), the merged metadata is obtained. This merged metadata is then written to the second level, freeing up the first level for subsequent flushing of metadata from memory to it. It's clear that this process requires both reading metadata from the disk and writing it back to disk after merging, resulting in read amplification and write amplification, increasing merging overhead and reducing merging efficiency.

[0127] For example, the apparatus provided in this application embodiment further includes: a fifth module, configured to, when the amount of metadata of the reference partition exceeds a threshold, retrieve the metadata corresponding to the reference partition from memory according to the identifier of the reference partition, and write the retrieved metadata to the reference partition to overwrite the metadata whose amount of data exceeds the threshold. Since this merging process directly performs an overwrite write, without needing to read metadata from the disk and merge it, read amplification and write amplification are avoided, resulting in lower overhead and higher efficiency in the merging process.

[0128] The metadata on the disk includes metadata from multiple partitions, and these partitions include a reference partition. This embodiment of the application makes it possible to merge metadata on a reference partition basis by dividing the memory and disk into multiple partitions. This merging method is a fine-grained, localized merging method. In other words, when metadata in a reference partition needs to be merged, there is no need to wait for other partitions; the metadata can be merged directly according to the requirements, offering high flexibility in merging.

[0129] In an exemplary embodiment, the data volume of the metadata of the reference partition exceeds a threshold. This could be due to the data volume of the metadata itself in the reference partition exceeding the threshold, or it could be due to the number of multiple sets of metadata in the reference partition exceeding a certain threshold, such as the number of SSTs exceeding the SST number threshold. This embodiment of the application does not limit this. Figure 9 As shown, when the metadata of the reference partition exceeds a threshold, the identifier of the reference partition can be obtained. Since there is a one-to-one correspondence between the multiple partitions on the disk and the multiple partitions included in the first part of memory, the reference partition can be determined from the multiple partitions included in the first part of memory based on its identifier. Therefore, the metadata corresponding to the reference partition can be obtained from the first part of memory. By directly writing the obtained metadata to the reference partition on the disk, the written metadata can overwrite the metadata in the reference partition whose data volume exceeds the threshold, meaning that the data volume of the metadata in the reference partition no longer exceeds the threshold.

[0130] As explained above, the first part of the memory stores the compressed metadata in its entirety. Therefore, the metadata corresponding to the reference partition obtained from the first part of the memory can refer to the compressed metadata. In this embodiment, the compressed metadata can be directly written to the reference partition on the disk, or the compressed metadata can be restored to its original state before being written to the disk. When restoring the compressed identifier portion, taking an example where the compressed identifier portion includes multiple levels of first characters, the multiple levels of first characters can be traversed to form the identifier portion. When restoring the compressed data portion, the three methods of obtaining the data described above can be referred to, and will not be elaborated here.

[0131] For example, during the process of overwriting metadata exceeding a threshold, writing metadata to the first part of memory is paused, and metadata is only written to the second part of memory. The reason for pausing writing to the first part of memory is that during the overwriting process, the metadata stored in the first part of memory needs to be written to a reference partition on disk. If new metadata continues to be written to the first part of memory during this process, it may cause data desynchronization and affect data consistency. After the overwriting process is completed, the metadata written to the second part of memory during the overwriting process is synchronized to the first part of memory. This ensures that the first part of memory stores the full amount of compressed metadata, and the metadata synchronized to the first part of memory will be written to the disk partition during the subsequent merging process.

[0132] For example, during the process of overwriting metadata exceeding a threshold, metadata is read from the second part of memory while reading metadata from the first part of memory is paused. This is because writing metadata to the first part of memory has been paused during the overwriting process. If the metadata to be read is metadata written during the overwriting process, it cannot be read from the first part of memory, affecting read efficiency. After the overwriting process ends, access to the first part of memory is resumed to read metadata from it. This is because after the overwriting process ends, the metadata written during the overwriting process is synchronized to the first part of memory, and the first part of memory resumes full storage of the compressed metadata, making reading from the first part of memory more efficient.

[0133] The foregoing described the various processes involved in processing metadata. These processes include modules 101, 102, 3, 4, and 5. The division of these modules into modules is merely illustrative and not intended to limit the scope of the embodiments described herein. In other words, in practical applications, the aforementioned functions can be assigned to different functional modules as needed to complete all or part of the functions described above.

[0134] For example, any module can be broken down into multiple sub-modules, which together implement the module's functionality. Alternatively, different modules can be combined, with the combined module performing the functions of the individual modules. Or, modules can be broken down and then combined, allowing for greater flexibility in the functionalities they can perform. For example, Figures 7 to 9The diagram shows four parts: a first part of memory, a second part of memory, a part on disk for storing metadata, and a part on disk for storing business data and WAL (Write-Ahead Log). In this embodiment, the first to fifth modules can be split and combined to obtain four modules corresponding one-to-one with the four parts. Each module implements the relevant functions of one of the parts corresponding to it.

[0135] In summary, the embodiments of this application store the compressed metadata in memory in its entirety, so that when querying the compressed metadata based on the received identifier portion, the data portion corresponding to the received identifier portion can be obtained and returned, without having to access the disk to query the metadata again, thereby reducing read amplification and effectively improving the metadata reading performance.

[0136] Furthermore, by writing WAL and business data to disk through a module deployed outside the file system, bypassing the file system, different WALs are avoided from overlapping. This allows for rapid WAL writing and quick transition to the metadata writing process, reducing write amplification and effectively improving metadata writing efficiency. During metadata merging, an overwrite write method effectively improves merging efficiency and reduces read and write amplification. During metadata reconstruction, metadata and WAL are retrieved from disk in parallel and filtered based on priority, effectively improving reconstruction efficiency. In this embodiment, the improvements in metadata read efficiency, write efficiency, merging efficiency, and reconstruction efficiency can all reach more than 100%.

[0137] The above describes the data processing apparatus provided in the embodiments of this application. Corresponding to the above apparatus, the embodiments of this application also provide a data processing method. Exemplarily, this method can be applied to... Figure 1 The data processing apparatus shown can also be applied to computer equipment; however, this application does not limit its application to this. Figure 10 As shown, the data processing method provided in this application embodiment includes the following steps 1001 and 1002.

[0138] Step 1001, Receive the identifier section.

[0139] Step 1002: Query the compressed metadata based on the received identifier portion to obtain the data portion corresponding to the received identifier portion, and return the obtained data portion. The compressed metadata is obtained by compressing the corresponding identifier portion and data portion in the metadata. The compressed metadata is stored in memory in its entirety.

[0140] In an exemplary embodiment, the compressed identifier portion corresponds to a first pointer, which is used to point to the compressed data portion. Querying the compressed metadata based on the received identifier portion to obtain the data portion corresponding to the received identifier portion includes: querying the compressed identifier portion based on the received identifier portion to obtain a reference identifier portion, which matches the received identifier portion; and obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion.

[0141] In one implementation, the compressed data portion includes multi-level characters. A second pointer corresponds to the lower-level character in two adjacent character levels, pointing to the higher-level character. A first pointer points to the lowest-level character in the multi-level characters. Obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion includes: determining the lowest-level character pointed to by the first pointer of the reference identifier portion; querying upwards level by level according to the second pointer corresponding to the lowest-level character until the highest-level character in the multi-level characters is obtained; and using the data portion composed of the lowest-level character to the highest-level character as the data portion corresponding to the received identifier portion.

[0142] In another embodiment, the compressed data portion includes a data portion group, which is obtained by compressing multiple data portions. A reference identifier portion also corresponds to a data portion identifier, which is used to indicate one of the multiple data portions. Obtaining the data portion corresponding to the received identifier portion based on a first pointer corresponding to the reference identifier portion includes: determining the data portion group pointed to by the first pointer corresponding to the reference identifier portion; decompressing the data portion group to obtain multiple data portions; and selecting the data portion indicated by the data portion identifier corresponding to the reference identifier portion as the data portion corresponding to the received identifier portion.

[0143] In another embodiment, the compressed data portion is index information, which is used to point to the data portion on the disk. Obtaining the data portion corresponding to the received identifier portion based on the first pointer corresponding to the reference identifier portion includes: determining the index information pointed to by the first pointer corresponding to the reference identifier portion, and using the data portion on the disk pointed to by the index information as the data portion corresponding to the received identifier portion.

[0144] For example, the metadata is hot metadata. Hot metadata is metadata used to indicate hot business data, or hot metadata is metadata located in an active volume, which is a volume on disk that is accessed more frequently than a threshold.

[0145] In an exemplary embodiment, the multilevel characters reside in a memory space, which includes at least one memory page and has a starting address. The second pointer corresponding to the non-highest-level character in the multilevel characters includes the offset of the character above the non-highest-level character relative to the starting address. The second pointer corresponding to the lowest-level character in the multilevel characters further includes the starting address.

[0146] For example, embodiments of this application also provide a metadata writing process, the method further including: writing business data and WAL to disk, generating metadata for indicating business data, wherein the WAL corresponds to the metadata.

[0147] This application embodiment also provides a metadata reconstruction process, the method further includes: writing metadata to disk, obtaining metadata and WAL in disk in parallel when the metadata in memory is cleared, filtering the metadata and WAL in disk obtained in parallel according to the priority of the metadata and WAL in disk, and restoring the metadata in memory according to the filtering results.

[0148] In an exemplary embodiment, the metadata in the disk includes metadata for multiple partitions, and the metadata for each partition includes multiple sets of metadata. The priority of any set of metadata is determined based on the position of any set within the partition in which it is located and the position of the partition in which it is located among multiple partitions.

[0149] In addition, the business data on the disk includes multiple data groups, and the WAL includes multiple WALs that correspond one-to-one with the multiple data groups. The priority of any WAL is determined according to the position of the data group corresponding to any WAL in the multiple data groups.

[0150] In an exemplary embodiment, this application also provides a metadata merging process. The metadata in the disk includes metadata of multiple partitions, and the multiple partitions include a reference partition. The method further includes: when the amount of metadata of the reference partition exceeds a threshold, retrieving the metadata corresponding to the reference partition from memory according to the identifier of the reference partition, and writing the retrieved metadata to the reference partition to overwrite the metadata whose amount of data exceeds the threshold.

[0151] It should be understood that the above Figure 10 The beneficial effects of the method shown are, and Figure 1 The data processing apparatus shown has the same beneficial effects in achieving its function. Furthermore, this method embodiment and the apparatus provided in the above embodiments belong to the same concept; their specific implementation process is detailed in the method embodiment and will not be repeated here.

[0152] Additionally, embodiments of this application provide a data processing device, which includes a memory and a processor; the memory stores at least one computer instruction, which is loaded and executed by the processor to enable the data processing device to perform its functions. Figure 10 The method shown is for processing data.

[0153] See Figure 11 , Figure 11 A schematic diagram of the structure of an exemplary data processing device 1100 of this application is shown. The data processing device 1100 includes at least one processor 1101, a memory 1103, and at least one network interface 1104.

[0154] Processor 1101 may be, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs), programmable logic devices (PLDs), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, or any combination thereof for implementing the scheme of this application. A PLD may be, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor supporting an advanced reduced instruction set machine (RISC) machine (ARM) architecture. It can implement or execute various logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0155] Optionally, the data processing device 1100 also includes a bus 1102. The bus 1102 is used to transfer information between the various components of the data processing device 1100. The bus 1102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 1102 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0156] The memory 1103 may be, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache.

[0157] By way of example, but not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0158] The memory 1103 may also be other types of storage devices capable of storing static information and instructions. Alternatively, it may be other types of dynamic storage devices capable of storing information and instructions. It may also be other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 1103 may exist independently, for example, and be connected to the processor 1101 via bus 1102. The memory 1103 may also be integrated with the processor 1101.

[0159] Network interface 1104 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), or wireless local area network (WLAN). Network interface 1104 may include wired network interfaces and wireless network interfaces. Specifically, network interface 1104 can be an Ethernet interface, such as Fast Ethernet (FE), Gigabit Ethernet (GE), Asynchronous Transfer Mode (ATM), WLAN, cellular network, or combinations thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of this application, network interface 1104 can be used by data processing device 1100 to communicate with other devices.

[0160] In specific implementations, as some embodiments, the processor 1101 may include one or more CPUs, such as Figure 11 The CPU0 and CPU1 shown are examples of processors. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0161] In specific implementations, as some embodiments, the data processing device 1100 may include multiple processors, such as... Figure 11 The processors 1101 and 1105 shown are illustrated. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0162] In some embodiments, memory 1103 is used to store program instructions 1110 for executing the scheme of this application, and processor 1101 can execute the program instructions 1110 stored in memory 1103. That is, data processing device 1100 can implement the method provided in the method embodiment through processor 1101 and program instructions 1110 in memory 1103, i.e. Figure 10 The method is illustrated. Program instructions 1110 may include one or more software modules. Optionally, processor 1101 itself may also store program instructions for executing the scheme of this application.

[0163] In specific implementation, the data processing device 1100 of this application can correspond to a computer device for performing the above-described method. The processor 1101 in the data processing device 1100 reads instructions from the memory 1103, causing... Figure 11 The data processing device 1100 shown is capable of performing all or part of the steps in the method embodiments.

[0164] The data processing device 1100 can also correspond to the above. Figure 1 The device shown, Figure 1 Each functional module in the illustrated device is implemented using software from the data processing device 1100. In other words, Figure 1 The device shown includes a functional module that is generated by the processor 1101 of the data processing device 1100 after reading the program instructions 1110 stored in the memory 1103.

[0165] in, Figure 10 Each step of the method shown is accomplished through integrated logic circuits in the hardware or instructions in the software form of the processor in the data processing device 1100. The steps of the method embodiments disclosed in this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media mature in the art. Since this storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method embodiments; to avoid repetition, they will not be described in detail here.

[0166] This application provides a computer program or computer program product, which includes: computer instructions that, when executed by a computer, cause the computer to perform the above-described... Figure 10 The method shown is for processing data.

[0167] For example, embodiments of this application provide a computer-readable storage medium that stores computer instructions, which, when executed on a computer, [are executed]. Figure 10 The data processing method shown is executed by a computer.

[0168] This application provides a chip, including a processor, for calling and executing instructions stored in memory, causing a computer equipped with the chip to perform the aforementioned operations. Figure 10 The method shown is for processing data.

[0169] This application also provides another chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through internal interconnection paths. The processor is used to execute code in the memory. When the code is executed, a computer with the chip installed performs the above-described actions. Figure 10 The method shown is for processing data.

[0170] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0171] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0172] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0173] In this application, the term "at least one" means one or more, and the term "multiple" means two or more; for example, multiple partitions means two or more partitions. The terms "system" and "network" are often used interchangeably in this document.

[0174] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0175] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0176] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0177] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. An apparatus for processing data, the apparatus comprising: The device comprises a first module and a second module, wherein, the first module is configured to receive an identification part; the second module is configured to query a compressed identification part in compressed metadata according to the received identification part to obtain a reference identification part, the reference identification part matches the received identification part, the compressed metadata is obtained by compressing a corresponding identification part and a data part in metadata, the compressed metadata is stored in a memory in full, and a compressed data part in the compressed metadata comprises multi-level characters, a lower level character in adjacent two levels of characters corresponds to a second pointer, and the second pointer is used to point to a higher level character in the adjacent two levels of characters; the second module is further configured to determine a lowest level character in the multi-level characters pointed to by a first pointer corresponding to the reference identification part, obtain a data part corresponding to the received identification part according to a second pointer corresponding to the lowest level character, and the obtained data part comprises the lowest level character to a highest level character in the multi-level characters; the second module is further configured to return the obtained data part.

2. The apparatus of claim 1, wherein, The device further comprises: a third module configured to write business data and a write-ahead log (WAL) into a disk, generate the metadata used to indicate the business data, and the WAL corresponds to the metadata, the first module and the second module are deployed on a file system, and the third module is deployed outside the file system.

3. The apparatus of claim 2, wherein, The device further comprises: a fourth module configured to write the metadata into the disk, acquire the metadata in the disk and the WAL in parallel in a case where the metadata in the memory is emptied, filter the metadata in the disk and the WAL acquired in parallel according to priorities of the metadata in the disk and the WAL, and restore the metadata in the memory according to a result obtained by filtering.

4. The apparatus of claim 3, wherein, The metadata in the disk comprises metadata of multiple partitions, each partition of metadata comprises multiple groups of metadata, and a priority of any group of metadata is determined according to a position of the any group in a partition where the any group is located and a position of the partition where the any group is located in the multiple partitions.

5. The apparatus of claim 3 or 4, wherein, The business data in the disk comprises multiple data groups, the WAL comprises multiple WALs corresponding to the multiple data groups one by one, and a priority of any WAL is determined according to a position of a data group corresponding to the any WAL in the multiple data groups.

6. The apparatus of any one of claims 1-4, wherein, The metadata in the disk comprises metadata of multiple partitions, the multiple partitions comprise a reference partition, and the device further comprises: a fifth module configured to acquire, in a case where a data amount of the metadata of the reference partition exceeds a threshold value, metadata corresponding to the reference partition from the memory according to an identification of the reference partition, and write the acquired metadata into the reference partition to cover the metadata whose data amount exceeds the threshold value.

7. The apparatus of claim 5, wherein, The metadata in the disk comprises metadata of multiple partitions, the multiple partitions comprise a reference partition, and the device further comprises: A fifth module is configured to, when the data amount of the metadata of the reference partition exceeds the threshold, acquire the metadata corresponding to the reference partition from the memory according to the identification of the reference partition, and write the acquired metadata into the reference partition to overwrite the metadata whose data amount exceeds the threshold.

8. The apparatus of any one of claims 1-4, 7, wherein, The second module is configured to query upwards level by level according to the second pointer corresponding to the lowest level character until a highest level character in the multi-level characters is obtained, and take a data part composed of the lowest level character to the highest level character as a data part corresponding to the received identification part.

9. The apparatus of claim 5, wherein, The second module is configured to query upwards level by level according to the second pointer corresponding to the lowest level character until a highest level character in the multi-level characters is obtained, and take a data part composed of the lowest level character to the highest level character as a data part corresponding to the received identification part.

10. The apparatus of claim 6, wherein, The second module is configured to query upwards level by level according to the second pointer corresponding to the lowest level character until a highest level character in the multi-level characters is obtained, and take a data part composed of the lowest level character to the highest level character as a data part corresponding to the received identification part.

11. The apparatus of any one of claims 1-4, 7, 9-10, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, and the reference identification part further corresponds to a data part identification used for indicating one data part in the plurality of data parts. The second module is further configured to determine a data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain the plurality of data parts, and take one data part indicated by the data part identification corresponding to the reference identification part as a data part corresponding to the received identification part in the plurality of data parts.

12. The apparatus of claim 5, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, and the reference identification part further corresponds to a data part identification used for indicating one data part in the plurality of data parts. The second module is further configured to determine a data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain the plurality of data parts, and take one data part indicated by the data part identification corresponding to the reference identification part as a data part corresponding to the received identification part in the plurality of data parts.

13. The apparatus of claim 6, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, and the reference identification part further corresponds to a data part identification used for indicating one data part in the plurality of data parts. The second module is further configured to determine a data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain the plurality of data parts, and identify one data part indicated by the data part identification corresponding to the reference identification part as the data part corresponding to the received identification part.

14. The apparatus of claim 8, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, and the reference identification part further corresponds to a data part identification used to indicate one data part in the plurality of data parts. The second module is further configured to determine a data part group pointed to by the first pointer corresponding to the reference identification part, decompress the data part group to obtain the plurality of data parts, and identify one data part indicated by the data part identification corresponding to the reference identification part as the data part corresponding to the received identification part.

15. The apparatus of any one of claims 1-4, 7, 9-10, 12-14, wherein, The compressed data part further comprises index information used to point to a data part in a disk. The second module is further configured to determine index information pointed to by the first pointer corresponding to the reference identification part, and identify the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

16. The apparatus of claim 5, wherein, The compressed data part is index information used to point to a data part in a disk. The second module is configured to determine index information pointed to by the first pointer corresponding to the reference identification part, and identify the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

17. The apparatus of claim 6, wherein, The compressed data part further comprises index information used to point to a data part in a disk. The second module is further configured to determine index information pointed to by the first pointer corresponding to the reference identification part, and identify the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

18. The apparatus of claim 8, wherein, The compressed data part further comprises index information used to point to a data part in a disk. The second module is further configured to determine index information pointed to by the first pointer corresponding to the reference identification part, and identify the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

19. The apparatus of claim 11, wherein, The compressed data part further comprises index information used to point to a data part in a disk. The second module is further configured to determine index information pointed to by the first pointer corresponding to the reference identification part, and identify the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

20. The apparatus of any one of claims 1-4, 7, 9-10, 12-14, 16-19, wherein, The metadata is hot metadata. The hot metadata is metadata used to indicate hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

21. The apparatus of claim 5, wherein, The metadata is hot metadata. The hot metadata is metadata used to indicate hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, the volume in the working state being a volume in the disk with a frequency of access exceeding a threshold.

22. The apparatus of claim 6, wherein, The metadata is hot metadata; The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, the volume in the working state being a volume in the disk with a frequency of access exceeding a threshold.

23. The apparatus of claim 8, wherein, The metadata is hot metadata; The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, the volume in the working state being a volume in the disk with a frequency of access exceeding a threshold.

24. The apparatus of claim 11, wherein, The metadata is hot metadata; The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, the volume in the working state being a volume in the disk with a frequency of access exceeding a threshold.

25. The apparatus of claim 15, wherein, The metadata is hot metadata; The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, the volume in the working state being a volume in the disk with a frequency of access exceeding a threshold.

26. The apparatus of any one of claims 1-4, 7, 9-10, 12-14, 16-19, 21-25, wherein, The multi-level character is located in a memory space in the memory, the memory space including at least one memory page, the memory space having a start address; The second pointer corresponding to a non-highest level character in the multi-level character includes an offset of a higher level character of the non-highest level character relative to the start address; The second pointer corresponding to a lowest level character in the multi-level character further includes the start address.

27. The apparatus of claim 5, wherein, The multi-level character is located in a memory space in the memory, the memory space including at least one memory page, the memory space having a start address; The second pointer corresponding to a non-highest level character in the multi-level character includes an offset of a higher level character of the non-highest level character relative to the start address; The second pointer corresponding to a lowest level character in the multi-level character further includes the start address.

28. The apparatus of claim 6, wherein, The multi-level character is located in a memory space in the memory, the memory space including at least one memory page, the memory space having a start address; The second pointer corresponding to a non-highest level character in the multi-level character includes an offset of a higher level character of the non-highest level character relative to the start address; The second pointer corresponding to a lowest level character in the multi-level character further includes the start address.

29. The apparatus of claim 8, wherein, The multi-level character is located in a memory space in the memory, the memory space including at least one memory page, the memory space having a start address; The second pointer corresponding to a non-highest level character in the multi-level character includes an offset of a higher level character of the non-highest level character relative to the start address; The second pointer corresponding to a lowest level character in the multi-level character further includes the start address.

30. The apparatus of claim 11, wherein, The multi-level character is located in a memory space in the memory, the memory space including at least one memory page, the memory space having a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

31. The apparatus of claim 15, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

32. The apparatus of claim 20, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

33. A method of processing data, characterized by, The method comprises: receiving an identification part; querying a compressed identification part in compressed metadata according to the received identification part to obtain a reference identification part, the reference identification part matches the received identification part, the compressed metadata is obtained by compressing a corresponding identification part and a data part in metadata, the full amount of the compressed metadata is stored in a memory, and a compressed data part in the compressed metadata comprises multi-level characters, a second pointer corresponding to a character of a lower level in adjacent two levels of characters, the second pointer is used to point to a character of a higher level in the adjacent two levels of characters; determining a lowest level character in the multi-level characters pointed to by a first pointer corresponding to the reference identification part, obtaining a data part corresponding to the received identification part according to a second pointer corresponding to the lowest level character, and the obtained data part comprises the lowest level character to a highest level character in the multi-level characters; returning the obtained data part.

34. The method of claim 33, wherein, The method further comprises: writing business data and a write-ahead log (WAL) into a disk to generate the metadata used to indicate the business data, and the WAL corresponds to the metadata.

35. The method of claim 34, wherein, The method further comprises: writing the metadata into the disk, in the case that metadata in the memory is emptied, obtaining the metadata in the disk and the WAL in parallel, screening the metadata in the disk and the WAL obtained in parallel according to priorities of the metadata in the disk and the WAL, and restoring the metadata in the memory according to a result obtained by screening.

36. The method of claim 35, wherein, The metadata in the disk comprises metadata of a plurality of partitions, metadata of each partition comprises a plurality of groups of metadata, and a priority of any group of metadata is determined according to a position of the any group in a partition where the any group is located and a position of the partition where the any group is located in the plurality of partitions.

37. The method of claim 35 or 36, wherein, The service data in the magnetic disk comprises a plurality of data groups, the WAL comprises a plurality of WALS corresponding to the plurality of data groups, and the priority of any one WAL is determined according to the position of the data group corresponding to the any one WAL in the plurality of data groups.

38. The method of any one of claims 33-36, wherein, The metadata in the magnetic disk comprises a plurality of partitioned metadata, the plurality of partitions comprises a reference partition, and the method further comprises: In a case where the data amount of the metadata of the reference partition exceeds a threshold, obtaining the metadata corresponding to the reference partition from the memory according to the identifier of the reference partition, and writing the obtained metadata into the reference partition to overwrite the metadata whose data amount exceeds the threshold.

39. The method of claim 37, wherein, The metadata in the magnetic disk comprises a plurality of partitioned metadata, the plurality of partitions comprises a reference partition, and the method further comprises: In a case where the data amount of the metadata of the reference partition exceeds a threshold, obtaining the metadata corresponding to the reference partition from the memory according to the identifier of the reference partition, and writing the obtained metadata into the reference partition to overwrite the metadata whose data amount exceeds the threshold.

40. The method of any one of claims 33-36, 39, wherein, The data part corresponding to the received identifier part is obtained according to the second pointer corresponding to the lowest level character, comprising: The data part corresponding to the received identifier part is obtained according to the second pointer corresponding to the lowest level character, comprising:

41. The method of claim 37, wherein, The data part corresponding to the received identifier part is obtained according to the second pointer corresponding to the lowest level character, comprising: The data part corresponding to the received identifier part is obtained according to the second pointer corresponding to the lowest level character, comprising:

42. The method of claim 38, wherein, The data part corresponding to the received identifier part is obtained according to the second pointer corresponding to the lowest level character, comprising: The compressed data part further comprises a data part group, the data part group is obtained by compressing a plurality of data parts, the reference identifier part further corresponds to a data part identifier, the data part identifier is used to indicate one data part in the plurality of data parts, and the method further comprises:

43. The method of any one of claims 33-36, 39, 41-42, wherein, The data part group pointed to by the first pointer corresponding to the reference identifier part is determined, the data part group is decompressed to obtain the plurality of data parts, and in the plurality of data parts, one data part indicated by the data part identifier corresponding to the reference identifier part is taken as the data part corresponding to the received identifier part. ​ 44. The method of claim 37, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, the reference identification part further corresponds to a data part identification, the data part identification is used to indicate one data part in the plurality of data parts, and the method further comprises: determining the data part group pointed to by the first pointer corresponding to the reference identification part, decompressing the data part group to obtain the plurality of data parts, and taking one data part indicated by the data part identification corresponding to the reference identification part as the data part corresponding to the received identification part in the plurality of data parts.

45. The method of claim 38, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, the reference identification part further corresponds to a data part identification, the data part identification is used to indicate one data part in the plurality of data parts, and the method further comprises: determining the data part group pointed to by the first pointer corresponding to the reference identification part, decompressing the data part group to obtain the plurality of data parts, and taking one data part indicated by the data part identification corresponding to the reference identification part as the data part corresponding to the received identification part in the plurality of data parts.

46. The method of claim 40, wherein, The compressed data part further comprises a data part group obtained by compressing a plurality of data parts, the reference identification part further corresponds to a data part identification, the data part identification is used to indicate one data part in the plurality of data parts, and the method further comprises: determining the data part group pointed to by the first pointer corresponding to the reference identification part, decompressing the data part group to obtain the plurality of data parts, and taking one data part indicated by the data part identification corresponding to the reference identification part as the data part corresponding to the received identification part in the plurality of data parts.

47. The method of any one of claims 33-36, 39, 41-42, 44-46, wherein, The compressed data part further comprises index information used to point to a data part in a disk, and the method further comprises: determining the index information pointed to by the first pointer corresponding to the reference identification part, and taking the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

48. The method of claim 37, wherein, The compressed data part further comprises index information used to point to a data part in a disk, and the method further comprises: determining the index information pointed to by the first pointer corresponding to the reference identification part, and taking the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

49. The method of claim 38, wherein, The compressed data part further comprises index information used to point to a data part in a disk, and the method further comprises: determining the index information pointed to by the first pointer corresponding to the reference identification part, and taking the data part in the disk pointed to by the index information as the data part corresponding to the received identification part.

50. The method of claim 40, wherein, The compressed data part further comprises index information used to point to a data part in a disk, and the method further comprises: determining the index information pointed to by the first pointer corresponding to the reference identification part, and taking the data part in the disk pointed to by the index information as the data part corresponding to the received identification part. The index information pointed by the first pointer corresponding to the reference identification part is determined, and a data part in the disk pointed by the index information is taken as a data part corresponding to the received identification part.

51. The method of claim 43, wherein, The compressed data part further comprises index information used for pointing to a data part in the disk, and the method further comprises: The index information pointed by the first pointer corresponding to the reference identification part is determined, and a data part in the disk pointed by the index information is taken as a data part corresponding to the received identification part.

52. The method of any one of claims 33-36, 39, 41-42, 44-46, 48-51, wherein, The metadata is hot metadata. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

53. The method of claim 37, wherein, The metadata is hot metadata. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

54. The method of claim 38, wherein, The metadata is hot metadata. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

55. The method of claim 40, wherein, The metadata is hot metadata. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

56. The method of claim 43, wherein, The metadata is hot metadata. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

57. The method of claim 47, wherein, The metadata is hot metadata. The hot metadata is metadata used for indicating hot service data, or the hot metadata is metadata located in a volume in a working state, and the volume in the working state is a volume in the disk with an access frequency exceeding a threshold.

58. The method of any one of claims 33-36, 39, 41-42, 44-46, 48-51, 53-57, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to a non-highest-level character in the multi-level character comprises an offset of a higher-level character of the non-highest-level character relative to the start address; The second pointer corresponding to a lowest-level character in the multi-level character further comprises the start address.

59. The method of claim 37, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to a non-highest-level character in the multi-level character comprises an offset of a higher-level character of the non-highest-level character relative to the start address; The second pointer corresponding to a lowest-level character in the multi-level character further comprises the start address.

60. The method of claim 38, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

61. The method of claim 40, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

62. The method of claim 43, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

63. The method of claim 47, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

64. The method of claim 52, wherein, The multi-level character is located in a memory space in the memory, the memory space comprises at least one memory page, and the memory space has a start address; The second pointer corresponding to the character of the non-highest level in the multi-level character comprises: an offset of the character of the next level relative to the start address; The second pointer corresponding to the character of the lowest level in the multi-level character further comprises: the start address.

65. An apparatus for processing data, the apparatus comprising: The device comprises a memory and a processor; the memory stores at least one computer instruction, the at least one computer instruction is loaded and executed by the processor, so that the device for processing data implements the method for processing data in any one of claims 33-64.

66. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer instruction, the at least one computer instruction is loaded and executed by the processor, so that the computer implements the method for processing data in any one of claims 33-64.

67. A computer program product, characterised in that, The computer program product comprises computer instructions, which are executed by the processor to make the computer implement the method for processing data in any one of claims 33-64.

Images