Metadata writing method and device and metadata reading method and device

Abstract

The invention provides a metadata writing method and device and a metadata reading method and device, and relates to the technical field of cloud computing, in particular to the technical field of storage. A specific implementation mode of the metadata writing method comprises the steps that a metadata writing request of a file is received, and the metadata writing request comprises metadata of the file; writing the metadata into a log file; writing the index node identifier and the secondary index address of the file into a primary index memory to generate a primary index of the file; and writing the name of the file, the log file address of the metadata and the metadata into a secondary index memory to generate a secondary index of the file.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing technology, in particular to the field of storage technology. Background Art

[0002] In today's era of cloud computing and big data, efficient and scalable storage systems have become the key cornerstone. Among them, LSM-Tree (Log-Structured-Merge-Tree) is an efficient KV (Key Value) storage system. Value, key-value) storage structure, which is widely used in the industry.

[0003] Currently, the LSM-Tree-based KV storage engine uses a single-machine storage method to store metadata of files in a distributed file system, resulting in low storage efficiency. Summary of the Invention

[0004] The embodiments of the present disclosure provide a metadata writing and metadata reading method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a metadata writing method, including: receiving a metadata writing request for a file, wherein the metadata writing request includes the metadata of the file; writing the metadata into a log file; writing the index node identifier and the secondary index address of the file into a primary index memory to generate a primary index of the file; writing the name of the file, the log file address of the metadata, and the metadata into a secondary index memory to generate a secondary index of the file.

[0006] In a second aspect, an embodiment of the present disclosure proposes a metadata reading method, including: receiving a metadata reading request for a file, wherein the metadata reading request includes an index node identifier of the file; querying a primary index memory based on the index node identifier to obtain a secondary index address of the file; querying a secondary index memory based on the secondary index address to obtain a metadata reading result of the file.

[0007] In a third aspect, an embodiment of the present disclosure proposes a metadata writing device, comprising: a receiving module, configured to receive a metadata writing request for a file, wherein the metadata writing request includes the metadata of the file; a first writing module, configured to write the metadata into a log file; a second writing module, configured to write the index node identifier and the secondary index address of the file into a primary index memory to generate a primary index of the file; a third writing module, configured to write the file name, the metadata log file address and the metadata into the secondary index memory to generate a secondary index of the file.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a metadata reading method, including: a receiving module, configured to receive a metadata reading request for a file, wherein the metadata reading request includes an index node identifier of the file; a first query module, configured to query a primary index memory based on the index node identifier to obtain a secondary index address of the file; and a first query module, configured to query a secondary index memory based on the secondary index address to obtain a metadata reading result of the file.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect or the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the method described in the first aspect or the second aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect or the second aspect when executed by a processor.

[0012] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects, and advantages of the present disclosure will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. Among them: Figure 1 is a flowchart of an embodiment of a metadata writing method according to the present disclosure; Figure 2 is a flowchart of another embodiment of a metadata writing method according to the present disclosure; Figure 3 is a flowchart of an embodiment of a metadata reading method according to the present disclosure; Figure 4 is a flowchart of another embodiment of the metadata reading method according to the present disclosure; Figure 5 It is a structural diagram of the three-level indexing mechanism; Figure 6 is a structural diagram of an embodiment of a metadata writing device according to the present disclosure; Figure 7 is a structural diagram of an embodiment of a metadata reading device according to the present disclosure; Figure 8 It is a block diagram of an electronic device used to implement the metadata writing and metadata reading methods of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0015] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0016] Figure 1 A process 100 of an embodiment of a metadata writing method according to the present disclosure is shown. The metadata writing method includes the following steps: Step 101: Receive a metadata writing request for a file.

[0017] In this embodiment, the execution subject of the metadata writing method may receive a metadata writing request for a file.

[0018] The metadata writing method is typically executed by a server. The server can be either hardware or software. If the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers or as a single server. If the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services) or as a single software program or software module. This is not specifically limited here.

[0019] The metadata write request may include file metadata. The file may be a file in a distributed file system. Metadata may be system data describing file characteristics, including but not limited to: file type, permissions, number of connections, file size, timestamp, access control list, etc.

[0020] Step 102: Write the metadata into a log file.

[0021] In this embodiment, the execution entity may write the metadata into a log file, which is a temporary file that can be used to record user operation information, including but not limited to write operations, read operations, delete operations, and update operations.

[0022] Step 103: Write the file's index node identifier and secondary index address into the primary index memory to generate a primary index for the file.

[0023] In this embodiment, the execution entity may write the inode identifier and the secondary index address of the file into the primary index memory to generate the primary index of the file.

[0024] The primary index resides in memory, and its storage format can be, for example, inode => secondary index address. The primary index requires a mapping from inode to secondary index, and since it is entirely in memory, it can use a common hash storage structure. The primary index can be persisted using a common hash structure, and its recovery solution can adopt a common design of checkpoints plus log playback. Index snapshots are read from the checkpoint file, loaded into memory, and then the log is played back. The log is recorded when users write data.

[0025] Step 104 : Write the file name, metadata log file address, and metadata into the secondary index memory to generate a secondary index for the file.

[0026] In this embodiment, the execution entity may write the file name, the metadata log file address, and the metadata into the secondary index memory to generate a secondary index for the file.

[0027] Typically, a key-value (KV) is inserted into the secondary index memory. The key in the KV can be the file name, and the value in the KV can be the log file address and metadata. The secondary index storage format can be, for example, a mapping of name => metadata and data storage path (name => [data block offset, inode, type, and other necessary information]).

[0028] Secondary indexes can use a tree structure for ordered storage. Secondary indexes are not persisted to disk and are restored asynchronously using a lazy method. The recovery process is as follows: First, start an asynchronous thread to traverse the first-level index in sequence and restore the corresponding second-level index in sequence.

[0029] Afterwards, secondary indexes with high popularity are restored first.

[0030] Then, all child nodes under the parent index node are scanned from RocksDB and the index structure is constructed in memory.

[0031] Finally, the data is replayed from the log, the index structure is constructed in memory, and the recovery is completed.

[0032] In some embodiments, a three-level indexing mechanism is used to store metadata. First, the file's inode identifier and secondary index address are written to the primary index memory to generate the file's primary index. Then, the file's name, metadata log file address, and metadata are written to the secondary index memory to generate the file's secondary index. Finally, the file's name, metadata log file address, and metadata are written to the tertiary index memory to generate the file's tertiary index. The three-level index can utilize a key-value storage engine, writing key metadata information to the key-value storage engine of the three-level index memory and writing metadata to the disk of the three-level index memory. The key-value storage engine can be, for example, RocksDB. RocksDB only stores key metadata information, which takes up approximately 24 bytes, thereby reducing the amount of data stored in RocksDB. Its storage format can be, for example: name => [datablock offset, inode, type, and other necessary information]. The full metadata on disk is not scanned, thus reducing to a point-by-point read design. The full metadata is not stored in RocksDB, but on a separate disk. For example, the size of online metadata is 200 to 500 bytes, so the metadata is directly stored at an address on the block device (a single block is 4 KB, and multiple full metadata items can be stored in one block).

[0033] In some embodiments, when a three-level index mechanism is used to store metadata, secondary indexes are degraded based on the memory usage of the secondary index memory. For example, if the secondary index memory is sufficient, all secondary indexes are stored in the secondary index memory, and the secondary indexes store the full metadata. If the memory usage reaches a memory usage threshold (e.g., n%, where n is a variable calculated by the total memory of a single machine and the average throughput over a period of time), all secondary indexes are stored in the secondary index memory, and the secondary indexes only store key metadata information (inode, pool, type, etc.). Optionally, all metadata that has not been used for a long time is asynchronously deleted. If the secondary index memory is tight (e.g., when the remaining memory is less than m GB, where m is calculated based on actual business pressure and machine configuration), only hot secondary indexes are stored in the secondary index memory, and hot secondary indexes only store key metadata information, and cold secondary indexes are asynchronously deprecated.

[0034] It's important to note that asynchronous flushing of log files can be enabled in the backend. To speed up flushing, each table can have its own log file, enabling concurrent flushing. Before the log file is flushed (the buffer is refreshed), the full in-memory metadata for the secondary index cannot be eliminated. (In extreme cases, it can be eliminated, and metadata can be read from the log file during reading.) After the log file is flushed, based on memory usage, only necessary metadata is stored in memory.

[0035] In some embodiments, upon receiving a metadata deletion request for a file, the secondary index of the file may be deleted from the secondary index memory. First, the deletion of the file's metadata is recorded in a log file; then, the secondary index of the file is marked for deletion from the secondary index memory; and finally, after the deletion log file is flushed, the secondary index of the file is deleted from the secondary index memory.

[0036] In some embodiments, storing all copies in the secondary index memory would waste memory space. To improve memory utilization, all copies can be stored in the index memory, with only some copies stored in the secondary index memory. For example, if there are three copies, all three copies can be stored in the memory, with only two copies storing the secondary index in memory, and the other copy not storing the secondary index in memory.

[0037] The disclosed embodiment proposes a new directory tree index structure that can support both a single machine scale of tens of billions and sequential traversal.

[0038] Continue to refer Figure 2 , which shows a process 200 of another embodiment of the metadata writing method according to the present disclosure. The metadata writing method includes the following steps: Step 201: Receive a metadata writing request for a file.

[0039] Step 202: Write the metadata into the log file.

[0040] Step 203: Write the file's index node identifier and secondary index address into the primary index memory to generate a primary index for the file.

[0041] Step 204: Write the file name, metadata log file address, and metadata into the secondary index memory to generate a secondary index for the file.

[0042] In this embodiment, the specific operations of steps 201-204 are already described in Figure 1 In the illustrated embodiment, steps 101-104 are described in detail and will not be repeated here.

[0043] Step 205: Write the file name, metadata log file address, and metadata into the three-level index memory to generate a three-level index for the file.

[0044] In this embodiment, the execution subject of the metadata writing method can write the file name, the metadata log file address and the metadata into the three-level index memory to generate a three-level index of the file.

[0045] The tertiary index can use a key-value storage engine, writing metadata key information to the key-value storage engine in the tertiary index memory and metadata to the disk in the tertiary index memory. The key-value storage engine can be RocksDB, for example. RocksDB only stores metadata key information, which takes up approximately 24 bytes, thereby reducing the amount of data stored in RocksDB. Its storage format can be, for example: name => [data block offset, inode, type, and other necessary information]. The full metadata on disk is not used for scanning, so it can be reduced to a point-to-point read design. Full metadata is not stored in RocksDB, but on a separate disk. For example, the online metadata size is 200 to 500 bytes, so the metadata is stored directly at an address on the block device (a single block is 4KB, and a block can store multiple full metadata entries).

[0046] Step 206 : In response to the memory usage of the secondary index memory reaching the memory usage threshold, non-critical information of metadata in the secondary index stored in the secondary index memory is deleted.

[0047] In this embodiment, when a three-level index mechanism is used to store metadata, the above-mentioned execution entity can degenerate the secondary index based on the memory usage of the secondary index memory. If the memory usage of the secondary index memory does not reach the memory usage threshold (such as n%, n is a variable, calculated by the total memory of a single machine and the average throughput over a period of time), it can be considered that the secondary index memory is sufficient. At this time, all secondary indexes can be stored in the secondary index memory, and the secondary index stores the full amount of metadata. If the memory usage reaches the memory usage threshold, it can be considered that the secondary index memory usage is too high. At this time, all secondary indexes can be stored in the secondary index memory, and the non-critical information of the metadata in the secondary index stored in the secondary index memory is deleted, so that the secondary index only stores the critical information of the metadata (inode, pool, type, etc.).

[0048] Step 207: Delete the secondary indexes stored in the secondary index memory that have not been used within a preset time period.

[0049] In this embodiment, the execution entity can delete the secondary indexes stored in the secondary index memory that have not been used within a preset time period, and asynchronously delete the full metadata that has not been used for a long time, thereby reducing the memory usage of the secondary index memory.

[0050] Step 208, in response to the remaining memory of the secondary index memory being less than the preset memory remaining threshold, delete the secondary indexes stored in the secondary index memory whose popularity is lower than the preset popularity threshold, and delete the non-critical information of the metadata in the secondary indexes stored in the secondary index memory whose popularity is not lower than the preset popularity threshold.

[0051] In this embodiment, when a three-level index mechanism is used to store metadata, the above-mentioned execution subject can degenerate the secondary index according to the memory usage of the secondary index memory. If the remaining memory of the secondary index memory is less than the preset memory remaining threshold (such as when the remaining memory is less than m GB, m is calculated based on the actual business pressure and machine configuration), it can be considered that the secondary index memory is tight. At this time, the secondary indexes whose heat is lower than the preset heat threshold stored in the secondary index memory can be deleted, and the non-critical information of the metadata in the secondary indexes whose heat is not lower than the preset heat threshold stored in the secondary index memory can be deleted. Only hot secondary indexes are stored in the secondary index memory, and the hot secondary indexes only store the key information of the metadata, and the cold secondary indexes are asynchronously eliminated, thereby reducing the memory usage of the secondary index memory.

[0052] It's important to note that asynchronous flushing of log files can be enabled in the backend. To speed up flushing, each table can have its own log file, enabling concurrent flushing. Before the log file is flushed (the buffer is refreshed), the full in-memory metadata for the secondary index cannot be eliminated. (In extreme cases, it can be eliminated, and metadata can be read from the log file during reading.) After the log file is flushed, based on memory usage, only essential metadata is stored in memory.

[0053] In some embodiments, when using a three-level indexing mechanism to store metadata, the snapshot process relies on logs and RocksDB. When taking a snapshot, all logs after the snapshot point are flushed to RocksDB, and subsequent snapshots read directly from RocksDB. When taking a snapshot, flushing of all newly written logs after the snapshot point is paused to prevent data deletion.

[0054] The disclosed embodiments propose a new directory tree index structure that can support both single-machine scale of tens of billions and sequential traversal. Using streamlined RocksDB to store information does not affect the read speed of the directory tree index, but also reduces the storage size of RocksDB, preventing RocksDB merges from impacting index performance. Furthermore, the secondary index uses an active degradation mechanism to ensure both system performance and normal operation when resources such as memory are insufficient.

[0055] Further references Figure 3 , which shows a process 300 of an embodiment of a metadata reading method according to the present disclosure. The metadata reading method includes the following steps: Step 301: Receive a request to read file metadata.

[0056] In this embodiment, the execution subject of the metadata reading method may receive a metadata reading request for a file.

[0057] The metadata reading method is typically executed by a server. The server can be either hardware or software. If the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. If the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. This is not specifically limited here.

[0058] A metadata read request may include the file's inode identifier. The file may be a file in a distributed file system. Metadata may be system data describing file characteristics, including but not limited to: file type, permissions, number of links, file size, timestamp, access control list, etc.

[0059] Step 302: query the primary index memory based on the index node identifier to obtain the secondary index address of the file.

[0060] In this embodiment, the execution entity may query the primary index memory based on the index node identifier to obtain the secondary index address of the file.

[0061] The primary index resides in memory, and its storage format can be, for example, inode => secondary index address. Therefore, querying the primary index in primary index memory yields a mapping from inode to secondary index. Primary indexes can be stored using common hash structures. Primary indexes can be persisted using a common hash structure, and their recovery solution can adopt a common design of checkpoints plus log replay. Index snapshots are read from checkpoint files, loaded into memory, and then the log is replayed. The log is recorded when users write data.

[0062] Step 303: query the secondary index memory based on the secondary index address to obtain the metadata reading result of the file.

[0063] In this embodiment, the execution entity may query the secondary index memory based on the secondary index address to obtain the metadata reading result of the file.

[0064] Typically, the secondary index memory stores the file name, metadata log file address, and metadata in a key-value (KV) format. The key in the KV can be the file name, and the value in the KV can be the metadata log file address and metadata. The secondary index storage format can be, for example, a mapping from name to metadata and data storage path (name => [data block offset, inode, type, and other necessary information]). Therefore, querying the secondary index in the secondary index memory yields a mapping from the file name to the metadata log file address and metadata.

[0065] Secondary indexes can use a tree structure for ordered storage. Secondary indexes are not persisted to disk and are restored asynchronously using a lazy method. The recovery process is as follows: First, start an asynchronous thread to traverse the first-level index in sequence and restore the corresponding second-level index in sequence.

[0066] Afterwards, secondary indexes with high popularity are restored first.

[0067] Then, all child nodes under the parent index node are scanned from RocksDB and the index structure is constructed in memory.

[0068] Finally, the data is replayed from the log, the index structure is constructed in memory, and the recovery is completed.

[0069] In some embodiments, a scan operation can be performed in the secondary index memory. For the case where the secondary index contains all sub-nodes, the secondary index memory can be scanned and the scan results can be returned directly from the secondary index memory. For the case where the secondary index contains only some sub-nodes, the secondary index memory and the RocksDB of the tertiary index can be scanned simultaneously, and the merged scan results can be returned. For the case where the secondary index is not in the secondary index memory, the disk of the tertiary index can be scanned, and the scan results can be returned from the disk. Among them, there is a flag bit in the secondary index to indicate whether the secondary index contains all sub-nodes. When the hot and cold indexes are eliminated, the sub-nodes in a secondary index need to be eliminated in order. At this time, each sub-node in the secondary index has 2 flag bits to mark whether it is continuous with the left entry and the right entry (that is, whether there are other entries eliminated between the two entries).

[0070] The disclosed embodiment proposes a new directory tree index structure that can support both a single machine scale of tens of billions and sequential traversal.

[0071] Further references Figure 4 , which shows a process 400 of another embodiment of the metadata reading method according to the present disclosure. The metadata reading method includes the following steps: Step 401: Receive a request to read metadata of a file.

[0072] Step 402: query the primary index memory based on the index node identifier to obtain the secondary index address of the file.

[0073] In this embodiment, the specific operations of steps 401-402 are already described in Figure 3 In the illustrated embodiment, steps 301 - 302 are described in detail and will not be repeated here.

[0074] Step 403: query the secondary index memory based on the secondary index address to determine whether the secondary index of the file is found.

[0075] In this embodiment, the execution subject of the metadata reading method can query the secondary index memory based on the secondary index address to determine whether the secondary index of the file is found.

[0076] When using a three-level index mechanism to store metadata, secondary indexes are degraded based on the memory usage of the secondary index memory. For example, if the secondary index memory is sufficient, all secondary indexes are stored in the secondary index memory, and the secondary indexes store the entire metadata. If the memory usage reaches a memory usage threshold (e.g., n%, where n is a variable calculated by the total memory of a single machine and the average throughput over a period of time), all secondary indexes are stored in the secondary index memory, and the secondary indexes only store key metadata information (such as inode, pool, type, etc.). In this case, if the secondary index of the file can be found based on the secondary index address, step 404 is executed. If the secondary index memory is limited (e.g., when the remaining memory is less than m GB, where m is calculated based on actual business pressure and machine configuration), only hot secondary indexes are stored in the secondary index memory, and hot secondary indexes only store key metadata information. Cold secondary indexes are asynchronously deprecated. If the secondary index of the file is a hot secondary index, the secondary index can be found based on the secondary index address, and step 404 is executed. If the secondary index of the file is a cold secondary index, the secondary index of the file cannot be found based on the secondary index address, and step 407 is executed.

[0077] Step 404: Determine whether the secondary index stores metadata of the file.

[0078] In this embodiment, when a secondary index of a file is found based on the secondary index address, the execution entity may determine whether metadata of the file is stored in the secondary index.

[0079] When using a three-level index mechanism to store metadata, the secondary index will be degraded based on the memory usage of the secondary index memory. For example, if the secondary index memory is sufficient, the secondary index memory stores all secondary indexes, and the secondary index stores the full metadata. In this case, the secondary index stores the full metadata of the file, and step 405 is executed. If the memory usage reaches the memory usage threshold, the secondary index memory stores all secondary indexes, and the secondary index only stores the key metadata information. In this case, the secondary index only stores the key metadata information of the file, and step 406 is executed.

[0080] Step 405: Return the metadata of the file.

[0081] In this embodiment, when a secondary index of a file is found and metadata of the file is stored in the secondary index, the execution entity may return the metadata of the file.

[0082] Step 406: Return key information of the metadata of the file, or query the metadata of the file stored in the three-level index in the three-level index memory, and return the metadata of the file.

[0083] In this embodiment, when the secondary index of the file is queried and the key information of the file's metadata is stored in the secondary index, the above-mentioned execution entity can return the key information of the file's metadata, or it can query the metadata of the file stored in the tertiary index in the tertiary index memory, and return the metadata of the file.

[0084] It should be noted that when file metadata is found in the third-level index, it is possible to choose whether to cache the file metadata based on the actual situation. For example, if the file metadata is frequently queried, the file metadata can be cached; if the file metadata is rarely queried, the file metadata may not be cached.

[0085] Step 407: query the metadata of the file stored in the third-level index in the third-level index memory, and return the metadata of the file.

[0086] In this embodiment, when the secondary index of the file is not found, the execution entity may query the metadata of the file stored in the third-level index in the third-level index memory and return the metadata of the file.

[0087] In some embodiments, the key-value storage engine in the third-level index memory is first queried to obtain metadata key information; then, based on the metadata key information, the disk in the third-level engine memory is queried to obtain the file metadata. For example, the file metadata offset is first queried from RocksDB, and then the disk is queried based on the offset to read the full metadata of the file.

[0088] The disclosed embodiments propose a new directory tree index structure that can support both single-machine scale of tens of billions and sequential traversal. Using streamlined RocksDB to store information does not affect the read speed of the directory tree index, but also reduces the storage size of RocksDB, preventing RocksDB merges from impacting index performance. Furthermore, the secondary index uses an active degradation mechanism to ensure both system performance and normal operation when resources such as memory are insufficient.

[0089] Figure 5 The schematic diagram of the structure of the three-level indexing mechanism is shown in FIG. Figure 5 As shown, the three-level index mechanism includes a primary index 502, secondary indexes 503 and 504, and a key-value storage engine 505 and a disk 506 in the three-level index. Operation information on the secondary indexes 503 and 504 can be recorded in the log file 501. The primary index 502 can store the index node identifier and the secondary index address of the file. The secondary indexes 503 and 504 can store the name of the file, the log file address of the metadata, and the metadata. The key-value storage engine 505 in the three-level index can store key information of the metadata, and the disk 506 in the three-level index can store the full metadata. The secondary indexes 503 and 504 can adopt an active degradation mechanism to degrade according to the memory usage.

[0090] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a metadata writing device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0091] like Figure 6 As shown, the metadata writing device 600 of this embodiment may include: a receiving module 601, a first writing module 602, a second writing module 603, and a third writing module 604. The receiving module 601 is configured to receive a metadata writing request for a file, wherein the metadata writing request includes the file's metadata; the first writing module 602 is configured to write the metadata into a log file; the second writing module 603 is configured to write the file's index node identifier and secondary index address into a primary index memory to generate a primary index for the file; and the third writing module 604 is configured to write the file's name, the metadata log file address, and the metadata into a secondary index memory to generate a secondary index for the file.

[0092] In this embodiment, the metadata writing device 600 includes the receiving module 601, the first writing module 602, the second writing module 603 and the third writing module 604. The specific processing and technical effects thereof can be referred to in the respective Figure 1The relevant descriptions of steps 101-104 in the corresponding embodiment are not repeated here.

[0093] In some optional implementations of this embodiment, the metadata writing device 600 also includes: a fourth writing module, configured to write the file name, metadata log file address and metadata into the three-level index memory to generate a three-level index of the file.

[0094] In some optional implementations of this embodiment, the fourth writing module is further configured to: write key information of the metadata into the key-value storage engine of the three-level index memory, and write the metadata into the disk of the three-level index memory.

[0095] In some optional implementations of this embodiment, the metadata writing device 600 further includes: a degradation module configured to degrade the secondary index according to the memory usage of the secondary index memory.

[0096] In some optional implementations of this embodiment, the degradation module is further configured to: in response to the memory usage of the secondary index memory reaching a memory usage threshold, delete non-critical information of metadata in the secondary index stored in the secondary index memory.

[0097] In some optional implementations of this embodiment, the degradation module is further configured to: delete the secondary indexes stored in the secondary index memory that have not been used within a preset time period.

[0098] In some optional implementations of this embodiment, the degradation module is further configured to: in response to the remaining memory of the secondary index memory being less than a preset memory remaining threshold, delete the secondary indexes stored in the secondary index memory whose popularity is lower than the preset popularity threshold, and delete the non-critical information of the metadata in the secondary indexes stored in the secondary index memory whose popularity is not lower than the preset popularity threshold.

[0099] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a metadata reading device. Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0100] like Figure 7As shown, the metadata reading device 700 of this embodiment may include: a receiving module 701, a first query module 702, and a second query module 703. The receiving module 701 is configured to receive a metadata reading request for a file, wherein the metadata reading request includes an index node identifier of the file; the first query module 702 is configured to query the primary index memory based on the index node identifier to obtain a secondary index address of the file; and the second query module 703 is configured to query the secondary index memory based on the secondary index address to obtain a metadata reading result for the file.

[0101] In this embodiment, the metadata reading device 700 includes the receiving module 701, the first query module 702, and the second query module 703. The specific processing and technical effects thereof can be referred to in the respective Figure 3 The relevant descriptions of steps 301-303 in the corresponding embodiment are not repeated here.

[0102] In some optional implementations of this embodiment, the second query module 703 is further configured to: in response to finding a secondary index of the file, and the secondary index storing the metadata of the file, return the metadata of the file.

[0103] In some optional implementations of this embodiment, the second query module 703 is further configured to: in response to querying the secondary index of the file, and the secondary index storing key information of the file's metadata, return the key information of the file's metadata, or query the metadata of the file stored in the tertiary index in the tertiary index memory, and return the file's metadata.

[0104] In some optional implementations of this embodiment, the second query module 703 is further configured to: in response to not finding the secondary index of the file, query the metadata of the file stored in the tertiary index in the tertiary index memory, and return the metadata of the file.

[0105] In some optional implementations of this embodiment, the second query module 703 is further configured to: query the key-value storage engine of the three-level index memory to obtain key information of the metadata; query the disk of the three-level engine memory based on the key information of the metadata to obtain the metadata of the file.

[0106] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0110] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the metadata writing method or the metadata reading method. For example, in some embodiments, the metadata writing method or the metadata reading method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the metadata writing method or the metadata reading method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the metadata writing method or the metadata reading method in any other appropriate manner (eg, by means of firmware).

[0112] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0117] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0119] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A metadata writing method, comprising: receiving a metadata writing request for a file, wherein the metadata writing request includes metadata of the file; Writing the metadata into a log file; Writing the index node identifier and the secondary index address of the file into the primary index memory to generate the primary index of the file; The name of the file, the log file address of the metadata, and the metadata are written into a secondary index memory to generate a secondary index of the file.

2. The method according to claim 1, wherein The method further comprises: The name of the file, the log file address of the metadata, and the metadata are written into a three-level index memory to generate a three-level index of the file.

3. The method according to claim 2, wherein: Writing the name of the file, the log file address of the metadata, and the metadata into the third-level index memory includes: The key information of the metadata is written into the key-value storage engine of the three-level index memory, and the metadata is written into the disk of the three-level index memory.

4. The method according to claim 2 or 3, wherein: The method further comprises: Degrading the secondary index according to the memory usage of the secondary index memory.

5. The method according to claim 4, wherein Degrading the secondary index according to the memory usage of the secondary index memory includes: In response to the memory usage of the secondary index memory reaching a memory usage threshold, non-critical information of metadata in the secondary index stored in the secondary index memory is deleted.

6. The method according to claim 5, wherein: Degrading the secondary index according to the memory usage of the secondary index memory includes: Delete the secondary indexes stored in the secondary index memory that have not been used within a preset time period.

7. The method according to claim 6, wherein: Degrading the secondary index according to the memory usage of the secondary index memory includes: In response to the remaining memory of the secondary index memory being less than a preset memory remaining threshold, the secondary indexes stored in the secondary index memory whose popularity is lower than the preset popularity threshold are deleted, and the non-critical information of the metadata in the secondary indexes stored in the secondary index memory whose popularity is not lower than the preset popularity threshold is deleted.

8. A metadata reading method, comprising: Receive a metadata read request for a file, wherein the metadata read request includes an index node identifier of the file; Querying the primary index memory based on the index node identifier to obtain the secondary index address of the file; The secondary index memory is queried based on the secondary index address to obtain the metadata reading result of the file.

9. The method according to claim 8, wherein The querying of the secondary index memory based on the secondary index address to obtain the metadata reading result of the file includes: In response to finding the secondary index of the file and storing the metadata of the file in the secondary index, the metadata of the file is returned.

10. The method according to claim 9, wherein: The querying of the secondary index memory based on the secondary index address to obtain the metadata reading result of the file includes: In response to querying the secondary index of the file, and the key information of the metadata of the file is stored in the secondary index, the key information of the metadata of the file is returned, or the metadata of the file stored in the tertiary index is queried in the tertiary index memory, and the metadata of the file is returned.

11. The method according to claim 10, wherein: The querying of the secondary index memory based on the secondary index address to obtain the metadata reading result of the file includes: In response to not finding the secondary index of the file, querying the metadata of the file stored in the tertiary index in the tertiary index memory, and returning the metadata of the file.

12. The method according to claim 10 or 11, wherein: The step of querying the metadata of the file stored in the third-level index in the third-level index memory includes: Query the key-value storage engine of the three-level index memory to obtain key information of the metadata; The disk in the third-level engine memory is queried based on the key information of the metadata to obtain the metadata of the file.

13. A metadata writing device, comprising: A receiving module is configured to receive a metadata writing request for a file, wherein the metadata writing request includes metadata of the file; A first writing module is configured to write the metadata into a log file; A second writing module is configured to write the index node identifier and the secondary index address of the file into the primary index memory to generate the primary index of the file; The third writing module is configured to write the name of the file, the log file address of the metadata, and the metadata into the secondary index memory to generate a secondary index of the file.

14. A metadata reading device, comprising: A receiving module is configured to receive a metadata reading request for a file, wherein the metadata reading request includes an index node identifier of the file; A first query module is configured to query the primary index memory based on the index node identifier to obtain a secondary index address of the file; The second query module is configured to query the secondary index memory based on the secondary index address to obtain the metadata reading result of the file.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or 8-12.

16. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being configured to cause the computer to execute the method of any one of claims 1-7 or 8-12.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7 or 8 to 12.

Images