A metadata processing method, device, equipment and computer storage medium

Abstract

Embodiments of the application disclose a metadata processing method, device and equipment and a storage medium, wherein the method comprises: a file protocol layer gateway receiving a metadata processing request sent by a file protocol layer; wherein the metadata processing request comprises at least path information of a metadata object to be processed and a type of a target operation on the metadata object; in the case that the type of the target operation belongs to a write operation, performing processing corresponding to the type of the target operation on the metadata object based on the path information of the metadata object to obtain a processing result; based on the type of the target operation, performing the following operations on the processing result and the metadata processing request: writing the processing result into a metadata cache; returning the processing result to the file protocol layer; and based on the type of the target operation, persisting the metadata processing request.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and includes, but is not limited to, a metadata processing method, apparatus, device, and computer storage medium. Background Technology

[0002] In current distributed file systems, the read / write performance of small files is a significant bottleneck. Compared to large files, the read / write performance of small files relies heavily on metadata operations. When a distributed file system primarily consists of small files, the system's read / write performance depends on the performance of metadata processing. However, metadata processing performance is limited by the semantics of metadata and has low concurrency, making it prone to reaching performance bottlenecks. This results in a large number of small files becoming a performance bottleneck. Summary of the Invention

[0003] In view of this, embodiments of this application provide a metadata processing method, apparatus, device, and computer storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a metadata processing method comprising: a file protocol layer gateway receiving a metadata processing request sent by a file protocol layer; wherein the metadata processing request includes at least path information of a metadata object to be processed and a type of target operation on the metadata object; if the type of the target operation is a write operation, the metadata object is processed according to the path information of the metadata object in accordance with the type of the target operation to obtain a processing result; based on the type of the target operation, the following operations are performed on the processing result and the metadata processing request: writing the processing result into a metadata cache; returning the processing result to the file protocol layer; and persisting the metadata processing request based on the type of the target operation.

[0006] Secondly, embodiments of this application provide a metadata processing apparatus, comprising: a receiving module, configured to receive a metadata processing request sent by a file protocol layer gateway; wherein the metadata processing request includes at least path information of a metadata object to be processed and a type of target operation on the metadata object; a processing module, configured to, if the type of target operation is a write operation, perform processing on the metadata object corresponding to the type of target operation based on the path information of the metadata object to obtain a processing result; and an operation module, configured to, based on the type of target operation, perform the following operations on the processing result and the metadata processing request: write the processing result into a metadata cache; return the processing result to the file protocol layer; and persist the metadata processing request based on the type of target operation.

[0007] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the metadata processing method described above.

[0008] Fourthly, embodiments of this application provide a computer storage medium storing executable instructions for implementing the above-described method when executed by a processor.

[0009] In this embodiment, a file protocol layer gateway is used to process the metadata object corresponding to the target operation, and the processing result is stored in a metadata cache, shortening the metadata read / write path. For most non-core metadata-insensitive operations, the metadata can be directly written to the metadata cache, and the latest metadata is read from the metadata cache during retrieval, thereby greatly reducing the latency of metadata operations. Based on the type of the target operation, the metadata processing request is persisted. For core metadata, it is appended directly to the persistent storage locally in the form of logs, which also shortens the read / write path compared to writing to the metadata server. Since the metadata request processing is completed at the file protocol layer gateway, the concurrency capability of metadata processing can be effectively improved. Attached Figure Description

[0010] Figure 1A This application provides a schematic diagram of a system architecture for distributed metadata processing.

[0011] Figure 1B A schematic diagram illustrating the implementation flow of a metadata processing method provided in an embodiment of this application;

[0012] Figure 2 A schematic diagram illustrating the implementation flow of a metadata processing method provided in an embodiment of this application;

[0013] Figure 3 This is a functional configuration diagram of a metadata processing method provided in an embodiment of this application;

[0014] Figure 4A A flowchart illustrating a metadata processing method provided in an embodiment of this application;

[0015] Figure 4B A schematic diagram of a request sequence provided for an embodiment of this application;

[0016] Figure 4C A schematic diagram illustrating the connection between a queue to be sent and a queue to be sent, provided for an embodiment of this application;

[0017] Figure 5This is a schematic diagram of the composition structure of the metadata processing device provided in the embodiments of this application;

[0018] Figure 6 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the invention will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0020] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0024] File: A unit for managing a piece of data.

[0025] File system: A management system that stores and accesses data on a storage device in the form of files.

[0026] Small file: A small file that contains very little data.

[0027] Metadata: Data generated by the file system when managing file data; that is, data about data.

[0028] Cache: A mechanism for storing content locally to speed up the reading and writing of content.

[0029] Metadata Cache: A cache that accelerates the reading and writing of metadata.

[0030] Client: A file system client that initiates file operations.

[0031] File Protocol Gateway (PGW): Acts as a file system client.

[0032] Metadata server (MDS): Manages the metadata of the file system.

[0033] Dcache: A persistent distributed data cache.

[0034] Jounrnal: A persistent method for writing data in an append-only manner.

[0035] Distributed file system (DFS): A file system consisting of multiple client / server nodes, allowing clients to manage files stored on server nodes.

[0036] EDS: A distributed storage system that provides object, block, and file storage capabilities.

[0037] Common Internet File System (CIFS): A file-sharing protocol for Windows.

[0038] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0039] It should be understood that the embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit the technical scope of this application.

[0040] Figure 1A This is a schematic diagram of a system architecture for distributed metadata processing provided in an embodiment of this application, such as... Figure 1A As shown in the diagram, the schematic includes: a file sharing protocol 101, a file protocol layer gateway 102, a persistent log 103, and a metadata server 104. The file protocol layer gateway 102 includes an application programming interface 1021 and a metadata cache 1022.

[0041] File sharing protocol 101 can be a file sharing protocol under Windows;

[0042] The file protocol layer gateway 102 can act as a file system client to initiate file operations; the application programming interface 1021 is used to receive and process metadata requests; and the metadata cache 1022 is a cache used to accelerate the reading and writing of metadata.

[0043] Persistent log 103 is a persistence method that uses an append-only approach to write data in a persistent distributed data cache.

[0044] Metadata server 104 is used to manage the metadata of the file system.

[0045] This application provides a metadata processing method, such as... Figure 1B As shown, the method includes:

[0046] Step S101: The file protocol layer gateway receives a metadata processing request sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object;

[0047] Data in any file system is divided into data and metadata. Data refers to the actual data in a regular file, while metadata refers to system data used to describe the characteristics of a file, such as access permissions, file owner, and the distribution information of file data blocks.

[0048] In some embodiments, such as Figure 1A As shown, the file protocol layer gateway 102 can act as a file system client to receive metadata processing requests sent by the file sharing protocol 101.

[0049] In some embodiments, the metadata object to be processed may be, for example, the location of the file, the size of the file, the space occupied by the file, etc.; correspondingly, the path information of the metadata object to be processed may be, for example, the path information storing the location information of the file, the path information storing the size of the file, the path information storing the space occupied by the file, etc.

[0050] Step S102: If the target operation is a write operation, the metadata object is processed according to the type of the target operation based on the path information of the metadata object to obtain the processing result.

[0051] In some embodiments, a write operation may be, for example, opening, creating, setting attribute information, creating extended attributes, or truncating.

[0052] In some embodiments, if the target operation is a write operation, the location where the metadata object is stored can be determined based on the path information of the metadata object. Once the location where the data object is stored is determined, the metadata object can be processed in accordance with the target operation. For example, the location information of the file can be set based on the path information of the file storage location.

[0053] Step S103: Based on the type of the target operation, perform the following operations on the processing result and the metadata processing request: write the processing result into the metadata cache; return the processing result to the file protocol layer; and persist the metadata processing request based on the type of the target operation.

[0054] The processing result is stored in a metadata cache. During implementation, for example, the location information of the configured file can be stored in the data cache. In some embodiments, such as... Figure 1A As shown, the metadata cache 1022 can be set on the file protocol layer gateway 102 and is used to accelerate the reading and writing of metadata.

[0055] The processing result is returned to the file protocol layer. In some embodiments, such as... Figure 1A As shown, after processing the metadata, the processing result is returned to file sharing protocol 101.

[0056] Based on the type of the target operation, the metadata processing request is persisted. The method for persisting the metadata request is determined according to the different types of the target operation. For example, if the target operation is a write operation and a core write operation (edit node number, edit size, change path), the metadata processing request is persisted immediately; if the target operation is a write operation and a non-core write operation (open, create, set attribute information, create extended attributes, truncate), the non-core write operation can be persisted periodically. In some embodiments, such as... Figure 1A As shown, persistent log 103 in a persistent distributed data cache can be used to persist metadata processing requests using an append-only persistence method. The persistent log 103 can be replaced with a local solid-state drive (SSD), sacrificing high availability for better log writing performance.

[0057] In this embodiment, a file protocol layer gateway is used to process the metadata object corresponding to the target operation, and the processing result is stored in a metadata cache, shortening the metadata read / write path. For most non-core metadata-insensitive operations, the metadata can be directly written to the metadata cache, and the latest metadata is read from the metadata cache during retrieval, thereby greatly reducing the latency of metadata operations. Based on the type of the target operation, the metadata processing request is persisted. For core metadata, it is appended directly to the persistent storage locally in the form of logs, which also shortens the read / write path compared to writing to the metadata server. Since the metadata request processing is completed at the file protocol layer gateway, the concurrency capability of metadata processing can be effectively improved.

[0058] Figure 2 A metadata processing method provided in this application embodiment further includes a metadata processing request that includes writing parameter values, such as... Figure 2 As shown, the method includes:

[0059] Step S201: The file protocol layer gateway receives a metadata processing request sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object;

[0060] Step S202: If the target operation is a write operation, and the metadata object is found based on the path information of the metadata object, the write parameter value is written based on the metadata object to obtain the write result as the processing result.

[0061] In some embodiments, a write operation may be, for example, opening, creating, setting attribute information, creating extended attributes, or truncating; correspondingly, the write parameter value may be the set attribute information, the created extended information, etc.

[0062] During implementation, for example, the file protocol layer gateway receives a metadata processing request sent by the file protocol layer to set attribute information. The metadata processing request includes path information for setting attribute information, setting attribute information, and setting attribute information. The setting attribute information is the write parameter.

[0063] In some embodiments, the metadata object can be determined first based on the path information of the metadata object, and then the write parameter value can be written to the metadata object. For example, if the data processing request is to set attribute information, the location of the attribute information to be written can be determined first based on the path information of the attribute information to be set, and then the location of the set attribute information to be written can be obtained to get the attribute information that has been written.

[0064] During implementation, the metadata object is first located based on its path. Once the metadata object is found, the write parameter value is written to it, and finally the write result is obtained.

[0065] Step S203: If the metadata object is not found based on the path information of the metadata object, a write error result is obtained;

[0066] During implementation, if the metadata object cannot be found based on the path information of the metadata object, the parameters cannot be written, resulting in a write error.

[0067] Step S204: When the type of the target operation is the second preset type, while writing the write result to the metadata cache, the metadata processing request is persisted, and then the write result is returned to the file protocol layer.

[0068] In some embodiments, the second preset type may be a core metadata processing request, and the user can set the core metadata request according to actual needs.

[0069] During implementation, if the type of the target operation is the second preset type, the metadata processing request can be persisted while the write result is written to the metadata cache, and finally the write result is returned to the file protocol layer.

[0070] In this embodiment, the metadata processing request also includes writing parameter values. This requires locating the metadata object based on its path. If the metadata object is found, the writing parameter values ​​are written to it, resulting in a write result. If the metadata object is not found based on its path, the parameters cannot be written, leading to a write error. This ensures that the writing parameter values ​​are effectively written to the corresponding metadata object, and a write result is obtained.

[0071] When the target operation is of the second preset type, the metadata processing request can be persisted while the result is written to the metadata cache, and finally the write result is returned to the file protocol layer. In this way, for core metadata requests, they are directly appended to the persistent storage locally in the form of logs. Compared with writing to the metadata server, this shortens the read and write path. At the same time, the core metadata requests are immediately and synchronously persisted, which can effectively ensure that core metadata requests are not lost.

[0072] This application provides a metadata processing method, which includes:

[0073] Step S211: The file protocol layer gateway receives a metadata processing request sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object;

[0074] Step S212: If the target operation is a write operation, and the metadata object is found based on the path information of the metadata object, the write parameter value is written based on the metadata object to obtain the write result as the processing result.

[0075] Step S213: If the metadata object is not found based on the path information of the metadata object, a write error result is obtained;

[0076] Step S214: If the type of the target operation is a first preset type, after writing the write result into the metadata cache and writing the metadata processing request into the persistence sequence of the metadata cache, the write result is returned to the file protocol layer.

[0077] In some embodiments, the first preset type can be a non-core metadata processing request, and users can set non-core metadata requests according to actual needs.

[0078] In some embodiments, when the type of the target operation is a first preset type, the metadata processing request of the target operation type of the first preset type can be encapsulated periodically or when the number of metadata requests reaches a certain threshold, and then the encapsulated request can be sent to the persistent log. In this way, the number of times the metadata request is written to the persistent log can be effectively reduced.

[0079] In some embodiments, the metadata cache sequence to be persisted may be a persistent sequence that is written after merging at least two metadata requests of type first preset type for target operations.

[0080] During implementation, when the type of the target operation is a first preset type, the metadata processing request can be written to the metadata cache's persistent sequence periodically or when the number of metadata requests reaches a certain threshold.

[0081] Step S215: After returning the writing result to the file protocol layer, persist the metadata processing request to obtain a persistent log.

[0082] In this embodiment, when the target operation is a non-core request, the metadata request can be processed first, then written into the metadata cache's persistence sequence, and the writing result can be returned to the file protocol layer. This effectively reduces metadata processing latency.

[0083] This application provides a metadata processing method, which includes:

[0084] Step S221: The file protocol layer gateway receives a metadata processing request sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object;

[0085] Step S222: If the target operation is a write operation, and the metadata object is found based on the path information of the metadata object, the write parameter value is written based on the metadata object to obtain the write result as the processing result.

[0086] Step S223: If the metadata object is not found based on the path information of the metadata object, a write error result is obtained;

[0087] Step S224: If the type of the target operation is a first preset type, after writing the write result into the metadata cache and writing the metadata processing request into the persistence sequence of the metadata cache, the write result is returned to the file protocol layer.

[0088] Step S225: After returning the writing result to the file protocol layer, in the persistent sequence, merge at least two metadata processing requests of the target operation of the first preset type, and send the merged metadata processing request to the persistent log.

[0089] In some embodiments, at least two metadata processing requests of type 1 preset type for target operations can be merged in the sequence to be persisted first, and then the merged metadata processing requests can be sent to the persistent log.

[0090] In this embodiment, at least two metadata processing requests of a first preset type targeting operation are first merged in the sequence to be persisted. Then, the merged metadata processing request is sent to the persistent log. By merging the metadata requests before sending them to the persistent log, the number of times metadata is sent to the persistent log is effectively reduced, thus reducing the number of interactions and improving the system's processing efficiency.

[0091] Figure 3A metadata processing method provided in this application embodiment further includes a metadata processing request that includes writing parameter values, such as... Figure 3 As shown, the method includes:

[0092] Step S301: The file protocol layer gateway receives a metadata processing request sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object;

[0093] Step S302: If the target operation is a write operation, and the metadata object is found based on the path information of the metadata object, the write parameter value is written based on the metadata object to obtain the write result as the processing result.

[0094] Step S303: If the metadata object is not found based on the path information of the metadata object, a write error result is obtained;

[0095] Step S304: If the type of the target operation is a first preset type, after writing the write result into the metadata cache and writing the metadata processing request into the persistence sequence of the metadata cache, the write result is returned to the file protocol layer.

[0096] Step S305: After returning the writing result to the file protocol layer, determine whether the second metadata processing request and the first metadata processing request in the sequence to be persisted meet the merging rule;

[0097] Step S306: If the merging rule is satisfied, merge the second metadata processing request with the first metadata processing request, and use the merged first metadata request as the updated first metadata request.

[0098] This process continues until all metadata processing requests in the sequence to be persisted are merged, or if the merging rules are not met, the first metadata processing request or the merged first metadata processing request is sent to the persistent log.

[0099] The preset merging rules include:

[0100] A creation request sequence is obtained when one of the following objectives is combined with the creation operation: opening, setting attribute information, creating extended attributes, and truncating.

[0101] An open request sequence is obtained when one of the following objectives is combined with the open operation: setting attribute information, creating extended attributes, and truncating.

[0102] A sequence of setting attribute information requests is obtained when the following objectives are combined with the setting attribute information: creating extended attributes, or truncation.

[0103] By combining the created extended attributes with the set attribute information, a truncated request sequence is obtained.

[0104] Table 3B provides a merging rule for an embodiment of this application. As shown in Table 3B, both horizontal and vertical headers include: creating, opening, setting attribute information, creating extended attributes, and truncating.

[0105] The request entered in the table is a combined request from both the horizontal and vertical headers. The "nothing" field in the table indicates that the request will not appear or cannot be combined.

[0106]

[0107] Table 3B

[0108] As shown in Table 3B, the merging rules can be understood as follows:

[0109] A creation request sequence is obtained by combining the opening, setting attribute information, creating extended attributes, or truncating with the creation.

[0110] Based on the combination of the set attribute information, the creation of extended attributes, or the truncation with the opening, an opening request sequence is obtained;

[0111] Based on the combination of the creation of extended attributes or the truncation with the setting attribute information, a setting attribute information request sequence is obtained;

[0112] Based on merging the creation extended attribute with the truncation, a truncation request sequence is obtained.

[0113] In this embodiment, if the merging rule is met, the second metadata processing request is merged with the first metadata processing request, and the merged first metadata request is used as the updated first metadata request. This process continues until all metadata processing requests in the sequence to be persisted are merged, or if the merging rule is not met, the first metadata processing request or the merged first metadata processing request is sent to the persistent log. This effectively merges non-core requests and sends them to the persistent log, while sending core requests directly to the persistent log, thus merging metadata requests and reducing the number of metadata interactions.

[0114] In distributed file systems, the performance of small files is a significant challenge; compared to large files, metadata plays a crucial role in the performance of small files. When a distributed file system primarily handles small files, its performance depends on the processing performance of metadata. However, metadata processing performance is limited by the semantics of metadata and has low concurrency, easily leading to a performance bottleneck. This results in a large number of small files becoming the primary processing bottleneck. Therefore, a method is needed to improve metadata processing capabilities.

[0115] A common optimization approach is to horizontally scale the metadata server, enabling it to handle files across multiple directories simultaneously, thus improving overall performance. While this method improves directory-level concurrency performance, it cannot enhance the file processing capacity within a single directory.

[0116] Figure 1A This is a schematic diagram of a system architecture for distributed metadata processing provided in an embodiment of this application, such as... Figure 1A As shown, a metadata cache 1022 and a persistent log 103 were added to the original architecture. Requests initiated from the file sharing protocol 101 reach the file protocol layer gateway 102 and are processed by the application interface 1021 before being uniformly entered into the memory metadata cache 1022. The metadata cache 1022 decides whether to return the request immediately based on the type of metadata request and the types of metadata involved.

[0117] For write requests, the metadata is written to the metadata cache, or further persisted to persistent log 103, and then returned.

[0118] For a read request, if the metadata is found in the metadata cache, it is returned immediately; otherwise, the read request is sent to the metadata server, and after the metadata server returns, the file protocol layer gateway returns to the upper layer.

[0119] The addition of metadata caching and persistent logs shortens the metadata IO path and reduces metadata latency.

[0120] Figure 4A This is a flowchart illustrating a metadata processing method provided in an embodiment of this application, as shown below. Figure 4A As shown, the method includes:

[0121] Step 1: File sharing protocol 101 sends a metadata processing request to file protocol layer gateway 101;

[0122] Step 2: The application programming interface 1021 in the file protocol layer gateway 102 writes the metadata request to the metadata cache 1021 in the file protocol layer gateway 102;

[0123] Step 3: The metadata cache writes the metadata processing request to the persistent log 103;

[0124] Step 4: The persistent log 103 sends the stored metadata processing request to the metadata cache 1022;

[0125] Step 5: Metadata cache 1022 sends the processing result to the application interface;

[0126] Step 6: Application Programming Interface 1021 returns the processing result to File Sharing Protocol 101;

[0127] Step 7: Persistent log writes the metadata processing request to the metadata server 104 and processes the metadata processing request in the metadata server.

[0128] This application embodiment can categorize metadata requests into non-core requests and core requests based on the user's actual needs. Specifically, it provides processing method A for non-core requests and processing method B for core requests.

[0129] Solution A: For non-core metadata writes, to accelerate metadata processing, do not persist them immediately. Instead, have the metadata cache periodically write the instructions to the persistent log.

[0130] The processing steps for processing method A are as follows:

[0131] Step S401: The metadata request is written to the metadata cache for processing, and the processing result is obtained;

[0132] like Figure 4A As shown, step 2 involves using the program interface 1021 to write the metadata request into the metadata cache 1021 in the file protocol layer gateway 102 for processing, and obtaining the processing result.

[0133] Step S402: Immediately return the processing result to the file sharing protocol, and then asynchronously write the metadata request to persistence;

[0134] like Figure 4A Steps 5 and 6 show that the processing result is returned to the file sharing protocol 101, and then the metadata request is asynchronously written to persistence as in step 7.

[0135] Requests that cache non-core metadata in the metadata cache can be merged and then written to the persistent log all at once.

[0136] Figure 4B This is a schematic diagram of a request sequence provided in an embodiment of this application, such as... Figure 4B As shown, it includes:

[0137] Metadata requests that are asynchronously written three times:

[0138] The first metadata request type is create, create extended attribute, truncate, set attribute information; the second metadata request type is write, get attribute, get extended attribute; the third metadata request type is attribute, set attribute information, create extended attribute, and close.

[0139] Figure 4C This application provides a schematic diagram illustrating the connection between a queue to be sent and a queue to be sent, as shown in the embodiment of the present application. Figure 4C As shown, the global queue 401 includes the connection between the pending send queue 402 and the sending queue 403. The pending send queue 402 is periodically added to the sending queue 403 in a one-time manner, and requests are merged. In this way, the process of merging requests can be seen in the global queue 401, while requests that cannot be merged remain as is in the global sending queue 402. This is the request processing from a global perspective.

[0140] Requests are processed from the perspective of a single file. Once a request is sent from the global queue 401, it is processed directly within the file. Therefore, file 1 has two queues: queue 1 to be sent and queue 2 to be sent. When queue 2 to be sent is empty, the next batch of requests that can be merged are sent, until queue 2 to be sent is empty.

[0141] Step S403: Then, asynchronously update the metadata server.

[0142] like Figure 4A As shown, step 7 involves asynchronously updating the metadata processing request to the metadata server 104.

[0143] This application provides an asynchronous sending mechanism, which is a basic method for sending instructions to update metadata from the metadata cache to the persistent log, and is implemented through an asynchronous sending queue.

[0144] Metadata updates for multiple files may coexist in the metadata cache, necessitating a global metadata instruction queue to store instructions awaiting persistence. Additionally, since instruction merging is required within each file, a file-level metadata instruction queue is also needed. Once a file's request is persisted to the persistent log, the associated requests are cleaned up from the file queue.

[0145] Solution B: For core metadata, security must be prioritized. For example, if a core request appends data and then needs to update the size, the updated metadata request in the MetaCache should be synchronously submitted to the Journal.

[0146] Step S411: The metadata request is written to the metadata cache for processing to obtain the processing result; the metadata processing request is synchronously submitted to the persistent log.

[0147] like Figure 4A As shown, step 2 involves using program interface 1021 to write the metadata request into the metadata cache 1021 in the file protocol layer gateway 102 for processing, and obtaining the processing result. Step 3 involves synchronously submitting the metadata processing request to the persistent log 103.

[0148] Step S412: Immediately return the processing result to the file sharing protocol;

[0149] like Figure 4A As shown, steps 5 and 6 involve immediately returning the processing result to file sharing protocol 101.

[0150] Step S413: Update the metadata server asynchronously.

[0151] like Figure 4A As shown, step 7 involves asynchronously updating the metadata processing request to the metadata server 104.

[0152] This application proposes an optimization method for improving the performance of small files in a distributed file system. This method, based on in-memory metadata caching, improves the concurrency of metadata processing by reducing metadata processing latency. Compared to traditional optimization methods, the following optimizations are made:

[0153] 1. The metadata I / O path is shortened. For most operations that are not sensitive to metadata, the metadata can be written directly to memory. When reading, the latest metadata is also read from memory, which greatly reduces the latency of metadata operations. For important metadata, it is written to persistent logs and directly appended to persistent storage locally. Compared with writing to the metadata server, this also shortens the I / O path.

[0154] 2. Metadata requests were merged, reducing the number of metadata interactions. By adopting metadata caching, metadata is synchronized to the persistent log asynchronously. Before committing, metadata requests are merged, thereby reducing the number of commits.

[0155] This application also enables horizontal scaling of file protocol layer gateways: here, horizontal scaling is done at the directory level. Typically, shared directories are used as the basic unit of scaling. To ensure the integrity of metadata caching, the shared directory and the node hosting the file protocol layer gateway must be bound together to prevent multiple nodes from simultaneously accessing and operating the shared directory, which could lead to incomplete metadata caching on the master node, causing metadata cache invalidation and preventing the shortening of metadata I / O paths. Therefore, during horizontal scaling, different file protocol layer gateways must be configured with different shared directories, ensuring that all directories are evenly distributed across the nodes.

[0156] This application can also perform request merging at the file protocol layer, reducing the number of network interactions and lowering latency.

[0157] Based on the foregoing embodiments, this application provides a metadata processing device, which includes various modules, each module including sub-modules, and each sub-module including units. It can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0158] Figure 5 This is a schematic diagram of the composition structure of the metadata processing device provided in the embodiments of this application, such as... Figure 5 As shown, the device 500 includes:

[0159] The receiving module 501 receives a metadata processing request sent by the file protocol layer from the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object;

[0160] Processing module 502 is used to process the metadata object according to the type of the target operation when the type of the target operation is a write operation, based on the path information of the metadata object, to obtain a processing result;

[0161] Operation module 502 is configured to perform the following operations on the processing result and the metadata processing request based on the type of the target operation: write the processing result into the metadata cache; return the processing result to the file protocol layer; and persist the metadata processing request based on the type of the target operation.

[0162] In some embodiments, the metadata processing request further includes a write parameter value. Correspondingly, the processing module 502 includes a write submodule and a get submodule. The write submodule is used to write the write parameter value based on the metadata object when the metadata object is found based on the path information of the metadata object, thereby obtaining a write result as the processing result. The get submodule is used to obtain a write result indicating a write error when the metadata object is not found based on the path information of the metadata object.

[0163] In some embodiments, the operation module 503 includes a write submodule, a persistence submodule, and a return submodule, wherein the write submodule is used to write the write result to the metadata cache when the type of the target operation is a second preset type; the persistence submodule is used to persist the metadata processing request at the same time; and the return submodule is used to return the write result to the file protocol layer.

[0164] In some embodiments, the writing submodule is further configured to write the writing result into the metadata cache when the type of the target operation is a first preset type; the return submodule is configured to return the writing result to the file protocol layer after writing the metadata processing request into the persistence sequence of the metadata cache; the persistence submodule is configured to persist the metadata processing request after returning the writing result to the file protocol layer to obtain a persistent log.

[0165] In some embodiments, the persistence submodule is further used in the sequence to be persisted to merge at least two metadata processing requests of the target operation of the first preset type, and send the merged metadata processing request to the persistence log.

[0166] In some embodiments, the persistence submodule includes a judgment unit and a merging unit, wherein the judgment unit is used to determine whether the second metadata processing request and the first metadata processing request in the sequence to be persisted meet the merging rules; the merging unit is used to merge the second metadata processing request and the first metadata processing request if the merging rules are met, and use the merged first metadata request as the updated first metadata request; and so on, until all metadata processing requests in the sequence to be persisted are merged, or if the merging rules are not met, the first metadata processing request or the merged first metadata processing request is sent to the persistence log.

[0167] In some embodiments, the preset merging rules include: when one of the following purpose operations is merged with the creation operation, a creation request sequence is obtained: the opening, the setting attribute information, the creation extended attribute, and the truncation; when one of the following purpose operations is merged with the opening operation, an opening request sequence is obtained: the setting attribute information, the creation extended attribute, and the truncation; when one of the following purpose operations is merged with the setting attribute information, a setting attribute information request sequence is obtained: the creation extended attribute and the truncation; when the creation extended attribute is merged with the setting attribute information, a truncation request sequence is obtained.

[0168] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0169] It should be noted that, in the embodiments of this application, if the above-described metadata processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0170] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the metadata processing method provided in the above embodiments.

[0171] Correspondingly, embodiments of this application provide an electronic device, Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the hardware entity of the device 600 includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the program, it implements the steps in the metadata processing method provided in the above embodiments.

[0172] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or already processed by the processor 602 and the various modules in the electronic device 600 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0173] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0174] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0175] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0177] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0179] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0180] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0181] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0182] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0183] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0184] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A metadata processing method, the method comprising: The file protocol layer gateway receives a metadata processing request sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object; If the target operation is a write operation, the metadata object is processed according to the type of the target operation based on the path information of the metadata object, and the processing result is obtained. Based on the type of the target operation, the following operations are performed on the processing result and the metadata processing request: The processing result is written to the metadata cache; the metadata cache is set at the file protocol layer gateway and is used to accelerate the reading and writing of metadata. The processing result is returned to the file protocol layer; Based on the type of the target operation, the metadata processing request is persisted; wherein, if the target operation type is a core write operation, the metadata processing request is persisted immediately, and if the target operation type is a non-core write operation, the metadata processing request is persisted periodically.

2. The method as described in claim 1, characterized in that, The metadata processing request also includes writing parameter values, corresponding to: The step of processing the metadata object based on the path information of the metadata object in accordance with the type of the target operation to obtain a processing result includes: when the metadata object is found based on the path information of the metadata object, writing the write parameter value based on the metadata object to obtain a write result as the processing result; If the metadata object cannot be found based on the path information of the metadata object, a write error result is obtained.

3. The method as described in claim 1 or 2, characterized in that, Based on the type of the target operation, the following operations are performed on the processing result and the metadata processing request, including: When the type of the target operation is the second preset type, while writing the write result to the metadata cache, the metadata processing request is persisted and the write result is returned to the file protocol layer.

4. The method as described in claim 1 or 2, characterized in that, Based on the type of the target operation, the following operations are performed on the processing result and the metadata processing request, including: When the type of the target operation is a first preset type, after writing the write result into the metadata cache and writing the metadata processing request into the persistence sequence of the metadata cache, the write result is returned to the file protocol layer. After the write result is returned to the file protocol layer, the metadata processing request is persisted to obtain a persistent log.

5. The method as described in claim 4, characterized in that, The step of persisting the metadata processing request to obtain a persistent log includes: In the sequence to be persisted, at least two metadata processing requests of the target operation of the first preset type are merged, and the merged metadata processing request is sent to the persistent log.

6. The method as described in claim 5, characterized in that, The step of merging at least two metadata processing requests of type 1 preset type for the target operation in the persistent sequence, and sending the merged metadata processing request to the persistent log, includes: Determine whether the second metadata processing request and the first metadata processing request in the sequence to be persisted meet the merging rules; If the merging rules are met, the second metadata processing request is merged with the first metadata processing request, and the merged first metadata request is used as the updated first metadata request. This process continues until all metadata processing requests in the sequence to be persisted are merged, or if the merging rules are not met, the first metadata processing request or the merged first metadata processing request is sent to the persistent log.

7. The method as described in claim 6, characterized in that, The preset merging rules include: When one of the following objectives is combined with the creation operation, a creation request sequence is obtained: Open, Set property information, Create extended property, Truncate; An open request sequence is obtained when one of the following objectives is combined with the open operation: setting attribute information, creating extended attributes, and truncating. A sequence of setting attribute information requests is obtained when the following objectives are combined with the setting attribute information: creating extended attributes, or truncation. By combining the created extended attributes with the set attribute information, a truncated request sequence is obtained.

8. A metadata processing apparatus, the apparatus comprising: The receiving module is used by the file protocol layer gateway to receive metadata processing requests sent by the file protocol layer; wherein, the metadata processing request includes at least the path information of the metadata object to be processed and the type of the target operation on the metadata object; The processing module is used to process the metadata object according to the type of the target operation when the target operation is a write operation, based on the path information of the metadata object, to obtain the processing result. An operation module is configured to perform the following operations on the processing result and the metadata processing request based on the type of the target operation: write the processing result into a metadata cache, the metadata cache being set in the file protocol layer gateway for accelerating read and write operations of metadata; return the processing result to the file protocol layer; and persist the metadata processing request based on the type of the target operation; wherein, if the target operation type is a core write operation, the metadata processing request is persisted immediately, and if the target operation type is a non-core write operation, the metadata processing request is persisted periodically.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The device stores executable instructions for causing a processor to execute, thereby implementing the steps of the method according to any one of claims 1 to 7.

Images