File attribute processing method, device and equipment of distributed file system and medium

Abstract

Embodiments of the present disclosure relate to a file attribute processing method and device of a distributed file system, an apparatus and a medium, wherein the method comprises: receiving, by a data node in the distributed file system, a file attribute query request of a client, the file attribute query request carrying target identification information of a target file, and being used to request to query attribute information of the target file; the target identification information being obtained from a metadata node in the distributed file system; determining, according to the target identification information, a target logical storage unit used to store the attribute information of the target file from at least one logical storage unit contained by the data node; obtaining, from the target logical storage unit, the attribute information of the target file matching the target identification information; and responding to the file attribute query request based on the attribute information of the target file. Embodiments of the present disclosure can offload file attributes from the metadata node to the data node, which can effectively reduce the pressure of the metadata node.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device and medium for processing file attributes of a distributed file system. Background Technology

[0002] Most existing distributed file systems divide the system into metadata nodes and data nodes. Metadata nodes store the directory tree and directory and file attributes, while data nodes store the data. Currently, attribute reading and writing operations rely on metadata nodes, placing significant pressure on them and becoming a bottleneck for the scalability of distributed file systems. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method, apparatus, device, and medium for processing file attributes in a distributed file system.

[0004] This disclosure provides a file attribute processing method for a distributed file system, comprising: receiving a file attribute query request from a client through a data node in the distributed file system, the file attribute query request carrying target identifier information of a target file for requesting to query the attribute information of the target file; the target identifier information being obtained from a metadata node in the distributed file system; determining a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node based on the target identifier information; the data node dynamically maintaining the attribute information of each file based on the at least one logical storage unit; obtaining the attribute information of the target file matching the target identifier information from the target logical storage unit; and responding to the file attribute query request based on the attribute information of the target file.

[0005] Optionally, the target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit; the step of determining the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node according to the target identification information includes: parsing the file identifier of the target file and extracting the embedded unit identifier from the file identifier of the target file; and determining the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node based on the unit identifier.

[0006] Optionally, the data node maintains the logical storage unit based on memory, and the method further includes: periodically writing the file attribute information stored in the logical storage unit into an attribute file maintained by the data node based on disk; obtaining the attribute information of the target file matching the target identification information from the target logical storage unit includes: searching for the attribute information of the target file matching the target identification information from the target logical storage unit; if the attribute information of the target file is not found from the target logical storage unit, obtaining the attribute information of the target file through the disk.

[0007] Optionally, the attribute file includes a plurality of first storage blocks and at least one second storage block, wherein the first storage blocks are used to store file attribute information, and the second storage block is used to store index information of the file attribute information in the first storage blocks and the location information of the first storage blocks; the step of obtaining the attribute information of the target file through the disk includes: determining a target storage block for storing the attribute information of the target file from the plurality of first storage blocks based on the index information stored in the second storage block; locating the target storage block based on the location information of the target storage block stored in the second storage block, and obtaining the attribute information of the target file from the target storage block.

[0008] Optionally, the method further includes: in response to receiving an attribute setting request from the client for the target file, writing the attribute setting information carried in the attribute setting request and the first auto-incrementing sequence number associated with the attribute setting information into a write-ahead log; updating the attribute information of the target file already recorded in the target logical storage unit based on the attribute setting information to obtain the latest attribute information of the target file, and associating the latest attribute information of the target file with the first auto-incrementing sequence number; detecting invalid attribute information of the target file existing in the write-ahead log based on the first auto-incrementing sequence number corresponding to the target file recorded in the target logical storage unit, and deleting the detected invalid attribute information; wherein, the second auto-incrementing sequence number corresponding to the invalid attribute information of the target file is less than the first auto-incrementing sequence number.

[0009] Optionally, the logical storage unit is located in the memory of the data node. If the attribute information of the target file also includes extended attributes, the attribute information of the target file recorded in the target logical storage unit includes: the non-extended attributes of the target file, and the position of the extended attributes of the target file in the write-ahead log.

[0010] This disclosure provides a file attribute processing method for a distributed file system, comprising: sending a target file identification information acquisition request to a metadata node of the distributed file system via a client; receiving target identification information of the target file returned by the metadata node, and determining a target logical storage unit based on the target identification information; wherein the target logical storage unit is a logical storage unit used to store attribute information of the target file; determining the target data node to which the target logical storage unit belongs based on an acquired target mapping relationship; wherein the target mapping relationship is a mapping relationship between logical storage units and data nodes; and acquiring the attribute information of the target file through the target data node.

[0011] Optionally, the method further includes: periodically obtaining the latest target mapping relationship through the metadata node.

[0012] This disclosure also provides a file attribute processing apparatus for a distributed file system, comprising: a request receiving module, configured to receive a file attribute query request from a client through a data node in the distributed file system, the file attribute query request carrying target identifier information of a target file for requesting to query attribute information of the target file; the target identifier information is obtained from a metadata node in the distributed file system; a storage unit determining module, configured to determine a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node based on the target identifier information; the data node dynamically maintains the attribute information of each file based on at least one logical storage unit; an attribute acquisition module, configured to acquire the attribute information of the target file that matches the target identifier information from the target logical storage unit; and a request response module, configured to respond to the file attribute query request based on the attribute information of the target file.

[0013] This disclosure also provides a file attribute processing apparatus for a distributed file system, comprising: a request sending module, configured to send a request to obtain the identification information of a target file to a metadata node of the distributed file system via a client; a unit determination module, configured to receive the target identification information of the target file returned by the metadata node, and determine a target logical storage unit based on the target identification information; wherein the target logical storage unit is a logical storage unit used to store the attribute information of the target file; a node determination module, configured to determine the target data node to which the target logical storage unit belongs based on an acquired target mapping relationship; wherein the target mapping relationship is a mapping relationship between logical storage units and data nodes, and a data node contains at least one logical storage unit; and an information acquisition module, configured to acquire the attribute information of the target file through the target data node.

[0014] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the file attribute processing method of the distributed file system provided in this disclosure.

[0015] This disclosure also provides a computer-readable storage medium storing a computer program for executing a file attribute processing method of a distributed file system as provided in this disclosure.

[0016] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the file attribute processing method of the distributed file system provided in this disclosure.

[0017] The technical solution provided in this disclosure does not store file attribute information in the metadata node of the distributed file system, but rather stores file attribute information in the logical storage unit of the data node of the distributed file system. The client can obtain target identification information through the metadata node, and based on the target identification information, obtain the target file attribute information through the data node containing the target logical storage unit (that is, the logical storage unit that stores the attribute information of the target file). This method can offload file attributes from the metadata node to the data node, and the client can obtain the required file attribute information through the data node, which can effectively reduce the pressure on the metadata node.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a file attribute processing method for a distributed file system provided in this embodiment of the disclosure;

[0022] Figure 2 This is a schematic diagram of the structure of an attribute file provided in an embodiment of the present disclosure;

[0023] Figure 3 A flowchart illustrating a file attribute processing method for a distributed file system provided in this embodiment of the disclosure;

[0024] Figure 4 This is a schematic diagram of the file attribute processing interaction flow of a distributed file system provided in an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of a file attribute processing device for a distributed file system provided in an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of a file attribute processing device for a distributed file system provided in an embodiment of the present disclosure;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] Figure 1 This is a flowchart illustrating a file attribute processing method for a distributed file system provided in this disclosure. This method can be executed by a file attribute processing device for the distributed file system, wherein the device can be implemented in software and / or hardware, and is generally integrated into an electronic device. For example, the electronic device can be a data node of the distributed file system, such as... Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0031] Step S102: The client's file attribute query request is received through the data node in the distributed file system. The file attribute query request carries the target identifier information of the target file, which is used to request the query of the target file's attribute information. The target identifier information is obtained from the metadata node in the distributed file system. That is, the client can obtain an identifier (i.e., the target identifier information) through the metadata node, provide this identifier information to the data node, and the data node can then determine the target file for which the attribute information needs to be queried based on the identifier information, and perform the query operation on the target file's attribute information.

[0032] Step S104: Based on the target identification information, determine the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained in the data node; the data node dynamically maintains the attribute information of each file based on at least one logical storage unit.

[0033] For example, embodiments of this disclosure can create multiple logical storage units when creating a distributed file system. Each storage unit has a unique unit identifier. The metadata node can evenly distribute several logical storage units across data nodes. In other words, each data node in the distributed file system contains one or more logical storage units. The metadata node records the mapping relationship between logical storage units and data nodes, that is, it records which logical storage units are on each data node. Data nodes can dynamically maintain the attribute information of each file based on at least one logical storage unit. For example, a data node can store received file attribute information in a logical storage unit and dynamically adjust the file attribute information in the logical storage unit based on user operations on the file. When a client creates a file through the metadata node, the metadata node can evenly distribute the file attribute information among multiple logical storage units for storage, such as preferentially allocating the file attribute information to logical storage units with low load. For example, when a client creates a new file, the metadata node can determine the logical storage unit (hereinafter referred to as the target unit) from multiple existing logical storage units based on a uniform allocation strategy, and generate the target identification information (which can be represented by a file identifier) ​​corresponding to the new file. It can also encode the unit identifier of the logical storage unit used to store the attribute information of the new file (i.e., the unit identifier of the target unit) within the target identification information. When the metadata node receives a request from the client to obtain the target file's identification information, it can provide the target file's target identification information to the client, allowing the client to request and query the target file's attribute information from the data node to which the target unit belongs based on the target identification information. Upon obtaining the attribute information of the new file, the metadata node can send a storage instruction (carrying the new file's attribute information and the new file's target identification information) to the data node to which the target unit belongs, so that the data node stores the new file's attribute information in the target unit. In some specific examples, the data node provided in this disclosure can respond to a file attribute storage instruction from a metadata node, and based on the file attribute information to be stored (such as the attribute information of the aforementioned new file) carried by the file attribute storage instruction and the file identifier corresponding to the file attribute information, store the file attribute information in a logical storage unit (such as the aforementioned target unit) corresponding to the unit identifier embedded in the file identifier. In this way, file attribute information can be offloaded to the data node for storage, reducing the pressure on the metadata node.

[0034] To facilitate simple and convenient identification of the logical storage unit corresponding to file attribute information, in some embodiments, the target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit. That is, in the embodiments of this disclosure, the unit identifier of the logical storage unit can be encoded into the file identifier. For example, the file identifier contains a total of M bits, and the first N bits of the M bits are the unit identifier, where M and N are both positive integers, and M is greater than N. Specifically, it can be flexibly set according to needs. For example, N = 16. The other bits in the M bits, excluding the first N bits, can be used to uniquely identify the file. Based on this, in the specific execution of the above-mentioned step of determining the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained in the data node according to the target identification information, the file identifier of the target file can be parsed to extract the embedded unit identifier from the file identifier of the target file, such as extracting the first 16 bits of the identifier as the unit identifier; then, based on the unit identifier, the target logical storage unit for storing the attribute information of the target file can be determined from at least one logical storage unit contained in the data node. The above methods can both uniquely identify a file based on its file identifier and explicitly indicate the logical storage unit used to store the file's attribute information based on its file identifier.

[0035] The file attribute information in this embodiment includes, but is not limited to, file size, modification time, user identifier, group identifier, and extended attributes. All file attributes are set in the logical storage unit of the data node to reduce the pressure on the metadata node as much as possible.

[0036] Step S106: Obtain the attribute information of the target file that matches the target identification information from the target logical storage unit. Since the target logical storage unit is a logical storage unit that stores the attribute information of the target file, after determining the target logical storage unit, the required attribute information of the target file can be directly obtained based on the target logical storage unit.

[0037] Step S108: Respond to the file attribute query request based on the attribute information of the target file.

[0038] The technical solution provided in this disclosure does not store file attribute information in the metadata node of the distributed file system, but rather stores file attribute information in the logical storage unit of the data node of the distributed file system. The client can obtain target identification information through the metadata node, and based on the target identification information, obtain the target file attribute information through the data node containing the target logical storage unit (that is, the logical storage unit that stores the attribute information of the target file). This method can offload file attributes from the metadata node to the data node, and the client can obtain the required file attribute information through the data node, which can effectively reduce the pressure on the metadata node.

[0039] In some implementations, data nodes maintain logical storage units based on memory. Building upon this, the method provided in this disclosure further includes periodically writing file attribute information stored in the logical storage units into an attribute file maintained by the data node on disk. In this approach, the attribute file resides on the data node's disk, and the file attribute information can be deleted from memory after being written to the attribute file. This effectively saves memory space. It is understood that memory space is limited; if all file attribute information were stored in memory, it would consume a large amount of memory, and the data node would need to load all file attribute information upon startup, slowing down the startup process. Therefore, this disclosure employs a lazy loading method. Specifically, the file attribute information stored in the logical storage units is periodically written to the attribute file on the data node's disk, and the file attribute information is only loaded from disk into memory when accessing it. This effectively saves memory space and improves startup speed.

[0040] Based on the foregoing, the steps of obtaining the attribute information of the target file that matches the target identification information from the target logical storage unit can be performed with reference to the following steps (1) and (2):

[0041] Step (1): Search for the attribute information of the target file that matches the target identification information in the target logical storage unit.

[0042] Step (2): If the attribute information of the target file cannot be found in the target logical storage unit, the attribute information of the target file is obtained through the disk. It can be understood that if the attribute information of the target file has been written from the target logical storage unit to the disk and the attribute information of the target file in the target logical storage unit has been cleared, then the attribute information of the target file cannot be found in the target logical storage unit. In this case, the attribute information of the target file can be obtained directly through the disk.

[0043] To improve the speed of retrieving file attribute information from the disk, in some implementations, the aforementioned attribute file includes multiple first storage blocks and at least one second storage block. The first storage blocks store file attribute information, and the second storage block stores the index information of the file attribute information within the first storage blocks and the location information of the first storage blocks. The index information of the file attribute information within the first storage blocks can be used to indicate the mapping relationship between the first storage blocks and the file attribute information, that is, to indicate which file attribute information is stored in each first storage block. The location information can be characterized based on two parameters: offset address and length. In practical applications, the above storage blocks can be managed based on a B+ tree. A B+ tree is a multi-way balanced search tree, a type of index structure, which can be used to efficiently and reliably store and retrieve file attribute information.

[0044] Based on the foregoing, the steps for obtaining the attribute information of the target file via disk can be performed with reference to steps A and B as follows:

[0045] Step A involves determining the target storage block for storing the attribute information of the target file from among multiple first storage blocks, based on the index information stored in the second storage block. In practical applications, the file attribute information in each first storage block can be sorted according to the size of the file identifier, and the target storage block for storing the attribute information of the target file can be quickly determined based on a binary search method.

[0046] Step B: Based on the location information of the target storage block stored in the second storage block, locate the target storage block and obtain the attribute information of the target file from the target storage block.

[0047] To further improve the speed of searching for file attribute information, the first storage block contains a storage area and an index area. That is, an additional index is added to the first storage block. The storage area stores file attribute information, and the index area stores the location of the file attribute information within the storage area. Based on this, the specific steps for obtaining the attribute information of the target file from the target storage block in step B above can be performed as follows: Steps 1 and 2.

[0048] Step 1: Based on the index area of ​​the target storage block, obtain the location of the target file's attribute information within the storage area of ​​the target storage block.

[0049] Step 2: Based on the location of the target file's attribute information within the target storage block's storage area, retrieve the target file's attribute information from the target storage block's storage area.

[0050] The above methods can help to further improve the speed of finding the attribute information of the target file from the target storage block and help to shorten the time spent reading and writing file attribute information.

[0051] For example, one can refer to Figure 2 The diagram illustrates a storage structure. Assume the attribute file includes storage blocks 1 through 7, where storage blocks 1 through 4 are the first storage blocks, and storage blocks 5 through 7 are the second storage blocks. Each of storage blocks 1 through 4 contains a storage area and an index area. The storage area stores the attribute information of each file, which corresponds to a file identifier. The index area stores the address of the attribute information for each file identifier, arranged in ascending order of file identifier. If we need to obtain the attribute information of file 3, i.e., the attribute information corresponding to file identifier 3, we can quickly search the storage block containing the required file attribute information using a binary search method. Assuming storage block 7 is the root node, storage block 7 indicates that storage block 5 contains the relevant information for file identifiers 1 through 4, and storage block 6 contains the relevant information for file identifiers 5 through 8. Based on this, we can quickly determine that the relevant information corresponding to file identifier 3 is located in storage block 5. Storage block 5, in turn, indicates that storage block 1 contains the relevant information for file identifiers 1 through 2, and storage block 2 contains the relevant information for file identifiers 3 through 4. Based on this, we can quickly determine that the relevant information corresponding to file identifier 3 is located in storage block 2. Then, the attribute information address of file identifier 3 can be obtained from the index area of ​​storage block 2, and the file attribute information of file identifier 3 can be obtained from the storage area based on that address. In this way, the required file attribute information can be obtained efficiently and accurately from multiple storage blocks within the attribute file.

[0052] In practical applications, when reading a storage block (the first or second storage block), the block can be cached in memory. Similarly, when searching for file attribute information corresponding to a file identifier within a storage block, this attribute information can be cached in memory. When a data node needs to obtain the attribute information of a target file, it first searches the cached file attribute information in memory. If the target file's attribute information is found, it is directly returned to the client. If not, a B+ tree search is performed, prioritizing the search in the memory storage block cache. If the target storage block is found, the required file attribute information is obtained based on it. If the target storage block is not found, it is read from disk, and then the target file's attribute information is retrieved from that block, cached in memory, and returned to the client. This method effectively avoids frequent disk reads, improves attribute query efficiency, and reduces query time.

[0053] In some embodiments, the above method further includes the following steps a to c:

[0054] Step a: In response to receiving a client's request to set attributes for a target file, the attribute setting information carried in the request and the associated first auto-incrementing sequence number are written to the write-ahead log. The write-ahead log is located on disk and is used to ensure data consistency and durability. When a data node receives a request to set file attribute information, it first records it in the write-ahead log and then updates the memory. In this embodiment, each attribute setting can be associated with an auto-incrementing sequence number, i.e., a progressively increasing sequence number, where the first auto-incrementing sequence number corresponding to a later attribute setting is greater than the first auto-incrementing sequence number of an earlier attribute setting.

[0055] Step b: Update the attribute information of the target file recorded in the target logical storage unit based on the attribute setting information to obtain the latest attribute information of the target file, and associate the latest attribute information of the target file with the first auto-incrementing sequence number.

[0056] Step c: Based on the first auto-incrementing sequence number corresponding to the target file recorded in the target logical storage unit, detect invalid attribute information of the target file existing in the write-ahead log, and delete the detected invalid attribute information; wherein, the second auto-incrementing sequence number corresponding to the invalid attribute information of the target file is less than the first auto-incrementing sequence number.

[0057] It is understood that attribute information includes various attributes such as file size, file modification time, user identifier, and extended attributes. Each attribute may be set individually or repeatedly. If it is set repeatedly, invalid attributes can be cleaned up to avoid consuming storage resources. For example, if the file size is set to 10 and then 9 successively, and the final file size is 9, the file size recorded in the target logical storage unit in memory is also the latest value of 9. In this case, the file size 10 in the write-ahead log can be deleted. This embodiment of the disclosure can quickly identify invalid attribute information in the write-ahead log by using an auto-incrementing sequence number. If the auto-incrementing sequence number corresponding to a certain file attribute recorded in the write-ahead log is less than the auto-incrementing sequence number corresponding to the same file attribute recorded in memory, then the file attribute recorded in the write-ahead log can be cleaned up. For example, if the auto-incrementing sequence number corresponding to the file size 9 in memory is 2, while the auto-incrementing sequence number corresponding to the file size 10 in the write-ahead log is 1, then the file size 10 in the write-ahead log is considered invalid attribute information and can be cleaned up. Through the above method, invalid data can be effectively cleaned up, avoiding resource waste.

[0058] Considering that some files also have extended attributes, which can be custom attributes, the difference between extended and non-extended attributes is that extended attributes may have larger values, thus occupying more storage space and making it inconvenient to store them entirely in memory. Since logical storage units reside in the memory of data nodes, when the target file's attribute information includes extended attributes, the target file's attribute information recorded in the target logical storage unit includes: the target file's non-extended attributes and the position of the target file's extended attributes in the write-ahead log. Non-extended attributes are all attributes of the target file other than extended attributes. For example, the maximum value of a single extended attribute may be 64KB, while each record in the write-ahead log is only 4KB. Therefore, extended attributes may occupy multiple records. To simplify processing, the position of the extended attributes in the write-ahead log can be recorded only in the logical storage unit in memory, that is, the offset address and length of the extended attributes in the write-ahead log. In this way, various types of attributes can be stored effectively, and even large extended attributes can be handled properly.

[0059] Figure 3 This is a flowchart illustrating a file attribute processing method for a distributed file system provided in this disclosure. This method can be executed by a file attribute processing device for the distributed file system, wherein the device can be implemented in software and / or hardware, and is generally integrated into an electronic device. For example, the electronic device can be a client, such as... Figure 3 As shown, the method mainly includes the following steps S302 to S308:

[0060] Step S302 involves the client sending a request to the metadata node of the distributed file system to retrieve the target file's identification information. In practice, the request may include the target file's name and path, thus identifying the target file through a combination of name and path. The client can retrieve the target file's identification information from the metadata node when it needs to obtain the target file's attribute information. This identification information can form a unique file identifier for the target file using a multi-bit identifier. In practice, the file identifiers for each file are assigned by the metadata node.

[0061] Step S304: Receive the target identification information of the target file returned by the metadata node, and determine the target logical storage unit based on the target identification information; wherein, the target logical storage unit is a logical storage unit used to store the attribute information of the target file. As mentioned above, the target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit. After obtaining the target identification information, the client can extract the unit identifier from it and accurately determine the target logical storage unit based on the unit identifier.

[0062] Step S306: Based on the acquired target mapping relationship, determine the target data node to which the target logical storage unit belongs; wherein, the target mapping relationship is the mapping relationship between logical storage units and data nodes, and a data node contains at least one logical storage unit. The target mapping relationship can clearly indicate which logical storage unit is located on which data node, and the client can quickly determine the target data node to which the target logical storage unit belongs based on the target mapping relationship, given the known target logical storage unit.

[0063] Considering the possibility of data node failures causing changes in the mapping relationship, the latest target mapping relationship can be periodically obtained from the metadata node. In practical applications, the client can continuously obtain the latest target mapping relationship from the metadata node based on a heartbeat mechanism. This method effectively ensures the accuracy of the obtained target mapping relationship, and based on this target mapping relationship, the target data node to which the target logical storage unit belongs can be accurately determined.

[0064] Step S308: Obtain the attribute information of the target file through the target data node. This embodiment of the disclosure does not restrict the client's subsequent operations based on the obtained attribute information of the target file.

[0065] The technical solution provided in this disclosure does not store file attribute information in the metadata node of the distributed file system, but rather stores file attribute information in the logical storage unit of the data node of the distributed file system. The client can obtain target identification information through the metadata node, and based on the target identification information, obtain the target file attribute information through the data node containing the target logical storage unit (that is, the logical storage unit that stores the attribute information of the target file). This method can offload file attributes from the metadata node to the data node, and the client can obtain the required file attribute information through the data node, which can effectively reduce the pressure on the metadata node.

[0066] For easier understanding, please refer to Figure 4 The diagram illustrates the file attribute processing interaction flow of a distributed file system, showing the interaction process between the client, metadata node, and target data node, mainly including the following steps S402 to S414:

[0067] In step S402, the client sends a request to the metadata node to obtain the identification information of the target file.

[0068] In step S404, the metadata node returns the file identifier of the target file to the client. The file identifier of the target file contains the unit identifier of the target logical storage unit.

[0069] In step S406, the client extracts the unit identifier of the target logical storage unit from the file identifier and determines the target data node to which the target logical storage unit belongs based on the mapping relationship between the logical storage unit and the data node.

[0070] In step S408, the client sends a file attribute query request to the target data node, and the file attribute query request carries the file identifier of the target file.

[0071] In step S410, the target data node determines the target file for the queried attribute and the target logical storage unit containing the attribute information of the target file based on the file identifier.

[0072] Step S412: The target data node obtains the attribute information of the target file based on the target logical storage unit.

[0073] In step S414, the target data node returns the attribute information of the target file to the client.

[0074] In summary, the file attribute processing method of the distributed file system provided in this disclosure can offload file attributes from the metadata node to the data node. The client can obtain the required file attribute information through the data node, which can effectively reduce the pressure on the metadata node and help increase the cluster size.

[0075] Corresponding to the aforementioned file attribute processing method for distributed file systems, this disclosure further provides a file attribute processing apparatus for distributed file systems. Figure 5 This is a schematic diagram of a file attribute processing device for a distributed file system provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, which can be a data node of the distributed file system, such as... Figure 5 As shown, the file attribute processing device of the distributed file system includes:

[0076] The request receiving module 502 is used to receive file attribute query requests from clients through data nodes in the distributed file system. The file attribute query request carries the target identifier information of the target file and is used to request the query of the attribute information of the target file. The target identifier information is obtained from the metadata node of the distributed file system.

[0077] The storage unit determination module 504 is used to determine, based on the target identification information, the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained in the data node; the data node dynamically maintains the attribute information of each file based on at least one logical storage unit.

[0078] The attribute acquisition module 506 is used to acquire attribute information of the target file that matches the target identification information from the target logical storage unit;

[0079] The request-response module 508 is used to respond to file attribute query requests based on the attribute information of the target file.

[0080] The aforementioned device can offload file attributes from the metadata node to the data node, allowing the client to obtain the required file attribute information through the data node, which can effectively reduce the pressure on the metadata node.

[0081] In some implementations, the target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit; the storage unit determination module 504 is specifically used to: parse the file identifier of the target file, extract the embedded unit identifier from the file identifier of the target file; and, based on the unit identifier, determine the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node.

[0082] In some implementations, the data node maintains the logical storage unit based on memory, and the device further includes: a disk writing module for periodically writing file attribute information stored in the logical storage unit into an attribute file maintained by the data node based on disk;

[0083] In some implementations, the attribute information acquisition module 504 is specifically used to: search for attribute information of the target file that matches the target identification information from the target logical storage unit; and, if the attribute information of the target file is not found from the target logical storage unit, acquire the attribute information of the target file through the disk.

[0084] In some embodiments, the attribute file includes a plurality of first storage blocks and at least one second storage block. The first storage blocks are used to store file attribute information, and the second storage blocks are used to store index information of the file attribute information in the first storage blocks and the location information of the first storage blocks. The attribute information acquisition module 504 is specifically used to: determine a target storage block for storing the attribute information of the target file from the plurality of first storage blocks based on the index information stored in the second storage block; locate the target storage block based on the location information of the target storage block stored in the second storage block, and acquire the attribute information of the target file from the target storage block.

[0085] In some implementations, the first storage block includes a storage area and an index area; the storage area is used to store file attribute information, and the index area is used to store the location of the file attribute information within the storage area; the attribute information acquisition module 504 is specifically used to: acquire the location of the target file's attribute information within the storage area of ​​the target storage block based on the index area of ​​the target storage block; and acquire the target file's attribute information from the storage area of ​​the target storage block based on the location of the target file's attribute information within the storage area of ​​the target storage block.

[0086] In some embodiments, the apparatus further includes an invalid information deletion module, configured to, in response to receiving an attribute setting request from the client for the target file, write the attribute setting information carried in the attribute setting request and a first auto-incrementing sequence number associated with the attribute setting information into a write-ahead log; update the attribute information of the target file already recorded in the target logical storage unit based on the attribute setting information to obtain the latest attribute information of the target file, and associate the latest attribute information of the target file with the first auto-incrementing sequence number; detect invalid attribute information of the target file existing in the write-ahead log based on the first auto-incrementing sequence number corresponding to the target file recorded in the target logical storage unit, and delete the detected invalid attribute information; wherein, the second auto-incrementing sequence number corresponding to the invalid attribute information of the target file is less than the first auto-incrementing sequence number.

[0087] In some implementations, the logical storage unit is located in the memory of the data node. When the attribute information of the target file also includes extended attributes, the attribute information of the target file recorded in the target logical storage unit includes: the non-extended attributes of the target file, and the position of the extended attributes of the target file in the write-ahead log.

[0088] Corresponding to the aforementioned file attribute processing method for distributed file systems, this disclosure further provides a file attribute processing apparatus for distributed file systems. Figure 6 This is a schematic diagram of a file attribute processing device for a distributed file system provided in an embodiment of the present disclosure. This device can be implemented by software and / or hardware, and is generally integrated into an electronic device, which can be a client, such as... Figure 6 As shown, the file attribute processing device of the distributed file system includes:

[0089] The request sending module 602 is used to send a request to the metadata node of the distributed file system to obtain the identification information of the target file through the client;

[0090] The unit determination module 604 is used to receive the target identification information of the target file returned by the metadata node, and determine the target logical storage unit for storing the attribute information of the target file based on the target identification information;

[0091] The node determination module 606 is used to determine the target data node to which the target logical storage unit belongs based on the acquired target mapping relationship; wherein, the target mapping relationship is the mapping relationship between logical storage units and data nodes, and a data node contains at least one logical storage unit;

[0092] The information acquisition module 608 is used to acquire the attribute information of the target file through the target data node.

[0093] The aforementioned device can offload file attributes from the metadata node to the data node, allowing the client to obtain the required file attribute information through the data node, which can effectively reduce the pressure on the metadata node.

[0094] In some embodiments, the apparatus includes a relationship acquisition module for periodically acquiring the latest target mapping relationship through the metadata node.

[0095] The file attribute processing device for the distributed file system provided in this disclosure can execute the file attribute processing method for the distributed file system provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0097] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0098] The following is for reference. Figure 7 This document illustrates a structural schematic diagram of an electronic device 700 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 7The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0099] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0100] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0101] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0102] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in embodiments of this disclosure.

[0104] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the methods described in this disclosure.

[0106] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0107] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0108] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0109] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing file attributes in a distributed file system, characterized in that, include: The file attribute query request is received from the client through the data node in the distributed file system. The file attribute query request carries the target identifier information of the target file and is used to request the query of the attribute information of the target file. The target identification information is obtained from the metadata node of the distributed file system; The target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit; the metadata node records the mapping relationship between the logical storage unit and the data node; The data node corresponding to the file attribute query request is determined by the client extracting the unit identifier of the target logical storage unit from the file identifier and based on the mapping relationship; Based on the target identification information, a target logical storage unit for storing the attribute information of the target file is determined from at least one logical storage unit contained in the data node; The data node dynamically maintains the attribute information of each file based on at least one logical storage unit; Obtain the attribute information of the target file that matches the target identification information from the target logical storage unit; The file attribute query request is responded to based on the attribute information of the target file.

2. The method according to claim 1, characterized in that, The step of determining a target logical storage unit for storing attribute information of the target file from at least one logical storage unit contained in the data node based on the target identification information includes: The file identifier of the target file is parsed, and the embedded unit identifier is extracted from the file identifier of the target file; Based on the unit identifier, a target logical storage unit for storing the attribute information of the target file is determined from at least one logical storage unit contained in the data node.

3. The method according to claim 1, characterized in that, The data node maintains the logical storage unit based on memory, and the method further includes: The file attribute information stored in the logical storage unit is periodically written into the attribute file maintained by the data node based on the disk; The step of retrieving the attribute information of the target file that matches the target identification information from the target logical storage unit includes: Search the target logical storage unit for attribute information of the target file that matches the target identification information; If the attribute information of the target file cannot be found in the target logical storage unit, the attribute information of the target file is obtained through the disk.

4. The method according to claim 3, characterized in that, The attribute file includes multiple first storage blocks and at least one second storage block. The first storage blocks are used to store file attribute information, and the second storage blocks are used to store the index information of the file attribute information in the first storage blocks and the position information of the first storage blocks. The step of obtaining the attribute information of the target file through the disk includes: Based on the index information stored in the second storage block, a target storage block for storing the attribute information of the target file is determined from the plurality of first storage blocks; Based on the location information of the target storage block stored in the second storage block, the target storage block is located, and the attribute information of the target file is obtained from the target storage block.

5. The method according to claim 1, characterized in that, The method further includes: In response to receiving the client's attribute setting request for the target file, the attribute setting information carried in the attribute setting request and the first auto-incrementing sequence number associated with the attribute setting information are written into the write-ahead log. The attribute information of the target file recorded in the target logical storage unit is updated based on the attribute setting information to obtain the latest attribute information of the target file, and the latest attribute information of the target file is associated with the first auto-incrementing sequence number. Based on the first auto-incrementing sequence number corresponding to the target file recorded in the target logical storage unit, invalid attribute information of the target file existing in the write-ahead log is detected and the detected invalid attribute information is deleted; wherein, the second auto-incrementing sequence number corresponding to the invalid attribute information of the target file is less than the first auto-incrementing sequence number.

6. The method according to claim 5, characterized in that, The logical storage unit is located in the memory of the data node. When the attribute information of the target file also includes extended attributes, the attribute information of the target file recorded in the target logical storage unit includes: the non-extended attributes of the target file, and the position of the extended attributes of the target file in the write-ahead log.

7. A method for processing file attributes in a distributed file system, characterized in that, include: The client sends a request to the metadata node of the distributed file system to obtain the identification information of the target file. The system receives the target identification information of the target file returned by the metadata node, and determines the target logical storage unit based on the target identification information; wherein, the target logical storage unit is a logical storage unit used to store the attribute information of the target file; the target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit; the metadata node records the mapping relationship between the logical storage unit and the data node; Based on the acquired target mapping relationship, the target data node to which the target logical storage unit belongs is determined; wherein, the target mapping relationship is the mapping relationship between the logical storage unit and the data node; The client sends a file attribute query request to the target data node, and obtains the attribute information of the target file through the target data node; the file attribute query request carries the target identifier information of the target file.

8. The method according to claim 7, characterized in that, The method further includes: The latest target mapping relationship is periodically obtained through the metadata node.

9. A file attribute processing device for a distributed file system, characterized in that, include: The request receiving module is used to receive file attribute query requests from clients through data nodes in the distributed file system. The file attribute query requests carry target identifier information of the target file and are used to request the query of the attribute information of the target file. The target identification information is obtained from the metadata node of the distributed file system; The target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit; the metadata node records the mapping relationship between the logical storage unit and the data node; The data node corresponding to the file attribute query request is determined by the client extracting the unit identifier of the target logical storage unit from the file identifier and based on the mapping relationship; A storage unit determination module is used to determine, based on the target identification information, a target logical storage unit for storing attribute information of the target file from at least one logical storage unit contained in the data node; The data node dynamically maintains the attribute information of each file based on at least one logical storage unit; The attribute acquisition module is used to acquire attribute information of the target file that matches the target identification information from the target logical storage unit; The request-response module is used to respond to the file attribute query request based on the attribute information of the target file.

10. A file attribute processing device for a distributed file system, characterized in that, include: The request sending module is used to send a request to the metadata node of the distributed file system to obtain the identification information of the target file through the client; The unit determination module is used to receive the target identification information of the target file returned by the metadata node, and determine the target logical storage unit based on the target identification information; wherein, the target logical storage unit is a logical storage unit used to store the attribute information of the target file; the target identification information is represented by the file identifier of the target file, and the file identifier of the target file embeds the unit identifier of the target logical storage unit; the metadata node records the mapping relationship between logical storage units and data nodes; The node determination module is used to determine the target data node to which the target logical storage unit belongs based on the acquired target mapping relationship; wherein, the target mapping relationship is the mapping relationship between logical storage units and data nodes, and a data node contains at least one logical storage unit; The information acquisition module is used to send a file attribute query request to the target data node through the client, and obtain the attribute information of the target file through the target data node; the file attribute query request carries the target identifier information of the target file.

11. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device is configured to execute the computer program in the storage device to implement the file attribute processing method of the distributed file system according to any one of claims 1-6 or the file attribute processing method of the distributed file system according to claim 7 or 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is used to execute the file attribute processing method of the distributed file system according to any one of claims 1-6 or the file attribute processing method of the distributed file system according to claim 7 or 8.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the file attribute processing method of the distributed file system according to any one of claims 1-6 or the file attribute processing method of the distributed file system according to claim 7 or 8.

Images