File attribute processing method and device of distributed file system, equipment and medium

Abstract

The embodiment of the invention relates to a file attribute processing method and device of a distributed file system, equipment and a medium, and the method comprises the steps that a file attribute query request of a client side is received through a data node in the distributed file system, and the file attribute query request carries target identification information of a target file; the query module requests to query attribute information of a target file; the target identification information is obtained from a metadata node in the distributed file system; according to the target identification information, determining a target logic storage unit used for storing attribute information of the target file from at least one logic storage unit contained in the data node; obtaining attribute information of a target file matched with the target identification information from a target logic storage unit; and responding to the file attribute query request based on the attribute information of the target file. According to the embodiment of the invention, the file attribute can be unloaded from the metadata node to the data node, so that the pressure of the metadata node can be effectively reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a file attribute processing method and device of a distributed file system, equipment and medium. BACKGROUND

[0002] Most of the existing distributed file systems divide metadata nodes and data nodes, the metadata nodes are used to store directory trees and attributes of directories and files, and the data nodes are used to store data. At present, attribute reading and writing and other related processing operations need to depend on metadata nodes, and the pressure of the metadata nodes is large, which becomes a bottleneck for the scale expansion of the distributed file system. SUMMARY

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a file attribute processing method, device, equipment and medium of a distributed file system.

[0004] The present disclosure provides a file attribute processing method of a distributed file system, comprising: receiving a file attribute query request of a client by a data node in the distributed file system, the file attribute query request carrying target identification information of a target file, and being used to request to query attribute information of the target file; the target identification information is obtained from a metadata node in the distributed file system; determining a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained by the data node according to the target identification information; the data node dynamically maintains attribute information of each file based on at least one logical storage unit; obtaining the attribute information of the target file matched with the target identification information from the target logical storage unit; and responding to the file attribute query request based on the attribute information of the target file.

[0005] Optionally, the target identification information is represented by a file identifier of the target file, and the file identifier of the target file is embedded with a unit identifier of the target logical storage unit; the determining of the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained by the data node according to the target identification information comprises: analyzing the file identifier of the target file, and extracting the embedded unit identifier from the file identifier of the target file; and determining the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained by the data node based on the unit identifier.

[0006] Optionally, the data node maintains the logical storage unit based on memory, and the method further comprises: periodically writing file attribute information stored in the logical storage unit into an attribute file maintained by the data node based on disk; and the attribute information of the target file matching the target identification information is obtained from the target logical storage unit, comprising: searching for the attribute information of the target file matching the target identification information from the target logical storage unit; and in the case that the attribute information of the target file is not found from the target logical storage unit, obtaining the attribute information of the target file through the disk.

[0007] Optionally, the attribute file comprises a plurality of first storage blocks and at least one second storage block, the first storage blocks are used to store file attribute information, and the second storage block is used to store index information of the file attribute information in the first storage blocks and position information of the first storage blocks; and the attribute information of the target file is obtained through the disk, comprising: determining a target storage block used to store the attribute information of the target file from the plurality of first storage blocks based on the index information stored in the second storage block; and locating the target storage block based on the position information of the target storage block stored in the second storage block, and obtaining the attribute information of the target file from the target storage block.

[0008] Optionally, the method further comprises: in response to receiving the attribute setting request of the client for the target file, writing attribute setting information carried by the attribute setting request and a first auto-incremental serial number associated with the attribute setting information into a prewrite log; updating the attribute information of the target file recorded in the target logical storage unit based on the attribute setting information, obtaining the latest attribute information of the target file, and associating the latest attribute information of the target file with the first auto-incremental serial number; detecting invalid attribute information of the target file existing in the prewrite log based on the first auto-incremental serial number corresponding to the target file recorded in the target logical storage unit, and deleting the detected invalid attribute information; wherein the second auto-incremental serial number corresponding to the invalid attribute information of the target file is less than the first auto-incremental serial number.

[0009] Optionally, the logical storage unit is located in the memory of the data node, and in the case that the attribute information of the target file further comprises extended attribute, the attribute information of the target file recorded in the target logical storage unit comprises: non-extended attribute of the target file, and position of the extended attribute of the target file in the prewrite log.

[0010] The embodiment of the present disclosure provides a file attribute processing method of a distributed file system, comprising: sending, by a client, an identification information acquisition request of a target file to a metadata node of the distributed file system; receiving target identification information of the target file returned by the metadata node, and determining a target logical storage unit based on the target identification information; wherein the target logical storage unit is a logical storage unit used for storing attribute information of the target file; determining a target data node to which the target logical storage unit belongs based on a target mapping relationship that has been acquired; wherein the target mapping relationship is a mapping relationship between a logical storage unit and a data node; and acquiring attribute information of the target file through the target data node.

[0011] Optionally, the method further comprises: periodically acquiring, by the metadata node, a latest target mapping relationship.

[0012] The embodiment of the present disclosure further provides a file attribute processing device of a distributed file system, comprising: a request receiving module, configured to receive a file attribute query request of a client through a data node in a distributed file system, wherein the file attribute query request carries target identification information of a target file, and is used for requesting to query attribute information of the target file; the target identification information is obtained from a metadata node in the distributed file system; a storage unit determining module, configured to determine, according to the target identification information, a target logical storage unit used for storing attribute information of the target file from at least one logical storage unit contained in the data node; the data node dynamically maintains attribute information of each file based on at least one logical storage unit; an attribute acquiring module, configured to acquire, from the target logical storage unit, attribute information of the target file matched with the target identification information; and a request responding module, configured to respond to the file attribute query request based on the attribute information of the target file.

[0013] The embodiment of the present disclosure further provides a file attribute processing device of a distributed file system, comprising: a request sending module, configured to send, by a client, an identification information acquisition request of a target file to a metadata node of the distributed file system; a unit determining module, configured to receive target identification information of the target file returned by the metadata node, and determine a target logical storage unit based on the target identification information; wherein the target logical storage unit is a logical storage unit used for storing attribute information of the target file; a node determining module, configured to determine a target data node to which the target logical storage unit belongs based on a target mapping relationship that has been acquired; wherein the target mapping relationship is a mapping relationship between a logical storage unit and a data node, and one data node contains at least one logical storage unit; and an information acquiring module, configured to acquire attribute information of the target file through the target data node.

[0014] The electronic device includes a processor, a memory for storing executable instructions for the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the file attribute processing method of the distributed file system.

[0015] The electronic device includes a processor, a memory for storing executable instructions for the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the file attribute processing method of the distributed file system.

[0016] The electronic device includes a processor, a memory for storing executable instructions for the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the file attribute processing method of the distributed file system.

[0017] The electronic device includes a processor, a memory for storing executable instructions for the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the file attribute processing method of the distributed file system.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure together with the specification.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0021] Figure 1 A flowchart of a file attribute processing method of a distributed file system provided by an embodiment of the present disclosure is shown in the figure.

[0022] Figure 2 A structural diagram of an attribute file provided by an embodiment of the present disclosure is shown in the following figure;

[0023] Figure 3 A flowchart of a file attribute processing method of a distributed file system provided by an embodiment of the present disclosure is shown in the following figure;

[0024] Figure 4 An interactive flowchart of a file attribute processing method of a distributed file system provided by an embodiment of the present disclosure is shown in the following figure;

[0025] Figure 5 A structural diagram of a file attribute processing device of a distributed file system provided by an embodiment of the present disclosure is shown in the following figure;

[0026] Figure 6 A structural diagram of a file attribute processing device of a distributed file system provided by an embodiment of the present disclosure is shown in the following figure;

[0027] Figure 7 A structural diagram of an electronic device provided by an embodiment of the present disclosure is shown in the following figure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the schemes of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the description are only some of the embodiments of the present disclosure, not all the embodiments.

[0030] Figure 1 A flowchart of a file attribute processing method of a distributed file system provided by an embodiment of the present disclosure is shown in the following figure, which can be executed by a file attribute processing device of a distributed file system, wherein the device can be implemented by software and / or hardware, and can be integrated in an electronic device, and the electronic device can be a data node of a distributed file system, such as Figure 1 As shown in the figure, the method mainly includes the following steps S102-S106:

[0031] In step S102, a file attribute query request of a client is received by a data node in the distributed file system, the file attribute query request carrying target identification information of a target file, used for requesting to query attribute information of the target file; the target identification information is obtained from a metadata node in the distributed file system. That is, the client can obtain an identification information (i.e. the target identification information) from the metadata node, provide the identification information to the data node, and the data node can determine the target file whose attribute information needs to be queried based on the identification information, and perform a query operation of the attribute information of the target file.

[0032] In step S104, a target logical storage unit used for storing attribute information of the target file is determined from at least one logical storage unit contained by the data node according to the target identification information; the data node dynamically maintains attribute information of each file based on the at least one logical storage unit.

[0033] Exemplarily, the embodiments of the present disclosure can create a plurality of logical storage units when creating a distributed file system, each storage unit has a unique unit identifier, and the metadata node can uniformly distribute a plurality of logical storage units to the data nodes. In other words, each data node in the distributed file system contains one or more logical storage units, and the mapping relationship between the logical storage units and the data nodes is recorded in the metadata node, that is, it records which logical storage units are on each data node. The data node can dynamically maintain the attribute information of each file based on at least one logical storage unit. Exemplarily, the data node can store the received file attribute information in the logical storage unit, and dynamically adjust the attribute information of the file in the logical storage unit based on the user's operation on the file. When the client creates a file through the metadata node, the metadata node can uniformly distribute the file attribute information to a plurality of logical storage units for storage, such as preferentially distributing the file attribute information to a logical storage unit with small load. Exemplarily, when the client creates a new file, the metadata node can determine a logical storage unit (referred to as a target unit) to store the attribute information of the new file based on the uniform distribution strategy from the existing plurality of logical storage units, and generate target identification information (which can be represented by a file identifier) corresponding to the new file. The unit identifier of the logical storage unit used to store the attribute information of the new file (that is, the unit identifier of the target unit) can also be encoded in the target identification information. The metadata node can provide the target identification information of the target file to the client when receiving the identification information acquisition request of the target file initiated by the client, so that the client can request the data node to which the target unit belongs to query the attribute information of the target file based on the target identification information. When the metadata node obtains the attribute information of the new file, it can initiate a storage instruction (carrying the attribute information of the new file and the target identification information of the new file) of the attribute information of the new file to the data node to which the target unit belongs, so that the data node stores the attribute information of the new file in the target unit. In some specific examples, the data node provided by the embodiments of the present disclosure can store the file attribute information in the logical storage unit (such as the target unit) corresponding to the unit identifier embedded in the file identifier based on the file attribute information to be stored (such as the attribute information of the new file) carried by the file attribute storage instruction of the metadata node and the file identifier corresponding to the file attribute information, in response to the file attribute storage instruction of the metadata node. Through the above-mentioned manner, the file attribute information can be offloaded to the data node for storage, and the pressure on the metadata node can be reduced.

[0034] To facilitate simple and convenient identification of the logical storage unit corresponding to the file attribute information, in some embodiments, the target identification information is represented by a file identifier of the target file, and the file identifier of the target file is embedded with a cell identifier of the target logical storage unit. That is, the embodiment of the present disclosure can encode the cell identifier of the logical storage unit into the file identifier, such as the file identifier including M bits in total, the first N bits of the M bits being the cell identifier, M and N being positive integers, M being greater than N, and the specific values can be flexibly set according to requirements, and exemplarily, N = 16. The number of bits in the M bits other than the first N bits can be used to uniquely identify the file. On this basis, the step of determining the target logical storage unit for storing the attribute information of the target file from the at least one logical storage unit included in the data node according to the target identification information can be specifically executed by parsing the file identifier of the target file, extracting the embedded cell identifier from the file identifier of the target file, such as extracting the first 16 bits as the cell identifier; and then determining the target logical storage unit for storing the attribute information of the target file from the at least one logical storage unit included in the data node based on the cell identifier. Through the above-mentioned manner, the file can be uniquely identified based on the file identifier, and the logical storage unit for storing the attribute information of the file can be explicitly indicated based on the file identifier.

[0035] The file attribute information of the embodiment of the present disclosure includes but is not limited to the size, modification time, user identifier, group identifier and extended attribute of the file, and all attributes of the file are set in the logical storage unit of the data node, so as to reduce the pressure on the metadata node as much as possible.

[0036] Step S106, obtaining the attribute information of the target file matching the target identification information from the target logical storage unit. Since the target logical storage unit is the logical storage unit for storing the attribute information of the target file, after the target logical storage unit is determined, the required attribute information of the target file can be directly obtained based on the target logical storage unit.

[0037] Step S108, responding to the file attribute query request based on the attribute information of the target file.

[0038] The technical solution provided by the embodiments of the present disclosure is not to store the file attribute information in the metadata node of the distributed file system, but to store the file attribute information in the logical storage unit of the data node of the distributed file system. The target identification information obtained by the client through the metadata node, and based on the target identification information, the attribute information of the target file is obtained through the data node containing the target logical storage unit (that is, the logical storage unit storing the attribute information of the target file). This way can unload the file attribute from the metadata node to the data node, and the client can obtain the required file attribute information through the data node, which can effectively reduce the pressure of the metadata node.

[0039] In some embodiments, the data node maintains the logical storage unit based on the memory. On this basis, the method provided by the embodiments of the present disclosure further includes: periodically writing the file attribute information stored in the logical storage unit into the attribute file maintained by the data node based on the disk. In this way, the attribute file is located in the disk of the data node, and the file attribute information can be deleted from the memory after being written into the attribute file. In this way, the memory space can be effectively saved. It can be understood that the memory space is limited. If all file attribute information is stored in the memory, a large amount of memory space will be occupied. In addition, the full amount of file attribute information also needs to be loaded when the data node starts, which slows down the starting speed. Therefore, the lazy loading method is adopted in the embodiments of the present disclosure. Specifically, the file attribute information stored in the logical storage unit is written into the attribute file in the disk of the data node at regular intervals. Only when the file attribute information is accessed, the file attribute information is loaded from the disk to the memory. In this way, the memory space can be effectively saved, and the starting speed can be improved.

[0040] On the basis of the foregoing, the step of obtaining the attribute information of the target file matching the target identification information from the target logical storage unit can be performed according to the following steps (1) and (2):

[0041] Step (1), searching for the attribute information of the target file matching the target identification information from the target logical storage unit.

[0042] Step (2), in the case that the attribute information of the target file is not found from the target logical storage unit, obtaining the attribute information of the target file through the disk. It can be understood that if the attribute information of the target file has been written from the target logical storage unit to the disk, and the attribute information of the target file in the target logical storage unit has been cleaned, the attribute information of the target file cannot be found from the target logical storage unit. At this time, the attribute information of the target file can be directly obtained through the disk.

[0043] In order to improve the speed of obtaining file attribute information from the disk, in some embodiments, the aforementioned attribute file includes a plurality of first storage blocks and at least one second storage block, the first storage blocks are used to store file attribute information, the second storage block is used to store index information of the file attribute information in the first storage blocks and position information of the first storage blocks, the index information of the file attribute information in the first storage blocks can be used to indicate the mapping relationship between the first storage blocks and the file attribute information, that is, to indicate which file attribute information is stored in each first storage block; the position information can be comprehensively represented based on two parameters of offset address and length. In actual application, the aforementioned storage blocks can be managed based on a B+ tree, the B+ tree is a multi-way balanced search tree and belongs to an index structure, and can be used to efficiently and reliably realize the storage and searching of file attribute information.

[0044] On the basis of the foregoing, the step of obtaining attribute information of a target file through the disk can be performed according to the following steps A and B:

[0045] Step A: determining a target storage block for storing attribute information of a target file from the plurality of first storage blocks based on the index information stored in the second storage block. In actual application, the file attribute information in each first storage block can be sorted according to the size of the file identifier, and the target storage block for storing attribute information of the target file can be quickly determined based on binary search.

[0046] Step B: locating the target storage block based on the position information of the target storage block stored in the second storage block, and obtaining attribute information of the target file from the target storage block.

[0047] In order to further improve the searching speed of file attribute information, the first storage block includes a storage area and an index area, that is, an additional index is added in the first storage block, the storage area is used to store file attribute information, and the index area is used to store the position of the file attribute information in the storage area; on this basis, the specific step of obtaining attribute information of the target file from the target storage block in step B can be performed according to the following steps 1 and 2:

[0048] Step 1: obtaining the position of attribute information of the target file in the storage area of the target storage block based on the index area of the target storage block.

[0049] Step 2: obtaining attribute information of the target file from the storage area of the target storage block based on the position of attribute information of the target file in the storage area of the target storage block.

[0050] In the above manner, the speed of searching attribute information of the target file from the target storage block is further improved, and the reading and writing time of file attribute information is shortened.

[0051] Exemplarily, the above can be performed according to the following steps:Figure 2 A storage structure diagram is shown, assuming that the attribute file includes storage block 1 to storage block 7, wherein storage block 1 to storage block 4 are all the first storage blocks, and storage block 5 to storage block 7 are all the second storage blocks, storage block 1 to 4 each include a storage area and an index area, the storage area is used to store file attribute information corresponding to a file identifier, and the index area is used to store attribute information addresses of each file identifier, which are arranged in ascending order of file identifiers. Assuming that the attribute information of file 3 needs to be obtained, that is, the attribute information corresponding to file identifier 3 needs to be obtained, the storage block containing the required file attribute information can be quickly found in the above storage structure based on binary search. Assuming that storage block 7 is taken as a root node, storage block 7 indicates that storage block 5 stores information related to file identifiers 1 to 4, and storage block 6 stores information related to file identifiers 5 to 8, based on which it can be quickly determined that the information related to file identifier 3 is located in storage block 5, and storage block 5 indicates that storage block 1 stores information related to file identifiers 1 to 2, and storage block 2 stores information related to file identifiers 3 to 4, based on which it can be quickly determined that the information related to file identifier 3 is located in storage block 2. Then the attribute information address of file identifier 3 can be obtained from the index area of storage block 2, and then the file attribute information of file identifier 3 can be obtained from the storage area based on the address. Through the above method, the required file attribute information can be efficiently and accurately obtained from multiple storage blocks in the attribute file.

[0052] In actual application, when reading a storage block (first storage block or second storage block), the storage block can be cached in the memory, and when searching for file attribute information corresponding to a file identifier from the storage block, the file attribute information can be cached in the memory. When a data node needs to obtain attribute information of a target file, the attribute information of the target file can be searched from the cached file attribute information in the memory first, if the attribute information of the target file is found, it is directly returned to the client, if the attribute information of the target file is not found, the B+ tree can be used for searching, specifically, the target storage block can be searched from the storage block cache in the memory first, if the target storage block is found, the required file attribute information is obtained based on the target storage block, if the target storage block is not found, the target storage block can be read from the disk, and the attribute information of the target file is searched from the target storage block, cached in the memory, and returned to the client. Through the above method, frequent reading of the disk can be effectively avoided, and attribute query efficiency can be improved, and attribute query time can be shortened.

[0053] In some embodiments, the above method further includes steps a to c:

[0054] Step a, in response to receiving a client attribute setting request for a target file, writing attribute setting information carried by the attribute setting request and a first auto-incrementing sequence number associated with the attribute setting information into a pre-write log. The pre-write log is located in a disk and can be used to ensure data consistency and persistence. When receiving a setting request for file attribute information, a data node will first record in the pre-write log and then update the memory. Embodiments of the present disclosure can associate an auto-incrementing sequence number with each attribute setting information, that is, a gradually increasing sequence number. The first auto-incrementing sequence number corresponding to the attribute setting information after is greater than the first auto-incrementing sequence number of the attribute setting information before.

[0055] Step b, updating the attribute information of the target file already recorded in the target logical storage unit based on the attribute setting information, obtaining the latest attribute information of the target file, and associating the latest attribute information of the target file with the first auto-incrementing sequence number.

[0056] Step c, detecting invalid attribute information of the target file existing in the pre-write log based on the first auto-incrementing sequence number corresponding to the target file recorded in the target logical storage unit, and deleting the detected invalid attribute information; wherein the second auto-incrementing sequence number corresponding to the invalid attribute information of the target file is less than the first auto-incrementing sequence number.

[0057] It can be understood that attribute information includes file size, file modification time, user identifier, extended attribute, and various attributes, each of which can be set individually or repeatedly. If it is repeatedly set, invalid attributes can be cleaned up to avoid occupying storage resources. For example, the file size is set to 10 and 9 in succession, and the final file size is 9. The file size recorded in the target logical storage unit in the memory is also the latest value 9. At this time, the file size 10 in the pre-write log can be deleted. Embodiments of the present disclosure can use the auto-incrementing sequence number to quickly identify invalid attribute information in the pre-write log. If the auto-incrementing sequence number corresponding to a file attribute recorded in the pre-write log is less than the auto-incrementing sequence number corresponding to the file attribute recorded in the memory, the file attribute recorded in the pre-write log can be cleaned up. For example, the auto-incrementing sequence number corresponding to the file size 9 in the memory is 2, and the auto-incrementing sequence number corresponding to the file size 10 in the pre-write log is 1. The file size 10 in the pre-write log is considered as invalid attribute information and can be cleaned up. Through the above manner, invalid data can be effectively cleaned up to avoid resource waste.

[0058] Considering that some files also have extended attributes, the extended attributes can be custom attributes, and the difference between the extended attributes and the non-extended attributes is that the values of the extended attributes can be larger, thus occupying larger storage space and being inconvenient to be placed in the memory entirely. The logical storage unit is located in the memory of the data node, and thus in the case where the attribute information of the target file also includes the extended attributes, the attribute information of the target file recorded in the target logical storage unit includes the non-extended attributes of the target file and the location of the extended attributes of the target file in the pre-write log. The non-extended attributes are attributes other than the extended attributes in all attributes of the target file. Exemplarily, the value of a single extended attribute can be up to 64 KB, and each record of the pre-write log can be only 4 KB, and thus the extended attributes can occupy multiple records. In order to simplify the processing, only the location of the extended attributes in the pre-write log, i.e., the offset address and the length of the extended attributes in the pre-write log, can be recorded in the logical storage unit in the memory, and in this way, various types of attributes can be effectively stored, and even larger extended attributes can be properly handled.

[0059] Figure 3 A flowchart of a file attribute processing method of a distributed file system provided by the embodiments of the present disclosure is provided, and the method can be executed by a file attribute processing device of the distributed file system. The device can be implemented by software and / or hardware, and can be integrated in an electronic device. Exemplarily, the electronic device can be a client, such as a computer. Figure 3 As shown in the figure, the method mainly includes the following steps S302-S308.

[0060] In actual application, the identification information acquisition request can carry the name and path of the target file, so as to refer to the target file by the name and path combination. The client can acquire the target identification information of the target file from the metadata node when the attribute information of the target file needs to be acquired, and the target identification information can be formed by a plurality of identifiers in the form of a file identifier unique to the target file. In actual application, the file identifier of each file is allocated by the metadata node.

[0061] In actual application, the identification information acquisition request can carry the name and path of the target file, so as to refer to the target file by the name and path combination. The client can acquire the target identification information of the target file from the metadata node when the attribute information of the target file needs to be acquired, and the target identification information can be formed by a plurality of identifiers in the form of a file identifier unique to the target file. In actual application, the file identifier of each file is allocated by the metadata node.

[0062] Step S306, based on the obtained target mapping relationship, determine the target data node to which the target logical storage unit belongs; wherein the target mapping relationship is the mapping relationship between the logical storage unit and the data node, and one data node contains at least one logical storage unit. The target mapping relationship can clearly indicate which logical storage unit is located on which data node, and the client can quickly determine the target data node to which the target logical storage unit belongs based on the target mapping relationship based on the known target logical storage unit.

[0063] Considering that there may be changes in the mapping relationship due to data node failure and other problems, the latest target mapping relationship can be obtained periodically through the metadata node. In actual application, the client can continuously obtain the latest target mapping relationship from the metadata node based on the heartbeat mechanism, and in this way, the accuracy of the obtained target mapping relationship can be effectively guaranteed, and the target data node to which the target logical storage unit belongs can be accurately determined based on the target mapping relationship.

[0064] Step S308, obtain the attribute information of the target file through the target data node. The operation of the client after obtaining the attribute information of the target file is not limited by the embodiments of the present disclosure.

[0065] The above technical solutions provided by the embodiments of the present disclosure do not store the file attribute information in the metadata node of the distributed file system, but store the file attribute information in the logical storage unit of the data node of the distributed file system. The client can obtain the target identification information through the metadata node, and based on the target identification information, obtain the attribute information of the target file through the data node containing the target logical storage unit (i.e. the logical storage unit storing the attribute information of the target file). This way can unload the file attribute from the metadata node to the data node, and the client can obtain the required file attribute information through the data node, which can effectively reduce the pressure of the metadata node.

[0066] For ease of understanding, a distributed file system file attribute processing interaction flow diagram can be referred to as shown in Figure 4 The interaction flow between the client, the metadata node, and the target data node is shown, mainly including the following steps S402-S414:

[0067] Step S402, the client sends an identification information obtaining request of a target file to the metadata node.

[0068] Step S404, the metadata node returns the file identifier of the target file to the client, and the file identifier of the target file is embedded with the unit identifier of the target logical storage unit.

[0069] Step S406, the client extracts the unit identifier of the target logical storage unit from the file identifier, and determines a target data node to which the target logical storage unit belongs based on a mapping relationship between the logical storage unit and the data node.

[0070] Step S408, the client sends a file attribute query request to the target data node, and the file attribute query request carries the file identifier of the target file.

[0071] Step S410, the target data node determines the target file to be queried for attributes and the target logical storage unit containing attribute information of the target file based on the file identifier.

[0072] Step S412, the target data node acquires the attribute information of the target file based on the target logical storage unit.

[0073] Step S414, the target data node returns the attribute information of the target file to the client.

[0074] To sum up, the file attribute processing method of the distributed file system provided by the embodiments of the present disclosure can offload file attributes from a metadata node to a data node, and the client can acquire the required file attribute information through the data node, which can effectively reduce the pressure of the metadata node and help to increase the cluster size.

[0075] Corresponding to the file attribute processing method of the distributed file system, the embodiments of the present disclosure further provide a file attribute processing device of a distributed file system, Figure 5 A structural diagram of the file attribute processing device of the distributed file system provided by the embodiments of the present disclosure, which can be implemented by software and / or hardware, and can be integrated in an electronic device, which can be a data node of a distributed file system, such as Figure 5 As shown in the figure, the file attribute processing device of the distributed file system comprises:

[0076] The request receiving module 502 is configured to receive a file attribute query request of a client through a data node in the distributed file system, the file attribute query request carrying target identification information of a target file, and the target identification information being obtained from a metadata node in the distributed file system, and the target identification information being used to request to query attribute information of the target file.

[0077] The storage unit determining module 504 is configured to determine a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit contained in the data node according to the target identification information, and the data node dynamically maintaining the attribute information of each file based on the at least one logical storage unit.

[0078] The attribute obtaining module 506 is configured to obtain attribute information of a target file matching the target identification information from the target logical storage unit.

[0079] The request responding module 508 is configured to respond to the file attribute query request based on the attribute information of the target file.

[0080] The above device can offload file attributes from a metadata node to a data node, and a client can obtain required file attribute information through the data node, thereby effectively reducing the pressure on the metadata node.

[0081] In some embodiments, the target identification information is represented by a file identifier of the target file, and the file identifier of the target file is embedded with a unit identifier of the target logical storage unit. The storage unit determining module 504 is specifically configured to: parse the file identifier of the target file, extract the embedded unit identifier from the file identifier of the target file; and determine the target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node based on the unit identifier.

[0082] In some embodiments, the data node maintains the logical storage units based on memory, and the device further includes a disk writing module configured to periodically write file attribute information stored by the logical storage unit into an attribute file maintained by the data node based on a disk.

[0083] In some embodiments, the attribute information obtaining module 504 is specifically configured to: search for the attribute information of the target file matching the target identification information from the target logical storage unit; and in a case where the attribute information of the target file is not found from the target logical storage unit, obtain the attribute information of the target file through the disk.

[0084] In some embodiments, the attribute file includes a plurality of first storage blocks and at least one second storage block, the first storage blocks are configured to store file attribute information, and the second storage block is configured to store index information of the file attribute information in the first storage blocks and position information of the first storage blocks. The attribute information obtaining module 504 is specifically configured to: determine a target storage block for storing the attribute information of the target file from the plurality of first storage blocks based on the index information stored by the second storage block; locate the target storage block based on the position information of the target storage block stored by the second storage block, and obtain the attribute information of the target file from the target storage block.

[0085] In some embodiments, the first storage block includes a storage area and an index area; the storage area is configured to store file attribute information, and the index area is configured to store the location of the file attribute information in the storage area; the attribute information obtaining module 504 is specifically configured to: based on the index area of the target storage block, obtain the location of the attribute information of the target file in the storage area of the target storage block; and based on the location of the attribute information of the target file in the storage area of the target storage block, obtain the attribute information of the target file from the storage area of the target storage block.

[0086] In some embodiments, the apparatus further includes an invalid information deleting module configured to, in response to receiving the attribute setting request of the target file from the client, write the attribute setting information carried by the attribute setting request and the first self-incrementing serial number associated with the attribute setting information into a prewrite log; update the attribute information of the target file recorded in the target logical storage unit based on the attribute setting information, to obtain the latest attribute information of the target file, and associate the latest attribute information of the target file with the first self-incrementing serial number; based on the first self-incrementing serial number corresponding to the target file recorded in the target logical storage unit, detect the invalid attribute information of the target file existing in the prewrite log, and delete the detected invalid attribute information; wherein the second self-incrementing serial number corresponding to the invalid attribute information of the target file is less than the first self-incrementing serial number.

[0087] In some embodiments, the logical storage unit is located in the memory of the data node, and in the case that the attribute information of the target file further includes extended attributes, the attribute information of the target file recorded in the target logical storage unit includes: the non-extended attribute of the target file, and the location of the extended attribute of the target file in the prewrite log.

[0088] Corresponding to the file attribute processing method of the foregoing distributed file system, the embodiments of the present disclosure further provide a file attribute processing apparatus of a distributed file system, Figure 6 A structural diagram of a file attribute processing apparatus of a distributed file system provided by the embodiments of the present disclosure, which can be implemented by software and / or hardware, and can be integrated in an electronic device, which can be a client, such as a computer, a mobile phone, a tablet computer, a server, etc. Figure 6 As shown in the figure, the file attribute processing apparatus of the distributed file system includes:

[0089] The request sending module 602 is configured to send an identification information obtaining request of a target file to a metadata node of a distributed file system through a client;

[0090] The unit determining module 604 is configured to receive target identification information of the target file returned by the metadata node, and determine a target logical storage unit for storing attribute information of the target file based on the target identification information.

[0091] The node determining module 606 is configured to determine a target data node to which the target logical storage unit belongs based on the obtained target mapping relationship. The target mapping relationship is a mapping relationship between a logical storage unit and a data node, and one data node contains at least one logical storage unit.

[0092] The information obtaining module 608 is configured to obtain the attribute information of the target file through the target data node.

[0093] The above device can offload file attributes from a metadata node to a data node, and a client can obtain required file attribute information through the data node, so that the pressure on the metadata node can be effectively reduced.

[0094] In some embodiments, the device comprises a relationship obtaining module configured to periodically obtain the latest target mapping relationship through the metadata node.

[0095] The file attribute processing device of the distributed file system provided in the embodiments of the present disclosure can execute the file attribute processing method of the distributed file system provided in any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device embodiments described above can refer to the corresponding process in the method embodiments, which will not be described here.

[0097] The embodiments of the present disclosure provide an electronic device, which comprises a storage device having a computer program stored thereon, and a processing device configured to execute the computer program in the storage device to implement the steps of any method in the present disclosure.

[0098] Reference will be made to the following description Figure 7 which shows a structural schematic diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital televisions, desktop computers, and the like. Figure 7The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0099] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0100] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0101] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0102] In addition to the method and device described above, the embodiments of the present disclosure can also be a computer program product, which includes computer program instructions that make the processor execute the method provided by the embodiments of the present disclosure when the processor is running. The computer program product can be written in any combination of one or more programming languages to execute the program code of the embodiments of the present disclosure, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] In addition, the embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the method provided by the embodiments of the present disclosure.

[0104] The computer readable storage medium can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of readable storage medium include: electrical connections having one or more wires, portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical fibers, portable compact disk read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0105] The embodiments of the present disclosure also provide a computer program product, which includes computer programs / instructions that are executed by a processor to implement the method in the embodiments of the present disclosure.

[0106] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0107] For example, in response to receiving the user's active request, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can voluntarily choose whether to provide personal information to the electronic device, application program, server or storage medium, etc. software or hardware that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0108] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0109] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0111] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing file attributes in a distributed file system, characterized in that: include: Receive a file attribute query request from a client through a data node in the distributed file system, wherein the file attribute query request carries target identification information of a target file and is used to request to query attribute information of the target file; The target identification information is obtained from a metadata node in the distributed file system; Determining, according to the target identification information, a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node; The data node dynamically maintains attribute information of each file based on at least one logical storage unit; Acquire attribute information of the target file that matches the target identification information from the target logical storage unit; The file attribute query request is responded to based on the attribute information of the target file.

2. The method according to claim 1, characterized in that The target identification information is represented by a file identifier of the target file, and the file identifier of the target file is embedded with a unit identifier of the target logical storage unit; and determining, based on the target identification information, a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node includes: Parsing the file identifier of the target file, and extracting the embedded unit identifier from the file identifier of the target file; Based on the unit identifier, a target logical storage unit for storing the attribute information of the target file is determined from at least one logical storage unit included in the data node.

3. The method according to claim 1, characterized in that The data node maintains the logical storage unit based on memory, and the method further includes: Periodically writing the file attribute information stored in the logical storage unit into the attribute file maintained by the data node based on the disk; The acquiring, from the target logical storage unit, the attribute information of the target file that matches the target identification information includes: Searching the target logical storage unit for attribute information of the target file that matches the target identification information; When the attribute information of the target file is not found in the target logical storage unit, the attribute information of the target file is obtained through the disk.

4. The method according to claim 3, characterized in that The attribute file includes a plurality of first storage blocks and at least one second storage block, wherein the first storage blocks are used to store file attribute information, and the second storage block is used to store index information of the file attribute information in the first storage blocks and location information of the first storage block; The acquiring the attribute information of the target file through the disk includes: Determining, based on the index information stored in the second storage block, a target storage block for storing the attribute information of the target file from the plurality of first storage blocks; Based on the location information of the target storage block stored in the second storage block, the target storage block is located, and the attribute information of the target file is obtained from the target storage block.

5. The method according to claim 1, wherein The method further comprises: In response to receiving a property setting request from the client for the target file, writing the property setting information carried in the property setting request and a first auto-increment sequence number associated with the property setting information into a write-ahead log; updating the attribute information of the target file recorded in the target logical storage unit based on the attribute setting information to obtain the latest attribute information of the target file, and associating the latest attribute information of the target file with the first auto-increment sequence number; Based on the first auto-increment sequence number corresponding to the target file recorded in the target logical storage unit, invalid attribute information of the target file existing in the write-ahead log is detected, and the detected invalid attribute information is deleted; wherein, the second auto-increment sequence number corresponding to the invalid attribute information of the target file is smaller than the first auto-increment sequence number.

6. The method according to claim 5, characterized in that The logical storage unit is located in the memory of the data node. When the attribute information of the target file also includes extended attributes, the attribute information of the target file recorded in the target logical storage unit includes: the non-extended attributes of the target file, and the position of the extended attributes of the target file in the write-ahead log.

7. A method for processing file attributes in a distributed file system, characterized in that: include: Sending a request for obtaining identification information of a target file to a metadata node of the distributed file system through a client; receiving target identification information of the target file returned by the metadata node, and determining a target logical storage unit based on the target identification information; wherein the target logical storage unit is a logical storage unit for storing attribute information of the target file; Determine the target data node to which the target logical storage unit belongs based on the acquired target mapping relationship; wherein the target mapping relationship is a mapping relationship between a logical storage unit and a data node; The attribute information of the target file is obtained through the target data node.

8. The method according to claim 7, characterized in that The method further comprises: The latest target mapping relationship is periodically obtained through the metadata node.

9. A file attribute processing device for a distributed file system, characterized in that: include: A request receiving module is used to receive a file attribute query request from a client through a data node in the distributed file system, wherein the file attribute query request carries target identification information of a target file and is used to request to query attribute information of the target file; The target identification information is obtained from a metadata node in the distributed file system; a storage unit determination module, configured to determine, based on the target identification information, a target logical storage unit for storing the attribute information of the target file from at least one logical storage unit included in the data node; The data node dynamically maintains attribute information of each file based on at least one logical storage unit; An attribute acquisition module, configured to acquire attribute information of the target file that matches the target identification information from the target logical storage unit; The request response module is used to respond to the file attribute query request based on the attribute information of the target file.

10. A file attribute processing device for a distributed file system, characterized in that: include: A request sending module, configured to send a request for obtaining identification information of a target file to a metadata node of the distributed file system through a client; a unit determination module, configured to receive the target identification information of the target file returned by the metadata node, and determine a target logical storage unit based on the target identification information; wherein the target logical storage unit is a logical storage unit for storing attribute information of the target file; a node determination module, configured to determine, based on the acquired target mapping relationship, a target data node to which the target logical storage unit belongs; wherein the target mapping relationship is a mapping relationship between a logical storage unit and a data node, and a data node includes at least one logical storage unit; The information acquisition module is used to acquire the attribute information of the target file through the target data node.

11. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the file attribute processing method of the distributed system according to any one of claims 1 to 6 or the file attribute processing method of the distributed file system according to claim 7 or 8.

12. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the file attribute processing method of a distributed system according to any one of claims 1 to 6 or the file attribute processing method of a distributed file system according to claim 7 or 8.

13. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the file attribute processing method of a distributed system according to any one of claims 1 to 6 or the file attribute processing method of a distributed file system according to claim 7 or 8.

Images