Method, device and equipment for acquiring attribute information of directory of distributed file system

Abstract

The invention provides a method, a device and equipment for acquiring attribute information of a directory of a distributed file system, and relates to the technical field of computers, in particular to the field of big data and distributed file systems. According to the specific implementation scheme, when an obtaining instruction is obtained, whether a log index of a directory indicated by the obtaining instruction is smaller than a transaction index of the directory or not and whether a queue index of the directory is larger than or equal to the transaction index or not are judged; if all the conditions are met, reading modification information of attributes of the directory from the memory queue; and determining the attribute information of the current directory according to the modification information of the attribute and the attribute file of the directory.

Description

Technical Field

[0001] The present disclosure relates to big data in computer technology and a distributed file system, and in particular to a method, device and equipment for acquiring attribute information of a directory of a distributed file system. Background Art

[0002] With the widespread application of distributed file systems, more and more users store data in distributed file systems, realizing online data sharing. There may be a large number of concurrent operations for shared data. The concurrent operations may include processing requests for the data, obtaining requests for attribute information of the data, etc.

[0003] Therefore, there is an urgent need for a method to quickly and accurately obtain attribute information. Summary of the invention

[0004] The present disclosure provides a method, apparatus and device for acquiring attribute information of a directory of a distributed file system.

[0005] According to a first aspect of the present disclosure, a method for obtaining attribute information of a directory of a distributed file system is provided, comprising:

[0006] In response to an acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, then determining the queue index in the memory queue corresponding to the directory; wherein the acquisition instruction indicates to acquire the attribute information of the directory;

[0007] When the queue index is greater than or equal to the transaction index, reading modification information of the attribute of the directory from the memory queue; and determining the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory;

[0008] The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

[0009] According to a second aspect of the present disclosure, a device for acquiring attribute information of a directory of a distributed file system is provided, comprising:

[0010] A first judgment unit, configured to respond to an acquisition instruction and, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, determine the queue index in the memory queue corresponding to the directory; wherein the acquisition instruction indicates to acquire the attribute information of the directory;

[0011] A second judgment unit is used to read the modification information of the attribute of the directory from the memory queue when the queue index is greater than or equal to the transaction index; and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory;

[0012] The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0022] Figure 4is a schematic diagram according to a fourth embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0024] Figure 6 is a schematic block diagram of an electronic device for implementing an embodiment of the present disclosure;

[0025] Figure 7 It is a scene graph of a distributed file system that can implement the embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] With the widespread application of distributed file systems, more and more users store data in distributed file systems, realizing online data sharing. In order to improve the operational efficiency of users accessing distributed file systems and increase the path resolution speed, index units have been introduced in distributed file systems to speed up path resolution.

[0028] However, with the addition of index units, operations such as creating and deleting directories have changed from only updating the metadata unit to simultaneously completing updates in the metadata unit and the index unit. This operation becomes a cross-unit two-phase commit (2PC) transaction. The complexity of its operation is therefore increased. During the execution of this 2PC transaction, the modification of the directory's attribute file in the metadata unit increases the possibility of transaction conflicts. For example, when multiple concurrent operations attempt to operate on the subdirectories of the parent directory under the same parent directory, the attribute file of the parent directory needs to be frequently accessed. In this case, the update of the attribute file of the parent directory may cause a large number of distributed transaction conflicts. In order to avoid these transaction conflicts, operations must be queued and executed serially. However, serial execution greatly limits the processing capacity of the distributed file system, resulting in the query rate per second (QPS) of operations under the same directory being only a few hundred, which has a low throughput problem.

[0029] To this end, the present application proposes a method for obtaining attribute information of a directory of a distributed file system. The method can improve the operation performance under the same directory and reduce transaction conflicts by writing the modification information of the attribute into a memory queue. The improvement of this performance will further improve the response efficiency of the user's operation request. Furthermore, for the user's attribute query request, the attribute information can be obtained quickly and accurately.

[0030] The present invention discloses a method, device and equipment for obtaining attribute information of a directory of a distributed file system, which are applied to big data and distributed file systems in computer technology to achieve the effect of quickly and accurately obtaining attribute information of a directory in a distributed file system.

[0031] It should be noted that the data in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0032] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2 and Figure 3 right Figure 1 The illustrated embodiment is further refined.

[0034] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, such as Figure 1 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0035] 101. In response to a get instruction, if it is determined that the log index of the directory indicated by the get instruction is less than the transaction index of the directory, then determine the queue index in the memory queue corresponding to the directory. The get instruction indicates to get the attribute information of the directory. The log index indicates the number of updates for the directory, and the transaction index indicates the number of operations on the directory. The memory queue stores the modification information of the attributes of the directory, and the queue index indicates the number of operations on the memory queue.

[0036] In this embodiment, after obtaining an acquisition instruction for querying the attribute information of a directory in a distributed file system, a query operation of the attribute information of the directory can be performed on a device node in the distributed file system in response to the acquisition instruction. In order to avoid transaction conflicts when the distributed file system processes concurrent operations on subdirectories under a directory, a memory queue is set in the device node for writing updated information on the attributes of the directory. Therefore, in the process of reading the attribute information of the directory, it is necessary to judge the read position. Among them, the reading method can include reading directly from the attribute file of the directory, and reading from the attribute file and the memory queue together. In the method that requires reading from the attribute file and the memory queue together, if the attribute information requested by the user is a non-numeric attribute, the latest non-numeric attribute can be directly obtained from the memory queue. If the attribute information requested by the user is a numeric attribute, since the queue records the modification information of the attribute, it is necessary to modify it based on the attribute value in the attribute file using the modification information in the memory queue to obtain the final attribute value.

[0037] When the user requests to obtain the attribute information of the directory, the log index and the transaction index of the directory may be obtained first.

[0038] The log index of the directory indicates the number of updates made to the directory. Optionally, the log index indicates the number of operations that have been updated to the log file of the directory. Optionally, the log index may be the maximum log entry that has been applied. In a distributed file system, the log entry that has been applied is the entry of the log that has been executed and written. That is, the 2PC transaction has been committed and the number of operations that have been executed has been completed.

[0039] Among them, the transaction index indicates the number of operations on the directory. The operation on the directory can specifically be an operation on a subdirectory of the directory. Optionally, the transaction index can be the maximum log entry that has been submitted. Compared with the log index, the transaction index may contain logs of operations that have been submitted but not yet applied. For example, the log of the operation that has been submitted but not yet applied can be: the operation has completed the update on the index unit and realized the submission of the transaction, but has not yet completed the update of the directory on the metadata unit. Therefore, the operation has not yet completed the submission of 2PC and has not been applied.

[0040] By comparing the log index of the directory and the transaction index of the directory, it can be determined whether the operations of the directory have completed the update of the corresponding directory.

[0041] If the log index is greater than or equal to the transaction index, it means that all operations on the directory have been committed. That is, all operations on the directory have completed the update of the directory. Therefore, the attribute information can be directly obtained from the attribute file of the directory.

[0042] Otherwise, if the log index is smaller than the transaction index, it means that only some operations of the directory have been submitted. That is, there are still some operations on the directory that have not completed the update of the directory. In the present disclosure, the modification information of the directory that has not completed the update is written in the memory queue. Therefore, when the log index is smaller than the transaction index, the queue index of the memory queue corresponding to the directory can be obtained. Optionally, the memory queue can store modification information of the attributes of the directory. Optionally. The queue index indicates the number of operations on the memory queue.

[0043] Optionally, the queue index can be obtained by obtaining the index of the modification information of the tail of the memory queue corresponding to the directory. Since the writing order of the memory queue is to write at the tail, the index of the tail is the queue index indicating the number of operations of the memory queue. This method can realize the rapid acquisition of the queue index and improve the efficiency of obtaining the queue index.

[0044] 102. When the queue index is greater than or equal to the transaction index, read the modification information of the attribute of the directory from the memory queue; and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory.

[0045] In this embodiment, since the process of the directory may have read the modification information from the queue, but the process is still processing the modification information and has not updated the property file of the directory. Therefore, after obtaining the queue index, it is necessary to further compare the queue index of the directory with the transaction index of the directory. If the queue index is greater than or equal to the transaction index, it means that there is no modification information stuck in the process. At this time, the modification information can be directly read from the queue, and the read modification information can be combined with the information in the property file of the directory to obtain the final property information of the directory. Otherwise, if the queue index is less than the transaction index, it means that there is a modification information stuck in the process. At this time, it is necessary to enter a waiting state and wait for the modification information in the process to be written into the property file of the directory.

[0046] Optionally, when the queue index is greater than or equal to the transaction index, the attribute information of the directory needs to be determined according to the attribute modification information and the attribute file of the directory, and the specific process may include:

[0047] 1021. Merge the modified information of the attributes to obtain summary information.

[0048] In this embodiment, since the modification information is usually written in a high concurrency situation, the number of the modification information is usually multiple. Therefore, after obtaining the modification information of the directory, the modification information can be merged to obtain summary information. When modifying the attributes of the summary information directory, the modification efficiency can be improved.

[0049] 1022. Merge the summary information with the attribute information in the attribute file of the directory to obtain final attribute information of the directory.

[0050] In this embodiment, the attribute information can be obtained from the attribute file of the directory. Then, the attribute information can be updated using the summary information to obtain the final attribute information, which is the final attribute information of the directory.

[0051] Through the above steps 1021 and 1022, the attribute information of the directory is obtained, and the attributes of the directory are quickly read and written while avoiding transaction conflicts in directory operations, thereby improving user experience.

[0052] In this embodiment, the reading of the attribute information of the directory is realized by comparing the log index, transaction index and queue index of the directory. The reading of the attribute information realizes accurate and fast reading of the attribute information when there is a conflict in the operation stored in the directory, improves the reading efficiency and accuracy of the attribute information, and improves the user experience.

[0053] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure, such as Figure 2 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0054] 201. In response to an acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, a queue index in a memory queue corresponding to the directory is determined. The acquisition instruction indicates acquisition of attribute information of the directory.

[0055] In this embodiment, the execution of step 201 is similar to the above step 101 and will not be repeated here.

[0056] 202. If it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory, attribute information is acquired from an attribute file of the directory.

[0057] In this embodiment, if it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory, it means that the operations of the directory have been applied. That is, the operations of the directory have completed the update of the directory. At this time, the attribute information can be directly obtained from the attribute file of the directory. By comparing the log index and the transaction index, it is possible to determine that the attribute information stored in the attribute file of the directory is the latest, and when it is determined that the information in the attribute file is the latest attribute information, the attribute information is directly obtained from the attribute file, which further improves the accurate and fast reading of the attribute information, improves the reading efficiency and accuracy of the attribute information, and improves the user experience.

[0058] 203. When the queue index is greater than or equal to the transaction index, read the modification information of the attribute of the directory from the memory queue; and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory.

[0059] In this embodiment, the execution of step 203 is similar to the above step 102, and will not be repeated here.

[0060] 204. When the queue index is less than the transaction index, the waiting state is entered. The waiting state indicates that the modification information in the memory queue has been read by the process but has not been written to the attribute file.

[0061] In this embodiment, when the queue index is greater than or equal to the transaction index, it indicates that the modification information has been read by the process but has not been updated to the property file of the directory. In order to avoid the problem of inaccurate property information read by the user caused by this situation, it can be achieved by further comparing the queue index and the transaction index. When the queue index is less than the transaction index, it indicates that the modification information has been read by the process but has not been written to the property file. At this time, since it is impossible to directly process the modification information that has been read by the process, it is necessary to wait for the process to complete the processing of the modification information. That is, enter the waiting state.

[0062] 205. In the waiting state, periodically determine whether the attribute information acquisition condition is satisfied. The attribute information acquisition condition represents the condition for exiting the waiting state and achieving the attribute information acquisition of the directory.

[0063] In this embodiment, in the waiting state, it can be periodically determined whether the attribute information acquisition condition is met. If the attribute information acquisition condition is met, the waiting state can be exited and the attribute information of the directory can be acquired. Otherwise, if the attribute information acquisition condition is not met, it is necessary to continue in the waiting state and wait for the attribute information acquisition condition to be met. Optionally, the period can be determined according to the read and write efficiency of the process.

[0064] Optionally, the attribute information acquisition condition represents a condition for realizing the attribute information acquisition of the directory. Optionally, the attribute information acquisition condition of the directory may include the conditions shown in step 202 and step 203. That is, the log index of the directory is greater than or equal to the transaction index of the directory, or the log index of the directory is less than the transaction index of the directory, and the queue index of the directory is greater than or equal to the transaction index of the directory.

[0065] Optionally, when the log index of a directory is greater than or equal to the transaction index of the directory, it means that the modification information of all operations of the directory has been written to the property file of the directory, and the update of the properties of the directory has been completed. Optionally, when the log index of a directory is less than the transaction index of the directory, and the queue index of the directory is greater than or equal to the transaction index of the directory, it means that the modification information in the process has been written, but the modification information of the directory still exists in the memory queue.

[0066] 206. If the attribute information acquisition condition is met, output the attribute information according to the satisfied attribute information acquisition condition.

[0067] In this embodiment, if the attribute information acquisition condition is met, the attribute information acquisition method can be determined according to the satisfied attribute information acquisition condition. Different attribute information acquisition conditions can correspond to different attribute information acquisition methods.

[0068] Optionally, if the log index of the directory is greater than or equal to the transaction index of the directory, the attribute information is obtained from the attribute file of the directory. If the log index of the directory is greater than or equal to the transaction index of the directory, it means that all modification information of the directory in the memory queue has been updated to the attribute file of the directory. Therefore, the attribute information can be directly obtained from the attribute file of the directory.

[0069] Optionally, if the log index of the directory is less than the transaction index of the directory, and the queue index of the directory is greater than or equal to the transaction index of the directory, the attribute information is obtained from the memory queue of the directory and the attribute file of the directory. If the log index of the directory is less than the transaction index of the directory, and the queue index of the directory is greater than or equal to the transaction index of the directory, it means that the modification information in the process has been updated to the attribute file of the directory. However, after the process reads the modification information of the directory from the memory queue, the modification information of the directory is written into the memory queue. At this time, the attribute information of the directory needs to be read from both the memory queue and the attribute file of the directory.

[0070] By setting the exit condition of the above waiting state, it can be further ensured that the attribute information of the directory is obtained in the waiting state, thereby further improving the reading efficiency and reading accuracy of the attribute information and improving the user experience.

[0071] By setting the waiting state in the above steps 204-206, the problem of inaccurate attribute information of the directory obtained by the user after the modification information is read by the process can be avoided, further improving the reading efficiency and accuracy of the attribute information and improving the user experience.

[0072] 207. If the duration of the waiting state is longer than a preset duration threshold, an acquisition abnormality reminder is generated. The acquisition abnormality reminder indicates that the acquisition of the attribute information of the directory indicated by the acquisition instruction fails.

[0073] In this embodiment, after entering the waiting state, timing can also be performed to determine the duration of the waiting state. If the duration is greater than a preset duration threshold, the acquisition request can be terminated, and an acquisition exception reminder can be generated and returned to the user. The acquisition exception reminder is used to inform the user that the attribute information of the directory has failed to be read. By recording the duration of the waiting state and returning a failure reminder when the duration of the waiting state is greater than a preset duration threshold, the problem of poor user experience caused by the user's acquisition request having no feedback for a long time can be avoided.

[0074] exist Figure 1 The first embodiment shown and Figure 2 On the basis of the second embodiment shown, when the attribute information of the directory is written in the ATTR file of the directory, an implementation method of the attribute information of the directory may include the following steps. When reading the attribute information of the directory, it is necessary to confirm whether the attribute ATTR of the directory is the latest written value. If the committed value that matches the most recent modification can be read, it means that the read committed isolation level is implemented.

[0075] In a distributed file system built on Raft, the Raft log is always serially ordered. It is similar to the redolog mechanism. Therefore, the present application can use the Raft log to ensure that the read operation can meet the read committed isolation level. Currently, two indexes, last_committed_index and known_applied_index, are provided in Raft. Among them, last_committed_index is used to record the largest log entry that has been submitted, but may not have been applied yet. known_applied_index is used to record the largest log entry that has been applied, which indicates that the execution and writing have been completed in the distributed transaction database.

[0076] When requesting to read the attribute ATTR of a directory, the relationship between the known_applied_index and the last_committed_index corresponding to ATTR can be checked to determine whether the attribute ATTR is the latest written value. The known_applied_index corresponding to ATTR can be recorded as the log index of the directory, and the last_committed_index can be recorded as the transaction index of the directory.

[0077] If the log index is greater than or equal to the transaction index, it indicates that the latest change of ATTR has been committed and applied, so the attribute can be safely read from ATTR. If the log index is less than the transaction index, the memory queue needs to be involved. Correspondingly, the maximum index at the end of the memory queue needs to be obtained as the queue index. If the queue index is greater than or equal to the transaction index, it means that the change of ATTR has not been fully applied, but the memory queue contains enough new data. Therefore, by reading the latest data from this queue, the attributes of the directory can be obtained. Optionally, for some numerical data, after reading the latest modification information in the memory queue, it is also necessary to obtain the data from ATTR, and obtain the final attribute information based on the data combined with the modification information. If the queue index is less than the transaction index, it means that the change of ATTR has not been applied, and the memory queue does not contain enough new data. At this time, you must wait until the log index of ATTR is greater than or equal to the transaction index, or the queue index exceeds the transaction index. If these conditions are not met within the set time threshold, the request to obtain the attribute information will return failure.

[0078] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure, Figure 3 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0079] 301. In response to an operation instruction on a directory, when a queue write condition is met, determine modification information of the directory to be written into the memory queue. The operation instruction indicates an operation on the directory. The queue write condition indicates a condition under which modification information needs to be generated and written into the queue according to the operation instruction on the directory. The modification information indicates modification of attribute information of the directory.

[0080] In this embodiment, when an operation instruction for a directory is obtained, it can be determined whether a queue write condition is met. The queue write condition indicates the condition under which modification information needs to be generated and written into the queue according to the operation instruction for the directory. That is, when the operation instruction satisfies the queue write condition, modification information of the directory to be written into the memory queue can be generated according to the operation instruction. Otherwise, if the operation instruction satisfies the queue write condition, a conventional method can be used to perform an update of the attributes of the directory under the operation. Optionally, the conventional method performs an update of the attributes of the directory under the operation, that is, directly updates the attribute file of the directory according to the operation.

[0081] Optionally, the queue write condition may include a conflict in the read and write events of the operation files of the directory. Alternatively, the queue write condition may include that the number of concurrent operations of the subdirectories under the directory is greater than a quantity threshold. For the above two situations, when the queue write condition is met, it means that the number of operations of the directory is large, and there are many conflicts when the attributes of the directory are updated according to the conventional method. Therefore, in order to avoid conflicts generated during the attribute update process of the directory, the operation instruction can be generated into modification information and written into the queue.

[0082] 302. Write the modification information into the memory queue, and update the queue index of the memory queue.

[0083] In this embodiment, after the modification information is generated according to the operation instruction, the modification information can be written into the memory queue. When writing into the memory queue, the writing of the modification information can be realized according to the time sequence. After the modification information is written into the memory queue, since the index of the tail of the memory queue changes, the queue index can be updated accordingly.

[0084] Through the execution of the above steps 301 and 302, it is possible to determine whether the operation information needs to be written into the memory queue, thereby improving the flexibility of using the memory queue. When the directory corresponds to high concurrency, the modification information corresponding to the operation instruction is written into the memory queue, thereby avoiding transaction conflicts caused by directly modifying the attributes of the directory, and improving the throughput of the distributed file system.

[0085] 303. When it is determined that the process of the directory is in an idle state, the process of the directory is called, the modification information of the directory in the memory queue is read, and the modified information read is updated to the directory.

[0086] In this embodiment, each directory may correspond to a process. When the process of the directory is in an idle state, if the process detects that data is written in the queue corresponding to the directory, the process may read the modification information of the directory in the memory queue. After the process reads the modification information, it may merge the modification information to obtain summary information, and then use the summary information to update the attribute information of the directory. By the idle process reading the modification information in the memory queue and updating the operation to the directory, asynchronous writing of data in the memory queue is realized, thereby avoiding transaction conflicts caused by directly modifying the attributes of the directory during the execution of the operation, and improving the throughput of the distributed file system.

[0087] Optionally, the process of updating the attribute information of the directory may include:

[0088] 3031. Read-write lock on the directory's attribute file.

[0089] In this embodiment, first, when a process requests a property file of a directory, it can set a read-write lock on the property file. The setting of the read-write lock can avoid conflicts when the process reads and writes property information in the property file.

[0090] 3032. Read the directory attribute information from the directory attribute file.

[0091] In this embodiment, the process can read the attribute information from the attribute file of the directory. The attribute information is the attribute information that has not been modified.

[0092] 3033. Modify the attribute information according to the modification information to obtain the modified attribute information.

[0093] In this embodiment, the attribute information read in step 3032 may be modified. The modification process may be performed based on the summary information of the modified information. For example, when the summary information includes modification information of a numerical attribute in the attribute information, the modification information may be superimposed on the numerical attribute. For another example, when the summary information includes modification information of a non-numerical attribute in the attribute information, the modification information may be used to replace the numerical attribute.

[0094] 3034. Write the modified attribute information into the attribute file of the directory.

[0095] In this embodiment, the modified attribute information can be rewritten into the attribute file to implement asynchronous modification of the attributes of the directory.

[0096] 3035. Release the read-write lock of the directory's attribute file.

[0097] In this embodiment, after the writing of the attribute information is completed, the read-write lock of the attribute file of the directory can be released, so that other processes can continue to read the attribute file of the directory.

[0098] In this embodiment, through the execution of the above steps 30311-3035, an asynchronous update of the property file is realized. The update information can be written through the asynchronous update, ensuring the data consistency of the metadata unit and the index unit in the distributed file system, and improving the throughput of the distributed file system.

[0099] exist Figure 3 On the basis of the third embodiment shown, when the attribute information of the directory is written in the ATTR file of the directory, taking the mkdir operation as an example, the process of realizing the asynchronous update of the ATTR through the memory queue may include the following steps. The execution of the following steps realizes the separation of ATTR from the participants of the 2PC transaction. The 2PC transaction initiated by mkdir can first create a directory. When the 2PC transaction initiated by mkdir is committed, the modification information of the directory's ATTR can be written to the memory queue. Based on the memory queue, the modification information of the ATTR can be taken out in batches from the queue, and after the modification information is merged, it is asynchronously written to the ATTR to realize the update of the ATTR.

[0100] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure, Figure 4 As shown, the present disclosure provides a device 400 for updating the attributes of a directory of a distributed file system, the method comprising:

[0101] The first judgment unit 401 is used to determine the queue index in the memory queue corresponding to the directory in response to the acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory. The acquisition instruction indicates to acquire the attribute information of the directory.

[0102] The second judgment unit 402 is used to read the modification information of the directory attribute from the memory queue when the queue index is greater than or equal to the transaction index, and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory.

[0103] The log index indicates the number of updates to the directory, the transaction index indicates the number of operations on the directory, the memory queue stores modification information of the directory's attributes, and the queue index indicates the number of operations on the memory queue.

[0104] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0105] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 5 As shown, the present disclosure provides a device 500 for updating the attributes of a directory of a distributed file system, the method comprising:

[0106] The first judgment unit 501 is used to determine the queue index in the memory queue corresponding to the directory in response to the acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory. The acquisition instruction indicates to acquire the attribute information of the directory.

[0107] The second judgment unit 502 is used to read the modification information of the directory attribute from the memory queue when the queue index is greater than or equal to the transaction index, and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory.

[0108] The log index indicates the number of updates to the directory, the transaction index indicates the number of operations on the directory, the memory queue stores modification information of the directory's attributes, and the queue index indicates the number of operations on the memory queue.

[0109] Optionally, the first judging unit 501 includes:

[0110] The acquisition module 5011 is used to obtain the index of the modification information at the tail of the memory queue corresponding to the directory, and use the index as the queue index.

[0111] Optionally, the second judging unit 502 includes:

[0112] The first merging module 5021 is used to merge the modified information of the attributes to obtain summary information.

[0113] The second merging module 5022 is used to merge the summary information with the attribute information in the attribute file of the directory to obtain the final attribute information of the directory.

[0114] Optionally, the attribute updating device 500 further includes:

[0115] The third judgment unit 503 is configured to obtain attribute information from an attribute file of the directory if it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory.

[0116] Optionally, the attribute updating device 500 further includes a fourth judgment unit 504; the fourth judgment unit 504 includes:

[0117] The first judgment module 5041 is used to enter a waiting state when the queue index is smaller than the transaction index. The waiting state indicates that the modification information in the memory queue has been read by the process but has not been written into the attribute file.

[0118] The second judgment module 5042 is used to periodically judge whether the attribute information acquisition condition is met in the waiting state. The attribute information acquisition condition represents the condition for exiting the waiting state and achieving the attribute information acquisition of the directory.

[0119] The third judgment module 5043 is used to output the attribute information according to the satisfied attribute information acquisition condition if the attribute information acquisition condition is met.

[0120] Optionally, the third determination module 5043 includes:

[0121] The first judgment submodule 50431 is used to obtain attribute information from the attribute file of the directory if the log index of the directory is greater than or equal to the transaction index of the directory.

[0122] The second judgment submodule 50432 is used to obtain attribute information from the directory's memory queue and the directory's attribute file if the directory's log index is smaller than the directory's transaction index and the directory's queue index is greater than or equal to the directory's transaction index.

[0123] Optionally, the fourth determining unit 504 further includes:

[0124] The fourth judgment module 5044 is configured to generate an acquisition abnormality reminder if the duration of the waiting state is longer than a preset duration threshold, wherein the acquisition abnormality reminder indicates that the acquisition of the attribute information of the directory indicated by the acquisition instruction fails.

[0125] Optionally, the attribute updating device 500 further includes: an updating unit 505 including:

[0126] The determination module 5051 is used to determine the modification information of the directory to be written into the memory queue in response to the operation instruction of the directory, when the queue writing condition is met. The operation instruction indicates to operate the directory. The queue writing condition indicates the condition that the modification information needs to be generated according to the operation instruction of the directory and written into the queue. The modification information indicates the modification of the attribute information of the directory.

[0127] The writing module 5052 is used to write the modification information into the memory queue and update the queue index of the memory queue.

[0128] Optionally, the updating unit 505 further includes:

[0129] The update module 5053 is used to call the process of the directory when it is determined that the process of the directory is in an idle state, read the modification information of the directory in the memory queue, and update the read attribute modification to the directory.

[0130] Optionally, the update module 5053 includes:

[0131] The locking submodule 50531 is used to read and write locks on the attribute files of the directory.

[0132] The reading submodule 50532 is used to read the attribute information of the directory from the attribute file of the directory.

[0133] The modification submodule 50533 is used to modify the attribute information according to the modification information to obtain the modified attribute information.

[0134] The writing submodule 50534 is used to write the modified attribute information into the attribute file of the directory.

[0135] The unlocking submodule 5053 is used to release the read-write lock of the attribute file of the directory.

[0136] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0138] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0139] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0140] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0141] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0142] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a method for obtaining attribute information of a directory of a distributed file system. For example, in some embodiments, the method for obtaining attribute information of a directory of a distributed file system may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for obtaining attribute information of a directory of a distributed file system described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for obtaining the attribute information of a directory of the distributed file system in any other appropriate manner (for example, by means of firmware).

[0143] Figure 7 is a scene graph of a distributed file system that can implement the embodiments of the present disclosure, such as Figure 7As shown, the distributed file system can obtain external operation requests. The operation request can create directories B, F, etc. Files BB, FF, etc. can also be inserted. Each operation request can correspond to an attribute update. Based on the operation request, the metadata can be inserted into the slice of the metadata unit. The metadata can be a directory or file included in the operation request. The operation request can also write the modification information of the attribute file of the parent directory in the metadata unit into the memory queue. The modification information can be obtained from the memory queue through process W, and the attribute update of the parent directory can be completed. And, based on the operation request, index information can also be inserted into the index unit.

[0144] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0149] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0150] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for obtaining attribute information of a directory of a distributed file system, comprising: In response to an acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, then determining the queue index in the memory queue corresponding to the directory; wherein the acquisition instruction indicates to acquire the attribute information of the directory; When the queue index is greater than or equal to the transaction index, reading modification information of the attribute of the directory from the memory queue; and determining the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory; The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

2. The method according to claim 1, wherein: The determining the queue index in the memory queue corresponding to the directory includes: The index of the modification information at the tail of the memory queue corresponding to the directory is obtained, and the index is used as the queue index.

3. The method according to claim 1, wherein: Determining the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory includes: Merging the modification information of the attributes to obtain summary information; The summary information is combined with the attribute information in the attribute file of the directory to obtain final attribute information of the directory.

4. The method according to any one of claims 1 to 3, further comprising: If it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory, the attribute information is acquired from the attribute file of the directory.

5. The method according to any one of claims 1 to 4, further comprising: When the queue index is less than the transaction index, entering a waiting state; The waiting state indicates that the modification information in the memory queue has been read by the process but has not been written into the attribute file; In the waiting state, periodically judging whether the attribute information acquisition condition is satisfied; The attribute information acquisition condition represents a condition for exiting the waiting state and achieving the attribute information acquisition of the directory; If the attribute information acquisition condition is satisfied, the attribute information is output according to the satisfied attribute information acquisition condition.

6. The method according to claim 5, wherein: If the attribute information acquisition condition is satisfied, outputting the attribute information according to the satisfied attribute information acquisition condition includes: If the log index of the directory is greater than or equal to the transaction index of the directory, obtaining the attribute information from the attribute file of the directory; If the log index of the directory is smaller than the transaction index of the directory, and the queue index of the directory is greater than or equal to the transaction index of the directory, the attribute information is obtained from the memory queue of the directory and the attribute file of the directory.

7. The method according to claim 5 or 6, wherein: The method further comprises: If the duration of the waiting state is longer than a preset duration threshold, an acquisition abnormality reminder is generated, where the acquisition abnormality reminder indicates that the acquisition of the attribute information of the directory indicated by the acquisition instruction fails.

8. The method according to any one of claims 1 to 7, further comprising: In response to an operation instruction on the directory, when a queue write condition is met, determining modification information of the directory to be written into the memory queue; wherein the operation instruction indicates an operation on the directory; the queue write condition indicates a condition under which modification information needs to be generated and written into the queue according to the operation instruction on the directory; and the modification information indicates modification of attribute information of the directory; The modification information is written into the memory queue, and the queue index of the memory queue is updated.

9. The method according to any one of claims 1 to 8, further comprising: When it is determined that the process of the directory is in an idle state, the process of the directory is called, the modification information of the directory in the memory queue is read, and the read modification information is updated to the directory.

10. The method according to claim 9, wherein: The read modification information is updated to the directory, including: A read-write lock is set on the attribute file of the directory; Reading attribute information of the directory from the attribute file of the directory; Modify the attribute information according to the modification information to obtain modified attribute information; Writing the modified attribute information into the attribute file of the directory; Release the read-write lock on the properties file of the directory.

11. A device for acquiring attribute information of a directory of a distributed file system, comprising: A first judgment unit, configured to respond to an acquisition instruction and, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, determine the queue index in the memory queue corresponding to the directory; wherein the acquisition instruction indicates to acquire the attribute information of the directory; A second judgment unit is used to read the modification information of the attribute of the directory from the memory queue when the queue index is greater than or equal to the transaction index; and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory; The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

14. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

Images