Method, device and equipment for obtaining attribute information of directory of distributed file system

Abstract

The present disclosure provides a method, device and equipment for obtaining attribute information of a directory of a distributed file system, relates to the technical field of computers, in particular to big data and the field of distributed file systems. The specific implementation scheme is as follows: when an obtaining instruction is obtained, it is determined whether a log index of a directory indicated by the obtaining instruction is less than a transaction index of the directory, and whether a queue index of the directory is greater than or equal to the transaction index. If the above conditions are all met, the modification information of the attribute of the directory is read from the memory queue; and the attribute information of the current directory is determined according to the modification information of the attribute and the attribute file of the directory.

Description

Technical Field

[0001] The present disclosure relates to big data and distributed file systems in computer technology, and in particular to a method, apparatus, and device for obtaining attribute information of a directory of a distributed file system. Background Art

[0002] With the widespread adoption of distributed file systems, more and more users are storing data there, enabling online data sharing. Shared data can be subject to a large number of concurrent operations. These concurrent operations can include requests to process the data and obtain its attribute information.

[0003] Therefore, there is an urgent need for a method to quickly and accurately obtain attribute information. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, and device for acquiring attribute information of a directory in a distributed file system.

[0005] According to a first aspect of the present disclosure, a method for obtaining attribute information of a directory in a distributed file system is provided, comprising:

[0006] In response to an acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, determining the queue index in the memory queue corresponding to the directory; wherein the acquisition instruction indicates acquisition of attribute information of the directory;

[0007] When the queue index is greater than or equal to the transaction index, reading the modification information of the attribute of the directory from the memory queue; and determining the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory;

[0008] The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

[0009] According to a second aspect of the present disclosure, a device for obtaining attribute information of a directory of a distributed file system is provided, comprising:

[0010] a first determining unit configured to, in response to an acquisition instruction, determine a queue index in a memory queue corresponding to the directory if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory; wherein the acquisition instruction indicates acquisition of attribute information of the directory;

[0011] a second judgment unit, configured to read, when the queue index is greater than or equal to the transaction index, attribute modification information of the directory from the memory queue; and determine attribute information of the directory based on the attribute modification information and the attribute file of the directory;

[0012] The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0022] Figure 4is a schematic diagram according to a fourth embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0024] Figure 6 is a schematic block diagram of an electronic device for implementing an embodiment of the present disclosure;

[0025] Figure 7 It is a scene graph of a distributed file system that can implement the embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] With the widespread adoption of distributed file systems, more and more users are storing data in them, enabling online data sharing. To improve user efficiency when accessing distributed file systems and increase the speed of path resolution, indexing units have been introduced to accelerate path resolution.

[0028] However, with the addition of index units, operations such as creating and deleting directories no longer require updating only the metadata unit, but now require simultaneous updates to both the metadata unit and the index unit. This operation becomes a cross-unit two-phase commit (2PC) transaction, increasing its operational complexity. The execution of this 2PC transaction increases the likelihood of transaction conflicts when modifying the directory's property files within the metadata unit. For example, when multiple concurrent operations attempt to operate on subdirectories within the same parent directory, frequent access to the parent directory's property files is required. In this case, updates to the parent directory's property files can potentially lead to numerous distributed transaction conflicts. To avoid these conflicts, operations must be queued and executed serially. However, serial execution significantly limits the processing power of the distributed file system, resulting in a query-per-second (QPS) rate of only a few hundred for operations within the same directory, resulting in low throughput.

[0029] To this end, this application proposes a method for obtaining attribute information for directories in a distributed file system. This method improves the performance of operations within the same directory and reduces transaction conflicts by writing attribute modification information to a memory queue. This improved performance will further enhance the efficiency of responding to user operation requests. Furthermore, attribute information can be quickly and accurately obtained for user attribute query requests.

[0030] The present disclosure provides a method, apparatus, and device for obtaining attribute information of a directory in a distributed file system, which are applied to big data and distributed file systems in computer technology to achieve the effect of quickly and accurately obtaining attribute information of a directory in a distributed file system.

[0031] It should be noted that the data in this embodiment is not targeted at a specific user and does not reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0032] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] In order to make readers more deeply understand the implementation principle of this disclosure, the following Figure 2 and Figure 3 right Figure 1 The illustrated embodiment is further refined.

[0034] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 1 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0035] 101. In response to a get instruction, if it is determined that the log index of the directory indicated by the get instruction is less than the transaction index of the directory, then determine the queue index in the memory queue corresponding to the directory. The get instruction instructs to obtain attribute information of the directory. The log index indicates the number of updates performed on the directory, and the transaction index indicates the number of operations performed on the directory. The memory queue stores modification information of the directory's attributes, and the queue index indicates the number of operations performed on the memory queue.

[0036] In this embodiment, after receiving an instruction to query the attribute information of a directory in a distributed file system, a query operation for the directory's attribute information can be executed on a device node in the distributed file system in response to the instruction. To avoid transaction conflicts when the distributed file system processes concurrent operations on subdirectories within a directory, a memory queue is provided in the device node for writing updated information about the directory's attributes. Therefore, when reading the directory's attribute information, it is necessary to determine the reading location. The reading method can include reading directly from the directory's attribute file or reading from both the attribute file and the memory queue. In the method requiring reading from both the attribute file and the memory queue, if the attribute information requested by the user is non-numeric, the latest non-numeric attribute can be directly obtained from the memory queue. If the attribute information requested by the user is numeric, since the queue records attribute modification information, it is necessary to modify the attribute value in the attribute file using the modification information in the memory queue to obtain the final attribute value.

[0037] When a user requests to obtain the attribute information of the directory, the log index and the transaction index of the directory may be obtained first.

[0038] The log index of the directory indicates the number of updates performed on the directory. Optionally, the log index indicates the number of operations that have been updated in the log file of the directory. Optionally, the log index can be the maximum log entry that has been applied. In a distributed file system, the applied log entry is the log entry that has been executed and written. In other words, the number of operations that have been executed after the 2PC transaction has been committed.

[0039] The transaction index indicates the number of operations on the directory. The operation on the directory may specifically be an operation on a subdirectory of the directory. Optionally, the transaction index may be the maximum log entry that has been submitted. Compared to the log index, the transaction index may contain logs of operations that have been submitted but not yet applied. For example, the log of the operation that has been submitted but not yet applied may be: the operation has completed the update on the index unit and achieved the submission of the transaction, but has not yet completed the update of the directory on the metadata unit. Therefore, the operation has not yet completed the submission and application of 2PC.

[0040] By comparing the log index of the directory and the transaction index of the directory, it can be determined whether all operations on the directory have completed the update of the corresponding directory.

[0041] If the log index is greater than or equal to the transaction index, then all operations on the directory have been committed. In other words, all operations on the directory have completed the directory update. Therefore, the attribute information can be directly obtained from the directory's attribute file.

[0042] Otherwise, if the log index is smaller than the transaction index, it indicates that only some operations on the directory have been submitted. In other words, some operations on the directory have not yet completed the update of the directory. In the present disclosure, the modification information of the incomplete directory update is written to the memory queue. Therefore, when the log index is smaller than the transaction index, the queue index of the memory queue corresponding to the directory can be obtained. Optionally, the memory queue can store modification information of the directory's attributes. Optionally, the queue index indicates the number of operations on the memory queue.

[0043] Alternatively, the queue index can be obtained by obtaining the index of the modification information at the end of the memory queue corresponding to the directory. Since the memory queue is written to the end first, the index at the end of the queue is the queue index indicating the number of operations on the memory queue. This method can quickly obtain the queue index and improve the efficiency of obtaining the queue index.

[0044] 102. When the queue index is greater than or equal to the transaction index, read the modification information of the directory attribute from the memory queue; and determine the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory.

[0045] In this embodiment, it is possible that the process of the directory has read the modification information from the queue, but the process is still processing the modification information and has not updated the directory's property file. Therefore, after obtaining the queue index, it is necessary to further compare the queue index of the directory with the transaction index of the directory. If the queue index is greater than or equal to the transaction index, it means that there is no modification information stuck in the process. At this time, the modification information can be read directly from the queue, and the read modification information can be combined with the information in the directory's property file to obtain the final directory's property information. Otherwise, if the queue index is less than the transaction index, it means that there is modification information stuck in the process. At this time, it is necessary to enter a waiting state and wait for the modification information in the process to be written to the directory's property file.

[0046] Optionally, when the queue index is greater than or equal to the transaction index, the directory attribute information needs to be determined based on the attribute modification information and the directory attribute file. The specific process may include:

[0047] 1021. Merge the modified information of the attributes to obtain summary information.

[0048] In this embodiment, since the modification information is usually written in a high-concurrency situation, the number of modification information is usually multiple. Therefore, after obtaining the modification information of the directory, the modification information can be merged to obtain summary information. When modifying the attributes of the directory with this summary information, the modification efficiency can be improved.

[0049] 1022. Merge the summary information with the attribute information in the directory attribute file to obtain final directory attribute information.

[0050] In this embodiment, the attribute information can be obtained from the attribute file of the directory. Then, the attribute information can be updated using the summary information to obtain the final attribute information. The attribute information is the final attribute information of the directory.

[0051] The above steps 1021 and 1022 are used to obtain the attribute information of the directory, thereby achieving fast reading and writing of the directory attributes while avoiding transaction conflicts in directory operations, thereby improving user experience.

[0052] In this embodiment, the directory's attribute information is read by comparing its log index, transaction index, and queue index. This allows for accurate and rapid reading of attribute information even when conflicts arise between directory operations, improving both efficiency and accuracy, and enhancing the user experience.

[0053] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure, as shown in Figure 2 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0054] 201. In response to a get instruction, if it is determined that the log index of the directory indicated by the get instruction is less than the transaction index of the directory, determine the queue index in the memory queue corresponding to the directory. The get instruction instructs obtaining attribute information of the directory.

[0055] In this embodiment, the execution of step 201 is similar to the above step 101 and will not be repeated here.

[0056] 202. If it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory, obtain attribute information from the attribute file of the directory.

[0057] In this embodiment, if it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory, it means that all operations on the directory have been applied. In other words, all operations on the directory have completed the update of the directory. At this point, the attribute information can be directly obtained from the directory's attribute file. By comparing the log index and the transaction index, it is possible to determine whether the attribute information stored in the directory's attribute file is the latest. When it is determined that the information in the attribute file is the latest attribute information, the attribute information is directly obtained from the attribute file, further improving the accurate and fast reading of attribute information, improving the efficiency and accuracy of reading attribute information, and enhancing the user experience.

[0058] 203. When the queue index is greater than or equal to the transaction index, read the directory attribute modification information from the memory queue; and determine the directory attribute information according to the attribute modification information and the directory attribute file.

[0059] In this embodiment, the execution of step 203 is similar to the above step 102 and will not be repeated here.

[0060] 204. When the queue index is less than the transaction index, the process enters the waiting state. The waiting state indicates that the modification information in the memory queue has been read by the process but has not been written to the attribute file.

[0061] In this embodiment, when the queue index is greater than or equal to the transaction index, this indicates that the modification information has been read by the process but has not yet been updated to the directory's properties file. To avoid the problem of inaccurate property information read by the user due to this situation, the queue index and transaction index can be further compared. When the queue index is less than the transaction index, this indicates that the modification information has been read by the process but has not yet been written to the properties file. In this case, since the modification information that has been read by the process cannot be directly processed, it is necessary to wait for the process to complete processing the modification information. In other words, the process enters a waiting state.

[0062] 205. In the waiting state, periodically determine whether an attribute information acquisition condition is satisfied. The attribute information acquisition condition represents a condition for exiting the waiting state and achieving the attribute information acquisition of the directory.

[0063] In this embodiment, in this waiting state, it is possible to periodically determine whether the attribute information acquisition condition is met. If the attribute information acquisition condition is met, the waiting state can be exited and the attribute information of the directory can be acquired. Otherwise, if the attribute information acquisition condition is not met, it is necessary to continue in the waiting state and wait for the attribute information acquisition condition to be met. Optionally, this period can be determined based on the read and write efficiency of the process.

[0064] Optionally, the attribute information acquisition condition represents a condition for acquiring the directory's attribute information. Optionally, the directory's attribute information acquisition condition may include the conditions shown in steps 202 and 203. Specifically, the directory's log index is greater than or equal to the directory's transaction index, or the directory's log index is less than the directory's transaction index, and the directory's queue index is greater than or equal to the directory's transaction index.

[0065] Optionally, when the directory's log index is greater than or equal to the directory's transaction index, it indicates that the modification information for all operations on the directory has been written to the directory's attribute file, completing the update of the directory's attributes. Optionally, when the directory's log index is less than the directory's transaction index, and the directory's queue index is greater than or equal to the directory's transaction index, it indicates that the modification information in the process has been written, but the directory's modification information still exists in the memory queue.

[0066] 206. If the attribute information acquisition condition is met, output the attribute information according to the satisfied attribute information acquisition condition.

[0067] In this embodiment, if the attribute information acquisition condition is met, the attribute information acquisition method can be determined based on the satisfied attribute information acquisition condition. Different attribute information acquisition conditions can correspond to different attribute information acquisition methods.

[0068] Optionally, if the directory's log index is greater than or equal to the directory's transaction index, the attribute information is retrieved from the directory's attribute file. If the directory's log index is greater than or equal to the directory's transaction index, this indicates that all modification information for the directory in the memory queue has been updated to the directory's attribute file. Therefore, the attribute information can be retrieved directly from the directory's attribute file.

[0069] Optionally, if the directory's log index is less than the directory's transaction index, and the directory's queue index is greater than or equal to the directory's transaction index, the attribute information is retrieved from the directory's memory queue and the directory's attribute file. If the directory's log index is less than the directory's transaction index, and the directory's queue index is greater than or equal to the directory's transaction index, the modification information in the process has already been updated in the directory's attribute file. However, after the process read the directory's modification information from the memory queue, the memory queue was rewritten with the directory's modification information. In this case, the directory's attribute information needs to be read from both the memory queue and the directory's attribute file.

[0070] By setting the exit condition of the above waiting state, it can be further ensured that the attribute information of the directory is obtained in the waiting state, thereby further improving the reading efficiency and reading accuracy of the attribute information and improving the user experience.

[0071] By setting the waiting state in steps 204-206, the problem of inaccurate attribute information of the directory obtained by the user after the modification information is read by the process can be avoided, thereby further improving the reading efficiency and accuracy of the attribute information and enhancing the user experience.

[0072] 207. If the duration of the waiting state is longer than a preset duration threshold, an acquisition abnormality reminder is generated. The acquisition abnormality reminder indicates that the acquisition of the attribute information of the directory indicated by the acquisition instruction fails.

[0073] In this embodiment, after entering the waiting state, timing can also be performed to determine the duration of the waiting state. If the duration is greater than a preset duration threshold, the acquisition request can be terminated, and an acquisition exception reminder can be generated and returned to the user. The acquisition exception reminder is used to inform the user that the attribute information of the sub-directory has failed to be read. By recording the duration of the waiting state and returning a failure reminder when the duration of the waiting state is greater than the preset duration threshold, the problem of a poor user experience caused by a user's acquisition request not receiving feedback for a long time can be avoided.

[0074] exist Figure 1 The first embodiment shown and Figure 2 Based on the second embodiment shown, when the directory's attribute information is written to the directory's ATTR file, one implementation of the directory's attribute information may include the following steps. When reading the directory's attribute information, it is necessary to confirm whether the directory's attribute ATTR is the most recently written value. If a committed value that matches the most recently modified value can be read, then the read committed isolation level has been achieved.

[0075] In a distributed file system built on Raft, the Raft log is always serially ordered. This is similar to the redolog mechanism. Therefore, this application can use the Raft log to ensure that read operations can meet the read-committed isolation level. Currently, two indexes are provided in Raft: last_committed_index and known_applied_index. Among them, last_committed_index is used to record the largest log entry that has been committed, but may not have been applied yet. Known_applied_index is used to record the largest log entry that has been applied, which indicates that the execution and write have been completed in the distributed transaction database.

[0076] When requesting to read a directory attribute ATTR, you can check the relationship between the known_applied_index and the last_committed_index corresponding to the ATTR to determine whether the attribute ATTR is the most recently written value. The known_applied_index corresponding to the ATTR can be recorded as the directory's log index, and the last_committed_index can be recorded as the directory's transaction index.

[0077] If the log index is greater than or equal to the transaction index, the latest change to ATTR has been committed and applied, so the attribute can be safely read from ATTR. If the log index is less than the transaction index, the memory queue must be involved. The highest index at the end of the memory queue must be used as the queue index. If the queue index is greater than or equal to the transaction index, the change to ATTR has not yet been fully applied, but the memory queue contains sufficiently recent data. Therefore, by reading the latest data from this queue, the directory's attributes can be retrieved. Optionally, for some numeric data, after reading the latest modification information from the memory queue, the data must also be retrieved from ATTR and combined with the modification information to obtain the final attribute information. If the queue index is less than the transaction index, the change to ATTR has not yet been applied, and the memory queue does not contain sufficiently recent data. In this case, it is necessary to wait until the log index of ATTR is greater than or equal to the transaction index, or until the queue index exceeds the transaction index. If these conditions are not met within the set time threshold, the request to retrieve the attribute information will fail.

[0078] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 3 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0079] 301. In response to an operation instruction on a directory, when a queue write condition is satisfied, determine modification information of the directory to be written to a memory queue. The operation instruction indicates an operation to be performed on the directory. The queue write condition indicates the conditions under which modification information needs to be generated and written to the queue based on the directory operation instruction. The modification information indicates modifications to attribute information of the directory.

[0080] In this embodiment, when an operation instruction for a directory is obtained, it can be determined whether a queue write condition is met. The queue write condition indicates the condition under which modification information needs to be generated and written to the queue based on the operation instruction for the directory. That is, when the operation instruction meets the queue write condition, modification information for the directory to be written to the memory queue can be generated based on the operation instruction. Otherwise, if the operation instruction meets the queue write condition, a conventional method can be used to perform an update of the directory attributes under the operation. Optionally, the conventional method performs an update of the directory attributes under the operation, that is, directly updates the directory attribute file based on the operation.

[0081] Optionally, the queue write condition may include a conflict in read and write events for operating files within the directory. Alternatively, the queue write condition may include a number of concurrent operations within subdirectories within the directory exceeding a threshold. In either of the above situations, if the queue write condition is met, it indicates that the directory has undergone a large number of operations, and that there are many conflicts when updating the directory's attributes using conventional methods. Therefore, to avoid conflicts arising during the directory attribute update process, the operation instructions may be converted into modification information and written into the queue.

[0082] 302. Write the modification information into the memory queue and update the queue index of the memory queue.

[0083] In this embodiment, after the modification information is generated according to the operation instruction, the modification information can be written into the memory queue. When writing into the memory queue, the modification information can be written according to the time sequence. After the modification information is written into the memory queue, since the index at the end of the memory queue changes, the queue index can be updated accordingly.

[0084] By executing the above steps 301 and 302, it is possible to determine whether the operation information needs to be written into the memory queue, thereby improving the flexibility of using the memory queue. When the directory corresponds to high concurrency, the modification information corresponding to the operation instruction is written into the memory queue, thereby avoiding transaction conflicts caused by directly modifying the directory attributes and improving the throughput of the distributed file system.

[0085] 303. When it is determined that the process of the directory is in an idle state, the process of the directory is called, the modification information of the directory in the memory queue is read, and the read modification information is updated in the directory.

[0086] In this embodiment, each directory can correspond to a process. When the process of the directory is in an idle state, if the process detects that data is written to the queue corresponding to the directory, the process can read the modification information of the directory in the memory queue. After the process reads the modification information, it can merge the modification information to obtain summary information, and then use the summary information to update the attribute information of the directory. By the idle process reading the modification information in the memory queue and updating the operation to the directory, asynchronous writing of data in the memory queue is achieved, avoiding transaction conflicts caused by directly modifying the attributes of the directory during the execution of the operation, and improving the throughput of the distributed file system.

[0087] Optionally, the process of updating the attribute information of the directory may include:

[0088] 3031. Read-write lock on the directory's attribute file.

[0089] In this embodiment, first, when a process requests a property file of a directory, it can set a read-write lock on the property file. The setting of the read-write lock can avoid conflicts when the process reads and writes property information in the property file.

[0090] 3032. Read the directory attribute information from the directory attribute file.

[0091] In this embodiment, the process can read the attribute information from the attribute file of the directory. The attribute information is the unmodified attribute information.

[0092] 3033. Modify the attribute information according to the modification information to obtain the modified attribute information.

[0093] In this embodiment, the attribute information read in step 3032 can be modified. This modification process can be performed based on the summary information of the modified information. For example, when the summary information includes modification information for a numerical attribute in the attribute information, the modification information can be added to the numerical attribute. For another example, when the summary information includes modification information for a non-numerical attribute in the attribute information, the modification information can be used to replace the numerical attribute.

[0094] 3034. Write the modified attribute information into the attribute file of the directory.

[0095] In this embodiment, the modified attribute information can be rewritten into the attribute file to implement asynchronous modification of the attributes of the directory.

[0096] 3035. Release the read-write lock of the directory's attribute file.

[0097] In this embodiment, after the writing of the attribute information is completed, the read-write lock of the attribute file of the directory can be released, so that other processes can continue to read the attribute file of the directory.

[0098] In this embodiment, through the execution of the above steps 30311-3035, the asynchronous update of the property file is realized. The writing of the update information can be realized through the asynchronous update, which ensures the data consistency of the metadata unit and the index unit in the distributed file system and improves the throughput of the distributed file system.

[0099] exist Figure 3 On the basis of the third embodiment shown, when the attribute information of the directory is written in the ATTR file of the directory, taking the mkdir operation as an example, the process of implementing the asynchronous update of the ATTR through the memory queue may include the following steps. The execution of the following steps realizes the separation of ATTR from the participants of the 2PC transaction. The 2PC transaction initiated by mkdir can first create a directory. When the 2PC transaction initiated by mkdir is committed, the modification information of the directory's ATTR can be written to the memory queue. Based on the memory queue, the modification information of the ATTR can be taken out in batches from the queue, and after merging the modification information, it can be asynchronously written to the ATTR to implement the update of the ATTR.

[0100] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure, as shown in Figure 4 As shown, the present disclosure provides a device 400 for updating attributes of a directory of a distributed file system, the method comprising:

[0101] The first judgment unit 401 is configured to determine the queue index in the memory queue corresponding to the directory in response to the acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory.

[0102] The second judgment unit 402 is configured to read the directory attribute modification information from the memory queue when the queue index is greater than or equal to the transaction index, and determine the directory attribute information based on the attribute modification information and the directory attribute file.

[0103] The log index indicates the number of updates to the directory, the transaction index indicates the number of operations on the directory, the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

[0104] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0105] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure, as shown in Figure 5 As shown, the present disclosure provides a device 500 for updating the attributes of a directory of a distributed file system, the method comprising:

[0106] The first judgment unit 501 is configured to determine the queue index in the memory queue corresponding to the directory in response to a get instruction, if it is determined that the log index of the directory indicated by the get instruction is less than the transaction index of the directory.

[0107] The second judgment unit 502 is configured to read the directory attribute modification information from the memory queue when the queue index is greater than or equal to the transaction index, and determine the directory attribute information based on the attribute modification information and the directory attribute file.

[0108] The log index indicates the number of updates to the directory, the transaction index indicates the number of operations on the directory, the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

[0109] Optionally, the first judging unit 501 includes:

[0110] The acquisition module 5011 is used to obtain the index of the modification information at the tail of the memory queue corresponding to the directory, and use the index as the queue index.

[0111] Optionally, the second judgment unit 502 includes:

[0112] The first merging module 5021 is used to merge the modified information of the attributes to obtain summary information.

[0113] The second merging module 5022 is used to merge the summary information with the attribute information in the directory attribute file to obtain the final directory attribute information.

[0114] Optionally, the attribute updating apparatus 500 further includes:

[0115] The third judgment unit 503 is configured to obtain attribute information from an attribute file of the directory if it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory.

[0116] Optionally, the attribute updating apparatus 500 further includes a fourth judgment unit 504; the fourth judgment unit 504 includes:

[0117] The first judgment module 5041 is configured to enter a waiting state when the queue index is smaller than the transaction index. The waiting state indicates that the modification information in the memory queue has been read by the process but has not been written into the attribute file.

[0118] The second judgment module 5042 is used to periodically judge whether the attribute information acquisition condition is met in the waiting state. The attribute information acquisition condition represents the condition for exiting the waiting state and achieving the attribute information acquisition of the directory.

[0119] The third judgment module 5043 is configured to output the attribute information according to the satisfied attribute information acquisition condition if the attribute information acquisition condition is satisfied.

[0120] Optionally, the third judgment module 5043 includes:

[0121] The first judgment submodule 50431 is configured to obtain attribute information from the attribute file of the directory if the log index of the directory is greater than or equal to the transaction index of the directory.

[0122] The second judgment submodule 50432 is used to obtain attribute information from the directory's memory queue and the directory's attribute file if the directory's log index is smaller than the directory's transaction index and the directory's queue index is greater than or equal to the directory's transaction index.

[0123] Optionally, the fourth determining unit 504 further includes:

[0124] The fourth judgment module 5044 is configured to generate an acquisition abnormality reminder if the duration of the waiting state is longer than a preset duration threshold, where the acquisition abnormality reminder indicates that acquisition of the attribute information of the directory indicated by the acquisition instruction has failed.

[0125] Optionally, the attribute updating apparatus 500 further includes an updating unit 505 including:

[0126] Determination module 5051 is configured to determine, in response to an operation instruction on a directory, directory modification information to be written to a memory queue when a queue write condition is satisfied. The operation instruction indicates an operation to be performed on the directory. The queue write condition indicates the conditions under which modification information needs to be generated and written to the queue based on the directory operation instruction. The modification information indicates modifications to the directory's attribute information.

[0127] The writing module 5052 is used to write the modification information into the memory queue and update the queue index of the memory queue.

[0128] Optionally, the updating unit 505 further includes:

[0129] The update module 5053 is used to call the process of the directory when it is determined that the process of the directory is in an idle state, read the modification information of the directory in the memory queue, and update the read attribute modification to the directory.

[0130] Optionally, the update module 5053 includes:

[0131] The locking submodule 50531 is used to set a read-write lock on the attribute file of the directory.

[0132] The reading submodule 50532 is used to read the attribute information of the directory from the attribute file of the directory.

[0133] The modification submodule 50533 is used to modify the attribute information according to the modification information to obtain the modified attribute information.

[0134] The writing submodule 50534 is used to write the modified attribute information into the attribute file of the directory.

[0135] The unlocking submodule 5053 is used to release the read-write lock of the attribute file of the directory.

[0136] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0138] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0139] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0141] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0142] The computing unit 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for obtaining attribute information of a directory in a distributed file system. For example, in some embodiments, the method for obtaining attribute information of a directory in a distributed file system can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for obtaining attribute information of a directory in a distributed file system described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for obtaining attribute information of a directory of a distributed file system in any other appropriate manner (for example, by means of firmware).

[0143] Figure 7 is a scene diagram of a distributed file system that can implement the embodiment of the present disclosure, such as Figure 7As shown, the distributed file system can obtain external operation requests. The operation request can create directories B, F, etc. It can also insert files BB, FF, etc. Each operation request can correspond to an attribute update. Based on the operation request, the metadata can be inserted into the fragment of the metadata unit. The metadata can be a directory or file included in the operation request. The operation request can also write the modification information of the attribute file of the parent directory in the metadata unit into the memory queue. Process W can obtain the modification information from the memory queue and complete the update of the attributes of the parent directory. In addition, based on the operation request, index information can also be inserted into the index unit.

[0144] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0149] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical servers and VPS services ("Virtual Private Servers" or "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0150] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0151] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for obtaining attribute information of a directory in a distributed file system, comprising: In response to an acquisition instruction, if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory, determining the queue index in the memory queue corresponding to the directory; wherein the acquisition instruction indicates acquisition of attribute information of the directory; When the queue index is greater than or equal to the transaction index, reading the modification information of the attribute of the directory from the memory queue; and determining the attribute information of the directory according to the modification information of the attribute and the attribute file of the directory; The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

2. The method according to claim 1, wherein Determining the queue index in the memory queue corresponding to the directory includes: The index of the modification information at the tail of the memory queue corresponding to the directory is obtained, and the index is used as the queue index.

3. The method according to claim 1, wherein The determining the attribute information of the directory according to the attribute modification information and the attribute file of the directory includes: Merging the modification information of the attributes to obtain summary information; The summary information is combined with the attribute information in the attribute file of the directory to obtain final attribute information of the directory.

4. The method according to any one of claims 1 to 3, further comprising: If it is determined that the log index of the directory indicated by the acquisition instruction is greater than or equal to the transaction index of the directory, the attribute information is acquired from the attribute file of the directory.

5. The method according to any one of claims 1 to 3, further comprising: When the queue index is less than the transaction index, entering a waiting state; The waiting state indicates that the modification information in the memory queue has been read by the process but has not been written into the attribute file; In the waiting state, periodically judging whether the attribute information acquisition condition is satisfied; The attribute information acquisition condition represents a condition for exiting the waiting state and achieving the attribute information acquisition of the directory; If the attribute information acquisition condition is satisfied, the attribute information is output according to the satisfied attribute information acquisition condition.

6. The method according to claim 5, wherein: If the attribute information acquisition condition is satisfied, outputting the attribute information according to the satisfied attribute information acquisition condition includes: If the log index of the directory is greater than or equal to the transaction index of the directory, obtaining the attribute information from the attribute file of the directory; If the log index of the directory is smaller than the transaction index of the directory, and the queue index of the directory is greater than or equal to the transaction index of the directory, the attribute information is obtained from the memory queue of the directory and the attribute file of the directory.

7. The method according to claim 5, wherein: The method further comprises: If the duration of the waiting state is longer than a preset duration threshold, an acquisition abnormality reminder is generated, where the acquisition abnormality reminder indicates that the acquisition of the attribute information of the directory indicated by the acquisition instruction fails.

8. The method according to any one of claims 1 to 3, 6 to 7, further comprising: In response to an operation instruction on the directory, when a queue write condition is satisfied, determining modification information of the directory to be written into the memory queue; wherein the operation instruction indicates an operation on the directory; the queue write condition indicates a condition under which modification information needs to be generated and written into the queue according to the operation instruction on the directory; and the modification information indicates a modification of attribute information of the directory; The modification information is written into the memory queue, and the queue index of the memory queue is updated.

9. The method according to any one of claims 1-3, 6-7, further comprising: When it is determined that the process of the directory is in an idle state, the process of the directory is called, the modification information of the directory in the memory queue is read, and the read modification information is updated to the directory.

10. The method according to claim 9, wherein: Update the read modification information to the directory, including: Set a read-write lock on the attribute file of the directory; Reading attribute information of the directory from the attribute file of the directory; Modify the attribute information according to the modification information to obtain modified attribute information; Writing the modified attribute information into the attribute file of the directory; Release the read-write lock on the properties file of the directory.

11. A device for obtaining attribute information of a directory in a distributed file system, comprising: a first determining unit configured to, in response to an acquisition instruction, determine a queue index in a memory queue corresponding to the directory if it is determined that the log index of the directory indicated by the acquisition instruction is less than the transaction index of the directory; wherein the acquisition instruction indicates acquisition of attribute information of the directory; a second judgment unit, configured to read, when the queue index is greater than or equal to the transaction index, attribute modification information of the directory from the memory queue; and determine attribute information of the directory based on the attribute modification information and the attribute file of the directory; The log index indicates the number of updates to the directory, and the transaction index indicates the number of operations on the directory; the memory queue stores modification information of directory attributes, and the queue index indicates the number of operations on the memory queue.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

14. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.

Images