Attribute updating method, device and equipment of directory of distributed file system and storage medium

Abstract

The disclosure provides an attribute updating method and device of a directory of a distributed file system, equipment and a storage medium, relates to the technical field of computers, in particular to big data and the field of distributed file systems. The specific implementation scheme is as follows: firstly, it can be judged whether the distributed system needs to execute the update of the attribute file of a certain directory through the incremental table. If the attribute file of the directory needs to be updated through the incremental table, it can be further determined that the update information of the attribute file of the directory needs to be updated using the incremental table. Then, the update information can be written into the incremental table. After the merging instruction is triggered, in response to the merging instruction, the update information of one or more directories can be obtained from the incremental table, and the update information of the directories is merged one by one. The updated update information can be used to update the attribute file of the directory.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to big data in computer technology, a distributed file system, and in particular to a method and device for updating attributes of a directory of a distributed file system, and a storage medium. BACKGROUND

[0002] With the increase of data, the distributed file system is more and more widely used. In the use of the distributed file system, there may be concurrent operations on the subdirectories under the same directory. After the operation on the subdirectories under the directory, the attribute file of the directory usually needs to be updated. In order to avoid conflicts when reading and writing the attribute file of the directory, the operations on the subdirectories under the same directory are usually queued and serialized.

[0003] However, the queuing and serialization processing easily leads to the problem of low throughput of the operations on the subdirectories under the same directory of the distributed file system. SUMMARY

[0004] The present disclosure provides a method and device for updating attributes of a directory of a distributed file system, and a storage medium.

[0005] According to a first aspect of the present disclosure, a method for updating attributes of a directory of a distributed file system is provided, comprising:

[0006] When it is determined that the attribute file of the directory of the distributed system needs to update attribute information, update information of the attribute file of the directory is determined, and the update information is written into a preset incremental table; wherein the distributed system comprises a plurality of directories, and the plurality of directories have a parent-child relationship; the update information indicates the modification content of the attribute file of the directory;

[0007] In response to a merge instruction, the update information in the preset incremental table is used to update the attribute file of the directory corresponding to the update information; wherein the merge instruction indicates that the update information in the preset incremental table is used to update the attribute file of the corresponding directory.

[0008] According to a second aspect of the present disclosure, a device for updating attributes of a directory of a distributed file system is provided, comprising:

[0009] The incremental unit is configured to, when it is determined that the attribute file of the directory of the distributed system needs to update attribute information, determine update information of the attribute file of the directory, and write the update information into a preset incremental table; wherein the distributed system comprises a plurality of directories, and the plurality of directories have a parent-child relationship; the update information indicates the modification content of the attribute file of the directory;

[0010] The updating unit is configured to, in response to a merge instruction, update the attribute file of the directory corresponding to the update information in the preset incremental table using the update information in the preset incremental table; wherein the merge instruction indicates that the attribute file of the corresponding directory is updated using the update information in the preset incremental table.

[0011] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0012] at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method of the first aspect.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to perform the method of the first aspect.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0017] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure;

[0021] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure;

[0022] Figure 6 is a schematic block diagram of an electronic device for implementing embodiments of the present disclosure;

[0023] Figure 7 is a scenario diagram of a distributed file system that can implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are presented for the purpose of illustration and description. It is to be understood that the embodiments described herein are merely exemplary and that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. As such, the exemplary embodiments described herein should not be construed as limiting the scope of the present disclosure. In the following description, for purposes of explanation and clarity, descriptions of well-known functions and constructions are omitted so as to not obscure the further disclosure.

[0025] With the increase of data volume, distributed file systems are increasingly widely used. In order to accelerate the speed of path resolution in the distributed file system, the prior art introduces a tree menu component (Dtree) into the distributed file system to accelerate path resolution. However, with the addition of the Dtree, the complexity of operations in the distributed file system is thus increased. In the distributed file system with the added Dtree, the operations performed in response to operation requests need to update both the Dtree and the directory entry (Dentry). The operations across the Dtree and the Dentry are two-phase commit (2PC) transactions.

[0026] When the operation request of a user is for operating a subdirectory under a directory, the update of the Dentry is to update the attribute file of the directory. At this time, the execution of the 2PC transaction increases the possibility of transaction conflict when updating the attribute file of the directory. In particular, for the case of concurrent operations on subdirectories under the same directory, the 2PC transaction needs to frequently read the attribute file of the directory, which further increases the possibility of transaction conflict. The operations can include creating a subdirectory under the directory, deleting a subdirectory, modifying the name of a subdirectory, modifying the permissions of a subdirectory, etc.

[0027] Currently, the transaction conflict is mainly avoided by queuing and serially processing concurrent operations on subdirectories under the same directory. However, in the queuing and serially processing manner of operation requests, the query rate per second (QPS) of the directory for operation requests can only be maintained at the level of several hundred, which is difficult to meet the performance requirements in a high-concurrency scenario.

[0028] To solve the above problems, the disclosure provides a directory attribute updating method of a distributed file system, which can be based on Delta updating to realize the updating of the attribute file of the directory, thereby avoiding transaction conflicts caused by frequent reading and writing of the attribute file of the directory when there are high-concurrency operations of the subdirectories under the same directory. The Delta updating can store the updating information of the metadata in the attribute file of the directory through a delta record, instead of directly updating the metadata by reading and writing the attribute file of the directory. The delta record can actually exist in the form of an incremental table. By periodically updating the accumulated updating information in the incremental table to the attribute file of the corresponding directory, asynchronous updating of the attribute file is realized. At the same time, the updating information that has been merged in the incremental table is deleted to reduce data redundancy and scanning burden.

[0029] In addition, since the existence of the delta record requires scanning the delta record to calculate the correct result when accessing the attribute file, in order to balance performance and accuracy, the disclosure can enable the delta record on demand. That is, Delta record is only enabled when there is a high-frequency contention condition in the directory to avoid unnecessary additional overhead.

[0030] By enabling the Delta record, the disclosure avoids the conflicts that may exist in the reading and writing process of the attribute file of the directory when the 2PC transaction of the distributed file system is executed, significantly improves the concurrent processing capability of the operations of the subdirectories under the same directory, and greatly improves the throughput of the operations of the distributed file system. Moreover, the on-demand delta record mechanism enables the system to dynamically adjust according to the actual contention condition, adapts to different use scenarios, and further improves the overall performance of the distributed file system.

[0031] The disclosure provides a directory attribute updating method, device and equipment of a distributed file system and a storage medium, which are applied to big data in computer technology and a distributed file system to improve the throughput of the operations of the subdirectories under the same directory.

[0032] It should be noted that the data in the embodiment is not for a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in the embodiment comes from a public data set.

[0033] In the technical solution of the disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0034] To enable the reader to more deeply understand the implementation principle of the disclosure, the following Figures 2-4 To Figure 1The illustrated embodiment is further refined.

[0035] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure, as Figure 1 The present disclosure provides a method for updating attributes of a directory of a distributed file system, the method comprising:

[0036] 101、In determining the attribute file of the directory of the distributed system needs to update the attribute information, then determine the update information of the attribute file of the directory, write the update information into the preset incremental table. Wherein, the distributed system includes a plurality of directories, and the plurality of directories have parent-child relationship. The update information indicates the modification content of the attribute file of the directory.

[0037] In this embodiment, it is applied to a distributed file system. The distributed file system can have a tree-shaped directory structure. Each directory in the tree-shaped directory structure can have a parent-child relationship. First, it can be determined whether the distributed system needs to perform the update of the attribute file of a certain directory through the incremental table. If it needs to update the attribute file of the directory through the incremental table, it can further determine the update information of the attribute file of the directory which needs to be updated by using the incremental table. Further, the update information can be written into the incremental table to update the attribute file of the directory using the incremental table. The update information is used to indicate the modification content of the attribute file of the directory. Alternatively, the incremental table can be as shown in Table 1.

[0038] Table 1

[0039] Directory Code Name Subdirectory Code Type Subdirectory Number Change …… 12 / _ATTR1 +3 …… 12 / _ATTR1$TS1 +1 …… 12 / _ATTR1$TS2 -1 …… 12 bar 21 dir …… 12 baz 22 dir ……

[0040] As shown in Table 1, the incremental table can include a plurality of fields. Among them, the fields of directory code, name, subdirectory code, type, etc. are used to indicate the information of the directory or subdirectory, and the fields of subdirectory quantity change, etc. are used to indicate the modification content of the attribute file of the directory. Optionally, in addition to the subdirectory quantity change (links) already exemplified in Table 1, the directory size (size), the permission, the file content last modification time (mtim e ), the file permission last modification time (tim c ), the file permission last modification time (tim e ), etc. can also be included. Among them, the primary key of the incremental table can be set as (pid, ATTR$ts). Pid is the directory code. ATTR is the name. The name can be the name of the attribute file. ts is the timestamp. The timestamp is the timestamp of generating the update information. Or, the timestamp can be the timestamp corresponding to the acquisition time of the operation request. $ is the connector between ATTR and ts. Optionally, the connector can also be replaced with "+", "-", "," and other symbols. Among them, the fields of subdirectory quantity change (links), directory size (size), etc. can correspond to numerical parameters, and the content of the field can be +1, -1, +3, -5, etc. The fields of permission, file content last modification time (mtim e ), file permission last modification time (tim c ), etc. can correspond to non-numerical parameters.

[0041] 102. In response to the merge instruction, the update information in the preset incremental table is used to update the attribute file of the directory corresponding to the update information. Among them, the merge instruction indicates that the update information in the preset incremental table is used to update the attribute file of the corresponding directory.

[0042] In this embodiment, after the merge instruction is triggered, the update of the attribute file of one or more directories corresponding to the merge instruction can be performed in response to the merge execution. Specifically, in response to the merge instruction, the update information of the one or more directories can be obtained from the incremental table, and the update information of the directory is merged one by one. The merged update information can be used to update the attribute file of the directory, thereby improving the update efficiency of the attribute file.

[0043] Optionally, a plurality of merge processes can be set in the distributed file system. Each merge process can be used to update the attribute file of one or more directories. The merge instruction is used to trigger the task of the merge process of one or more directories. Each task corresponds to a directory. When the task is triggered, the merge process will perform the merge of the update information of the directory corresponding to the task and the update of the attribute file of the directory.

[0044] For example, assume that there is a directory C, and there are 5 mkdir operations in a high concurrency environment, which are used to create D1 to D55 subdirectories under directory C. Assume that the second threshold is 5, then the concurrent increment state can be entered. That is, an update information is generated each time the mkdir operation is performed. The update information can be written into the increment table. For example, the update information can be (C_pid, _ATTR_ts1, num+1), (C_pid, _ATTR_ts2, num+1), …, (C_pid, _ATTR_ts5, num+1). These update information, as participants of the 2PC transaction, can update the attribute file after the transaction is executed. Within a set merging period, the distributed file system can scan the update information in the increment table, and merge to obtain the num+5 information, and update the num+5 information to the attribute file of C. The update information is deleted to ensure the consistency of the attribute file and the increment table.

[0045] In the embodiment, by determining the update information of the attribute file of the directory when it is determined that the attribute file of the directory needs to be updated using the increment table, writing the update information into the preset increment table, and responding to the merging instruction, obtaining the update information of the directory indicated by the merging instruction from the increment table, merging the update information of the directory, and updating the merged update information to the attribute file of the directory, the conflict that can be caused when the attribute file of the target is frequently read and written is avoided, the processing capability of the concurrent operation of the subdirectories under the same directory is improved, and the throughput of the distributed file system is improved.

[0046] On the basis of the embodiment, the triggering condition of the merging instruction at least includes the following two conditions:

[0047] First, the merging instruction of the directory is periodically generated according to a preset period of the directory.

[0048] Optionally, each directory can correspond to a preset period. For example, the preset period can be 1s, 10s, etc. According to the preset period, the merging instruction of the directory can be periodically triggered. In response to the merging instruction, the merging task of the directory can be triggered in the merging process corresponding to the directory.

[0049] Second, in response to a user request, if it is detected that the number of update information corresponding to the directory in the increment table is greater than or equal to a third threshold during the traversal of the increment table, the merging instruction of the directory is generated.

[0050] Optionally, if the user requests to read the file attribute of the directory, the incremental table is traversed to obtain the update information corresponding to the file attribute of the directory in the incremental table while reading the file attribute of the directory. According to the file attribute and the update information, the latest information of the file attribute can be obtained, and the latest information is fed back to the user. During the traversal of the incremental table, if it is detected that the number of update information corresponding to the directory is greater than or equal to a third threshold, the merging instruction of the directory can be triggered. Optionally, the directory can be the directory requested to be read by the user. Alternatively, the directory can also be any directory other than the directory requested to be read by the user. In response to the merging instruction, the merging task of the directory can be triggered in the merging process corresponding to the directory. Optionally, if there are multiple merging tasks to be processed in the merging process, the merging task can be advanced to realize the rapid processing of the update information of the directory in the incremental table.

[0051] The generation processes of the two merging instructions are independent of each other.

[0052] Through the generation processes of the two merging instructions, on the basis of ensuring that the update information of the directory is periodically processed, when it is found that the number of update information of the directory is particularly large, the update information of the directory is preferentially merged and processed, the processing efficiency of the update information is improved, and the decrease of the processing efficiency caused by too much update information of a directory is avoided.

[0053] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure, as Figure 2 shown, the present disclosure provides a method for updating the attribute of a directory of a distributed file system. The method is used to implement the step 101 of determining the update information of the attribute file of the directory of the distributed system when it is determined that the attribute file of the directory of the distributed system needs to update the attribute information. The method comprises the following steps:

[0054] 201, when it is determined that the read-write event of the attribute file of the directory of the distributed system conflicts, the update information of the directory is generated according to the operation information of the subdirectory triggering the read-write event. The read-write event represents that the subdirectory requests to read or write the attribute file. The operation information represents the information generated by the operation on the subdirectory.

[0055] In the embodiment, firstly, it can be judged whether the read-write events of the attribute file of the directory of the distributed system conflict. If the conflict occurs, it indicates that the attribute file of the directory of the distributed system needs to be updated by means of the incremental table. Further, according to the operation information of the subdirectory triggering the read-write event, the update information of the attribute file of the directory needing to be written into the incremental table can be determined. The subdirectory is a subdirectory under the directory. The operation information is information generated when the operation corresponding to the operation request of the subdirectory generated by the user is performed. In this way, the conflict of the attribute file in the read-write process can be resolved, so that the concurrent operation of the subdirectory can be better performed, and the throughput of the distributed file system can be improved.

[0056] Specifically, the step 201 can determine the generated update information needing to be written into the incremental table according to the number of conflicts of the attribute file of the directory occurring within the first time length, or the frequency of the conflicts of the attribute file of the directory, and specifically can include the following steps:

[0057] 2011, if the number of conflicts occurring within the first time length is less than or equal to a first threshold value, enter the conflict incremental state of the directory, and the conflict write state represents that the update information of the directory is generated according to the operation information of the subdirectory corresponding to the read-write event of the conflict.

[0058] In the embodiment, the number of conflicts occurring within the first time length before the current time can be counted. If the number of conflicts is less than or equal to the first threshold value, it indicates that there are a small number of conflicts. At this time, the conflict incremental state can be entered. The conflict incremental state can determine the subdirectory of the conflict according to the conflict. Further, according to the operation information of the subdirectory, the update information of the directory is generated and written into the incremental table. The conflict incremental state can reduce the case of writing into the incremental table under the bearing condition of the read-write of the attribute file of the directory, and reduce the calculation power consumption generated when the incremental table is read when the attribute file is requested.

[0059] 2012, if the number of conflicts occurring within the first time length is greater than the first threshold value, enter the first all-incremental state of the directory, and the first all-incremental state represents that the update information of the directory is generated according to the operation information of all subdirectories triggering the read-write event of the attribute file.

[0060] In the embodiment, if the number of conflicts in the first time period before the current time is greater than the first threshold, it is considered that there are a large number of conflicts. At this time, the first full incremental state can be entered. In the first full incremental state, the operation information of all subdirectories under the directory can be directly generated into update information of the directory and written into the incremental table. On the one hand, this processing manner can avoid more conflicts from occurring when conflicts occur frequently, and avoid further expansion of abnormalities when a large number of conflicts occur. On the other hand, the frequent conflicts at this time mean that there are a large number of concurrent operations on the subdirectories under the directory. This processing manner can reduce the calculation power consumed by the statistics of the number of conflicts, and apply the calculation power to the concurrent operations of the subdirectories.

[0061] In summary, through steps 2011 and 2012, the restriction of converting the operation information of the subdirectories into the update information of the directory is realized under different conflict frequencies, and the processing efficiency of the distributed operating system is flexibly improved.

[0062] Optionally, after entering the first full incremental state in step 2012, the following steps can be further included:

[0063] 2013. If the attribute file of the directory has no read-write event in the second time period, the first full incremental state of the directory is exited.

[0064] In the embodiment, the time period in which the attribute file of the directory has no read-write event can be recorded. If the time period reaches the second time period, it is considered that the high-concurrency operation time of the subdirectories of the directory has passed. At this time, the first full incremental state of the directory can be exited. The exit of the first full incremental state can avoid writing the operation of the subdirectory into the incremental table as update information when there are a small number of operations of the subdirectory. This manner can further realize flexible control of writing into the incremental table, and improve the flexibility and efficiency of the attribute file of the directory in the update process of the distributed operating system.

[0065] 202. When it is determined that there are concurrent operations on the subdirectories under the directory of the distributed system, update information of the directory is generated according to operation information of the concurrent operations of the subdirectories. The concurrent operation represents an operation triggered in the same time period. The operation information represents information generated by the operation on the subdirectory.

[0066] In this embodiment, firstly, it can be judged whether there is a concurrent operation on a subdirectory under the directory of the distributed system. Alternatively, the concurrent operation can be specifically an operation on a subdirectory triggered in the same time period. Since the operation on the subdirectory usually needs to update the attribute file of the directory according to the operation, when there is a concurrent operation, the attribute file of the directory can be updated in the same time period due to the concurrent multiple operations, thereby causing a conflict in the update of the attribute file. Therefore, in order to better avoid the occurrence of the conflict, the update information of the directory can be generated according to the operation information of the concurrent operation on the subdirectory. The operation information is information generated when the operation on the subdirectory is performed. The operation can specifically include deleting the subdirectory, adding the subdirectory, modifying the name of the subdirectory, and the like. In this way, the concurrent operation on the subdirectory under the directory can be better optimized and processed, and the conflict in the update of the attribute file by the concurrent operation can be avoided.

[0067] Specifically, the step 202 can determine the update information of the generated increment table according to the number of the concurrent operations on the subdirectory under the directory, and the update information can specifically include the following steps:

[0068] 2021, if it is determined that the number of the concurrent operations on the subdirectory under the directory of the distributed system is greater than or equal to the second threshold value and less than the third threshold value, the concurrent increment state of the directory is entered, and the concurrent increment state indicates that the update information of the directory is generated according to the concurrent operation on the subdirectory under the directory.

[0069] In this embodiment, the number of the concurrent operations on the subdirectory under the directory can be acquired. If the number is greater than or equal to the second threshold value and less than the third threshold value, it indicates that there is a certain amount of concurrency, but not too much. At this time, the concurrent increment state of the directory can be entered. In the concurrent increment state, the update information of the directory can be generated according to the concurrent operation on the subdirectory under the directory and written into the increment table. At this time, if the concurrent operation on the subdirectory under the directory is normally executed, a small amount of conflict in the update process of the attribute file can be caused. Therefore, in order to avoid the conflict, the update information can be generated according to the operation information of the concurrent operation, thereby improving the execution efficiency of the distributed file system on the concurrent operation.

[0070] 2022, if it is determined that the number of the concurrent operations on the subdirectory under the directory of the distributed system is greater than or equal to the third threshold value, the second overall increment state of the directory is entered, and the second overall increment state indicates that the update information of the directory is generated according to the operation on the subdirectory under the directory.

[0071] In this embodiment, if the number of concurrent operations of the subdirectories under the directory is greater than or equal to the third threshold value, it indicates that there are a large number of concurrent operations. At this time, in order to avoid the conflicts that may occur when updating the attribute file of the directory due to a large number of concurrent operations, and further reduce other computing power consumption, the second full incremental state can be entered. In the second full incremental state, the operation information of the operations of the subdirectories under the directory can be all correspondingly generated into update information, and the update information is written into the incremental table. In this way, conflicts can be avoided when updating the attribute file in a high concurrency.

[0072] In summary, through steps 2021 and 2022, the restriction of converting the operation information of the subdirectories into the update information of the directory is realized under different concurrency numbers, and the processing efficiency of the distributed operating system is flexibly improved.

[0073] Optionally, after entering the second full incremental state in step 2022, the following steps can be further included:

[0074] 2023. If no operation of the subdirectories occurs in the directory within the third time length, the second full incremental state of the directory is exited.

[0075] In this embodiment, the time length during which no operation of the subdirectories occurs under the directory can be recorded. If the time length reaches the third time length, it indicates that the high-concurrency operation time of the subdirectories of the directory has passed. At this time, the second full incremental state of the directory can be exited. The exit of the second full incremental state can avoid writing the update information generated by the operation of the subdirectories into the incremental table when there are a small number of operations of the subdirectories. In this way, the flexible control of the writing of the incremental table can be further realized, and the flexibility and efficiency of the attribute file of the directory of the distributed operating system in the updating process can be improved.

[0076] In this embodiment, by generating the update information of the directory according to the operation information of the subdirectories triggering the read-write event when the read-write event of the attribute file conflicts, or generating the update information of the directory according to the operation information of the concurrent operations of the subdirectories when it is determined that there are concurrent operations of the subdirectories under the directory, the control of the generation of the update information and the mechanism of enabling the incremental table on demand are realized. In this way, the system can be dynamically adjusted according to the actual contention situation, adapt to different use scenarios, and flexibly optimize the overall performance.

[0077] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure, as Figure 3 As shown in the figure, the present disclosure provides a directory attribute updating method of a distributed file system, which is used to realize the process of updating the attribute file of the directory by using the update information in the preset incremental table in step 102. The method comprises the following steps:

[0078] 301. Determine whether the update information corresponds to the same directory according to the primary key of the update information in the delta table.

[0079] In this embodiment, the delta table can be as shown in Table 1 in the embodiment shown in the figure. In the delta table, the target code and the name can constitute the primary key of the update information in the delta table. Since the name contains the name of the attribute file and the timestamp, the primary key can be used to determine whether the update information corresponds to the same directory according to the directory code and the name of the attribute file in the name. If the directory code and the name of the attribute file in the primary key of the two pieces of update information are consistent, it means that the two pieces of update information correspond to the same directory. Figure 1

[0080] 302. Merge the update information of the same directory to obtain the final information of the directory.

[0081] In this embodiment, the multiple pieces of update information of the same directory can be merged to obtain the final information of the directory. The merging process can be performed separately for numerical parameters and non-numerical parameters. Alternatively, for numerical parameters, the numerical parameters in the multiple pieces of update information can be accumulated to obtain the numerical parameters in the final information. For non-numerical parameters, the non-numerical parameters in the update information can be iterated according to the timestamp of the update information to obtain the non-numerical parameters in the final information. The timestamp of the update information is the timestamp in the name in the primary key.

[0082] For example, the update information with the prefix (C pid, ATTR) can be obtained by traversing the delta table, and it is determined that these pieces of update information correspond to the directory C. The multiple pieces of update information corresponding to the directory C can be merged. For numerical parameters such as links and size in the directory C, the update information records the addition and subtraction increments of the numerical parameters. At this time, the addition and subtraction increments of the numerical parameters can be accumulated to obtain the final addition and subtraction increments. For non-numerical parameters such as permissions, mtim e 、 c tim e and the like, the update information records the result of the record. At this time, the latest record result can be obtained by simply covering the operation to obtain the final result.

[0083] 303. Write the final information into the attribute file of the directory.

[0084] In this embodiment, after obtaining the final information of the directory, the final information of the directory can be used to update the attribute file of the directory, thereby realizing the merging of the update information of the directory.

[0085] ​In the embodiment, by merging the update information in the incremental table into the attribute file of the directory, the conflict of the attribute file is avoided, the update of the attribute file is realized, and the processing efficiency of the distributed file system for the operation of the subdirectory under the directory is improved.

[0086] Figure 4 is a schematic diagram according to the fourth embodiment of the disclosure, as Figure 4 shown, the disclosure provides an attribute updating device 400 of a directory of a distributed file system, comprising:

[0087] The incremental unit 410 is configured to, when it is determined that the attribute file of the directory of the distributed system needs to update attribute information, determine update information of the attribute file of the directory, and write the update information into a preset incremental table. The distributed system comprises a plurality of directories, and the plurality of directories have a parent-child relationship. The update information indicates the modification content of the attribute file of the directory.

[0088] The update unit 420 is configured to, in response to a merging instruction, update the attribute file of the directory corresponding to the update information by using the update information in the preset incremental table. The merging instruction indicates that the attribute file of the corresponding directory is updated by using the update information in the preset incremental table.

[0089] The device of the embodiment can execute the technical solutions in the above method, and the specific implementation process and technical principles are the same, which will not be repeated here.

[0090] Figure 5 is a schematic diagram according to the fifth embodiment of the disclosure, as Figure 5 shown, the disclosure provides an attribute updating device 500 of a directory of a distributed file system, comprising:

[0091] The incremental unit 510 is configured to, when it is determined that the attribute file of the directory of the distributed system needs to update attribute information, determine update information of the attribute file of the directory, and write the update information into a preset incremental table. The distributed system comprises a plurality of directories, and the plurality of directories have a parent-child relationship. The update information indicates the modification content of the attribute file of the directory.

[0092] The update unit 520 is configured to, in response to a merging instruction, update the attribute file of the directory corresponding to the update information by using the update information in the preset incremental table. The merging instruction indicates that the attribute file of the corresponding directory is updated by using the update information in the preset incremental table.

[0093] Optionally, the incremental unit 510 comprises:

[0094] The conflict increment module 511 is configured to, when determining that the read-write events of the attribute file of the directory of the distributed system conflict, generate the update information of the directory according to operation information of a subdirectory triggering the read-write event. The read-write event represents that the subdirectory requests to read or write the attribute file. The operation information represents information generated by an operation on the subdirectory.

[0095] Optionally, the conflict increment module 511 includes:

[0096] The first conflict increment submodule 5111 is configured to, if the number of conflicts occurring in the first time length is less than or equal to a first threshold, enter a conflict increment state of the directory. The conflict write-in state represents that the update information of the directory is generated according to the operation information of the subdirectory triggering the conflicting read-write event.

[0097] Optionally, the conflict increment module 511 includes:

[0098] The second conflict increment submodule 5112 is configured to, if the number of conflicts occurring in the first time length is greater than the first threshold, enter a first full increment state of the directory. The first full increment state represents that the update information of the directory is generated according to the operation information of all subdirectories triggering the read-write event of the attribute file.

[0099] Optionally, the conflict increment module 511 includes:

[0100] The exit conflict increment submodule 5113 is configured to, if the attribute file of the directory has no read-write event in a second time length, exit the first full increment state of the directory.

[0101] Optionally, the increment unit 510 includes:

[0102] The concurrent increment module 512 is configured to, when determining that there is a concurrent operation on a subdirectory under the directory of the distributed system, generate the update information of the directory according to operation information of the concurrent operation on the subdirectory. The concurrent operation represents an operation triggered in the same time period. The operation information represents information generated by an operation on the subdirectory.

[0103] Optionally, the concurrent increment module 512 includes:

[0104] The first concurrent increment submodule 5121 is configured to, when determining that the number of concurrent operations on the subdirectory under the directory of the distributed system is greater than or equal to a second threshold and less than a third threshold, enter a concurrent increment state of the directory. The concurrent increment state represents that the update information of the directory is generated according to the operation on the concurrent subdirectory under the directory.

[0105] Optionally, the concurrent increment module 512 includes:

[0106] The second concurrent increment sub-module 5122 is configured to: when the number of concurrent operations on the sub-directories under the directory of the distributed system is greater than or equal to the third threshold value, enter a second full increment state of the directory, and the second full increment state is configured to generate the update information of the directory according to the operations on the sub-directories under the directory.

[0107] Optionally, the concurrent increment module 512 includes:

[0108] The exit concurrent increment sub-module 5123 is configured to: if no operation on the sub-directories of the directory occurs within the third time length, exit the second full increment state of the directory.

[0109] Optionally, the update unit 520 includes:

[0110] The query module 521 is configured to: determine the update information corresponding to the same directory according to the primary keys of the update information in the increment table.

[0111] The merge module 522 is configured to: merge the update information of the same directory to obtain the final information of the directory.

[0112] The update module 523 is configured to: write the final information into the attribute file of the directory.

[0113] Optionally, the merge module 522 includes:

[0114] The first merge sub-module 5221 is configured to: accumulate the numerical parameters in the update information to obtain the numerical parameters in the final information.

[0115] The second merge sub-module 5222 is configured to: iterate the non-numerical parameters in the update information according to the time stamps of the update information to obtain the non-numerical parameters in the final information.

[0116] Optionally, the attribute update apparatus 500 further includes:

[0117] The instruction generation unit 530 is configured to: periodically generate the merge instruction of the directory according to a preset period of the directory. Alternatively, in response to a user request, during the traversal of the increment table, if it is detected that the number of the update information corresponding to the directory in the increment table is greater than or equal to the third threshold value, the merge instruction of the directory is generated.

[0118] The apparatus of the embodiment can execute the technical solutions in the above method, and the specific implementation process and technical principles are the same, which will not be described here again.

[0119] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0121] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0123] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the attribute update method of a directory of a distributed file system. For example, in some embodiments, the attribute update method of a directory of a distributed file system can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the attribute update method of a directory of a distributed file system described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the attribute update method of a directory of a distributed file system by any other suitable means, such as by means of firmware.

[0125] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0126] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0129] Figure 7 is a scenario diagram of a distributed file system that can implement embodiments of the present disclosure, as shown in Figure 7 As shown, the distributed file system can include two parts, a tree unit (Dtree) and a directory unit (Dentry). When a subdirectory F needs to be created under a directory C, a node of F needs to be inserted in the tree unit. Also, the attribute of F needs to be updated in the directory unit, and an interface of F needs to be inserted in the parent node C of F and the attribute of the parent node of F needs to be updated. Figure 7 The view in the file system explorer can also be included in the

[0130] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0132] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0133] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for updating the attributes of a directory in a distributed file system, characterized in that: include: When it is determined that the attribute information of the attribute file of a directory of the distributed system needs to be updated, the update information of the attribute file of the directory is determined, and the update information is written into a preset incremental table; wherein the distributed system includes multiple directories, and the multiple directories have a parent-child relationship; the update information indicates the modified content of the attribute file of the directory; In response to a merge instruction, using the update information in the preset incremental table to update the property file of the directory corresponding to the update information; wherein the merge instruction instructs to use the update information in the preset incremental table to update the property file of the corresponding directory; When it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, determining the update information of the attribute file of the directory includes: When it is determined that a conflict occurs in a read / write event of an attribute file of a directory of the distributed system, the update information of the directory is generated according to the operation information of the subdirectory that triggers the read / write event; wherein the read / write event represents that the subdirectory requests to read / write the attribute file; and the operation information represents information generated by the operation on the subdirectory; or When it is determined that there are concurrent operations on sub-directories under the directory of the distributed system, the update information of the directory is generated based on the operation information of the concurrent operations on the sub-directories; the concurrent operations represent operations triggered within the same time period; and the operation information represents information generated by the operations on the sub-directories.

2. The method according to claim 1, wherein When it is determined that a conflict occurs in the read / write events of the attribute file of the directory of the distributed system, generating the update information of the directory according to the operation information of the subdirectory that triggers the read / write event, includes: If the number of conflicts occurring within the first time period is less than or equal to the first threshold, the directory enters a conflict increment state, and the conflict write state represents the generation of update information of the directory based on the operation information of the sub-directory corresponding to the conflicting read and write events.

3. The method according to claim 1, wherein When it is determined that a conflict occurs in the read / write events of the attribute file of the directory of the distributed system, generating the update information of the directory according to the operation information of the subdirectory that triggers the read / write event, includes: If the number of conflicts occurring within the first time period is greater than the first threshold, the directory enters the first full incremental state, where the first full incremental state represents the generation of the update information of the directory based on the operation information of all sub-directories that trigger the read and write events of the attribute file.

4. The method according to claim 3, wherein: The method further comprises: If no read or write event occurs in the attribute file of the directory within the second time period, the first full incremental state of the directory is exited.

5. The method according to claim 1, wherein When it is determined that there is a concurrent operation on a subdirectory under the directory of the distributed system, generating the update information of the directory according to the operation information of the concurrent operation on the subdirectory includes: If it is determined that the number of concurrent operations of the sub-directories under the directory of the distributed system is greater than or equal to the second threshold and less than the third threshold, the directory enters a concurrent incremental state, and the concurrent incremental state represents the generation of update information of the directory based on the concurrent operations of the sub-directories under the directory.

6. The method according to claim 1, wherein When it is determined that there is a concurrent operation on a subdirectory under the directory of the distributed system, generating the update information of the directory according to the operation information of the concurrent operation on the subdirectory includes: If it is determined that the number of concurrent operations of the sub-directories under the directory of the distributed system is greater than or equal to the third threshold, the directory enters the second full incremental state, and the second full incremental state represents the generation of update information of the directory based on the operations of the sub-directories under the directory.

7. The method according to claim 3, wherein: The method further comprises: If no sub-directory operation occurs in the directory within the third time period, the second full incremental state of the directory is exited.

8. The method according to claim 1, wherein The updating of the property file of the directory corresponding to the update information using the update information in the preset incremental table includes: Determining the update information corresponding to the same directory according to the primary key of the update information in the incremental table; Merging the updated information of the same directory to obtain final information of the directory; The final information is written into the property file of the directory.

9. The method according to claim 8, wherein The merging of the update information of the same directory to obtain final information of the directory includes: Accumulating the numerical parameters in the update information to obtain the numerical parameters in the final information; Iterate the non-numeric parameters in the update information according to the timestamp of the update information to obtain the non-numeric parameters in the final information.

10. The method according to any one of claims 1 to 9, wherein The method further comprises: periodically generating the merging instruction for the directory according to a preset period of the directory; or, During traversal of the incremental table in response to a user request, if it is detected that the number of update information corresponding to the directory in the incremental table is greater than or equal to a third threshold, the merge instruction of the directory is generated.

11. A device for updating attributes of a directory in a distributed file system, comprising: An increment unit is configured to, when determining that attribute information of an attribute file of a directory of a distributed system needs to be updated, determine update information of the attribute file of the directory, and write the update information into a preset increment table; wherein the distributed system includes a plurality of directories, and the plurality of directories have a parent-child relationship; and the update information indicates the modified content of the attribute file of the directory; an updating unit, configured to update the property file of the directory corresponding to the update information using the update information in the preset incremental table in response to a merge instruction; wherein the merge instruction instructs to update the property file of the corresponding directory using the update information in the preset incremental table; The incremental unit is specifically used to generate the update information of the directory based on the operation information of the sub-directory that triggers the read and write event when it is determined that a conflict occurs in the read and write events of the attribute file of the directory of the distributed system; wherein the read and write event represents that the sub-directory requests to read and write the attribute file; the operation information represents the information generated by the operation on the sub-directory; or, when it is determined that there are concurrent operations on the sub-directory under the directory of the distributed system, generate the update information of the directory based on the operation information of the concurrent operations of the sub-directory; the concurrent operation represents the operation triggered within the same time period; the operation information represents the information generated by the operation on the sub-directory.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

14. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.

Images