Attribute updating method and device for directory of distributed file system, equipment and storage medium

Abstract

The invention provides an attribute updating method and device for a directory of a distributed file system, equipment and a storage medium, and relates to the technical field of computers, in particular to the field of big data and distributed file systems. According to the specific implementation scheme, firstly, whether the distributed system needs to execute updating of an attribute file of a certain directory in an incremental table mode or not can be judged; and if the attribute file of the directory needs to be updated through the incremental table, updating information needing to update the attribute file of the directory by using the incremental table can be further determined. Further, the update information may be written into an incremental table. After a merging instruction is triggered, in response to the merging instruction, the update information of the one or more directories can be obtained from the incremental table, and the update information of the directories is merged one by one. The merged update information may be used to update attribute files of the directory.

Description

Technical Field

[0001] The present disclosure relates to big data in computer technology and a distributed file system, and in particular to a method, device, equipment and storage medium for updating the attributes of a directory of a distributed file system. Background Art

[0002] As the amount of data increases, the application of distributed file systems is becoming more and more widespread. In the use of distributed file systems, there may be concurrent operations on subdirectories under the same directory. After operating on subdirectories under a directory, the directory's property files usually need to be updated. In order to avoid conflicts when reading and writing directory property files, operations on subdirectories under the same directory are usually queued and serialized.

[0003] However, the queuing and serialization process may easily lead to a low throughput problem when operating subdirectories under the same directory of a distributed file system. Summary of the invention

[0004] The present disclosure provides a method, apparatus, device and storage medium for updating the attributes of a directory of a distributed file system.

[0005] According to a first aspect of the present disclosure, a method for updating attributes of a directory of a distributed file system is provided, comprising:

[0006] When it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, the update information of the attribute file of the directory is determined, and the update information is written into a preset incremental table; wherein the distributed system includes a plurality of directories, and the plurality of directories have a parent-child relationship; the update information indicates the modified content of the attribute file of the directory;

[0007] In response to a merge instruction, the property file of the directory corresponding to the update information is updated using the update information in the preset incremental table; wherein the merge instruction instructs to update the property file of the corresponding directory using the update information in the preset incremental table.

[0008] According to a second aspect of the present disclosure, a device for updating attributes of a directory of a distributed file system is provided, comprising:

[0009] An incremental unit, configured to determine the update information of the attribute file of the directory of the distributed system when it is determined that the attribute information of the attribute file of the directory needs to be updated, and write the update information into a preset incremental table; wherein the distributed system includes a plurality of directories, and the plurality of directories have a parent-child relationship; the update information indicates the modified content of the attribute file of the directory;

[0010] An update unit is used to respond to a merge instruction and use the update information in the preset incremental table to update the property file of the directory corresponding to the update information; wherein the merge instruction indicates to use the update information in the preset incremental table to update the property file of the corresponding directory.

[0011] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0012] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0021] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0022] Figure 6 is a schematic block diagram of an electronic device for implementing an embodiment of the present disclosure;

[0023] Figure 7 It is a scene graph of a distributed file system that can implement the embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] With the increase of data volume, the application of distributed file systems is becoming more and more extensive. In order to speed up the resolution of paths in distributed file systems, the prior art has introduced a tree menu component (Dtree) in distributed file systems to speed up path resolution. However, with the addition of Dtree, the complexity of operations in distributed file systems has increased. In a distributed file system with Dtree added, in response to the operation performed by the operation request, both Dtree and directory entry (Dentry) need to be updated. This operation across Dtree and Dentry is a two-phase commit (2PC) transaction.

[0026] When the user's operation request is used to operate a subdirectory under a directory, the update of Dentry is to update the property file of the directory. At this time, the execution of the 2PC transaction will increase the possibility of transaction conflict when updating the property file of the directory. In particular, in the case of concurrent operations on subdirectories under the same directory, the 2PC transaction needs to frequently read the property file of the directory, which will further increase the possibility of transaction conflict. Among them, the operation may include creating a subdirectory, deleting a subdirectory, modifying the name of a subdirectory, modifying the permissions of a subdirectory, etc. under the directory.

[0027] Currently, transaction conflicts are mainly avoided by queuing and serializing concurrent operations of subdirectories under the same directory. However, when queuing and serializing operation requests, the query rate per second (QPS) of the directory's operation request throughput can only be maintained at a few hundred, which is difficult to meet the performance requirements in high-concurrency scenarios.

[0028] To solve the above problems, the present disclosure proposes a method for updating the attributes of a directory of a distributed file system. The method can implement the update of the attribute file of the directory based on incremental (Delta) updates, thereby avoiding transaction conflicts caused by frequent reading and writing of the attribute file of the directory when there are high-concurrency operations of sub-directories under the same directory. The Delta update can store the update information of the metadata in the attribute file of the directory through delta records, rather than directly updating the metadata by reading and writing the attribute file of the directory. The delta record can actually exist in the form of an incremental table. By periodically merging and updating the accumulated update information in the incremental table into the attribute file of the corresponding directory, asynchronous updates to the attribute file are implemented. At the same time, the merged update information in the incremental table is deleted to reduce data redundancy and scanning burden.

[0029] In addition, since the existence of delta records requires scanning delta records to calculate the correct result when accessing the attribute file, in order to balance performance and accuracy, the present disclosure can enable delta records on demand, that is, enable delta records only when there is a high frequency of contention in the directory to avoid unnecessary additional overhead.

[0030] The present disclosure avoids possible conflicts in the reading and writing of directory attribute files during the execution of 2PC transactions of the distributed file system by enabling Delta records, significantly improves the concurrent processing capability of operations on subdirectories under the same directory, and greatly improves the throughput of distributed file system operations. In addition, the delta record mechanism is enabled on demand, so that the system can be dynamically adjusted according to the actual contention situation, adapt to different usage scenarios, and further improve the overall performance of the distributed file system.

[0031] The present invention discloses a method, device, equipment and storage medium for updating the attributes of a directory of a distributed file system, which are applied to big data and distributed file systems in computer technology to improve the throughput of operations on subdirectories under the same directory.

[0032] It should be noted that the data in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0033] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2-Figure 4 right Figure 1The illustrated embodiment is further refined.

[0035] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, such as Figure 1 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method comprising:

[0036] 101. When it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, the update information of the attribute file of the directory is determined, and the update information is written into a preset incremental table. The distributed system includes multiple directories, and the multiple directories have a parent-child relationship. The update information indicates the modification content of the attribute file of the directory.

[0037] In this embodiment, it is applied to a distributed file system. The distributed file system may have a tree-shaped directory structure. The directories in the tree-shaped directory structure may have a parent-child relationship. First, it may be determined whether the distributed system needs to update the property file of a certain directory by means of an incremental table. If it is necessary to update the property file of the directory by means of an incremental table, it may be further determined that the update information for updating the property file of the directory using the incremental table is required. Furthermore, the update information may be written into the incremental table to update the property file of the directory using the incremental table. The update information is used to indicate the modified content of the property file of the directory. Optionally, the incremental table may be as shown in Table 1.

[0038] Table 1

[0039] Catalog Code name Subdirectory Encoding type Subdirectory quantity changes …… 12 / _ATTR1 +3 …… 12 / _ATTR1$TS1 +1 …… 12 / _ATTR1$TS2 -1 …… 12 bar 21 dir …… 12 baz 22 dir ……

[0040] As shown in Table 1, the incremental table may include multiple fields. Among them, fields such as directory code, name, subdirectory code, type, etc. are used to indicate the information of the directory or subdirectory, and fields such as subdirectory quantity change are used to indicate the modification content of the directory attribute file. Optionally, in addition to the subdirectory quantity change (links) already exemplified in Table 1, directory size (size), permissions, file content last modification time (mtime), file permissions last modification time (ctime), etc. may also be included. Among them, the primary key of the incremental table may be set to (pid, ATTR$ts). Pid is the directory code. ATTR is the name. The name may be the name of the attribute file. ts is the timestamp. The timestamp is the timestamp for generating the update information. Alternatively, the timestamp may be the timestamp corresponding to the acquisition time of the operation request. $ is the connector between ATTR and ts. Optionally, the connector may also be replaced by other symbols such as "+", "-", "," etc. Among them, the fields such as the change in the number of subdirectories (links) and the size of the directory (size) can correspond to numerical parameters, and the content of the field can be +1, -1, +3, -5, etc. The last modification time of the permissions and file content (mtim e ), the last modification time of file permissions ( c Tim e ) can correspond to non-numeric parameters.

[0041] 102. In response to the merge instruction, update the property file of the directory corresponding to the update information using the update information in the preset incremental table. The merge instruction indicates to update the property file of the corresponding directory using the update information in the preset incremental table.

[0042] In this embodiment, after the merge instruction is triggered, in response to the merge execution, the property file of one or more directories corresponding to the merge instruction may be updated. Specifically, in response to the merge instruction, the update information of the one or more directories may be obtained from the incremental table, and the update information of the directory may be merged directory by directory. The merged update information may be used to update the property file of the directory, thereby improving the update efficiency of the property file.

[0043] Optionally, multiple merge processes may be provided in the distributed file system. Each merge process may be used to update the property files of one or more directories. The merge instruction is used to trigger the task of the merge process of one or more directories. Each task corresponds to a directory. When the task is triggered, the merge process will execute the merge of the update information of the directory corresponding to the task, and the update of the property files of the directory.

[0044] For example, suppose there is a directory C, and there are 5 mkdir operations in a high-concurrency environment, which are used to create 5 subdirectories D1 to D5 under directory C. Assuming that the second threshold is 5, the concurrent incremental state can be entered. That is, each mkdir operation generates an update information. The update information can be written into the incremental table. For example, the update information can be (C_pid, _ATTR_ts1, num+1), (C_pid, _ATTR_ts2, num+1), ..., (C_pid, _ATTR_ts5, num+1). As participants in the 2PC transaction, these update information can update the attribute file after the transaction is executed. Within the set merge cycle, the distributed file system can scan the update information in the incremental table, merge the information of num+5, and update the information of num+5 to the attribute file of C. Delete these update information to ensure the consistency between the incremental table and the attribute file.

[0045] In this embodiment, when it is determined that an incremental table is needed to be used to update the attribute file of a directory, the update information of the attribute file of the directory is determined by modifying the content of the attribute file of the directory; the update information is written into a preset incremental table; in response to a merge instruction, the update information of the directory indicated by the merge instruction is obtained from the incremental table; the update information of the directory is merged, and the merged update information is updated into the attribute file of the directory, thereby avoiding conflicts that may arise when the target attribute file is frequently read and written, thereby improving the processing capability of concurrent operations of sub-directories under the same directory and improving the throughput of the distributed file system.

[0046] Based on the embodiment, the triggering conditions of the merge instruction include at least the following two:

[0047] First, according to a preset cycle of the directory, a directory merging instruction is periodically generated.

[0048] Optionally, each directory may correspond to a preset period. For example, the preset period may be once every 1 second, once every 10 seconds, etc. According to the preset period, a merge instruction of the directory may be periodically triggered. In response to the merge instruction, a merge task of the directory may be triggered in the merge process corresponding to the directory.

[0049] Second, in response to a user request, during traversal of the incremental table, if it is detected that the number of update information corresponding to the directory in the incremental table is greater than or equal to a third threshold, a directory merging instruction is generated.

[0050] Optionally, if the user requests to read the file attributes of a directory, the incremental table will be traversed while reading the file attributes of the directory to obtain the update information corresponding to the file attributes of the directory in the incremental table. Based on the file attributes and the update information, the latest information of the file attributes can be obtained, and the latest information can be fed back to the user. Wherein, in the process of traversing the incremental table, if it is detected that the number of update information corresponding to the directory is greater than or equal to the third threshold, the merge instruction of the directory can be triggered. Optionally, the directory can be the directory requested to be read by the user. Alternatively, the directory can also be any directory other than the directory requested to be read by the user. In response to the merge instruction, a merge task of the directory can be triggered in the merge process corresponding to the directory. Optionally, if there are multiple merge tasks to be processed in the merge process, the merge task can be advanced to achieve rapid processing of the update information of the directory in the incremental table.

[0051] The generation processes of the above two merge instructions are independent of each other.

[0052] Through the generation process of the above two merge instructions, on the basis of ensuring the periodic processing of the directory update information, when it is found that the number of directory update information is particularly large, it is preferentially merged to improve the processing efficiency of the update information and avoid the reduction of processing efficiency due to too much update information in a directory.

[0053] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure, such as Figure 2 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, the method is used to implement the above step 101, when it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, to determine the update information of the attribute file of the directory. The method includes:

[0054] 201. When it is determined that a conflict occurs in the read / write event of the attribute file of the directory of the distributed system, the directory update information is generated according to the operation information of the subdirectory that triggers the read / write event. The read / write event indicates that the subdirectory requests to read / write the attribute file. The operation information indicates the information generated by the operation on the subdirectory.

[0055] In this embodiment, it is first possible to determine whether a conflict occurs in the read and write events of the attribute files of the directory of the distributed system. If a conflict occurs, it means that the attribute files of the directory of the distributed system need to be updated by means of an incremental table. Further, based on the operation information of the subdirectory that triggers the read and write event, the update information of the attribute files of the directory that needs to be written into the incremental table can be determined. Among them, the subdirectory is a subdirectory under the directory. Among them, the operation information is the information generated when the operation corresponding to the operation request of the subdirectory generated by the user is executed. In this way, the conflict of the attribute files in the reading and writing process can be resolved, so that the concurrent operations of the subdirectories can be better executed, and the throughput of the distributed file system can be improved.

[0056] Specifically, the above step 201 may determine the generated update information that needs to be written into the incremental table according to the number of conflicts in the attribute files of the directory within the first time period, or the frequency of conflicts in the attribute files of the directory, and may specifically include the following steps:

[0057] 2011. If the number of conflicts occurring within the first time period is less than or equal to the first threshold, the directory enters a conflict increment state. The conflict write state represents that update information of the directory is generated according to operation information of the subdirectory corresponding to the conflicting read and write events.

[0058] In this embodiment, the number of conflicts that occurred within the first time period before the current moment can be counted. If the number of conflicts is less than or equal to the first threshold, it means that there are a small number of conflicts. At this time, the conflict incremental state can be entered. The conflict incremental state can determine the subdirectory in which the conflict occurs based on the conflict that occurs. Furthermore, based on the operation information of the subdirectory, update information for the directory is generated, and the update information is written to the incremental table. The conflict incremental state can reduce the situation where the incremental table is written when the attribute file of the directory can be read and written, and reduce the computing power consumption generated by reading the incremental table when requesting the attribute file.

[0059] 2012. If the number of conflicts occurring within the first time period is greater than the first threshold, the directory enters a first full incremental state, where the first full incremental state represents generating directory update information based on operation information of all subdirectories that trigger read and write events of the attribute file.

[0060] In this embodiment, if statistics show that the number of conflicts within the first time period before the current moment is greater than the first threshold, it means that there are a large number of conflicts. At this time, the first full incremental state can be entered. In this first full incremental state, the operation information of all sub-directories under the directory can be directly used to generate update information for the directory and write it into the incremental table. This processing method, on the one hand, can avoid the occurrence of more conflicts when conflicts occur frequently, and avoid further expansion of anomalies after a large number of conflicts occur. On the other hand, the frequent conflicts at this time mean that there are a large number of concurrent operations on sub-directories under the directory. This processing method can reduce the computing power consumed by the statistics of the number of conflicts, and apply the computing power to the concurrent operations of sub-directories.

[0061] In summary, through the above steps 2011 and 2012, the restriction on converting the operation information of the sub-directory into the update information of the directory under different conflict frequencies is achieved, and the processing efficiency of the distributed operating system is flexibly improved.

[0062] Optionally, after entering the first full increment state in step 2012, the following steps may also be included:

[0063] 2013. If no read or write event occurs in the attribute file of the directory within the second time period, the first full incremental state of the directory is exited.

[0064] In this embodiment, the duration of no read or write events of the attribute file of the directory can be recorded. If the duration reaches the second duration, it means that the high concurrent operation time of the subdirectories of the directory has passed. At this time, the first full incremental state of the directory can be exited. The exit of the first full incremental state can avoid writing the update information generated by the operation of the subdirectory into the incremental table when there are a small number of subdirectories. This method can further realize flexible control of the writing of the incremental table, and improve the flexibility and efficiency of the attribute file of the directory of the distributed operating system during the update process.

[0065] 202. When it is determined that there are concurrent operations on subdirectories under the directory of the distributed system, update information of the directory is generated according to operation information of the concurrent operations on the subdirectories. The concurrent operations represent operations triggered within the same time period. The operation information represents information generated by the operations on the subdirectories.

[0066] In this embodiment, first, it can be determined whether there are concurrent operations on subdirectories under the directory of the distributed system. Optionally, the concurrent operation can specifically be an operation on the subdirectory that is triggered within the same time period. Since when operating on a subdirectory, it is usually necessary to update the property file of the directory according to the operation. When there are concurrent operations, the property file of the directory may be updated within the same time period due to multiple concurrent operations, thereby causing conflicts in the update of the property file. Therefore, in order to better avoid the occurrence of conflicts, the update information of the directory can be generated according to the operation information of the concurrent operations of the subdirectory. The operation information is the information generated when operating on the subdirectory. The operation can specifically include deleting a subdirectory, adding a new subdirectory, modifying the name of the subdirectory, etc. In this way, it is possible to better optimize the concurrent operations of the subdirectories under the directory to avoid conflicts when the property files are updated by these concurrent operations.

[0067] Specifically, the above step 202 may determine the generated update information that needs to be written into the incremental table according to the number of concurrent operations on the subdirectories under the directory, and may specifically include the following steps:

[0068] 2021. If it is determined that the number of concurrent operations of the sub-directories under the directory of the distributed system is greater than or equal to the second threshold and less than the third threshold, the directory enters a concurrent incremental state. The concurrent incremental state represents the generation of directory update information based on the concurrent operations of the sub-directories under the directory.

[0069] In this embodiment, the number of concurrent operations of the subdirectories under the directory can be obtained. If the number is greater than or equal to the second threshold value and less than the third threshold value, it means that there is a certain amount of concurrency, but not very much. At this time, the concurrent incremental state of the directory can be entered. In this concurrent incremental state, the operations of the subdirectories under the concurrent directory can be used to generate update information of the directory and write it into the incremental table. At this time, if the concurrent operations of the subdirectories under the directory are performed normally, conflicts may occur in the update process of a small number of attribute files. Therefore, in order to avoid conflicts, update information can be generated based on the operation information of the concurrent operations, thereby improving the execution efficiency of the distributed file system for the concurrent operations.

[0070] 2022. If it is determined that the number of concurrent operations of the subdirectories under the directory of the distributed system is greater than or equal to the third threshold, the directory enters a second full incremental state, where the second full incremental state represents generating update information of the directory based on operations of the subdirectories under the directory.

[0071] In this embodiment, if the number of concurrent operations of the subdirectories under the directory is greater than or equal to the third threshold, it means that there are a large number of concurrent operations. At this time, in order to avoid conflicts that may arise when a large number of concurrent operations update the property files of the directory and further reduce other computing power losses, the second full incremental state can be entered. In this second full incremental state, the operation information of the operations of the subdirectories under the directory can all be generated corresponding to update information, and the update information can be written into the incremental table. This processing method can avoid conflicts when updating property files in high concurrency.

[0072] In summary, through the above steps 2021 and 2022, the restriction on converting the operation information of the sub-directory into the update information of the directory under different concurrency numbers is achieved, and the processing efficiency of the distributed operating system is flexibly improved.

[0073] Optionally, after entering the second full increment state in step 2022, the following steps may also be included:

[0074] 2023. If no sub-directory operation occurs in the directory within the third time period, exit the second full incremental state of the directory.

[0075] In this implementation, the duration of time when no sub-directory operations occur under the directory can be recorded. If the duration reaches the third duration, it means that the high concurrent operation time of the sub-directories of the directory has passed. At this time, the second full incremental state of the directory can be exited. The exit from the second full incremental state can avoid writing the update information generated by the operation of the sub-directory into the incremental table when there are a small number of sub-directories. This method can further realize flexible control of the writing of the incremental table, and improve the flexibility and efficiency of the attribute file of the directory of the distributed operating system during the update process.

[0076] In this embodiment, when a conflict occurs in the read and write events of the attribute files, the update information of the directory is generated according to the operation information of the sub-directory that triggers the read and write events, or when it is determined that there are concurrent operations on the sub-directories under the directory, the update information of the directory is generated according to the operation information of the concurrent operations of the sub-directories, thereby controlling the generation of update information and enabling the mechanism of incremental tables on demand, so that the system can dynamically adjust according to the actual contention situation, adapt to different usage scenarios, and flexibly optimize the overall performance.

[0077] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure, Figure 3 As shown, the present disclosure provides a method for updating the attributes of a directory of a distributed file system, which is used to implement the process of updating the attribute file of the directory using the update information in the preset incremental table in the above step 102. The method includes:

[0078] 301. Determine the update information corresponding to the same directory according to the primary key of the update information in the incremental table.

[0079] In this embodiment, the increment table can be as follows Figure 1 As shown in Table 1 of the illustrated embodiment. In the incremental table, the target code and the name can constitute the primary key of the update information in the incremental table. Since the name contains two parts, the name of the attribute file and the timestamp. Therefore, it is possible to determine whether the update information corresponds to the same directory based on the two parts, the directory code in the primary key and the name of the attribute file in the name. If the two parts, the directory code in the primary key and the name of the attribute file in the name, of the two update information are consistent, it means that the two update information correspond to the same directory.

[0080] 302. Merge the update information of the same directory to obtain the final information of the directory.

[0081] In this embodiment, multiple update information of the same directory can be merged to obtain the final information of the directory. The merging process can be processed separately for numerical parameters and non-numerical parameters. Optionally, for numerical parameters, the numerical parameters in multiple update information can be accumulated to obtain the numerical parameters in the final information. For non-numerical parameters, the non-numerical parameters in the update information can be iterated according to the timestamp of the update information to obtain the non-numerical parameters in the final information. The timestamp of the update information is the timestamp in the name in the primary key.

[0082] For example, by traversing the incremental table, we can obtain update information with a prefix of (C pid, ATTR), and determine that these update information all correspond to directory C. We can merge multiple update information corresponding to directory C. For numerical parameters such as links and size, the update information records the addition and subtraction increments of the numerical parameters. At this time, we can accumulate the addition and subtraction increments of the numerical parameters to obtain the final addition and subtraction increments. For permissions, mtim e , c Tim e For non-numeric parameters such as , the update information records the result of the record. At this time, the result of the latest record can be taken as the final result through a simple overwrite operation.

[0083] 303. Write the final information to the directory's property file.

[0084] In this embodiment, after obtaining the final information of the directory, the final information of the directory may be used to update the property file of the directory, thereby realizing the merging of the updated information of the directory.

[0085] In this embodiment, by merging the update information in the incremental table into the property file of the directory, the conflict of the property file is avoided, the property file is updated, and the processing efficiency of the distributed file system for sub-directory operations under the directory is improved.

[0086] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure, Figure 4 As shown, the present disclosure provides a device 400 for updating the attributes of a directory of a distributed file system, comprising:

[0087] The incremental unit 410 is used to determine the update information of the attribute file of the directory when it is determined that the attribute information of the attribute file of the distributed system needs to be updated, and write the update information into a preset incremental table. The distributed system includes multiple directories, and the multiple directories have a parent-child relationship. The update information indicates the modified content of the attribute file of the directory.

[0088] The updating unit 420 is used to update the property file of the directory corresponding to the update information using the update information in the preset incremental table in response to the merge instruction. The merge instruction indicates to update the property file of the corresponding directory using the update information in the preset incremental table.

[0089] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0090] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 5 As shown, the present disclosure provides a device 500 for updating the attributes of a directory of a distributed file system, comprising:

[0091] The incremental unit 510 is used to determine the update information of the attribute file of the directory when it is determined that the attribute information of the attribute file of the distributed system needs to be updated, and write the update information into a preset incremental table. The distributed system includes multiple directories, and the multiple directories have a parent-child relationship. The update information indicates the modified content of the attribute file of the directory.

[0092] The updating unit 520 is used to update the property file of the directory corresponding to the update information using the update information in the preset incremental table in response to the merge instruction. The merge instruction indicates to update the property file of the corresponding directory using the update information in the preset incremental table.

[0093] Optionally, the increment unit 510 includes:

[0094] The conflict increment module 511 is used to generate directory update information according to the operation information of the subdirectory that triggers the read / write event when it is determined that the read / write event of the attribute file of the directory of the distributed system conflicts. The read / write event represents the subdirectory requesting to read / write the attribute file. The operation information represents the information generated by the operation on the subdirectory.

[0095] Optionally, the conflict increment module 511 includes:

[0096] The first conflict increment submodule 5111 is used to enter the conflict increment state of the directory if the number of conflicts occurring within the first time period is less than or equal to the first threshold. The conflict write state represents the generation of directory update information based on the operation information of the subdirectory corresponding to the conflicting read and write events.

[0097] Optionally, the conflict increment module 511 includes:

[0098] The second conflict incremental submodule 5112 is used to enter the first full incremental state of the directory if the number of conflicts occurring within the first time period is greater than the first threshold. The first full incremental state represents the generation of directory update information based on the operation information of all subdirectories that trigger the read and write events of the attribute file.

[0099] Optionally, the conflict increment module 511 includes:

[0100] The conflict-exit incremental submodule 5113 is used to exit the first full incremental state of the directory if no read or write event occurs in the attribute file of the directory within the second time period.

[0101] Optionally, the increment unit 510 includes:

[0102] The concurrent increment module 512 is used to generate directory update information based on the operation information of the concurrent operation of the subdirectory when it is determined that there is a concurrent operation on the subdirectory under the directory of the distributed system. The concurrent operation represents the operation triggered in the same time period. The operation information represents the information generated by the operation on the subdirectory.

[0103] Optionally, the concurrent increment module 512 includes:

[0104] The first concurrent incremental submodule 5121 is used to enter the concurrent incremental state of the directory if it is determined that the number of concurrent operations of the subdirectories under the directory of the distributed system is greater than or equal to the second threshold and less than the third threshold. The concurrent incremental state represents the generation of directory update information based on the concurrent operations of the subdirectories under the directory.

[0105] Optionally, the concurrent increment module 512 includes:

[0106] The second concurrent incremental submodule 5122 is used to enter the second full incremental state of the directory when the number of concurrent operations on the subdirectories under the directory of the distributed system is greater than or equal to the third threshold. The second full incremental state represents the generation of directory update information based on the operations on the subdirectories under the directory.

[0107] Optionally, the concurrent increment module 512 includes:

[0108] The concurrent incremental exit submodule 5123 is used to exit the second full incremental state of the directory if no sub-directory operation occurs in the directory within the third time period.

[0109] Optionally, the updating unit 520 includes:

[0110] The query module 521 is used to determine the update information corresponding to the same directory according to the primary key of the update information in the incremental table.

[0111] The merging module 522 is used to merge the update information of the same directory to obtain the final information of the directory.

[0112] The updating module 523 is used to write the final information into the attribute file of the directory.

[0113] Optionally, the merging module 522 includes:

[0114] The first merging submodule 5221 is used to accumulate the numerical parameters in the update information to obtain the numerical parameters in the final information.

[0115] The second merging submodule 5222 is used to iterate the non-numeric parameters in the update information according to the timestamp of the update information to obtain the non-numeric parameters in the final information.

[0116] Optionally, the attribute updating device 500 further includes:

[0117] The instruction generation unit 530 is used to periodically generate a directory merge instruction according to a preset cycle of the directory. Alternatively, in response to a user request, during the traversal of the incremental table, if it is detected that the number of update information corresponding to the directory in the incremental table is greater than or equal to a third threshold, then generate a directory merge instruction.

[0118] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0121] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0122] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0123] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a method for updating the attributes of a directory of a distributed file system. For example, in some embodiments, the method for updating the attributes of a directory of a distributed file system may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for updating the attributes of a directory of a distributed file system described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the attribute update method of the directory of the distributed file system in any other appropriate manner (for example, by means of firmware).

[0125] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] Figure 7 is a scene graph of a distributed file system that can implement the embodiments of the present disclosure, such as Figure 7 As shown, the distributed file system may include two parts: a tree unit (Dtree) and a directory unit (Dentry). When a subdirectory F needs to be created under directory C, the node of F needs to be inserted into the tree unit. Also, the attributes of F need to be updated in the directory unit, and the interface of F needs to be inserted into the parent node C of F and the parent node attributes of F need to be updated. Figure 7 You can also include views in a file system browser.

[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for updating the attributes of a directory in a distributed file system, characterized in that: include: When it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, the update information of the attribute file of the directory is determined, and the update information is written into a preset incremental table; wherein the distributed system includes a plurality of directories, and the plurality of directories have a parent-child relationship; the update information indicates the modified content of the attribute file of the directory; In response to a merge instruction, the property file of the directory corresponding to the update information is updated using the update information in the preset incremental table; wherein the merge instruction instructs to update the property file of the corresponding directory using the update information in the preset incremental table.

2. The method according to claim 1, wherein: When it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, determining the update information of the attribute file of the directory includes: When it is determined that a conflict occurs in the read and write events of the attribute file of the directory of the distributed system, the update information of the directory is generated according to the operation information of the sub-directory that triggers the read and write events; wherein the read and write events represent that the sub-directory requests to read and write the attribute file; and the operation information represents the information generated by the operation on the sub-directory.

3. The method according to claim 2, wherein: When it is determined that a conflict occurs in the read / write events of the attribute file of the directory of the distributed system, the update information of the directory is generated according to the operation information of the subdirectory that triggers the read / write event, including: If the number of conflicts occurring within the first time period is less than or equal to the first threshold, the directory enters a conflict increment state, and the conflict write state represents generating the update information of the directory based on the operation information of the subdirectory corresponding to the conflicting read and write events.

4. The method according to claim 2, wherein: When it is determined that a conflict occurs in the read / write events of the attribute file of the directory of the distributed system, the update information of the directory is generated according to the operation information of the subdirectory that triggers the read / write event, including: If the number of conflicts occurring within the first time period is greater than a first threshold, the directory enters a first full incremental state, wherein the first full incremental state represents generation of update information of the directory based on operation information of all sub-directories that trigger the read and write events of the attribute file.

5. The method according to claim 4, wherein: The method further comprises: If no read or write event occurs in the attribute file of the directory within the second time period, the first full incremental state of the directory is exited.

6. The method according to claim 1, wherein: When it is determined that the attribute file of the directory of the distributed system needs to update the attribute information, determining the update information of the attribute file of the directory includes: When it is determined that there are concurrent operations on sub-directories under the directory of the distributed system, the update information of the directory is generated according to the operation information of the concurrent operations on the sub-directories; the concurrent operations represent operations triggered within the same time period; and the operation information represents information generated by the operations on the sub-directories.

7. The method according to claim 6, wherein: When it is determined that there is a concurrent operation on a subdirectory under the directory of the distributed system, generating the update information of the directory according to the operation information of the concurrent operation of the subdirectory includes: If it is determined that the number of concurrent operations of the sub-directories under the directory of the distributed system is greater than or equal to the second threshold and less than the third threshold, the directory enters a concurrent incremental state, and the concurrent incremental state represents the generation of update information of the directory based on the concurrent operations of the sub-directories under the directory.

8. The method according to claim 6, wherein: When it is determined that there is a concurrent operation on a subdirectory under the directory of the distributed system, generating the update information of the directory according to the operation information of the concurrent operation of the subdirectory includes: If it is determined that the number of concurrent operations of the subdirectories under the directory of the distributed system is greater than or equal to the third threshold, the directory enters the second full incremental state, and the second full incremental state represents the generation of update information of the directory based on the operations of the subdirectories under the directory.

9. The method according to claim 4, wherein: The method further comprises: If no sub-directory operation occurs in the directory within the third time period, the second full incremental state of the directory is exited.

10. The method according to any one of claims 1 to 9, wherein: The updating the property file of the directory corresponding to the update information using the update information in the preset incremental table includes: Determining the update information corresponding to the same directory according to the primary key of the update information in the incremental table; Merging the update information of the same directory to obtain final information of the directory; The final information is written to the properties file of the directory.

11. The method according to claim 10, wherein: The merging of the update information of the same directory to obtain final information of the directory includes: Accumulating the numerical parameters in the update information to obtain the numerical parameters in the final information; According to the timestamp of the update information, the non-numeric parameters in the update information are iterated to obtain the non-numeric parameters in the final information.

12. The method according to any one of claims 1 to 11, wherein: The method further comprises: periodically generating the merging instruction of the directory according to a preset period of the directory; or, During traversal of the incremental table in response to a user request, if it is detected that the number of update information corresponding to the directory in the incremental table is greater than or equal to a third threshold, the merge instruction of the directory is generated.

13. A device for updating the attributes of a directory of a distributed file system, comprising: An incremental unit, configured to determine the update information of the attribute file of the directory of the distributed system when it is determined that the attribute information of the attribute file of the directory needs to be updated, and write the update information into a preset incremental table; wherein the distributed system includes a plurality of directories, and the plurality of directories have a parent-child relationship; the update information indicates the modified content of the attribute file of the directory; An update unit is used to respond to a merge instruction and use the update information in the preset incremental table to update the property file of the directory corresponding to the update information; wherein the merge instruction indicates to use the update information in the preset incremental table to update the property file of the corresponding directory.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

16. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 12 when executed by a processor.

Images